How does an AI SRE read OpenTelemetry log records while investigating a cron job that stopped?

Tares receives OpenTelemetry log records through its otlp connector and labels them by service, so an AI SRE reads the cron job’s log lines on the same timeline as the alert. You create one otlp source, point your exporter at Tares, and the agent reads the service’s logs and alerts on one clock. The agent can be a Tares agent woken by a trigger or your own MCP client investigating on demand.

How it works

The otlp connector receives OTLP/HTTP logs, traces and metrics at /v1/logs, /v1/traces and /v1/metrics. One source ingests every service. Resource attributes such as service.name become labels, so whatever an OpenTelemetry SDK or Collector exports correlates by service with no per-service setup.

The label is what ties the pieces together. A label reads its value from a path in the event:

• resourceAttributes.<key> for resource attributes, for example resourceAttributes.service.name.
• attributes.<key> for the attributes of a log record, span or data point.
• text for the rendered log line.

There is no bare service field, so a label written as field: service gets no value from an OTLP event. Use resourceAttributes.service.name as the primary label.

In an AI SRE setup, your logs, metrics and alerts all carry the same service label. Tares merges them into one timeline per service. An alert source such as prometheus_alerts or alertmanager brings in the alert. A trigger keyed by service fires when the alert event arrives and wakes an agent. That agent reads the correlated timeline, so the cron job’s log lines and the alert sit side by side on one clock. You can compare what the job logged last against the time the alert fired. The agent writes its conclusion back to the service’s timeline, and the next investigation starts from it.

Set it up

1. Make sure the cron job’s OpenTelemetry resource sets service.name. That value becomes the service label.
2. Start Tares locally and open the console.

tares up

3. Create the source under Sources → Add source → otlp, or ask a connected agent: “Use tares to create an OTLP source keyed by service.” There is no required config. The label is the only thing to set. You can also send the same object with POST /api/sources.

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: resourceAttributes.service.name, primary: true }

4. Point the exporter at Tares. Use OTLP/HTTP, or OTLP/gRPC on port 4317 if you installed the otlp-grpc extra.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

5. If your Tares has auth on, the exporter sends an Authorization: Bearer header holding an ingest-scoped API key. You can set it with OTEL_EXPORTER_OTLP_HEADERS. Without auth, no header is needed.
6. Check that data arrives. Recent events shows logs as they come in, and Explore lists your services under the service key.
7. Add an alert source and a trigger keyed by service, as the AI SRE guide walks through. The trigger wakes a Tares agent when the alert for the cron job’s service fires.
8. To investigate yourself, start the MCP endpoint in a second shell and connect your client.

tares mcp

Then ask something like:

Use tares: what happened to nightly-report in the last 15 minutes?

The client answers from the correlated read of that service, which holds the log lines and the alert in the same window. The full connector reference is on the OpenTelemetry connector page.

What you get

• One source for every service. The cron job’s logs arrive next to your other OpenTelemetry data without extra setup per service.
• A single timeline per service, with log lines and alerts on one clock, so the agent does not stitch separate tools together.
• A first look that starts on its own. When the trigger fires, a Tares agent reads the timeline and writes a finding onto the service, with the likely cause tied to evidence lines and a next action.
• On-demand investigation. Any MCP client you connect reads the same timeline and the earlier finding, so it starts from the previous conclusion.
• Traces and metrics from the same exporter under the same service label, if you send them.

When something else fits

• If the cron job writes to a container’s stdout and you have no OpenTelemetry pipeline, the docker_logs connector tails a running container’s logs and fits better.
• If the jobs run in several places and emit no OpenTelemetry data, you can post JSON events to a webhook source instead, though that is a different setup from OTLP log records.

Related