Which OpenTelemetry log records does an AI SRE search when Alertmanager alert groups fire?

Tares gives an AI SRE the OpenTelemetry log records that carry the same service label as the alert, on that service’s correlated timeline. Alertmanager alerts and OTLP logs both land as events keyed by service, so the agent reads one timeline for the service that fired. You choose the span, for example the last 15 minutes, when you read it.

How it works

Two sources feed one timeline per service:

• Alertmanager. Every alert in a group that Alertmanager posts becomes its own event. The event type is the alert name, the text is the alert summary, and the alert labels become Tares labels plus a status of firing or resolved. The entity key comes from a label, checked in the order namespace, service, job, instance, unless you declare a primary label or set key_label.
• OTLP. One OpenTelemetry source ingests every service. Resource attributes become labels, and service comes from service.name.

When both sources carry the same service label, their events sit on one timeline for that service, with metrics, logs and alerts merged on one clock. That timeline is what the agent reads. A trigger wakes an agent with it, and an MCP client gets it by asking for a service over a span, such as “what happened to api-server in the last 15 minutes?”. The Alertmanager connector page covers the alert side in full.

The key choice matters. Alertmanager checks namespace before service, so an alert that carries both is keyed by namespace unless you set key_label to service. Set it, or declare service as the primary label, so alerts and logs share the same key.

Set it up

1. Create the OTLP source keyed by service. In the console go to Sources, Add source, otlp, or use this object:

   - name: otel
     connector: otlp
     config:
       labels:
         - { name: service, field: service, primary: true }

2. Point your exporter or OpenTelemetry Collector at Tares.

   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
   # or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

3. Create an Alertmanager source under Sources, Add source, Alertmanager alerts. The source page shows a receiver block with the ingest URL filled in. Add it to alertmanager.yml and reload Alertmanager:

   receivers:
     - name: tares
       webhook_configs:
         - url: http://localhost:8787/ingest/<ingest_key>
           send_resolved: true
   route:
     receiver: tares

4. Set the alert label to key on, key_label, to service, so alert events join the same timeline as the OTLP logs.

5. If Tares runs with auth on, both senders use a Bearer Authorization header holding an ingest-scoped API key. Without auth, no key is needed.

6. Send a test batch, then open Recent events and Fields to confirm the alert labels include service:

   curl -X POST http://localhost:8787/ingest/<ingest_key> \
     -H 'Content-Type: application/json' \
     -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

7. Add a trigger over the alert source, keyed by service, and subscribe an agent to it. The trigger wakes the agent with the correlated timeline for that service. The Build an AI SRE guide walks through the full chain.

What you get

• Log records tied to the service that fired, because the OTLP service label and the alert’s service label are the same key.
• Alerts, logs and metrics for that service on one clock, so you can compare when the first alert arrived with what the logs show around it.
• The full alert, including its fingerprint, kept in the event payload.
• Resolved alerts on the same timeline when send_resolved is on, so the end of an incident shows next to its start.
• The same view for interactive use. A connected MCP client reads the service over a span you name and answers from it.

When something else fits

• If you want a cheap first level that looks at your sources every few minutes before any alert fires, use the schedule trigger and triage agent in Watch, then escalate.
• If your alerts are evaluated in Prometheus and you do not run Alertmanager, the prometheus_alerts connector polls them instead, though you lose Alertmanager’s grouping and silencing.
• If your logs come from a container and not from OpenTelemetry, the docker_logs connector can feed the same service timeline.

Related