How do I pair Alertmanager alert groups with OpenTelemetry log records for an AI SRE?

Tares pairs Alertmanager alerts with OpenTelemetry logs when you key both sources by the same service label, so each firing alert sits on one timeline next to the log lines from the same period. Create an alertmanager source and an otlp source, make service the primary label on both, and point Alertmanager and your OTel exporter at Tares. An agent or trigger then works from one correlated read per service instead of two separate feeds.

How it works

Both connectors are push connectors, so Tares receives data and never polls for it.

• Alertmanager. Alertmanager posts a batch of alerts to the source’s ingest URL. Tares fans the alerts array out into one event per alert. event_type is the alert name, text is the summary, and the alert labels become Tares labels plus a synthetic status of firing or resolved. The full alert, including fingerprint, stays in the payload.
• OpenTelemetry. The OpenTelemetry connector receives OTLP/HTTP logs, traces and metrics. Resource attributes become labels, and service.name becomes the service label. One source ingests every service.
• The shared key. Tares correlates sources by their entity key. For OTLP the natural key is service. For the Alertmanager connector the key is derived automatically from namespace, then service, then job, then instance, or from a label you declare primary. If your alerts carry a namespace label, that label wins by default. Declare service as the primary label on the alertmanager source so alerts key on the same value as your logs.

The values must match. If the alert label is checkout and service.name is checkout, the alert and the logs land on the same timeline. The AI SRE guide builds the full incident-response agent on this per-service timeline.

Set it up

1. Start Tares locally and open the console.

tares up

2. Create the OTLP source under Sources → Add source → otlp, or ask a connected agent to create an OTLP source keyed by service. It needs no config beyond the label.

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: service, primary: true }

3. Point your exporter, an instrumented app or an OpenTelemetry Collector, at the daemon.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

4. Create the Alertmanager source under Sources → Add source → Alertmanager alerts, with service as the primary label. The source page shows a receiver block with the ingest URL filled in.

- name: alerts
  connector: alertmanager
  config:
    labels:
      - { name: service, field: service, primary: true }

5. Add the receiver to your alertmanager.yml and reload Alertmanager.

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

6. Send a test batch whose service matches a service that exports logs, then check Recent events and Explore.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","service":"checkout"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

On a secured instance, started with tares up --auth, both senders use a Bearer Authorization header holding an ingest-scoped API key. You need it only when Tares has auth on. For OTLP you can set it with OTEL_EXPORTER_OTLP_HEADERS.

If you prefer to leave the labels block out, the alertmanager source also has an advanced key_label field that names the alert label to key on.

What you get

• A firing alert and the logs from the same service appear on one timeline in Explore, merged on one clock.
• Resolved alerts arrive too when send_resolved is true, so the timeline shows both the start and the end of the incident.
• Alertmanager keeps owning grouping, deduplication and silencing. Tares records what it sends and correlates it with your other sources.
• A trigger over both sources, keyed by service, can wake an agent when an alert event arrives. The agent reads the correlated evidence rather than hunting through two systems.
• A connected MCP client can ask what happened to a service in the last 15 minutes and answer from the same read.
• One OTLP source covers every service, so adding a new service needs no new Tares setup, only a matching service.name.

When something else fits

• If you run Prometheus alerting rules directly and do not need Alertmanager’s grouping and silencing, the prometheus_alerts connector polls fired alerts without a receiver.
• If your logs come from containers rather than an OpenTelemetry pipeline, the docker_logs connector tails a container’s logs and can carry the same service label.

Related