How do I line up Alertmanager alert groups with OpenTelemetry metric series on one timeline?

Tares lines them up when you key both sources by the same service label. Alertmanager alerts and OTLP metric points then land on one timeline per service, merged on one clock in Explore. Alertmanager pushes alerts to an ingest URL, your OpenTelemetry exporter or Collector sends metrics to the OTLP endpoint, and Tares correlates the two.

How it works

Tares correlates events from different sources when they share a key. Here the key is the service.

• Alertmanager source. The Alertmanager connector receives each batch Alertmanager posts and fans the alerts array out into one event per alert. The event type is the alert name, the text is the alert summary, and the alert labels become Tares labels. Tares adds a status label that is firing or resolved. The full alert, including its fingerprint, stays in the payload.
• OTLP source. The OpenTelemetry connector receives OTLP/HTTP logs, traces and metrics. One source ingests every service. Resource attributes such as service.name become labels, and the service label is the natural key.
• The shared key. The Alertmanager source derives its entity key from the first alert label it finds in this order: namespace, service, job, instance. If your alerts carry namespace, that label wins over service. Set key_label to service so the alert key matches the OTLP service label.

Once both sources use the same service value, Explore shows the alert and the metric points for that service merged on one clock. The alert tells you what fired, and the metric points show how the service got there. This is the same per-service timeline the AI SRE guide builds its incident agent on.

Set it up

1. Start Tares locally.

   tares up

2. Create the OTLP source under Sources, Add source, otlp, or ask a connected agent to create an OTLP source keyed by service. Mark service as the primary label.

   - name: otel
     connector: otlp
     config:
       labels:
         - { name: service, field: service, primary: true }

3. Point your exporter at Tares. For OTLP/gRPC, install the otlp-grpc extra first. The receiver listens on :4317.

   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
   # or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

4. Create the Alertmanager source under Sources, Add source, Alertmanager alerts. The source page shows a receiver block with your ingest URL filled in. Set key_label to service in the source config so alerts key on the same label as your metrics.

   - name: alerts
     connector: alertmanager
     config:
       key_label: service

5. Add the receiver to your alertmanager.yml and reload Alertmanager. Keep send_resolved: true so resolved alerts reach the timeline too.

   receivers:
     - name: tares
       webhook_configs:
         - url: http://localhost:8787/ingest/<ingest_key>
           send_resolved: true
   route:
     receiver: tares

6. If your Tares instance has auth on, both senders (Alertmanager and the OTLP exporter) use a Bearer Authorization header holding an ingest-scoped API key. Without auth you need nothing extra.

7. Check both sources. Recent events on each source shows what has arrived, and Fields on the Alertmanager source lists the alert labels. Open Explore, find your service under the service key, and confirm alerts and metric points appear together. Make sure the alert label values match the OTel service.name values exactly. A mismatch puts them on two separate timelines.

What you get

• One timeline per service with Alertmanager alerts and OpenTelemetry metric points merged on one clock in Explore.
• Alertmanager’s own grouping, silencing and resolved tracking, kept as it is. Tares records what Alertmanager pushes, one event per alert, with firing and resolved status.
• Every service from your OTel SDKs and Collector on the same OTLP source, with no per-service setup.
• Logs and traces from the same OTLP source on that service’s timeline too, if you export them.
• A single read that a connected agent or a trigger-woken agent can work from, so the investigation starts from the alert and the metrics that led to it.

When something else fits

• If your metrics live in a Prometheus server and you do not export them over OTLP, use the Prometheus connector. You pick metrics by name or label, with no PromQL, and key them by service the same way.
• If you do not need Alertmanager’s grouping and want Tares to read fired rules directly, the prometheus_alerts connector polls them instead. It is described in the AI SRE guide’s source table.

Related