How do I line up Alertmanager alert groups with Prometheus metrics on one timeline?

Tares puts Alertmanager alerts and Prometheus metrics on one timeline when both sources carry the same entity key, such as a service label. Alertmanager pushes each alert to Tares as its own event, and Tares polls the Prometheus metrics you pick. In Explore, you pick the entity and see what fired next to how the metric got there, on one clock.

How it works

Tares correlates sources by a shared entity key, so the work is making both sources resolve to the same key value.

• Alertmanager is a push source. You point a webhook receiver at the source’s ingest URL. Alertmanager has already grouped and deduped the alerts. Tares fans the alerts array out into one event per alert. The event_type is the alert name, the text is its summary, and the alert labels become Tares labels plus a status of firing or resolved. The full alert, including its fingerprint, stays in the payload. See the Alertmanager connector.
• Prometheus is a poll source. You pick metrics by name or by label, with no PromQL to write. Each series becomes a metric event on every poll tick, and the numeric value is stored as a number-typed value label. See the Prometheus connector.
• The key is what joins them. For Alertmanager, Tares derives the key from a label in this order: namespace, service, job, instance. You can declare a primary label instead, or set key_label to name the alert label to key on. For Prometheus, the primary label in your config is the key, for example service read from metric.service.

When both sources produce the same value, for example api-server, their events merge into one timeline for that entity. This is the same per-service correlation the AI SRE guide builds on.

Set it up

1. Start Tares and open the console at http://127.0.0.1:8787.

tares up

2. Add the metrics. Go to Sources → Add source → Prometheus, enter the url, click Discover, and tick the metrics you want. Tares proposes the entity key and labels from a sample. The generated config looks like this. Note the primary service label and the number-typed value:

- name: metrics
  connector: prometheus
  poll: 10s
  config:
    url: http://localhost:9090
    queries:
      - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
        by_name: true
    labels:
      - { name: service, field: metric.service, primary: true }
      - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

3. Add the alerts. Go to Sources → Add source → Alertmanager alerts, name it and click Create. The source page shows a receiver block with the ingest URL already filled in. Add it to your alertmanager.yml and reload Alertmanager. Keep send_resolved: true so resolved alerts land on the timeline too:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

4. If your instance runs with auth on, the Alertmanager receiver sends a Bearer Authorization header holding an ingest-scoped API key. Without auth, you need no header.

5. Make the keys match. If your alerts carry a service label, the Alertmanager source already keys on it, because service comes before job and instance in the derivation order. If namespace is also present, it wins, so set key_label on the Alertmanager source to the label your metrics use. Check that the metric label feeding your primary label holds the same value as the alert label.

6. Send a test alert and check Recent events on the Alertmanager source:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

7. Open Fields on each source to see the labels. Then open Explore, pick the entity both sources are keyed on, and read the alert events and metric events on one clock.

What you get

• One timeline per entity where a firing alert sits next to the metric series that led up to it, and the resolved event sits next to the recovery.
• Alertmanager’s own grouping, silencing and resolved handling, delivered in real time, with the full alert kept in the payload.
• Only the metrics you picked, ingested as typed events. Triggers can then aggregate the value label with max, avg or sum, filtered to one metric by event_type.
• One correlated read for a person in Explore or for an agent. You can wake an agent from a trigger and hand it the same merged timeline.

When something else fits

• If you do not run Alertmanager and only need the alerts your Prometheus rules fire, the prometheus_alerts connector polls them directly. Choose Alertmanager when you want its grouping, silencing and real-time delivery.
• If you need a computed metric such as a p99 rather than a raw series, a queries row can hold a single PromQL expression with an explicit event_type and field.

Related