How do I tie Alertmanager alert groups to the right service?

Tares keys every Alertmanager alert on one of its labels, so the alerts land on the same entity as the metrics and logs that carry that label. Point the receiver at an Alertmanager source and, if the default key is not the one you want, set key_label to the label your other sources use, such as service. Tares then merges alerts, metrics and logs into one timeline per service.

How it works

The Alertmanager connector is a push receiver. Alertmanager POSTs a batch of alerts to the source’s ingest URL, and Tares splits the alerts array into one event per alert:

• event_type is the alert name.
• text is the alert summary.
• The alert’s labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert, including its fingerprint, stays in the payload.

The entity key comes from a label. Tares looks for namespace, then service, then job, then instance, or it uses a label you declare as primary. You can also set key_label to name the alert label to key on. The default is namespace or service if present.

The order matters. If your alerts carry both namespace and service, the default keys on namespace. Set key_label to service when your metrics and logs are keyed by service.

Timelines join because sources share the same key label. In the AI SRE guide, the metrics, logs and alerts sources are all keyed by the same service label, which gives one timeline per service with all three merged on one clock.

Set it up

1. Decide which label your metrics and logs sources use as their primary label. This guide uses service.
2. Create the source. In the console go to Sources → Add source → Alertmanager alerts, name it and click Create. Or ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” The agent calls create_source and returns the ingest URL.
3. Add the receiver to your alertmanager.yml and reload Alertmanager. The source page shows this block with the ingest URL already filled in:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

Keep send_resolved: true so resolved alerts arrive and close out the firing ones. On a secured instance (tares up --auth), the sender uses a Bearer Authorization header holding an ingest-scoped API key. It is needed only when Tares has auth on, and the source page shows the auth block when the instance requires a token.

4. Set the key label on the source when the default key is not the one you want:

- name: alerts
  connector: alertmanager
  config:
    key_label: service

5. Send a test batch to the ingest endpoint and check Recent events:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

6. Open Fields on the source. It lists the alert labels that arrive, so you can promote namespace or service to correlate on and confirm it matches the key your metrics and logs sources use.

What you get

• Each alert lands on the entity named by its key label, next to that entity’s metrics and logs.
• Firing and resolved alerts sit on the same timeline, because status is recorded on every event.
• Triggers and agents that key on the shared label see the alert together with the evidence around it, in one read.
• Alertmanager keeps its own grouping, silencing and dedupe. Tares records what Alertmanager sends and correlates it with your other sources.
• The full alert, including its fingerprint, is kept in the payload, so nothing is lost when you pick a different key later.

When something else fits

• If you want Tares to poll Prometheus for fired alerts instead of receiving Alertmanager’s pushes, use prometheus_alerts. You give up Alertmanager’s grouping, silencing and real-time delivery.
• If the alerts come from a sender other than Alertmanager, or you want to map your own payload fields, the webhook connector lets you declare a primary label per payload field.

Related