What does an event from Alertmanager alert groups look like once it reaches Tares?

Tares turns every alert in an Alertmanager group into its own event. The event type is the alert name, the text is the alert summary, and the alert labels become Tares labels plus a status label of firing or resolved. The full alert, including its fingerprint, stays in the event payload, so you keep everything Alertmanager sent.

How it works

Alertmanager has already evaluated, grouped and deduped your alerts before it calls Tares. With send_resolved on, it also tracks the move from firing to resolved. Tares records what Alertmanager pushes and correlates it with your other sources, as described in AI SRE.

Alertmanager POSTs a batch to the source’s ingest URL, a JSON object with an alerts array. Tares fans that array out into one event per alert:

• Event type: the alert name, for example PostgresDown.
• Text: the alert’s summary annotation.
• Labels: the alert’s own labels become Tares labels, and Tares adds a synthetic status label that is firing or resolved.
• Payload: the full alert is kept, including fingerprint.
• Entity key: Tares derives it from a label, trying namespace, then service, then job, then instance. If you declare a primary label, that label is used instead. The advanced key_label option names the alert label to key on.

For the test alert in the setup steps, the event looks like this:

• event type: PostgresDown
• text: Postgres is down
• labels: alertname, severity (critical), namespace (prod) and status (firing)
• entity key: prod, taken from namespace

When the same alert later arrives as resolved, it comes in as a new event with status set to resolved. That lets you see both states of an alert next to your logs, metrics and deploys.

Set it up

1. Create the source. In the console, go to Sources, then Add source, then Alertmanager alerts, name it and select Create. Or ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” The agent calls create_source and returns the ingest URL. The source page also shows a copy-paste receiver block with the URL filled in.
2. Add the receiver to your alertmanager.yml, then reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If your Tares instance has auth on, Alertmanager must send a Bearer Authorization header holding an ingest-scoped API key. Add it in the http_config of the webhook_configs entry. The source page shows this block when the instance requires a token. Without auth, skip this step.
4. Send a test batch to the ingest endpoint:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

5. Open the source page and check Recent events. You should see one event for the alert. Fields lists the alert labels, so you can promote namespace or service as the label to correlate on.

The Alertmanager connector is a push connector, so it has no polling config. The only option is key_label.

See the Alertmanager connector page for the full reference.

What you get

• One event per alert, not one per group, so each alert can be filtered, counted and triggered on separately.
• A readable event: the alert name as the type and the summary as the text.
• Alert labels as Tares labels, with a status label to separate firing from resolved.
• The full alert, fingerprint included, in the payload.
• An entity key chosen for you from namespace, service, job or instance, so alerts line up with events from your other sources on the same service.
• Real-time delivery, because Alertmanager pushes to Tares as it routes.

When something else fits

• If you want Tares to poll Prometheus for fired alerts instead of receiving them from Alertmanager, use the prometheus_alerts connector. Choose the Alertmanager connector when you want Alertmanager’s grouping, silencing and resolved handling.
• For alert tools that have no connector of their own in Tares today, such as Datadog, Sentry or PagerDuty, point the vendor’s webhook at a webhook connector source. You declare labels for the payload fields you want, set an event type or an event type field, and optionally a text template.

Related