How does each Alertmanager alert become an event in Tares?

Tares fans out the alerts array of each Alertmanager batch into one event per alert. Each event has event_type set to the alert name, text set to the alert summary, the alert labels kept as labels plus a synthetic status of firing or resolved, and the full alert, including its fingerprint, kept in the payload. You get one event for every alert Alertmanager routes, however it grouped them.

How it works

Alertmanager sends one HTTP POST per notification. The body is a batch: a JSON object with an alerts array and some group-level fields. The Alertmanager connector is a push receiver, so Tares does no polling. It reads the batch at the source’s ingest URL and splits it.

For every entry in the alerts array, Tares creates one event:

• event_type is the alert name, for example PostgresDown.
• text is the alert’s summary annotation, for example Postgres is down.
• labels are the alert’s own labels, such as severity and namespace, kept as Tares labels.
• status is a synthetic label added by Tares. Its value is firing or resolved, taken from the alert’s state.
• payload holds the full alert, including the fingerprint, so nothing Alertmanager sent is lost.

A batch of five alerts therefore gives you five events, each with its own type, text and labels. A firing alert and its later resolved notification are separate events with the same alert name, told apart by the status label. Resolved notifications arrive only when the receiver sets send_resolved to true.

Alertmanager has already evaluated, grouped and deduplicated the alerts before it posts them. Tares records what Alertmanager pushes. It does not regroup or re-evaluate anything.

Each event also needs an entity key. Tares derives it from a label, trying namespace, then service, then job, then instance, or from a label you declare as primary. The advanced key_label field lets you name a different alert label to key on.

Set it up

1. In Tares, go to Sources, then Add source, then Alertmanager alerts. Name the source and select Create. Or ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” Either way you get an ingest URL. The source page also shows a ready receiver block with that URL filled in. See AI SRE for how alert events fit into the wider workflow.
2. Add the receiver to your alertmanager.yml, with your own ingest key in the URL, and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If your Tares instance has auth on, the sender needs a Bearer Authorization header holding an ingest-scoped API key. Add it to the webhook_configs entry. The source page shows the exact block. If auth is off, skip this step.
4. Send a test batch to the ingest endpoint to see the fan-out for yourself:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

5. Open Recent events on the source. You see one event with event_type PostgresDown, text Postgres is down, and the labels severity, namespace and a status of firing.
6. Open Fields. It lists the alert labels, so you can promote namespace or service and correlate on it.

What you get

• One event per alert, so a grouped notification with many alerts never hides an individual alert inside one record.
• A readable event: the alert name as the type and the summary as the text.
• Every alert label available as a Tares label, plus a status label that separates firing from resolved.
• The complete alert in the payload, fingerprint included, for the details the labels do not carry.
• An entity key derived from the labels you already use, with key_label as an override.
• Alertmanager’s grouping, silencing and resolved handling stay in place, because Tares records what Alertmanager decides to send. Once the events are in, Tares correlates them with your other sources, and promoting a label such as namespace or service in Fields gives you the key to correlate on.

When something else fits

If you want Tares to read alerts that Prometheus rules fire directly, without Alertmanager’s grouping, silencing and resolved handling, use the prometheus_alerts connector, which polls every 30 seconds by default. If you want real-time delivery with Alertmanager semantics, stay with the alertmanager connector described here. For a sender that is not Alertmanager, the webhook connector stores whatever JSON it posts, as one event per post.

Related