What should Alertmanager alert groups about error spikes include for an AI SRE?

Send Tares alerts that carry a service label, a severity label, a summary annotation and a start time, so the AI SRE can tie each error spike to evidence. Tares turns every alert in an Alertmanager batch into one event, with the alert name as the event type, the summary as the text and the labels as Tares labels. The full alert, including its start time and fingerprint, stays in the payload.

How it works

Alertmanager has already evaluated, grouped and deduped the alerts. When it POSTs a batch to your Tares ingest URL, Tares fans out the alerts array into one event per alert. Each event is built like this:

• Event type: the alert name, such as HighErrorRate.
• Text: the alert’s summary annotation.
• Labels: every label on the alert, plus a synthetic status of firing or resolved.
• Payload: the complete alert, including startsAt and fingerprint.

So each part of your alert has a job:

• A service name. Tares keys each alert to an entity from a label, checking namespace, then service, then job, then instance, or a label you declare as primary. Use the same service value your metrics and log sources use. The alert then lands on the same per-service timeline as the 5xx rate and the error logs, and the agent reads one correlated view. If your alerts carry both namespace and service and you want to key on service, set the advanced key_label field or declare service as the primary label.
• A severity. A severity label such as critical or warning is kept as a Tares label, so triggers and agents can tell a page-worthy spike from background noise.
• A summary. The summary annotation becomes the event text. Write it as a sentence an agent can quote, for example the symptom and the service it affects.
• A start time. startsAt travels in the payload, so the agent can compare when the alert began with when errors, deploys or config changes appeared on the timeline.
• A resolved signal. Set send_resolved: true so the resolved event arrives and the agent sees when the spike ended.

A stable alert name matters too. Because the alert name becomes the event type, a name like HighErrorRate reads clearly as a type of event on the timeline. The Alertmanager connector page lists every field.

Set it up

1. In the console, go to Sources, choose Add source, pick Alertmanager alerts, name it and select Create. The source page shows a receiver block with your ingest URL filled in. You can also ask a connected agent to create the source and give you the webhook URL.
2. Add the receiver to your alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If Tares runs with auth on (tares up --auth), the sender needs a Bearer Authorization header holding an ingest-scoped API key. Add it to the webhook_configs entry through Alertmanager’s http_config. The source page shows the exact block. Without auth, you need no key.
4. In your alerting rules, give every error spike alert a service label that matches your other sources, a severity label and a summary annotation. Keep your own rule names, so the alert name stays meaningful as the event type.
5. Send a test batch and check Recent events:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

6. Open Fields on the source. It shows the alert labels, so you can promote namespace or service to correlate on.

For the full path from alerts to an agent that names a root cause, follow the AI SRE guide.

What you get

• One event per alert. A grouped notification about an error spike becomes individual events, each with its own labels, status and summary.
• Alerts beside evidence. With a shared service label, the alert sits on the same timeline as your metrics and logs, so the agent can tie the failure to specific lines.
• A clear start and end. The start time in the payload and the resolved status mark the window of the spike.
• A trigger you can use. In the AI SRE guide, a trigger fires when an alert event arrives, per service, and wakes an agent. That agent takes a first look, ties the failure to evidence and writes a finding back onto the timeline.
• Alerting stays where it is. Alertmanager keeps grouping, silencing and resolved tracking. Tares records what it pushes and correlates it with your other sources.

When something else fits

• If you do not run Alertmanager and your Prometheus rules fire alerts directly, the prometheus_alerts connector polls them instead, but you lose Alertmanager’s grouping, silencing and real-time delivery.
• If an alert has no service, namespace, job or instance label, add one, or declare a primary label, so Tares can key the alert to an entity.

Related