How do I test Alertmanager alert groups about error spikes before a real incident?

Tares lets you test an Alertmanager error spike alert before a real incident by posting a sample batch to the source’s ingest URL and reading the result under Recent events and Fields. Each alert in the batch becomes one event, with the alert name as the event type and its labels mapped into Tares labels. You see right away whether the labels you want to correlate on arrive as expected.

How it works

The Alertmanager source is a push receiver. Alertmanager POSTs a batch shaped like { alerts: [...], ... } to the source’s ingest endpoint, and Tares fans the alerts array out into one event per alert:

• event_type is the alert name.
• text is the alert’s summary annotation.
• The alert’s labels become Tares labels, plus a synthetic status label that is firing or resolved.
• The full alert, including fingerprint, is kept in the event payload.

The entity key is derived automatically from a label, in the order namespace, service, job, instance, or from a label you declare as primary. If your error spike alerts carry a different label that identifies the affected service, set key_label on the source.

Because the source only records what Alertmanager pushes, you can send the same shaped batch yourself with curl and get the same kind of event. For a push source, the test_source tool takes no payload and only returns the ingest endpoint. It does not check your label mapping. To check the mapping, post a sample payload and read the event. The Alertmanager connector page has the full reference, and the inbound webhook page describes the same ingest and verify flow for generic JSON.

Set it up

1. Create the source. Go to Sources → Add source → Alertmanager alerts, name it and click Create. Or ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” The agent calls create_source and returns the ingest URL.

2. Copy the ingest URL from the source page. It has the form /ingest/<ingest_key>. The source page also shows a copy-paste alertmanager.yml receiver block with the URL already filled in.

3. Send a sample batch. Use the example below as it is, or change the alertname, severity, namespace and summary values to match what your error spike rule produces.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

4. Open the source page and read Recent events. You see one event for the alert you posted, with the alert name as its type and the summary as its text.

5. Open Fields. It lists the alert labels that arrived. Promote namespace or service, whichever identifies the affected system, so Tares can correlate on it.

6. Add the receiver to your alertmanager.yml and reload Alertmanager. Point it at the same ingest URL:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

7. If Tares runs with auth on, the sender needs a Bearer Authorization header holding an ingest-scoped API key. On a local instance that means it was started with tares up --auth. The source page shows the matching auth block for the receiver. Without auth on, you need no key.

A curl post goes straight to the ingest URL and does not pass through Alertmanager. It tests the label mapping and the event. Your Alertmanager routing and receiver config are a separate part to confirm once the receiver is live.

What you get

• A visible event in Recent events for an error spike alert, created from a payload you control, before any real incident.
• A check of the label mapping: the event shows which labels came through, and Fields shows how often each one arrives.
• A chosen entity key, either one of the automatic labels or a primary label or key_label you set, so error spikes from one service correlate together.
• Both firing and resolved statuses recorded, because the receiver uses send_resolved: true. Post one batch with "status":"firing" and another with "status":"resolved" to see both.
• The full alert, including fingerprint, stored in the event payload.

When something else fits

• If you want Prometheus alerts without running Alertmanager, the prometheus_alerts connector polls fired alerts instead. Alertmanager is the better fit when you want its grouping, silencing and resolved handling with real-time delivery.
• If the sender is not Alertmanager, such as a CI system or your own service, use the inbound webhook connector and declare labels per payload field.
• For the wider picture of how alert sources feed investigations, see AI SRE.

Related