How do I test Alertmanager alert groups about high latency before a real incident?

Tares lets you test high latency alert groups from Alertmanager by posting a sample batch to the source’s ingest URL and reading the result on the source page. You see each alert as an event under Recent events, and Fields shows how its labels were mapped. The test_source tool does not do this check for push sources, so a sample payload is the test.

How it works

The alertmanager connector is a push receiver. You point an Alertmanager webhook_configs url at the source’s ingest endpoint, and every alert Alertmanager routes there becomes an event. Alertmanager has already evaluated, grouped and deduped the alerts. With send_resolved on, it also tracks firing to resolved. Tares records what Alertmanager pushes and correlates it with your other sources. See AI SRE for how those events are used.

When a batch arrives, Tares fans the alerts array out into one event per alert:

• event_type is the alert name.
• text is the alert’s summary annotation.
• The alert’s labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert, including its fingerprint, stays in the payload.

The entity key is derived automatically from a label, in this order: namespace, service, job, instance. A label you declare as primary takes precedence. The optional key_label field lets you name the alert label to key on.

For a push source, test_source takes no payload. It only returns the ingest endpoint, so it does not check your label mapping. A sample payload posted to the ingest URL exercises the same path a real alert takes.

Set it up

1. Create the source. Ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” The agent calls create_source and returns the ingest URL. Or open Sources, then Add source, then Alertmanager alerts, name it and click Create. The source page shows a copy-paste receiver block with the ingest URL filled in.

2. Add the receiver to your alertmanager.yml and reload Alertmanager. Do this now so the route is ready for a real latency alert:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If your Tares instance has auth on (tares up --auth), the sender needs a Bearer Authorization header holding an ingest-scoped API key. Alertmanager takes this in the webhook_configs entry. The source page shows the exact block when the instance requires a token. Without auth, skip this step.

4. Post a sample high latency batch to the ingest URL. Use your real alert name and labels so the test matches what Alertmanager will send:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"HighLatency","severity":"warning","namespace":"prod"},"annotations":{"summary":"p99 latency is high"},"startsAt":"2024-01-01T00:00:00Z"}]}'

5. Open the source page and read Recent events. You should see one event whose type is HighLatency and whose text is the summary you sent.

6. Open Fields. It lists the alert labels that arrived. Promote namespace or service to a label so Tares can correlate the latency alert with your other sources.

7. Repeat step 4 with status set to resolved. This confirms the resolved path before a real incident, and the event shows the resolved status.

What you get

• Confidence that the ingest URL, the receiver block and the auth setting work before an alert fires for real.
• A visible event on the timeline for the sample alert, with the alert name as its type and the summary as its text.
• A clear view of the label mapping on the Fields tab, so you choose the entity key on purpose and not by accident.
• A tested path for firing and resolved states.
• A source that is ready to correlate latency alerts with your other sources when a real incident starts.

The sample event is an ordinary event in the source, so use clearly marked test values, such as a distinct summary, to tell it apart from real alerts later.

When something else fits

• If the sender is not Alertmanager, such as a CI system or your own service, use the webhook connector. It stores any JSON payload and lets you declare labels per field. You test it the same way, by posting a sample payload and reading Recent events and Fields.
• If you want Tares to poll alerts that Prometheus rules have fired, instead of receiving Alertmanager’s grouped, silenced and resolved deliveries, use the prometheus_alerts connector. The Alertmanager connector page covers when each one fits.

Related