How do I test Alertmanager alert groups about slow database queries before a real incident?

Tares lets you test slow-query alerts before a real incident by posting a sample Alertmanager batch to your source’s ingest URL. You then check Recent events and Fields on the source page. Each alert in the batch becomes one event, so you see the event type, text and labels Tares will use for a real alert. For a push source, test_source only returns the ingest endpoint, so the sample post is how you check the label mapping.

How it works

The alertmanager connector is a push receiver. Alertmanager has already evaluated, grouped and deduped the alerts. Tares records what it pushes and correlates it with your other sources.

Alertmanager POSTs a batch that looks like { alerts: [...], ... }. Tares fans the alerts array out into one event per alert:

• event_type is the alert name, for example a slow-query alert’s alertname.
• text is the alert’s summary annotation.
• The alert’s labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert, including fingerprint, is kept in the payload.

The entity key is derived from a label, in this order: namespace, service, job, instance. A label you declare as primary takes its place. The advanced key_label field picks a different alert label to key on.

Because you post the batch yourself, you can send it as often as you like and change the labels until the mapping looks right. The same ingest endpoint is what a real Alertmanager receiver posts to. The webhook connector works the same way for other senders.

Set it up

1. Create the source. Ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” Or open Sources, Add source, Alertmanager alerts, name it and select Create. The source page shows the ingest URL.

2. Post a sample slow-query alert to the ingest URL. Use the labels your real alerts carry, such as alertname, severity and namespace:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresSlowQueries","severity":"warning","namespace":"prod"},"annotations":{"summary":"Queries on the orders database are slow"},"startsAt":"2024-01-01T00:00:00Z"}]}'

3. If your instance runs with tares up --auth, the sender must include a Bearer Authorization header holding an ingest-scoped API key. Without auth on, no header is needed.

4. Open the source page and check Recent events. You should see one event whose type is PostgresSlowQueries and whose text is the summary.

5. Open Fields. It lists the alert labels that arrived. Promote namespace or service to a label so Tares can correlate the slow-query alert with your other sources.

6. Post the same alert again with status set to resolved. You get a second event for the same alert with the resolved status.

7. When the mapping looks right, add the receiver to your alertmanager.yml and reload Alertmanager. The source page shows this block with your ingest URL filled in:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

For the full field list and auth details, see the Alertmanager connector page.

What you get

• A visible event for each sample alert, with the event type, summary text and labels as Tares will record them.
• A check of your label mapping. Fields shows which alert labels arrive, so you choose the primary label and the labels to correlate on before anything fires for real.
• Firing and resolved events you can compare side by side, because Tares adds the status label.
• The full alert kept in the payload, including fingerprint.
• A repeatable test. You can post a new batch whenever you change a label or an annotation.

For how these events feed investigation and escalation, see AI SRE.

When something else fits

• If you only need the ingest endpoint, test_source returns it for a push source. It takes no payload, so it does not check the label mapping. Use a sample post for that.
• If Prometheus rules should be read directly, without Alertmanager’s grouping and silencing, the prometheus_alerts connector polls fired alerts instead.
• If the sender is not Alertmanager, the webhook connector takes arbitrary JSON and lets you map fields with labels.

Related