How do I test Alertmanager alert groups about 5xx errors on an API before a real incident?

Tares lets you test Alertmanager alert groups about 5xx errors by posting a sample batch to the source’s ingest URL, then reading the event under Recent events. Alertmanager sends a batch with an alerts array, and Tares turns each alert into one event. You check that the event type, text, status and labels match your 5xx alert before a real one fires. You do not need to trigger a real outage.

How it works

An Alertmanager source in Tares is a push receiver. Alertmanager has already evaluated, grouped and deduplicated the alerts, so Tares only records what it receives.

• Alertmanager POSTs a batch that looks like { alerts: [...], ... } to the source’s ingest URL.
• Tares fans out the alerts array into one event per alert.
• event_type is the alert name, and text is the alert’s summary.
• The alert’s labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert, including fingerprint, stays in the payload.
• The entity key comes from a label, checked in the order namespace, service, job, instance, or from a label you declare primary. The advanced key_label field lets you pick the alert label to key on.

Because the ingest URL accepts any matching POST, a curl request stands in for Alertmanager. The event you get is the same one a real 5xx alert would create. The Alertmanager connector page lists every field.

Set it up

1. Create the source. Ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” Or open Sources, then Add source, then Alertmanager alerts, name it and Create. The source page shows a copy-paste receiver block with your ingest URL filled in.

2. Add the receiver to alertmanager.yml and reload Alertmanager. Do this once your test looks right, so real alerts follow the same path:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If your Tares instance has auth on (tares up --auth), the sender uses a Bearer Authorization header holding an ingest-scoped API key. On an instance without auth you can skip it. The source page shows the auth block when the instance requires a token.

4. Post a sample batch that looks like your 5xx alert. Match the alert name, labels and summary your Prometheus rule produces:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"ApiHigh5xxRate","severity":"critical","namespace":"prod","service":"api"},"annotations":{"summary":"API 5xx rate is high"},"startsAt":"2024-01-01T00:00:00Z"}]}'

5. Open the source page and read Recent events. Check that the event type is your alert name, the text is your summary, and the labels include namespace, service and severity, with status set to firing.

6. Open Fields. It lists the alert labels that arrived, so you can promote namespace or service to the label you correlate on. If the entity key is not the one you want, set key_label.

7. Post the same batch again with status set to resolved. This confirms that a resolved alert arrives as its own event with the resolved status, which is what send_resolved: true produces in real use.

An agent can also call test_source on this source. For a push source it takes no payload and only returns the ingest endpoint, so it does not check your label mapping. Use the posted payload for that check.

What you get

• Proof that the ingest URL, and the token if you use one, accepts your alerts before an incident.
• A visible event for a 5xx alert, with the event type, text, labels and status you expect.
• A label mapping you have checked. Fields shows which labels exist, so you choose the entity key before real traffic arrives.
• Confidence that firing and resolved alerts both arrive.
• A source that Tares correlates with your other sources once real alerts flow. You can then build on it with the guidance in AI SRE.

When something else fits

• If you want to post a payload with a shape of your own rather than Alertmanager’s batch, the webhook connector takes any JSON and lets you declare labels per field.
• If you do not need Alertmanager’s grouping, silencing and real-time delivery, the prometheus_alerts connector polls the alerts that Prometheus rules have fired.

Related