How do I test Grafana alert rules about 5xx errors on an API before a real incident?

Tares lets you test Grafana alert rules for API 5xx errors before a real incident by posting a Grafana-shaped sample payload to a webhook source’s ingest URL. You then check the label mapping under Fields and the event under Recent events on the source page. Tares has no connector of its own for Grafana Alerting today, so Grafana reaches Tares through the webhook connector and a Grafana contact point. Nothing has to break in production for you to see the result.

How it works

Grafana Alerting sends notifications through a Webhook contact point. Tares receives them on a webhook source, which stores each POST lossless and maps the payload fields you declare as labels.

• A Grafana notification arrives as one event. The alerts list stays inside the stored payload.
• Top-level fields such as status, receiver and title work for labels and for text_template.
• The source page shows Recent events, which lists what arrived, and Fields, which shows which payload fields arrive and how often.
• For a push source, test_source takes no payload. It only returns the ingest endpoint, so it does not check the label mapping. To check the mapping you post a sample payload yourself.

Because a test POST goes through the same ingest URL a Grafana contact point uses, a payload that looks right on the timeline tells you the Tares side is ready for the real alert rule on your API’s 5xx errors. For more on how Tares uses events like this, see AI SRE.

Set it up

1. Create the source. In the console go to Sources, then Add source, then webhook. Or ask a connected agent to create a webhook source and give you the ingest URL. Declare a label for each top-level field you want to correlate on and mark one primary.

- name: grafana-alerts
  connector: webhook
  config:
    event_type_field: status
    text_template: '{title} is {status}'
    labels:
      - { name: receiver, field: receiver, primary: true }
      - { name: status, field: status }

2. Copy the ingest endpoint from the source page.

3. Post a sample payload shaped like a Grafana notification for a 5xx rule on your API. The alert name and annotation here are examples, so use the names of your own rule.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"receiver":"tares","status":"firing","orgId":1,"alerts":[{"status":"firing","labels":{"alertname":"Api5xxRate"},"annotations":{"summary":"5xx errors on the API"},"startsAt":"2024-01-01T00:00:00Z","fingerprint":"abc123"}],"title":"Api5xxRate","state":"alerting","message":"5xx errors on the API"}'

When Tares has auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. Add that header to the test request the same way you will in Grafana.

4. Open the source page. Under Recent events confirm the event arrived and the text reads the way you want. Under Fields confirm receiver and status show up, so the labels map to real values.

5. Wire up Grafana. Add a Webhook contact point with the Tares ingest URL and the POST method. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key. Then route alerts to the contact point from a notification policy.

6. Run the same check from Grafana. Use the contact point’s test option or let your 5xx rule fire in a non-production setting, then look at Recent events again. The event from Grafana should look like your sample.

What you get

• Proof that the ingest URL, the credential and the label mapping work before the first real 5xx alert.
• A visible event on the source’s Recent events list, with the full Grafana payload kept, including the alerts list.
• A Fields view that shows which payload fields arrive, so you can fix a label or the primary choice now and not during an incident.
• A tested path you can reuse. Once the sample and the Grafana test both land the same way, the rule’s notifications reach Tares the way you expect.

When something else fits

• If you want one event per alert, with its labels as Tares labels, create a separate alertmanager source with its own ingest URL. Grafana’s default payload is Alertmanager compatible, so it can take the same notification.
• If the payload fields you need to map sit inside the alerts list rather than at the top level, the alertmanager source is the better fit, because it splits that list into separate events.

Related