How do I test Grafana alert rules about failed deploys before a real incident?

Tares lets you post a sample Grafana payload to your webhook source’s ingest URL and check the result under Recent events and Fields on the source page. That shows you the label mapping and the event for a failed deploy before a real alert fires. You can do it with one curl command, and you can ask a connected agent to create the source. The AI SRE guide covers what to build on top of it.

How it works

Grafana Alerting reaches Tares through the webhook connector and a Grafana contact point setup. Tares has no connector of its own for Grafana Alerting today.

• A Grafana notification arrives as one event. The alerts list stays inside the lossless payload.
• Top-level fields such as status, receiver and title work for labels and for text_template.
• Labels map payload fields into the event. You mark one label as primary, and it keys the entity.
• The source page shows Recent events, which lists what arrived, and Fields, which shows which payload fields arrive and how often.

Because a test post goes to the same ingest URL a real Grafana contact point uses, what you check now is what a real alert will do.

Grafana’s default payload is Alertmanager compatible. If you want one event per alert instead of one per notification, create a separate alertmanager source. It has its own ingest URL and fans the alerts array out into one event each.

Set it up

1. Create the webhook source. Go to Sources → Add source → webhook, or ask a connected agent: “Use tares to create a webhook source for my Grafana alerts.” The agent calls create_source and returns the ingest URL. Declare labels for the payload fields you want to correlate on, and mark one as primary:

   - name: grafana-alerts
     connector: webhook
     config:
       event_type_field: status
       text_template: '{title}: {status}'
       labels:
         - { name: receiver, field: receiver, primary: true }
         - { name: status, field: status }

2. Copy the ingest endpoint from the source page.

3. Post a sample payload shaped like a Grafana notification for a failed deploy:

   curl -X POST http://localhost:8787/ingest/<ingest_key> \
     -H 'Content-Type: application/json' \
     -d '{"receiver":"tares","status":"firing","orgId":1,"alerts":[{"status":"firing","labels":{"alertname":"DeployFailed","severity":"critical"},"annotations":{"summary":"Deploy failed"},"startsAt":"2024-01-01T00:00:00Z"}],"title":"[FIRING] DeployFailed","state":"alerting","message":"Deploy failed"}'

   If your Tares instance has auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. Add that header to the curl command. Without auth, you do not need it.

4. Open the source page and check Recent events. Confirm the event is there, and that its text and event type read the way you want.

5. Open Fields and check that receiver, status and title arrive. Adjust your labels and text_template until the mapping matches, then post again. Send a second payload with "status":"resolved" to see how a resolved notification looks.

6. When the mapping is right, wire up Grafana. Add a Webhook contact point with the Tares ingest URL and the POST method. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key. Then route the failed-deploy alert rule to that contact point from a notification policy.

A connected agent can also run test_source on the source. For a push source such as this one, test_source takes no payload and only returns the ingest endpoint, so it does not check the label mapping. Use the sample post and the Recent events and Fields views for that.

What you get

• Confidence in the label mapping before an incident. You see which fields land as labels and which key the entity.
• A readable event. You see the event type and the rendered text for a failed deploy, and you can fix the text_template while nothing is on fire.
• A repeatable check. You can post the same payload again after any change to the source, then read Recent events and Fields again.
• A clear path to production. The Grafana contact point sends to the same ingest URL you already tested.

When something else fits

• If you want one event per alert rather than one per notification, point Grafana at a separate alertmanager source and use the same Grafana payload.
• If you want to check a real Grafana delivery end to end, do that after the sample post, once the mapping is correct, by routing a test rule through your notification policy.

Related