How do I test Grafana alert rules about crash loops before a real incident?

Tares lets you test crash loop alerts from Grafana before a real incident by posting a sample Grafana payload to your webhook source’s ingest URL. You then read the event under Recent events and check the label mapping under Fields. Grafana Alerting reaches Tares through the webhook connector and a Grafana contact point, because Tares has no connector of its own for Grafana Alerting today.

How it works

A Grafana webhook contact point POSTs one notification to the ingest URL of a Tares webhook source. Tares stores the whole payload lossless, with the alerts list inside it, as a single event. The labels you declare on the source map payload fields into the event, and a field can be top level, such as status, receiver and title, or a one-level dotted path, such as commonLabels.service.

For a push source, test_source takes no payload. It only returns the ingest endpoint, so it cannot check your label mapping. To test the mapping, post a sample payload yourself and read the result on the source page:

• Recent events shows the event Tares recorded.
• Fields shows which payload fields arrive and how often, so you can see whether commonLabels.service is there.

This tests the Tares side of the path: the URL, the authentication and the mapping. Your crash loop rule itself still lives and evaluates in Grafana. Read more on the webhook connector page. For the wider picture of watching services with Tares, see AI SRE.

Set it up

1. Create a webhook source for Grafana. Use Sources → Add source → webhook, or ask a connected agent to create a webhook source and give you the ingest URL. Key it on the service with a primary label, and keep status and receiver as plain labels.

- name: grafana-alerts
  connector: webhook
  config:
    text_template: '{title} {status}'
    labels:
      - { name: service, field: commonLabels.service, primary: true }
      - { name: status, field: status }
      - { name: receiver, field: receiver }

2. Copy the ingest endpoint from the source page.

3. Post a sample crash loop notification in the shape Grafana sends, with a service entry in commonLabels:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"receiver":"tares","status":"firing","orgId":1,"alerts":[{"status":"firing","labels":{"alertname":"PodCrashLooping","service":"checkout"},"annotations":{"summary":"checkout pod is crash looping"},"startsAt":"2024-01-01T00:00:00Z","fingerprint":"abc123"}],"commonLabels":{"service":"checkout"},"title":"[FIRING:1] PodCrashLooping","state":"alerting","message":"checkout pod is crash looping"}'

4. If Tares has auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. Add that header to the request. Without auth you can post as shown.

5. Open the source page. Under Recent events, confirm the event arrived and that its service label reads checkout. Under Fields, confirm commonLabels.service is listed. Post the same payload with status set to resolved to see the resolved case.

6. Wire up Grafana. Add a Webhook contact point with the Tares ingest URL and the POST method. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key. Then route alerts to the contact point from a notification policy.

7. Compare a real delivery with your sample. Once your crash loop rule fires or resolves, check Fields again to confirm the live notification carries the same fields as your test.

What you get

• Proof that the ingest URL, the authentication and your label mapping work before any crash loop happens.
• A visible service label on the event, so you know which entity the alert is filed under.
• The full Grafana notification kept lossless, including the alerts list and each alert’s fingerprint, so nothing you may want later is dropped.
• A repeatable test: post the firing and the resolved sample whenever you change the source’s labels or text template.

When something else fits

• If you want one event per alert instead of one event per notification, create a separate alertmanager source with its own ingest URL. Grafana’s default payload is Alertmanager compatible, so it can take the same payload. That connector derives its entity key from labels such as namespace or service.
• If your crash loop signal comes from Prometheus rules rather than Grafana, point Alertmanager at Tares instead of going through Grafana.

Related