How do I test Grafana alert rules about slow database queries before a real incident?

Tares lets you test a Grafana alert for slow database queries by posting a sample Grafana payload to a webhook source’s ingest URL and reading the result under Recent events and Fields. Grafana Alerting reaches Tares through the webhook connector and a vendor setup, and Tares has no connector of its own for it today. Once the sample event and its labels look right, point a Grafana contact point at the same URL.

How it works

A Grafana contact point sends each notification as one JSON POST. In Tares that POST lands on a webhook source and is stored lossless, so the whole alerts list is kept inside the event payload.

• Top-level fields such as status, receiver and title work for labels and for text_template.
• The per-alert labels and annotations sit inside the alerts list, so they are not top-level fields.
• test_source on a push source takes no payload and only returns the ingest endpoint. It does not check your label mapping. A sample POST does.
• The default Grafana payload is Alertmanager compatible. If you want one event per alert, with each alert’s labels as Tares labels, create a separate alertmanager source. It has its own ingest URL and takes the same Grafana payload.

The test runs entirely on the Tares side, so you can validate the mapping before Grafana ever fires a rule. For where this fits in a wider monitoring setup, see AI SRE.

Set it up

1. Create the webhook source. Ask a connected agent to create a webhook source for your Grafana alerts, or go to Sources → Add source → webhook. Declare a label per field you want to correlate on and mark one primary.

- name: grafana-slow-queries
  connector: webhook
  config:
    event_type_field: status
    text_template: '{title}: {status}'
    labels:
      - { name: title, field: title, primary: true }
      - { name: status, field: status }
      - { name: receiver, field: receiver }

2. Copy the ingest endpoint from the source page. On a local Tares, start it with tares up first.

3. Post a sample Grafana payload that describes a slow query alert. Replace the ingest key with your own.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"receiver": "tares", "status": "firing", "orgId": 1, "alerts": [{"status": "firing", "labels": {"alertname": "SlowQueries", "severity": "critical"}, "annotations": {"summary": "Database queries are slow"}, "startsAt": "2024-01-01T00:00:00Z"}], "title": "SlowQueries", "state": "alerting", "message": "Database queries are slow"}'

4. If your Tares has auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. Add that header to the curl call with -H. Without auth, no header is needed.

5. Open the source page. Check Recent events for the new event, then read Fields to see which payload fields arrived and how often. Confirm that the event type, the text and the labels match what you declared, and adjust the labels if not.

6. Send a second sample with "status": "resolved" to confirm that the resolved notification shows up as its own event.

7. Connect Grafana. Add a Webhook contact point with the Tares ingest URL and the POST method. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key. Then route your slow query alert rule to that contact point from a notification policy.

If you prefer one event per alert, repeat steps 1 to 5 with an alertmanager source and send the same payload to its ingest URL. The alertmanager page shows that source’s test batch, and its Fields view lists the alert labels you can promote to correlate on.

What you get

• A confirmed mapping from Grafana’s payload to Tares labels, event type and text, checked before an incident.
• A visible event on the source’s Recent events list for both the firing and the resolved state.
• A Fields view that shows which payload fields arrive, so you pick a reliable primary label.
• A contact point and notification policy that send real slow query alerts to the same source you already tested.
• The choice between one event per notification on a webhook source and one event per alert on an alertmanager source.

When something else fits

• Use an alertmanager source instead of a webhook source when you want each alert as its own event with its labels mapped, since it fans out the alerts array.
• Test with a real Grafana notification once the sample looks right, because only a live rule proves the contact point, notification policy and auth header together.

Related