How do I test Grafana alert rules about a disk filling up before a real incident?

Tares lets you test a Grafana disk-filling alert before a real incident by posting a sample Grafana payload to a webhook source’s ingest URL. You then check the label mapping on the Fields tab and the event in Recent events. test_source only returns the ingest endpoint for a push source, so the sample post is what checks your mapping.

How it works

Tares has no connector of its own for Grafana Alerting today. Grafana reaches Tares through the webhook connector and a vendor setup: a Grafana Webhook contact point POSTs to the source’s ingest URL.

• Grafana sends one notification as a JSON body with receiver, status, alerts, groupLabels, commonLabels, title, state and message.
• Tares stores that notification as one event, with the alerts list kept inside the lossless payload.
• Top-level fields such as status, receiver and title work for labels and for text_template.
• A one-level dotted field such as commonLabels.service is a valid label path.

Because the test is a plain POST, you can run it before any Grafana rule fires. If you ask a connected agent to run test_source on a push source, it takes no payload and returns only the ingest endpoint. It does not check the label mapping. To check the mapping, post a sample payload and read Recent events and Fields on the source page.

Part of the AI SRE workflow is knowing what an alert will look like in Tares before it matters, and this test gives you that.

Set it up

1. Create a webhook source. Ask a connected agent to create a webhook source for your Grafana alerts, or use Sources → Add source → webhook. Declare the labels below and mark service as primary:

- name: grafana-alerts
  connector: webhook
  config:
    text_template: '{status}: {title}'
    labels:
      - { name: service, field: commonLabels.service, primary: true }
      - { name: receiver, field: receiver }
      - { name: status, field: status }

The service label is the primary label, so events are keyed by the service your alert rule names. receiver and status are plain labels and are not primary.

2. Copy the ingest endpoint from the source page. It has the form /ingest/<ingest_key>.

3. Post a sample Grafana notification for a disk filling up. The shape follows the default Grafana payload, with the service in commonLabels:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"receiver": "tares", "status": "firing", "orgId": 1, "alerts": [{"status": "firing", "labels": {"alertname": "DiskFillingUp", "service": "api-server"}, "annotations": {"summary": "Disk is filling up"}, "startsAt": "2024-01-01T00:00:00Z"}], "commonLabels": {"alertname": "DiskFillingUp", "service": "api-server"}, "title": "DiskFillingUp", "state": "alerting", "message": "Disk is filling up"}'

If your instance has auth on, send an ingest-scoped API key as a Bearer Authorization header on the request. Without auth you do not need it.

4. Open the source page. Check Recent events for one event with your text_template rendering, and check Fields to see that commonLabels.service, receiver and status arrive with the values you expect. If a label shows no value, fix its field path and post again.

5. In Grafana, add a Webhook contact point with the Tares ingest URL and the POST method. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key. Make sure the disk alert rule sets a service label, and route alerts to the contact point from a notification policy that groups alerts by service. That keeps commonLabels.service meaningful on real notifications.

If you want one event per alert instead of one per notification, create a separate alertmanager source. It has its own ingest URL and takes the same Grafana payload, because the default Grafana payload is Alertmanager compatible. Test it the same way, with a batch in the alerts array, and check Recent events and Fields.

What you get

• You see how a disk-filling alert lands in Tares before a real incident does, with no waiting for a rule to fire.
• You confirm that the service, receiver and status labels carry the values you intended.
• You see the rendered event text on the timeline, so you can adjust text_template first.
• You have a repeatable curl command. Change the status to resolved and post again to see how a resolved notification looks.
• You have a working contact point and notification policy to point at the same source when you go live.

When something else fits

• Use an alertmanager source instead of the webhook source when you want one event per alert, with the alert name as the event type and the alert labels as Tares labels.
• If Grafana alerts already flow through an Alertmanager you run, point that Alertmanager at an alertmanager source rather than adding a second path from Grafana.
• If you only want to confirm that the ingest endpoint is reachable, the endpoint test_source returns is enough. It does not check labels.

Related