How do I test Grafana alert rules about high latency before a real incident?

Tares lets you test a Grafana high latency alert before an incident by posting a sample Grafana payload to the ingest URL of a webhook source. Then you check the label mapping under Fields and the event under Recent events. Grafana Alerting reaches Tares through the webhook connector and a Grafana-side setup, because Tares has no connector of its own for Grafana today.

How it works

Grafana sends alert notifications to a Grafana webhook contact point. In Tares, a webhook source receives that POST at its ingest endpoint and stores the payload lossless.

• A Grafana notification arrives as one event, with the alerts list inside the stored payload.
• Top-level fields such as status, receiver and title work for labels and for text_template.
• The labels you declare map payload fields into the event, and one label is marked primary.
• For a push source, test_source takes no payload and only returns the ingest endpoint. It does not check the label mapping, so you post a sample payload and read the source page instead.

The default Grafana payload is Alertmanager compatible. If you prefer one event per alert, a separate alertmanager source with its own ingest URL can take the same Grafana payload.

Set it up

1. Create a webhook source in Tares. Use Sources → Add source → webhook, or ask a connected agent to create the source and return the ingest URL. Declare labels for the top-level fields you want to correlate on, and mark one as primary:

- name: grafana-alerts
  connector: webhook
  config:
    event_type_field: status
    text_template: '{title}: {status}'
    labels:
      - { name: receiver, field: receiver, primary: true }
      - { name: status, field: status }

2. Copy the ingest endpoint from the source page.

3. Post a sample payload shaped like a Grafana notification, with a latency alert in the alerts list:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"receiver": "tares", "status": "firing", "orgId": 1, "alerts": [{"status": "firing", "labels": {"alertname": "HighLatency"}, "annotations": {"summary": "p95 latency is high"}, "startsAt": "2024-01-01T00:00:00Z"}], "title": "High latency", "state": "alerting", "message": "p95 latency is high"}'

4. If Tares has auth on, the sender needs an Authorization header with the Bearer scheme, holding an ingest-scoped API key. Creating a push source in the console mints one and shows it once.

5. Open the source page. Under Recent events, confirm the event arrived with the event type and text you expect. Under Fields, confirm the payload fields you mapped show up and pick a reliable primary label. Change the labels or text_template and post again until the event looks right.

6. Wire up Grafana. Add a Webhook contact point with the Tares ingest URL and the POST method. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to the ingest-scoped API key. Then route alerts to the contact point from a notification policy.

To get one event per alert instead, create an alertmanager source and post the same payload to its ingest URL. Its page shows each alert’s labels plus a synthetic status under Fields.

What you get

• A label mapping you have checked on a sample payload, so the labels and text template are settled before an incident.
• A visible event under Recent events, with the full Grafana payload kept lossless, including the alerts list.
• A Fields view that shows which payload fields arrive and how often, so you pick a reliable key.
• A contact point and notification policy in Grafana that deliver to the same ingest URL you tested. The AI SRE guide shows what to build on the events once they flow.

When something else fits

• If you want one event per alert, with the alert’s name as the event type, use the alertmanager connector and post the same Grafana payload to its ingest URL.
• If your high latency alerts come from Prometheus rules and an Alertmanager you already run, point Alertmanager at Tares directly and skip the Grafana contact point.

Related