How do I test Grafana alert rules about a cron job that stopped before a real incident?

Tares lets you test a Grafana alert for a stopped cron job by posting a sample Grafana payload to a webhook source’s ingest URL and reading the result in Recent events and Fields. Grafana reaches Tares through the webhook connector and a vendor setup, and Tares has no connector of its own for Grafana today. For a push source, test_source only returns the ingest endpoint and does not check label mapping, so the sample post is the real check. You confirm the mapping first and connect the Grafana contact point afterward.

How it works

Grafana sends a notification as one JSON object to a Webhook contact point. The object has top-level fields such as receiver, status, title, state and message, plus an alerts list. Tares receives that object on a webhook source and stores it lossless, with the alerts list inside the payload.

• Top-level fields such as status, receiver and title work for labels and for the text template.
• You declare one label per field you want to correlate on and mark one as primary.
• A push source has no polling. Whatever you post to its ingest endpoint becomes an event, so a hand-made sample behaves the same as a real Grafana delivery.
• test_source on a push source takes no payload and returns only the ingest endpoint. To check the mapping, post a sample and read Recent events and Fields on the source page.

Grafana’s default payload is compatible with Alertmanager. If you would rather get one event per alert, create a separate alertmanager source, which has its own ingest URL, and post the same Grafana payload there. The webhook source gives you one event per notification.

For the wider picture of how alert sources feed investigation, see AI SRE.

Set it up

1. Create the source. Ask a connected agent: “Use tares to create a webhook source for my Grafana alerts and give me the ingest URL.” Or go to Sources → Add source → webhook in the console. Declare labels for the top-level fields your Grafana payload carries, with one primary:

- name: grafana-cron
  connector: webhook
  config:
    event_type_field: status
    text_template: '{title}: {message}'
    labels:
      - { name: receiver, field: receiver, primary: true }
      - { name: status, field: status }

2. Copy the ingest endpoint from the source page.

3. Post a sample that looks like what Grafana sends when your cron rule fires. Replace the ingest key with yours:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"receiver":"tares","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"CronJobStopped","severity":"critical","namespace":"prod"},"annotations":{"summary":"Nightly backup cron has not run"},"startsAt":"2024-01-01T00:00:00Z"}],"title":"Cron job stopped","state":"alerting","message":"Nightly backup cron has not run"}'

If your instance runs with auth on, the request also needs an Authorization header with the scheme Bearer and an ingest-scoped API key. Add it with curl’s header option.

4. Open the source page. In Recent events, confirm the event arrived and that its event type and text read the way you want. In Fields, confirm that receiver, status and the other fields you mapped show up. If a label is empty or the text is wrong, edit the source and post the sample again.

5. Post the same sample with status set to resolved. This shows how the recovery notification looks next to the firing one.

6. Connect Grafana. Add a Webhook contact point with the Tares ingest URL and the POST method. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key. Then route alerts to the contact point from a notification policy.

What you get

• A verified label mapping before any real incident. You see which fields arrive and how often, so you pick a reliable primary label.
• A known event shape for a stopped cron job, firing and resolved, so you can write triggers and agent prompts against fields you have already seen.
• A source that is ready the moment the Grafana contact point is live. Real notifications land on the same source and follow the same mapping.
• A clear choice between one event per notification (webhook source) and one event per alert (alertmanager source), made while the stakes are low.

When something else fits

• If you want one event per alert and Alertmanager-style resolved tracking from Grafana, use a separate alertmanager source and post the same payload to its ingest URL.
• If the cron job runs under a system you can already poll or tail, a different connector can watch it directly instead of waiting for a Grafana notification.

Related