What should Grafana alert rules about a cron job that stopped include for an AI SRE?

Grafana alert rules about a stopped cron job should set a service label, a severity label and a summary annotation, so Tares can tie the alert to evidence. Tares keys each event on the service label, which lets your agent line the alert up with that service’s metrics, logs and deploys. The start time comes with every alert Grafana sends, so you do not have to add it.

How it works

Grafana Alerting reaches Tares through the webhook connector and a Grafana contact point. Tares has no connector of its own for Grafana Alerting today.

• Grafana posts one notification per group of alerts. Tares stores it as one event and keeps the whole payload lossless, including the alerts list.
• Each alert in that list carries its labels, its annotations, startsAt, endsAt and a fingerprint. Your agent can read the summary annotation and the start time from there.
• Top-level fields such as status, receiver and title work for labels and for text_template.
• commonLabels holds only the labels shared by every alert in the notification. For commonLabels.service to be filled, every alert rule must set a service label, and the notification policy must group alerts by service. Then each notification covers one service.

The service label is what places the event on a service’s timeline. If your metrics, logs and deploy sources use the same service label with the same values, the cron job alert appears next to that evidence. This is the per-service timeline described in Build an AI SRE.

Set it up

1. In each Grafana alert rule for a cron job, set these labels and annotations:
   • Label service: the name of the service that owns the job, spelled exactly as in your other Tares sources.
   • Label severity: for example critical or warning.
   • Annotation summary: a plain sentence that says which job stopped and what it was expected to do, for example that the nightly invoice job has not run.
   • The start time needs no setup. Grafana sends startsAt for every alert.
2. In the notification policy that handles these rules, group alerts by service.
3. In Tares, go to Sources → Add source → webhook and create a source like this. The service label is primary:

- name: grafana-alerts
  connector: webhook
  config:
    text_template: '{status}: {title}'
    labels:
      - { name: service, field: commonLabels.service, primary: true }
      - { name: severity, field: commonLabels.severity }
      - { name: status, field: status }

4. Copy the ingest endpoint from the source page. In Grafana, add a Webhook contact point with that URL and the POST method.
5. When Tares has auth on, set the contact point’s Authorization header scheme to Bearer and the credentials to an ingest-scoped API key.
6. Route the cron job alerts to the contact point from the notification policy.
7. Check the mapping by posting a sample notification, then open Recent events and Fields on the source page:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"status":"firing","title":"Nightly invoice job stopped","commonLabels":{"service":"billing","severity":"critical"}}'

See the webhook connector page for the full set of options.

What you get

• Every stopped-cron notification lands as one event on the timeline of the service named in the service label.
• The event text reads from status and title, so a firing alert and its resolved notice are easy to tell apart.
• The severity is available as a label on the event.
• The full payload stays intact, so your agent can read the summary annotation and startsAt for each alert and compare that start time with the last good log lines, metrics and deploys on the same timeline.
• When the alert and your other sources share the service label, the agent reads one correlated view and can tie the stopped job to a specific change or error instead of guessing.

When something else fits

• If you want one Tares event per alert instead of one per notification, create a separate alertmanager source with its own ingest URL. Grafana’s default payload is Alertmanager compatible, so it can take the same notifications.
• If your policy cannot group by service, so one notification mixes several services, commonLabels.service will be empty for the services that differ. Group by service first, or use the alertmanager source.

Related