What should Alertmanager alert groups about a cron job that stopped include for an AI SRE?

Tares gets the most from a stopped-cron alert when it carries an alert name, a service label, a severity label, a summary annotation and a start time. Tares turns each alert in a group into one event, so these fields decide whether the agent can tie the alert to logs, metrics and deploys. Set send_resolved: true so the recovery lands on the same timeline.

How it works

Alertmanager POSTs a batch to the Tares ingest URL. Tares fans the alerts array out into one event per alert, so a group of five alerts becomes five events. Each field you send has a job:

• Alert name. It becomes the event type, so name the failure plainly, for example the cron job and what it did (stopped running).
• Summary annotation. It becomes the event text, the line the agent reads first. Say which job stopped and where.
• Labels. Every alert label becomes a Tares label, and Tares adds a status label that is firing or resolved.
• Severity label. It lets a trigger or an agent tell a critical stop from a warning.
• Start time. The alert’s startsAt stays in the stored payload with the rest of the alert, including its fingerprint. The agent can compare it with deploys and log lines on the same clock.
• Service label. This is the one that ties the alert to evidence. Tares builds one timeline per entity, and the entity key comes from a label in this order: namespace, service, job, instance, or a label you declare as primary.

The order matters. If your alert carries both namespace and service, Tares keys on namespace by default. To key on the service, set the key_label option on the source to the label you want. The AI SRE guide keeps metrics, logs and alerts keyed by the same service label, so the stopped-cron alert lands next to that service’s other evidence.

Set it up

1. In the console, go to Sources, then Add source, then Alertmanager alerts. Name it and create it. You can also ask a connected agent to create an Alertmanager source and give you the webhook URL. The source page shows a receiver block with your ingest URL filled in.
2. Add the receiver to alertmanager.yml and reload Alertmanager. Keep send_resolved: true so the agent sees the cron job start again.

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If your Tares instance runs with auth on, the sender needs a Bearer Authorization header holding an ingest-scoped API key. Without auth, the receiver block above is enough.
4. In your alert rule, put the fields from the list above on every stopped-cron alert: a clear alert name, a service label, a severity label and a summary annotation. Alertmanager supplies the start time itself.
5. If you also send namespace, set key_label on the source to service so the alert keys on the service.
6. Send a test batch shaped like a stopped-cron alert, then check Recent events:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"NightlyBackupStopped","severity":"critical","service":"backup"},"annotations":{"summary":"The nightly backup job stopped running"},"startsAt":"2024-01-01T00:00:00Z"}]}'

7. Open Fields on the source. It lists the alert labels, so you can promote service and correlate on it.

The Alertmanager connector page has the full receiver setup and the key_label option.

What you get

• One event per alert, with the alert name as the event type and your summary as the text, so the agent reads a plain sentence about the stopped job.
• A status label that flips from firing to resolved, so the agent knows whether the cron job is back.
• The alert on the same per-service timeline as your metrics, logs and deploys, which lets the agent place the stop against what changed just before it.
• The full alert kept in the payload, including its start time and fingerprint, for anything you did not promote to a label.
• A severity you can filter on in a trigger, so a critical stop wakes an agent and a low-priority one stays on the timeline.

When something else fits

• If you do not run Alertmanager and only want the alerts Prometheus rules have fired, the prometheus_alerts connector polls them, but you lose Alertmanager’s grouping, silencing and real-time delivery.
• If the sender is not Alertmanager, such as a deploy pipeline or a custom job runner, the webhook connector takes any JSON you post and lets you choose the labels and text template yourself.
• If one alert group mixes many unrelated jobs, split them with separate labels or routes, so each stopped job keys to the right service.

Related