How do I route Alertmanager alert groups about a cron job that stopped to an AI SRE?

Tares turns each alert in an Alertmanager group into its own event, so you can add a trigger that matches only the alert name for your stopped cron job. Only matching alerts fire the trigger and wake the agent subscribed to it. Name the trigger after the symptom so every firing reads as that symptom.

How it works

• Alertmanager POSTs a batch to the ingest URL of an alertmanager source. Tares fans the alerts array out into one event per alert.
• Each event has event_type set to the alert name, text set to the alert summary, and the alert labels as Tares labels, plus a synthetic status of firing or resolved. The full alert, including fingerprint, stays in the payload.
• The entity key comes from a label, checked in this order: namespace, then service, then job, then instance, or a primary label you declare. If namespace is present on the alert, it wins. To key on a different label, set key_label on the source.
• A trigger lists the source in sources, filters on event_type for the alert name of your stopped cron job, and wakes the agent subscribed to it. Alerts for other symptoms never match, so they never wake that agent.
• Alertmanager has already grouped, deduped and tracked firing to resolved before Tares sees the alert. Tares records what Alertmanager sends and correlates it with your other sources.

Set it up

1. Create the source. Go to Sources → Add source → Alertmanager alerts, name it and click Create. Or ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” The source page shows a receiver block with the ingest URL filled in.
2. Add the receiver to your alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. The sender uses a Bearer Authorization header holding an ingest-scoped API key. You need it only when Tares has auth on. The source page shows the auth block for you in that case.
4. Optional: if you want Tares to key the events on a label other than namespace, set key_label on the source. Without it, namespace wins whenever the alert carries it.

- name: alerts
  connector: alertmanager
  config:
    key_label: job

5. Send a test alert shaped like your cron alert, then check Recent events. The event type should be your alert name. Fields shows the labels, so you can promote one to correlate on.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"CronJobStopped","severity":"critical","namespace":"prod"},"annotations":{"summary":"Nightly cron job has stopped"},"startsAt":"2024-01-01T00:00:00Z"}]}'

6. Create the trigger. Replace CronJobStopped with the alert name from your own Prometheus rule. The trigger name states the symptom, so its firings on the trigger page and under Firings read as a stopped cron job.

- name: cron-job-stopped
  sources: [alerts]
  filters:
    - { field: event_type, op: eq, value: CronJobStopped }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 5m

7. Subscribe a Tares agent to the trigger, or point it at your own agent service over a webhook or a Slack channel. Only firings of cron-job-stopped wake it. Write the agent prompt for this symptom, for example: check whether the job ran, when it last ran, and what changed before it stopped.
8. If you have several alert names for the same symptom, list them with op: in and a list of values instead of eq.

What you get

• The agent wakes only for the one symptom you chose. Disk, latency and other alerts that share the Alertmanager receiver do not wake it.
• Each firing is recorded under a trigger named for the symptom, so you can tell at a glance what woke the agent.
• The agent reads the stopped-job alert next to your other sources on one timeline, so it can compare it with deploys, logs and metrics. Use the AI SRE guide to wire those sources and choose who gets woken.
• The cooldown stops a burst of repeated alerts from waking the agent again right away.
• Alertmanager keeps owning grouping, silencing and resolved tracking. Tares does not re-threshold anything.

When something else fits

• If the cron job itself can send an HTTP request, post to a webhook source from the job. This works when you have no alert rule for it.
• If you want the agent to see every alert, drop the event_type filter and let the whole stream through, or use several triggers with different filters.
• For a Prometheus rule you want Tares to poll without Alertmanager, use the prometheus_alerts connector. Alertmanager gives you real-time delivery and its grouping and silencing.

Related