How do I wake an AI SRE once when Alertmanager alert groups report a cron job that stopped?

Tares wakes your AI SRE once per stopped cron job when you pair the alertmanager connector with a trigger that has a cooldown. Each alert Alertmanager sends becomes one event, and the trigger’s cooldown allows one firing per entity in that gap. Resolved alerts are still recorded on the entity’s timeline with a status of resolved.

How it works

The pieces fit together like this:

• Alertmanager sends batches. You point a webhook_configs receiver at the ingest URL of an alertmanager source. Tares fans out the alerts array into one event per alert. The event_type is the alert name, the text is the summary, and the alert labels become Tares labels plus a synthetic status of firing or resolved. The full alert, including its fingerprint, stays in the payload.
• Each alert gets an entity key. Tares derives it from a label, in the order namespace, service, job, instance, or from a label you declare primary. To key on something else, set key_label on the source. For a cron job, make sure the key names the job, so each job is its own entity.
• The trigger watches the source. It checks its condition as events arrive, and group_by: [key_value] evaluates it per entity.
• The cooldown holds repeats back. cooldown is the minimum gap between firings for the same entity. A repeating alert for the same stopped cron job fires one investigation, and a different job still fires on its own.
• The agent gets context. When the trigger fires, Tares POSTs a dispatch to each subscriber. Its payload is the entity’s correlated timeline across the trigger’s sources over context_window, so the agent starts with the evidence already in hand.

For the wider picture of how agents investigate, see AI SRE.

Set it up

1. Create the source. In Sources → Add source → Alertmanager alerts, name it and Create. Or ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.”

2. Add the receiver to alertmanager.yml and reload Alertmanager. Keep send_resolved: true so resolved alerts reach Tares too:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. Add authentication if Tares has auth on. The sender uses a Bearer Authorization header holding an ingest-scoped API key, set under http_config in the webhook_configs entry. It is needed only on a secured instance (tares up --auth).

4. Check the pipeline. Send a test batch, then look at Recent events:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

Fields shows the alert labels, so you can promote the one that names your cron job and correlate on it.

5. Define the trigger. Add it under Triggers → Add trigger or in the catalog YAML. Replace the source name and the alert name (CronJobStopped here) with yours:

triggers:
  - name: cron_stopped
    sources: [cron-alerts]
    filters:
      - { field: event_type, op: eq, value: CronJobStopped }
      - { field: status, op: eq, value: firing }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: cron_stopped
      context_window: 15m
    cooldown: 1h

6. Subscribe the agent. A Tares agent wakes when a project wires it to the trigger and you turn it on there. An external agent calls subscribe(trigger, url) over MCP, or registers with POST /subscribe. Your endpoint must answer with a 2xx and should dedupe on dispatch_id, because delivery is at-least-once.

The Alertmanager connector page covers the receiver and key_label in full. The triggers guide covers conditions, subscriptions and delivery.

What you get

• One investigation per stopped cron job within the cooldown, however many times Alertmanager repeats the alert.
• Independent firing per job, because the condition and the cooldown work per entity.
• A dispatch whose payload is the correlated timeline over context_window, so the agent does not need a follow-up call.
• A record of every firing, even with zero subscribers, under Firings in the console, on the trigger’s own page and on the project’s Activity. Each dispatch has a page with the delivery outcome and the exact payload sent.
• Resolved alerts recorded as events with status set to resolved, so the timeline shows when the job came back. The status filter in step 5 keeps them from counting toward a firing. Filters also narrow which events are handed over, so a firing-only filter keeps resolved events out of the dispatch payload. They stay recorded on the source, and you see them in Recent events. To hand resolved events to the agent as well, drop the status filter. A resolved event then counts toward the condition like any other event.

When something else fits

• If you want a trigger to fire on a metric threshold instead of an alert, use a metrics source and a numeric condition as in the triggers guide.
• If you need Prometheus alerts without running Alertmanager, the prometheus_alerts connector polls them, but it does not give you Alertmanager’s grouping, silencing and real-time delivery.
• If you want a periodic look at the cron alerts rather than a wake-up on each one, give the trigger a schedule with condition: {every: 10m}.

Related