How do I wake an AI SRE once when alerts from Prometheus report a cron job that stopped?

Tares wakes your AI SRE once per stopped cron job when you poll Prometheus alerts with the prometheus_alerts connector and add a trigger with a cooldown. The trigger evaluates per entity, so a repeating alert starts one investigation instead of one per poll. When the alert clears, Tares records a resolved event on the source’s timeline.

How it works

• The prometheus_alerts connector polls the alerts your Prometheus rules have already fired, every 30 seconds by default. Prometheus decides what counts as an alert, and Tares needs no Alertmanager and no PromQL.
• State is tracked across polls. Each alert fires once and re-emits only when its state changes, so you do not get one copy per tick.
• Each alert becomes an event with event_type set to the alert name, text set to its summary, its labels as Tares labels, and a synthetic state of firing, pending or resolved.
• A trigger watches the source and evaluates a condition as events arrive. With group_by: [key_value] it evaluates per entity, so each cron job fires independently.
• cooldown sets the minimum gap between firings for the same entity. If the alert flaps or keeps repeating inside that gap, the same cron job does not wake the agent again.
• When the alert disappears, the connector emits a resolved event. You see it under Recent events on the source, next to the firing event for the same alert.

The label that names the cron job is the primary label on the source and the key_field on the trigger. Use the same label in both, so the entity is the same everywhere. The examples below use service, taken from the label your stopped-cron alert carries.

Set it up

1. Add the source. Ask a connected agent to ingest the firing alerts from your Prometheus, or use Sources → Add source → Prometheus alerts, enter the url and choose Discover. Discover proposes only severity and state, because an alert’s dynamic labels exist only while it fires. Once real alerts have been ingested, add the label that names the cron job from the Fields tab. Mark that label as primary and keep severity as a plain label.

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: service, field: labels.service, primary: true }
      - { name: severity, field: labels.severity }
      - { name: state, field: state }

2. Define the trigger on the project’s page, under Triggers → Add trigger, or in the catalog YAML. It counts firing events for your cron alert, per service, and waits 30 minutes before firing again for the same one. Replace the event_type value with the name of your alert rule.

triggers:
  - name: cron_stopped
    sources: [alerts]
    key_field: service
    filters:
      - { field: event_type, op: eq, value: CronJobStopped }
      - { field: state, op: eq, value: firing }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: cron_stopped
      context_window: 15m
    cooldown: 30m

3. Wire the agent. A Tares agent is woken by the trigger in a project that wires it there, once you turn it on. An external agent registers a webhook with the subscribe tool over MCP, giving the trigger name and its endpoint URL.

4. Verify. Open the source and check Recent events and Fields. Open the trigger’s page, or Firings in the console, to see each firing and its delivery status.

Each dispatch is a JSON body with trigger, key, fired_at and a payload holding the entity’s correlated timeline over context_window. Delivery is at-least-once, so dedupe on dispatch_id and acknowledge with a 2xx.

What you get

• One investigation per stopped cron job. The condition is evaluated per service, and the cooldown holds back repeat firings for that same job.
• Independent handling. Two different cron jobs stopping in the same window each wake the agent once.
• A timeline you can read. The firing event and the later resolved event both sit on the source, so you can see when the job stopped and when the alert cleared.
• Context in the dispatch. The payload carries the entity’s correlated timeline across the trigger’s sources, so the agent does not need a follow-up call.
• A record of every firing, with per-subscriber delivery status, even when nobody is subscribed.

The trigger above filters on state equal to firing, so resolved events do not count toward the condition and do not wake the agent. They are still stored and listed on the source.

When something else fits

• If you want Tares to look at the source on a clock instead of on a condition, give the trigger a schedule such as every: 10m. See Watch, then escalate.
• If you need the raw alert stream with its correlation across several sources, start from the AI SRE guide and add those sources to the same trigger.
• If your alerts already flow through Alertmanager and you want them pushed, the alertmanager connector takes them at a Tares ingest URL. Polling with prometheus_alerts is simpler when Tares can reach Prometheus directly.

Related