How do I route alerts from Prometheus about a cron job that stopped to an AI SRE?

Tares polls the alerts your Prometheus rules have fired with the prometheus_alerts connector, and a trigger filtered to one alert name wakes your AI SRE only for the cron job that stopped. Other alerts still land on the service timeline but do not wake the agent. You name the trigger after the symptom, so every firing reads as that symptom. The agent you subscribe to it starts from the correlated timeline for the affected service.

How it works

• Prometheus keeps owning alerting. Your rule for the stopped cron job fires in Prometheus, and the prometheus_alerts source polls /api/v1/alerts every 30 seconds by default.
• Each active alert becomes one event. Its event_type is the alert name, its text is the summary, and its labels become Tares labels plus a state of firing, pending or resolved.
• State is tracked across polls, so an alert fires once and emits a resolved event when it clears. It does not repeat every tick.
• The source declares service as its primary label, read from labels.service. Your alerting rule should put the cron job’s name or its owning service in that label. severity stays a plain label.
• A trigger narrows the stream to the symptom with filters on event_type and state. When the condition holds, it wakes whatever is subscribed to it. The Build an AI SRE guide shows the full chain: source, trigger, agent, finding written back to the timeline.

Set it up

1. Make sure your Prometheus alerting rule for the stopped cron job carries a service label. Note the rule’s alert name. This page uses CronJobStopped as the example, so replace it with your own.
2. Add the source under Sources → Add source → Prometheus alerts. Enter the url and click Discover, which lists your alerting rules and the severities present. Discover proposes only severity and state. Once real alerts have been ingested, add service from the Fields tab and mark it primary. The source ends up like this:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: service, field: labels.service, primary: true }
      - { name: severity, field: labels.severity }
      - { name: state, field: state }

You can also ask a connected agent to ingest the firing alerts from your Prometheus, or POST /api/sources with the same object. See the Prometheus alerts connector for every field. 3. Open the source and check Recent events. You should see one event per active alert, with CronJobStopped as the event type when that rule fires. 4. Create the trigger. Name it for the symptom, watch the alerts source, and filter on the alert name and on firing so pending and resolved events do not wake the agent:

- name: cron-job-stopped
  sources: [alerts]
  filters:
    - { field: event_type, op: eq, value: CronJobStopped }
    - { field: state, op: eq, value: firing }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 15m

5. Subscribe the responder to the trigger. Use a Tares agent with a prompt written for stopped jobs, your own agent service over a webhook, or a Slack channel. The Build an AI SRE guide covers all three choices.
6. Open the trigger’s page to see its firings. When your rule fires, you see a firing under cron-job-stopped and the agent run it woke.

What you get

• Only the alerts you named wake the agent. Latency, error-rate and other alerts from the same Prometheus stay on the timeline without waking it.
• Each firing is recorded under a trigger called cron-job-stopped, so the symptom is clear in the console without opening the alert.
• The agent reads one timeline for the affected service, with alerts, metrics and logs merged on one clock when you have those sources keyed by the same service label.
• The cooldown keeps a flapping cron alert from waking the agent repeatedly.
• When the alert clears, a resolved event lands on the timeline, so the agent and your team see when the job recovered.

When something else fits

• If Prometheus does not evaluate the rule that detects the stopped job, add a rule there first. This route ingests alerts Prometheus has already fired.
• If the job can send an HTTP request itself, it can POST JSON to a webhook source and land a success or failure event on the same timeline. That event is useful context for the agent, though it is a separate source from your Prometheus alerts.
• If your alerts reach you through Alertmanager rather than the Prometheus alerts endpoint, use the alertmanager connector instead and keep the same trigger shape.

Related