What should Prometheus alerts about a cron job that stopped include for an AI SRE?

Tares gives an AI SRE the most to work with when a stopped-cron-job alert carries a service label, a severity label, a summary annotation and a clear firing state. The service label ties the alert to the job’s metrics and logs on one timeline. The summary becomes the event text, and severity lets you filter. Tares polls the alerts your Prometheus rules have fired and keeps each one as an event with its state changes.

How it works

The prometheus_alerts connector polls the alerts endpoint of your Prometheus server, every 30 seconds by default. Prometheus decides what counts as an alert. Tares ingests what fired, correlates it with the other evidence and can wake an agent. Read more in Prometheus alerts.

Each active alert becomes one event:

• event_type is the alert name, so name the rule after the job, for example a name that says the nightly backup stopped.
• text is the alert’s summary.
• The alert’s labels become Tares labels, plus a synthetic state of firing, pending or resolved.

Tares tracks state across polls. An alert fires once and emits again only when its state changes. A resolved event arrives when it clears. The agent sees when the alert started and, later, when it ended, without a copy on every tick.

What to put on the alert for a stopped cron job:

• A service label. Use the same name your metrics and logs carry for the job or the service that runs it. This is the label Tares correlates on. In an AI SRE setup, all sources are keyed by the same service label, which gives one timeline per service with metrics, logs and alerts on one clock.
• A severity label. It tells the agent how urgent the alert is. You can also set severities on the source, for example critical,warning, to ingest only those.
• A summary annotation. One plain sentence such as which job stopped and what it should have done. It arrives as the event text, so the agent reads it first.
• A start time. Each state change is an event on the timeline, so the agent can place the failure against logs and metrics and see when it cleared. If you also want alerts that are still waiting out their for: duration, turn on include_pending.

Set it up

1. Start Tares and open the console.

tares up

2. Go to Sources → Add source → Prometheus alerts. Enter the url of your Prometheus server and select Discover. It lists your alerting rules and the severities present, and it proposes the labels that are always there, severity and state.
3. Make sure the rule for the stopped cron job sets a service label (and a severity label) and a summary annotation. A rule that omits the service label leaves the alert off the job’s timeline.
4. Let the alert fire once. An alert’s dynamic labels such as service only exist while it is firing. Open the Fields tab and promote service so you can correlate on it.
5. Declare service as the primary label and keep severity as a plain label:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: service, field: labels.service, primary: true }
      - { name: severity, field: labels.severity }
      - { name: state, field: state }

6. Key your metrics and log sources by the same service label, so the alert, the job’s logs and its metrics land on one timeline.
7. Open the source and check Recent events. You see one event per active alert and a resolved event when it clears. Fields shows the labels the alerts carry.

You can also ask a connected agent to ingest the firing alerts from your Prometheus. It runs discovery and then creates the source for you.

What you get

• An event per stopped-cron-job alert, named after the alert, with the summary as its text.
• The alert on the same per-service timeline as the job’s metrics and logs, so an agent can tie the alert to evidence instead of reading it alone.
• A single firing event, then a resolved event when the job runs again, rather than a repeat on every poll.
• A severity label you can filter on at the source.
• An agent that can read what broke and since when, and write its finding back to that timeline for the next investigation.

When something else fits

• If your alerts already go through Alertmanager and you want them pushed to Tares, the alertmanager connector takes them at an ingest URL instead of polling Prometheus.
• If a cron job can only report by calling a URL when it finishes or fails, a webhook source fits better than a Prometheus rule.

Related