What should an AI SRE read next to alerts from Prometheus for a cron job that stopped?

Tares puts a Prometheus alert for a stopped cron job on one timeline beside the same service’s metrics, logs and changes, so an AI SRE reads that timeline next. The alert says that something stopped. The neighbouring evidence says when it stopped and why. The agent cites those lines in its diagnosis instead of guessing from the alert text.

How it works

The prometheus_alerts connector polls Prometheus’s own alerts endpoint, by default every 30 seconds. It reads the alerts your alerting rules have already fired, so Prometheus keeps deciding what counts as an alert and Tares does not re-threshold anything.

Each alert becomes an event. The event_type is the alert name, the text is its summary, and the alert’s labels become Tares labels, plus a state of firing, pending or resolved. Tares tracks state across polls, so a stopped cron job produces one firing event and a resolved event when it clears, not a copy on every tick.

The alert joins a timeline through the service label. service is a label the firing alert carries, so your rule must set it. Give your metrics source, your log source and any deploy source the same service label and they merge on one clock with the alert. This is what an AI SRE reads next, in this order:

• The service’s metrics. Read what the job last reported and when the numbers flattened, so you can tell a job that never ran from one that ran and failed.
• The service’s logs. Read the last lines before the gap. Logs reach Tares from a container with the docker_logs connector or from your OpenTelemetry pipeline over OTLP.
• Recent deploys and config changes. Most real root causes are a change. A webhook source with event_type: deploy puts each deploy on the service’s timeline, and a second one with event_type: config covers config changes.
• Earlier findings. Agents write their conclusions back to the timeline, so a new investigation starts from the previous one.

Set it up

1. Make sure the Prometheus alerting rule for the cron job sets a service label. Dynamic labels such as service exist only while an alert is firing, and discovery proposes only severity and state.

2. Start Tares and open the console, then add the alerts source under Sources → Add source → Prometheus alerts. Enter the url and use Discover to list your alerting rules.

   tares up

3. Once a real alert has been ingested, open the source’s Fields tab and promote service to the primary label. The source then looks like this:

   - name: alerts
     connector: prometheus_alerts
     poll: 30s
     config:
       url: http://localhost:9090
       severities: critical,warning   # blank = all
       labels:
         - { name: service, field: labels.service, primary: true }
         - { name: severity, field: labels.severity }
         - { name: state, field: state }

4. Give your metrics source, your log source and your deploy webhook source the same primary service label, so every signal lands on the same per-service timeline.

5. Start the MCP endpoint in a second shell and connect your client, as in the AI SRE guide.

   tares mcp

6. Ask the client for the service the alert names, for example: “Use tares: what happened to the nightly job’s service in the last 15 minutes?” The client calls read with the service name and answers from the correlated timeline.

7. Open the source and check Recent events. It shows one event per active alert and resolved events as they clear, and Fields shows the labels the alerts carry.

If you want a trigger to wake an agent when the alert arrives, the AI SRE guide shows the incident trigger that fires on an alert event per service. The agent then reads the same timeline without anyone asking.

What you get

• One timeline per service with the stopped-job alert, its metrics, its logs and its recent changes on the same clock.
• A diagnosis that ties the failure to specific evidence lines, such as the last log line before the gap or a deploy that landed shortly before it, instead of restating the alert.
• One firing event and one resolved event per incident, so the timeline stays readable.
• A conclusion written back as a finding, so the next investigation, yours or an agent’s, starts from it.
• Prometheus stays in charge of alerting. Tares ingests what fired and correlates it. The Prometheus alerts connector page lists every field.

When something else fits

• If you route alerts through Alertmanager and want it to push them, use the alertmanager connector instead. It has its own ingest URL and receives what Alertmanager sends.
• If the alert does not carry a service label and you cannot add one to the rule, it cannot join the service’s timeline. Add the label to the rule first.
• If you only need to see the alerts that fired and nothing around them, the alerts source alone is enough. Add metrics, logs and deploys when you want the diagnosis to cite evidence.

Related