What should an AI SRE read next to Alertmanager alert groups for a cron job that stopped?

Tares gives an AI SRE the same service’s metrics, logs and recent deploys on one timeline beside the Alertmanager alert, so the diagnosis cites evidence. The alert says the cron job stopped. The neighbouring events help the agent say why and since when. Every source shares one service label, so the agent reads a single correlated timeline.

How it works

Tares has an alertmanager connector that receives what Alertmanager routes to a webhook receiver. Alertmanager has already evaluated, grouped and deduped the alerts. With send_resolved it also tracks firing to resolved. Tares records what it pushes and correlates it with your other sources.

Each POST from Alertmanager is a batch. Tares fans the alerts array out into one event per alert:

• event_type is the alert name.
• text is the alert summary.
• The alert labels become Tares labels, plus a status of firing or resolved.
• The full alert, including fingerprint, stays in the payload.

The entity key decides which timeline an event lands on. Tares derives it from a label in the order namespace, service, job, instance, or from a label you declare as primary. For the join to work, the alert source, the metrics, the logs and the deploy events must all be keyed by the same service label. If the alert keys on namespace while everything else keys on service, the alert lands on a different timeline from the evidence.

For a stopped cron job, the agent reads these next to the alert group:

• Metrics for the same service, polled from Prometheus with the prometheus connector or any metric you pick by name or label.
• Logs for the same service, from a container with docker_logs or from your OpenTelemetry pipeline with OTLP.
• Recent changes, such as deploys and config changes, posted to a webhook source with event_type: deploy or event_type: config.
• Earlier findings that agents wrote back onto that service’s timeline.

The agent reads these as one timeline on one clock. It can tie the stop to a specific log line, a metric that went flat or a deploy that landed just before.

Set it up

1. Create the alert source. In the console go to Sources, Add source, Alertmanager alerts, name it and create it. Or ask a connected agent to create an Alertmanager source and give you the webhook URL. The source page shows a receiver block with the ingest URL filled in.

2. Add the receiver to alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

If Tares has auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. Without auth, the receiver above is enough.

3. Set the key to the service label. Make sure your cron alert rules carry a service label. Send an alert, open the source’s Fields view and promote service to the source’s primary label. Do not leave the key on namespace. The metrics, logs and deploy sources in the next step use service as their primary label, and the alert has to use the same one to share their timeline. The key_label field on the source also lets you name the alert label to key on.

4. Add the evidence sources, each with service as its primary label:

   • Metrics for the cron job’s service, with the prometheus connector.
   • Logs, with docker_logs or OTLP. For OTLP the service label reads resourceAttributes.service.name.
   • A webhook source for deploys, with a primary service label and event_type: deploy. Post to it from your CD pipeline on every deploy. A second one with event_type: config covers config changes.

5. Check the join. Send a test alert and look at Recent events. Then open Explore, pick the service and confirm the alert, metrics, logs and deploys appear together.

6. Wake an agent. Add a trigger over these sources, keyed by service, that fires when an alert event arrives. Subscribe an agent to it, or connect your own MCP client and ask what happened to the service. The full walkthrough is in Build an AI SRE. The connector details are on the Alertmanager connector page.

What you get

• One timeline per service, with the Alertmanager alert group, metrics, logs and deploys merged on one clock.
• Alerts that resolve show up as resolved events, so the agent sees when the cron job came back, if it did.
• A diagnosis that cites evidence lines instead of repeating the alert text. It can say the job stopped, name the log line or the deploy beside it, and suggest a next action.
• The conclusion written back as a finding on the service’s timeline, so the next investigation starts from it.
• Your alerting stays in Alertmanager. Tares ingests what it routes and does not re-threshold anything.

When something else fits

• If you do not run Alertmanager and rely on Prometheus rules alone, the prometheus_alerts connector polls fired alerts, but you lose Alertmanager’s grouping and silencing and its real-time delivery.
• If the cron job runs outside Prometheus, have it post a heartbeat or failure event to a webhook source, and keep the same service label.

Related