What should an AI SRE read next to CloudWatch alarms for a cron job that stopped?

Tares lets an AI SRE read the same service’s metrics, logs and recent deploys next to a CloudWatch alarm, all on one timeline. The alarm tells you the cron job stopped. The neighbouring evidence tells you when it stopped and what changed just before. The diagnosis then cites those lines instead of guessing. CloudWatch reaches Tares through the webhook connector and an EventBridge setup.

How it works

A stopped cron job rarely explains itself. The alarm says a job missed its schedule, and the cause is usually somewhere else: a deploy, a config change, an error in the job’s logs, or a metric that flatlined.

Tares keeps one timeline per service. Every source that carries the same service label lands on that timeline, merged on one clock. When an agent reads the service, it gets the alarm event and the evidence around it in one read.

For a stopped cron job, put these beside the alarm:

• Metrics for the same service, for example from a prometheus source, so you can see where the job’s output dropped to zero.
• Logs for the same service, for example from docker_logs or OTLP, so you can see the last run’s output and any error before it went quiet.
• Deploys and config changes, posted by your CD pipeline to a webhook source, because most root causes are a change. The agent can then see that a deploy landed shortly before the job stopped.
• The job’s own runs. A cron job can POST to a webhook source itself, so its last successful run is an event on the timeline.

Tares has no connector of its own for CloudWatch today. The alarm arrives through the webhook connector: EventBridge posts the alarm state change as JSON to a Tares ingest URL. The webhook connector parses it whatever the Content-Type. CloudWatch keeps owning the alarm, and Tares ingests what fired and correlates it with the evidence. This follows the pattern in Build an AI SRE.

Set it up

1. Create a webhook source for the alarms. Add it under Sources, then Add source, then webhook, and copy the ingest URL. The detail-type field is top level, so it can be the event type field. detail.alarmName is a one-level dotted field, so it works as a label.

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: service, field: detail.alarmName, primary: true }

This joins the timeline when the alarm name matches the service name your other sources use. If your alarm names differ, use an input transformer on the rule target to flatten a field you choose. detail.state.value is two levels deep, so it needs the same flattening if you want it as a label or in a text template.

2. Send alarms through EventBridge. Create an API destination whose endpoint is the Tares ingest URL (it must be public HTTPS) and whose method is POST. Give it a connection with API key authorization, header name Authorization, and the value Bearer followed by an ingest-scoped API key. The key only matters when Tares has auth on. Then create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as its target.

3. Label the evidence with the same service. Give your metrics and log sources a service label with the same values. Add a second webhook source for deploys, with a primary service label, and call it from your CD pipeline on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

4. Let an agent read it. Start the MCP endpoint and connect your client. The client calls read with the service name and answers from the correlated timeline. You can also create a trigger over these sources, keyed by service, to wake an agent when an alarm arrives.

tares mcp

What you get

• The alarm event sits on the service’s timeline beside its metrics, logs and deploys, so one read shows what happened around the missed run.
• A diagnosis that points to evidence lines, such as a deploy minutes before the job stopped or an error in the last log lines, instead of restating the alarm.
• A finding written back to the timeline, so the next investigation of this job starts from the previous conclusion.
• No change to your alerting. CloudWatch still decides when the alarm fires.

When something else fits

• CloudWatch alarm actions that publish to SNS do not fit. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed. Use the EventBridge route above.
• If your alerts come from Prometheus rules, a prometheus_alerts or alertmanager source is the closer fit than routing them through a webhook.
• If the alarm names do not map to services and you cannot flatten the event, the alarm will not share a timeline with the other evidence until you add a matching label.

Related