What should an AI SRE read next to CloudWatch alarms for crash loops?

Tares puts the CloudWatch alarm, the same service’s metrics, its logs and its recent deploys on one timeline, so an AI SRE reads all of it before naming a crash-loop cause. The alarm reaches Tares through the webhook connector and an EventBridge setup, because Tares has no connector of its own for CloudWatch today. The diagnosis then cites evidence lines instead of repeating the alarm.

How it works

A CloudWatch alarm tells you that a service is unhealthy. It does not tell you why. Tares correlates sources that carry the same label, usually service, into one timeline per service, and that timeline is what your agent reads. For a crash loop, read these next to the alarm:

• The service’s metrics. A Prometheus source is not needed for the alarm itself. Metrics you pick by name or label from a Prometheus server land on the same timeline, so you can see what moved around the time the alarm went to ALARM.
• The service’s logs. Container logs from the docker_logs connector, or OpenTelemetry logs over OTLP, show what the service printed while it was failing.
• Recent changes. Most real root causes are a change. A webhook source that your CD pipeline posts to on every deploy puts a deploy event on the same timeline, so the agent can see a deploy land shortly before the alarm.

The alarm travels like this: CloudWatch publishes a state change event, an EventBridge rule matches it, and an API destination POSTs it as a JSON body to a Tares webhook source. The webhook connector parses it whatever the Content-Type. source and detail-type are top-level fields. detail.alarmName works as a one-level dotted label. detail.state.value is two levels deep, so flatten it with an input transformer on the rule target if you want it as a label or in a text template.

The AI SRE guide shows how a trigger wakes an agent with this correlated timeline and how the agent writes its finding back.

Set it up

1. In EventBridge, create an API destination whose endpoint is the Tares ingest URL and whose method is POST. The endpoint must be public HTTPS.
2. Give it a connection, which EventBridge requires on every API destination. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key value only matters when Tares has auth on.
3. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as its target.
4. In Tares, add a webhook source for the alarms. Use a label that matches the one your metrics and logs sources use, so the alarm lands on the same timeline. This example reads the alarm name as the service, which works when your alarms are named after the service. Otherwise flatten a service field with an input transformer on the rule target.

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: service, field: detail.alarmName, primary: true }
      - { name: region, field: region }

5. Add the metrics and logs sources for the same service, keyed by the same label, as the AI SRE guide describes.
6. Add a second webhook source for deploys, with a primary service label, and run one line in your CD pipeline on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

7. Post a test alarm event to the ingest URL and check Recent events and Fields on the source page to confirm the labels arrive.

Do not route the alarm through an SNS action. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed, so that route does not fit.

What you get

• One timeline per service with the alarm, the metrics, the logs and the deploys on a single clock.
• A diagnosis that ties the crash loop to specific evidence lines, such as a deploy that landed before the alarm or an error repeated in the logs, instead of restating that the alarm fired.
• A finding written back to the timeline, so the next investigation starts from the previous conclusion.
• CloudWatch stays where your alerting lives. Tares ingests what fired and correlates it with the evidence around it.

When something else fits

• If your alarms already flow through Alertmanager or Prometheus rules, use the alertmanager or prometheus_alerts connectors instead, since those have their own connectors.
• If you can only publish alarm actions to SNS, you need a different path into Tares, such as an EventBridge rule on the alarm state change event or OTLP.

Related