What should an AI SRE read next to CloudWatch alarms for failed deploys?

Tares puts the alarm on one timeline beside the same service’s metrics, logs and deploy events, so an AI SRE reads the surrounding evidence and not only the alarm. CloudWatch alarms reach Tares through the webhook connector and an EventBridge setup, because Tares has no connector of its own for CloudWatch today. Add a deploy source from your CD pipeline and the agent can cite the change that landed just before the alarm fired.

How it works

A CloudWatch alarm tells you something is wrong. It rarely tells you why. For a failed deploy, the evidence sits beside the alarm:

• Metrics for the same service, such as error rate and latency, from your Prometheus source.
• Logs for the same service, such as container logs or an OpenTelemetry pipeline.
• Recent changes, meaning deploy and config events posted by your CD pipeline.
• The alarm itself, with its state and reason.

Tares merges sources that carry the same label, usually service, into one timeline per service on one clock. An agent reads that timeline in a single call and can tie its diagnosis to specific lines, for example a deploy at 14:02, errors from 14:04 and the alarm at 14:05. The AI SRE guide builds this end to end.

The alarm arrives like this:

• EventBridge matches the CloudWatch Alarm State Change event and posts it as a JSON body to a Tares ingest URL.
• The webhook connector parses the body whatever the Content-Type and stores it lossless.
• source and detail-type are top-level fields. detail.alarmName works as a one-level dotted label. detail.state.value is two levels deep, so flatten it with an input transformer on the rule target if you want it as a label or in a text template.

CloudWatch alarm actions that publish to SNS do not fit. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed.

Set it up

1. Create a webhook source for the alarms. Give it a primary service label with a constant value, so its events land on that service’s timeline. Add a second label for the alarm name.

- name: checkout-alarms
  connector: webhook
  config:
    labels:
      - { name: service, const: checkout, primary: true }
      - { name: alarm, field: detail.alarmName }

2. Copy the ingest URL from the source page. It must be public HTTPS for EventBridge to reach it.
3. In EventBridge, create an API destination whose endpoint is that ingest URL and whose method is POST.
4. Give the destination a connection, which EventBridge requires. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key value only matters when Tares has auth on.
5. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as its target.
6. Add the change signal. Create a second webhook source with the same service label, a primary label read from the payload’s service field, event_type: deploy and a text template such as deploy {commit} by {author}: {message}. Then post to it from your CD pipeline on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

7. Make sure your metrics and log sources carry the same service label, and start the MCP endpoint with tares mcp. Then ask your connected agent what happened to the service in the last 15 minutes. It reads the correlated timeline, alarm and deploy events included.

If you want an agent woken when something fires, the AI SRE guide shows a trigger over sources keyed by service. Build yours the same way over your own sources.

What you get

• One timeline per service with the alarm, the metrics, the logs and the deploy events on the same clock.
• Diagnoses that cite evidence: the deploy that landed, the metric that moved and the log lines around the alarm state change.
• Alarm payloads stored lossless, so every field EventBridge sends is available later, and the Fields view on the source page shows which ones arrive.
• A finding written back to the timeline, so the next investigation starts from the previous conclusion.

When something else fits

• If your alerts already flow through Alertmanager or Prometheus rules, the alertmanager and prometheus_alerts connectors fit better than routing CloudWatch alarms through EventBridge.
• If you only want to be notified of an alarm and need no surrounding evidence, CloudWatch’s own notification actions are enough.

Related