What should an AI SRE read next to CloudWatch alarms for a disk filling up?

Tares puts the CloudWatch alarm, the same service’s metrics and logs, and its recent deploys and config changes on one timeline, so an AI SRE reads all of it together. The alarm says a disk is filling up. The metrics show how fast, the logs show what is writing, and the changes show what moved just before. Every claim in the diagnosis then points at a line of evidence.

How it works

CloudWatch alarms reach Tares through the webhook connector and an AWS-side setup in EventBridge. Tares has no connector of its own for CloudWatch today. EventBridge posts the alarm state change as a JSON body to a webhook source’s ingest URL, and the connector stores it lossless.

Tares correlates by label. Every source that carries the same service label lands on that service’s timeline, merged on one clock. The agent reads these next to the alarm:

• Metrics for the service, from a prometheus source or from OTLP.
• Logs for the service, from docker_logs or from OTLP.
• Recent changes, from a webhook source that your CD pipeline posts a deploy event to on every deploy. A second webhook source can carry config changes the same way.

Most real root causes are a change, so a deploy event next to the alarm often explains why the disk started filling when it did. A trigger over these sources, keyed by service, wakes an agent when the alarm event arrives. The agent reads the correlated timeline, ties its conclusion to specific evidence lines and writes a finding back onto the timeline. The next investigation starts from that finding. For the full walkthrough, see Build an AI SRE.

Set it up

1. Create a webhook source for the alarms. A CloudWatch alarm carries no service field, so give the source a constant service label for the service it covers. Add the alarm name as a second label. detail.alarmName works as a one-level dotted label, and detail-type is a top-level field, so it can be the event type field.

   - name: cloudwatch-alarms
     connector: webhook
     config:
       event_type_field: detail-type
       labels:
         - { name: service, const: checkout, primary: true }
         - { name: alarm, field: detail.alarmName }

   Save it and copy the ingest URL from the source page. See the webhook connector page for the options.

2. In EventBridge, create an API destination. Its endpoint is the Tares ingest URL, which must be public HTTPS, and its method is POST. Give it a connection, which EventBridge requires on every API destination. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key value only matters when Tares has auth on.

3. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Set the API destination as its target. The state value sits two levels deep at detail.state.value. If you want it as a label or in text_template, flatten it with an input transformer on the rule target.

4. Label the service’s metrics and logs sources with the same service label. For OTLP sources, the label is field: resourceAttributes.service.name.

5. Add the change signal. Create a second webhook source with a primary service label and event_type: deploy, then add one line to your CD pipeline:

   curl -X POST https://<your-tares>/ingest/<ingest_key> \
     -H 'content-type: application/json' \
     -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

6. Connect your agent and ask it about the service. Start the MCP endpoint with the command below, then point your client at it.

   tares mcp

   Per-client setup is on the AI SRE guide.

What you get

• One timeline per service that shows the alarm, the metrics, the logs and the deploys in time order.
• A diagnosis that cites evidence. The agent can say the alarm fired at one time, the service’s metrics were moving before it, a log pattern matches, and a deploy landed shortly before.
• A finding written back to the timeline, so the next investigation starts from the previous conclusion.
• Alarm events stored lossless, so fields you did not label are still in the payload.
• Sources that stay yours. CloudWatch keeps owning alarming, and Tares ingests what fired and correlates it with the evidence.

When something else fits

• CloudWatch alarm actions that publish to SNS are a different route and do not fit. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed. Use the EventBridge route above.
• If the metrics and logs already live in Prometheus and Alertmanager, point the alerts at the alertmanager or prometheus_alerts connector instead of going through EventBridge.
• If you only want a notification when the alarm fires, you do not need a timeline. The timeline pays off when an agent has to explain the cause.

Related