What should an AI SRE read next to CloudWatch alarms for error spikes?

Tares puts a CloudWatch alarm on one timeline per service, beside that service’s metrics, logs and recent deploys, so an AI SRE reads them together and cites evidence. The alarm is the trigger. The metrics show the size of the spike, the logs show the failing requests, and deploy events show what changed just before. Tares has no connector of its own for CloudWatch today, so alarms reach Tares through the webhook connector and an EventBridge setup.

How it works

An error-spike alarm tells you that something crossed a threshold. It does not tell you why. Tares correlates sources that carry the same label, usually service, into one timeline, and an agent reads that timeline in a single call. The AI SRE guide builds this end to end. For an error spike, put these on the timeline next to the alarm:

• The alarm itself. EventBridge posts the CloudWatch Alarm State Change event to a Tares webhook source. The payload is stored lossless, so the alarm name, the state and the reason stay readable.
• The service’s metrics. A prometheus source carries the 5xx rate, latency and dependency health. If you send OpenTelemetry metrics, an otlp source does the same job.
• The service’s logs. A docker_logs source or an otlp source shows the failing requests, so the diagnosis can point at specific log lines.
• Recent changes. A second webhook source, called from your CD pipeline on every deploy, adds deploy events. Config changes can use another webhook source the same way. Most error spikes follow a change, so this source often settles the diagnosis.

Tares does not re-threshold your alarm. CloudWatch keeps owning alerting, and Tares ingests what fired and correlates it with the evidence around it.

Set it up

1. Create a webhook source in Tares for the alarms. Add a label for the alarm name and mark it primary. detail.alarmName is one level deep, so it works as a label. detail-type is a top-level field, so it can be the event type field.

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: service, field: detail.alarmName, primary: true }

This label only joins the service timeline if the alarm name matches the service value your metrics and logs sources use. Name your alarms after the service. detail.state.value is two levels deep, so to use it as a label or in a text template, flatten it with an input transformer on the rule target.

2. In EventBridge, create an API destination. Its endpoint is the Tares ingest URL, which must be public HTTPS, and its method is POST.
3. Give the destination a connection, which EventBridge requires on every API destination. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key only matters when Tares has auth on.
4. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Set the API destination as its target.
5. Post a sample event to the ingest URL, then check Recent events and Fields on the source page.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"detail-type":"CloudWatch Alarm State Change","source":"aws.cloudwatch","detail":{"alarmName":"checkout","state":{"value":"ALARM"}}}'

6. Add the evidence sources, keyed by the same service label: metrics, logs and a deploy webhook. A deploy call from your pipeline looks like this:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

Use the EventBridge route, not CloudWatch alarm actions that publish to SNS. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed.

What you get

• One timeline per service where the alarm, the metric that moved, the error logs and the latest deploy sit on the same clock.
• A diagnosis that cites evidence lines. It can say that the alarm fired at 14:04, errors began after the deploy at 14:02, and the logs show the failing calls.
• A trigger that wakes an agent when the alarm event arrives, so the first look is done before anyone opens a dashboard.
• A written finding on the service’s timeline, so the next investigation starts from the previous conclusion instead of from scratch.

When something else fits

• If you already run Alertmanager or Prometheus rules for the same service, the alertmanager and prometheus_alerts connectors fit better than routing through EventBridge.
• If you only need CloudWatch to notify a person, a plain SNS notification is enough. Use the EventBridge webhook route when you want an agent to read the alarm with the surrounding evidence.

Related