How do I wake an AI SRE once when CloudWatch alarms report error spikes?

Tares wakes your AI SRE once per alarm when you route CloudWatch alarm state changes through EventBridge into a webhook source and put a trigger with a cooldown on top. The cooldown keeps a repeating alarm from starting a new investigation each time it fires. Resolved events still arrive and are stored on the source. Tares has no connector of its own for CloudWatch today, so the alarms reach Tares through the webhook connector and an EventBridge setup.

How it works

CloudWatch alarm state changes go to Tares in three hops.

• EventBridge delivers the alarm. A rule on the default event bus matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Its target is an API destination that POSTs the event to your Tares ingest URL.
• A webhook source stores it. The webhook connector parses the JSON body whatever the Content-Type and stores the payload lossless. The labels you declare map payload fields into the envelope. source and detail-type are top-level fields, and detail.alarmName works as a one-level dotted label.
• A trigger decides when to wake the agent. A trigger watches the source and evaluates its condition as events arrive. With group_by: [key_value] it evaluates per entity, here per alarm. The cooldown is the minimum gap between firings for the same entity, so an alarm that keeps reporting produces one dispatch per cooldown window, not one per event.

Each dispatch carries the entity’s correlated timeline over context_window, so the agent starts with context. This is the pattern behind AI SRE.

Set it up

1. Add a webhook source for the alarms. Declare a primary label for the alarm name and a label for the state. Save it and copy the ingest endpoint from the source page.

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: alarm, field: detail.alarmName, primary: true }
      - { name: state, field: state }

2. In EventBridge, create an API destination. Its endpoint is the Tares ingest URL, which must be public HTTPS, and its method is POST.
3. Give the API destination a connection, which EventBridge requires. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key only matters when Tares has auth on.
4. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Set the API destination as its target.
5. detail.state.value is two levels deep, so it cannot be a label directly. Add an input transformer on the rule target that writes it to a top-level field named state. That gives the state label above a value.
6. Add the trigger. It counts only alarms in the ALARM state, evaluates per alarm, and waits 30 minutes before waking the agent again for the same alarm.

triggers:
  - name: error_spike
    sources: [cloudwatch-alarms]
    key_field: alarm
    filters:
      - { field: state, op: eq, value: ALARM }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 1m
      group_by: [key_value]
    emit:
      kind: error_spike
      context_window: 15m
    cooldown: 30m

7. Subscribe the agent. A Tares agent is woken by the trigger once you turn it on in a project that wires it up. An external agent registers over MCP:

Use tares to subscribe to the error_spike trigger and deliver to https://my-agent.example.com/hook.

8. Test the path by posting a sample alarm to the ingest URL, then check Recent events and Fields on the source page.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"detail-type":"CloudWatch Alarm State Change","source":"aws.cloudwatch","state":"ALARM","detail":{"alarmName":"checkout-5xx"}}'

What you get

• One investigation per alarm, not one per event. While the cooldown runs, repeated ALARM events for the same alarm do not start another dispatch.
• Independent alarms. Each alarm is its own entity, so a spike on one does not suppress another.
• Resolved events kept. OK events from EventBridge are ingested and stored on the source like any other event. They do not count toward the condition, so a recovery never wakes the agent.
• Context in the dispatch. The POST carries the entity’s correlated timeline, so the agent does not need a follow-up call.
• A record of every firing. Firings and per-subscriber delivery status show in the console under Firings. Delivery is at-least-once, so dedupe on dispatch_id and acknowledge with a 2xx.

When something else fits

• If your alarms publish through SNS, this route does not fit. An SNS HTTPS subscription cannot send a custom header, and its subscription confirmation message would never be confirmed. Use the EventBridge route above, or OTLP or http_poll against an API.
• If you want a scheduled summary rather than a wake on a condition, give the trigger a schedule such as condition: {every: 10m}, as described on the triggers page.

Related