How do I route CloudWatch alarms about error spikes to an AI SRE?

Tares receives CloudWatch alarms through the webhook connector and an EventBridge setup, and a trigger filtered to your error-spike alarms wakes the agent only for those. Every other alarm is still stored on the timeline, so nothing is lost, but it does not wake anyone. Tares has no connector of its own for CloudWatch today, so the route is EventBridge, then a webhook source, then a trigger named for the symptom.

How it works

CloudWatch alarm state changes reach Tares through the webhook connector. An EventBridge rule matches each alarm state change and sends it as a JSON body to an API destination. The destination points at the ingest URL of a Tares webhook source. The webhook connector parses the body whatever the Content-Type and stores it lossless.

Filtering happens in a Tares trigger, not in CloudWatch:

• The webhook source maps payload fields into labels. detail.alarmName works as a one-level dotted label, so the alarm name becomes a label you can filter on.
• detail.state.value is two levels deep. Flatten it with an input transformer on the rule target if you want the alarm state as a label.
• A trigger lists the source, filters down to the alarms that mean an error spike, and counts matches in a window. Only a match fires it.
• The trigger wakes the agent you chose. Name the trigger after the symptom, for example error-spike. Its firings then appear under that name on the trigger’s page and under Firings.

This fits the split in AI SRE: Tares does the ingestion, correlation and waking, and your alerting stays where it is.

Use the EventBridge route, not SNS. A CloudWatch alarm action that publishes to SNS does not fit, because an SNS HTTPS subscription cannot send a custom header and its SubscriptionConfirmation message would never be confirmed.

Set it up

1. In Tares, create a webhook source and copy its ingest URL. It must be public HTTPS for EventBridge to reach it. Declare a label for the alarm name, mark it primary, and a label for the flattened alarm state:

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: alarm, field: detail.alarmName, primary: true }
      - { name: state, field: state }

2. In EventBridge, create an API destination whose endpoint is that ingest URL and whose method is POST.
3. Give the destination a connection, which EventBridge requires on every API destination. Use API key authorization with the header name Authorization and the value Bearer followed by an ingest-scoped API key. The key only matters when Tares has auth on.
4. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Set the API destination as its target.
5. On the rule target, add an input transformer that flattens the alarm state into a top-level field named state, so the state label above has a value.
6. Create the trigger that narrows the stream to one symptom. List the alarm names that mean an error spike, and keep only the ALARM state so recoveries do not wake the agent:

- name: error-spike
  sources: [cloudwatch-alarms]
  filters:
    - { field: alarm, op: in, value: [checkout-5xx-errors, api-error-rate] }
    - { field: state, op: eq, value: ALARM }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 10m

7. Point the trigger at who gets woken: a Tares agent, your own agent service over a webhook, or a Slack channel. Then change one alarm to ALARM and check Recent events and Fields on the source page to confirm the labels arrive. Check the trigger’s Firings to confirm it fired.

What you get

• Only alarms that match your error-spike filter wake the agent. Alarms for other symptoms stay on the timeline as evidence and do not interrupt anyone.
• The cooldown keeps a flapping alarm from waking the agent over and over inside ten minutes.
• The firing carries the symptom’s name, so you can see at a glance that an error spike woke the agent, not a generic alert.
• The alarm sits on one timeline next to your other sources when you label them with the same key, so the agent starts from correlated evidence instead of one alarm message.
• Because the payload is stored lossless, fields you did not map into labels are still there when you read the event.

When something else fits

• If your alarms already flow through Alertmanager, use the alertmanager connector. It makes one event per alert and has its own ingest URL.
• If you want an agent to act on every alarm regardless of symptom, drop the filters and keep only the source and the count condition.
• If you only want to investigate on demand, skip the trigger and ask a connected MCP client to read the timeline.

Related