How do I route CloudWatch alarms about crash loops to an AI SRE?

Tares receives CloudWatch alarms through the webhook connector and an EventBridge API destination, and a trigger that filters on the alarm name wakes your agent only for crash loops. Tares has no connector of its own for CloudWatch today, so the route is a webhook source plus a vendor setup. Name the trigger after the symptom and every firing shows up under that name. Other alarms still land on the timeline, they just do not wake the agent.

How it works

The path has three parts.

• EventBridge delivers the alarm. A rule on the default event bus matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Its target is an API destination that POSTs the event to your Tares ingest URL, which must be public HTTPS.
• A webhook source stores it. The webhook connector accepts the JSON body whatever the Content-Type and stores it lossless. source and detail-type are top-level fields. detail.alarmName is one level deep, so it works as a label.
• A trigger filters it. The trigger watches the webhook source, keeps only the events whose alarm label matches your crash-loop alarms, and wakes the agent you subscribed to it. Everything else stays on the timeline as evidence.

The filter is the point. CloudWatch sends every alarm state change through the same rule, so the symptom has to be chosen on the Tares side. Pick the alarms that describe crash loops, for example one on a container restart count, and list their names in the filter.

detail.state.value is two levels deep, so it does not work directly as a label or in text_template. If you want the alarm state in a label, flatten it with an input transformer on the rule target first. event_type_field and text_template read top-level fields only.

Set it up

1. Create the API destination in EventBridge. Set the endpoint to the Tares ingest URL and the method to POST. EventBridge requires a connection on every API destination. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key only matters when Tares has auth on.
2. Create the rule. On the default event bus, match source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as the target. Add an input transformer if you want the alarm state as a flat field.
3. Add the webhook source in Tares. Create it under Sources → Add source → webhook, or ask a connected agent. Copy the ingest URL for step 1. Declare a label for the alarm name and mark it primary:

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: alarm, field: detail.alarmName, primary: true }
      - { name: region, field: region }

4. Add the trigger that names the symptom. Replace the alarm names with your own crash-loop alarms:

- name: crash-loops
  sources: [cloudwatch-alarms]
  filters:
    - { field: alarm, op: in, value: [checkout-crash-loop, payments-crash-loop] }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 5m

5. Subscribe your agent to the trigger. Choose who gets woken: a Tares agent, your own agent service over a webhook, or a Slack channel. The AI SRE guide walks through the full loop from trigger to finding.
6. Check the mapping. Fire a test alarm, then open Recent events and Fields on the source page. They show which payload fields arrive, so you can confirm the alarm label is filled before you rely on the filter.

If your alarms reach you through an SNS topic, use the EventBridge route above instead. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed, so that route does not fit.

What you get

• Only crash-loop alarms wake the agent. Disk, latency and billing alarms still arrive and sit on the timeline, so the agent can read them when it investigates.
• A trigger named for the symptom. Each firing is listed on the trigger’s page under the name crash-loops, so you see at a glance why the agent woke.
• Lossless storage of each alarm event, so the full CloudWatch payload is available to the agent, not only the labels you mapped.
• A cooldown that keeps a flapping alarm from waking the agent again and again.
• A finding the agent writes back to the timeline, so the next investigation starts from the previous conclusion.

When something else fits

• If you want every alarm to wake the agent, drop the filter and keep the trigger broad. You then rely on the agent to sort out which alarms matter.
• If your crash-loop signal lives in Prometheus rather than CloudWatch, the prometheus_alerts or alertmanager connectors are a closer fit than a webhook source.
• If you only want to ask questions on demand, skip the trigger and let a connected MCP client read the timeline.

Related