How do I wake an AI SRE once when CloudWatch alarms report crash loops?

Tares wakes your AI SRE once per crash-loop alarm when you route CloudWatch alarm state changes through EventBridge into a webhook source and put a trigger with a cooldown on it. The cooldown applies per entity, so an alarm that keeps repeating fires one investigation per cooldown window. Later events from the same alarm are still stored on that alarm’s timeline. Tares has no connector of its own for CloudWatch today, so the alarms reach Tares through the webhook connector and an EventBridge setup on the AWS side.

How it works

CloudWatch alarms reach Tares through the webhook connector and an EventBridge rule. A trigger then watches that source and wakes your agent.

• Delivery. An EventBridge rule on the default event bus matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Its target is an API destination that POSTs each event to your Tares ingest URL.
• Storage. The webhook connector parses the JSON body whatever the Content-Type and stores it lossless. source and detail-type are top-level fields. detail.alarmName works as a one-level dotted label.
• Entity. You mark the alarm name as the primary label. The trigger uses it as the entity, so each alarm is tracked on its own.
• Cooldown. The trigger’s cooldown is the minimum gap between firings for the same entity. A crash-loop alarm that keeps posting fires one dispatch, and the next one can only fire after the cooldown has passed.
• Timeline. Every event is stored whether or not the trigger fires. When the trigger does fire, the dispatch carries the entity’s correlated timeline over context_window, so the agent sees the other events of that alarm without a follow-up call.

This is the general trigger mechanism, applied to a webhook source. For the wider picture of how an agent investigates, see AI SRE.

Set it up

1. Create the webhook source. Label the alarm name from detail.alarmName and mark it primary. Read the event type from the top-level detail-type field.

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: alarm, field: detail.alarmName, primary: true }

2. Copy the ingest URL from the source page. It must be public HTTPS for EventBridge to reach it.
3. Create the API destination in EventBridge. Set the endpoint to the Tares ingest URL and the method to POST.
4. Give it a connection. EventBridge requires one on every API destination. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key only matters when Tares has auth on.
5. Create the rule on the default event bus. Match source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as the target.
6. Add the trigger. It watches the source per alarm, counts matching events, and waits 30 minutes before the same alarm can fire again.

triggers:
  - name: crash_loop_alarm
    sources: [cloudwatch-alarms]
    key_field: alarm
    filters:
      - { field: alarm, op: contains, value: crashloop }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: crash_loop
      context_window: 15m
    cooldown: 30m

7. Wake your agent. Turn the trigger on for a Tares agent in its project, or register a webhook for an outside agent with the subscribe tool. Acknowledge each dispatch with a 2xx. Delivery is at-least-once, so dedupe on dispatch_id.

The filter assumes your crash-loop alarms have crashloop in their names. Change the value to match your naming.

If you want only the alarming state to wake the agent, flatten detail.state.value with an input transformer on the rule target. It is two levels deep, so it cannot be a label as it is. Then declare it as a label and add a filter on it with op: eq.

Do not use a CloudWatch alarm action that publishes to SNS. An SNS HTTPS subscription cannot send a custom header, and its subscription confirmation message would never be confirmed.

What you get

• One investigation per crash-loop alarm per cooldown window, instead of one per repeated notification.
• Independent handling of each alarm, because the condition is evaluated per entity.
• A complete timeline. Every alarm event the rule delivers is stored lossless under the alarm’s primary label, including later state changes after the first alarm.
• Context in the dispatch. The agent receives the alarm’s correlated timeline over context_window.
• A record of each firing. Every firing is logged, even with zero subscribers, with per-subscriber delivery status under Firings in the console.

When something else fits

• If you want a regular review of the alarms rather than a reaction to them, give the trigger a schedule with condition: {every: 10m}. It fires once per interval with a summary of the window.
• If your crash loops also show up as metrics or logs in Prometheus, Docker or OTLP sources, add those sources to the same trigger so the agent gets them on one timeline.
• If a repeating alarm needs a fresh look sooner, shorten the cooldown. If you want fewer wake-ups, lengthen it.

Related