How do I wake an AI SRE once when CloudWatch alarms report 5xx errors on an API?

Tares wakes an AI SRE once for a CloudWatch 5xx alarm with a webhook source fed by EventBridge and a trigger that has a cooldown per entity. While the alarm keeps repeating, the cooldown holds back extra firings for that alarm. Resolved events are still stored on the source’s timeline.

How it works

CloudWatch reaches Tares through the webhook connector and a vendor setup in EventBridge. Tares has no connector of its own for CloudWatch today.

• EventBridge matches CloudWatch alarm state changes and posts each one as a JSON body to the Tares ingest URL. The webhook connector parses it whatever the Content-Type.
• Tares stores every event lossless on the webhook source. This includes the OK event that arrives when the alarm resolves.
• A trigger watches the source and checks its condition as events arrive. With group_by: [key_value] the condition is evaluated per entity, so each alarm fires independently.
• cooldown is the minimum gap between firings for the same entity. A repeating ALARM state for one API alarm starts one investigation, and the next firing for that alarm waits until the cooldown passes.
• When the trigger fires, Tares wakes the subscribed agent with the entity’s correlated timeline over the trigger’s context_window, so the agent starts with context and needs no follow-up call.

This is one pattern from AI SRE: the alarm is the symptom, and the trigger decides when it is worth waking an agent.

Set it up

1. Create a webhook source and copy its ingest URL. See the webhook connector. Use detail-type as the event type. detail.alarmName works as a one-level dotted label. detail.state.value is two levels deep, so you cannot label it directly.

- name: cloudwatch_alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: alarm, field: detail.alarmName, primary: true }
      - { name: state, field: state }

2. In EventBridge, create an API destination whose endpoint is the Tares ingest URL. The URL must be public HTTPS, and the method is POST.
3. Give the destination a connection. EventBridge requires one on every API destination. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key only matters when Tares has auth on.
4. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Set the API destination as the rule’s target.
5. On the rule target, add an input transformer that flattens detail.state.value into a top-level field named state. That is what the state label above reads.
6. Add a trigger that counts ALARM events per alarm and sets a cooldown. Change cooldown to the quiet period you want.

triggers:
  - name: api_5xx_alarm
    sources: [cloudwatch_alarms]
    key_field: alarm
    filters:
      - { field: state, op: eq, value: ALARM }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: api_5xx_alarm
      context_window: 30m
    cooldown: 30m

7. Wake an agent. A Tares agent in the project is woken by this trigger once you wire and turn it on there. An external agent registers a webhook with the subscribe tool, for example: use tares to subscribe to the api_5xx_alarm trigger and deliver to your agent’s URL.
8. Post a test alarm event to the ingest URL, then check Recent events and Fields on the source page to confirm the labels carry values.

What you get

• One investigation per 5xx alarm instead of one per repeated notification. The cooldown is set per entity, so a second API’s alarm is not held back by the first.
• Resolved events on the timeline. The OK event is ingested and stored like any other, so it appears in the source’s Recent events and in the history you read during or after an incident. The trigger filter only decides which events count toward firing and which are handed over in a dispatch.
• A dispatch that carries the correlated timeline for the alarm, so the agent sees the surrounding events without querying anything.
• A record of every firing, even with zero subscribers, under Firings in the console, with per-subscriber delivery status.
• Delivery you can rely on: dispatches are at-least-once, so dedupe on dispatch_id. Acknowledge with a 2xx, because 5xx responses and transport errors are retried with backoff and a 4xx is not.

When something else fits

• CloudWatch alarm actions that publish to SNS do not fit this route. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed. Use the EventBridge route above.
• If you want a response to a numeric 5xx rate rather than an alarm state, send that metric to a source with a typed field and use an aggregate such as max in the trigger.
• If the alarms are noisy for reasons unrelated to the API, narrow them with a filter on the alarm label before you shorten the cooldown.

Related