Skip to Content

How do I route CloudWatch alarms about high latency to an AI SRE?

Tares receives CloudWatch alarms through the webhook connector and an EventBridge setup, and a trigger with filters lets only the high latency alarms wake your agent. Tares has no connector of its own for CloudWatch today, so EventBridge posts alarm state changes to a webhook source’s ingest URL. You name the trigger for the symptom and filter on the alarm and its state, so unrelated alarms and recoveries stay on the timeline without waking anyone.

How it works

  • EventBridge matches CloudWatch alarm events and posts each one as a JSON body to the Tares ingest URL of a webhook source. The webhook connector parses the body whatever the Content-Type and stores it lossless.
  • source and detail-type are top-level fields. detail.alarmName is a one-level dotted field, so it works as a label.
  • detail.state.value is two levels deep. Flatten it with an input transformer on the rule target if you want the alarm state as a label.
  • A trigger watches the source. Its filters keep only the alarms you name for the symptom, and its condition and cooldown decide when it fires. When it fires, it wakes the agent subscribed to it, as described in Build an AI SRE.

Alarm actions that publish to SNS are a different route and do not fit. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed. Use an EventBridge rule with an API destination.

Set it up

  1. Create a webhook source with a label for the alarm name and, if you flatten it, one for the state. Save it and copy the ingest endpoint from the source page.
- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: alarm, field: detail.alarmName, primary: true }
      - { name: state, field: state }
  1. In EventBridge, create an API destination whose endpoint is the Tares ingest URL and whose method is POST. The endpoint must be public HTTPS.
  2. Give the API destination a connection, which EventBridge requires. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key only matters when Tares has auth on.
  3. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Set the API destination as its target. On the target, add an input transformer that copies detail.state.value into a top-level field named state.
  4. Test the mapping by posting a sample alarm to the ingest URL. Then check Recent events and Fields on the source page.
curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"source": "aws.cloudwatch", "detail-type": "CloudWatch Alarm State Change", "state": "ALARM", "detail": {"alarmName": "checkout-high-latency"}}'
  1. Add the trigger that carries the symptom. Name it for the symptom. List the alarm names that mean high latency and keep only the ALARM state, so recoveries do not wake the agent.
- name: high-latency
  sources: [cloudwatch-alarms]
  filters:
    - { field: alarm, op: in, value: [checkout-high-latency, api-high-latency] }
    - { field: state, op: eq, value: ALARM }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 15m
  1. Subscribe an agent to the trigger. Use a Tares agent whose prompt says the symptom is high latency, or your own agent service over a webhook. Either way the trigger is called high-latency, and its firings show under that name on the trigger’s page and under Firings.

What you get

  • Only the alarms you list for high latency fire the trigger. Other CloudWatch alarms still land on the source’s timeline as evidence, but they do not wake the agent.
  • Recoveries and state changes you did not filter in stay quiet, because the trigger keeps only the ALARM state.
  • The cooldown keeps a flapping alarm from waking the agent repeatedly.
  • Every firing is recorded under a trigger named for the symptom, so you can see at a glance why the agent woke.
  • The alarm sits on the same timeline as your metrics, logs and deploys when they share a label. The agent reads that correlated evidence instead of the bare alarm.

When something else fits

  • If you already run Prometheus rules for latency, the prometheus_alerts or alertmanager connectors are a closer fit than CloudWatch plus EventBridge.
  • If you want every alarm to wake the agent, leave out the filters and keep only the condition and cooldown.
  • If your alarms reach you only through SNS, add an EventBridge rule for the alarm state change, because the SNS route does not work with Tares.

Still stuck? Get help

Last updated on