How do I route CloudWatch alarms about 5xx errors on an API to an AI SRE?

Tares receives CloudWatch alarms through the webhook connector, with an EventBridge API destination as the vendor setup, and a trigger filtered to your API’s 5xx alarms wakes the agent. Alarms for other symptoms still land on the timeline but do not fire that trigger. You name the trigger after the symptom, so each firing and the agent subscribed to it point at the 5xx problem.

How it works

Tares has no connector of its own for CloudWatch today. The alarms reach Tares through the webhook connector and a setup on the AWS side.

• Delivery. An EventBridge rule matches CloudWatch alarm state changes and sends each one to an API destination that points at your Tares ingest URL. EventBridge posts the event as a JSON body, and the webhook connector parses it whatever the Content-Type.
• Fields. source and detail-type are top-level fields. detail.alarmName is one level deep, so it works as a label. detail.state.value is two levels deep, so flatten it with an input transformer on the rule target if you want it as a label or in a text template.
• Filtering. A trigger reads the source and keeps only events whose alarm label matches the 5xx alarms you list. Everything else is stored on the timeline but does not wake anything.
• Naming. The trigger’s name is the symptom, for example api-5xx-alarm. It shows on the trigger’s page and under Firings, and your agent subscribes to that trigger.

This route uses an EventBridge rule, not a CloudWatch alarm action that publishes to SNS. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed.

Set it up

1. Create the webhook source in Tares. Add a source with the webhook connector and a primary label on the alarm name. Copy the ingest endpoint from the source page.

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: alarm, field: detail.alarmName, primary: true }

2. Create the API destination in EventBridge. Set the endpoint to the Tares ingest URL, which must be public HTTPS, and the method to POST.
3. Give it a connection. EventBridge requires one on every API destination. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key only matters when Tares has auth on.
4. Create a rule on the default event bus. Match source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as the target.
5. Check the mapping. Fire a test alarm, then open the source page. Recent events shows what arrived and Fields shows which payload fields come through and how often.
6. Add the symptom trigger. List the names of your API’s 5xx alarms in the filter. Replace the example names with your own.

- name: api-5xx-alarm
  sources: [cloudwatch-alarms]
  filters:
    - { field: alarm, op: in, value: [api-5xx-prod, api-5xx-staging] }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 5m

7. Pick who gets woken. Subscribe a Tares agent to the trigger, point it at your own agent service over a webhook, or send it to a Slack channel. The AI SRE guide covers each choice.

The condition counts matching events in a five minute window, and the cooldown keeps a flapping alarm from waking the agent again straight away. If you also want to separate ALARM from OK, flatten detail.state.value into a top-level field in the input transformer, add a label for it, and add a second filter on that label.

What you get

• Only your API’s 5xx alarms wake the agent. Latency, disk and other alarms from the same account stay on the timeline as context.
• Each firing is recorded under the trigger’s name, so you read the symptom straight from the Firings list.
• The agent reads one correlated timeline instead of a bare alarm message, with the alarm event next to your other sources.
• Adding a new 5xx alarm is a one-line change to the filter list.
• The EventBridge connection carries the credential, so Tares needs no AWS access of its own.

When something else fits

• If you need a wider net, drop the filter and let every alarm fire one trigger, then let the agent decide what matters.
• If your alarms already flow through Alertmanager or Prometheus alerts, use those connectors instead.
• If the 5xx signal is a metric and not an alarm, ingest it over OTLP or poll the API with http_poll and put the threshold in the trigger.

Related