What should CloudWatch alarms about high latency include for an AI SRE?

Tares reads a CloudWatch high latency alarm best when it carries a service, the alarm name, the state reason and the alarm time, mapped on a webhook source. Tares has no connector of its own for CloudWatch today, so the alarm reaches Tares through the webhook connector and an EventBridge setup. The agent then reads the alarm next to the other evidence for that service.

How it works

EventBridge posts the alarm event to a Tares webhook source as a JSON body. The webhook connector parses it whatever the Content-Type and stores it lossless. Labels map payload fields into the envelope so the alarm lands on the right timeline.

The event holds these pieces:

• Service. The event has no service field. Give the source a constant service label, so every alarm on that source lands on that service’s timeline, whatever the alarm is named. Keep one source per service for this.
• Alarm name. detail.alarmName is a one-level dotted field, so it works as a label. It tells the agent which alarm fired.
• Summary. detail.state.reason explains why the alarm changed state. It is two levels deep, so flatten it with an input transformer on the rule target if you want it as a label or in text_template.
• Start time. time is a top-level field. Point event_time_field at it so the event sits on the timeline at the moment the alarm fired.
• State. detail.state.value shows ALARM or its resolved counterpart. It is also two levels deep, so it needs the same flattening.

The event EventBridge posts has no severity field, so Tares does not read a severity from it.

If you would rather read detail.alarmName as the service label, the alarm name must equal the service name for the alarm to share that service’s timeline. A constant label avoids that naming rule.

See the webhook connector page for the full list of source options.

Set it up

1. Create a webhook source for the service, with a constant service label marked primary, the alarm name as a second label and the event time field set.

- name: checkout-latency-alarms
  connector: webhook
  config:
    event_time_field: time
    labels:
      - { name: service, const: checkout, primary: true }
      - { name: alarm, field: detail.alarmName }
      - { name: detail_type, field: detail-type }

2. Copy the ingest URL from the source page. It must be public HTTPS for EventBridge to reach it.
3. In EventBridge, create an API destination whose endpoint is the Tares ingest URL and whose method is POST.
4. Give the API destination a connection, which EventBridge requires on every API destination. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key value only matters when Tares has auth on.
5. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Set the API destination as its target.
6. If you want the state reason or state value as a label, add an input transformer on the rule target that flattens those fields to the top level, then add labels for them.
7. Trigger the alarm, then open Recent events and Fields on the source page to confirm the labels arrive.

Repeat for each service you want its own timeline. The AI SRE guide shows how the sources, triggers and agent fit together once the alarms arrive.

What you get

• Every high latency alarm on the source lands on that service’s timeline, next to its metrics and logs.
• The agent sees which alarm fired, from the alarm label, and the exact time it started.
• With the reason flattened, the agent also sees why CloudWatch changed the alarm state, so it can tie the alert to evidence lines.
• The full EventBridge event stays in the lossless payload, so fields you did not label are still there to read.
• A trigger on the service can wake an agent when an alarm event arrives.

When something else fits

• CloudWatch alarm actions that publish to SNS do not fit. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed. Use the EventBridge route above.
• If you want the service to come from the alarm itself rather than a fixed source, read detail.alarmName as the service label and name each alarm exactly as the service.
• If your latency metrics already reach Tares through OTLP or Prometheus, the alarm adds the fired event on top of that evidence rather than replacing it.

Related