What should CloudWatch alarms about 5xx errors on an API include for an AI SRE?

Tares gives an AI SRE the most to work with when each CloudWatch 5xx alarm event carries a service name, the alarm name, the state and its reason, and the time it happened. CloudWatch alarms reach Tares through the webhook connector and an EventBridge setup, because Tares has no connector of its own for CloudWatch today. The event already holds the alarm name, state, reason and time. You add the service name with a label on the Tares source.

How it works

An EventBridge rule on the default event bus matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. It posts each alarm event as a JSON body to a Tares webhook source through an API destination. The webhook connector parses the body whatever the Content-Type and stores it lossless.

The event has these fields:

• Top level: version, id, detail-type, source, account, time, region and resources, where resources holds the alarm ARN.
• One level down: detail.alarmName. A one-level dotted path works as a label.
• Two levels down: detail.state.value and detail.state.reason. You can use these as a label or in text_template only after you flatten them with an input transformer on the rule target.

Tares correlates evidence per service when your sources share a label such as service, as described in Build an AI SRE. The CloudWatch event has no service field, so you set the service on the Tares source. A label with const gives every event from that source the same value. That ties the alarm to the metrics, logs and deploys you already key by the same service label.

What each piece gives the agent:

• Service name: a const label on the source, one source per API.
• Alarm name: detail.alarmName as a label. It names the check that tripped, so name your 5xx alarms for the API and the condition.
• State and reason: detail.state.value and detail.state.reason, once flattened. The reason is the one-line explanation of why the alarm moved state.
• Start time: the top-level time field, plus the Tares timeline clock for the other sources.

The event as shown has no severity field. Tares does not invent one. If your team grades 5xx alarms, put the grade in the alarm name so it arrives in the alarm name label.

Set it up

1. In Tares, create a webhook source for the API (Sources → Add source → webhook). Mark the service label primary. Copy the ingest URL from the source page.

- name: api-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    event_time_field: time
    labels:
      - { name: service, const: checkout-api, primary: true }
      - { name: alarm, field: detail.alarmName }
      - { name: region, field: region }

2. In EventBridge, create an API destination whose endpoint is the Tares ingest URL. The endpoint must be public HTTPS and the method is POST.
3. Give the destination a connection, which EventBridge requires on every API destination. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key value only matters when Tares has auth on.
4. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Set the API destination as its target.
5. If you want the state value or the reason as a label or in text_template, add an input transformer on the rule target that flattens them. Then declare the labels over the flattened field names you chose.
6. Test the mapping by posting a sample event to the ingest URL. Then read Recent events and Fields on the source page to see which fields arrive and how often.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"detail-type": "CloudWatch Alarm State Change", "source": "aws.cloudwatch", "time": "2024-01-01T00:00:00Z", "region": "eu-west-1", "detail": {"alarmName": "checkout-api-5xx-high"}}'

The webhook connector page lists every option the source accepts.

What you get

• Each 5xx alarm lands as one event on the API’s timeline, stored lossless with the full CloudWatch payload.
• The agent sees the service, the alarm name and the region as labels, so it can read the alarm next to the logs and metrics for the same service.
• The alarm ARN, account and the state change are kept in the payload for the agent to cite.
• A trigger can fire on the alarm event and wake an agent with the correlated timeline, as in the AI SRE walkthrough. CloudWatch keeps owning the alarm and Tares ingests what fired.

When something else fits

• If your alarms publish only to SNS, this route does not fit. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed. Send the alarm through EventBridge as above.
• If you want the alarm values as labels without a transformer, send OTLP telemetry or poll an API with the http_poll connector instead.

Related