Which labels on CloudWatch alarms about 5xx errors on an API tie it to the right service?

Tares ties a CloudWatch 5xx alarm to the right service through a label named service, marked primary on the webhook source and carried by every other source. The alarm, metrics, logs and deploys then land on one entity timeline.

How it works

CloudWatch alarms reach Tares through the webhook connector and an EventBridge API destination. Tares has no connector of its own for CloudWatch today. The alarm arrives as a JSON event with source and detail-type at the top level, and detail.alarmName, the alarm ARN in resources, region and account in the body. The alarm gives no service name of its own, so you declare one.

Tares correlates by labels. A label is a named axis on an event. The label marked primary: true is the key, and an entity is a pair such as (service, api-server). Every source that carries service with the same value for the same API joins that entity’s timeline.

For a 5xx alarm on one API, use these labels on the webhook source:

• service: the key. Set it with const when the source is dedicated to one API, so every alarm event from it carries that value. This is the label that ties the alarm to the right service.
• alarm: read from detail.alarmName. One-level dotted fields work as labels, so you can tell the 5xx alarm from a latency alarm on the same service.

The state is two levels deep (detail.state.value). Flatten it with an input transformer on the rule target if you want it as a label or in text_template. The webhook connector stores the full payload lossless, so nothing is lost either way.

The angle that matters most is consistency. Pick one label name and one value per service, and use them everywhere. A metrics source that says service: api-server and an alarm source that says service: api-gateway produce two entities, and the correlation never happens. For the full model, see Concepts.

Set it up

1. Create a webhook source for the alarms with service as the primary label. The alarm label reads the one-level dotted field. The event type comes from the top-level detail-type field.

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: service, const: api-server, primary: true }
      - { name: alarm, field: detail.alarmName }

Save it and copy the ingest URL from the source page. See the webhook connector for every option.

2. In EventBridge, create an API destination whose endpoint is the Tares ingest URL (it must be public HTTPS) and whose method is POST.
3. Give the destination a connection, which EventBridge requires. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key matters only when Tares has auth on.
4. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Set the API destination as its target. Alarm actions that publish to SNS do not fit this route. An SNS HTTPS subscription cannot send a custom header, and its confirmation message would never be confirmed.
5. Give every other source the same label. Examples: a docker_logs source with the label { name: service, const: api-server, primary: true }, a prometheus source that reads service from the metric labels, and a deploy webhook source with a primary service label read from the payload’s service field. Your CD pipeline then posts each deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

6. Open the source’s Fields tab. It shows how many sampled events carry each label and its top values. Confirm that service is present on every alarm event and that its values match the other sources. Then open Explore and pick api-server.

What you get

• One timeline per service. When the 5xx alarm moves to ALARM, you read the alarm, the error-rate metric, the request logs and the latest deploy for api-server merged on one clock, in a single read.
• Fast checks for “did a deploy land before the errors started”, because the deploy event sits on the same timeline as the alarm.
• Triggers keyed by the same label. A trigger watches several sources and groups by entity, so one firing carries the whole timeline for that service. See Build an AI SRE for the full incident loop built on this.
• Findings that stay with the service. An agent’s conclusion is written back with the same label, so the next investigation starts from it.
• Indexed lookups. Selecting events by service is a lookup, not a text scan.

When something else fits

• If one CloudWatch source carries alarms for several APIs, a fixed const would label them all the same service. Split the alarms into one webhook source per API, or flatten a service value into the event with an input transformer so you can read it as a field.
• If the metrics, logs and deploys do not share any one label value, choose the label they do share, such as repo for code changes, and key the timeline on that instead.

Related