Which labels on CloudWatch alarms about high latency tie it to the right service?

Tares ties a CloudWatch high latency alarm to its service through a label named service, marked as the key, that every source for that service also carries. Alarm, metrics, logs and recent deploys then land on one timeline for that service. The alarm reaches Tares through the webhook connector and an EventBridge setup, and detail.alarmName is a useful second label.

How it works

A CloudWatch alarm arrives as one event on a webhook source, and the labels you declare on that source say which entity it belongs to. The label marked primary: true is the key, and Tares reads the entity timeline by that key. Tares has no connector of its own for CloudWatch today, so the alarm reaches Tares through the webhook connector and a vendor setup in EventBridge.

The alarm payload has these fields to work with:

• source and detail-type are top-level fields. The detail-type value is CloudWatch Alarm State Change, so it works as the event_type_field.
• detail.alarmName is one level deep, so it works as a dotted label.
• detail.state.value is two levels deep. Flatten it with an input transformer on the rule target if you want it as a label or in text_template.

The alarm event has no service field of its own to read. Make service the key on the alarm source with a const label, one alarm source per service, and keep detail.alarmName as a plain label so you can still tell which alarm fired. Then give every other source for that service a label with the same name service, marked primary. A read for service returns the alarm, the metrics, the logs and the deploys merged in time order. See Concepts for how labels, keys and entities fit together.

Set it up

1. Create a webhook source for the service’s alarms, with service as the primary label. See the webhook connector for the console and API routes. Copy the ingest endpoint from the source page.

- name: checkout-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: service, const: checkout, primary: true }
      - { name: alarm, field: detail.alarmName }

2. In EventBridge, create an API destination whose endpoint is the Tares ingest URL (it must be public HTTPS) and whose method is POST.
3. Give it a connection, which EventBridge requires on every API destination. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key value only matters when Tares has auth on.
4. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Set the API destination as its target.
5. Give the other sources the same key name, so they all agree on one value for the service:
   • Metrics from the Prometheus connector can read it from the metric labels, for example { name: service, field: metric.service, primary: true }.
   • Docker logs can use a constant, for example { name: service, const: api-server, primary: true }.
   • A deploy webhook can read service from the payload your pipeline posts.
6. Make the value identical everywhere. Alarms for checkout need checkout on every source, not a variant of it, or the events land on separate entities.
7. Post a test alarm, or wait for one, then open the alarm source’s Recent events and Fields. Fields shows how many events carry each label, so you can confirm the key is reliably present. Then open Explore, pick service and the value, and check that alarm, metrics, logs and deploys share one timeline.

What you get

• One timeline per service. A high latency alarm sits next to the metric that tripped it, the request logs around it and the deploy that landed just before.
• A key that triggers can use. A trigger takes the primary label of its first source as the entity key by default, so a firing carries that service’s timeline across all the sources it watches.
• Alarm names you can filter on. The alarm label separates the latency alarm from other alarms on the same service.
• Findings that stick. An agent that diagnoses the latency writes its finding onto the same service timeline, so the next investigation starts from it. Build an AI SRE walks through the full loop with sources keyed by one service label.

When something else fits

• If your alarm names or ARNs already identify the service, you can read detail.alarmName as the primary label instead of a constant. Every other source must then carry that same value.
• CloudWatch alarm actions that publish to SNS do not fit this setup. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed. Use an EventBridge rule with an API destination instead.
• If you run Alertmanager or Grafana Alerting for latency, the alert already carries a service label. Use the alertmanager connector for that route.

Related