Skip to Content

Which labels on CloudWatch alarms about error spikes tie it to the right service?

Tares ties a CloudWatch error-spike alarm to its service through a service label that every source for that service carries, marked primary: true so it becomes the key. Put the same label on the CloudWatch webhook source, your metrics, your logs and your deploy events, and they land on one entity timeline. The alarm’s detail.alarmName makes a good second label, because it is a one-level dotted field the webhook connector can read.

How it works

CloudWatch reaches Tares through the webhook connector and an EventBridge setup on the AWS side. Tares has no connector of its own for CloudWatch today. EventBridge posts the alarm event as a JSON body to the Tares ingest URL, and the webhook connector stores it lossless.

A label is a named axis on each event, declared as a const (a fixed value for every event from the source) or a field (read per event). The label marked primary: true is the key, and the entity is the pair of label and value, such as (service, checkout). A read for { service: "checkout" } returns that service’s events from every source, merged in time order. See Concepts for the full model.

What each source uses for service:

  • CloudWatch alarms (webhook source): the alarm event has no service field of its own. Use a const label when one source serves one service. The top-level fields source and detail-type work as they are, and detail.alarmName works as a one-level dotted label. detail.state.value is two levels deep, so flatten it with an input transformer on the rule target if you want it as a label or in text_template.
  • Metrics (prometheus): read service from the metric labels.
  • Logs: a docker_logs source takes a const label such as service set to the service name. An OTLP source reads it from resourceAttributes.service.name.
  • Deploys and config changes (webhook): a primary service label read from the payload’s service field.

Because the label name and value match across sources, the alarm, the metrics, the logs and the recent changes share one timeline. A trigger over these sources can use key_field to name the label that identifies the entity. Its default is the primary label of the first source.

Set it up

  1. Pick one label name every source will carry for the same service. Use service, and use the same spelling of each service name everywhere, for example checkout.
  2. Create a webhook source for the CloudWatch alarms of that service. Mark service as primary and add the alarm name as a second label.
- name: cloudwatch-checkout
  connector: webhook
  config:
    labels:
      - { name: service, const: checkout, primary: true }
      - { name: alarm, field: detail.alarmName }
  1. Copy the ingest URL from the source page. In EventBridge, create an API destination whose endpoint is that URL (it must be public HTTPS) and whose method is POST.
  2. Give the API destination a connection, which EventBridge requires. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key value only matters when Tares has auth on.
  3. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as its target. Alarm actions that publish to SNS do not fit this route, because an SNS HTTPS subscription cannot send a custom header and its confirmation message would never be confirmed.
  4. Give your other sources the same service label marked primary: Prometheus metrics with service read from the metric labels, logs with a service label, and a deploy webhook source with a service field in its payload. Each deploy pipeline step posts one event:
curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'
  1. Check the result on each source’s Fields tab. It shows how many sampled events carry service and its top values, so you can confirm the label is reliably present. Then open Explore, pick (service, checkout) and read the merged timeline.

What you get

  • One timeline per service. When the error-spike alarm fires, the alarm event sits next to the error-rate metrics, the log lines and the latest deploy for the same service.
  • A key you can trigger on. A trigger over these sources groups by the service entity, so one firing carries that service’s whole timeline to the agent it wakes.
  • Findings that stay attached. An agent’s finding is a labeled event too, so the next read of (service, checkout) starts from the earlier conclusion. The AI SRE guide walks through this loop end to end.
  • A queryable alarm name. The alarm label lets you narrow a read to one alarm within a service.

When something else fits

  • If one CloudWatch source must cover many services and the service name sits in a field deeper than one level, flatten it with an input transformer on the rule target first, then read it as a field label. A const label fits only one source per service.
  • If your alarms already flow through Alertmanager, the alertmanager connector may suit you better, and it makes one event per alert.

Still stuck? Get help

Last updated on