Which labels on CloudWatch alarms about slow database queries tie it to the right service?

Tares ties a CloudWatch alarm about slow database queries to the right service through one shared label, such as service, marked primary on every source. Give the alarm source, your metrics, your logs and your deploy events that same label, and they land on one entity timeline. The alarm reaches Tares through the webhook connector and an EventBridge setup.

How it works

Every event in Tares carries labels, and an entity is a label and value pair such as service and checkout. The label you mark primary: true is the key, and Tares stores its value as key_value. A read for service: checkout returns that service’s alarms, metrics, logs and deploys merged in time order. See Concepts for the full model.

CloudWatch reaches Tares through the webhook connector and a vendor setup in EventBridge. Tares has no connector of its own for CloudWatch today. The webhook connector stores the alarm event lossless and lets you map payload fields to labels:

• source and detail-type are top-level fields.
• detail.alarmName works as a one-level dotted label.
• detail.state.value is two levels deep. To use it as a label or in text_template, flatten it with an input transformer on the rule target.

The alarm payload carries an alarm name, but it has no field that names your service. So pick the label every source can carry for the same service, call it service, and mark it as the key on each source. Use const when one ingest URL receives only one service’s alarms. Otherwise flatten the value you need into a top-level field with the input transformer and read it with field. The alarm name is a useful second label, but keep it secondary. It narrows a read to one alarm and does not tie the alarm to the other signals.

Set it up

1. Choose the shared label. Use service unless your metrics and logs already agree on another name. Check each source’s Fields tab to confirm the label is present on most events, so it works as a reliable key.
2. In EventBridge, create an API destination whose endpoint is the Tares ingest URL. It must be public HTTPS, and the method is POST.
3. Give the API destination a connection, which EventBridge requires. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key only matters when Tares has auth on.
4. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as its target.
5. Create the webhook source for the alarms with service as the primary label:

- name: cloudwatch-checkout
  connector: webhook
  config:
    labels:
      - { name: service, const: checkout, primary: true }
      - { name: alarm, field: detail.alarmName }

6. Give your other sources the same service label as the key. Your Prometheus metrics read it from the metric labels, your logs carry it per container or per OpenTelemetry resource, and a deploy webhook reads it from the payload:

labels:
  - { name: service, field: service, primary: true }

7. Send each deploy from your pipeline so recent changes land on the same timeline:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

8. Trigger a test alarm, then open Explore, pick the entity service and checkout, and check that the alarm, metrics, logs and the deploy appear together. Recent events on the alarm source shows whether the labels were filled.

The AI SRE guide walks through the same pattern end to end, with every source keyed by service.

What you get

• One timeline per service. A slow query alarm sits next to the latency metric, the database error logs and the deploy that landed minutes earlier.
• A key that is the same on every source, so a trigger keyed by service sees the alarm and the evidence together.
• A narrower read when you need it. Adding the alarm label to a selector returns only that alarm’s events for the service.
• Findings that agents write back onto the same entity, so the next investigation starts from the previous conclusion.

When something else fits

• If your alarms have no service identity in the payload and you cannot add one with an input transformer, key on a label that exists, such as the alarm name, and accept that correlation is per alarm.
• If you publish alarms through SNS, this route does not fit. An SNS HTTPS subscription cannot send a custom header, and its subscription confirmation message would never be confirmed. Use EventBridge with an API destination, or send the alarm data over OTLP or from your own service.

Related