Which labels on CloudWatch alarms about a disk filling up tie it to the right service?

Tares ties a CloudWatch disk alarm to the right service through one label that every source carries for that service, usually service, marked primary: true so it becomes the key. Put the same label on the alarm source, your metrics, your logs and your deploys, and they all land on one entity timeline. CloudWatch alarms reach Tares through the webhook connector and an EventBridge setup, so you declare the service label on that webhook source yourself.

How it works

Tares has no connector of its own for CloudWatch today. Alarms reach Tares through the webhook connector and a vendor setup in EventBridge: a rule matches CloudWatch alarm state changes and posts them as JSON to a Tares ingest URL.

The label is what ties events together. In Concepts, an entity is a label and value pair, such as service and checkout. The label marked primary: true is the key, and a read of service: checkout returns that service’s alerts, metrics, logs and deploys merged in time order.

A disk alarm is an ordinary alarm state change event, so the same rules apply. The event carries these fields:

• source and detail-type at the top level.
• detail.alarmName, which works as a one-level dotted label.
• resources, account and region.
• detail.state.value, which is two levels deep, so flatten it with an input transformer on the rule target if you want it as a label or in text_template.

The alarm name and the other payload fields describe the alarm. The service label is what places it on the right entity. You have two ways to set it:

• Use const on the label when one webhook source receives the alarms for one service. Every event from that source gets the same value.
• Use field when the posted JSON carries a flat top-level field with the service name, for example one your input transformer adds. Keep the payload flat, because top-level fields are the safe choice for labels and templates.

Then give every other source the same label name. The metrics, logs and deploys sources must carry service with the same values, or they will not merge with the alarm.

Set it up

1. In EventBridge, create an API destination whose endpoint is the Tares ingest URL (it must be public HTTPS) and whose method is POST.
2. Give it a connection, which EventBridge requires. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key only matters when Tares has auth on.
3. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as its target.
4. Create a webhook source in Tares and declare the labels. Mark service as primary:

- name: cloudwatch-alarms
  connector: webhook
  config:
    labels:
      - { name: service, const: checkout, primary: true }
      - { name: alarm, field: detail.alarmName }

5. Give your other sources the same service label. On an OTLP source the service name is a path into the event, so the label reads field: resourceAttributes.service.name. A Docker logs source can use a const label, and a deploy webhook source can read service from its payload.
6. Open the source page and check Recent events and Fields. Fields shows how many sampled events carry each label and its top values, so you can confirm that service is present on the alarm events and on every other source.
7. Read the entity in Explore, or ask a connected agent for service: checkout. The alarm, metrics, logs and recent deploys appear on one timeline. The AI SRE guide shows how a trigger keyed by service wakes an agent on that timeline.

What you get

• One timeline per service. A disk alarm firing sits next to the metrics, the log lines and the last deploy for the same service, merged on one clock.
• A reliable key. Because service is marked primary on every source, key_value is set from it, and triggers keyed by service group events per service.
• A check before you rely on it. The Fields tab tells you whether the label really arrives on enough events.
• A lossless record. The webhook source stores the original CloudWatch event unmodified, so fields you did not label stay available in the payload.
• A reusable pattern. The same service label works for every vendor you route through the webhook connector, so one convention covers all your alert sources.

When something else fits

• If you set the alarm to publish to SNS, that route does not fit. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed. Use the EventBridge route above.
• If you want one source per service, use the const label. If one source carries several services, use a field label instead, and make sure that field is flat and present on every event.
• If your alerts already come from Prometheus or Alertmanager, use the prometheus_alerts or alertmanager connector and label them by service the same way.

Related