Which labels on CloudWatch alarms about a cron job that stopped tie it to the right service?

Tares ties a stopped-cron-job alarm to the right service through a service label that every source for that service carries, marked primary: true so it becomes the key. Declare it on the CloudWatch webhook source as a fixed const value. Declare the same service label on your metrics, logs and deploy sources. The alarm, the metrics, the logs and recent changes then land on one entity timeline.

How it works

CloudWatch reaches Tares through the webhook connector and an EventBridge setup on the AWS side. Tares has no connector of its own for CloudWatch today. EventBridge posts the alarm event as a JSON body to the ingest URL of a webhook source. The payload is stored lossless, and the labels you declare map it into the envelope.

A label is a named axis on an event. The one marked primary: true is the key, and its value becomes key_value. An entity is a label and value pair, such as (service, checkout). A read for { service: "checkout" } returns that service’s events from every source, merged in time order. For a stopped cron job, the label choice comes down to this:

• service as a const label on the CloudWatch source. It tags every alarm that arrives through that source with one fixed service name. The CloudWatch event carries no service field you can rely on, so a fixed value is the dependable way to put the alarm on the right entity.
• detail.alarmName as a second label. It is a one-level dotted field, so it works as a label. It lets you narrow a read to one alarm, but it is a different axis from service, so it is not the key.
• service on every other source. The key only joins sources that carry the same label. Metrics, logs and deploys must declare service too.

Because one const value applies to the whole source, use one webhook source per service. Each source has its own ingest URL, so each service gets its own EventBridge API destination pointing at it.

Set it up

1. In EventBridge, create an API destination whose endpoint is the Tares ingest URL (it must be public HTTPS) and whose method is POST.
2. Give it a connection, which EventBridge requires on every API destination. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key value only matters when Tares has auth on.
3. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Set the API destination as its target.
4. In Sources, Add source, webhook, create the source for the service that owns the cron job. Mark the service label as primary and give it a fixed value:

- name: checkout-cloudwatch
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: service, const: checkout, primary: true }
      - { name: alarm, field: detail.alarmName }

5. Declare the same service label on the other sources for that service, so they share the key. A deploy source is the simplest to add, and each deploy lands on the service’s timeline when your pipeline posts a payload with a service field:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

6. Trigger a test alarm state change, then open the source page. Recent events shows what arrived and Fields shows which payload fields come through and how often, so you can confirm the labels are present.

Do not use CloudWatch alarm actions that publish to SNS. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed. To use detail.state.value as a label or in text_template, flatten it first with an input transformer on the rule target, because it is two levels deep.

For the full walkthrough of keying metrics, logs, alerts and deploys by service, see Build an AI SRE. The source options are on the webhook connector page, and the terms are defined in Concepts.

What you get

• One entity per service. A read for { service: "checkout" } returns the alarm that says the cron job stopped, the metrics around it, the logs and the latest deploy, all on one clock.
• A key you can check. The source’s Fields tab shows each label’s coverage, so you can confirm service is present on the events before you rely on it.
• Triggers and agents that work from the same timeline. A trigger keyed by service hands the agent the whole correlated picture, so the agent sees a deploy followed by the alarm.
• The alarm name kept as its own label, so you can still tell which alarm fired within a service.

When something else fits

• If one CloudWatch source must cover many services, a const label does not fit. Split the sources by service, or post events whose body already carries the service in a top-level field and declare a field label for it.
• If the alarm should be read by alarm rather than by service, make alarm the primary label. Then the cron job’s alarm is no longer on the same key as your other sources.

Related