What should CloudWatch alarms about failed deploys include for an AI SRE?

Send Tares a CloudWatch alarm event that you label with a service name and severity, and that carries the alarm name, region and event time. CloudWatch reaches Tares through the webhook connector and an EventBridge setup, and Tares has no connector of its own for CloudWatch today. The labels put the alarm on the same per-service timeline as your deploy events, metrics and logs, so an agent can read it next to that evidence.

How it works

CloudWatch reaches Tares through the webhook connector and a vendor setup in EventBridge. EventBridge posts the alarm as a JSON body to a webhook source’s ingest URL, and the webhook connector stores the payload lossless. You declare labels to map payload fields into the envelope.

The alarm event has these fields to work with:

• Top level: source, detail-type, time, region and account. These work directly for labels, text_template and event_time_field.
• One level deep: detail.alarmName works as a dotted label.
• Two levels deep: detail.state.value can’t be a label or appear in text_template as it is. Flatten it with an input transformer on the rule target first.

For a failed-deploy alarm, map the four things you want to carry like this:

• Service name: a primary service label. Tares builds one timeline per service from sources that share this label, as the AI SRE guide shows. If one source handles one service’s alarms, give it a constant service value.
• Severity: a severity label. A constant works the same way on a source that handles one kind of alarm.
• Summary: a text_template over top-level fields, so each event renders as readable text.
• Start time: event_time_field pointed at time, so the event lands on the timeline at the time the event carries.

Also label detail.alarmName and region, so the alarm name and region are searchable fields on the event.

Set it up

1. Create a webhook source for the failed-deploy alarms (Sources → Add source → webhook) and copy its ingest endpoint from the source page. This example uses a constant service and severity, so create one source per service:

- name: checkout-deploy-alarms
  connector: webhook
  config:
    event_type: failed_deploy_alarm
    event_time_field: time
    text_template: 'alarm event from {source} in {region}'
    labels:
      - { name: service, const: checkout, primary: true }
      - { name: severity, const: critical }
      - { name: alarm, field: detail.alarmName }
      - { name: region, field: region }

2. In EventBridge, create an API destination whose endpoint is the Tares ingest URL and whose method is POST. The endpoint must be public HTTPS.
3. Give the API destination a connection, which EventBridge requires. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key value only matters when Tares has auth on.
4. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Set the API destination as its target.
5. If you want the alarm state as a label or in text_template, add an input transformer to the rule target that flattens detail.state.value to a top-level field. Then add a label or a template placeholder for it.
6. Trigger the alarm or send a test event, then open the source page. Recent events shows what arrived and Fields shows which payload fields come through and how often. Use them to check that your labels read real values.

See the webhook connector page for every field the source accepts.

What you get

• A failed-deploy alarm that arrives as one event on the service’s timeline, with its time, alarm name and region as fields you can read.
• Alarm events that sit beside your deploy events, metrics and logs for the same service, because they share the service label.
• A readable one-line text for each event from your text_template.
• The full original payload, stored lossless, so fields you did not label are still there.
• Nothing to maintain on the Tares side beyond one webhook source per service or alarm family.

When something else fits

• CloudWatch alarm actions that publish to SNS do not fit. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed. Use the EventBridge route above.
• If the failed deploy comes from your CD pipeline instead, post a deploy event straight to a webhook source with a service field. You then skip EventBridge entirely.
• If your alarms already come from Prometheus rules, the prometheus_alerts or alertmanager connectors take them directly.

Related