How do I test CloudWatch alarms about failed deploys before a real incident?

Tares lets you test your failed-deploy CloudWatch alarm route by posting a sample event to the source’s ingest URL and checking Recent events and Fields. CloudWatch reaches Tares through the webhook connector and an EventBridge setup, because Tares has no connector of its own for CloudWatch today. You see the event on the source page and fix the label mapping before an alarm fires for real.

How it works

A CloudWatch alarm state change goes to Tares in three hops:

• A CloudWatch alarm for your deploy failures changes state. EventBridge sees it as an event with source aws.cloudwatch and detail-type CloudWatch Alarm State Change.
• An EventBridge rule on the default event bus matches that event and sends it to an API destination. The destination is a public HTTPS endpoint, the Tares ingest URL, with the POST method.
• A Tares webhook source receives the JSON body. It stores the payload lossless and maps the fields you declare as labels into the event.

EventBridge posts a JSON body, and the webhook connector parses it whatever the Content-Type. source and detail-type are top-level fields. detail.alarmName works as a one-level dotted label. detail.state.value is two levels deep, so flatten it with an input transformer on the rule target if you want it as a label or in text_template.

The ingest endpoint is the same kind of endpoint the Alertmanager source uses, so the test method is the same: post a payload, then read what arrived. Because the endpoint accepts any JSON, you can run this check before any EventBridge rule exists.

Set it up

1. Create the webhook source. In the console, go to Sources, Add source, webhook. You can also ask a connected agent to create it, as described in the AI SRE guide. Declare a label per field you want to correlate on and mark one as primary. Then copy the ingest endpoint from the source page.

- name: deploy-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: alarm, field: detail.alarmName, primary: true }
      - { name: source, field: source }

2. Post a sample alarm event to the ingest URL. Use the fields from the real CloudWatch event, and name the alarm after your deploy check.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"version": "0", "detail-type": "CloudWatch Alarm State Change", "source": "aws.cloudwatch", "detail": {"alarmName": "deploy-failed-web", "state": {"value": "ALARM"}}}'

3. If Tares has auth on, the sender needs a Bearer Authorization header holding an ingest-scoped API key. A local instance without auth needs no header. In EventBridge, the connection that holds this header uses API key authorization with header name Authorization and the value Bearer followed by the key.

4. Open the source page. Check Recent events for the new event, and check that the alarm label shows deploy-failed-web. Check Fields to see which payload fields arrive.

5. Fix the mapping where needed. If you want the alarm state as a label, flatten detail.state.value with an input transformer on the rule target, then post a sample in the flattened shape and check again.

6. Wire the real route. In EventBridge, create an API destination with the Tares ingest URL as its endpoint and POST as its method. Give it a connection. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as the target.

For a push source, test_source takes no payload and only returns the ingest endpoint, so it does not check label mapping. Use it to get the URL, and use the posted sample to check the mapping.

Route only the EventBridge path. A CloudWatch alarm action that publishes to SNS does not fit, because an SNS HTTPS subscription cannot send a custom header and its SubscriptionConfirmation message would never be confirmed.

What you get

• A tested source before any deploy fails, with the ingest URL, labels and primary label confirmed.
• A visible event in Recent events, so you know the endpoint accepts your payload shape.
• A Fields view that shows which payload fields arrive and how often, so you can pick a reliable key.
• A clear decision on flattening detail.state.value, made while nothing is on fire.
• A source ready for the first real alarm state change to arrive through the EventBridge rule.

When something else fits

• If your alarms come from Prometheus rules, use the Alertmanager source. It makes one event per alert and keeps firing and resolved status.
• If you only want to confirm that the endpoint accepts JSON, the plain curl from the webhook page is enough, without any CloudWatch fields.

Related