How do I test CloudWatch alarms about crash loops before a real incident?

Tares lets you rehearse a crash loop alarm by posting a CloudWatch-shaped sample event to the ingest URL of a webhook source. You then read the label mapping on the source page and the event on the timeline. CloudWatch reaches Tares through the webhook connector and an EventBridge setup, since Tares has no connector of its own for CloudWatch today.

How it works

CloudWatch alarm state changes travel through Amazon EventBridge. An API destination posts each event as a JSON body to the ingest URL of a Tares webhook source. The connector parses the body whatever the Content-Type and stores it lossless.

Because the real delivery and your test are the same HTTP POST, a sample you send by hand exercises the same path as a live alarm. Three details shape what you see:

• source and detail-type are top-level fields, so event_type_field can read detail-type.
• detail.alarmName is a one-level dotted field, so it works as a label. That is where your crash loop alarm name lands.
• detail.state.value is two levels deep. To use it as a label or in text_template, flatten it with an input transformer on the EventBridge rule target.

The test_source tool does not help with the mapping here. For a push source it takes no payload and only returns the ingest endpoint, so it cannot check labels. Post a sample and read Recent events and Fields on the source page instead.

Set it up

1. Create the webhook source. Ask a connected agent to create a webhook source for CloudWatch alarms, or use Sources → Add source → webhook. Declare a label for the alarm name and mark it primary:

- name: ci-events
  connector: webhook
  config:
    event_type_field: action
    text_template: '{action} by {sender}'
    labels:
      - { name: pipeline, field: pipeline, primary: true }

Adapt it to your alarm: name the source for your cluster, set event_type_field to detail-type, and map a label named alarm to the field detail.alarmName with primary: true. Copy the ingest endpoint from the source page.

2. Post a sample crash loop alarm. This payload follows the CloudWatch alarm state change shape, with an alarm name you choose for the test:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"version":"0","detail-type":"CloudWatch Alarm State Change","source":"aws.cloudwatch","time":"2024-01-01T00:00:00Z","detail":{"alarmName":"checkout-pod-crash-loop","state":{"value":"ALARM","reason":"Restart count above threshold"}}}'

3. If your Tares instance has auth on, the sender needs an Authorization header with the value Bearer followed by an ingest-scoped API key. Add that header to the curl call. With auth off, the key value does not matter.

4. Open the source page. In Recent events, confirm the event arrived and carries the event type and the alarm label. In Fields, check that detail.alarmName shows up and how often it arrives. Fix the label mapping now, while nothing is on fire.

5. Open the timeline and find the event next to your other sources. This tells you whether the crash loop alarm will sit beside deploy, log or metric events when a real one fires.

6. Wire up the real path. In EventBridge, create an API destination whose endpoint is the Tares ingest URL (it must be public HTTPS) and whose method is POST. Give it a connection with API key authorization, header name Authorization, and the value Bearer followed by an ingest-scoped API key. Then create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change, with the API destination as its target.

7. Repeat the sample post, changing the state value to OK, to see how a recovery looks in Recent events.

The verify pattern matches the test batch on the Alertmanager connector page: post a sample, then read the events and fields.

What you get

• A checked label mapping. You see that the alarm name arrives as a label before an alarm fires at night.
• A visible event on the timeline, so you know a crash loop alarm will be correlated with your other sources.
• A list of the fields that arrive and how often, which helps you pick a reliable primary label.
• Early warning about deep fields. If detail.state.value is missing from your labels, you know to add an input transformer on the rule target.
• A repeatable drill you can run again after you rename alarms or change the rule.

For how this feeds the wider workflow, see AI SRE.

When something else fits

• If your alarms publish to SNS instead of EventBridge, this route does not fit. An SNS HTTPS subscription cannot send a custom header, and its subscription confirmation message would never be confirmed. Route the alarm through an EventBridge rule instead.
• If you want the real alarm fired end to end, change the alarm threshold in a non-production account and let CloudWatch send the event. The hand-posted sample only proves the Tares side.

Related