How do I test CloudWatch alarms about error spikes before a real incident?

Tares tests CloudWatch alarms for error spikes when you post a sample alarm event to a webhook source’s ingest URL and check Recent events and Fields. You see the event and your label mapping before any real alarm fires. CloudWatch reaches Tares through the webhook connector and an EventBridge setup, because Tares has no connector of its own for CloudWatch today.

How it works

A CloudWatch alarm state change reaches Tares as an EventBridge event. An EventBridge rule matches it and sends it to an API destination that points at the ingest URL of a Tares webhook source. The webhook connector stores the JSON body lossless and parses it whatever the Content-Type.

The same path works for a test. You post a sample payload with the same shape to the ingest URL, and Tares treats it like a real alarm event.

• source and detail-type are top-level fields, so they work as the event type field or as labels.
• detail.alarmName is one level deep, so it works as a dotted label.
• detail.state.value is two levels deep. To use the ALARM state as a label or in a text template, flatten it with an input transformer on the rule target.
• test_source is not the right check here. For a push source it takes no payload and only returns the ingest endpoint, so it does not check the label mapping. Post a sample payload and read the source page instead.

Set it up

1. Ask a connected agent to create the source, or add it in the console under Sources, Add source, webhook:

   Use tares to create a webhook source for my CloudWatch alarms and give me the ingest URL.

   The agent calls create_source and returns the ingest URL.

2. Declare a label for each field you want to correlate on, and mark one as primary. This example keys on the alarm name and reads the event type from the top-level detail-type field:

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: alarm, field: detail.alarmName, primary: true }

3. Post a sample error-spike alarm to the ingest URL. The body follows the CloudWatch alarm event shape that EventBridge sends:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"version": "0", "id": "test-1", "detail-type": "CloudWatch Alarm State Change", "source": "aws.cloudwatch", "time": "2024-01-01T00:00:00Z", "region": "us-east-1", "resources": ["arn:aws:cloudwatch:us-east-1:123456789012:alarm:checkout-5xx-errors"], "detail": {"alarmName": "checkout-5xx-errors", "state": {"value": "ALARM", "reason": "Error count above threshold"}}}'

If Tares has auth on, add an Authorization Bearer header that holds an ingest-scoped API key. Without auth the key value does not matter.

4. Open the source page. Under Recent events, check that the event arrived with the event type and the alarm label you expect. Under Fields, check which payload fields arrive and how often, and pick a reliable key.

5. If you need the ALARM state as a label, flatten detail.state.value with an input transformer on the rule target, then post a flattened sample and check Fields again.

6. Wire up the real path. In EventBridge, create an API destination whose endpoint is the Tares ingest URL (it must be public HTTPS) and whose method is POST. Give it a connection with API key authorization, header name Authorization and the value Bearer followed by an ingest-scoped API key. Then create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as its target.

What you get

• You confirm the label mapping before an incident, so the alarm name keys the entity you expect.
• You see the exact event Tares stores, with the full payload kept lossless.
• You learn early whether a field is too deep to use, such as detail.state.value, and fix it with an input transformer.
• Your CloudWatch alarm events sit next to your other sources, so Tares can correlate them when a real spike happens. For the wider picture, see AI SRE.

When something else fits

• CloudWatch alarm actions that publish to SNS do not fit this route. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed. Use an EventBridge rule with an API destination instead.
• If your error-spike alerts come through Prometheus and Alertmanager, the Alertmanager connector gives one event per alert and has its own ingest URL.

Related