How do I test CloudWatch alarms about high latency before a real incident?

Tares lets you test a CloudWatch high latency alarm by posting a sample alarm event to your webhook source’s ingest URL and reading the result under Recent events and Fields. CloudWatch reaches Tares through the webhook connector and an EventBridge setup, because Tares has no connector of its own for CloudWatch today. Once the sample event shows the labels you expect, you wire EventBridge to the same ingest URL for real alarm state changes.

How it works

A CloudWatch alarm state change arrives as an EventBridge event, and EventBridge delivers it as a JSON body to the ingest URL of a Tares webhook source. The connector parses the body whatever the Content-Type and stores the payload lossless.

• source and detail-type are top-level fields, so you can read them directly.
• detail.alarmName is a one-level dotted field, so it works as a label.
• detail.state.value is two levels deep, so it does not work as a label or in text_template as it arrives. Flatten it with an input transformer on the EventBridge rule target if you want it.

For a push source, test_source takes no payload and only returns the ingest endpoint. It does not check your label mapping. To check the mapping, post a sample payload yourself and read the result on the source page.

The curl test below posts straight to the ingest URL. It does not go through EventBridge. It tells you how Tares maps a CloudWatch-shaped event, not whether your EventBridge rule and API destination are wired correctly. You check that part in step 5.

Set it up

1. Create a webhook source, in the console under Sources, Add source, webhook, or by asking a connected agent to create one. Declare a label for detail.alarmName and mark it primary. Read the event type from the top-level detail-type field.

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: alarm, field: detail.alarmName, primary: true }
      - { name: source, field: source }

2. Save the source and copy the ingest endpoint from the source page. It looks like http://localhost:8787/ingest/<ingest_key> on a local Tares.

3. Post a sample alarm event shaped like the real one. Use an alarm name that says what it is, so you can find it later.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"version": "0", "detail-type": "CloudWatch Alarm State Change", "source": "aws.cloudwatch", "detail": {"alarmName": "checkout-high-latency", "state": {"value": "ALARM", "reason": "Threshold crossed"}}}'

On an instance with auth on, the sender also needs an Authorization: Bearer header holding an ingest-scoped API key. Without auth, the key value does not matter.

4. Open the source page. Under Recent events, find your sample event and check that the event type is the alarm state change and that the alarm label shows your alarm name. Under Fields, check which payload fields arrived. Adjust the labels and post again until the mapping is right.

5. Connect the real route, using the same ingest URL:

   • In EventBridge, create an API destination whose endpoint is the Tares ingest URL and whose method is POST. The URL must be public HTTPS.
   • Give it a connection with API key authorization, header name Authorization and the value Bearer followed by an ingest-scoped API key. The value only matters when Tares has auth on.
   • Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as its target.

6. Trigger or wait for a real alarm state change and confirm in Recent events that it looks like your sample.

What you get

• You see how a CloudWatch high latency alarm lands in Tares before a real incident does, including the event type and the alarm label.
• Fields shows which payload fields arrive and how often, so you pick a reliable primary label.
• You learn early which nested fields, such as detail.state.value, need flattening with an input transformer.
• Real alarm events reach Tares through the EventBridge route in step 5, and you can read them alongside your other sources. For the wider picture of how Tares correlates sources, see AI SRE.

When something else fits

• CloudWatch alarm actions that publish to SNS do not fit this route. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed. Use the EventBridge route above instead.
• If your alerts already pass through Prometheus, the alertmanager connector gives you one event per alert and has its own sample batch you can post to test it.

Related