How do I test CloudWatch alarms about 5xx errors on an API before a real incident?

Tares lets you test CloudWatch 5xx alarms by posting a sample alarm event to a webhook source’s ingest URL, then checking Recent events and Fields. CloudWatch reaches Tares through the webhook connector and an EventBridge setup, because Tares has no connector of its own for CloudWatch today. Once the sample shows the labels you want, you wire up EventBridge and wait for a real alarm state change.

How it works

CloudWatch alarms reach Tares through the webhook connector and a vendor setup in EventBridge. An EventBridge rule matches alarm state changes, and an API destination posts each one as a JSON body to the source’s ingest URL. The webhook connector parses it whatever the Content-Type and stores the payload lossless.

Two things matter for testing:

• source and detail-type are top-level fields, so they work directly for labels and event types.
• detail.alarmName is one level deep and works as a dotted label. detail.state.value is two levels deep, so it does not work as a label or in text_template unless you flatten it with an input transformer on the rule target.

For a push source like this one, test_source takes no payload and only returns the ingest endpoint. It does not check the label mapping. To check the mapping, post a sample payload yourself and read Recent events and Fields on the source page. This fits the wider approach in AI SRE: get the alert into Tares first, then let correlation work on it.

Set it up

1. Create a webhook source for the alarms. Ask a connected agent: “Use tares to create a webhook source for my CloudWatch alarms and give me the ingest URL.” Or use Sources → Add source → webhook in the console. Declare a label per field you want to correlate on and mark one primary:

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: alarm, field: detail.alarmName, primary: true }
      - { name: source, field: source }

2. Copy the ingest endpoint from the source page. It has the form /ingest/<ingest_key>. If you call test_source here, it returns only that endpoint.

3. Post a sample alarm event for your API’s 5xx alarm. The body follows the CloudWatch alarm event shape, with the alarm name set to the one you use for 5xx errors:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"source": "aws.cloudwatch", "detail-type": "CloudWatch Alarm State Change", "detail": {"alarmName": "api-5xx-errors", "state": {"value": "ALARM"}}}'

If your Tares instance has auth on, the sender also needs an Authorization: Bearer header holding an ingest-scoped API key. The key only matters when Tares has auth on.

4. Open the source page. Under Recent events, confirm the event arrived with the event type you expect. Under Fields, confirm the alarm label carries the alarm name. If you want the alarm state as a label, flatten detail.state.value with an input transformer on the EventBridge rule target, then post a flattened sample and check again.

5. Wire up the real path in EventBridge:

   • Create an API destination whose endpoint is the Tares ingest URL. It must be public HTTPS, and the method is POST.
   • Give it a connection, which EventBridge requires on every API destination. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key.
   • Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as its target.

6. When the alarm next changes state, the rule delivers the event to the same ingest URL. Check Recent events again to see it arrive.

What you get

• A checked label mapping before an incident, so the alarm name is already a label on the day the 5xx alarm fires.
• A visible list of which payload fields arrive and how often, from Fields, so you can pick a reliable key.
• Knowledge up front that detail.state.value needs flattening, instead of finding out during an outage.
• An alarm source that is ready for correlation with your other sources, as described in AI SRE.

When something else fits

• If your alarms publish to SNS instead of EventBridge, this route does not fit. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed. Route the alarms through an EventBridge rule with an API destination instead.
• If the 5xx signal is a Prometheus rule routed through Alertmanager, use the alertmanager connector, which makes one event per alert.

Related