How do I test CloudWatch alarms about a disk filling up before a real incident?

Tares lets you test a CloudWatch disk alarm by posting a sample EventBridge alarm event to your webhook source’s ingest URL and reading the result under Recent events and Fields. Tares has no connector of its own for CloudWatch today. Alarms reach Tares through the webhook connector and an EventBridge setup. You confirm that your labels map correctly before any real disk fills up.

How it works

CloudWatch alarms reach Tares through the webhook connector and a vendor setup in AWS. An EventBridge rule matches the alarm state change event and sends it to an API destination, which POSTs it to the ingest URL of a Tares webhook source.

• The webhook connector parses the JSON body whatever the Content-Type and stores the payload lossless.
• source and detail-type are top-level fields, so event_type_field can read them.
• detail.alarmName is a one-level dotted field, so it works as a label.
• detail.state.value is two levels deep. To use it as a label or in text_template, flatten it with an input transformer on the rule target.

Because a test is just an HTTP POST, you can send the same shape of body yourself and see exactly what Tares records. For a push source, test_source takes no payload and only returns the ingest endpoint, so it does not check the label mapping. The sample POST is the way to check the mapping.

Alarm actions that publish to SNS are a different route and do not fit. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed.

Set it up

1. Create the webhook source. Ask a connected agent to create a webhook source and give you the ingest URL, or use Sources → Add source → webhook in the console. Declare a label per field you want to correlate on and mark one primary:

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: alarm, field: detail.alarmName, primary: true }
      - { name: region, field: region }

2. Send a sample alarm event to the ingest URL. The body follows the EventBridge alarm shape, with a disk alarm name as an example:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"version":"0","detail-type":"CloudWatch Alarm State Change","source":"aws.cloudwatch","region":"us-east-1","detail":{"alarmName":"DiskFillingUp","state":{"value":"ALARM","reason":"Disk usage above threshold"}}}'

3. If your Tares has auth on (tares up --auth), add an Authorization: Bearer header holding an ingest-scoped API key to the request. Without auth on, the key value does not matter.

4. Open the source page and check Recent events. You should see one event whose type is CloudWatch Alarm State Change and whose alarm label is DiskFillingUp.

5. Open Fields on the same page. It shows which payload fields arrive and how often, so you can confirm detail.alarmName and region are present and pick a reliable key. Adjust the labels and send the sample again until the event looks right.

6. Wire up the live route in AWS:

   • In EventBridge, create an API destination whose endpoint is the Tares ingest URL and whose method is POST. The endpoint must be public HTTPS.
   • Give it a connection, which EventBridge requires on every API destination. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key value only matters when Tares has auth on.
   • Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as its target.

7. Trigger the disk alarm in a safe way, for example by setting it to the ALARM state, and confirm a second event appears under Recent events. Compare it with the one from your sample and, if detail.state.value is missing as a label, add the input transformer that flattens it.

What you get

• A tested label mapping for your disk alarms before an incident, with the alarm name as the primary label.
• A visible list of the fields that arrive, so you know which ones you can correlate on.
• A recorded event in Recent events that you can compare against the event from a real alarm.
• The full alarm payload stored lossless, so fields you did not map are still there later.
• A route you can repeat: change the alarm name or region in the sample and post again whenever you add another disk alarm.

To see how Tares uses these events once they arrive, start from AI SRE.

When something else fits

• If your disk alerts come from Prometheus rules through Alertmanager, use the Alertmanager connector. It fans a batch out into one event per alert and tracks firing and resolved.
• If your alarms can only publish to SNS, the EventBridge API destination route described here does not apply to them.

Related