How do I test CloudWatch alarms about slow database queries before a real incident?

Tares lets you test a CloudWatch slow-query alarm by posting a sample alarm event to a webhook source’s ingest URL and reading the result on the source page. The event shows under Recent events, and Fields shows whether your label mapping picked up the alarm name. You find mapping mistakes before a real alarm fires. Tares has no connector of its own for CloudWatch today, so alarms reach Tares through the webhook connector and an EventBridge setup.

How it works

CloudWatch alarm state changes reach Tares as EventBridge events, posted as a JSON body to the ingest URL of a webhook source. The webhook connector parses the body whatever the Content-Type and stores the payload lossless. You declare labels to map payload fields into the event.

The event has these top-level fields:

• source, for example aws.cloudwatch
• detail-type, for example CloudWatch Alarm State Change
• detail, which holds alarmName and a nested state

A label can read detail.alarmName, because one-level dotted fields work. detail.state.value is two levels deep, so it does not work as a label or in text_template unless you flatten it first with an input transformer on the EventBridge rule target.

For a push source, test_source takes no payload. It only returns the ingest endpoint, so it does not check your label mapping. To test the mapping, post a sample payload yourself and read Recent events and Fields on the source page.

Set it up

1. Create a webhook source for your alarms. In Sources → Add source → webhook, or by asking a connected agent to create one, declare a label for the alarm name and mark it primary. Keep detail-type as the event type field, since it is a top-level field.

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: alarm, field: detail.alarmName, primary: true }
      - { name: source, field: source }

2. Copy the ingest endpoint from the source page. It has the form /ingest/<ingest_key> on your Tares address.

3. Post a sample alarm shaped like a real CloudWatch event. Use an alarm name that matches your slow-query alarm.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"version":"0","detail-type":"CloudWatch Alarm State Change","source":"aws.cloudwatch","time":"2024-01-01T00:00:00Z","detail":{"alarmName":"rds-slow-queries","state":{"value":"ALARM","reason":"Threshold crossed"}}}'

If Tares has auth on, the request also needs an Authorization header with the value Bearer followed by an ingest-scoped API key. Without auth, no key is needed.

4. Open the source page and check Recent events. Your sample should appear with the event type CloudWatch Alarm State Change and the alarm label set to rds-slow-queries.

5. Open Fields and confirm which payload fields arrive. If you want the alarm state as a label, flatten detail.state.value with an input transformer on the rule target, then post again and recheck.

6. Wire up the real path in EventBridge once the mapping looks right:

   • Create an API destination whose endpoint is the Tares ingest URL and whose method is POST. The endpoint must be public HTTPS.
   • Give it a connection that uses API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key value only matters when Tares has auth on.
   • Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change, with the API destination as its target.

A local address such as http://localhost:8787 is fine for the curl test, but EventBridge cannot reach it. For the live route you need a public HTTPS URL.

Alarm actions that publish to SNS do not fit this route. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed.

What you get

• You see a CloudWatch-shaped alarm as an event in Tares before any real slow-query incident.
• You confirm that the alarm name arrives as the label you correlate on, and that the primary label keys the entity the way you expect.
• You learn early which fields need flattening, such as detail.state.value, instead of discovering it during an incident.
• You can repeat the post as often as you like while you adjust labels, because Fields shows what arrives and how often.

Once the source behaves, the same alarms feed the rest of your setup. Start from AI SRE to see how events from several sources are correlated and acted on.

When something else fits

• If your alarms are already routed through Alertmanager, use the alertmanager connector. It makes one event per alert and has its own ingest URL, and you verify it with the same kind of curl post.
• If you only want to confirm that a source exists and report its ingest endpoint, test_source is enough. It does not check label mapping, so for that, post a sample payload.

Related