Skip to Content

How do I test CloudWatch alarms about a cron job that stopped before a real incident?

Tares lets you rehearse a stopped-cron alarm by posting a CloudWatch-shaped sample payload to the webhook source’s ingest URL. You then check the label mapping on the event it records. CloudWatch reaches Tares through the webhook connector and an EventBridge setup, and Tares has no connector of its own for CloudWatch today. Do the test first, so the real alarm lands on a source you have already checked.

How it works

  • CloudWatch alarm state changes reach Tares through EventBridge. A rule matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and an API destination posts the event as a JSON body to the Tares ingest URL.
  • The webhook connector stores the payload lossless and parses it whatever the Content-Type. source and detail-type are top-level fields. detail.alarmName works as a one-level dotted label.
  • detail.state.value is two levels deep. To use it as a label or in text_template, flatten it with an input transformer on the rule target.
  • A sample POST with the same shape goes through the same path as a real alarm. It shows you what Tares records before the cron job ever stops.
  • For a push source, test_source takes no payload and only returns the ingest endpoint, so it does not check the label mapping. Use a posted sample payload for that check.

Alarm actions that publish to SNS are a different route and do not fit. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed.

Set it up

  1. Create a webhook source for the alarms. Ask a connected agent: “Use tares to create a webhook source for my CloudWatch alarms.” The agent calls create_source and returns the ingest URL. You can also use Sources, Add source, webhook in the console.
  2. Declare a label for the alarm name and mark it primary. Read the event type from the top-level detail-type field:
- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: alarm, field: detail.alarmName, primary: true }
  1. Post a sample alarm for the stopped cron job to the ingest endpoint. The alarm name here is only an example:
curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"version": "0", "detail-type": "CloudWatch Alarm State Change", "source": "aws.cloudwatch", "detail": {"alarmName": "nightly-backup-cron-stopped", "state": {"value": "ALARM", "reason": "No heartbeat"}}}'
  1. Open the source page and read Recent events. Check that the event type is CloudWatch Alarm State Change and that the alarm label holds the alarm name. Then open Fields to see which payload fields arrive and how often.
  2. If you want the alarm state as a label, add an input transformer on the EventBridge rule target that lifts detail.state.value to a top-level or one-level field. Post a sample with that shape and check Fields again.
  3. Wire up the real route. In EventBridge, create an API destination whose endpoint is the Tares ingest URL (it must be public HTTPS) and whose method is POST. Give it a connection with API key authorization, header name Authorization and the value Bearer followed by an ingest-scoped API key. The key value only matters when Tares has auth on. Create a rule on the default event bus that matches the source and detail-type above, and set the API destination as its target.
  4. Repeat the sample POST against the public URL if you want to confirm that the URL is reachable. When Tares has auth on, send the same ingest-scoped key as Authorization: Bearer <key>.

What you get

  • You know the label mapping works before the real alarm fires, so the stopped cron job arrives keyed by its alarm name.
  • You see the exact event Tares records, including the lossless payload, in Recent events.
  • Fields shows you which payload fields to promote to labels, and which ones need flattening first.
  • The alarm events sit alongside your other sources, so AI SRE can correlate a stopped cron job with whatever else changed around it.
  • The EventBridge route is set up with the right authentication before you depend on it.

When something else fits

  • If your alarms come from Prometheus rules routed through Alertmanager, use the alertmanager connector. It makes one event per alert, with its own ingest URL.
  • If you only want to prove the ingest URL accepts data, any small JSON POST is enough. Use the CloudWatch-shaped sample when you want to check labels.

Still stuck? Get help

Last updated on