How do I route CloudWatch alarms about a cron job that stopped to an AI SRE?

Tares receives CloudWatch alarms through the webhook connector and a vendor setup in EventBridge, and a trigger filtered to your cron alarms wakes the agent only for that symptom. Tares has no connector of its own for CloudWatch today. You send alarm state changes to a webhook source, then write a trigger that matches only the alarm names for the stopped cron job. Every other alarm lands in the source but does not wake the agent.

How it works

CloudWatch alarm state changes reach Tares in three hops:

• An EventBridge rule on the default event bus matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change.
• The rule’s target is an API destination that POSTs the event as a JSON body to your Tares ingest URL. The webhook connector parses it whatever the Content-Type and stores it lossless.
• A trigger on that webhook source filters the stream. Only events that match the alarm names you list count toward the condition, so only those wake the agent.

source and detail-type are top-level fields in the event. detail.alarmName is one level deep, so it works as a label. detail.state.value is two levels deep. If you want the alarm state as a label or in text_template, flatten it to a top-level field with an input transformer on the rule target.

CloudWatch alarm actions that publish to SNS are a different route and do not fit. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed. Use EventBridge.

Set it up

1. In Tares, create a webhook source and copy its ingest URL. Declare a label for the alarm name and mark it primary:

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type: cloudwatch_alarm
    labels:
      - { name: alarm, field: detail.alarmName, primary: true }

2. In EventBridge, create an API destination. Its endpoint is the Tares ingest URL, which must be public HTTPS, and its method is POST.
3. Give the API destination a connection, which EventBridge requires on every API destination. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key only matters when Tares has auth on.
4. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Set the API destination as its target.
5. Check the mapping by posting a sample event to the source and reading Recent events and Fields on the source page:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"version": "0", "detail-type": "CloudWatch Alarm State Change", "source": "aws.cloudwatch", "detail": {"alarmName": "nightly-backup-heartbeat"}}'

6. Add a trigger that names the symptom and filters on the alarm label. List the alarms that mean the cron job stopped. The names below are examples, so use your own:

- name: cron-job-stopped
  sources: [cloudwatch-alarms]
  filters:
    - { field: alarm, op: in, value: [nightly-backup-heartbeat, invoice-export-heartbeat] }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 15m

7. Subscribe your agent to the cron-job-stopped trigger. Use a Tares agent, your own agent service over a webhook, or a Slack channel, as described in Build an AI SRE. The cooldown keeps a flapping alarm from waking the agent again for 15 minutes after a firing.

What you get

• Only alarms for the cron job you named wake the agent. Other CloudWatch alarms still arrive and stay in the source, so you can read them later.
• The trigger is named for the symptom, cron-job-stopped, so each firing in the console shows which symptom woke the agent.
• The full CloudWatch event stays in the stored payload, including the alarm reason, the previous state and the configuration.
• The trigger decides who gets woken: a Tares agent, your own agent service, or a Slack channel.
• You can add the next symptom, such as a queue that stopped draining, as another trigger on the same webhook source with its own alarm names.

When something else fits

• If you want one event per alert from Alertmanager-style routing, use the alertmanager connector instead. It has its own ingest URL.
• If you only need CloudWatch data and not alarms, http_poll against an API or OTLP can carry it into Tares.
• If the cron job posts its own heartbeat, point it straight at a webhook source with curl and skip CloudWatch.

Related