How do I wake an AI SRE once when CloudWatch alarms report a cron job that stopped?

Tares wakes your AI SRE once per stopped cron job when you send CloudWatch alarm events to a webhook source and put a per-entity cooldown on a trigger. The first alarm for an alarm name fires one investigation, and repeats inside the cooldown stay quiet. Every event, including the resolved one, is stored on the timeline that the next dispatch hands over.

How it works

CloudWatch reaches Tares through the webhook connector and a vendor setup in AWS. Tares has no connector of its own for CloudWatch today. EventBridge posts each alarm state change as a JSON body to the source’s ingest URL, and the webhook connector stores it lossless.

Two parts keep the wake-up to one:

• A label that names the entity. The alarm name is at detail.alarmName, a one-level dotted field, so it works as the primary label. Each cron job’s alarm becomes its own entity.
• A trigger with a cooldown. A trigger evaluates its condition per entity with group_by: [key_value]. The cooldown sets the minimum gap between firings for the same entity, so a repeating alarm for one cron job starts one investigation while a different job’s alarm still fires on its own.

Every event the source ingests is stored, whether or not it fires the trigger. When the trigger fires, the dispatch carries the entity’s correlated timeline over context_window. A resolved event for the same alarm name is part of that history, so your agent sees the alarm and the recovery together. This is the pattern behind AI SRE: the agent is woken with context instead of polling.

Set it up

1. Create a webhook source for the alarms, in Sources → Add source → webhook or from the catalog YAML, and copy its ingest URL.

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: alarm, field: detail.alarmName, primary: true }

2. In EventBridge, create an API destination. Its endpoint is the Tares ingest URL, which must be public HTTPS, and its method is POST.
3. Give the API destination a connection, which EventBridge requires. Use API key authorization with the header name Authorization and the value Bearer followed by an ingest-scoped API key. The key only matters when Tares has auth on.
4. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Set the API destination as its target.
5. Add a trigger that watches the source, keys on the alarm label and sets a cooldown. This one narrows to alarms whose name contains nightly-, so name your cron alarms with a shared prefix and change the value to match.

triggers:
  - name: cron_stopped
    sources: [cloudwatch-alarms]
    key_field: alarm
    filters:
      - { field: alarm, op: contains, value: nightly- }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: cron_stopped
      context_window: 15m
    cooldown: 1h

6. Wake an agent from it. A Tares agent is woken by the trigger in a project that wires it up. An external agent registers a webhook with the subscribe tool. Dispatches are at-least-once, so dedupe on dispatch_id.
7. Send a test alarm, then check Recent events and Fields on the source page to confirm that alarm carries the alarm name.

Use the SNS route only if you accept that it does not fit. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed, so use EventBridge as above.

What you get

• One investigation per stopped cron job. The cooldown holds back repeats for the same alarm name for an hour, and other alarms fire independently.
• A dispatch whose payload is the alarm’s correlated timeline, so the agent starts with context and makes no follow-up call.
• Resolved events on the timeline. CloudWatch sends a state change when the alarm returns to OK, and the source stores it like any other event. An alarm and its recovery inside the window arrive together in the dispatch.
• A record of every firing, with its delivery status, in the console under Firings, even when nobody is subscribed.

When something else fits

• If a recovery that arrives after the cooldown should not wake anyone, flatten the state value into a top-level field with an input transformer on the rule target. detail.state.value is two levels deep, so it cannot be a label as it is. Then add a label for it and a trigger filter on the ALARM value. The dispatch then carries only the events that match the filters.
• If you have more than a few alarms, key on the alarm label and keep a shared naming prefix. One trigger then covers all of them.
• If you want a regular review instead of a reaction, give the trigger a schedule with condition: {every: 10m}.

Related