What should CloudWatch alarms about a disk filling up include for an AI SRE?

Tares works best when a disk-filling alarm carries a service name, a severity, a summary and a start time as flat top-level fields. CloudWatch reaches Tares through the webhook connector and an EventBridge setup on the AWS side, because Tares has no connector of its own for CloudWatch today. With those four fields, the alarm lands on the same per-service timeline as your metrics and logs, so the agent can tie it to evidence.

How it works

EventBridge posts each alarm state change to a Tares webhook source as a JSON body. The event already carries detail.alarmName, detail.state.reason, time, region, account and the alarm ARN under resources. It has no service name or severity of its own.

Tares reads labels from top-level fields and from one-level dotted fields. detail.alarmName is one level deep, so it works as a label directly. detail.state.reason and detail.state.value are two levels deep, so they only work as a label or in text_template after you flatten them. Do that with an input transformer on the EventBridge rule target.

The AI SRE guide keys every source on a shared service label, and Tares builds one timeline per service. If the alarm carries the same service name as your metrics and logs, it shows up on that service’s timeline next to them.

For a disk alarm, shape the transformer output to carry:

• Service name: the value that matches the service label on your other sources. This is what joins the alarm to the right timeline.
• Severity: so the agent and your triggers can tell a warning at 80 percent from a critical alarm near full.
• Summary: a short human line. The alarm’s state reason (detail.state.reason) is a good source for it.
• Start time: a top-level timestamp for when the alarm changed state. The event’s own time field is top level, so you can use it too.

You choose the names of these fields in the transformer. The webhook source maps whatever you name to labels.

Set it up

1. In EventBridge, create an API destination whose endpoint is your Tares ingest URL and whose method is POST. The endpoint must be public HTTPS.
2. Give the API destination a connection, which EventBridge requires. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key only matters when Tares has auth on.
3. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Set the API destination as its target.
4. On the rule target, add an input transformer that flattens the event into top-level fields for the service name, severity, summary and start time. Include any other field you want as a label.
5. Create a webhook source in Tares under Sources, Add source, webhook. Declare a label per field, with the service label marked primary:

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_time_field: startTime
    text_template: '{summary}'
    labels:
      - { name: service, field: service, primary: true }
      - { name: severity, field: severity }

6. Trigger a test alarm, then open the source page. Recent events shows what arrived, and Fields shows which payload fields come through and how often. Fix the transformer if a label comes up empty.

Use an EventBridge rule for this. An alarm action that publishes to SNS does not fit, because an SNS HTTPS subscription cannot send a custom header and its SubscriptionConfirmation message would never be confirmed.

What you get

• Each disk alarm lands as an event on the timeline of the service it names, merged on one clock with that service’s metrics and logs.
• The agent reads the summary and the start time, then looks at what else happened on that service around that moment, such as a deploy or a burst of log writes.
• Severity is a label, so you can filter on it in a trigger and decide which alarms wake an agent.
• The full CloudWatch event is stored lossless, so fields you did not map are still there to read.
• The alarm name and region stay available for your own labels if you keep them in the transformer output.

When something else fits

• If the disk metric is already in a Prometheus server, the AI SRE guide shows how to ingest the metrics and the alerts that Prometheus rules fire, with no EventBridge setup.
• If your service sends its own HTTP calls, post a flat JSON payload straight to a webhook source and skip the transformer.

Related