How do I pair CloudWatch alarms with logs from Docker containers for an AI SRE?

Tares pairs CloudWatch alarms with Docker container logs by giving both sources the same service label, so a firing alarm arrives next to the log lines from the same window. CloudWatch reaches Tares through the webhook connector and an EventBridge API destination, a vendor setup on the AWS side. The docker_logs connector tails the container. Your AI SRE then reads one timeline per service.

How it works

• One label joins the two sources. Both sources declare a service label and mark it primary. Tares merges events that carry the same service value onto one timeline, in time order. This is the same per-service correlation described in Build an AI SRE.
• Alarms arrive through the webhook connector. Tares has no connector of its own for CloudWatch today. An EventBridge rule matches CloudWatch alarm state changes and sends each one as a JSON POST to the source’s ingest URL. The webhook connector stores the payload lossless and parses it whatever the Content-Type.
• Logs arrive through docker_logs. The docker_logs connector tails a running container, one event per line, and polls every 5 seconds in the example below. taresd needs access to the Docker daemon, so run it on the host or mount the Docker socket.
• The label value comes from the alarm name. source and detail-type are top-level fields in the EventBridge event. detail.alarmName is one level deep, so it works as a label field. detail.state.value is two levels deep, so flatten it with an input transformer on the rule target if you want it as a label or in text_template.

Set it up

1. Tail the container. Create a docker_logs source and set a constant service label. Use the service name you will also use for the alarm.

- name: logs
  connector: docker_logs
  poll: 5s
  config:
    container: tares-demo-api-server   # the container name (docker ps)
    labels:
      - { name: service, const: api-server, primary: true }

2. Create the alarm source. Add a webhook source whose service label reads the alarm name. Name the CloudWatch alarm after the service, here api-server, so the two values match. If your alarm names differ from your service names, flatten a service value to a top-level field with an input transformer on the rule target and point the label at that field.

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: service, field: detail.alarmName, primary: true }

3. Copy the ingest URL. Take the ingest endpoint from the source page. EventBridge needs it to be public HTTPS.
4. Create the API destination. In EventBridge, create an API destination whose endpoint is the Tares ingest URL and whose method is POST.
5. Add the connection. EventBridge requires one on every API destination. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key value only matters when Tares has auth on.
6. Create the rule. On the default event bus, match source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Set the API destination as the rule’s target.
7. Test the mapping. Post a sample event to the ingest URL, then check Recent events and Fields on the source page.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"detail-type": "CloudWatch Alarm State Change", "source": "aws.cloudwatch", "detail": {"alarmName": "api-server"}}'

8. Read the timeline. Connect an MCP client with tares mcp and ask what happened to api-server in the last 15 minutes. You can also add a trigger over both sources, keyed by service, to wake an agent when an alarm event arrives.

Use the EventBridge route for alarms. CloudWatch alarm actions that publish to SNS do not fit. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed.

What you get

• A firing alarm and the container’s log lines for the same service on one clock, in one read.
• An agent that starts from the alarm and the surrounding log lines instead of querying CloudWatch and Docker separately.
• A lossless copy of every alarm payload, so fields you did not map are still there to read.
• A setup that grows by adding sources: metrics, deploys or more containers join the same timeline once they carry the same service label.

When something else fits

• If your alerts come from Prometheus rules or Alertmanager, use the prometheus_alerts or alertmanager connector instead of EventBridge and a webhook.
• If your logs come from an OpenTelemetry pipeline instead of a container on your Docker host, use the otlp connector for them and keep the same service label.

Related