Which logs from Docker containers does an AI SRE search when CloudWatch alarms fire?

Tares gives the AI SRE the Docker logs of the service that the CloudWatch alarm names, on the same timeline as the alarm event. You label the container’s logs and the alarm with the same service value, so the agent reads one service’s evidence instead of every container’s output. CloudWatch reaches Tares through the webhook connector and an EventBridge setup, and the container logs come from the docker_logs connector.

How it works

Tares builds one timeline per service. Every source that carries the same service label lands on that service’s timeline, merged on one clock. That is what narrows the search.

• The Docker logs source tails one container, one event per line, and tags each line with a constant service label.
• The CloudWatch alarm arrives as an event through the webhook connector. Tares has no connector of its own for CloudWatch today. EventBridge posts the alarm state change to a Tares ingest URL, and a webhook source maps the alarm into a label.
• When the alarm event lands, it sits on the service’s timeline next to that service’s log lines. A trigger on the alarm source can wake an agent with that correlated timeline, as described in Build an AI SRE.
• A container whose logs carry a different service value stays off that timeline, so its lines are not part of the read.

You can also narrow at ingest. The match field keeps only lines that match a regex such as ERROR|WARN, and drop skips lines such as access logs.

Set it up

1. Start Tares and open the console.

tares up

2. Add a docker_logs source for the container behind the alarming service. Set container to the name from docker ps, and give the source a service label as the primary label. Run taresd on the host so it can read the Docker daemon.

- name: logs
  connector: docker_logs
  poll: 5s
  config:
    container: tares-demo-api-server   # the container name (docker ps)
    labels:
      - { name: service, const: api-server, primary: true }

3. Add a webhook source for the alarms with the same service label. EventBridge puts the alarm name in detail.alarmName, a one-level dotted field, and detail-type is a top-level field, so it can set the event type. Name the alarm after the service value on the container, here api-server, so both sources land on one timeline. If your alarm names differ, use an input transformer on the rule target to flatten a service field into the payload.

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: service, field: detail.alarmName, primary: true }

4. In EventBridge, create an API destination whose endpoint is the ingest URL from the webhook source page. It must be public HTTPS, and the method is POST.
5. Give the destination a connection, which EventBridge requires. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key value only matters when Tares has auth on.
6. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as its target. Alarm actions that publish to SNS do not fit, because an SNS HTTPS subscription cannot send a custom header and its confirmation message would never be confirmed.
7. Check Recent events on both sources. Then ask a connected agent: “Use tares: what happened to api-server in the last 15 minutes?” The agent reads the api-server timeline, with the alarm and the container’s log lines together.

What you get

• An investigation that starts from the alerting service. The agent reads that service’s logs and the alarm, not a search across every container.
• A time frame you control. The timeline is merged on one clock, so you can ask for the last 15 minutes and see the first alarm event and the log lines around it.
• Less noise when you add match or drop filters on the Docker source.
• One path for later alarms. Each new alarm from the same rule lands on the same timeline.
• A finding the agent writes back to the timeline, so the next investigation starts from the previous conclusion.

When something else fits

• If you want an agent to notice a problem that no alarm covers, use a scheduled first-level agent as in Watch, then escalate.
• If your logs come from somewhere other than a local Docker daemon, such as an OpenTelemetry pipeline, use the matching connector and keep the same service label.

Related