Which logs from Docker containers does an AI SRE search when Sentry issue alerts fire?

Tares narrows the search to the Docker container logs that carry the same service label as the Sentry alert. When a Sentry issue alert fires, the agent reads that one service’s timeline, not every container’s logs. Sentry reaches Tares through the webhook connector and a Sentry setup, and the logs come from a docker_logs source.

How it works

Tares keeps one timeline per service. Every source that carries the same service label lands on that service’s timeline, merged on one clock. The Build an AI SRE guide wires metrics, logs and alerts this way.

For Sentry issue alerts, two sources share the label:

• Sentry reaches Tares through the webhook connector and a Sentry setup. Tares has no connector of its own for Sentry today. Sentry posts to the source’s ingest URL, and Tares stores the payload lossless. The webhook connector page covers labels and payload mapping.
• Docker logs come from a docker_logs source that tails one running container, one event per line. The Docker logs connector page lists its options.

When a trigger on the Sentry source fires, the agent reads the timeline for the alerting service. That read holds the Sentry event and the log lines from that service’s container, side by side. Containers you did not label with that service stay out of the read.

You narrow further at two points:

• At ingestion. match keeps only lines matching a regex, such as ERROR|WARN. drop skips lines matching a regex, such as access logs.
• At read time. The agent asks for a window. The example question in the AI SRE guide asks what happened to a service in the last 15 minutes, so the agent starts from the minutes around the alert.

Set it up

1. Tail the container of the service you care about, and give it a service label. Set container to the name from docker ps. Tares needs access to the Docker daemon, so run taresd on the host.

- name: logs
  connector: docker_logs
  poll: 5s
  config:
    container: tares-demo-api-server   # the container name (docker ps)
    labels:
      - { name: service, const: api-server, primary: true }

2. Optionally add match or drop under config so the timeline holds the lines that help during an incident. If the container runs under Compose, add compose_file so the tail follows restarts.

3. In Sentry, add an integration under Settings, Developer Settings. Set its Webhook URL to the Tares ingest URL. Subscribe it to the resources you want, such as issues or errors. Turn on Alert Action to use it from alert rules.

4. Create a webhook source for Sentry with the same service value. Sentry signs its deliveries, so the signature is the authentication. Set the signature scheme to hmac_sha256, the signature header to Sentry-Hook-Signature, and the signing secret to the integration Client Secret. The source needs no Tares API key. action is a top-level field, so it can be the event_type_field. The signing options are described in the Tares release notes.

- name: sentry
  connector: webhook
  config:
    event_type_field: action
    signature_scheme: hmac_sha256
    signature_header: Sentry-Hook-Signature
    signing_secret: <client-secret>
    labels:
      - { name: service, const: api-server, primary: true }

5. Add a trigger over the Sentry source, keyed by service, so an alert wakes the agent for that one service.

triggers:
  - name: sentry-issue
    sources: [sentry]
    key_field: service
    condition: { aggregate: count, predicate: "> 0", window: 5m }
    cooldown: 5m

6. Point an agent at the trigger. Open the service in Explore to see the Sentry event and the container logs on one timeline.

What you get

• An agent that starts from the alerting service’s own container logs and the Sentry event, not from every log you collect.
• One correlated timeline per service, so the Sentry issue and the log lines around it read in order.
• Less noise in each read, because match and drop filters keep routine lines such as access logs out of the timeline.
• Signed Sentry deliveries, with the signature as the credential.
• A finding the agent writes back onto the same timeline, so the next investigation starts from it.

When something else fits

• If you want an agent to look over your sources on a schedule and only escalate when something moves, use the pattern in Watch, then escalate.
• If your containers send logs through an OpenTelemetry pipeline, an OTLP source can replace docker_logs. Keep the same service label.

Related