Which OpenTelemetry log records does an AI SRE search when CloudWatch alarms fire?

Tares gives an AI SRE the OpenTelemetry log records of the one service whose CloudWatch alarm fired, merged on a single timeline with the alarm event. The service label is the key, so the agent reads that service’s logs for the window around the alarm and does not search every log you ingest. CloudWatch alarms reach Tares through the webhook connector and an EventBridge setup, and the logs arrive through the otlp connector.

How it works

Both signals carry a service label, and Tares builds one timeline per label value. This is the pattern in the AI SRE guide.

• Logs: an OpenTelemetry source receives OTLP logs, traces and metrics. It reads the service name from the resource attribute resourceAttributes.service.name, so every service that exports OTLP lands under its own service value with no per-service setup.
• Alarms: CloudWatch reaches Tares through the webhook connector and an EventBridge setup. Tares has no connector of its own for CloudWatch today. EventBridge posts the alarm state change as a JSON body, and the webhook source stores it lossless.
• The join: the webhook source declares a service label too. When an alarm event arrives, a trigger keyed by service fires for that service only. The agent then reads the correlated timeline for that service: its OTel log records, the alarm event and any other sources you labeled the same way.

The label value on the alarm side has to equal the service.name your applications export. The alarm payload names the alarm in detail.alarmName, which works as a one-level dotted label, so name your alarms after the service they guard. The nested field detail.state.value is two levels deep. Flatten it with an input transformer on the rule target if you want the alarm state as a label or in text_template.

Set it up

1. Create an OTLP source keyed by service:

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: resourceAttributes.service.name, primary: true }

2. Point each exporter or OpenTelemetry Collector at Tares:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

3. Create a webhook source for the alarms. The detail-type and source fields are top level, and the service label reads the alarm name:

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: service, field: detail.alarmName, primary: true }

4. In EventBridge, create an API destination whose endpoint is the Tares ingest URL. It must be public HTTPS, and the method is POST.
5. Give the API destination a connection, which EventBridge requires. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key only matters when Tares has auth on.
6. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as its target.
7. Add a trigger over the alarm source, keyed by service, and wake an agent with it. The agent reads the timeline for the alerting service. If you read interactively, ask for a window such as the last 15 minutes. Open Recent events on each source to confirm the alarm and the logs carry the same service value.

What you get

• The first look at an alarm starts from one service’s log records and the alarm event, side by side on one clock, and not from a search across all logs.
• Other services’ logs stay out of the agent’s context, so the evidence it cites belongs to the service that alerted.
• Every OTLP service shares one source, so a new service gets this behavior as soon as it exports logs with a service.name.
• The agent writes its finding back onto the same service timeline. The next investigation of that service starts from the previous conclusion.

When something else fits

• If your alarm names do not match your service.name values, the two sources will not share a timeline. Rename the alarms, or add a flat service field with an input transformer on the rule target, before you rely on this.
• CloudWatch alarm actions that publish to SNS do not fit. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed. Use the EventBridge route above.
• If you want a cheap, frequent check for services that have no alarm yet, see Watch, then escalate, which puts a triage agent in front of the root-cause agent.

Related