How do I pair CloudWatch alarms with OpenTelemetry log records for an AI SRE?

Tares pairs the two by giving both sources the same service label, so a firing alarm lands on one per-service timeline next to the OpenTelemetry log lines from the same window. OTLP logs arrive through the otlp connector. CloudWatch alarms arrive through the webhook connector, fed by an EventBridge API destination. An agent or trigger then reads one correlated timeline per service.

How it works

Tares correlates events by label. Every source that carries the same primary label, here service, writes onto the same timeline for that value, and the timeline merges the events on one clock.

• Logs: the OpenTelemetry connector receives OTLP/HTTP logs, traces and metrics. One source covers every service. The service label reads resourceAttributes.service.name, which your SDK or Collector already sets.
• Alarms: Tares has no connector of its own for CloudWatch today. The alarm reaches Tares through the webhook connector and an EventBridge setup on the AWS side. EventBridge posts the alarm state change as a JSON body, and the webhook connector parses it whatever the Content-Type.
• Matching: the alarm event carries source and detail-type as top-level fields, and detail.alarmName as a one-level dotted field. A label can read detail.alarmName, but detail.state.value is two levels deep. If you want it as a label or in a text template, flatten it with an input transformer on the rule target.

The service label on the alarm source reads detail.alarmName. The two sources therefore join on one timeline only when the alarm name is the same string as the service.name your telemetry reports. Name the alarm after the service, for example checkout, if you want that match.

Alarm actions that publish to SNS do not fit. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed. Use EventBridge.

Set it up

1. Create the OTLP source, keyed by service. This is the YAML the OTLP page shows:

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: resourceAttributes.service.name, primary: true }

2. Point your exporter at Tares. The sender uses a Bearer Authorization header holding an ingest-scoped API key, needed only when Tares has auth on.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

3. Create a webhook source for the alarms. Give it a service label marked primary, which makes it the same key as the OTLP source. Save it and copy the ingest URL from the source page.

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: service, field: detail.alarmName, primary: true }
      - { name: source, field: source }

4. In EventBridge, create an API destination whose endpoint is the Tares ingest URL and whose method is POST. The endpoint must be public HTTPS.
5. Give the API destination a connection, which EventBridge requires on every API destination. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key value only matters when Tares has auth on.
6. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Set the API destination as its target. Add an input transformer on the target if you want nested fields such as the alarm state as labels.
7. Check the result. Open Recent events and Fields on each source page to see what arrives. Then open Explore, pick the service, and confirm that alarm events and log lines sit on the same timeline.
8. Wake an agent from the pair. The AI SRE guide shows how to build a trigger over sources keyed by service, subscribe an agent to it, and connect an MCP client to read the same evidence yourself.

What you get

• One timeline per service, with the alarm state change and the OTLP log lines from the same window merged on one clock.
• An agent or trigger that reads the correlated evidence instead of querying CloudWatch and a log store separately.
• The alarm payload stored lossless, including the full detail object, even for fields you do not map to labels.
• Alerting stays where it is. CloudWatch keeps owning the alarm thresholds, and Tares ingests what fired and correlates it with your logs.
• A shared pattern. Any other source you add with the same primary service label joins the same timeline.

When something else fits

• If your alarm names do not match service.name and you cannot rename them, the label join will not line up. Flatten the fields you need with the input transformer first, or key both sources by a field the two payloads share.
• If your alerts come from Prometheus or Alertmanager rather than CloudWatch, use the prometheus_alerts or alertmanager connector instead of the webhook route.

Related