How do I line up CloudWatch alarms with OpenTelemetry metric series on one timeline?

Tares puts CloudWatch alarms and OpenTelemetry metric series on one timeline when both sources carry the same primary service label. The alarms arrive through a webhook source fed by an EventBridge API destination, and the metrics arrive through an OTLP source. Open Explore, pick the service, and you see the alarm and the metric points that led to it on one clock.

How it works

Tares correlates events by label. Two sources that share a primary service label merge into one per-service timeline, as described in Build an AI SRE.

• Metrics: the OTLP connector receives OTLP/HTTP metrics at /v1/metrics. Resource attributes become labels, so service.name becomes the service label with no per-service setup.
• Alarms: Tares has no connector of its own for CloudWatch today. Alarm state changes reach Tares through the webhook connector and an EventBridge setup on the AWS side. EventBridge posts the alarm event as a JSON body, and the webhook connector parses it whatever the Content-Type. The payload is stored lossless.
• The shared key: the CloudWatch payload has no top-level service field. Give the webhook source a service label with a constant value that matches the service.name your exporter sends. Both sources then land under the same service.
• Field depth: source and detail-type are top-level payload fields. detail.alarmName works as a one-level dotted label. detail.state.value is two levels deep, so flatten it with an input transformer on the rule target if you want the alarm state as a label or in text_template.

CloudWatch alarm actions that publish to SNS do not fit this route. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed.

Set it up

1. Create the OTLP source. In Sources, Add source, otlp, add a service label as primary and save. No other config is needed.

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: service, primary: true }

2. Point your exporter at Tares. Use the OTLP/HTTP endpoint, or the gRPC endpoint if you installed the otlp-grpc extra.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

3. Create the webhook source for alarms. Add a primary service label with a constant value equal to your service.name. Add an alarm label from detail.alarmName. Read the event type from detail-type, which is a top-level field. Save, then copy the ingest URL from the source page.

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: service, const: api-server, primary: true }
      - { name: alarm, field: detail.alarmName }

4. Create the EventBridge API destination. Set its endpoint to the Tares ingest URL, which must be public HTTPS, and its method to POST.
5. Add the connection. EventBridge requires one on every API destination. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key value only matters when Tares has auth on.
6. Create the rule. On the default event bus, match source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as the target.
7. Check the result. Recent events on each source shows what arrives, and Fields shows which payload fields come in. In Explore, pick your service under the service key.

What you get

• One timeline per service with CloudWatch alarm events and OpenTelemetry metric points merged on the same clock.
• The alarm shows what fired, and the metric points just before it show how the service got there.
• The same correlated read for you in the console and for any connected agent, so an investigation starts from alarm and metric together.
• Room to add more signals on the same service label later. For example, the Prometheus connector turns the metrics you pick into events that join on the same key.

When something else fits

• If your alarms already flow through Alertmanager or Prometheus rules, the alertmanager or prometheus_alerts connectors fit better than an EventBridge route.
• If you only need the metric series, the OTLP source alone is enough, and you can skip the webhook setup.
• If your only route out of CloudWatch is SNS, this setup does not work. Use an EventBridge rule instead.

Related