Which OpenTelemetry log records does an AI SRE search when Datadog monitors fire?

Tares gives the AI SRE the correlated timeline of the service whose Datadog monitor fired, with that service’s OpenTelemetry log records next to the monitor event. The Datadog event reaches Tares through the webhook connector, and the log records arrive through the otlp connector. Both sources carry the same service label, so one read for that service returns both signals on one clock.

How it works

Tares has no connector of its own for Datadog today. Datadog monitors reach Tares through the webhook connector and a Datadog-side setup. OpenTelemetry logs arrive through the otlp connector.

The two sources meet on one label:

• The otlp source turns the resource attribute service.name into the service label. One source ingests every service, so there is no per-service setup.
• The Datadog webhook source declares a service label over a field in the JSON payload you write, and marks it primary.
• Tares merges sources that carry the same label into one timeline per service. In the AI SRE guide, metrics, logs and alerts for one service appear merged on one clock.

A trigger keyed by service fires when the monitor event arrives. The agent it wakes reads that service’s timeline. A connected client does the same with read and a service, as in the question “what happened to api-server in the last 15 minutes?”. The window comes from your question or the agent’s prompt, so you choose how many minutes to look at.

The webhook source can also read an ISO-8601 timestamp from the payload through event_time_field. The Datadog event then lands on the timeline at its own time, and you can see where the first event sits among the log records.

Set it up

1. Create the otlp source. It needs no config beyond labels, and service is the natural key.

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: service, primary: true }

2. Point your exporter or OpenTelemetry Collector at Tares.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

3. Create a webhook source for Datadog with a service label. Keep the payload flat, because event_type_field and text_template read top-level fields only. Copy the ingest endpoint from the source page.

- name: datadog-monitors
  connector: webhook
  config:
    labels:
      - { name: service, field: service, primary: true }

4. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the payload as JSON from Datadog variables, and include a service field that holds the alerting service’s name, matching the service.name your OpenTelemetry logs carry.
5. When Tares has auth on, add custom headers as JSON in Datadog with an Authorization Bearer header holding an ingest-scoped API key. Exporters on a secured instance send the same kind of key.
6. Put @webhook- followed by your webhook name in the monitor message so the monitor calls the webhook.
7. Add a trigger keyed by service over the Datadog source, so a monitor event wakes the agent for that service.

triggers:
  - name: escalate
    sources: [datadog-monitors]
    key_field: service
    condition: {aggregate: count, predicate: "> 0", window: 5m}
    cooldown: 30m

8. Check Recent events on both sources, and open Explore to see the service’s timeline with the monitor event and the log records together.

What you get

• One timeline per service, with the Datadog monitor event and the OpenTelemetry log records for that service on the same clock.
• An agent that is woken per service, so the investigation starts from the service that alerted.
• A time window you control through the question or the agent prompt.
• Findings written back to the same service timeline, so the next investigation starts from the previous conclusion.
• Log records from every instrumented service in one otlp source, with no per-service source to maintain.

When something else fits

• If you want a cheap first level that checks your sources on a schedule and escalates only when something moves, see Watch, then escalate.
• If Datadog payloads cannot carry the service name, add the field in the webhook payload first. Without a shared service label the monitor event and the log records do not land on the same timeline.

Related