Which OpenTelemetry log records does an AI SRE search when Grafana alert rules fire?

Tares gives an AI SRE the OpenTelemetry log records of the service the Grafana alert points at, read from that service’s timeline instead of a search across every log you ingest. Logs, metrics and alerts for one service share one clock, so the agent looks at that service around the first alert event. Grafana reaches Tares through the webhook connector and a vendor setup, and OpenTelemetry logs arrive through the otlp connector.

How it works

• One source for all services. The OpenTelemetry connector receives OTLP/HTTP logs, traces and metrics. Resource attributes such as service.name become labels, so every service your SDKs or Collector export correlates by service with no per-service setup.
• One timeline per service. When the OTLP source and your alert source carry the same service label, Tares merges logs, metrics and alerts for that service on one clock. That timeline is what the agent reads, not the whole log stream.
• Grafana arrives as one event. Grafana Alerting reaches Tares through the webhook connector and a vendor setup. Tares has no connector of its own for Grafana Alerting today. The notification lands as one event with the alerts list inside the lossless payload. Top-level fields such as status, receiver and title work for labels and text_template.
• A trigger wakes the agent. A trigger over your sources, keyed by service, fires when an alert event arrives and wakes an agent with the correlated timeline. The agent can also read on demand. Its read call takes the service, and you set the span, for example the last 15 minutes.
• The window is yours to set. You choose how far back the agent reads. Keep it to the minutes around the first alert event, and the log records it sees are the alerting service’s, in that stretch only.

Set it up

1. Create an OTLP source keyed by service. The service label reads from the resource attribute service.name. A label with field: service gets no value from an OTLP event, so use the full path:

   - name: otel
     connector: otlp
     config:
       labels:
         - { name: service, field: resourceAttributes.service.name, primary: true }

2. Point your exporter at Tares:

   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
   # or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

   When Tares has auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key.

3. Create a webhook source for Grafana and copy its ingest URL. Give it a label for the field that carries the service, and mark one label primary. Use top-level payload fields for labels. If the service name sits only inside the alerts list, an alertmanager source can take the same Grafana payload instead. It makes one event per alert and has its own ingest URL.

4. In Grafana, add a Webhook contact point with the Tares ingest URL and the POST method. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key.

5. In Grafana, route alerts to that contact point from a notification policy.

6. Build a trigger over the Grafana and OTLP sources, keyed by service, as described in Build an AI SRE. Choose who gets woken: a Tares agent, your own agent service over a webhook, or a Slack channel.

7. Check the result. Open Recent events on each source, then Explore and pick the service. Or ask a connected MCP client:

   Use tares: what happened to checkout in the last 15 minutes?

   The client reads the correlated timeline for that service, including its OpenTelemetry log lines.

What you get

• When a Grafana rule fires, the agent starts from the alerting service’s own log records, not from every service’s logs.
• Log lines, metrics and the alert sit on one clock, so the agent sees what the service logged right before and after the first event.
• You decide the window, so a narrow read keeps the evidence small and readable.
• Every service you export over OTLP is correlated by the same service label, so a new service needs no extra source.
• The agent’s conclusion is written back to the same service’s timeline, and the next investigation starts from it.

When something else fits

• If your Grafana rules do not cover a failure, a triage agent that checks your sources every few minutes can catch it. See Watch, then escalate.
• If you want one event per alert instead of one per notification, send the Grafana payload to an alertmanager source.

Related