How do I pair Grafana alert rules with OpenTelemetry log records for an AI SRE?

Tares pairs Grafana alerts with OpenTelemetry logs by keying both sources on the same service label. A firing alert then lands on the same per-service timeline as the log lines from that window. Grafana reaches Tares through the webhook connector and a Grafana contact point, and the logs arrive through the OTLP connector.

How it works

Tares builds one timeline per service from sources that carry the same primary label.

• Logs: the OpenTelemetry connector takes OTLP logs, traces and metrics. One source ingests every service, and the service.name resource attribute becomes the service label through the path resourceAttributes.service.name.
• Alerts: Tares has no connector of its own for Grafana Alerting today. Grafana reaches Tares through the webhook connector and a Grafana webhook contact point. The notification arrives as one event, with the alerts list inside the lossless payload. Top-level fields such as status, receiver and title work for text templates.
• The shared key: you give the Grafana webhook source a service label with a constant value that matches the service’s service.name. Alerts and log lines then share a key, and a read of that service returns both on one clock.

This is the correlation described in Build an AI SRE: sources keyed by the same label, a trigger that wakes on the alert, and an agent that reads the merged timeline.

Set it up

1. Create the OTLP source. Add a source of type otlp with a service label read from the resource attribute. Use the path resourceAttributes.service.name, because a label with field: service gets no value from an OTLP event.

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: resourceAttributes.service.name, primary: true }

2. Point the exporter at Tares. Set the OTLP/HTTP endpoint on your app or OpenTelemetry Collector.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

3. Create a webhook source for the service’s alerts. Give it a primary service label with a constant value equal to that service’s service.name, for example checkout. Copy the ingest endpoint from the source page. For a second service, create a second webhook source with its own constant.

- name: grafana-checkout
  connector: webhook
  config:
    event_type: grafana_alert
    text_template: '{title} ({status})'
    labels:
      - { name: service, const: checkout, primary: true }
      - { name: status, field: status }

4. Add the Grafana contact point. In Grafana, add a Webhook contact point with the Tares ingest URL and the POST method. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key.

5. Route the alerts. Use a notification policy to send that service’s alert rules to the contact point.

6. Add a trigger over both sources. The trigger wakes your agent when an alert event arrives.

- name: checkout-incident
  sources: [grafana-checkout, otel]
  filters:
    - { field: event_type, op: eq, value: grafana_alert }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 5m

7. Verify. Open Recent events on each source, then open Explore and pick the service. The alert and the log lines appear together.

What you get

• One timeline per service with Grafana alerts and OpenTelemetry log lines on the same clock.
• A firing alert that arrives next to the logs from the same window, so an agent reads the evidence in a single call.
• Grafana keeps owning alert rules. Tares ingests what fired, correlates it and handles the waking.
• The full Grafana notification stored lossless, including the alerts list with its labels and annotations.
• The same service key for traces and metrics from the OTLP source, if you export them.

When something else fits

• If you want one event per alert instead of one event per notification, use a separate alertmanager source with its own ingest URL. Grafana’s default payload is Alertmanager compatible, so it can take the same payload.
• If your alerts come from Prometheus rules rather than Grafana, the guide Build an AI SRE covers the Prometheus alerts and Alertmanager connectors.

Related