How do I line up Grafana alert rules with OpenTelemetry metric series on one timeline?

Tares puts Grafana alert notifications and OpenTelemetry metric points on one clock by giving both sources the same primary service label. Grafana alerts arrive through the webhook connector, and OpenTelemetry metrics arrive through the OTLP connector. Open Explore for that service and you see what fired next to the metric movement that led to it.

How it works

Tares builds one timeline per key. Every source that carries the same primary label, here service, merges into the same timeline for that value, so an alert and a metric for one service sit side by side.

• Metrics side. The OpenTelemetry connector receives OTLP/HTTP logs, traces and metrics. One source covers every service, because resource attributes such as service.name become labels. The service label comes from service.name.
• Alert side. Grafana Alerting reaches Tares through the webhook connector and a vendor setup. Tares has no connector of its own for Grafana Alerting today. The notification arrives as one event, with the alerts list kept inside the lossless payload. Top-level fields such as status, receiver and title work for labels and for text_template.
• The shared key. The OTLP source gets service from service.name. The webhook source gets it from a constant label, so each Grafana contact point feeds a webhook source that names one service. Both sources then share the key.

If your metrics sit in a Prometheus server instead, the Prometheus connector polls the series you pick and joins them on the same kind of key. The timeline works the same way.

Set it up

1. Start Tares and open the console.

   tares up

2. Create the OTLP source under Sources, Add source, otlp. It has no required config, and service is the natural key.

   - name: otel
     connector: otlp
     config:
       labels:
         - { name: service, field: service, primary: true }

3. Point your OpenTelemetry exporter at Tares.

   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
   # or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

4. Create a webhook source for the Grafana alerts of one service. Use a constant service label with the same value your OTLP metrics carry in service.name, and mark it primary. Build the text from top-level payload fields.

   - name: grafana-alerts
     connector: webhook
     config:
       text_template: '{title}: {status}'
       labels:
         - { name: service, const: api-server, primary: true }
         - { name: status, field: status }

5. Copy the ingest URL from the source page. In Grafana, add a Webhook contact point with that URL and the POST method.

6. Route the alert rules for that service to the contact point from a notification policy. Repeat steps 4 to 6 for each service you want to line up.

7. When Tares has auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. For Grafana, set the Authorization header scheme to Bearer and the credentials to that key. For the OpenTelemetry exporter, send the same header. Without auth on, you need no key.

8. Check the result. Recent events on each source shows what arrives, and Explore lists the service under the service key.

Alertmanager-compatible Grafana payloads can also go to a separate alertmanager source, which makes one event per alert and has its own ingest URL. Use it if you want each alert as its own event instead of one event per notification.

What you get

• One timeline for a service, with the Grafana alert and the OpenTelemetry metric points merged on one clock in Explore.
• The alert tells you what fired, and the metric points around it show how the service got there.
• The full Grafana payload stored lossless, including the alerts list, labels and annotations.
• Metrics, traces and logs from OTLP on the same key, so one source covers every service you instrument.
• The same timeline available to an agent, which is what the AI SRE guide builds on: a trigger wakes an agent with one correlated read of metrics and alerts for the service.

When something else fits

• If your alerts come from Prometheus rules or Alertmanager rather than Grafana, use the Prometheus connector for the series and the matching alert connector, so you skip the webhook setup.
• If you want one event per firing alert instead of one per notification, point Grafana’s Alertmanager-compatible payload at an alertmanager source.

Related