How do I pair alerts from Prometheus with Grafana Loki logs for an AI SRE?

In Tares, you pair Prometheus alerts with Grafana Loki logs by giving both sources the same service label, so one timeline holds the alert and its logs. Add a prometheus_alerts source and a loki source, and label each by service. A trigger wakes your agent when an alert fires, and the agent reads the alert and the log lines together from one per-service timeline.

How it works

Tares correlates by label. Sources that carry the same service label merge into one timeline per service, with every event on one clock. You use two sources:

• prometheus_alerts polls Prometheus’s own alerts, every 30 seconds by default. Each active alert becomes one event with event_type set to the alert name, text set to its summary, its labels as Tares labels, and a synthetic state of firing, pending or resolved. An alert fires once, re-emits only on a state change, and emits a resolved event when it clears.
• loki reads log lines from a Loki instance, including Grafana Cloud. You select them with a LogQL query, and Tares polls every 5 seconds by default.

Prometheus keeps owning alerting. Tares ingests what fired, correlates it with the logs, and handles the waking. Nothing is re-thresholded. For the full pattern, see the Build an AI SRE guide.

Set it up

1. Install Tares and start it.

   uv tool install tares          # or: brew install glassflow/tap/tares
   tares up

2. Add the alerts source. In the console, go to Sources, Add source, Prometheus alerts, enter the url plus any auth, and click Discover. It lists your alerting rules and the severities present. The equivalent source looks like this:

   - name: alerts
     connector: prometheus_alerts
     poll: 30s
     config:
       url: http://localhost:9090
       severities: critical,warning   # blank = all
       labels:
         - { name: severity, field: labels.severity, primary: true }
         - { name: state, field: state }

3. Add the service label to the alerts source. Discover only proposes the labels every alert is guaranteed to carry, severity and state. An alert’s own labels, such as service or namespace, exist only while the alert is firing. Once real alerts have been ingested, open the source’s Fields tab and promote service to a label. Make it the primary label. Check Recent events to confirm alerts are arriving. The Prometheus alerts page covers each field.

4. Add the Loki source under Sources, Add source. Set the url, write a query in LogQL that selects the service’s logs, and add bearer_token, or username and password, or tenant_id if your Loki needs them. Give it a service label, marked primary, with the same values your alerts use, so api-server in Loki matches api-server in the alert.

5. Create a trigger over both sources, keyed by service, that fires when an alert event arrives. The AI SRE guide uses sum(alert_active) > 0 over 1m, per service.

6. Pick who gets woken: a Tares agent, your own agent service over a webhook, or a Slack channel.

7. Open Explore, pick a service, and confirm the alert and the log lines appear together. To investigate interactively, run tares mcp and ask your connected client what happened to that service. It reads the correlated timeline for the service.

What you get

• One timeline per service, with the firing alert and the Loki log lines merged on one clock.
• A single read that gives your agent the alert, its state and the surrounding logs, so the investigation starts from evidence instead of from queries against each system.
• Alerts that fire once per state change, plus a resolved event when they clear, so the timeline is not flooded with repeats.
• A trigger that wakes your agent when an alert lands, so nobody has to page it.
• Findings that your agent writes back onto the same service timeline, so the next investigation starts from the previous conclusion.
• The same data available to an MCP client for interactive questions.

When something else fits

• If you route alerts through Alertmanager and want it to push them to Tares, use the alertmanager connector instead of prometheus_alerts. The service label works the same way.
• If your logs come from a container on the same host, the docker_logs connector tails them directly. If you already run OpenTelemetry, OTLP carries logs, traces and metrics. You still key them by service, and the Build an AI SRE guide shows the options side by side.
• If you also need metrics on the timeline, add the prometheus connector with the same service label.

Related