Which Grafana Loki logs does an AI SRE search when alerts from Prometheus fire?

Tares gives an AI SRE the Grafana Loki logs that carry the same service label as the Prometheus alert that fired, on one timeline with the alert. The agent reads that service’s recent window, not every log you ingest. You get a first look built from the alerting service’s own evidence.

How it works

Tares correlates by label. When the Prometheus alerts source and the Loki source both carry a service label, their events land on one timeline per service, merged on one clock.

• The alert is the anchor. The prometheus_alerts connector polls Prometheus every 30 seconds by default. It emits one event per active alert, keyed by a label, and tracks state across polls. Each alert fires once and emits a resolved event when it clears. The firing event is the first event on the timeline.
• Loki supplies the log lines. The loki connector reads log lines from a Loki instance, including Grafana Cloud. A LogQL query selects the lines, and Tares polls every 5 seconds by default.
• A trigger wakes the agent per service. A trigger watches the alerts source keyed by service. When an alert event arrives, it wakes the agent for that service only. See Build an AI SRE for the full wiring.
• The agent reads one service, not everything. The agent asks for the correlated read of that service, for example the last 15 minutes of api-server. It gets the alert, the metrics and the Loki lines that share the service label, ordered in time. Logs from other services stay out of the read.

An alert’s dynamic labels, such as namespace or service, exist only while it is firing. You add them from the Fields tab once real alerts have been ingested.

Set it up

1. Add the alerts source. In the console, go to Sources → Add source → Prometheus alerts, enter the url and use Discover. The source looks like this:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

2. Let some alerts ingest, then open the source’s Fields tab and promote service (or the label your alert rules use for the service name) so it becomes a label. Details are on the Prometheus alerts page.
3. Add the Loki source under Sources → Add source. Enter the Loki url and a LogQL query that selects the logs you want. Add bearer_token, username, password or tenant_id if your Loki needs them. Give it a service label, read from a field of the log line or set as a constant, so its lines join the alert’s timeline.
4. Keep the label name identical on every source. The same service value on the alert and on the Loki lines is what joins them.
5. Add a trigger over the alerts source, keyed by service, and subscribe an agent to it. The guide on building an AI SRE walks through this with a demo stack.
6. Connect an MCP client with tares mcp and ask what happened to the service. The client calls read with the service:

{"service": "api-server"}

7. Open the source and check Recent events and Fields to confirm the alert and the log lines carry the same service value.

What you get

• A first look that starts from the alerting service. The agent sees that service’s alert, metrics and Loki lines together, not a search across all your logs.
• A short, readable window around the first event, so the agent ties the failure to specific evidence lines.
• One firing per alert. State changes are tracked, so a long-running alert does not wake the agent on every poll, and a resolved event lands on the timeline when it clears.
• A finding written back on the service’s timeline. The next investigation, by an agent or by you through MCP, starts from that conclusion.
• Prometheus keeps owning alerting. Tares ingests what fired and correlates it, and nothing is re-thresholded.

When something else fits

• If you want a check before any alert fires, use a schedule trigger with a cheap triage agent that hands off to a root-cause agent. Watch, then escalate shows how.
• If your alerts reach you through Alertmanager rather than Prometheus’s own alert list, use the alertmanager connector for the alert side and keep the same service label on the Loki source.
• If your logs are not in Loki, such as a single container’s output or an OpenTelemetry pipeline, use the docker_logs or OTLP connector for the log side. The correlation by service works the same way.

Related