How does an AI SRE read Grafana Loki logs while investigating error spikes?

Tares reads Grafana Loki logs with its Loki connector, which polls the log lines your LogQL query selects. Label the source by service and its lines join that service’s metrics and alerts on one timeline. When an error spike alert fires, your agent reads the log lines from the same window as the alert.

How it works

• Loki is a polled source. The Loki connector runs your LogQL query against a Loki instance, including Grafana Cloud, every 5 seconds by default, and stores the log lines it returns as events.
• The service label ties logs to the alert. Every source in an AI SRE setup carries the same service label. Tares builds one timeline per service from all sources that carry it, so Loki lines, metrics and alerts sit on one clock.
• The alert wakes the agent. Prometheus or Alertmanager keeps owning alerting. When an alert event such as HighErrorRate arrives, a trigger fires and wakes your agent. Tares does not re-threshold anything.
• The agent reads one correlated view. The agent, or an MCP client you connect, reads the service’s timeline and sees the error log lines next to the alert and the metric that tripped it.
• The conclusion is written back. The agent records a finding on the same timeline, so the next investigation starts from it.

The Build an AI SRE guide walks through this chain end to end with a demo system.

Set it up

1. Install and start Tares.

   uv tool install tares          # or: brew install glassflow/tap/tares
   tares up

2. Add the Loki source. In the console at http://127.0.0.1:8787, go to Sources → Add source and choose the Loki connector. Set url to your Loki instance and query to the LogQL selector for the logs you want, for example the error lines of the services you watch.

3. Add credentials if Loki needs them. The connector accepts bearer_token, or username and password, and a tenant_id for multi-tenant Loki. Use these for Grafana Cloud as well.

4. Narrow the stream if needed. The match and drop keys let you filter which lines come in, so an error spike is not buried in noise.

5. Label the source by service. Give the source a primary service label. Use the same label name on your metrics and alert sources, because that shared label is what merges them into one timeline:

   labels:
     - { name: service, const: api-server, primary: true }

   Use const when one query covers one service. Use field to read the service from a field on each line when one query covers several.

6. Keep alerts on the same label. Add an alert source, either prometheus_alerts or alertmanager, keyed by service, and create a trigger over your sources that fires when an alert event arrives.

7. Choose who gets woken. Pick a Tares agent, your own agent service over a webhook, or a Slack channel for each trigger.

8. Connect an MCP client to investigate on demand.

   tares mcp        # MCP endpoint on http://127.0.0.1:8788/mcp

   Then ask your client: “Use tares: what happened to api-server in the last 15 minutes?” It answers from the correlated read, with the Loki lines in it.

What you get

• Log lines from Loki on the same timeline as the alert and the metric behind it, so an error spike shows its error messages next to the moment it started.
• An agent that is woken by the alert and starts with the correlated evidence, with no per-system queries to write first.
• A finding written back onto the service timeline. The next investigation, yours or an agent’s, reads the evidence and that finding together.
• One service label across metrics, logs and alerts, so adding another service means adding another label value, not another view.
• Loki stays your log store. Tares reads from it and does not replace it.

When something else fits

• You want Tares to receive logs rather than poll Loki. If your logs already flow through an OpenTelemetry pipeline, the OTLP connector receives logs, traces and metrics directly. If you run containers locally, the docker_logs connector tails a container’s logs.
• You have no alerting yet. The trigger in this setup reacts to fired alerts. Add a source such as prometheus_alerts or alertmanager, or write triggers over the typed fields of your sources.
• You need deploys in the same window. Add a webhook source with a service label and post one event per deploy from your CD pipeline, so the agent sees the change next to the error spike.

The Build an AI SRE guide covers these choices step by step.

Related