How does an AI SRE read Grafana Loki logs while investigating high latency?

Tares reads Grafana Loki logs with the loki connector, which polls a LogQL query and places the log lines on the same per-service timeline as your metrics and alerts. When a high latency alert fires, the agent reads the log lines from that same window. You label the Loki source with service, and the agent asks Tares for that service instead of querying Loki itself.

How it works

The loki connector is a polled connector. Tares pulls from your Loki instance on an interval, every 5 seconds by default, using a LogQL query you choose. Grafana Cloud works too.

• One query selects the lines. The connector takes a url and a query. For authentication it accepts bearer_token, or username and password, and it accepts tenant_id for multi-tenant setups. Tares holds that credential for the upstream.
• Filters trim the stream. The match and drop options and a key option are available on the source, so the timeline carries the lines you care about.
• The service label joins the signals. Every source that carries the same service label lands on one timeline for that service. Loki logs, Prometheus metrics and Prometheus alerts then sit on one clock, so a p99 spike, the HighLatency alert and the log lines around it appear side by side.
• The agent reads one correlated view. A Tares agent, or any MCP client you connect, reads the timeline for a service and gets metrics, logs and alerts merged in time order.
• A trigger wakes the agent. A trigger keyed by service fires on the alert, or on a threshold over a typed field such as max(p99_ms) > 1000 for latency. The woken agent starts from the correlated timeline and writes its finding back to it.

The AI SRE guide shows this whole loop with a demo system. It uses container logs there, and Loki takes the log slot in the same way.

Set it up

1. Start Tares and open the console.

   tares up

2. Go to Sources, then Add source, and choose the loki connector. Enter your Loki url and your LogQL query. Add bearer_token, or username and password, if your Loki needs them, and tenant_id if it is multi-tenant. Keep the default 5 second poll unless you want a different interval.

3. Add a label named service and mark it as the primary label. Use a constant value if the query selects one service, or read the value from a field when the query spans several services.

4. Give your other sources the same service label. Your Prometheus metrics source and your Prometheus alerts or Alertmanager source should all use the same service names as the Loki source, so they join the same timeline.

5. Create a trigger over those sources, keyed by service. Either react to fired alerts, as the guide does, or threshold the latency field directly.

6. Pick who gets woken: a Tares agent with a prompt written for your stack, your own agent service over a webhook, or a Slack channel.

7. Connect an MCP client in a second shell so you can investigate interactively.

   tares mcp

8. Ask your client what happened to the service in the last 15 minutes. It reads the correlated timeline, including the Loki lines, and answers from that evidence.

Open Explore and pick the service to check the result. The Loki lines should appear next to the metrics and alerts on one clock. The What the agent gets view shows the exact text an agent receives.

What you get

• Log lines from Loki sit in the same time window as the latency alert, so the agent does not stitch separate tools together.
• The agent investigates one service at a time, because every signal shares the service label.
• A trigger can wake an agent the moment latency alerts fire, so the first look is already on the timeline when you open it.
• The agent’s conclusion is written back to the service timeline. The next investigation, yours or an agent’s, starts from that finding instead of from scratch.
• Alerting stays where it is. Prometheus or Alertmanager keeps owning the thresholds, and Tares ingests what fired and correlates it with the logs.

To read the same evidence interactively, see Build an AI SRE for the MCP client steps.

When something else fits

• If your logs live in containers on the host, the docker_logs connector tails a running container without needing a Loki instance.
• If your logs already flow through an OpenTelemetry pipeline, the OTLP connector receives them directly.
• If you need a one-off query against Loki outside any incident, query Loki directly. Tares is the better fit when you want logs, metrics and alerts correlated per service for an agent.

Related