How does an AI SRE read Grafana Loki logs while investigating crash loops?

Tares reads Grafana Loki with its Loki connector, which polls the log lines your LogQL query selects every 5 seconds by default, Grafana Cloud included. Label the source by service and the lines join that service’s timeline in Tares. An agent investigating a crash loop then reads them next to the alert for the same service. A trigger-woken Tares agent and an MCP client both read that timeline.

How it works

• A Loki source selects the lines. You give the Loki connector a LogQL query, and Tares polls Loki for the matching log lines every 5 seconds by default. It works with a self-hosted Loki instance and with Grafana Cloud.
• Labels key the source to a service. Every source carries labels, and one of them is primary. Sources that share a service label are merged into one timeline per service, on one clock. The AI SRE guide builds this with metrics, logs and alerts all keyed by service.
• A trigger wakes the agent. In the guide, a trigger watches the sources for a service and fires when an alert event arrives. Prometheus keeps owning alerting, and Tares ingests what fired and correlates it with the evidence.
• The agent reads one correlated timeline. A Tares agent such as incident-first-look is woken by the trigger and reads the timeline. It ties the failure to specific evidence lines and writes a finding back onto the service’s timeline.
• An MCP client can pull the same evidence. The client calls read with a service name and answers from the correlated read, including earlier findings.

Set it up

1. Start Tares and open the console at http://127.0.0.1:8787.

   tares up

2. Go to Sources → Add source and choose the Loki connector. Set url to your Loki instance. Set query to the LogQL query that selects the service’s log lines, for example the lines of the workload that is crash looping.

3. Add authentication if your Loki needs it. The connector accepts bearer_token, or username and password, and tenant_id for a multi-tenant setup. It also has match, drop and key options.

4. Add a label named service and mark it primary. Use the same service label on your metrics and alert sources, so the log lines, the metrics and the alerts for one service land on one timeline.

5. Keep your alerting where it is. The AI SRE guide ingests alerts with the prometheus_alerts or alertmanager connector and uses a trigger over the sources, keyed by service, that fires when an alert event arrives.

6. Choose who investigates. A Tares agent works from a prompt you edit for your stack, with no infrastructure to run. Your own agent service can also be woken over a webhook.

7. To investigate on demand, start the MCP endpoint and connect your client.

   tares mcp

   Then ask a question such as “Use tares: what happened to api-server in the last 15 minutes?” The client calls read like this:

   {"service": "api-server"}

   Swap in the service that is crash looping.

What you get

• One read per service. The agent opens the timeline for the crash-looping service instead of querying each system separately. The timeline is merged on one clock from the sources that carry that service label.
• Log lines as evidence. Because Loki lines sit on the same timeline as the alert, the agent can tie its conclusion to specific lines.
• An investigation that starts on its own. When the trigger fires, the Tares agent wakes, reads the timeline and writes a finding. Nobody has to page it.
• Findings that carry forward. The finding is written onto the service’s timeline. The next investigation, including one from your MCP client, reads the evidence and the earlier finding and starts from the previous conclusion.
• Delivery where your team looks. You can send each firing and finding to a Slack channel, or have Tares POST the timeline to your own agent service.

When something else fits

• Your logs live in a container on the same host. The docker_logs connector tails a running container directly, and the guide’s demo uses it. Use it when you do not run Loki.
• Your logs already flow through OpenTelemetry. The OTLP connector receives logs, traces and metrics over OTLP, so you can send them straight to Tares.
• You want to investigate by hand in Grafana. Tares adds value when an agent needs the logs, metrics and alerts together. For a one-off look at a single log stream, Grafana remains a good place to read it.
• You want to see the full flow first. The AI SRE guide walks through a demo system on a laptop, with a trigger, an agent and a finding, before you point it at your own stack.

Related