How does an AI SRE read Grafana Loki logs while investigating 5xx errors on an API?

Tares reads your Grafana Loki logs with its loki connector and puts them on the same per-service timeline as your metrics and alerts. When a 5xx alert fires, the agent reads the log lines from that window in one correlated read. You select the API’s lines with a LogQL query, label the source by service, and the agent investigates from evidence instead of from a blank page.

How it works

The loki connector polls a Loki instance, including Grafana Cloud, and runs the LogQL query you give it. It polls every 5 seconds by default. Each source you add carries a primary service label. Sources that share that label merge into one timeline per service, on one clock.

For a 5xx investigation the pieces fit together like this:

• Logs: a loki source selects the API’s log lines with your LogQL query.
• Metrics and alerts: your metrics source and your alerts source carry the same service label. Prometheus or Alertmanager keeps owning alerting, and Tares ingests what fires.
• Trigger: a trigger watches the sources keyed by service and fires when an alert event arrives for that service.
• Agent: the Tares agent that the trigger wakes reads the correlated timeline. The error-rate alert and the error log lines sit side by side in the same window. The agent writes a finding back onto the service’s timeline, so the next investigation starts from that conclusion.

You can also pull instead of waiting for a push. An MCP client calls read for a service and answers from the same correlated timeline. The AI SRE guide walks through the full loop with a demo system.

Set it up

1. Install Tares and start it, then start the MCP endpoint in a second shell so your client can read the timeline.

tares up
tares mcp

2. In the console, go to Sources → Add source and choose the loki connector. Set url to your Loki instance and query to a LogQL selector for the API’s logs, for example the stream that carries its request logs.
3. Add credentials for your Loki if it needs them: bearer_token, or username and password. Set tenant_id if you use one. For Grafana Cloud, use the same keys with your Grafana Cloud Loki URL.
4. Narrow the lines to what matters for 5xx work. Use match to keep lines and drop to discard noise such as health checks. The default poll interval is 5 seconds.
5. Give the source a primary service label, with the same service name your metrics and alerts sources use. Matching labels are what put the logs on the same timeline as the alert. If you have no alerts or metrics source yet, add one with the prometheus, prometheus_alerts or alertmanager connector and key it by service too.
6. Create a trigger over those sources, keyed by service, that fires when an alert event arrives. Subscribe a Tares agent to it, or send the firing to your own agent over a webhook.
7. Connect your MCP client and ask about the service. The client calls read with the service name:

{"service": "api-server"}

In Explore, pick the service to see logs, metrics and alerts merged on one clock. What the agent gets shows the exact text an agent receives.

What you get

• Logs in the alert window. When the 5xx alert fires, the error log lines from Loki are already on the timeline next to it, so the agent ties the failure to specific evidence lines.
• One read instead of several. The agent does not query Loki, Prometheus and your alerting separately. It reads one correlated timeline per service.
• A finding that persists. The agent’s conclusion is written back onto the service’s timeline, and the next investigation, human or agent, starts from it.
• Both ways to investigate. A trigger wakes an agent automatically, and an MCP client can pull the same evidence on demand.
• Control over the noise. match and drop keep the timeline to the lines that help.

When something else fits

• If your API logs run as containers on a host you control, the docker_logs connector tails a container’s logs directly. If your logs already flow through OpenTelemetry, the OTLP connector receives them.
• If you only need to be paged on the alert itself, keep that in your existing alerting. Add Loki when you want the agent to read the lines behind the alert.

Related