What should I look for in Grafana Loki logs when the problem is high latency?

Tares polls your Loki logs and puts them on one service timeline, so you look for changes in slow-request, timeout and error lines instead of reading raw text. Give the loki source the same service label as your metrics and alerts, and an agent reads logs, the latency metric and the alert together. Ask it how line counts changed across the window, then use the lines themselves as evidence.

How it works

The loki connector reads log lines from a Loki instance, Grafana Cloud included. You select the lines with a LogQL query, and Tares polls them every 5 seconds by default. The connector accepts these keys:

• url and query, for the instance and the lines you select.
• bearer_token, username and password, for the credentials Tares holds for Loki.
• tenant_id, for a multi-tenant Loki.
• match and drop, to keep or discard lines.
• key, to set the entity key.

Tares merges every source that carries the same service label into one timeline per service. A Loki source, a Prometheus source and an alert source for api-server end up on one clock. This is the read an agent works from, as described in Use AI to monitor your systems.

For a latency problem, these are the lines worth looking for:

• Slow request lines. Lines that carry a duration, or wording about a request taking long.
• Timeout and retry lines. Lines that show a caller giving up or trying again.
• Database connection lines. Pool exhaustion or waiting for a connection often shows up as latency.
• Dependency lines. Errors from a downstream service you call.
• Deploy or config lines. A change shortly before the slowdown is the most common root cause.

A window of raw lines is long and repetitive. What helps an agent is the change: how many slow-request or timeout lines there were before the window and how many there are now. Ask for counts per line type, then ask it to quote a few lines as evidence.

Set it up

1. Write a LogQL query that selects the logs of the service you are worried about. Keep it to one service, so the counts mean something.
2. Add a loki source in the console under Sources, Add source. Fill in url and query, add credentials such as bearer_token, or username and password, if your Loki needs them, and set tenant_id if you use one. Use match and drop to keep latency-relevant lines and discard noise.
3. Give the source a primary service label with the same value your other sources use. That label is what puts the logs on the same timeline as the metrics.
4. Start Tares and open the console.

tares up

5. Add the signals that tell you whether the logs match reality: a Prometheus metrics source for latency, and a Prometheus alerts or Alertmanager source for the alert that fired. Add a deploy source too, because a change is often the cause. Build an AI SRE walks through all of these step by step with a demo.
6. Optionally add a trigger over those sources keyed by service. For latency you can threshold a typed field directly, for example max(p99_ms) > 1000, or keep reacting to the alert your alerting already fires. Triggers also support count aggregates over a window.
7. Connect your MCP client with tares mcp, then ask a question that points the agent at counts.

tares mcp

For example: “Use tares: what happened to api-server in the last 15 minutes? Compare slow-request and timeout line counts before and during the window, and quote two lines as evidence.”

What you get

• Loki lines, the latency metric, the alert and any deploy on one timeline per service, so you do not switch between a log view and a dashboard.
• A diagnosis that states what changed in the logs, such as more timeout lines after a deploy, with example lines behind it.
• A finding written back to the service timeline, so the next investigation, yours or another agent’s, starts from that conclusion instead of from scratch.
• Alerting that stays where it is. Prometheus or Alertmanager keeps owning the alert, and Tares ingests what fired and handles the waking.
• Trigger-driven wake-ups. When the condition holds, a Tares agent or your own agent service starts with the correlated timeline already in front of it.

When something else fits

• If you only need to run an ad hoc LogQL query once, your existing Loki tooling is quicker. Tares is for correlating logs with metrics, alerts and deploys, and for waking an agent.
• If the container logs are not in Loki, the Docker logs, OTLP and Vercel connectors cover other log paths, and the same service label still joins them to the timeline. Use AI to monitor your systems lists them.

Related