How do I label Grafana Loki logs so an AI SRE can narrow high latency to one service?

Tares lets you give your Loki log lines a service label, taken from a field or set as a constant and marked primary, so a read on that service returns the logs next to its metrics and alerts. For high latency, the agent asks for one service and sees that service’s p99, its latency alert and its log lines on one timeline. The label lives on the Loki source, so you set it once and every later read and trigger uses it.

How it works

• The loki connector polls your Loki instance, Grafana Cloud included, with a LogQL query. It polls every 5 seconds by default.
• Every source in Tares declares labels. A label has a name and gets its value either from a field of the event or from a fixed const. One label is marked primary.
• Sources that carry the same primary label key are merged into one timeline per service. Metrics, logs and alerts sit on one clock, as described in Build an AI SRE.
• A read for one service returns that merged timeline. During a latency incident the agent does not search logs across your whole estate. It reads the one service and sees the p99 metric, the latency alert and the log lines together.
• Use a const when the LogQL query selects the logs of a single service. Use a field when the log lines themselves carry the service name.
• The Loki source also accepts url, query, bearer_token, username, password and tenant_id. Use them to reach your Loki or Grafana Cloud with the credentials it expects.

Set it up

1. Install Tares and start it locally.

uv tool install tares          # or: brew install glassflow/tap/tares
tares up

2. Open the console at http://127.0.0.1:8787, go to Sources and choose Add source. Pick the loki connector. Give the source a name, set url to your Loki address and query to the LogQL selector for the logs you want. Add bearer_token, or username and password, and tenant_id if your Loki needs them.

3. Add a label named service to the source. Give it a const with the service name if the query selects one service. Give it a field pointing at the service name inside the log event if the query covers several services. Mark it primary.

4. Use the same label name, service, on your metrics and alert sources, with matching values, so all three share one key. The demo in the AI SRE guide keys its metrics, logs and alerts sources by the same service label for this reason.

5. Start the MCP endpoint in a second shell and connect your client.

tares mcp

6. Ask your client about the slow service, for example: “Use tares: why is api-server slow?” The client calls read with the service and gets the correlated timeline.

{"service": "api-server"}

7. If you want the agent woken automatically, add a trigger over your sources, keyed by service. The guide shows max(p99_ms) > 1000 as a latency condition on the typed metric fields.

What you get

• Log lines that share a key with the service’s metrics and alerts, so one read on that service returns all three together.
• A high-latency investigation that starts from one service. The agent sees the p99 climb, the alert that fired and the log lines from the same window on one clock.
• Findings the agent writes land on the same service timeline, so the next investigation starts from the previous conclusion instead of from scratch.
• One consistent label to build triggers on. A trigger keyed by service fires per service and wakes a Tares agent, your own agent service over a webhook, or a Slack channel.
• A setup you can change later. The label is ordinary source configuration, so you can switch from a const to a field without touching your Loki.

When something else fits

• If your logs already flow through OpenTelemetry, use the otlp connector. Its service label reads resourceAttributes.service.name, so there is no separate Loki poll to configure.
• If the logs come from a container on the same host, the docker_logs connector tails that container directly and takes a const service label.
• If you only need a quick look at one incident and have no Loki, the guide’s demo sources let you try the correlated read on a laptop before you wire your own systems.

Related