How do I label Grafana Loki logs so an AI SRE can narrow 5xx errors on an API to one service?

Tares lets you label Grafana Loki logs by declaring a label from a field or a constant on the loki source and marking it primary. Use the same label name your metrics and alerts sources use, usually service. A read on that service then returns the API’s 5xx log lines, metrics and alerts together on one timeline.

How it works

The loki connector reads log lines from a Loki instance, including Grafana Cloud. A LogQL query that you set on the source selects the lines, and Tares polls them every 5 seconds by default.

Correlation in Tares comes from labels, not from the connector type:

• Each source declares labels. A label takes its value from a field of the event, or from a constant.
• One label is marked primary. It is the key the source is filed under.
• Sources that carry the same primary label name and value share one timeline. In the AI SRE guide, the metrics, logs and alerts sources all carry service, so Tares builds one timeline per service.
• A read for one service returns metrics, logs and alerts merged on one clock. Triggers and agents work from that same timeline.

For 5xx errors on an API, this means the log lines from the API get the same service value as the 5xx rate metric and the alert that fires on it, for example api-server. When the agent asks about that service, it sees the error lines next to the metric that climbed and the alert that fired. Other services’ logs stay out of the read.

Use a constant when the LogQL query already selects one service’s logs. Use a field when one query returns lines for several services and each event carries the service name.

Set it up

1. Start Tares and open the console.

   tares up

2. Go to Sources, then Add source, and choose the loki connector. Set the Loki url and a LogQL query that selects the API’s logs. If your Loki needs credentials, the connector accepts bearer_token, or username and password.

3. Add a label named service. Give it a constant value such as api-server, or point it at the event field that holds the service name.

4. Mark that label primary.

5. Check that your metrics source (for example prometheus) and your alerts source (prometheus_alerts or alertmanager) use the same label name and the same values. The label name must match exactly across all three sources for the logs to join the timeline.

6. Start the MCP endpoint in a second shell and connect your client.

   tares mcp

7. Ask your client about the service, for example: “Use tares: what happened to api-server in the last 15 minutes?” The client calls read with the service set to api-server and answers from the correlated read. In the console, open Explore and pick the service to see the same merged timeline.

If a trigger should wake an agent on 5xx trouble, key the trigger by the same service label. The incident trigger in the AI SRE guide watches its sources keyed by service and fires per service.

What you get

• Loki log lines for the API filed under one service, next to that service’s metrics and alerts.
• One read per service. The agent narrows a 5xx spike to the service you named, without separate queries to Loki, Prometheus and your alerting.
• Evidence the agent can tie to specific lines. In the AI SRE guide, the agent’s finding names the likely cause and links it to evidence lines on the timeline.
• Per-service triggers. A trigger keyed by the shared label fires for the service whose alert arrived, and wakes the agent you picked for that trigger.
• Findings written back to the same service timeline, so the next investigation starts from the previous conclusion.

When something else fits

• If your logs come through an OpenTelemetry pipeline, the otlp connector is a better fit. If they come from a container on the same host, use docker_logs. Both take labels the same way, so the same primary service label applies.
• If you only need to look at logs on their own, without joining them to metrics and alerts, Loki’s own query tools are enough. The shared label matters when you want one correlated read per service.

Related