How do I label Grafana Loki logs so an AI SRE can narrow a disk filling up to one service?

Tares lets you declare a service label on your Loki source, from a field or a constant, and mark it primary. The log lines then share a key with that service’s metrics and alerts. A read on that service returns them together, so an AI SRE looking at a disk filling up sees one timeline for one service instead of every log line in Loki.

How it works

Tares correlates signals per service. Every source that carries the same primary label value lands on one timeline for that service, merged on one clock.

• A loki source polls log lines from your Loki instance with a LogQL query, every 5 seconds by default. Grafana Cloud works too.
• A label is declared with name, and either field (take the value from the event) or const (a fixed value). primary: true marks the label that keys the timeline.
• Your metrics source (prometheus) and your alerts source (prometheus_alerts or alertmanager) carry a service label as well. The matching value is what joins the three.
• A read on that service returns the metrics, the log lines and the alerts together. An agent works from that single read, whether a trigger woke it or you asked through an MCP client.

For a disk filling up, this means the filesystem errors in the logs, the disk metric and the alert that fired sit side by side for the one service that owns the disk. The AI SRE guide shows this pattern end to end with three sources keyed by the same service label.

Set it up

1. Start Tares and open the console.

   tares up

2. Go to Sources, then Add source, and pick the loki connector. You can also ask your connected agent to set the source up.

3. Fill in the Loki connection. Set url and query, the LogQL selector for the service’s logs. If your Loki needs credentials, Tares accepts bearer_token, or username and password, and tenant_id for a multi-tenant setup.

4. Declare the service label and mark it primary. If one Loki query returns only one service’s logs, use a constant. If the lines carry the service name, declare the label from a field of the event instead. Either way, keep the label name service.

   - name: disk_logs
     connector: loki
     poll: 5s
     config:
       url: http://localhost:3100
       query: '{app="api-server"}'
       labels:
         - { name: service, const: api-server, primary: true }

5. Give your metrics and alerts sources the same service label, so the values match exactly. A prometheus source can read it from the metric labels, for example:

   labels:
     - { name: service, field: metric.service, primary: true }

   Do the same for your prometheus_alerts or alertmanager source.

6. Start the MCP endpoint in a second shell and connect your client.

   tares mcp

7. Ask about the service by name, for example: “Use tares: what happened to api-server in the last 15 minutes?” The client calls read with {"service": "api-server"} and answers from the correlated timeline. Open Explore and pick the service to see the same merged view in the console.

What you get

• One timeline per service, with Loki log lines, metrics and alerts merged on one clock.
• A disk alert that arrives next to the log lines and the metric around it, so the agent ties its conclusion to specific evidence lines for that service.
• Triggers keyed by service, so a fired alert can wake an agent that starts from the correlated timeline with no extra reads.
• Findings written back onto the service’s timeline, so the next investigation of that service starts from the previous conclusion.
• The same read from the console in Explore and from any MCP client.

When something else fits

• If the logs come from a container on the host rather than from Loki, the docker_logs connector tails the container directly and takes the same service label.
• If your logs already flow through an OpenTelemetry pipeline, the otlp connector receives them and you label by resourceAttributes.service.name.
• If the service name is not in your logs or metrics at all, fix that at the source first, because the join depends on matching service values.

Related