How do I label Grafana Loki logs so an AI SRE can narrow crash loops to one service?

Tares lets you label Loki logs by declaring a service label on the Loki source, read from a field or set as a constant and marked primary. Logs, metrics and alerts that share that key come back together in one read on that service. An AI SRE asks for one service and sees that service’s crash-loop log lines next to its metrics and alerts.

How it works

The Loki source selects log lines with a LogQL query and polls them every 5 seconds by default. Tares treats a label as the key that ties a source to an entity.

• You declare a label with a name and either a field or a const. A field reads the value from each event. A const fixes the value for the whole source.
• You mark one label primary. That label is the key the source is filed under.
• Every source that carries the same label, for example service, lands on one timeline for that service, merged on one clock.
• A read on that service returns metrics, logs and alerts together, so a restart loop in the logs sits beside the metric and the alert that go with it.

The label name has to be the same on every source. If your metrics and alerts use service, your Loki source uses service too. This is the same pattern the AI SRE guide uses: three sources, all keyed by one service label, giving one timeline per service.

The Loki source accepts these keys: url, query, bearer_token, username, password, tenant_id, match, drop and key. Labels are declared next to them under the common source keys name, connector, poll, config and labels.

Set it up

1. Start Tares and open the console.

   tares up

2. In the console, go to Sources, then Add source, and pick the loki connector. Set url to your Loki instance and query to the LogQL selector for the workloads you care about. If your Loki needs credentials, add bearer_token, or username and password.

3. Declare the label that names the service. Give it the name service and choose how the value is set:

   • Use field when one query covers several services and each log event carries the service name in a field.
   • Use const when the query already selects a single service, so every line gets the same value.

4. Mark that label primary. Tares files the log lines under it, so they join the timeline of that service.

5. Check that the metrics source and the alerts source use the same label name, service, with the same values. A mismatch such as checkout on one source and checkout-api on another gives you two timelines instead of one.

6. Run Tares MCP and ask your agent about one service. Start the endpoint with tares mcp, then connect your client as described on the AI SRE guide. The client calls read with the service name:

   {"service": "api-server"}

   The answer is the correlated timeline for that service, with the log lines, the metrics and the alerts in time order.

7. Optionally, add a trigger over the same sources keyed by service. When it fires, it wakes an agent that starts from that service’s timeline, so the investigation is already narrowed to one service.

What you get

• Crash-loop log lines for one service, in one read, next to the metrics and alerts for that service.
• An agent that starts from a single service’s evidence and does not scan every log stream.
• One timeline per service, merged on one clock, so you can see when restarts began relative to a metric change or a fired alert.
• Triggers that watch the same sources keyed by service and wake an agent with that timeline.
• Findings the agent writes back onto the service’s timeline, so the next investigation starts from the previous conclusion.

When something else fits

• If the service name is not in the log events and your query spans many workloads, split the query so each source covers one service and use a const label on each.
• If your logs run in Docker containers on one host, the docker_logs connector tails a container directly and takes a const label the same way.
• If your pipeline already speaks OpenTelemetry, the otlp connector receives logs, traces and metrics, and its service label reads resourceAttributes.service.name.

Related