How do I label Grafana Loki logs so an AI SRE can narrow failed deploys to one service?

Tares lets you declare a label on your Loki source, either from a field or a constant, and mark it primary. Log lines that carry the same primary key as your metrics, alerts and deploy events land on one timeline per service. A read on that service then returns the failed deploy and the logs around it together. Use the same label name, such as service, on every source.

How it works

Every Tares source declares labels. A label has a name and takes its value from a field in the event or from a fixed const. One label is marked primary, and that label is the key Tares uses to group events.

When the logs, the metrics and the alerts share the same primary key, Tares merges them into one timeline per service on a single clock. This is the model behind the AI SRE guide: three sources keyed by the same service label give you one read per service.

The loki connector polls log lines from a Loki instance, including Grafana Cloud. It selects them with a LogQL query and polls every 5 seconds by default. Its config keys are:

• url and query
• bearer_token, or username and password
• tenant_id
• match, drop and key

Whether you use field or const depends on your data:

• Use const when the query already selects the logs of a single service. The label then holds a fixed value such as the service name.
• Use field when one query returns lines from several services and each event carries the service name. The label then reads the value from the event.

Failed deploys need a second source. A webhook source with the same primary service label receives a deploy event from your CD pipeline. The deploy then sits on the same timeline as the error logs, so you can see a deploy land and the errors start after it.

Set it up

1. Create or edit your Loki source in the console under Sources, then Add source. Set the url and the LogQL query in its config. Add tenant_id and the credential keys your Loki needs.
2. Declare the primary label under the source’s config. The block below is a constant service label on a loki source. Use field instead of const if each event carries the service name.

- name: logs
  connector: loki
  poll: 5s
  config:
    labels:
      - { name: service, const: api-server, primary: true }

3. Use the same label name and the same values on your metrics and alerts sources, so that all of them key on service. The AI SRE guide keys its Prometheus, log and alert sources this way.
4. Add a webhook source for deploys. Give it a primary service label read from the payload’s service field, event_type: deploy, and a text template like deploy {commit} by {author}: {message}. Creating it gives you a stable ingest URL.
5. Add one line to your CD pipeline, run on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

6. Start the MCP endpoint with tares mcp and connect your client. Then ask it about one service:

Use tares: what happened to checkout in the last 15 minutes?

The client calls read for that service and answers from the correlated timeline. Open Explore and pick the service to see the same merged view in the console.

What you get

• One timeline per service. The failed deploy, the Loki log lines, the metrics and the alerts sit on one clock, and the agent starts from that evidence.
• A narrowed read. Asking about checkout returns only the checkout events, not logs from every service in the Loki instance.
• Triggers keyed by service. A trigger can watch the sources together and wake an agent with the correlated timeline, as the incident trigger does in the guide.
• Findings on the right service. What the agent concludes is written back onto that service’s timeline, so the next investigation starts from the last conclusion.

When something else fits

• If your logs already flow through OpenTelemetry, the OTLP connector may fit better than polling Loki. Its service label reads the path resourceAttributes.service.name.
• If you only need logs from one running container, the docker_logs connector tails it directly and takes a constant label.
• If one service is all you watch and you never combine signals, a single source with a constant primary label is enough.

Related