How do I label Grafana Loki logs by service so they join the right timeline?

In Tares, you label Grafana Loki logs by service by adding a service label to the loki source and marking it primary. Declare the label from a field or from a constant. Use the same label name on your metrics and alerts sources, and Tares merges all three on one timeline per service.

How it works

The loki source reads log lines from a Loki instance, including Grafana Cloud. It selects them with a LogQL query and polls every 5 seconds by default.

A label tells Tares which entity each log line belongs to. You declare it in the source’s labels list with a name, and give it a value in one of two ways:

• field reads the value from a field of the event. Use it when the service name travels with each line.
• const sets a fixed value. Use it when one Loki query covers exactly one service.

Mark one label primary: true. The primary label is the key the timeline is built on. Every source that carries the same primary label, such as service, lands on the same per-service timeline. That is how a log line from Loki sits next to the metrics and alerts for the same service, on one clock. The AI SRE guide builds this end to end with three sources keyed by one service label.

Set it up

1. In the console, go to Sources, then Add source, and choose the loki connector.
2. Set url to your Loki instance and query to the LogQL selector for the logs you want. If Loki needs authentication, the source accepts bearer_token, or username and password, and tenant_id for multi-tenant setups.
3. Add a service label and mark it primary. For a query that returns one service’s logs, declare it from a constant:

- name: logs
  connector: loki
  poll: 5s
  config:
    # url and query go here
    labels:
      - { name: service, const: api-server, primary: true }

4. If the service name comes with each event, read it from a field instead of a constant. Replace the label line with one that uses field:

labels:
  - { name: service, field: service, primary: true }

5. Give your metrics and alerts sources the same primary service label. For example, a Prometheus source reads it from the metric labels:

labels:
  - { name: service, field: metric.service, primary: true }

6. Open Explore and pick the service. Logs, metrics and alerts appear merged on one timeline.

Keep the label name identical across sources. A log source labeled service and a metrics source labeled svc do not share a key, so they stay on separate timelines.

What you get

• One timeline per service. Loki log lines appear beside the metrics and alerts that carry the same primary label, on one clock.
• One correlated read for agents. Triggers and agents work from that merged timeline, so an agent investigating a service sees its logs next to the alert that fired.
• Triggers keyed by service. A trigger over your sources can watch them per service, because every source shares the key.
• Fewer queries to write. You label once in the source, and you do not stitch logs to metrics by hand at investigation time.

To see the whole chain working, follow the AI SRE guide, which wires metrics, logs and alerts under one service label and wakes an agent from a trigger.

When something else fits

Use const when one source covers one service, and field when one query returns lines for many services. If a single Loki query mixes services and the lines carry no field that names the service, narrow the query per service and create one source for each, each with its own constant label.

If your logs live in a container on the same host, the docker_logs connector tails a running container directly and takes the same kind of label. If your logs already flow through an OpenTelemetry pipeline, the OTLP connector receives them and takes a service label the same way. Both join the same per-service timeline when the primary label matches.

Related