How can an AI SRE match a jump in Prometheus metrics to the lines in Grafana Loki logs?

Tares puts Prometheus metrics and Grafana Loki log lines for the same service on one timeline with one clock, so an AI SRE reads both in a single call. You key both sources by the same service label, and the agent sees when the metric moved next to what the log lines were doing at that moment. The read covers one window for the service, and you can wake the agent from a trigger on the metric value.

How it works

Each source in Tares carries labels, and sources that share a key label are merged into one timeline per service.

• Prometheus metrics. The prometheus connector polls the metrics you pick by name or by a label they carry. Each series becomes a metric event on every poll tick, and you write no PromQL. The numeric value is stored as a number-typed value label.
• Loki logs. The loki connector polls log lines from a Loki instance, including Grafana Cloud, using a LogQL query. It polls every 5 seconds by default. Its settings are url, query, bearer_token, username, password, tenant_id, match, drop and key.
• One key. Both sources carry a service label, so metric events and log events for api-server land on the same timeline, merged on one clock.
• One read. The agent calls read with the service, for example {"service": "api-server"}, and gets the correlated timeline for that window. The metric event where value changes and the log events around it sit in time order in the same answer.

The agent does not stitch two tools together or line up timestamps from two systems. The ordering is already in the read.

Set it up

1. Install Tares and start it locally.

uv tool install tares          # or: brew install glassflow/tap/tares
tares up

2. Add the Prometheus source. In the console go to Sources → Add source → Prometheus, enter the url and any auth, choose Discover, and tick the metrics you want. Tares proposes the entity key and labels from a sample. The generated config looks like this:

- name: metrics
  connector: prometheus
  poll: 10s
  config:
    url: http://localhost:9090
    queries:
      - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
        by_name: true
    labels:
      - { name: service, field: metric.service, primary: true }
      - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

See Prometheus for every field.

3. Add the Loki source. Choose Sources → Add source and pick Loki. Give it the Loki url and a LogQL query that selects the service’s logs. Add bearer_token, or username and password, if your Loki needs auth, and tenant_id if you use one. Give it a primary service label, so it joins the metric events.

4. Check the join. Open Explore and pick the service. Metric events and log lines appear on one clock. What the agent gets shows the exact text an agent receives.

5. Start the MCP endpoint in a second shell and point your client at http://127.0.0.1:8788/mcp.

tares mcp

6. Ask your agent a question tied to the service and the window:

Use tares: what happened to api-server in the last 15 minutes?

The client calls read for the service and answers from the correlated timeline. For the full build, including the demo stack, see Build an AI SRE.

What you get

• One read for both signals. The agent gets metrics and Loki lines for the service in one window, ordered on one clock, instead of querying Prometheus and Loki separately.
• Metric and log timing side by side. When the value of a series changes, the log events from that period sit next to it. The agent can tie its explanation to specific evidence lines.
• No PromQL. You pick metrics by name or label, and Tares builds the selector.
• A way to wake the agent. Because value is a number label, a trigger can aggregate it with max, avg or sum. Filter the trigger to one metric by event_type, then aggregate value. When it fires, the agent wakes with the correlated timeline already in hand.
• Conclusions that persist. An agent in the AI SRE setup writes its finding back onto the service’s timeline. The next investigation reads the evidence and the earlier finding together.

When something else fits

• If you already alert from Prometheus rules, ingest what fires with the prometheus_alerts connector or the alertmanager connector next to your metrics, and the alert lands on the same timeline.
• If your logs come from containers rather than Loki, the docker_logs connector tails a running container’s logs into the same per-service timeline.

Related