How does an AI SRE read Grafana Loki logs while investigating failed deploys?

Tares polls Grafana Loki with a LogQL query, labels the lines by service, and puts them on one timeline with the alert and deploy events your AI SRE agent reads. When a deploy fails, the agent sees the log lines next to the alert and the deploy event on the same clock. This works with a self-hosted Loki and with Grafana Cloud.

How it works

The loki connector is a polled connector. Tares pulls log lines from your Loki instance with a LogQL query, every 5 seconds by default, and holds the credential for Loki itself.

• Source settings. The connector takes url, query, bearer_token, username, password, tenant_id, match, drop and key. Use bearer_token or username and password for the credential. Use tenant_id for a multi-tenant Loki, and match and drop to keep or discard lines.
• One label per service. Every source in the AI SRE setup carries the same service label, and that label marks the primary key. Sources that share it merge into one timeline per service.
• One clock. Metrics, logs and alerts for a service are merged on one clock, so a log line that appears two minutes after a deploy sits two minutes after it on the timeline.
• Deploys as events. Add a webhook source with a primary service label and event_type: deploy. Each POST from your CD pipeline lands as a deploy event on that service’s timeline.
• Triggers wake the agent. A trigger keyed by service watches the sources. When an alert event arrives, it wakes an agent. The agent reads the correlated timeline and writes its conclusion back onto the same timeline.

Tares keeps alerting where it is. Your alert rules still decide what fires, and Tares ingests the alerts and correlates them with the log lines around them.

Set it up

1. Start Tares. Install it with uv tool install tares or brew install glassflow/tap/tares, then start it:

   tares up

2. Add the Loki source. In the console, go to Sources, choose Add source and pick loki. Set url to your Loki address, or your Grafana Cloud Loki address. Add the credential as bearer_token, or as username and password. Set query to a LogQL selector for the services you deploy. Add tenant_id if your Loki needs one.

3. Label it by service. Add a label named service, read from a field of the log line or set as a constant, and mark it primary. Use the same label name on every source you want on that timeline.

4. Match your alert source. Add your alert source, prometheus_alerts or alertmanager, with the same service label. Then the alert for a failed deploy and its log lines meet on one timeline.

5. Record deploys. Create a webhook source with a primary service label, event_type: deploy and a text template that reads your deploy fields. Creating it gives you a stable ingest URL. Add one line to your CD pipeline, run on every deploy:

   curl -X POST https://<your-tares>/ingest/<ingest_key> \
     -H 'content-type: application/json' \
     -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

6. Choose who investigates. Create a trigger keyed by service over these sources and pick what it wakes. That can be a Tares agent with a prompt you edit, your own agent service over a webhook, or a Slack channel.

7. Investigate on demand too. Start the MCP endpoint and connect your client to it:

   tares mcp

   Then ask about the service and the time window, for example what happened to checkout in the last 15 minutes. The client reads the correlated timeline, including the Loki log lines.

What you get

• Log lines beside the alert. The agent reads error logs from Loki on the same timeline as the alert that fired and the metrics around it. It does not need a separate query tool for each system.
• The change in view. With the deploy webhook in place, the agent sees a sequence such as deploy landed at 14:02 and errors started at 14:04. For a failed deploy, that points at the change.
• A written conclusion. The agent writes a finding onto the service’s timeline. It names what broke, since when, and the likely cause tied to evidence lines, and it suggests a next action. The next investigation, by an agent or by you through MCP, starts from that finding instead of from scratch.
• Nothing re-thresholded. Your existing alerts keep deciding what is an incident. Tares adds the evidence and does the waking.

When something else fits

• If your logs already flow through an OpenTelemetry pipeline, the otlp connector receives them directly, and you can skip polling Loki.
• If the failing deploys run in local containers, the docker_logs connector tails a container’s logs straight into Tares.
• If you only need a one-off look at a log line, query Loki in Grafana. Tares pays off when you want logs, alerts and deploys correlated per service and an agent woken automatically.

Related