What should I look for in Grafana Loki logs when the problem is failed deploys?

Tares reads your Grafana Loki logs with the loki connector and puts them on the service’s timeline next to your deploy events, so you can see which log lines start when a deploy lands. You choose the lines with a LogQL query, and a trigger can count matching events over a window and wake an agent. The agent then starts from the correlated timeline instead of raw log searches.

How it works

The loki connector reads log lines from a Loki instance, including Grafana Cloud. It runs your LogQL query and polls every 5 seconds by default. Its settings include the Loki url, the query and, for multi-tenant setups, a tenant_id. It also has match and drop settings to narrow the lines that come in.

What counts as a telltale line for a failed deploy is defined by your own services: the error or failure lines they log when a release goes wrong. Your LogQL query decides which of those lines reach Tares, so write it around the wording your services use.

Give the Loki source the same service label as your other sources. That label puts logs, metrics, alerts and deploys on one timeline per service. Most real root causes are a change, so the deploy source is the one worth adding first. A webhook source called from your CD pipeline records each deploy as an event on the same timeline. Logs that start misbehaving right after a deploy event then sit next to it on one clock.

Counting is the job of a trigger. A trigger watches the sources for a service and fires on a condition you choose. Conditions use an aggregate such as count, a predicate such as > 0 and a window such as 5m, with a cooldown so repeated firings stay quiet. Pointed at your failure lines, a count condition turns “the failure lines are piling up” into a firing instead of something you spot by scrolling. The firing wakes a Tares agent, or posts the timeline to your own agent service over a webhook. The agent starts with the correlated timeline already in front of it, so it does not spend its first minutes querying each system. It writes a finding with what broke, since when, the likely cause tied to evidence lines and a next action.

Set it up

1. Start Tares and open the console.

tares up

2. In the console, go to Sources and add a source with the loki connector. Set the Loki url and a LogQL query that selects the lines your services emit when a deploy fails. Add a tenant_id if your Loki needs one.
3. Give the source a primary service label so its lines land on that service’s timeline.
4. Add the deploy marker. Create a webhook source with a primary service label, event_type: deploy and a text template like deploy {commit} by {author}: {message}. Creating it gives you a stable ingest URL. Then add one line to your CD pipeline, run on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

5. Create a trigger over the Loki source, keyed by service. Use a count condition over a window and a cooldown, and choose whether it wakes a Tares agent, your own agent service over a webhook, or a Slack channel.

The AI SRE guide walks through the same pattern step by step with a demo system. It then repeats it against your real systems, including the deploy webhook. Use AI to monitor your systems covers the overall picture of sources, triggers and agents.

What you get

• Your Loki log lines and your deploy events on one timeline per service, so a change in the logs lines up with the deploy that preceded it.
• A trigger that reacts to a count of matching lines over a window, so you do not have to watch a dashboard for the pile-up.
• A finding on the timeline that says what broke, since when, the likely cause tied to evidence lines and what to do next.
• The same timeline readable from Claude Code, Cursor or any other MCP client when you want to investigate yourself, starting from the agent’s finding instead of from scratch.

When something else fits

• If Prometheus or Alertmanager already alerts on the failed deploy, connect those alert sources too. They keep owning alerting, and Tares ingests what fired and correlates it with your logs.
• If your services log through OpenTelemetry or Docker rather than Loki, use the OTLP or Docker logs connector for the same timeline.

Related