What should I look for in Grafana Loki logs when the problem is error spikes?

In Grafana Loki logs, look for a burst of error lines that starts at a specific moment, and Tares brings those lines into one service timeline for an agent to read. The Loki connector selects lines with a LogQL query and polls every 5 seconds by default. Give the source a service label and the error logs sit next to the metrics, alerts and deploys for that service. The agent starts from that timeline and writes what broke, since when, and the likely cause tied to evidence lines.

How it works

An error spike shows up in logs as a change over time. Quiet lines become a run of error lines from one service, usually at the same moment as a rise in the 5xx rate and an alert. The lines to read first are the ones at the start of the burst, because they sit closest to the cause.

Tares helps you see that change in context:

• Loki as a source. The Loki connector reads log lines from a Loki instance, including Grafana Cloud, using a LogQL query you choose. It polls every 5 seconds by default.
• Filtering at the door. The match and drop keys let you filter which lines come in, so the timeline holds the lines that matter for errors.
• One timeline per service. Every source that carries the same service label lands on one timeline, merged on one clock. Logs, metrics, alerts and deploys for a service are read together.
• Triggers over windows. A trigger watches the sources for a service and fires on a condition you choose. Trigger conditions use an aggregate, a predicate and a window, so a rule can ask whether the count of matching events in the last few minutes crosses a line.
• An agent that starts informed. When a trigger fires, the agent already has the correlated timeline in front of it, so it does not spend its first minutes querying each system. It writes a finding with what broke, since when, the likely cause tied to evidence lines, and a next action.

Your agent starts from the correlated timeline and the finding, not from a raw dump of log lines. See Use AI to monitor your systems for the full flow.

Set it up

1. Install Tares and start it:

   uv tool install tares          # or: brew install glassflow/tap/tares
   tares up

2. Open the console at http://127.0.0.1:8787, go to Sources, and choose Add source. Pick the Loki connector.

3. Fill in the connection keys: url for your Loki instance and query for the LogQL selection. For authentication use bearer_token, or username and password. Set tenant_id if your Loki uses tenants. Use match and drop to filter lines.

4. Add a primary service label so the Loki lines join the same timeline as your other sources for that service. Add your metrics and alert sources with the same label, and a deploy source if you can, because most real root causes are a change. A deploy event next to the first error line shows the cause right away.

5. Create a trigger over those sources, keyed by service, with a condition for the spike you care about. Subscribe a Tares agent to it, or send the timeline to your own agent service over a webhook.

6. Rehearse the flow before you point it at production. The demo causes an error spike on purpose:

   curl -s -XPOST localhost:8080/demo/inject -H 'content-type: application/json' \
     -d '{"scenario": "error_spike"}'

   Within about 30 seconds the 5xx rate climbs, the alert fires, the trigger wakes the agent, and the finding lands on the timeline. The demo reads Docker logs and Prometheus, and the flow is the same with Loki as the log source. Build an AI SRE walks through it step by step and then points the setup at your real systems.

What you get

• Error lines from Loki on the same clock as the 5xx rate, the alert that fired and the deploy that landed.
• A trigger that wakes an agent when your condition on the service is met, without anyone paging it.
• A finding on the service’s timeline that says what broke, since when, the likely cause with evidence lines, and a next action.
• The same timeline and finding for any MCP client, such as Claude Code or Cursor, so your own investigation starts from the last conclusion.
• Loki stays your log store. Tares reads from it and does not replace it.

When something else fits

• If your logs are not in Loki, use the connector that matches where they live. Use AI to monitor your systems lists Docker logs, OpenTelemetry (OTLP) and Vercel logs as log sources.
• If you want alert evaluation itself, keep it in Prometheus or Alertmanager. Tares ingests what fired and correlates it with the logs, and it does not re-threshold anything.
• If you only need to browse raw lines or run ad hoc LogQL, Grafana remains the better place for that. Tares is for putting the lines in context and handing them to an agent.

Related