What should I look for in Grafana Loki logs when the problem is crash loops?

Tares brings your Grafana Loki logs onto one timeline per service, so you can look for the lines a crash loop leaves: a repeated startup line, a fatal or panic line, and kill or exit messages. The loki connector polls a LogQL query every 5 seconds by default, including on Grafana Cloud. An agent reads those lines next to your alerts and deploys, then writes a finding that ties the failure to specific evidence lines.

How it works

The loki connector polls a Loki instance with the LogQL query you give it and turns the results into events on a Tares source. It works with Grafana Cloud as well as your own Loki. The connector takes these keys: url, query, bearer_token, username, password, tenant_id, match, drop and key. Use match and drop to narrow which lines come in.

Give the source the same service label as your other sources. That label puts Loki logs on the same timeline as metrics, alerts and deploys for the service.

The telltale lines for a crash loop are:

• A startup line that repeats. The same banner, such as “starting server” or “listening on”, shows up again and again within a short window.
• A fatal line right before each restart. Look for panic, fatal, uncaught exception or out of memory messages.
• Exit or kill messages. An exit code, a signal or a killed-by-the-runtime message that appears once per restart.
• A failed dependency at boot. Connection refused or timeout errors from the first seconds after each start, which often explain why the process cannot stay up.

When an alert fires, a trigger wakes a Tares agent, or your own agent service over a webhook. The agent starts with the correlated timeline already in front of it. It writes what broke, since when, the likely cause tied to evidence lines, and a next action. A trigger condition can also count events over a window, using aggregate, predicate and window, for example a count above 0 over 5m.

Set it up

1. Install Tares with uv tool install tares, or with Homebrew using brew install glassflow/tap/tares, then start it.

tares up

2. Open the console at http://127.0.0.1:8787 and go to Sources, then Add source. Choose the loki connector and fill in url and query. Write the LogQL query so it selects the logs of the crashing workload. Add tenant_id, username, password or bearer_token if your Loki needs them.
3. Add a label named service and mark it primary, so the logs join the service’s timeline. Use the same service label on your metrics and alert sources.
4. Use match to keep the lines you care about and drop to remove noise. Startup, fatal and exit lines are the ones that show a crash loop.
5. Add a trigger on the service that fires on the alert for the restarts or the failing pods, and choose who it wakes: a Tares agent, your own agent service over a webhook, or a Slack channel. The Build an AI SRE guide walks through this wiring end to end.
6. To investigate yourself from an MCP client such as Claude Code, start the MCP endpoint in a second shell.

tares mcp

7. Ask your client what happened to the service in the last 15 minutes. It reads the correlated timeline, including the Loki lines, and any earlier finding.

What you get

• Loki log lines on the same clock as your alerts, metrics and deploys, so a restart pattern sits next to the alert and the change that came before it.
• A finding on the service’s timeline that names what broke, since when and the likely cause, tied to evidence lines such as the fatal line before each restart.
• A clear next action in the finding, so the next investigation, yours or another agent’s, starts from that conclusion and not from a blank page.
• Deploy events beside the logs, so you can see whether the loop began right after a release.

When something else fits

• If the crash loop is visible in container logs on a single host, the Docker logs connector tails a running container directly and needs no Loki.
• If you only need to know that restarts are happening, let Prometheus or Alertmanager keep owning that alert and use Tares to correlate the logs and wake the agent when it fires.

Related