What should I look for in Grafana Loki logs when a cron job has stopped?

Tares reads your Grafana Loki logs with the loki connector, so you can look for the lines a cron job should write and notice when they stop. The telltale sign is a gap: the job’s start and finish lines vanish at a point in time, often with no error. Give an agent a window of those logs and ask it for counts before and after the gap plus the last line it saw, instead of making it read raw lines.

How it works

The loki connector reads log lines from a Loki instance, including Grafana Cloud. You select lines with a LogQL query, and Tares polls them every 5 seconds by default. The lines land on the timeline of the service they belong to, next to your other sources, such as deploys, metrics and alerts.

A stopped cron job rarely logs its own death, so the evidence is mostly what is missing:

• The job’s regular lines disappear. Look for the start and finish lines the job writes on each run. At some moment they stop, and nothing replaces them.
• The last line before the gap. An error, a lock message, a timeout or an exit message is the likeliest cause. If the last line is a normal finish, the job stopped being scheduled rather than failing.
• Lines from the scheduler or the host around the same time. A restart, a redeploy or a container change near the gap points at the cause.
• A change just before the gap. Most real root causes are a change, so check what shipped shortly before the job went quiet. The Use AI to monitor your systems guide explains why the deploy source is the one worth adding first.

If you hand an agent the raw lines, it spends its effort on repetitive output and can miss what is absent. A window of logs works better when you ask for a change in counts: how many of the job’s lines appeared in each interval before the gap, how many after, and what the final line was. A drop to zero is the answer in one sentence.

Set it up

1. Install Tares and start it locally, then open the console at http://127.0.0.1:8787.

uv tool install tares      # or: brew install glassflow/tap/tares
tares up

2. In the console, go to Sources → Add source and choose the loki connector. Set url to your Loki instance and query to a LogQL query that selects the cron job’s lines. Add bearer_token, or username and password, if your Loki requires them, and tenant_id if it uses one.
3. Use match and drop to keep the lines that matter and discard noise, so the window the agent reads is mostly the job’s own output.
4. Give the source a primary service label, the same one your other sources use. That label puts the cron job’s logs on the same timeline as your deploys and alerts. The Build an AI SRE guide shows how sources keyed by service become one timeline.
5. Start the MCP endpoint in a second shell and point your MCP client at it.

tares mcp

6. Ask the agent for counts, not a read-through of the lines:

Use tares: for the nightly-report service, count the job’s start and finish lines per 15 minutes over the last 24 hours, tell me when they stopped, and quote the last line before the gap.

7. To have an agent look without you asking, add a trigger over the source and choose its condition. Triggers use an aggregate, a predicate and a window, and you pick the condition that fits your job’s schedule.

What you get

• Loki log lines for the cron job on one timeline per service, next to the other sources you connect.
• A clear way to spot the symptom: the job’s regular lines stop, and the last line before the gap is easy to find.
• An agent answer built from counts per interval and a single quoted line, so you see when the job stopped and what it last said without reading the whole window.
• The same evidence from any MCP client, so you can investigate on demand, or let a trigger wake an agent that writes its finding onto the timeline.

When something else fits

• If the job itself should report whether it ran, a heartbeat or alert from your scheduler may catch a missed run sooner than logs. Tares can ingest those alerts alongside your logs.
• If the job writes nothing to Loki on a normal run, there are no lines to count. Add start and finish output to the job first, or use a source that sees the job another way.

Related