How does an AI SRE read Grafana Loki logs while investigating a cron job that stopped?

Tares reads Grafana Loki logs through its loki connector, which polls a LogQL query and puts the log lines on the same per-service timeline as your alerts. When the alert for a stopped cron job fires, your agent reads the job’s last log lines next to it, on one clock. You label the Loki source by service, so the agent asks for that service and gets logs and alert together.

How it works

The loki connector is a polled source. Tares runs your LogQL query against a Loki instance, including Grafana Cloud, every 5 seconds by default, and stores each log line as an event.

The connector takes these settings:

• url and query, the Loki address and the LogQL selector.
• bearer_token, or username and password, for credentials.
• tenant_id for multi-tenant Loki.
• match, drop and key to shape which lines you keep.

Every source in Tares can carry a label. When your Loki logs, your metrics and your alerts all carry the same service label, Tares merges them into one timeline per service. That is what the agent reads. A cron job that stopped usually leaves a trail: a last successful run, an error line, or a gap where lines used to be. The correlated read shows the alert and the surrounding log lines together, so the agent can tie its conclusion to specific lines.

There are two ways the agent gets there, as described in Build an AI SRE:

• Push: a trigger watches your sources, keyed by service. When the alert event arrives, it wakes an agent, which reads the timeline and writes a finding back.
• Pull: you ask your own MCP client a question, and it calls the read tool with the service name.

Set it up

1. Install Tares and start it. Open the console at http://127.0.0.1:8787.

   tares up

2. Go to Sources, Add source and choose the loki connector. Set url to your Loki or Grafana Cloud address. Set query to a LogQL selector that matches the cron job’s output. Add bearer_token, or username and password, if your Loki needs credentials, and tenant_id if it is multi-tenant.

3. Add a service label and mark it primary. Use a constant value, such as the cron job’s service name, or read it from a field on the log line. Give your alert source, which can be prometheus_alerts or alertmanager, the same service label with the same value. Matching labels are what put logs and alerts on one timeline.

4. Open Explore and pick the service. Check that the alert event and the log lines for the cron job sit side by side.

5. Create a trigger over the sources, keyed by service, that fires when the alert event arrives, and subscribe an agent to it. Prometheus or Alertmanager keeps owning the alerting. Tares ingests what fired and wakes the agent.

6. To investigate by hand, start the MCP endpoint in a second shell and connect your client.

   tares mcp

7. Ask your client a question about the service:

   Use tares: what happened to nightly-report in the last 15 minutes?

The client reads the correlated timeline for that service and answers from the log lines, the alert and any earlier finding.

What you get

• Loki log lines and the cron job’s alert on one timeline, so the agent reads both over the same window.
• A single read per service. The agent does not query Loki, your metrics and your alerts separately.
• Findings tied to specific log lines, written back to the service’s timeline. The next investigation starts from the previous conclusion.
• A trigger that wakes the agent when the alert arrives, so the first look is on the timeline before anyone opens a dashboard.
• The same evidence for interactive questions from any MCP client.

When something else fits

• If your cron job writes to a container’s logs and you do not run Loki, the docker_logs connector tails a container directly. OTLP fits when the job ships OpenTelemetry logs.
• If the job reports through a CI or deploy system rather than logs, send its events to a webhook source labeled with the same service.

Related