What should I look for in OpenTelemetry log records when a cron job has stopped?

Tares receives your OpenTelemetry log records and puts them on one timeline per service, so you can check whether a cron job’s usual lines stopped. The telltale is that the job’s start and completion lines no longer appear while the service’s other lines keep arriving. Ask a connected agent for counts of those lines across two time windows instead of having it read raw lines.

How it works

The otlp connector receives OTLP/HTTP logs, traces and metrics. One source ingests every service. Resource attributes such as service.name become labels, so the cron job’s logs land on the same timeline as the rest of that service. Read more on the OpenTelemetry connector page.

A log label reads its value from one of three places in the event:

• resourceAttributes.service.name for resource attributes.
• A path starting with attributes for the attributes of a log record.
• text for the rendered log line.

There is no bare service field, so a label with field: service gets no value from an OTLP event.

What to look for when a cron job stopped

• The job’s start line. A scheduled job logs something at the beginning of each run. Find the most recent one and note when it was.
• The job’s completion line. A start line with no completion line points to a run that began and never finished. Neither line points to a run that never began.
• An error line just before the last run. An exception, a timeout or a failed connection right before the lines end suggests the job crashed.
• Other lines from the same service. If they keep arriving, the service is alive and only the job has gone quiet.
• A change near the last run. A deploy or config change close to the last good run is the first thing worth checking. Most real root causes are a change, which is why Use AI to monitor your systems suggests adding a deploy source first.

Counts instead of raw lines

A healthy job repeats the same few lines on every run, so the raw lines say little. What matters is the change in how many there are. Give the agent a window of logs and ask it to count the job’s start and completion lines in the recent window and in the window before it. A drop from several per window to zero is the signal, and the agent reports one comparison rather than reading hundreds of identical lines. The agent starts from the correlated timeline for the service, so it does not need to query each system first.

Set it up

1. Install Tares and start it locally.

   uv tool install tares      # or: brew install glassflow/tap/tares
   tares up

2. Create an OTLP source keyed by service. In the console, go to Sources → Add source → otlp, or ask a connected agent: “Use tares to create an OTLP source keyed by service.” The source looks like this:

   - name: otel
     connector: otlp
     config:
       labels:
         - { name: service, field: resourceAttributes.service.name, primary: true }

3. Point the exporter that runs your cron job’s service at Tares.

   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
   # or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

4. If your Tares has auth on, the exporter sends an ingest-scoped API key in an Authorization: Bearer header. On an instance without auth you need no key.

5. Check that records arrive. Recent events shows logs as they come in, and Explore lists your services under the service key.

6. Start the MCP endpoint in a second shell and connect your agent client, as described in Build an AI SRE.

   tares mcp

7. Ask for counts, not lines:

   Use tares: for the service that runs the nightly job, compare how many start and completion lines it logged in the last hour with the hour before, and say when the last one was.

What you get

• Your cron job’s logs sit on one timeline with the service’s other logs, metrics, alerts and deploys, all labeled by the same service.
• You can tell “the job crashed”, “the job never started” and “the service itself is down” apart by checking which lines are still arriving.
• The agent gives you a short count comparison and the time of the last run, so you see the change without reading raw lines.
• If a deploy or config change landed near the last run, it appears on the same timeline for the agent to tie to the stop.
• The agent can write its conclusion back as a finding, so the next investigation starts from it.

When something else fits

• If the job writes no log lines at all, there is nothing for the otlp connector to receive. Add logging to the job, or use a source that reads a different signal.
• If your logs are container output rather than OpenTelemetry, the Docker logs connector puts them on the same per-service timeline.

Related