What should I look for in OpenTelemetry log records when the problem is 5xx errors on an API?

Tares puts your OpenTelemetry log records on one timeline per service, so you can spot 5xx errors on an API by their text, attributes and timing. Look for error lines that start at one moment, the status code in the record text or attributes, and what else changed on the same clock, such as an alert or a deploy. Tares can fire a trigger on a condition over a time window, so an agent starts from the correlated timeline instead of raw lines.

How it works

The OTLP connector receives OTLP/HTTP logs, traces and metrics at /v1/logs, /v1/traces and /v1/metrics. One source ingests every service. Resource attributes such as service.name become labels, so the API’s logs correlate with its metrics and alerts by service with no per-service setup.

For a 5xx problem, a log record gives you three places to look:

• The log line. The text field holds the rendered line. Look for the status code and the error message your API writes when a request fails.
• Record attributes. Attributes on the log record are read with paths that start with attributes.. If your instrumentation puts the status code or route there, you can label on it.
• The service. resourceAttributes.service.name tells you which service the lines belong to. There is no bare service field, so a label that uses field: service gets no value from an OTLP event.

The telltale signs are in the shape over time, not in any single line:

• A burst of failing-request lines that begins at a specific minute.
• The same error message repeating across many records.
• One route or one dependency named in most of the failing lines.
• A metric, an alert or a deploy event landing on the same timeline just before the burst.

Tares merges logs, metrics, alerts and deploys on one clock per service. In the demo, the 5xx rate climbs, the HighErrorRate alert fires, and the alert event lands next to the error logs and the metric that tripped it.

Counts matter more than lines. A trigger watches the sources for a service and fires on a condition over a window, and the AI SRE guide names an error-rate predicate for 5xx storms. When the trigger wakes a Tares agent, the agent starts with the correlated timeline already in front of it. It then writes a finding that says what broke, since when, and which evidence lines point to the cause. You can ask a connected agent the same thing yourself, for example: what happened to api-server in the last 15 minutes?

Set it up

1. Install Tares and start it locally.

uv tool install tares                # or: pipx install tares
tares up

2. Create an OTLP source keyed by service. In the console go to Sources, then Add source, then otlp. There is no required config. Or ask a connected agent: use tares to create an OTLP source keyed by service.

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: resourceAttributes.service.name, primary: true }

3. Point your API’s exporter, or your OpenTelemetry Collector, at the daemon.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

4. If Tares has auth on, the exporter sends a Bearer Authorization header that holds an ingest-scoped API key. Without auth, no header is needed.

5. Give your other sources the same service label so they share the API’s timeline: metrics from Prometheus, alerts from Prometheus alerts or Alertmanager, and deploys from a webhook. The deploy source is the one worth adding first, because most root causes are a change.

6. Check the result. Recent events shows logs as they arrive, and Explore lists your services under the service key. Pick the API and read its merged timeline. What the agent gets shows the exact text an agent receives.

7. To read the timeline from your own MCP client, start the endpoint in a second shell.

tares mcp

The Build an AI SRE guide walks through the full chain with a demo that causes a 5xx spike on purpose. Use AI to monitor your systems gives the shorter overview.

What you get

• Your API’s log records sit on the same per-service timeline as its metrics, alerts and deploys, so a 5xx burst reads in context.
• A trigger over a window of events wakes an agent when the failures start. Nobody has to page it.
• The agent starts from the correlated timeline and writes a finding that ties the failure to specific evidence lines and suggests a next action.
• The finding stays on the timeline. The next investigation, yours or another agent’s, starts from that conclusion instead of from scratch.
• Prometheus or Alertmanager keeps owning alerting. Tares ingests what fired and handles the waking.

When something else fits

• If the problem is a metric, such as a rising 5xx rate, start with the Prometheus connector and use logs for the detail.
• If your API writes to Docker container logs rather than OTLP, the Docker logs connector tails them into the same timeline.

Related