Which OpenTelemetry metric series show high latency?

Tares receives your OpenTelemetry metric series over OTLP and labels each one by service, so you can find the latency series per service in Recent events and Explore. Resource attributes such as service.name become labels, so every service that exports to Tares lines up on the same service key. For trigger aggregation of a numeric latency value, the Prometheus connector stores the value as a number-typed label that a trigger can take the max or avg of.

How it works

The OpenTelemetry connector is the otlp connector. It receives OTLP/HTTP logs, traces and metrics at POST /v1/{logs,traces,metrics}. One source ingests every service you instrument.

• Labels come from resource attributes. The service label comes from service.name, and it is the natural key. Metrics, logs and traces from the same service share it, so a latency series lines up with the logs and spans of the service that produced it.
• No per-service setup. Everything an OTel SDK or Collector exports correlates by service. The connector has no connector-specific fields, only labels.
• Metric points arrive as events. Recent events shows spans, logs and metric points as they arrive. Explore lists your services under the service key.
• Numeric aggregation in triggers is documented for Prometheus. The Prometheus connector stores each metric value as a number-typed value label, so a trigger can max, avg or sum it. You filter the trigger to one metric by event_type, then aggregate value.

To find which series show high latency, open Recent events on your OTLP source and look at the metric points per service. Open Explore and pick a service under the service key to see everything that service sends. Use the service label to compare the latency of one service against the others.

For a latency number you aggregate in a trigger, use the Prometheus connector. You pick metrics by name or by a label they carry, and Tares builds the selector, so there is no PromQL to write. You can drop series by name with the exclude option, for example _bucket. A queries row can also be one PromQL expression, such as a histogram_quantile p99. Give it an event_type and a field, and Tares emits a computed latency series. Its text can render a line such as p99 {service}={val}ms.

Set it up

1. Create an OTLP source keyed by service. In the console, go to Sources, Add source, otlp. Or ask a connected agent: “Use tares to create an OTLP source keyed by service.” The source looks like this:

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: service, primary: true }

2. Point your exporter at the daemon. Use OTLP/HTTP, or gRPC if you installed the otlp-grpc extra, which listens on :4317:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

3. If your Tares instance has auth on (tares up --auth), the exporter sends a Bearer Authorization header holding an ingest-scoped API key. Without auth you skip this step:

OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <key>

4. Check the result. Recent events shows metric points as they arrive, and Explore lists your services under the service key. Read the latency series of each service from there.

5. To alert on a numeric latency value, add the Prometheus connector alongside. Open Sources, Add source, Prometheus, enter the url, choose Discover, and pick the latency metrics by name or by label. Tares stores the value as a number-typed value label. In a trigger, filter to the metric by event_type, aggregate value with max or avg over a window, and set a cooldown so you hear about a sustained rise once. The trigger fields are sources, filters, condition with aggregate, predicate and window, and cooldown.

For the wider picture of how metrics, logs and alerts work together, see AI SRE.

What you get

• One source for every service. Each OpenTelemetry service lands under the same service key, so you compare latency across services without extra configuration.
• Metrics next to logs and spans. The latency series of a service sits on the same key as its logs and traces, so you move from a slow series to the events around it.
• A visible feed. Recent events shows metric points as they arrive, and Explore shows what each service sends.
• A numeric value for triggers through Prometheus. With the Prometheus connector, you aggregate value with max, avg or sum in a trigger. You act on a sustained rise and not on one slow request.
• Computed latency when you need it. A histogram_quantile p99 as a computed series gives you a ready latency number with its own event_type.

When something else fits

• If your latency already lives in a Prometheus server, the Prometheus connector is the better fit. It stores the metric value as a number-typed value label, and a trigger aggregates it directly.
• If you only need to see which services export metrics and compare them, the OTLP source on its own is enough. Use Recent events and Explore with the service key.
• If you want a p99 rather than raw histogram series, use a computed series on the Prometheus connector with an event_type and a field.

Related