Which Prometheus metrics show high latency?

Tares ingests the Prometheus latency metrics you pick, such as request duration or response time series, and stores each reading as a number-typed value label. A trigger filters to one metric by event type and aggregates that value with max, avg or sum, so you can fire on high latency. You pick metrics by name or by a label they carry, and you write no PromQL.

How it works

The Tares prometheus connector polls your Prometheus server on an interval. Each series becomes one metric event per poll tick, joinable with logs, deploys and alerts on the same key.

• Metric names. You choose the latency series from the list Tares discovers on your server. Type a metric name or a prefix_* pattern and tick the matches. Which names appear depends on what your exporters publish.
• Labels. Pick a label and Tares lists the metrics that carry it. The service label from metric.service is the usual entity key, so a slow metric lines up with the service it belongs to.
• The value. The reading is stored as a number-typed value label, proposed automatically, so a trigger can aggregate it. Number labels are excluded from entity faceting.
• Event names. Your picks compile to one selector, and each event is named by its own metric name. That name is what a trigger filters on.

Histogram data needs one decision. Raw bucket series are counts, not a latency figure, and the exclude option drops series whose names contain a substring such as _bucket. When you want a p99 rather than a raw series, add a computed series. It is a single PromQL expression, such as a histogram_quantile, with an explicit event_type and field, and an optional text line like p99 {service}={val}ms.

Set it up

1. Start Tares and make sure your Prometheus server is reachable, for example at http://localhost:9090.

tares up

2. Ask a connected agent to ingest the metrics, or open Sources → Add source → Prometheus in the console. Name the latency metrics you want, for example the duration series your services export. The agent calls discover_source, then test_source, then create_source. In the console, enter the url (plus any auth) and choose Discover.
3. Pick the metrics by name or by label, then confirm. Tares proposes the entity key and labels from a sample. The generated source looks like this:

- name: metrics
  connector: prometheus
  poll: 10s
  config:
    url: http://localhost:9090
    queries:
      - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
        by_name: true
    labels:
      - { name: service, field: metric.service, primary: true }
      - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

The two metric names in the selector are the example’s picks. Yours list the latency metrics you chose. If your Prometheus sits behind managed auth, set bearer_token, or username and password.

4. Create a trigger on the source. Filter it to the one latency metric by event_type, then aggregate value with max, avg or sum over a window. Use max to catch a slow spike and avg for sustained slowness. Pick a threshold in the unit your metric reports, whether seconds or milliseconds.
5. Open the source to verify. Recent events shows one event per series per poll. Fields shows the labels such as metric.service and the numeric value. Pick the entity in Explore to watch the metric move.

What you get

• Latency series from Prometheus stored as events, keyed by service, next to your logs, deploys and alerts.
• A numeric value you can aggregate in a trigger, so high latency becomes a condition rather than a chart you watch.
• A way to scope a trigger to a single metric through its event type, so one slow endpoint does not mix with unrelated series.
• A computed p99 series when a raw metric is not the number you want to alert on.
• Discovery that lists your server’s metrics and labels, so you recognize the latency series by name instead of writing selectors.

Start from the AI SRE guide to see how these events feed the rest of the investigation. The full field list is on the Prometheus connector page.

When something else fits

• If your services export latency through OpenTelemetry rather than a Prometheus server, use the OTLP connector. It receives OTLP metrics and turns resource attributes such as service.name into the service label.
• If you only need the alerts your Prometheus rules already fire, the prometheus_alerts connector polls firing and resolved alerts instead of raw metric values.

Related