Which Prometheus metrics show error spikes?

Tares ingests whichever Prometheus metrics you pick, so the error metrics on your server become events you can watch for spikes. You choose them by name or by a label they carry, with no PromQL. Tares stores each value as a number-typed value label, and a trigger aggregates it with max, avg or sum.

How it works

• You pick the error metrics. Tares connects to your Prometheus server and lists its metrics and labels. You choose the ones that carry errors, either by metric name (a full name or a prefix_* pattern) or by a label. Picking by label lists every metric that carries that label, and you tick the ones you want.
• Each series becomes an event. On every poll tick, each series of a picked metric becomes a metric event. With a generated basket, each event is named by its own metric name.
• The value is a number. The metric value is stored as a number-typed value label, proposed automatically. The labels the series carries, such as metric.service, are available alongside it, and you can key the source on one of them so the metrics join with logs, deploys and alerts for the same entity.
• Triggers aggregate value. Filter a trigger to one metric by event_type, then aggregate value with max, avg or sum over the window you choose.
• Raw or computed. Raw ingestion covers the pick-metrics flow. If you want a derived rate rather than the raw value, a queries row can hold a single PromQL expression with an explicit event_type and field. This is the advanced option for a computed series.

To keep noise out of the basket, the exclude field on a query entry takes comma-separated substrings to drop by metric name, for example _bucket.

Set it up

1. Make sure you have a reachable Prometheus server and know its base URL, for example http://localhost:9090.
2. Ask a connected agent to do it. For example: “Use tares to ingest the pg_* metrics from the Prometheus at localhost:9090.” Name the error metrics you want instead. The agent lists the metrics and labels, samples your picks, runs one test poll and creates the source. It never writes PromQL.
3. Or use the console. Go to Sources → Add source → Prometheus, enter the url (plus any auth) and choose Discover. Pick metrics by name or by label, then confirm. Tares proposes the entity key and labels from a sample.
4. Check the generated config. It looks like this, with your own picks in the matcher:

- name: metrics
  connector: prometheus
  poll: 10s
  config:
    url: http://localhost:9090
    queries:
      - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
        by_name: true
    labels:
      - { name: service, field: metric.service, primary: true }
      - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

5. If your Prometheus needs authentication, set bearer_token, or username with password. Series that carry no key label fall under default_key, which defaults to unknown.
6. Open the source and check Recent events, which shows one event per series per poll. Fields shows the labels and the numeric value.
7. Add a trigger. Filter it to one error metric by event_type, aggregate value with max, avg or sum, and set the predicate, window and cooldown to match what a spike means for that metric. Decide first whether you alert on the raw value or on a derived rate, and use a computed series for the rate.

The full field list for the source is on the Prometheus connector page.

What you get

• Error metrics from your Prometheus server stored as events, one per series per poll, each carrying a numeric value.
• A per-entity view. The primary label, such as service, keys the series so you can watch a metric move in Explore and correlate it with logs, deploys and alerts on the same key.
• Triggers that act on the numbers. Filter by event_type, then use max, avg or sum over a window to decide what counts as a spike.
• A pick-from-a-list setup. Discovery shows the metric names and labels your server really exposes, so you choose the error metrics from that list rather than guessing.
• A way to chart derived values. A computed series gives you a rate or a quantile instead of the raw number.

Number labels such as value are excluded from entity faceting, so they do not clutter the entity view.

When something else fits

• You want Prometheus rules to decide. If you already have alert rules that fire when errors rise, ingest those alerts with the prometheus_alerts connector, or receive them from Alertmanager. You then work from the alerts themselves rather than the raw metrics.
• Your telemetry comes from OpenTelemetry. If services export metrics through an SDK or Collector, the OpenTelemetry connector receives OTLP metrics, logs and traces. One source covers every service, and service.name becomes the service label.
• You need a computed number. If the raw series does not show the spike, for example you need a rate or a p99, use the computed series option on the Prometheus connector rather than raw ingestion.

Related