Which Prometheus metrics show 5xx errors on an API?

Tares ingests the Prometheus metrics your API exposes for 5xx responses, picked by name or by a label they carry, with no PromQL to write. Each series becomes an event with a number-typed value label. A trigger can filter to one metric and aggregate that value. Which metric names show 5xx depends on what your API exports, and Tares lists them for you to pick.

How it works

Tares does not ship a fixed list of 5xx metric names. Your API and its exporter decide the names, and the Prometheus connector discovers them on your server.

• Discovery. Given the Prometheus url, Tares lists the server’s metrics and labels for you to pick from.
• Pick by name. Type a metric name or a prefix_* pattern and tick the matches. If your API prefixes its error metrics, a pattern finds them in one step.
• Pick by label. Pick a label, such as the one your API uses for the response code. Tares lists the metrics that carry it, and you tick the ones that show 5xx errors.
• One event per series per poll. Your picks compile to a single selector, and each event is named by its own metric name.
• A number you can aggregate. The metric value is stored as a number-typed value label, so a trigger can use max, avg or sum on it.

To narrow the metrics you ingest, the exclude option on a query entry drops metrics by substring in the name, for example _bucket.

This fits into the wider AI SRE workflow: the 5xx series are joinable with logs, deploys and alerts on the same key.

Set it up

1. Start Tares locally.

   tares up

2. Ask a connected agent to do the discovery for you:

   Use tares to ingest the metrics from the Prometheus at localhost:9090 that show 5xx errors on my API.

   The agent calls discover_source to list the metrics and labels, test_source for one poll, then create_source. You name the metrics. It never writes PromQL.

3. Or use the console. Open Sources → Add source → Prometheus, enter the url (plus any auth) and choose Discover. Pick by name or by label, tick the metrics that carry your 5xx counts, and confirm. Tares proposes the entity key and labels from a sample.

4. Check the generated config. It looks like this, with your own metric names in the selector:

   - name: metrics
     connector: prometheus
     poll: 10s
     config:
       url: http://localhost:9090
       queries:
         - promql: '{__name__=~"(your_5xx_metric)"}'   # generated from your picks
           by_name: true
       labels:
         - { name: service, field: metric.service, primary: true }
         - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

   You can also POST /api/sources with the same object.

5. Verify. Open the source. Recent events shows one event per series per poll, and Fields shows the labels (metric.service, and any others your picks carry) and the numeric value. Watch the metric move in Explore by picking the entity the series is keyed on.

6. Add a trigger that aggregates the value. Filter it to one metric by event_type, then aggregate value:

   - name: api-5xx
     sources: [metrics]
     filters:
       - { field: event_type, op: eq, value: your_5xx_metric }
     condition: { aggregate: max, field: value, predicate: "> your_limit", window: 5m }

   Replace your_5xx_metric with the metric name you picked and your_limit with the number that counts as too many errors for your API.

What you get

• A metric event for each 5xx series on every poll, named by its own metric name.
• A number-typed value label that triggers can aggregate with max, avg or sum.
• A trigger scoped to one metric through its event_type, so other metrics in the same source do not count toward the condition.
• Events keyed on the same entity as your logs, deploys and alerts, so you can line up a spike in errors with what changed.
• A config that the console or agent generates, so you do not hand-write selectors.

When something else fits

• If you want a computed metric, such as a derived rate or a histogram_quantile, a queries row can hold a single PromQL expression with an explicit event_type and field. This is the advanced escape hatch.
• If your API exports OpenTelemetry metrics instead of exposing them to Prometheus, use the OTLP connector. It receives logs, traces and metrics, and service.name becomes the service label.

Related