Which Prometheus metrics show crash loops?

Tares ingests the Prometheus metrics you pick, so you choose the restart-count series that reveal a crash loop and Tares stores each value as a number you can aggregate. You select metrics by name or by a label they carry, and you write no PromQL. A trigger then filters to one metric by event type and runs max, avg or sum over its value.

How it works

The prometheus connector polls your Prometheus server on an interval. Each series becomes one metric event per poll tick, and it joins with logs, deploys and alerts on the same key.

The metric names come from your own server. Give Tares the base URL and discovery lists the server’s metrics and labels for you to pick from. Use it to find the series that count container or process restarts in your setup.

• By name: type a metric name or a prefix_* pattern, then tick the matches.
• By label: pick a label, see the metrics that carry it, and tick the ones you want.
• Value: the numeric value is stored as a number-typed value label, so a trigger can aggregate it.
• Key: a label such as metric.service can serve as the primary label, so each service has its own series history. Series with no key label fall under default_key, which defaults to unknown.

Your picks compile to a single selector that ingests every chosen metric, and each event is named by its own metric name. Number labels are excluded from entity faceting, so value is for aggregating, not for grouping.

Set it up

1. Start Tares if it is not running yet.

tares up

2. Open Sources → Add source → Prometheus. Enter the url of your Prometheus server (for example http://localhost:9090) plus any auth, then choose Discover. If your server sits behind a proxy, bearer_token, or username with password, are available.
3. Pick the restart metrics, by name or by label. Confirm, and Tares proposes the entity key and labels from a sample. You can also ask a connected agent, for example: “Use tares to ingest the restart metrics from the Prometheus at localhost:9090.” The agent discovers the metrics, samples your picks, tests one poll and creates the source.
4. Check the generated config. It looks like this, with your own picks in the selector:

- name: metrics
  connector: prometheus
  poll: 10s
  config:
    url: http://localhost:9090
    queries:
      - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
        by_name: true
    labels:
      - { name: service, field: metric.service, primary: true }
      - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

5. Open the source. Recent events shows one event per series per poll, and Fields shows the labels and the numeric value. Watch a metric move in Explore by picking the entity the series is keyed on.
6. Create a trigger that reads the metric. Filter it to the restart metric by event_type, then aggregate value with max, avg or sum over a window, with a cooldown so a loop that keeps restarting does not fire repeatedly. See the Prometheus connector page for the full field reference.

What you get

• Restart metrics from your Prometheus server stored as events, one per series per poll, named by their metric name.
• A number-typed value you can aggregate in a trigger, so a rising restart count can start an investigation.
• The same key as your logs, deploys and alerts, so a restarting service lines up with the deploy or error that came with it. This is the correlation that AI SRE builds on.
• A source that discovery and an agent set up for you, with no query language at any step.

If you need a derived figure rather than a raw counter, a queries row can hold one PromQL expression, such as a rate, with an explicit event_type and field. This is the advanced escape hatch. The pick-metrics flow covers raw ingestion without it.

When something else fits

• If your services already export OpenTelemetry metrics, the OTLP connector receives them over OTLP/HTTP, and service.name becomes a service label, so you need no Prometheus server in the path.
• If the Prometheus rules for your restarts already fire alerts, the prometheus_alerts connector reads those fired alerts instead of raw series.

Related