How do I compare Prometheus metrics with Postgres tables when I investigate a service?

Tares puts Prometheus metrics and Postgres rows on one timeline when both sources carry the same value in their primary label, such as a service name. The metric shows how the service behaved, and the rows show the data it touched. Add a prometheus source and a postgres source, give each a primary label that holds the same value, and read both side by side for that entity.

How it works

Each source turns what it reads into events, and events that share a label value sit on one timeline.

• The prometheus connector polls the metrics you pick and stores one event per series per poll. The metric value is stored as a number-typed value label, so a trigger can aggregate it with max, avg or sum.
• The postgres connector polls a table incrementally and stores one event per new or changed row. A cursor column that only grows, such as an id or updated_at, tracks where it left off. A column such as tenant_id becomes the entity key through a primary label.
• Both are joinable on the same key. The metric and the row line up only when the primary label on each side holds the same value. If the values differ, the two sources stay on separate timelines.
• For an API instead of a table, the HTTP API (poll) connector stores one event per item the endpoint returns. It follows the same rule: give its primary label the same value.

This fits the investigation work described in AI SRE.

Set it up

1. Start Tares locally if you have not already.

   tares up

2. Add the metrics. Go to Sources → Add source → Prometheus, enter the url of your Prometheus server and press Discover. Pick metrics by name, such as a prefix_* pattern, or by a label they carry. You write no PromQL. Tares proposes the entity key and labels from a sample. The Prometheus connector page lists every option. The generated config looks like this:

   - name: metrics
     connector: prometheus
     poll: 10s
     config:
       url: http://localhost:9090
       queries:
         - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
           by_name: true
       labels:
         - { name: service, field: metric.service, primary: true }
         - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

3. Add the table. Go to Sources → Add source → Postgres, enter the connection URL (dsn) and press Discover. Pick the table, and Tares proposes the cursor, entity key and labels from its columns. Choose which columns to pull, then save. See the Postgres connector page for all fields.

   - name: orders
     connector: postgres
     poll: 10s
     config:
       dsn: postgresql://user:pass@host:5432/appdb
       table: orders
       cursor_column: updated_at
       columns: id,tenant_id,status,amount,updated_at   # empty = every column
       labels:
         - { name: tenant_id, field: tenant_id, primary: true }
         - { name: status, field: status }

4. Make the keys agree. The two examples above use different primary labels, service and tenant_id, so on their own they do not share a timeline. Choose a primary label on each side whose values are the same. Open the Prometheus source and read Fields to see which labels your series carry. Then choose the table column that holds the same value and mark it as the Postgres source’s primary label.

5. Verify each source. Open it and check Recent events. On the Postgres source, Fields reports each column’s coverage, so you can confirm the key column is reliably present.

6. Investigate. In Explore, pick the entity the series is keyed on. Watch the metric move and look at the rows that changed for the same entity.

What you get

• One timeline per entity, with metric events and table rows side by side, so you can see when a metric moved and which rows changed around that time.
• Your application’s source-of-truth data next to the service’s behavior, without exporting either side.
• Incremental reads. The Postgres source picks up only new or changed rows, and the credentials stay in the source: the dsn is stored as a secret.
• A numeric value on every metric event, so you can later aggregate it in a trigger. Filter the trigger to one metric by event_type, then aggregate value.
• The same pattern for API data. Add an http_poll source whose primary label matches, and its items join the timeline too.

When something else fits

• If the metric series and the table share no common value, such as a service name, the two will not align, and a computed or relabeled key is not something to guess at. Fix the key first.
• If you need a derived metric such as a rate or a p99, a Prometheus queries row can hold a single PromQL expression with an explicit event_type and field. Raw metric picks need no query language.
• If the data you need is exposed through an HTTP endpoint rather than a table, use the HTTP API (poll) connector.

Related