How do I compare Prometheus metrics with JSON API endpoints when I investigate a service?

Tares lets you compare Prometheus metrics with JSON API endpoints by ingesting both into one timeline and giving them the same label value. The metric shows how the service behaved, and the API items show what it touched. They line up when both carry the same primary label value, such as the service name or a tenant_id the metric also carries.

How it works

Tares has a prometheus connector and an http_poll connector, and both write events to the same timeline.

• Prometheus metrics. Tares polls your Prometheus server on an interval. You pick metrics by name or by a label they carry, and there is no PromQL to write. Each series becomes one metric event per poll. The numeric value is stored as a number-typed value label, so a trigger can take its max, avg or sum.
• JSON API endpoints. The http_poll connector calls any JSON endpoint on a schedule and stores one event per item. An object in the reply is one event, and a list is one event per entry. Nested values are reachable by dotted name, for example current_weather.windspeed.
• The join. Events from different sources share a timeline when they carry the same value in a label that is marked primary. Declare a primary label on each source and fill it with the same value, for example the service name. If you also read a table with the Postgres connector, a tenant_id primary label works the same way, provided the metric series carries tenant_id too.

The payload of a polled item is stored as received, so you can declare more labels later and they still reach every field.

Set it up

1. Add the metrics source. In the console, go to Sources → Add source → Prometheus, enter the url and any auth, and press Discover. Pick metrics by name or by label. Tares proposes the entity key and labels from a sample. You can also ask a connected agent to ingest the metrics you name. The generated config looks like this, and a service label is the primary key:

- name: metrics
  connector: prometheus
  poll: 10s
  config:
    url: http://localhost:9090
    queries:
      - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
        by_name: true
    labels:
      - { name: service, field: metric.service, primary: true }
      - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

2. Add the API source. Go to Sources → Add source → HTTP API (poll). Fill in the method, the URL, a credential if the API needs one, and press Discover. Tares calls the API once, shows the reply and proposes what becomes an event. A polled source looks like this:

- name: weather_berlin
  connector: http_poll
  poll: 10m
  config:
    url: https://api.open-meteo.com/v1/forecast?latitude=52.52&longitude=13.41&current_weather=true
    event_type: weather
    event_time_field: current_weather.time
    id_field: current_weather.time
    text_template: 'Berlin wind {current_weather.windspeed} km/h, {current_weather.temperature} C'
    labels:
      - { name: city, const: berlin, primary: true }
      - { name: windspeed, field: current_weather.windspeed, type: number }

3. Make the key match. In the API source, mark the label that names the service or tenant as primary, using either a field from the item or a const. Give it the same value the metric series carries. In the example above, city is the primary label for the weather items. For your own service you would use the same label value on both sources.

4. Set id_field on the API source. An item whose value was seen in recent polls is not stored again, so use the item’s own id for a list API.

5. Verify. Open each source. Recent events shows what each poll stored, and Fields lists every field seen with its coverage, dotted names included. Then watch both in Explore by picking the shared entity. For a first guide to the whole flow, see AI SRE.

What you get

• One timeline per service or tenant, with metric points and API items side by side.
• Metric values you can aggregate in triggers. Filter a trigger to one metric by event_type, then aggregate value.
• API fields you can use the same way. A trigger on a number label such as max(windspeed) > 62 over 15m fires from the next reading that crosses it.
• Polling that protects the upstream. http_poll refuses a poll under 10 seconds. It backs off on a 429 or any 5xx, and the source’s health says when the next try is.
• Credentials stored as secrets. http_poll sends one credential in the header you name with credential_header, and the Prometheus connector takes an optional bearer token or basic auth. Neither is returned by the API.

When the metric moves, you can see in the same view which orders, readings or status items arrived around that time.

When something else fits

• If the items you want to compare live in a database table rather than behind an API, the Postgres connector reads new and changed rows on a cursor column and joins on a primary label like tenant_id.
• If the system pushes events to you instead of waiting to be polled, use the webhook connector. For an API Tares should call on a schedule, use the HTTP API connector.
• If you need a derived metric such as a p99 rather than raw series, a Prometheus queries row can hold a single PromQL expression with an explicit event_type and field.

Related