Can an AI SRE explain a metric change in Prometheus metrics with JSON API endpoints?

Yes, Tares lets an AI SRE read a Prometheus metric change next to a polled JSON endpoint, when both sources carry the same label value such as tenant_id. The agent sets the movement in the metric beside what changed in the API data for that value. The JSON endpoint adds business context only for the value both sources share.

How it works

Tares ingests each source as events on a shared timeline, and the join happens on a label value both sources carry. This is the core of the AI SRE approach: one key, several sources.

• Prometheus side. The Prometheus connector turns each series you pick into a metric event on every poll tick. You choose metrics by name or by a label they carry, and you write no PromQL. The metric value is stored as a number-typed value label, so a trigger can aggregate it.
• JSON API side. The HTTP API (poll) connector calls an endpoint on its schedule and stores one event per item. A list in the reply becomes one event per entry. Nested values are reachable by dotted name, so a label can read a field deep in the reply.
• The shared key. Mark the same label, for example tenant_id, as primary on both sources. Tares joins the metric events and the API events on that value.
• The limit of the join. An API field that has no counterpart label in the metric adds nothing for that metric. Context lines up only for values both sources carry.

With both sources on the timeline, a connected agent can put a movement in the metric next to the changes in the API data for that tenant. It can also say when the two do not line up, for example when the metric moved and nothing changed in the API data for that value.

Set it up

1. Start Tares locally and open the console.

   tares up

2. Add the Prometheus source. Go to Sources → Add source → Prometheus, enter the url and any auth, press Discover, and pick the metrics. Or ask a connected agent to ingest the metrics you name. The agent calls discover_source, test_source and create_source, and never writes PromQL.

3. Make the metric’s key label the shared one. The metric series must carry the label, here tenant_id, and the source marks it primary:

   - name: metrics
     connector: prometheus
     poll: 10s
     config:
       url: http://localhost:9090
       queries:
         - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
           by_name: true
       labels:
         - { name: tenant_id, field: metric.tenant_id, primary: true }
         - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

4. Add the JSON endpoint. Go to Sources → Add source → HTTP API (poll), fill in the method and URL, add a credential if the API wants one, and press Discover. Tares calls the API once, shows the reply, and fills in what becomes an event. The credential travels in one header, stored as a secret.

5. Give the API events the same primary label. Set id_field so an item already seen in recent polls is not stored again, and event_time_field so events carry the API’s own time:

   - name: tenant_changes
     connector: http_poll
     poll: 5m
     config:
       url: https://api.example.com/v1/readings
       event_type: api_reading
       id_field: id
       labels:
         - { name: tenant_id, field: tenant_id, primary: true }

6. Verify both sources. Open each source: Recent events shows the events, and Fields lists every field with its coverage. Confirm that tenant_id shows up reliably on both. Then watch the metric move in Explore by picking the entity the series is keyed on.

The connector refuses a poll under 10 seconds for the JSON endpoint. On a 429 or any 5xx it backs off, and the source’s health says when the next try is.

What you get

• A metric movement and the API changes for the same tenant on one timeline, so the agent explains the movement with data instead of guessing.
• An honest answer when the two disagree. If the API shows no change for that value, the agent can report that the metric moved without a matching change.
• Every API field stays reachable. The payload is stored as received, so labels you declare later can still reach every field of events already stored.
• No query language on the metric side. You pick metrics, and Tares builds the selector.
• Numeric metric values that triggers can max, avg or sum, filtered to one metric by event_type.

When something else fits

• If the business context lives in your own database, use the Postgres connector. It reads new and changed rows incrementally on a cursor column, and an entity column such as tenant_id becomes the key through a primary label.
• If the metric and the API share no label value, the join has nothing to match. Add the label to the metric, or use a field the API already returns.
• If the metric alone answers the question, the Prometheus source by itself is enough.

Related