How can an AI SRE see who was affected by the errors in Grafana Loki logs using JSON API endpoints?

Tares puts your Loki error lines and your JSON API rows on one timeline when both carry the same label value, such as a tenant_id. The rows on that tenant’s timeline then show who the errors touched. You add a Loki source for the logs and an http_poll source for the JSON endpoint, and give both a primary label with the same name and values. An AI SRE reading that timeline sees the errors and the affected customers together.

How it works

Tares groups events by the value of each source’s primary label. Two sources land on the same timeline when their primary labels share a name and a value.

• Loki source. The loki connector polls log lines from a Loki instance with a LogQL query, every 5 seconds by default, and it works with Grafana Cloud. Declare a label named tenant_id on it that reads the field where the log line carries the tenant, and mark it primary.
• JSON endpoint source. The HTTP API (poll) connector calls an endpoint on a schedule and stores one event per item. payload_key points at the list in the reply, for example data.items, and id_field stops an item from being stored twice. Nested values are reachable by dotted name, so a label can read a field several levels down.
• Same label, same timeline. Give the JSON items a primary label that is also named tenant_id. When an error line for tenant acme arrives, the items for acme sit on the same timeline, so you can see which customer, order or account the errors touched.

The label has to exist on both sides. Log lines that do not carry the tenant value have no tenant timeline to join, so check that the field is in your log lines before you rely on this.

If the affected-user data lives in a database instead of an API, the Postgres table connector does the same job. It reads new and changed rows on a cursor column and turns an entity column such as tenant_id into the key through a primary label.

Set it up

1. Add the Loki source. Set the Loki url and your LogQL query so it selects the error lines. Add bearer_token, or username and password, if your Loki needs them. Declare a primary label named tenant_id that points at the log field carrying the tenant.
2. Check the field. Open the source and look at Fields, which lists every field seen with its coverage. Confirm the tenant field is present on the error lines.
3. Add the JSON endpoint as an http_poll source, with a primary label named tenant_id that reads the matching field of each item:

- name: affected_users
  connector: http_poll
  poll: 5m
  config:
    url: https://api.example.com/v1/readings
    payload_key: data.items
    id_field: id
    credential: abc123
    credential_header: X-API-Key
    labels:
      - { name: tenant_id, field: tenant_id, primary: true }

In the UI, go to Sources → Add source → HTTP API (poll), fill in the request and press Discover. Tares calls the API once, shows the reply and fills in what becomes an event. You can also POST /api/sources with the same object. 4. Or read the rows from Postgres with the same label name:

- name: orders
  connector: postgres
  poll: 10s
  config:
    dsn: postgresql://user:pass@host:5432/appdb
    table: orders
    cursor_column: updated_at
    columns: id,tenant_id,status,amount,updated_at   # empty = every column
    labels:
      - { name: tenant_id, field: tenant_id, primary: true }
      - { name: status, field: status }

5. Verify. On the JSON or Postgres source, Recent events shows the rows on the next poll. Open a tenant’s timeline and check that the error lines and the rows appear together.

You can also ask a connected agent to do the setup. For Postgres, it discovers the tables, proposes the cursor and labels, tests the source and creates it.

What you get

• One timeline per tenant that holds both the Loki error lines and the rows from your API or database.
• An AI SRE that can answer “who was hit by these errors” from that timeline, with the customer data next to the errors.
• Rows stored as received, so labels you declare later can still reach every field.
• Deduplicated API items through id_field, and incremental Postgres reads through the cursor column, so each change shows up once.
• Backoff on rate limits for polled APIs. A 429 or 5xx pauses requests, and the source health says when the next try is.

For how an AI SRE uses these timelines end to end, see AI SRE.

When something else fits

• If the log lines carry no value that also appears in your user data, there is nothing to join on. Add the identifier to the logs first, or use a different shared label.
• If your system pushes events to Tares instead of exposing an endpoint, the webhook connector fits better than polling.
• If the data is already in a table, prefer the Postgres table connector over building a JSON endpoint in front of it.

Related