How can an AI SRE see who was affected by errors in OpenTelemetry logs using JSON API endpoints?

Tares shows an AI SRE who was affected by errors in OpenTelemetry logs when the logs and a polled JSON API both carry the same tenant_id label. You add an otlp source for the log records and an http_poll source for the JSON endpoint, and give both a label with the same name and value. The error lines and the API items for that tenant then sit on one timeline, so the AI SRE reads who the errors touched next to the errors themselves.

How it works

Two sources feed one timeline, joined by a label value.

• The otlp source receives OTLP/HTTP logs at POST /v1/logs. A label reads its value by path. For the attributes of a log record the path is attributes.<key>, so a tenant_id attribute on the log line becomes attributes.tenant_id.
• The http_poll source calls your JSON endpoint on a schedule and stores one event per item. payload_key points at the list in the reply, and a label reads tenant_id from each item. Nested values are reachable by dotted name.
• Both sources mark tenant_id as the primary label. An error line for tenant t-42 and an API item for tenant t-42 share that tenant’s timeline.

The join depends on the label. A log line only lands on a tenant’s timeline when it carries that tenant’s value, so make sure your application puts the tenant on the log record as an attribute. When an AI SRE investigates an error burst, it reads the timeline of the affected tenant and sees the surrounding items, such as orders, sessions or statuses from your API, beside the error lines.

For the wider workflow, see AI SRE.

Set it up

1. Create the log source. In Sources → Add source → otlp, or with POST /api/sources, use this object. The label path is attributes.tenant_id, not a bare tenant_id, because a label with a bare field name gets no value from an OTLP event.

- name: otel
  connector: otlp
  config:
    labels:
      - { name: tenant_id, field: attributes.tenant_id, primary: true }

2. Point your exporter at Tares. The OTLP/HTTP endpoint serves /v1/logs. For gRPC, install the otlp-grpc extra and use port 4317.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

The exporter uses a Bearer Authorization header holding an ingest-scoped API key. You need it only when Tares has auth on. Details are on the OpenTelemetry connector page.

3. Create the JSON source. In Sources → Add source → HTTP API (poll), enter the method and URL, then press Discover. Tares calls the API once, shows the reply and fills in what becomes an event. You can also create it directly:

- name: affected_users
  connector: http_poll
  poll: 2m
  config:
    url: https://api.example.com/v1/readings
    payload_key: data.items
    event_type: api_reading
    id_field: id
    event_time_field: updated_at
    text_template: '{tenant_id}: {status}'
    labels:
      - { name: tenant_id, field: tenant_id, primary: true }
      - { name: status, field: status }

id_field is the deduplication key, so an item seen in recent polls is not stored again. If the API needs a credential, set credential and credential_header. The value is stored as a secret and never returned by the API. Extra headers go in headers. The minimum poll is 10 seconds. See the HTTP API (poll) page for all fields.

4. Check the join. On each source, open Recent events to see events arrive, then open Fields to check that tenant_id has high coverage. For the log source, also confirm that your error lines actually carry the attribute.

5. Ask your connected agent to look at a tenant. For example: show me the errors for this tenant and the latest items from the affected_users source.

What you get

• Error lines and API items for the same tenant on one timeline, so you see who an error touched without joining data by hand.
• One otlp source for every service. Each tenant’s errors join the same timeline whichever service emitted them, as long as the log record carries the tenant attribute.
• Items from your own JSON endpoint, stored as received, so labels you declare later can still reach every field.
• Automatic backoff when the API rate limits: Tares waits after a 429 or any 5xx and retries, and the source’s health shows the next try time.
• Source health that shows a config error for any other 4xx, with the status and a slice of the reply.

When something else fits

• If the data about affected users lives in a database table instead of behind an API, use the Postgres connector. It reads new and changed rows on a cursor column such as updated_at and takes tenant_id as the primary label, so the rows join the same tenant timeline.
• If your log lines carry no tenant or user value, there is nothing to join on yet. Add the attribute to your log records first, then use this setup.

Related