How do I read JSON API endpoints next to logs from Docker containers when an AI SRE investigates?

Tares puts JSON API items and Docker container log lines on one timeline when both sources carry the same label value, such as service api-server. The AI SRE then reads that one value and gets the log lines and the API items together. You set this up with a docker_logs source and an http_poll source that share a label.

How it works

Each source turns what it reads into events and stamps them with labels. The primary label is the entity key, and events from different sources that carry the same value for it belong to the same entity.

• The Docker logs connector tails a running container and stores one event per log line. Optional match and drop regex filters trim the lines.
• The HTTP API (poll) connector calls a JSON endpoint on a schedule and stores one event per item. A list in the reply becomes one event per entry. Nested values are reachable by dotted name, so a label can read a field such as current_weather.windspeed.
• The payload is stored as received, so labels you declare later can still reach every field.

The join is the label value. If the logs source labels its lines service = api-server and the API source labels its items service = api-server, both land on the api-server timeline. When an AI SRE investigates api-server, one read of that value returns the log lines and the API items in time order. If the two sources use different values, they stay on separate timelines, so pick the value once and reuse it.

For the broader picture of how investigations use these timelines, see AI SRE.

Set it up

1. Decide the shared value. The examples below use service with the value api-server. Any label name works, as long as both sources declare it with the same value.

2. Add the Docker logs source. Ask a connected agent to find your running containers and ingest the api-server’s logs, or go to Sources → Add source → docker_logs and press Discover. Tares needs access to the Docker daemon, so run taresd on the host or mount the Docker socket. The source sets the shared label as a constant and marks it primary:

- name: logs
  connector: docker_logs
  poll: 5s
  config:
    container: tares-demo-api-server   # the container name (docker ps)
    labels:
      - { name: service, const: api-server, primary: true }

3. Add the API source. Go to Sources → Add source → HTTP API (poll), fill in the method and URL, and press Discover. Tares calls the endpoint once, shows the reply, and fills in what becomes an event. Declare the same label with the same value:

- name: api-readings
  connector: http_poll
  poll: 5m
  config:
    url: https://api.example.com/v1/readings
    labels:
      - { name: service, const: api-server, primary: true }

4. Add a credential if the API needs one. Put the whole value in credential and name the header in credential_header, which defaults to Authorization. Tares stores the credential as a secret. Extra headers go in headers.

5. Keep deduplication in mind. Set id_field to the item’s own id so a repeated item is not stored again. If it is empty, every poll stores every item.

6. Verify. Open each source in the console. Recent events shows lines and items on the next poll, and Fields lists every field seen. Then read the api-server value and check that both kinds of events appear together.

If you want a value that changes per line, such as a tenant, label_pattern on the logs source takes a regex with named groups and gives each line its own label context. The API items then need to carry that same label value to share its timeline.

What you get

• One read of a label value returns log lines and API items together, so the AI SRE does not stitch two feeds by hand.
• Log lines arrive within the poll interval of the logs source, and API items arrive on the schedule of the API source.
• API replies are stored as received, so you can add labels later and reach any field, including nested ones.
• Rate limits are handled for you. The API connector refuses a poll under 10 seconds, and on a 429 or any 5xx it backs off, doubling the wait from the poll interval up to an hour. A Retry-After value is honored, and the source’s health shows when the next try is.
• Other 4xx replies show up as a config error on the source’s health, with the status and a slice of the reply.

When something else fits

• If the data you want next to the logs is rows in a table rather than an API, use the Postgres table connector. It reads new and changed rows on a cursor column and uses a primary label such as tenant_id as the key. Rows join the logs only when the log lines carry the same label value.
• If a system should call Tares instead of Tares calling it, use the webhook connector rather than polling.

Related