How do I turn a nested list in a JSON reply into one event per item with http_poll?

Tares turns a nested list into one event per item when you set payload_key on an http_poll source to the path of that list, such as data.items. A list at that path gives one event per entry, and an object gives one event. event_time_field sets each event’s time, text_template writes its summary, and dotted names reach nested values.

How it works

The HTTP API (poll) connector calls your endpoint on the source’s schedule and stores what comes back as events. Four settings decide the shape of those events.

• payload_key names the part of the reply that is the payload, for example data or data.items. If it is empty, the entire JSON is the payload. When the key points at an object, you get one event. When it points at a list, you get one event per entry.
• event_time_field names the field in each item that holds the event’s time, in ISO-8601 or epoch format. If it is empty, the event time is the poll time.
• text_template is the one-line summary of each event. You put field names from the item in braces, and dotted names work inside them. If it is empty, Tares stores the payload’s JSON as the summary.
• id_field is the deduplication key. An item whose value was seen in recent polls is not stored again. For a list API, use each entry’s own id. If it is empty, every poll stores every item again.

Dotted names reach nested values everywhere in Tares: in event_time_field, in text_template, in labels, and in trigger filters and conditions. A reading such as current_weather.windspeed can be a label or a field a trigger watches.

Tares stores the payload as received. Labels you declare later can still reach every field, including ones you did not use in the first version of the source.

Set it up

1. Open Sources → Add source → HTTP API (poll). Enter the method, the URL and, if the API needs it, a credential. You can add extra headers, and a body for POST.
2. Press Discover. Tares calls the API once, shows the reply it got and fills in what becomes an event. Check that the list you want is the part it picked, then set payload_key to its path if needed, such as data.items.
3. Pick the entity key from the fields Tares saw, then press Create.
4. Or declare the source in YAML. This example reads a list at data.items, takes the time from each item, deduplicates on the item’s own id and writes a summary line. The field names after field: and in the template stand for fields in your own reply.

- name: readings
  connector: http_poll
  poll: 10m
  config:
    url: https://api.example.com/v1/readings
    payload_key: data.items
    event_type: reading
    event_time_field: time
    id_field: id
    text_template: 'Reading {id} at {location.name}: {value}'
    labels:
      - { name: location, field: location.name, primary: true }
      - { name: value, field: value, type: number }

5. Send the same object with POST /api/sources if you create sources through the API.
6. Keep poll at 10 seconds or more. Tares refuses a shorter interval when you save the source.
7. Open the source to check the result. Recent events shows each poll’s events with the summary line, and Fields lists every field seen, dotted names included, with its coverage.

If the API needs authentication, put the whole value in credential and the header name in credential_header. Tares stores the credential as a secret and never returns it through the API or exports it. Headers that are not secret go in headers.

You can also ask a connected agent, or describe the goal in the guided setup. The plan includes an http_poll source, and discover_source reads one reply and fills in the rest.

What you get

• One event per list entry, each with its own time, so a poll that returns twenty items stores twenty events instead of one large blob.
• A readable summary line on every event, built from nested values with dotted names.
• No repeated events across polls when id_field is set, because an item seen in recent polls is not stored again.
• Labels and triggers that watch nested fields. A trigger on a number label such as max(windspeed) > 62 over 15m fires from the next reading that crosses it. AI SRE shows how events like these feed monitoring and escalation.
• Backoff handled for you. On a 429 or any 5xx, Tares waits before the next request and doubles the wait from the poll interval up to an hour. A 429 with Retry-After uses that value, and the source’s health shows when the next try is. Any other 4xx appears as a config error on the source’s health, with the status and a slice of the reply.

When something else fits

• If the system calls Tares instead of Tares calling it, use the webhook connector.
• If a dedicated connector already covers the system, such as Postgres rows or Prometheus metrics, use that connector instead of reading its API with http_poll.
• If you want one event for the whole reply, leave payload_key empty or point it at an object, and you get a single event.

Related