How do I stop an http_poll source from storing the same reading again on every poll?

Tares stops repeat storage when you set id_field on the http_poll source. An item whose id_field value was seen in recent polls is not stored again, and an empty id_field stores every poll. Use the entry’s id for a list API, or the reading’s own time for a single reading that changes only every few minutes.

How it works

An http_poll source calls your endpoint on its poll schedule and stores one event per item in the reply. payload_key decides what counts as an item: an object is one event, and a list is one event per entry.

id_field is the deduplication key. Tares reads that field from each item and compares the value with the values seen in recent polls. If the value is already there, the item is not stored again. If it is new, the item is stored as an event.

Which field you pick depends on the shape of the reply:

• A list API. Use the entry’s id. Each entry keeps its id across polls, so only new entries become events.
• A single reading that changes every few minutes. Use the reading’s own time. The time stays the same until the API publishes a new reading, so each poll in between is skipped.
• Empty. Every poll stores every item, even when nothing changed.

Nested values are reachable by dotted name, so id_field can point at a value such as current_weather.time.

Set it up

1. Open the source, or the one you are about to create, under Sources → Add source → HTTP API (poll). Fill in the request and press Discover. Tares calls the API once, shows the reply and lists the fields it saw, so you can pick the one that identifies an item.
2. Set id_field to that field. For a list API, use the entry’s id. For a single reading, use the field that holds the reading’s time.
3. Set event_time_field to the same time field when the reply has one, so the stored event carries the reading’s time and not the poll time.
4. Set poll to an interval of 10 seconds or more. The connector refuses anything shorter when the source is saved.
5. Save the source. The example below polls a weather reading every 10 minutes and stores a new event only when current_weather.time changes:

- name: weather_berlin
  connector: http_poll
  poll: 10m
  config:
    url: https://api.open-meteo.com/v1/forecast?latitude=52.52&longitude=13.41&current_weather=true
    event_type: weather
    event_time_field: current_weather.time
    id_field: current_weather.time
    text_template: 'Berlin wind {current_weather.windspeed} km/h, {current_weather.temperature} C'
    labels:
      - { name: city, const: berlin, primary: true }
      - { name: windspeed, field: current_weather.windspeed, type: number }

You can also send the same object with POST /api/sources. For the full list of fields, see HTTP API (poll).

6. Check the result. Open the source and read Recent events, which shows each poll’s events with the summary line. Fields lists every field seen with its coverage, so you can confirm the field you chose is present in every item.

What you get

• One stored event per distinct reading, not one per poll.
• Counts, aggregates and trigger conditions that reflect real readings. A trigger on a number label such as max(windspeed) > 62 over 15m fires from the next reading that crosses it, without repeated copies skewing the window.
• A shorter, cleaner Recent events list on the source.
• The freedom to poll often enough to catch a new reading quickly, since repeated values are skipped.
• The payload is still stored as received, so labels you declare later can reach every field of the readings that were stored.

When something else fits

• If you want every poll stored, even when the value is unchanged, leave id_field empty. This suits a case where each poll is itself a data point you want to count.
• If the system can call Tares itself, the webhook connector is a better fit than polling, because it sends events when they happen. See HTTP API (poll) for the polling side. For setting up monitoring end to end, start from AI SRE.

Related