Which column should I use as the cursor when polling a Postgres table into Tares?

Tares polls a Postgres table with a cursor_column that only ever grows: an autoincrement id catches inserts, and updated_at catches inserts and updates. Discover reads the table’s columns and picks the cursor for you. The time_column setting defaults to the cursor when it is a timestamp, and to ingest time when it is not.

How it works

The Postgres connector reads a table incrementally and creates one event per new or changed row. The cursor remembers the highest value of the cursor column it has seen. On each poll Tares asks only for rows beyond that value, so it never rereads the whole table.

This is why the cursor column must be monotonic, a value that only ever grows. Pick it by what you need to see:

• An autoincrement id catches inserts. Every new row gets a higher id, so each one arrives as an event. Edits to an existing row keep the same id, so they do not show up again.
• updated_at catches inserts and updates. A row that changes gets a newer timestamp, so it comes through again as a new event. This works when your application sets updated_at on every write.

A column that can go down or stay flat is a poor cursor, because rows can fall behind the cursor and never be read.

The cursor column and the time column are two separate settings:

• cursor_column decides which rows Tares fetches on each poll.
• time_column decides the timestamp used as the event time. If you leave it out, Tares uses the cursor when it is a timestamp column, and ingest time otherwise.

So with updated_at as the cursor, event time follows updated_at with no extra setting. With an integer id as the cursor, events carry the time Tares ingested them, unless you name a timestamp column such as created_at in time_column.

The cursor, key and time columns are always pulled, even if you list only a few columns in columns.

Set it up

1. In Tares, open Sources, choose Add source, then Postgres.
2. Enter the Connection URL (dsn) in the form postgresql://user:pass@host:port/dbname and select Discover. The Postgres must be reachable from the Tares host. The driver ships with Tares, so there is nothing extra to install.
3. Pick your table from the list Discover shows. Tares reads that table’s columns and proposes the cursor, the entity key and the labels.
4. Check the proposed cursor. Keep an id if you only care about new rows. Switch to updated_at if you also want changes to existing rows.
5. Choose which columns to pull. All columns are pulled by default.
6. Set time_column only if the cursor is not a timestamp and you want event time from a column such as created_at.
7. Save the source.

You can also write the source as YAML, or send the same object with POST /api/sources. This example uses updated_at as the cursor and picks tenant_id as the primary label:

- name: orders
  connector: postgres
  poll: 10s
  config:
    dsn: postgresql://user:pass@host:5432/appdb
    table: orders
    cursor_column: updated_at
    columns: id,tenant_id,status,amount,updated_at   # empty = every column
    labels:
      - { name: tenant_id, field: tenant_id, primary: true }
      - { name: status, field: status }

The dsn is stored as a secret. It is redacted in the API and left out of catalog exports.

You can also ask a connected agent: use Tares to ingest the orders table at your connection URL, keyed by tenant_id. The agent calls discover_source to list tables and propose the cursor and labels, then test_source to fetch sample rows in a dry run, then create_source.

To confirm it works, open the source. Recent events shows rows on the next poll. Fields reports the coverage of each column, which tells you whether the key column is reliably present.

What you get

• One event per new or changed row of your application’s source-of-truth table, on the same timeline as your other sources.
• A cursor that Discover proposes from the table’s columns, so you start from a sensible choice and only change it if your needs differ.
• Event time that follows your data. With a timestamp cursor, event time comes from that column. With an id cursor, it is ingest time unless you set time_column.
• Entity keys from a primary label such as tenant_id, so rows group by the entity they belong to.
• Incremental reads on every poll, so Tares fetches only rows beyond the cursor. You can also set limit to cap the rows fetched per poll.

These events feed the monitoring and triage described in AI SRE, next to logs, metrics and alerts.

When something else fits

• If your table has no column that only grows, such as a table that is rewritten in place with no timestamp or id, the cursor cannot track it. Expose a view or add an updated_at column that your application maintains.
• If you need to see edits to existing rows, use updated_at rather than an id, since an id only catches inserts.
• If the data lives behind an API rather than a database, the http_poll connector reads a JSON endpoint on a schedule instead.

Related