How do I add Postgres tables to the evidence an AI SRE reads when Alertmanager alert groups fire?

Tares lets you add a Postgres source next to an Alertmanager source, so rows from your own tables sit in the same timeline as the alerts Alertmanager routes. The alertmanager connector receives each alert as an event, and the postgres connector polls a table and records one event per new or changed row. You get a second signal beyond telemetry: the tenants, orders or accounts that changed while the alert fired.

How it works

Two sources feed one timeline, and Tares correlates what they record.

• Alertmanager is push. You point a webhook_configs.url at the source’s ingest endpoint. Alertmanager has already evaluated, grouped and deduplicated, and with send_resolved it tracks firing to resolved. Tares fans the alerts array of each batch out into one event per alert. The event_type is the alert name, the text is its summary, and the labels become Tares labels plus a synthetic status of firing or resolved. The full alert, including fingerprint, stays in the payload.
• Postgres is poll. Tares reads the table incrementally on a cursor column that only grows, such as an autoincrement id or updated_at. Each new or changed row becomes one event. A column such as tenant_id becomes the entity key through a primary label.
• The alert key is derived for you. The Alertmanager source keys on namespace, then service, then job, then instance, or on a declared primary label. Use key_label if you want a different alert label.

For the wider picture of incident work in Tares, see AI SRE.

Set it up

1. Create the Alertmanager source. Ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” Or open Sources → Add source → Alertmanager alerts, name it and press Create. The source page shows a receiver block with the ingest URL filled in. The Alertmanager connector page covers every field.

2. Add the receiver to alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. Add authentication if Tares has auth on. Alertmanager sends a Bearer Authorization header holding an ingest-scoped API key in the webhook_configs entry. The source page prints this block for you when the instance needs a token. Without auth on, you need nothing extra.

4. Create the Postgres source. Ask a connected agent: “Use tares to ingest the orders table at the connection URL, keyed by tenant_id.” Or open Sources → Add source → Postgres, enter the Connection URL and press Discover. Tares lists the tables it can see. Pick one and it proposes the cursor, entity key and labels from the columns. The result looks like this, and the Postgres connector page lists every field:

- name: orders
  connector: postgres
  poll: 10s
  config:
    dsn: postgresql://user:pass@host:5432/appdb
    table: orders
    cursor_column: updated_at
    columns: id,tenant_id,status,amount,updated_at   # empty = every column
    labels:
      - { name: tenant_id, field: tenant_id, primary: true }
      - { name: status, field: status }

The dsn is stored as a secret, redacted in the API and left out of catalog exports. The Postgres host must be reachable from the Tares host, and the driver ships with Tares.

5. Pick columns on purpose. Pull only the columns that show who or what an incident touched. The cursor, key and time columns are always included. Set limit to cap the rows fetched per poll.

6. Verify the alert side. Send a test batch to the ingest endpoint, then check Recent events:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

7. Verify the table side. Open the Postgres source. Recent events shows rows on the next poll, and Fields reports each column’s coverage. Use it to confirm the key column is reliably present. On the Alertmanager source, Fields shows the alert labels, so you can promote namespace or service as the label to correlate on.

What you get

• Every alert Alertmanager routes arrives as its own event, with firing and resolved status, in real time.
• Rows from your application’s tables appear in the same timeline, so you can see which tenants, orders or accounts changed around an alert.
• Each Postgres row is one event with the columns you chose. The cursor means a row is read once, and later changes to it arrive as new events when the cursor column moves.
• The full Alertmanager alert, including its fingerprint, is kept in the payload, so labels you declare later can still reach every field.
• Tares correlates what both sources record with your other sources, and you promote the label that links alerts to table rows from the Fields view.

When something else fits

• If the second signal is an API response rather than a table, use the HTTP API (poll) connector. It polls any JSON endpoint and stores one event per item.
• If you only need alerts that Prometheus rules have fired and do not run Alertmanager, the prometheus_alerts connector polls them instead. Choose alertmanager when you want its grouping, silencing and resolved semantics with real-time delivery.
• If you want a table’s rows to be a ticket or incident record rather than live application state, the Postgres connector still reads them, but you will want to pick the columns with care.

Related