Will a Prometheus alert be ingested again on every poll while it keeps firing?

No, Tares ingests a firing Prometheus alert once, not on every poll. The prometheus_alerts connector tracks state across polls, re-emits an alert only when its state changes, and emits a resolved event when the alert disappears. The severities and include_pending settings decide which alerts are ingested at all.

How it works

The Prometheus alerts connector polls Prometheus’s own alerts endpoint, /api/v1/alerts, every 30 seconds by default. It reads the alerts your Prometheus alerting rules have already fired. Prometheus decides what counts as an alert. Tares does not run PromQL and does not need Alertmanager.

Between polls, Tares remembers the state of each alert. That gives you this behavior:

• A new alert produces one event the first time it is seen.
• An alert that keeps firing produces no further events on later polls. You do not get one copy per tick.
• An alert that changes state, for example from pending to firing, is emitted again, because the state changed.
• An alert that disappears produces one resolved event.

Each event carries the alert name as its event_type and the alert summary as its text. The alert’s labels become Tares labels, plus a synthetic state label whose value is firing, pending or resolved.

Two settings control what is ingested:

• severities is a comma-separated list such as critical,warning. Leave it blank to ingest every severity.
• include_pending is a boolean. Turn it on to also ingest alerts that are still pending, meaning they are waiting out their for: duration. Leave it off and pending alerts are not ingested.

Set it up

1. Make sure you have a reachable Prometheus server with alerting rules configured.
2. Add the source. You can ask a connected agent, for example: “Use tares to ingest the firing alerts from the Prometheus at localhost:9090.” The agent calls discover_source to list the configured rules and the severities present, then calls create_source.
3. Or add it in the UI under Sources, Add source, Prometheus alerts. Enter the url and any auth, then choose Discover. Optionally filter by severity, and toggle include pending.
4. Or write the source yourself, or send the same object with POST /api/sources:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

5. If your Prometheus needs authentication, set bearer_token, or username together with password.
6. Verify. Open the source. Recent events shows one event per active alert, and resolved events as alerts clear. The Fields tab shows the labels the alerts carry.

Discover proposes only the labels that are always present, severity and state. Labels such as namespace or service exist only while an alert is firing. Add them from the Fields tab once real alerts have been ingested, and you can then correlate on them.

What you get

• One event per active alert, not a repeat on every poll, so your event stream stays readable while an incident lasts.
• A state change, such as pending to firing, shows up as a new event.
• A resolved event when the alert clears, so you can see when an incident ended.
• Control over noise: filter by severities, and decide with include_pending whether alerts still waiting out their for: duration are ingested.
• Alerts that Tares can correlate with your other sources and can dispatch to an agent.

When something else fits

• If you already route alerts through Alertmanager, the alertmanager connector takes what Alertmanager sends to a webhook receiver pointed at a Tares ingest URL.
• If you want raw metric values rather than fired alerts, the prometheus connector polls metrics you pick by name or label.

Related