What does an event from alerts from Prometheus look like once it reaches Tares?

Tares turns each Prometheus alert into one event: the alert name is the event type, the summary is the text, and the alert’s labels become Tares labels plus a state label. The state is firing, pending or resolved. The prometheus_alerts connector polls Prometheus every 30 seconds by default and emits an event only when an alert’s state changes.

How it works

The prometheus_alerts connector polls the /api/v1/alerts endpoint of your Prometheus server. That endpoint lists the alerts your alerting rules have already fired. Prometheus decides what counts as an alert. Tares records it, correlates it with your other sources and can dispatch it to an agent. You need no Alertmanager and no PromQL.

Each alert becomes an event with these parts:

• Event type: the alert name.
• Text: the alert’s summary.
• Labels: the alert’s own labels, as Tares labels, plus a synthetic state label with the value firing, pending or resolved.
• Entity key: a label you mark primary. It is the key Tares uses to group the events of one entity.

Tares tracks state across polls. An alert fires once and re-emits only when its state changes. When the alert disappears from Prometheus, Tares emits a resolved event instead of repeating the old one on every tick.

Pending alerts, the ones still waiting out their for: duration, are left out by default. Turn on include_pending to ingest them too.

Two labels are always present on every alert, severity and state, so those are the ones Tares proposes. Dynamic labels such as namespace and service exist only while an alert is firing. You add them from the Fields tab after real alerts have arrived.

Set it up

1. Open Sources, choose Add source, then Prometheus alerts. Enter the Prometheus url and any authentication, then run Discover. Discover lists your configured alerting rules and the severities present. You can also ask a connected agent: “Use tares to ingest the firing alerts from the Prometheus at localhost:9090.” The agent calls discover_source, then create_source.
2. Optionally filter by severities (blank means all) and toggle include_pending. If your Prometheus sits behind managed or proxy authentication, set bearer_token, or username and password.
3. Declare the labels you want on each event. This is the source definition Tares produces:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

4. You can also send the same object with POST /api/sources.
5. Open the source and check Recent events. You see one event per active alert, and resolved events as alerts clear. Open Fields to see which labels the alerts carry, then promote namespace or service to correlate on them.

For general setup of this connector, see Prometheus alerts. For where it fits in the larger picture, see AI SRE.

What you get

• One event per active alert, with a readable event type and text, so you can scan alerts in Recent events without opening a payload.
• A state label you can filter and trigger on, with values firing, pending and resolved.
• No repeated copies of a long-running alert, because events appear only on a state change.
• Alert labels that you promote into Tares labels, so alerts correlate with logs, metrics, deploys and other sources through a shared entity such as a service or namespace.
• Alerts that an agent can receive and act on, because the events live next to everything else Tares ingests.

When something else fits

• If you want Alertmanager’s grouping, silencing and resolved semantics with real-time delivery, use the Alertmanager connector. It receives a batch from a webhook receiver and fans it into one event per alert. The event type is the alert name, the text is the summary, and the status label is firing or resolved. The full alert, including its fingerprint, stays in the payload.
• If the alert comes from a vendor that Tares has no connector of its own for today, send it through the webhook connector. You declare labels that map payload fields into the event, and the payload is stored lossless.

Related