Should I ingest alerts through the Alertmanager connector or the prometheus_alerts connector?

With Tares, use the alertmanager connector for Alertmanager’s grouping, silencing and real-time delivery, and prometheus_alerts to poll Prometheus directly. Alertmanager pushes alerts to a Tares ingest URL as it routes them. The prometheus_alerts connector reads Prometheus’s own alerts endpoint every 30 seconds by default, needs no Alertmanager and no PromQL, and tracks each alert’s state across polls.

How it works

The two connectors differ in who delivers the alert and who decides what it means.

alertmanager (push)

• You point a webhook_configs.url in Alertmanager at the source’s ingest endpoint. Every alert Alertmanager routes becomes an event as it is sent.
• Alertmanager has already evaluated, grouped and deduped the alerts. With send_resolved it also tracks firing to resolved. Tares records what it pushes and correlates it with your other sources.
• Alertmanager posts a batch. Tares fans the alerts array out into one event per alert. The event_type is the alert name, the text is its summary, and the alert labels become Tares labels plus a synthetic status of firing or resolved. The full alert, including its fingerprint, stays in the payload.
• The entity key comes from a label automatically, in the order namespace, service, job, instance, or from a label you declare as primary. The optional key_label field picks a different alert label.

prometheus_alerts (poll)

• Tares polls Prometheus’s own /api/v1/alerts, the alerts its alerting rules have already fired. There is no Alertmanager in the path and no PromQL to write. Prometheus decides what counts as an alert.
• You get one event per active alert, keyed by a label, plus a resolved event when it clears.
• State is tracked across polls. Each alert fires once, re-emits only on a state change, and emits resolved when it disappears, not one copy every tick.
• The event has the alert name as event_type, its summary as text, its labels as Tares labels, and a synthetic state of firing, pending or resolved.
• You can filter with severities (blank means all) and turn on include_pending to also take alerts still waiting out their for: duration. Discovery lists the configured alerting rules and the severities present when you give it a url.

The synthetic field is status on the alertmanager connector and state on prometheus_alerts. Keep that in mind when you write triggers over both.

Set it up

1. Pick the connector. If you run Alertmanager and want its grouping, silencing and resolved semantics, use the alertmanager connector. If you want Tares to read Prometheus directly, use the prometheus_alerts connector, which needs a reachable Prometheus server with alerting rules configured.

2. For Alertmanager, go to Sources → Add source → Alertmanager alerts, name it and click Create. The source page shows a receiver block with the ingest URL filled in. You can also ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” Add the receiver to your alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. When Tares has auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key, added to the webhook_configs entry through its http_config. The source page shows the auth block for you when the instance requires a token.

4. Send a test batch, then check Recent events:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

5. For prometheus_alerts, go to Sources → Add source → Prometheus alerts, enter the url and any auth Prometheus needs, and click Discover. Optionally filter by severity and toggle include pending. The source looks like this:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

6. Open the source and check Recent events and Fields. Discover proposes only the labels that are always present, severity and state. Dynamic labels such as namespace and service exist only while an alert fires, so promote them from the Fields tab once real alerts have been ingested.

What you get

• Every alert becomes an event with the alert name as its type, its summary as text and its labels attached, so Tares can correlate it with your other sources.
• With alertmanager, you get alerts the moment Alertmanager routes them, already grouped and deduped, with silencing honored and resolved notifications when send_resolved is on.
• With prometheus_alerts, you get one event per alert and a resolved event when it clears, without running Alertmanager and without writing PromQL.
• Either way, the alerts feed the rest of your setup. The AI SRE guide shows how Tares correlates them and can dispatch them to an agent.

When something else fits

• Choose prometheus_alerts if you have no Alertmanager, or you want Prometheus to be the only system Tares talks to. Delivery follows the poll interval, 30 seconds by default, not real time.
• Choose alertmanager if you rely on Alertmanager’s grouping or silencing, or you need alerts delivered as they happen.
• If you only need raw metric values rather than fired alerts, a metrics connector fits better than either of these.

Related