How do I test alerts from Prometheus about failed deploys before a real incident?

Tares lets you check your Prometheus deploy alerts before an incident by creating a prometheus_alerts source, running Discover, and reading Recent events and Fields. Discover lists your alerting rules and severities, so you can confirm your failed-deploy rule is visible. Recent events shows each active alert as an event, and Fields shows the labels you can map. If you route alerts through Alertmanager instead, a separate Alertmanager source takes a test batch with curl.

How it works

The prometheus_alerts connector polls Prometheus’s own alerts endpoint, /api/v1/alerts, every 30 seconds by default. It reads the alerts your alerting rules have already fired, so it needs no Alertmanager and no PromQL. Prometheus decides what counts as an alert, and Tares records and correlates it.

• Each active alert becomes one event. The event_type is the alert name, the text is its summary, and its labels become Tares labels plus a synthetic state of firing, pending or resolved.
• State is tracked across polls. An alert fires once, re-emits only on a state change, and emits a resolved event when it clears.
• Discover proposes only the labels guaranteed to exist, severity and state. Per-alert labels such as namespace or service exist only while an alert is firing, so you add them from the Fields tab once an alert has been ingested.

Because this source polls, it has no ingest URL of its own. A test batch posted to an ingest URL belongs to a push source, such as the Alertmanager connector or the webhook connector. The Prometheus alerts connector page covers the polled path in full.

Set it up

1. Create the source. In the console, go to Sources → Add source → Prometheus alerts. Or ask a connected agent: “Use tares to ingest the firing alerts from the Prometheus at localhost:9090.” Enter the url and any auth, such as bearer_token or username and password.
2. Click Discover. It lists the configured alerting rules and the severities present. Confirm that the rule that watches your failed deploys is in the list.
3. Set the filters. Leave severity blank to ingest every severity, or list the ones you want, such as critical,warning. If your deploy alert uses a severity that is not in the list, it is not ingested. Turn on include pending to also ingest alerts still waiting out their for: duration.
4. Save the source with the labels Discover proposes:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

You can also POST /api/sources with the same object. 5. Open the source and check Recent events. You should see one event per active alert, named after your deploy alert, with state set to firing or pending. Resolved alerts appear as resolved events as they clear. 6. Open Fields and look at the labels the deploy alert carries. Promote namespace or service so Tares can correlate the alert with your other sources.

If Alertmanager delivers your deploy alerts, create an Alertmanager source instead. Add the receiver block it shows to alertmanager.yml, then send a test batch to its ingest URL and check Recent events on that source:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

The event shows event_type as the alert name, the summary as text, and a status label. Fields lists the alert labels, so you can promote namespace or service.

What you get

• You know before an incident that your failed-deploy rule is visible to Tares, because Discover lists it.
• You know which severities are ingested, because you set the filter yourself and see the result in Recent events.
• You see the exact event Tares builds from an alert: the alert name as the event type, the summary as text, and the labels with state.
• You pick the labels to correlate on from real data in Fields, so a deploy alert lines up with the service it belongs to.
• If you use Alertmanager, you confirm its source records a batch as one event per alert before the first real page.

For how alerts then trigger agents and escalation, see AI SRE.

When something else fits

• Use the Alertmanager connector when you want Alertmanager’s grouping, silencing and resolved handling with real-time delivery. It also gives you an ingest URL for test batches.
• Use the webhook connector when the deploy failure comes from a CI system or your own service rather than a Prometheus rule. A connected agent can run test_source with a sample payload to check the label mapping before you point the producer at the ingest URL.

Related