Skip to Content

How do I test alerts from Prometheus about 5xx errors on an API before a real incident?

Tares lets you test 5xx alerts from Prometheus before a real incident by reading Recent events and Fields on the source page, and by posting a sample batch to an ingest URL. The prometheus_alerts connector polls Prometheus for the alerts its rules have fired, so you see the alert name, the labels and a state of firing, pending or resolved. For a push source, you post a sample payload to its ingest URL and read the same two tabs to check the label mapping. Start with the AI SRE guide if you want the wider picture.

How it works

  • The Prometheus alerts connector polls Prometheus’s own /api/v1/alerts every 30 seconds by default. Prometheus decides what counts as an alert. Tares records it and correlates it with your other sources.
  • Each alert becomes one event. The event_type is the alert name, the text is its summary, and the alert labels become Tares labels plus a synthetic state of firing, pending or resolved.
  • State is tracked across polls, so an alert fires once and emits a resolved event when it clears, not a copy on every tick.
  • Discover proposes only the labels guaranteed to be present, severity and state. Per-alert labels such as namespace and service exist only while an alert is firing, so you add them from the Fields tab once alerts have been ingested.
  • Push sources work differently. The Alertmanager and webhook sources take a POST at their ingest URL. For a push source, test_source takes no payload and only returns the ingest endpoint, so it does not check the label mapping. To check the mapping, post a sample payload and read Recent events and Fields.

Set it up

  1. In the console, go to Sources → Add source → Prometheus alerts. Enter the Prometheus url and any auth, then click Discover. It lists the configured alerting rules and the severities present, so you can confirm your 5xx rule is there.
  2. Filter by severity if you want, and turn on include pending to also ingest alerts still waiting out their for: duration. The source looks like this:
- name: alerts connector: prometheus_alerts poll: 30s config: url: http://localhost:9090 severities: critical,warning # blank = all labels: - { name: severity, field: labels.severity, primary: true } - { name: state, field: state }
  1. Open the source and read Recent events. You see one event per active alert, and resolved events as they clear. Check that the alert name and summary are the ones your 5xx rule produces.
  2. Open Fields to see the labels the alerts carry. Promote namespace or service to labels so the API’s alerts correlate with your other sources.
  3. To rehearse the push path without waiting for Prometheus, create an Alertmanager source with Sources → Add source → Alertmanager alerts. Its page shows a receiver block with the ingest URL filled in. Post a sample batch to that ingest URL. Replace the alert name and labels with the ones your 5xx rule uses:
curl -X POST http://localhost:8787/ingest/<ingest_key> \ -H 'Content-Type: application/json' \ -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'
  1. Read Recent events and Fields on that source page. Tares fans the alerts array out into one event each, with the alert name as the event type, the summary as the text and a status label of firing or resolved.
  2. On a secured instance, add an ingest-scoped API key to the webhook_configs entry of the receiver, as the Alertmanager page shows.

For a sender that is not Alertmanager, the same method works with a webhook source. Post a JSON sample to /ingest/<ingest_key>, then read Recent events and Fields to see which payload fields arrive and how often.

What you get

  • You see how a 5xx alert looks in Tares before an incident: its event type, text, severity and state.
  • You confirm which labels the alerts carry, and you promote the ones you correlate on, such as namespace or service.
  • You see firing and resolved events as separate events, so you know what a cleared alert looks like.
  • You get a repeatable rehearsal. A sample batch posted to an ingest URL lands as events you can read on the source page.

When something else fits

  • Use the Alertmanager source instead of polling when you want Alertmanager’s grouping, silencing and resolved handling, and real-time delivery.
  • Use the webhook source when the sender is another system that can POST JSON, and map the payload fields with labels.
  • If your Prometheus has no alerting rules yet, Discover lists none. Add the 5xx rule in Prometheus first.

Still stuck? Get help

Last updated on