How do I test alerts from Prometheus about slow database queries before a real incident?

Tares lets you rehearse your slow query alerts before an incident by polling your Prometheus rules and sending a sample alert batch. You then check the label mapping and the event in Recent events. The prometheus_alerts source is polled and has no ingest URL, so the sample payload goes to an Alertmanager or webhook source.

How it works

The prometheus_alerts source polls the /api/v1/alerts endpoint of your Prometheus server, every 30 seconds in the example below. It reads the alerts your alerting rules have already fired. Prometheus decides what counts as an alert. Tares records it and correlates it with your other sources, as covered in AI SRE.

Each active alert becomes one event:

• event_type is the alert name.
• text is the alert summary.
• The alert labels become Tares labels, plus a synthetic state label that is firing, pending or resolved.

State is tracked across polls. An alert fires once, re-emits only when its state changes, and produces a resolved event when it clears.

Because the source polls, you cannot post a sample payload to it. For a payload test you use a push source instead. An Alertmanager source turns each alert in a posted batch into one event. A webhook source stores any JSON you post and maps the fields you declare as labels.

The two sources label state differently. The Alertmanager source adds status, and prometheus_alerts adds state. A sample batch therefore proves the event shape and your label choices, and the polled source proves the real alert names from your rules.

Set it up

1. Create the polled source. Open Sources → Add source → Prometheus alerts, enter the Prometheus url and any auth, and click Discover. Discover lists your configured alerting rules and the severities present, so you can confirm your slow query rule is there. You can also ask a connected agent: “Use tares to ingest the firing alerts from the Prometheus at localhost:9090.” The resulting source looks like this:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

2. Choose which alerts to see. Set severity to match your slow query rule, or leave it blank for all. Turn on include pending to also ingest alerts that are still waiting out their for: duration. A slow query alert then shows up before it fires.

3. Send a sample batch. Create an Alertmanager source with Sources → Add source → Alertmanager alerts, or ask your agent: “Use tares to create an Alertmanager source and give me the webhook URL.” Then post a test batch to its ingest URL:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

Edit the alert name, labels and summary in the payload so they match your slow query rule. On a secured instance started with tares up --auth, the sender uses a Bearer Authorization header holding an ingest-scoped API key.

4. Test a custom shape with a webhook source. Ask your agent to run test_source with a sample payload, which checks the label mapping before you save anything. Then post the same payload to the webhook ingest URL.

5. Check the result in the source page:

   • Recent events shows one event per alert, with the alert name as the event type and the summary as the text.
   • Fields shows the labels the alerts carry.

6. Promote the labels you want to correlate on. Discover proposes only the labels guaranteed to exist, severity and state. Dynamic labels such as namespace or service exist only while an alert is firing, so add them from Fields once an alert has been ingested.

What you get

• You confirm that Tares sees your slow query rule before it matters, through Discover on the polled source.
• You see the event the alert produces, with its event type, summary and labels, in Recent events.
• You choose the primary label and the correlation labels from real field data in Fields, not from guesses.
• You see the pending, firing and resolved lifecycle of the alert, each state emitted once and not repeated on every poll.
• You learn whether your setup correlates the alert with your other sources, such as database logs or deploys, with time to adjust.

When something else fits

• If you want Alertmanager’s grouping, silencing and resolved handling with real-time delivery, use the Alertmanager source for the live path as well as for testing. Use Prometheus alerts when you want polling with no Alertmanager.
• If your producer is not Prometheus or Alertmanager, a webhook source accepts any JSON and lets you shape the event type, text and labels yourself.

Related