How do I test Prometheus alerts about a stopped cron job before a real incident?

Tares lets you test a stopped-cron alert by creating a prometheus_alerts source, letting a test alert fire, and checking Recent events and Fields on the source page. To rehearse the label mapping with a sample payload, post it to the ingest URL of an Alertmanager or webhook source. You see the event, its labels and its status on the timeline before a real cron job fails.

How it works

The prometheus_alerts connector polls Prometheus at /api/v1/alerts every 30 seconds by default. It reads the alerts your alerting rules have already fired, so Prometheus decides what counts as an alert and Tares records and correlates it.

• Each active alert becomes one event. event_type is the alert name, text is its summary, and the alert labels become Tares labels plus a synthetic state of firing, pending or resolved.
• State is tracked across polls. An alert fires once, re-emits only when its state changes, and produces a resolved event when it clears.
• severity and state are always present, so Tares proposes them for you. Dynamic labels such as namespace or service exist only while the alert is firing, so you add them from the Fields tab after a real alert has been ingested.

A polled source has no ingest URL, so a sample payload goes to a push source instead. The Alertmanager connector turns each alert in a posted batch into one event. The webhook connector stores any JSON you post. For a push source, test_source only returns the ingest endpoint, so it does not check your label mapping. Post a payload and read the page instead.

Set it up

1. Create the source. Open Sources, Add source, Prometheus alerts, enter the Prometheus url, and click Discover to list your alerting rules and severities. You can also ask a connected agent to ingest the firing alerts from your Prometheus. Turn on include pending so you see the cron alert while it waits out its for: duration. In YAML the source looks like this, and you can add include_pending: true under config:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

2. Make the stopped-cron alert fire in a test setup. Use the alerting rule you plan to rely on, pointed at a test cron job that you let stop. Make sure the severity is one your severities filter lets through, or leave it blank.
3. Open the source and read Recent events. You should see one event named after your rule, first as pending if you enabled it, then as firing.
4. Open Fields and check which labels the alert carries. Promote the ones you want to correlate on, such as namespace or service, and mark one primary.
5. Let the alert clear, for example by running the cron job again. Confirm that a resolved event appears.
6. Rehearse the label mapping with a sample payload. Create an Alertmanager source from Sources, Add source, Alertmanager alerts, copy its ingest URL, and post a test batch. The example below uses an alert called PostgresDown. Replace the alertname, the labels and the summary with the ones your cron alert uses.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

7. Read Recent events and Fields on that source. The event shows the alert name as its type, the summary as its text, and the labels plus a status of firing or resolved. If your instance runs with tares up --auth, the sender must also present an ingest-scoped API key.

The Alertmanager source uses status where the polled source uses state. Treat the rehearsal as a check on label names and event shape, then confirm the final mapping on the prometheus_alerts source itself.

What you get

• Proof that a stopped-cron alert reaches Tares as an event with the alert name, summary and labels you expect.
• A confirmed set of labels, with one primary, so the alert correlates with your other sources by service or namespace.
• Confirmation that the pending, firing and resolved transitions each show up once, not once per poll.
• A rehearsed payload shape you can reuse when you wire up Alertmanager delivery. Start from the AI SRE guide to decide what Tares should do once the alert lands.

When something else fits

• Use the Alertmanager connector instead when you want Alertmanager’s grouping, silencing and resolved handling with real-time delivery, since you can also test it directly with a posted batch.
• Use the webhook connector when the cron job itself should report in, posting JSON to an ingest URL on every run.
• Stay with the Prometheus alerts connector when you have no Alertmanager and want Prometheus to stay the one deciding what is an alert.

Related