How do I test Alertmanager alert groups about a cron job that stopped before a real incident?

Tares lets you test Alertmanager alert groups before a real incident by posting a sample batch to the source’s ingest URL and reading the result under Recent events. Create an Alertmanager source, send the batch with curl, then check the event and the labels it carries. The same ingest URL goes into your Alertmanager receiver once the test looks right.

How it works

The alertmanager connector is a push receiver. Alertmanager POSTs a batch shaped like { alerts: [...], ... } to the source’s ingest endpoint, and Tares turns each entry in the alerts array into its own event:

• event_type is the alert name.
• text is the alert’s summary.
• The alert’s labels become Tares labels, plus a synthetic status label that is firing or resolved.
• The full alert, including fingerprint, stays in the payload.

The entity key comes from a label automatically, in this order: namespace, service, job, instance. A label you mark primary takes precedence. If your cron alerts carry a different label that identifies the job, set the advanced key_label field to it.

For a push source, test_source takes no payload and only returns the ingest endpoint. It does not check your label mapping. To check the mapping, post a sample payload yourself and read Recent events and Fields on the source page. The webhook connector works the same way if you would rather send cron results from the job itself.

Set it up

1. Create the source. Ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” Or open Sources → Add source → Alertmanager alerts, name it and select Create. Either way you get an ingest URL of the form /ingest/<ingest_key>.
2. Post a sample batch to that URL. This is the example from the Alertmanager connector page:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

3. Make it describe your cron job. Replace the alertname value, the summary text and the labels with the ones your cron alert rule produces. Use the same label names your real alert uses, so the test exercises the mapping you will rely on. Send a second batch with status set to resolved to see how a resolved alert arrives.
4. Open the source page and check Recent events. Confirm the event_type matches your alert name, the text shows the summary, and the labels and status are what you expect.
5. Open Fields. It shows the alert labels, so you can promote a label such as namespace or service to correlate on.
6. When the mapping looks right, add the receiver to your alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

7. If your instance runs with auth on (tares up --auth), the sender uses a Bearer Authorization header holding an ingest-scoped API key. The source page shows the auth block in its copy-paste receiver. Without auth, no key is needed.

For the wider picture of how sources feed investigations, see AI SRE.

What you get

• A way to check your Alertmanager wiring and label mapping before a cron job actually stops.
• Events under Recent events that show the alert name, summary, labels and status for the batch you sent.
• A Fields view of the alert labels, so you choose the key to correlate on from labels you have seen arrive.
• A tested ingest URL and receiver block you can reuse in alertmanager.yml with send_resolved: true, so firing and resolved alerts both reach Tares.

When something else fits

• If the cron job reports its own result and you do not route it through Alertmanager, post JSON straight to a source with the webhook connector, where you declare the labels yourself.
• If you want Prometheus alerts without Alertmanager’s grouping and silencing, the prometheus_alerts connector polls them instead.

Related