How do I test alerts from Prometheus about 5xx errors on an API before a real incident?

Tares lets you test 5xx alerts before a real incident by sending a sample alert batch to a source’s ingest URL and checking the label mapping and the event. Give the sample alert a service label such as api-server and set key_label to service on the source, so the event lands on the API’s own entity. The polled prometheus_alerts source has no ingest URL, so the test goes through an Alertmanager source that uses the same label values.

How it works

• The Prometheus alerts connector polls Prometheus and records alerts that its rules have already fired. It has no ingest URL, so you cannot post a sample to it.
• The Alertmanager connector is a push receiver. Every alert in a batch you POST to its ingest URL becomes one event. The event_type is the alert name, the text is the summary, and the alert labels become Tares labels plus a synthetic status of firing or resolved.
• Tares keys each event on a label. By default it tries namespace, then service, then job, then instance. A test alert that carries namespace would key on namespace and miss the API’s service. Setting key_label to service on the source keys it on service instead.
• The same value has to appear on every source that describes the API. If the sample alert says api-server, your Prometheus alerts source and any log source for the API use api-server too, so Tares correlates them on one entity.
• You can also check the label mapping with test_source and a sample payload. The webhook connector page describes this check, and you can run it through a connected agent.

Set it up

1. Create a test source. Ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” Or open Sources → Add source → Alertmanager alerts, name it and Create.
2. Set the key label to service, either in the advanced key_label field or in the source config:

- name: api-alerts
  connector: alertmanager
  config:
    key_label: service

3. Send a sample 5xx alert to the ingest URL. The service label carries the value your other sources use for the API. On a secured instance, send an ingest-scoped API key with the request.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"ApiHigh5xxRate","severity":"critical","namespace":"prod","service":"api-server"},"annotations":{"summary":"API 5xx error rate is high"},"startsAt":"2024-01-01T00:00:00Z"}]}'

4. Open the source and check Recent events. You see one event named ApiHigh5xxRate with the summary as its text and status set to firing. Open Fields to confirm the labels service, severity and namespace arrived. Confirm the event sits under the api-server entity, not under prod.
5. Send the same body with "status":"resolved" to check that the resolved event shows up too.
6. Make the polled source use the same value. Add a service label from labels.service on your Prometheus alerts source:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: service, field: labels.service, primary: true }
      - { name: state, field: state }

An alert’s dynamic labels only exist while it is firing. If service is not offered yet, add it from the Fields tab once real alerts have been ingested. 7. When the mapping looks right, point your real receiver at the same ingest URL and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

What you get

• You see how a 5xx alert maps to Tares labels before anything breaks: event_type, text, status and the labels you care about.
• The test event lands on the api-server entity, so you know a real alert will sit beside your other sources for that API.
• You confirm that firing and resolved events both arrive.
• You learn which fields exist from the Fields tab and can promote service or namespace to correlate on.
• Once your real alerts are flowing, the setup is part of your wider AI SRE workflow.

When something else fits

• If you want real alerts through Alertmanager’s grouping, silencing and resolved handling with real-time delivery, use the Alertmanager source for production too, not only for the test.
• If you want Tares to read Prometheus directly with no Alertmanager, keep the prometheus_alerts source and check it under Recent events once Prometheus fires an alert.
• If your alert sender is not Alertmanager, a webhook source takes any JSON, and test_source with a sample payload checks its label mapping first.

Related