How do I test PagerDuty incidents about 5xx errors on an API before a real incident?

Tares lets you test PagerDuty incidents about 5xx errors on your API by posting a sample payload to a webhook source’s ingest URL and checking Recent events and Fields. PagerDuty reaches Tares through the webhook connector and a vendor setup, and Tares has no connector of its own for PagerDuty today. You see how your labels map before PagerDuty sends anything real. The check takes one curl command.

How it works

• A PagerDuty v3 webhook subscription POSTs JSON to the Tares ingest URL of a webhook source. The payload is stored lossless, so the full incident stays on the event.
• Labels map payload fields into the event. PagerDuty nests its fields under event, so use one-level dotted fields such as event.event_type.
• The source uses a fixed event_type, because event_type_field reads top-level fields only.
• The 5xx detail sits deeper in the payload, so it stays in the stored payload and is not a label. A service label with a constant value names your API and is the primary label.
• Do not set signature_scheme on this source. PagerDuty signs with a v1= prefixed header that hmac_sha256 does not accept.
• test_source on a push source takes no payload and only returns the ingest endpoint, so it does not check the label mapping. To check the mapping, post a sample payload yourself and read Recent events and Fields on the source page.

Set it up

1. Start Tares locally if it is not running yet.

tares up

2. Create a webhook source under Sources → Add source → webhook, or ask a connected agent to create it. Declare these labels. The service label is a constant that names your API and is the primary one. The incident_event label reads the PagerDuty event name and is not primary.

- name: pagerduty-api
  connector: webhook
  config:
    event_type: pagerduty_incident
    labels:
      - { name: service, const: api, primary: true }
      - { name: incident_event, field: event.event_type }
      - { name: resource_type, field: event.resource_type }

3. Copy the ingest endpoint from the source page and post a sample triggered incident. The payload follows the PagerDuty v3 shape, and the data text is your own sample about 5xx errors.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"event":{"id":"TEST1","event_type":"incident.triggered","resource_type":"incident","occurred_at":"2026-10-20T10:00:00Z","data":{"title":"5xx errors on the API"}}}'

4. On a secured instance (tares up --auth), send an ingest-scoped API key as an Authorization Bearer header with the same request. Without auth on, no header is needed.
5. Open the source page. In Recent events, confirm the event arrived with service set to api and incident_event set to incident.triggered. Open Fields to see which payload fields arrive and how often.
6. Post the same payload again with incident.resolved as the event_type. Recent events now shows a second event whose incident_event label has the new value.
7. When the mapping looks right, wire up PagerDuty:
   • Create a v3 webhook subscription, in the console or with the Webhook Subscriptions API, whose delivery URL is the Tares ingest URL.
   • When Tares has auth on, add a custom header Authorization: Bearer followed by your ingest-scoped API key to the delivery method.
   • Choose the events, such as incident.triggered and incident.resolved, and the scope: account, team or service.

What you get

• Proof that your ingest URL accepts the PagerDuty payload shape and, on a secured instance, your ingest key.
• A checked label mapping: you see service, incident_event and resource_type on real stored events, and Fields shows which payload fields arrive.
• Triggered and resolved samples that you can tell apart by their incident_event value.
• A source that is ready for the live subscription, so the first real 5xx incident lands in a source you have already exercised.

For how Tares uses events from your sources once they arrive, see AI SRE.

When something else fits

• If your 5xx alerts reach you through Alertmanager rather than PagerDuty, the alertmanager connector turns each alert in a batch into its own event and has its own test curl.
• If you only need to confirm that PagerDuty can reach Tares at all, one sample post to the ingest URL is enough. Skip the extra labels until you know which fields you want to correlate on.

Related