How do I test PagerDuty incidents about error spikes before a real incident?

Tares lets you test PagerDuty incidents for error spikes by posting a sample payload to the ingest URL of a webhook source and reading Recent events. You see the event, its labels and the fields that arrive before PagerDuty sends anything real. Tares has no connector of its own for PagerDuty today, so the incidents reach Tares through the webhook connector and a PagerDuty v3 webhook subscription.

How it works

PagerDuty incidents reach Tares through the webhook connector. PagerDuty delivers a JSON body shaped like this, and Tares stores it lossless:

• event.id, event.event_type (for example incident.triggered), event.resource_type, event.occurred_at
• event.agent, event.client and event.data, which are nested objects

Because the payload is stored as it arrives, a test post and a real delivery look the same. That makes a sample post a faithful rehearsal. You declare labels to map payload fields into the event. For PagerDuty, labels use one-level dotted fields such as event.event_type. Set a fixed event_type on the source, because event_type_field reads top-level fields only and the PagerDuty type sits under event.

For a push source, test_source takes no payload and only returns the ingest endpoint, so it does not check the label mapping. To check the mapping, post a sample payload and read Recent events and Fields on the source page.

Authentication is a header. PagerDuty signs its deliveries with a v1= prefixed header that hmac_sha256 does not accept, so leave signature_scheme unset. When Tares has auth on, the credential is an Authorization Bearer header holding an ingest-scoped API key. When auth is off, no header is needed.

Set it up

1. Start Tares locally if it is not running, or open your Tares Cloud workspace.

   tares up

2. Create a webhook source for PagerDuty. In the console, go to Sources → Add source → webhook, or ask a connected agent to create it. Declare a label per field you want to correlate on and mark one primary:

   - name: pagerduty-incidents
     connector: webhook
     config:
       event_type: pagerduty_incident
       labels:
         - { name: resource_type, field: event.resource_type, primary: true }
         - { name: incident_event, field: event.event_type }

3. Copy the ingest endpoint from the source page. It ends in your ingest key.

4. Post a sample incident. This one mimics an incident.triggered delivery for an error spike:

   curl -X POST http://localhost:8787/ingest/<ingest_key> \
     -H 'Content-Type: application/json' \
     -d '{"event": {"id": "test-1", "event_type": "incident.triggered", "resource_type": "incident", "occurred_at": "2024-01-01T00:00:00Z"}}'

   On a secured instance (tares up --auth), add an Authorization Bearer header holding an ingest-scoped API key to the request.

5. Open the source page. Under Recent events, confirm the event arrived with the event type you set and the labels you declared. Under Fields, check which payload fields arrive and how often, then fix any label whose field path is wrong and post again.

6. When the mapping looks right, wire up PagerDuty. Create a v3 webhook subscription, in the console or with the Webhook Subscriptions API, whose delivery URL is the Tares ingest URL. Add the Bearer header to the delivery method if Tares has auth on. Choose events such as incident.triggered and incident.resolved. For the first run, scope the subscription to the one service that carries your error-spike alerts, not the whole account.

The AI SRE guide shows what to build on top of the events once they flow.

What you get

• A checked label mapping before a real page fires, so a wrong field path shows up in a test, not during an incident.
• A visible event in Recent events that confirms the ingest URL, the event type and, on a secured instance, the header all work.
• A Fields view that shows which PagerDuty fields arrive, so you pick a reliable primary label.
• The full PagerDuty body kept in the payload. Nested parts such as event.data stay available there even though they are deeper than a one-level label.
• A source you reuse unchanged for the real subscription, because the test and the live delivery use the same ingest URL.

When something else fits

• If your error-spike alerts come from Prometheus and you only want PagerDuty to page on them, send those alerts through the Alertmanager connector. It makes one event per alert and has its own ingest URL and Verify step.
• If you want to test only that the ingest URL is reachable, the plain curl post in step 4 is enough. You do not need a PagerDuty subscription until the mapping looks right.

Related