How do I test PagerDuty incidents about crash loops before a real incident?

Tares lets you test PagerDuty crash loop incidents by posting a sample payload to your webhook source’s ingest URL and reading the result under Recent events and Fields. PagerDuty reaches Tares through the webhook connector and a vendor setup, and Tares has no connector of its own for PagerDuty today. Run the check before you create the PagerDuty subscription, so the label mapping is already proven when a real incident arrives.

How it works

• PagerDuty sends incidents as v3 webhook deliveries to the ingest URL of a Tares webhook source. The full payload is stored lossless.
• Labels map payload fields into the event. PagerDuty nests everything under event, so labels use one-level dotted fields such as event.event_type. Fields two levels deep, such as those inside event.data, stay in the stored payload but cannot be labels.
• The event type is a fixed value on the source, because event_type_field reads top-level fields only.
• The primary label is a constant, service set to crash-loops. Every incident from this source then lands on one entity you can correlate with your other sources. The PagerDuty event type is not a good primary label, because it has only a few values and would give you nothing useful to group on, so it stays a plain label.
• For a push source, test_source takes no payload and only returns the ingest endpoint. It does not check the label mapping. To check the mapping, post a sample payload yourself and read the source page.
• The signing options for the webhook connector do not fit PagerDuty. PagerDuty signs with a v1= prefixed header that hmac_sha256 does not accept, so leave signature_scheme unset.

Set it up

1. Create the source. In the console, go to Sources → Add source → webhook. Or ask a connected agent to create a webhook source and give you the ingest URL. Use this definition:

- name: pagerduty-crash-loops
  connector: webhook
  config:
    event_type: pagerduty_incident
    labels:
      - { name: service, const: crash-loops, primary: true }
      - { name: incident_event, field: event.event_type }
      - { name: resource_type, field: event.resource_type }

2. Copy the ingest endpoint from the source page. If you call test_source, expect only that endpoint back.
3. Post a sample that has the shape of a PagerDuty v3 delivery. Replace the empty data object with data copied from a real PagerDuty incident if you have one:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"event": {"id": "TEST1", "event_type": "incident.triggered", "resource_type": "incident", "occurred_at": "2024-01-01T00:00:00Z", "data": {}}}'

If Tares has auth on, send an ingest-scoped API key in an Authorization Bearer header with the request. That header is the credential. 4. Open the source page. Under Recent events, confirm the event arrived. Check that the service label reads crash-loops and that incident_event reads incident.triggered. Fields shows which payload fields arrive and how often. 5. Post the same payload again with incident.resolved as the event type. Confirm incident_event changes while service stays the same. 6. Connect PagerDuty. Create a v3 webhook subscription, in the console or with the Webhook Subscriptions API, whose delivery URL is the Tares ingest URL. When Tares has auth on, add a custom header Authorization with the Bearer scheme and your ingest-scoped API key to the delivery method. Choose incident.triggered and incident.resolved, then pick the scope: account, team or service.

For the general webhook options, see the Inbound webhook connector. For how sources like this feed investigations, start from AI SRE.

What you get

• You know the ingest URL accepts your payload shape, and that the credential works, before an incident fires.
• You see exactly which labels each incident carries. Triggered and resolved incidents share the crash-loops entity and differ in incident_event.
• You can spot unmapped fields early. Anything under event.data that you want as a label has to be reached some other way, because only one-level dotted fields work.
• When a real crash loop incident arrives, it appears in Recent events on the same source, mapped the way you tested.

When something else fits

• If you already route alerts through Alertmanager, the Alertmanager connector fits better. It creates one event per alert and has its own ingest URL and verify step.
• If you need the incident title or service name as a label, flatten it before delivery. The webhook source only reads one-level dotted fields.

Related