How do I test PagerDuty incidents about a disk filling up before a real incident?

Tares lets you test PagerDuty incident handling by posting a sample PagerDuty payload to the ingest URL of a webhook source and checking Recent events and Fields on the source page. PagerDuty reaches Tares through the webhook connector and a vendor setup, so you can run this before you create the PagerDuty subscription. When the event shows the labels you expect, the real incident.triggered delivery for a disk alert lands the same way.

How it works

PagerDuty reaches Tares through the webhook connector. Tares has no connector of its own for PagerDuty today, so you create a webhook source and point a PagerDuty v3 webhook subscription at its ingest URL.

• The webhook source stores each payload lossless and maps fields into labels that you declare.
• A PagerDuty delivery wraps everything in an event object, with id, event_type, resource_type, occurred_at and data inside it. Labels read one-level dotted fields such as event.event_type.
• event_type_field reads top-level fields only, so you set a fixed event_type on the source instead.
• The primary label is the entity key that incidents correlate on. Give it a constant service value. Do not make event.event_type the primary label: it has only a few values, such as incident.triggered and incident.resolved, so every incident would share one entity. Keep it as a non-primary label.
• test_source on a push source takes no payload and only returns the ingest endpoint, so it does not check your label mapping. A posted sample payload does.

The Inbound webhook connector page covers the source options in full. For the wider picture of how Tares watches your systems, see AI SRE.

Set it up

1. Create the webhook source. In the console choose Sources, Add source, webhook, or ask a connected agent to create a webhook source and give you the ingest URL. Use this definition:

- name: pagerduty-incidents
  connector: webhook
  config:
    event_type: pagerduty_incident
    labels:
      - { name: service, const: disk-alerts, primary: true }
      - { name: incident_event, field: event.event_type }
      - { name: resource_type, field: event.resource_type }

2. Copy the ingest endpoint from the source page. Do not set signature_scheme on this source. PagerDuty signs with a v1= prefixed header that hmac_sha256 does not accept, so the credential is the Bearer header.

3. Post a sample disk incident to the ingest URL. The id names the scenario so you can spot it in the list:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"event": {"id": "TEST-disk-filling-up", "event_type": "incident.triggered", "resource_type": "incident", "occurred_at": "2024-01-01T00:00:00Z", "data": {}}}'

4. If Tares has auth on, add an Authorization: Bearer header holding an ingest-scoped API key to the request. If auth is off, the call above works as written.

5. Open the source page. Check Recent events for the new event, and check Fields to see which payload fields arrived and how often. Confirm that service shows disk-alerts, incident_event shows incident.triggered and resource_type shows incident.

6. Post a second sample with event_type set to incident.resolved and confirm that incident_event changes. This tells you the resolve path maps correctly.

7. Create the real subscription. In PagerDuty, create a v3 webhook subscription, in the console or with the Webhook Subscriptions API, whose delivery URL is the Tares ingest URL. When Tares has auth on, add a custom header Authorization: Bearer with the ingest-scoped API key to the delivery method. Choose the events, such as incident.triggered and incident.resolved, and the scope: account, team or service.

What you get

• A checked label mapping before any real incident fires, so the first disk alert is not the first time you see how it lands.
• Proof that the ingest URL and the auth header work, because the sample arrives in Recent events.
• A primary label that groups incidents under one meaningful service key, with the PagerDuty event type kept as a separate label you can filter on.
• Lossless payloads, so fields you did not map are still stored and show up under Fields.
• A test you can repeat after any change to the source or the subscription by posting the same sample again.

When something else fits

• If your disk alerts start in Alertmanager rather than PagerDuty, use the Alertmanager connector. It makes one event per alert and has its own verify step.
• If you want to see how real PagerDuty deliveries are shaped, send a live test event from the subscription and read Fields, then adjust your labels to match.

Related