How do I test PagerDuty incidents about slow database queries before a real incident?

Tares lets you test PagerDuty incident handling before a real incident by posting a sample payload to a webhook source’s ingest URL. You then read the label mapping and the event on the source page. PagerDuty reaches Tares through the webhook connector and a vendor setup, and Tares has no connector of its own for PagerDuty today. The test needs no PagerDuty account activity, only a curl call.

How it works

PagerDuty delivers v3 webhook subscriptions as JSON, and the webhook connector stores every payload lossless. The payload has an event object with id, event_type (for example incident.triggered), resource_type, occurred_at and data.

The webhook source maps fields into labels, and the label mapping is what you check in a test:

• Labels use one-level dotted fields such as event.event_type.
• The event type is fixed in the source config, because event_type_field reads top-level fields only and PagerDuty nests the type under event.
• The service label is a constant with the primary flag. It describes the database incidents this source carries.
• Do not set signature_scheme. PagerDuty signs with a v1= prefixed header that hmac_sha256 does not accept, so the credential is an Authorization Bearer header holding an ingest-scoped API key, needed only when Tares has auth on.

For a push source, test_source takes no payload and only returns the ingest endpoint, so it does not check the label mapping. Posting a sample payload does. See the webhook connector page for the full field list. If you route PagerDuty-style alerts through Alertmanager instead, the Alertmanager connector fans each alert out into its own event.

Set it up

1. Create a webhook source for the incidents. In the console, use Sources → Add source → webhook, or ask a connected agent to create it with create_source. This is the config:

- name: pagerduty-incidents
  connector: webhook
  config:
    event_type: pagerduty_incident
    labels:
      - { name: service, const: database, primary: true }
      - { name: incident_event, field: event.event_type }
      - { name: resource_type, field: event.resource_type }

2. Save the source and copy the ingest endpoint from the source page. Calling test_source here returns that endpoint, which is all it does for a push source.

3. Post a sample incident that has the same shape PagerDuty sends:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"event": {"id": "test-slow-queries-1", "event_type": "incident.triggered", "resource_type": "incident", "occurred_at": "2026-10-20T09:00:00Z", "data": {}}}'

On a secured instance (tares up --auth), send an ingest-scoped API key as Authorization: Bearer followed by the key.

4. Open the source page and read Recent events. Your sample appears as an event with the fixed event type and the labels you declared.

5. Open Fields. It shows which payload fields arrive and how often, so you can confirm that event.event_type and event.resource_type carry values.

6. Post a second sample with incident.resolved as the event_type and check that incident_event changes on the new event.

7. In PagerDuty, create a v3 webhook subscription whose delivery URL is the ingest URL. When Tares has auth on, add a custom header Authorization: Bearer followed by the ingest-scoped API key to the delivery method. Choose events such as incident.triggered and incident.resolved, and pick the scope: account, team or service. A service scope lets you limit deliveries to the service that owns your database.

What you get

• A tested webhook source with a label mapping you have seen on a real event before any incident fires.
• A check of which payload fields arrive, from Fields, so you can adjust labels without waiting for PagerDuty.
• The incident open and resolve events as separate entries in Recent events, each carrying the incident_event label.
• A clear auth setup: a Bearer header with an ingest-scoped API key when Tares has auth on.

This setup is part of the wider alerting workflow described in AI SRE.

When something else fits

• If your alerts come from Prometheus rules through Alertmanager, use the Alertmanager connector, which makes one event per alert and has its own ingest URL.
• If you want a label on a deeper field such as event.data, post a sample first and check Fields. Labels use one-level dotted fields, so a deeper value may need a different approach.

Related