How do I route PagerDuty incidents to an AI SRE?

Tares receives PagerDuty incidents through the webhook connector and a PagerDuty v3 webhook subscription. A trigger then wakes your AI SRE agent with the correlated timeline. Tares has no connector of its own for PagerDuty today, so the webhook connector is the route.

How it works

PagerDuty reaches Tares through the webhook connector and a vendor setup on the PagerDuty side. The flow has three parts:

• Delivery. A PagerDuty v3 webhook subscription POSTs each incident event to the ingest URL of a Tares webhook source. The payload is stored lossless, with a top-level event object that holds event_type (for example incident.triggered), resource_type, occurred_at and data.
• Mapping. Labels on the source map payload fields into the event. Labels read one-level dotted fields such as event.event_type. The event_type_field option reads top-level fields only, so you set a fixed event_type on the source instead.
• Waking. A trigger watches the source. When it fires, it wakes the agent you chose, as described in Build an AI SRE: a Tares agent, your own agent service over a webhook, or a Slack channel.

Tares builds one timeline per service from sources that carry the same service label. The agent reads that correlated timeline when it wakes.

PagerDuty signs its deliveries with a v1= prefixed header, which the hmac_sha256 signature scheme does not accept. Leave signature_scheme unset. The credential is the Authorization Bearer header.

Set it up

1. Create the webhook source. In the console, go to Sources → Add source → webhook, or ask a connected agent to create it. Declare labels for the fields you want to correlate on and mark one primary.

   - name: pagerduty-incidents
     connector: webhook
     config:
       event_type: incident
       labels:
         - { name: kind, field: event.event_type, primary: true }
         - { name: resource, field: event.resource_type }

   Save it and copy the ingest endpoint from the source page.

2. Create the PagerDuty subscription. In PagerDuty, create a v3 webhook subscription, in the console or with the Webhook Subscriptions API. Set its delivery URL to the Tares ingest URL. Choose the events, such as incident.triggered and incident.resolved, and the scope: account, team or service. Subscribe only to incident.triggered if you want the agent woken for new incidents alone.

3. Add the credential if Tares has auth on. Add a custom header to the delivery method with the scheme Bearer and an ingest-scoped API key as the value. Creating a push source in the console mints such a key and shows it once. If Tares has no auth, skip this step.

4. Send a test event. Post a PagerDuty-shaped payload to the ingest URL and check Recent events. Fields shows which payload fields arrive.

   curl -X POST http://localhost:8787/ingest/<ingest_key> \
     -H 'Content-Type: application/json' \
     -d '{"event": {"event_type": "incident.triggered", "resource_type": "incident"}}'

5. Put the incident on a service timeline. If your payload carries a field that names the affected service, declare a label called service from it, the same label your metrics, logs and alerts sources use. Events that share the service label land on that service’s timeline.

6. Create the trigger. Watch the source and fire when an incident event arrives:

   - name: pagerduty-incident
     sources: [pagerduty-incidents]
     condition: { aggregate: count, predicate: "> 0", window: 5m }
     cooldown: 5m

7. Pick who gets woken. Subscribe a Tares agent to the trigger, point it at your own agent service over a webhook, or send it to a Slack channel.

What you get

• Every PagerDuty incident event you subscribe to arrives in Tares as a stored event, with the full payload kept.
• You can inspect arrivals under Recent events and Fields before you rely on the route.
• A trigger wakes your agent when an incident event lands. The agent reads the correlated timeline for the service, so it starts from evidence rather than from the page alone.
• Once the source carries the same service label as your other sources, the incident sits on that service’s timeline next to the metrics, logs and alerts that share the label.
• The agent’s finding is written back, so the next investigation starts from the previous conclusion.
• You choose the events and the scope in PagerDuty, so you control how much reaches Tares.

When something else fits

• If your alerts come from Prometheus or Alertmanager, use the prometheus_alerts or alertmanager connector instead. Tares ingests what those fire, and no webhook setup is needed on a vendor side.
• If you want the agent to threshold metrics itself, build the trigger on typed fields from a metrics source rather than on incident events. The AI SRE guide walks through both.

Related