How do I wake an AI SRE once when PagerDuty incidents report 5xx errors on an API?

Tares wakes your AI SRE once per API when you send PagerDuty incidents to a webhook source and add a trigger with a cooldown per entity. The first incident.triggered event fires an investigation, and repeats for the same API stay quiet until the cooldown ends. Resolved events are still ingested and stored. This is part of the AI SRE workflow.

How it works

PagerDuty reaches Tares through the webhook connector and a PagerDuty v3 webhook subscription. Tares has no connector of its own for PagerDuty today.

• PagerDuty posts each incident event to the source’s ingest URL. Tares stores the payload lossless and maps the fields you declare as labels.
• The payload nests everything under event. Labels use one-level dotted fields such as event.event_type, and the source uses a fixed event_type, because event_type_field reads top-level fields only.
• A constant label names the API, so every event from the source belongs to one entity. Scope the PagerDuty subscription to the service that reports the 5xx errors, and use one source per API.
• A trigger watches the source and evaluates its condition per entity with group_by: [key_value]. The cooldown is the minimum gap between firings for the same entity, so a repeating alert starts one investigation.
• A filter on the incident event label counts only incident.triggered. Events such as incident.resolved are ingested and stored in the source like any other event, and they do not fire the trigger.

Set no signature scheme on this source. PagerDuty signs with a v1= prefixed header that hmac_sha256 does not accept, so the Authorization header is the credential.

Set it up

1. Create the webhook source. Add it under Sources → Add source → webhook, or ask a connected agent to create it, then copy the ingest endpoint from the source page.

- name: pagerduty-api
  connector: webhook
  config:
    event_type: pagerduty_incident
    labels:
      - { name: service, const: checkout-api, primary: true }
      - { name: incident_event, field: event.event_type }

2. In PagerDuty, create a v3 webhook subscription, in the console or with the Webhook Subscriptions API. Set its delivery URL to the Tares ingest URL.
3. Choose the events incident.triggered and incident.resolved, and set the scope to the service for that API (account and team scopes also exist).
4. If Tares has auth on, add a custom header to the delivery method: an Authorization header with the Bearer scheme, holding an ingest-scoped API key. On an instance without auth the header is not needed.
5. Post a test event to check the mapping, then read Recent events and Fields on the source page.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"event": {"id": "test-1", "event_type": "incident.triggered", "resource_type": "incident"}}'

6. Add the trigger on the project’s page, under Triggers → Add trigger, or in the catalog YAML. Subscribe your agent to it, or turn it on in a project that wires it to a Tares agent.

triggers:
  - name: api_5xx_incident
    sources: [pagerduty-api]
    key_field: service
    filters:
      - { field: incident_event, op: eq, value: incident.triggered }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: api_5xx_incident
      context_window: 15m
    cooldown: 30m

Pick a cooldown that matches how long PagerDuty keeps re-notifying. While it runs, further triggered events for the same API do not fire again.

What you get

• One dispatch per API per cooldown period. Tares POSTs a JSON body to each subscriber, with the entity’s correlated timeline over context_window as the payload, so the agent starts with context and makes no follow-up call.
• Quiet repeats. A flapping PagerDuty alert for the same API does not start a second investigation inside the cooldown.
• Resolved events stored. Each incident.resolved event is ingested and appears in Recent events on the source, so you can see when the incident closed.
• A full record. Every firing is logged in the console under Firings, on the trigger’s page and on the project’s Activity, even with zero subscribers. Delivery is at-least-once, so dedupe on dispatch_id.
• Independent APIs. With one source per API and group_by: [key_value], each API fires on its own.

When something else fits

• If you want every event, resolved ones included, to count toward firing, remove the filter. Resolved events then fire the trigger too, so keep the cooldown.
• If your 5xx signal comes from metrics rather than PagerDuty, a trigger on a numeric field such as rate_5xx with max and a predicate fits better, and the same cooldown applies.
• If you want a scheduled summary instead of a condition, give the trigger a schedule with every.

Related