How do I wake an AI SRE once when PagerDuty incidents report crash loops?

Tares wakes your AI SRE from PagerDuty crash-loop incidents with a trigger whose cooldown sets the minimum gap between firings for the same entity. PagerDuty reaches Tares through the webhook connector and a vendor setup, because Tares has no connector of its own for PagerDuty today. Resolved events are stored on the source alongside the triggered ones. Start from AI SRE if you want the wider picture.

How it works

• PagerDuty delivers to a webhook source. A v3 webhook subscription posts each incident event to the Tares ingest URL. The webhook connector stores every payload lossless.
• Labels name the entity. A label maps a payload field into the event. PagerDuty nests its fields under event, so use one-level dotted fields such as event.event_type. Set a fixed event_type on the source, because event_type_field reads top-level fields only.
• The trigger picks the entity. A trigger watches the source. key_field names the entity, and by default it is the source’s primary label. With group_by: [key_value] the condition is evaluated per entity.
• The cooldown spaces the firings. cooldown is the minimum gap between firings for the same entity. It stops a flapping or repeating condition from firing again and again for that entity.
• Resolved events stay on the source. If you subscribe PagerDuty to incident.resolved as well as incident.triggered, both are stored. A trigger filter decides which events count toward the condition and which are handed over.

Set it up

1. In PagerDuty, create a v3 webhook subscription, in the console or with the Webhook Subscriptions API. Set its delivery URL to the Tares ingest URL.
2. Choose the events, such as incident.triggered and incident.resolved. Choose the scope: account, team or service. To keep this to crash loops, scope the subscription to the service that reports them.
3. If Tares has auth on, add a custom header to the delivery method. It is an Authorization Bearer header holding an ingest-scoped API key. Do not set signature_scheme. PagerDuty signs with a v1= prefixed header that hmac_sha256 does not accept, so the Bearer header is the credential.
4. Add a webhook source with a fixed event type and dotted labels. The label named incident_event is the one the trigger filters on.

- name: pagerduty_incidents
  connector: webhook
  config:
    event_type: pagerduty_incident
    labels:
      - { name: incident_event, field: event.event_type, primary: true }
      - { name: resource, field: event.resource_type }

5. Post a sample event and check Recent events and Fields on the source page.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"event": {"event_type": "incident.triggered", "resource_type": "incident"}}'

6. Add a trigger with a cooldown. The filter counts only triggered events, so a resolved event does not fire it. The emit block sets how much timeline the agent receives.

triggers:
  - name: crash_loop_incident
    sources: [pagerduty_incidents]
    key_field: incident_event
    filters:
      - { field: incident_event, op: eq, value: incident.triggered }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: crash_loop_incident
      context_window: 15m
    cooldown: 30m

7. Wire your agent to the trigger in its project, or register a webhook with the subscribe tool. Acknowledge each dispatch with a 2xx. Delivery is at-least-once, so dedupe on dispatch_id.

In this example the primary label is incident_event, so the entity is the event type and the cooldown spaces firings across all incidents of that type. To give each service its own cooldown, add a label for a one-level field that identifies the service and set key_field to it.

What you get

• A repeating PagerDuty alert for the same entity does not fire the trigger again until the cooldown has passed.
• Each firing is logged, even with zero subscribers. You see it under Firings, on the trigger’s page and in the project’s activity, with the exact payload that was sent.
• The dispatch carries the entity’s correlated timeline over context_window, so the agent has context without a follow-up call.
• Resolved incidents are stored on the source with the triggered ones, so the recorded events show when an incident cleared.

When something else fits

• If you want a periodic review of the source instead of a reaction to each incident, give the trigger a schedule such as condition: {every: 10m} rather than a cooldown.
• If your alerts already pass through Alertmanager, its connector makes one event per alert and may fit better than PagerDuty webhooks.

Related