How do I wake an AI SRE once when PagerDuty incidents report high latency?

Tares wakes your AI SRE once per entity when you receive PagerDuty incidents through a webhook source and add a trigger with a cooldown. A repeating high latency alert fires one investigation, and the cooldown holds back further firings for the same entity until it ends. Resolved events still arrive at the source and are stored there.

How it works

PagerDuty reaches Tares through the webhook connector and a vendor setup. Tares has no connector of its own for PagerDuty today. A v3 webhook subscription in PagerDuty POSTs each incident event to the ingest URL of a Tares webhook source, and Tares stores the payload lossless.

A trigger watches that source and checks its condition as events arrive. When the condition holds, Tares POSTs one dispatch to each subscriber and starts the agents wired to the trigger in the project. The dispatch carries the entity’s correlated timeline over the context_window.

The cooldown is the minimum gap between firings for the same entity. An entity is the value of the trigger’s key label, and group_by: [key_value] evaluates the condition per entity. While the cooldown runs, more incident.triggered events for that entity do not start another investigation.

These rules shape the setup:

• Labels on a PagerDuty source use one-level dotted fields such as event.event_type.
• Set a fixed event_type, because event_type_field reads top-level fields only.
• Do not set signature_scheme. PagerDuty signs with a v1= prefixed header that hmac_sha256 does not accept.
• When Tares has auth on, the credential is an Authorization Bearer header holding an ingest-scoped API key.

Set it up

1. In PagerDuty, create a v3 webhook subscription, in the console or with the Webhook Subscriptions API. Set its delivery URL to the Tares ingest URL. Choose the events incident.triggered and incident.resolved, and scope the subscription to the service that reports high latency, so only that service’s incidents arrive.
2. When Tares has auth on, add a custom header to the delivery method. It is an Authorization header with the scheme Bearer and an ingest-scoped API key as the value.
3. Add the webhook source under Sources → Add source → webhook, or in the catalog YAML. The label named incident_event reads the PagerDuty event type. The constant primary label names the entity the cooldown applies to:

- name: pagerduty-incidents
  connector: webhook
  config:
    event_type: pagerduty_event
    labels:
      - { name: service, const: checkout-latency, primary: true }
      - { name: incident_event, field: event.event_type }
      - { name: resource, field: event.resource_type }

4. Add the trigger on the project’s page, under Triggers → Add trigger, or in the catalog YAML. The filter keeps only new incidents, the condition counts them per entity, and the cooldown holds back repeats:

triggers:
  - name: pagerduty_high_latency
    sources: [pagerduty-incidents]
    key_field: service
    filters:
      - { field: incident_event, op: eq, value: incident.triggered }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: high_latency
      context_window: 30m
    cooldown: 30m

5. Turn on the agent in the project that wires it to this trigger. An external agent subscribes over MCP with the subscribe tool instead.
6. Post a sample event and check Recent events and Fields on the source page:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"event": {"event_type": "incident.triggered", "resource_type": "incident"}}'

Pick a cooldown that matches how long one investigation usually takes. The 30 minutes above is an example value, so change it to suit your incidents.

What you get

• One investigation per entity instead of one per repeated alert. The first incident.triggered event fires the trigger and the cooldown covers the repeats.
• A dispatch with the entity’s correlated timeline over context_window, so the agent starts with context and makes no follow-up call.
• Resolved events on the source. Because the webhook stores payloads lossless, incident.resolved events are kept as events and you can read them under Recent events. The filter in the example decides what counts toward the condition and what is handed over, so it keeps resolved events out of the firing condition and out of the dispatch. To change that, adjust the filters list.
• A record of every firing. Firings and per-subscriber delivery status show in the console under Firings, on the trigger’s page and in the project’s Activity, even with zero subscribers.
• Safe delivery handling. Dispatches are at-least-once, so dedupe on dispatch_id. Acknowledge with a 2xx, because 5xx responses are retried with backoff.

When something else fits

• If you want a timer instead of a condition, give the trigger a schedule such as condition: {every: 10m}. It fires once per interval and hands over a summary of the window, which suits a watch-then-escalate setup.
• If your latency data comes from metrics, aggregate a numeric field, for example max(p99_ms) > 1000, rather than counting PagerDuty incidents.

Related