What should PagerDuty incidents about error spikes include for an AI SRE?

Tares gets the most from a PagerDuty error-spike incident when it carries a service name, an event type and a start time, because those become labels you can line up with your metrics and logs. PagerDuty reaches Tares through the webhook connector and a vendor setup, and Tares has no connector of its own for PagerDuty today. Tares stores the full payload lossless, so everything else PagerDuty sends stays with the event.

How it works

PagerDuty posts a v3 webhook payload to a Tares ingest URL. A webhook source stores the payload as received and maps the fields you declare into labels. The payload has this shape:

• event.id, the event identifier.
• event.event_type, such as incident.triggered or incident.resolved.
• event.resource_type, such as incident.
• event.occurred_at, the time the event happened, which is your start time.
• event.agent, event.client and event.data, which carry the rest.

The AI SRE guide keys every source by one shared service label, so metrics, logs and alerts for a service sit on one timeline. Give the PagerDuty source the same service label and its events can sit on that timeline too.

Labels on a webhook source read one-level dotted fields such as event.event_type and event.occurred_at. They cannot reach deeper into event.data. Everything inside event.data is still kept in the lossless payload, and the source page’s Fields view shows which payload fields arrive and how often.

The PagerDuty payload nests everything under event, so event_type_field, which reads top-level fields only, does not apply. Set a fixed event_type instead. Do not set signature_scheme. PagerDuty signs with a v1= prefixed header that hmac_sha256 does not accept, so the Authorization Bearer header is the credential.

Set it up

1. In Tares, create a webhook source and copy its ingest URL. The webhook connector page covers the source form and the ingest endpoint.
2. Give the source a primary service label. PagerDuty lets you scope a subscription to the account, a team or a service. Scope the subscription to one service and set the label as a constant, so every incident from that subscription carries the service name you use on your other sources.
3. Add labels for the event type and the start time, using the one-level dotted fields event.event_type and event.occurred_at.
4. In PagerDuty, create a v3 webhook subscription, in the console or with the Webhook Subscriptions API. Set its delivery URL to the Tares ingest URL.
5. Choose the events, such as incident.triggered and incident.resolved, and the scope.
6. When Tares has auth on, add a custom header Authorization: Bearer <ingest-api-key> to the delivery method, with an ingest-scoped API key.

The source looks like this:

- name: pagerduty-checkout
  connector: webhook
  config:
    event_type: incident
    labels:
      - { name: service, const: checkout, primary: true }
      - { name: event_type, field: event.event_type }
      - { name: started, field: event.occurred_at }

To check the mapping before PagerDuty sends anything, post a sample with the same shape, then read Recent events and Fields on the source page:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"event": {"id": "test-1", "event_type": "incident.triggered", "resource_type": "incident", "occurred_at": "2024-01-01T00:00:00Z"}}'

What you get

• A service label that matches the label on your metrics and logs sources. The incident can then be read next to the 5xx rate and the error logs for the same service.
• A start time on every incident from event.occurred_at. You can compare it with when the error rate began to climb.
• An event type such as incident.triggered or incident.resolved, so a firing and its resolution are distinguishable on the timeline.
• The full PagerDuty payload, kept lossless, including event.data, event.agent and event.client.

The guide’s demo uses an error_spike scenario. A HighErrorRate alert lands in the same timeline as the error logs and the metric that tripped it, and your PagerDuty events can follow the same pattern.

When something else fits

• If your alerts already come from Prometheus rules, the guide’s prometheus_alerts or alertmanager sources keep the alert labels as they are. PagerDuty then adds the paging side on top.
• If you want the service name read from the payload instead of set as a constant, you need a field that sits one level into the payload. A constant label per service-scoped subscription is the route that works today.

Related