What should an AI SRE read next to PagerDuty incidents for 5xx errors on an API?

Tares lets an AI SRE read the same service’s metrics, logs and recent deploys on one timeline next to the PagerDuty incident. PagerDuty reaches Tares through the webhook connector and a vendor setup. Each source carries the same service label, so the agent cites the 5xx rate, the error log lines and the deploy that landed just before, not only the alert.

How it works

A PagerDuty incident tells you that something broke and when. It does not tell you why. Tares correlates sources by a shared label, so you add the evidence an on-call engineer would open first, all keyed by the same service label:

• Metrics: the 5xx rate, p99 latency, database connections and dependency health, from a prometheus source.
• Logs: the API’s request and error logs, from a docker_logs source or an OTLP pipeline.
• Recent changes: a deploy event from your CD pipeline and, if you want, config changes, each posted to a webhook source.
• The incident itself: PagerDuty webhook deliveries, received by a webhook source.

Because every source carries service, Tares merges them into one timeline per service on one clock. The agent reads that timeline, so it can say “deploy landed at 14:02, 5xx rate climbed at 14:04, the logs show the same error” and point to each line.

PagerDuty reaches Tares through the webhook connector and a vendor setup. Tares has no connector of its own for PagerDuty today. PagerDuty posts a v3 payload with an event object holding event_type, resource_type, occurred_at and data. The webhook connector stores that payload lossless. The full pattern is in Build an AI SRE.

Set it up

1. Create the PagerDuty source in Tares. Add a webhook source scoped to your API. PagerDuty’s event_type sits inside the event object, and event_type_field reads top-level fields only. So set a fixed event_type and use one-level dotted fields for labels. A constant service label ties the incidents to the same timeline as your metrics and logs. Save it and copy the ingest URL from the source page.

- name: pagerduty-incidents
  connector: webhook
  config:
    event_type: pagerduty_incident
    labels:
      - { name: service, const: api-server, primary: true }
      - { name: pd_event, field: event.event_type }

2. Point PagerDuty at it. In PagerDuty, create a v3 webhook subscription, in the console or with the Webhook Subscriptions API, whose delivery URL is the Tares ingest URL. Choose events such as incident.triggered and incident.resolved, and set the scope to the account, a team or the API’s service.
3. Add the credential if Tares has auth on. Add a custom header Authorization: Bearer followed by an ingest-scoped API key to the delivery method. This header is the credential. Do not set signature_scheme on this source. PagerDuty signs with a v1= prefixed header that hmac_sha256 does not accept.
4. Add the metrics and log sources. Create a prometheus source for the 5xx rate and latency, and a docker_logs source or an OTLP pipeline for the API’s logs. Give each the same service label value as above.
5. Add the change source. Create a second webhook source with a primary service label read from the payload’s service field, event_type: deploy, and the text template deploy {commit} by {author}: {message}. Then add one line to your CD pipeline that runs on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

6. Check the mapping. Post a sample payload and read Recent events and Fields on each source page. See Inbound webhook for the options.

What you get

• One timeline for the API with the PagerDuty incident, the 5xx metrics, the error logs and the deploy events side by side.
• A diagnosis that cites evidence lines, such as the metric that tripped and the log lines at the same moment, so you can check it.
• A visible link between a change and the incident. Most real root causes are a change, and the deploy event puts it on the same clock as the first 5xx.
• A resolved event on the timeline when PagerDuty sends incident.resolved, so the next investigation sees how the last one ended.
• Findings the agent writes back to the timeline, so each investigation starts from the previous conclusion.

When something else fits

• If your alerts already come from Prometheus rules, the prometheus_alerts or alertmanager connector gives Tares the fired alerts directly. PagerDuty then adds on-call context, not the trigger.
• If you need a firing alert to wake an agent automatically, add a trigger over these sources. The AI SRE guide shows the wiring.
• If you only want a one-off look at an incident, a connected MCP client can read the same service timeline on demand without any trigger.

Related