What should an AI SRE read next to PagerDuty incidents for high latency?

With Tares, an AI SRE should read the PagerDuty incident next to the same service’s latency metrics, request logs and recent deploys on one timeline. Tares reaches PagerDuty through the webhook connector and a vendor setup, and a shared service label joins everything. The agent then names a cause and cites the evidence lines behind it.

How it works

Tares has no connector of its own for PagerDuty today. PagerDuty reaches Tares through the webhook connector: you create a v3 webhook subscription in PagerDuty that delivers to a Tares ingest URL. Tares stores each payload lossless. The payload has an event object with an id, an event_type such as incident.triggered, a resource_type and an occurred_at time, plus agent, client and data.

The alert is only the start of the diagnosis. These are the pieces that make it useful:

• The PagerDuty source carries a const service label marked primary. That label is what joins the incident to that service’s timeline.
• Metrics sources supply the latency numbers, for example p99, labeled with the same service.
• Log sources supply the service’s request logs, labeled with the same service.
• A deploy source records each deploy as an event on the service, so “deploy landed, latency rose two minutes later” is visible in order.

Because every source carries the same service label, Tares merges them into one timeline per service on one clock. A trigger over that service can wake an agent, or you can ask a connected MCP client to read the service on demand. The agent answers from the incident, the metrics, the logs and the deploys together, and writes its conclusion back to the timeline for the next investigation. The AI SRE guide builds this loop end to end.

Set it up

1. Create a webhook source for PagerDuty. Use a const service label marked primary for the service you are watching, and an event_type label read from event.event_type. Labels use one-level dotted fields.

- name: pagerduty
  connector: webhook
  config:
    labels:
      - { name: service, const: checkout, primary: true }
      - { name: event_type, field: event.event_type }

2. Copy the ingest endpoint from the source page. In PagerDuty, create a v3 webhook subscription, in the console or with the Webhook Subscriptions API, whose delivery URL is that Tares ingest URL.
3. Choose the events, such as incident.triggered and incident.resolved, and the scope: account, team or service. Scope it to the service the const label names.
4. When Tares has auth on, add a custom header Authorization with the value Bearer followed by an ingest-scoped API key to the delivery method. That header is the credential. Do not set signature_scheme on this source, because PagerDuty signs with a v1= prefixed header that hmac_sha256 does not accept.
5. Check the mapping by posting a sample payload, then look at Recent events and Fields on the source page.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"event": {"event_type": "incident.triggered", "resource_type": "incident"}}'

6. Key your metrics and log sources by the same service label. Use the Prometheus connector for latency metrics, reading service from the metric labels, and Docker logs or OTLP for request logs.
7. Add the signal that explains most latency regressions, a change. Create a second webhook source for deploys with a primary service label read from the payload’s service field.

- name: deploys
  connector: webhook
  config:
    event_type: deploy
    text_template: 'deploy {commit} by {author}: {message}'
    labels:
      - { name: service, field: service, primary: true }

8. Add one line to your CD pipeline so every deploy posts to that source.

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

9. Start the MCP endpoint with tares mcp, connect your client, and ask: “Use tares: what happened to checkout in the last 15 minutes?”

What you get

• One timeline per service where the PagerDuty incident sits beside latency metrics, request logs and recent deploys.
• A diagnosis that cites specific evidence lines instead of repeating the alert text.
• Deploy timing next to the first latency rise, so a change shows up as a likely cause.
• Findings written back to the timeline, so the next investigation starts from the previous conclusion.
• Resolved events from PagerDuty on the same timeline, so you can see when the incident closed relative to the evidence.
• The choice of who gets woken: a Tares agent, your own agent service over a webhook, or a Slack channel.

To react to the incident automatically, add a trigger over the sources for that service. The AI SRE guide shows how to threshold typed fields directly, for example max(p99_ms) > 1000 for latency.

When something else fits

• If Prometheus rules already own your alerting, you can ingest those alerts directly instead of routing them through PagerDuty. The AI SRE guide covers that path.
• If you only need a one-off look at a single service, connect an MCP client and read the service on demand. You do not need a trigger.
• If a service has no metrics, logs or deploy events labeled with its service name, the incident arrives alone. Add those sources first, because the evidence is what turns the alert into a diagnosis.

Related