What should an AI SRE read next to PagerDuty incidents for slow database queries?

Tares puts a PagerDuty incident on one timeline with the same service’s metrics, logs and recent deploys, so the AI SRE diagnoses slow queries from cited evidence. PagerDuty reaches Tares through the webhook connector and a vendor setup. Tares has no connector of its own for PagerDuty today. The agent reads the incident, then the database metrics, the service logs and the latest changes for the same service.

How it works

A PagerDuty incident only says that something is wrong. To explain slow queries, the agent needs the evidence around it, and Tares keeps that evidence together by service. Every source that carries the same service label lands on one timeline for that service, merged on one clock. The agent reads that single timeline instead of querying each system separately.

For slow database queries, put these four things on the timeline:

• The PagerDuty incident. A v3 webhook subscription in PagerDuty delivers events such as incident.triggered and incident.resolved to a Tares webhook source.
• Database metrics. The prometheus connector polls the metrics you pick by name or label, so you need no PromQL. Aggregate the metric value in triggers.
• Service logs. The docker_logs connector tails a container’s logs. An OpenTelemetry pipeline into the otlp connector covers services that are not in Docker.
• Recent changes. A second webhook source receives a deploy event from your CD pipeline. Most real root causes are a change, so this is the signal that turns “queries are slow” into “queries got slow two minutes after this deploy”.

The PagerDuty payload nests its data under event. Labels on the webhook source can use one-level dotted fields such as event.event_type, and event_type_field reads top-level fields only. Set a fixed event_type and give the source a constant service label, scoping the PagerDuty subscription to that service. The AI SRE guide shows the full loop: a trigger wakes an agent, the agent reads the timeline and writes its finding back for the next investigation.

Set it up

1. Create the PagerDuty source. Add a webhook source with a fixed event type and a constant service label. Copy its ingest URL from the source page. See the webhook connector for the full field list.

   - name: pagerduty
     connector: webhook
     config:
       event_type: incident
       labels:
         - { name: service, const: checkout, primary: true }
         - { name: pd_event, field: event.event_type }

2. Point PagerDuty at it. In PagerDuty, create a v3 webhook subscription, in the console or with the Webhook Subscriptions API, whose delivery URL is the Tares ingest URL. Choose events such as incident.triggered and incident.resolved, and scope the subscription to the service. When Tares has auth on, add a custom header to the delivery method: an Authorization Bearer header holding an ingest-scoped API key. Do not set a signature scheme on this source. PagerDuty signs with a v1= prefixed header that hmac_sha256 does not accept, so the Bearer header is the credential.

3. Add the database metrics. Poll the Postgres metrics you care about from Prometheus and read the service from the metric labels.

   - name: metrics
     connector: prometheus
     poll: 10s
     config:
       url: http://localhost:9090
       queries:
         - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
           by_name: true
       labels:
         - { name: service, field: metric.service, primary: true }
         - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

4. Add the service logs. Use docker_logs for a container, or otlp for an OpenTelemetry pipeline. Give the source the same service label.

   - name: logs
     connector: docker_logs
     poll: 5s
     config:
       container: tares-demo-api-server   # the container name (docker ps)
       labels:
         - { name: service, const: api-server, primary: true }

5. Add deploys. Create a webhook source with a primary service label read from the payload’s service field, event_type: deploy, and the text template deploy {commit} by {author}: {message}. Then add one line to your CD pipeline, run on every deploy:

   curl -X POST https://<your-tares>/ingest/<ingest_key> \
     -H 'content-type: application/json' \
     -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

6. Check the timeline. Make sure the service value is the same on every source. Open the service in Explore to see the incident, metrics, logs and deploys merged on one clock. Then add a trigger over these sources, keyed by service, to wake your agent.

What you get

• One timeline per service, with the PagerDuty incident next to the metrics, logs and deploys that explain it.
• A diagnosis that cites evidence lines: the metric that moved, the log lines around it and the deploy that landed just before.
• The same read for a connected MCP client, so you can investigate interactively from the same evidence.
• A finding written back to the timeline, so the next investigation starts from the previous conclusion instead of from scratch.
• Alerting that stays in PagerDuty. Tares ingests what fired, correlates it and handles the waking.

When something else fits

• If the incident is the only signal you need, a plain webhook source that records the PagerDuty events is enough, and you can add metrics, logs and deploys later.
• If your services are not labeled consistently, fix the service label first, because the correlation depends on it.

Related