How do I line up PagerDuty incidents with Prometheus metrics on one timeline?

Tares puts PagerDuty incidents and Prometheus metrics on one timeline by giving both sources the same service label. The incident shows what fired, and the metric events next to it show how the service got there. PagerDuty reaches Tares through the webhook connector and a vendor setup, and Prometheus is polled by its own connector.

How it works

Tares builds one timeline per service from every source that carries the same key label. You set up two sources and label both with service.

• Prometheus metrics. The Prometheus connector polls the metrics you pick by name or by label. You write no PromQL. Each series becomes a metric event on every poll tick, and the numeric value is stored as a number-typed value label.
• PagerDuty incidents. Tares has no connector of its own for PagerDuty today. A PagerDuty v3 webhook subscription posts to the ingest URL of a webhook source, and the payload is stored lossless. Labels map payload fields into the envelope.
• The join. Both sources carry service, so incident events and metric events appear together on that service’s timeline, merged on one clock. This is the same per-service timeline the AI SRE guide builds its agents on.

The PagerDuty payload nests everything under event, with fields such as event.event_type, event.id, event.resource_type and event.occurred_at. Labels read one-level dotted fields such as these. The webhook event_type_field reads top-level fields only, so you set a fixed event_type instead.

Set it up

1. Install and start Tares. Install with uv tool install tares, pipx install tares or brew install glassflow/tap/tares, then start it:

   tares up

2. Add the Prometheus source. In Sources, Add source, Prometheus, enter the Prometheus URL, run Discover and tick the metrics you want. Tares generates the config. Make service the primary label and keep the numeric value label:

   - name: metrics
     connector: prometheus
     poll: 10s
     config:
       url: http://localhost:9090
       queries:
         - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
           by_name: true
       labels:
         - { name: service, field: metric.service, primary: true }
         - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

3. Add the PagerDuty webhook source. In Sources, Add source, webhook, declare labels and save. Then copy the ingest endpoint from the source page. If you scope the PagerDuty subscription to one service, give the source a constant service label with that service’s name, the same value your metrics carry:

   - name: pagerduty-incidents
     connector: webhook
     config:
       event_type: pagerduty_incident
       labels:
         - { name: service, const: api-server, primary: true }
         - { name: incident_event, field: event.event_type }
         - { name: resource, field: event.resource_type }

4. Create the PagerDuty subscription. In PagerDuty, create a v3 webhook subscription, in the console or with the Webhook Subscriptions API. Set its delivery URL to the Tares ingest URL. Choose events such as incident.triggered and incident.resolved, and a scope: account, team or service.

5. Add the credential if Tares has auth on. Add a custom header Authorization: Bearer followed by an ingest-scoped API key to the delivery method. Do not set a signature scheme on the source. PagerDuty signs with a v1= prefixed header that hmac_sha256 does not accept, so the Bearer header is the credential.

6. Verify. Trigger a test incident. Check Recent events on each source, and check Fields on the webhook source to confirm the labels arrive. Then open Explore, pick the service, and read incidents and metrics together.

What you get

• One timeline per service with PagerDuty incident events and Prometheus metric events side by side.
• Triggered and resolved incidents next to the metric movement around them, so you see the climb before the page.
• A numeric value on every metric event, so a trigger over the service can aggregate it with max, avg or sum.
• The full PagerDuty payload stored lossless, with the labels you declared available for correlation.
• The same single read for a connected agent or MCP client, so an investigation starts from the correlated evidence instead of from two separate tools.

When something else fits

• If you only need the incident list, PagerDuty’s own console already shows it. Tares adds value when you want it merged with metrics and other sources per service.
• If you need a computed metric such as a p99, use an advanced queries row in the Prometheus connector with an explicit event_type and field.
• If PagerDuty incidents span many services, use one webhook source per service scope, or point the subscription at the scope you need.

Related