How do I secure the connection that carries PagerDuty incidents into Tares?

Tares secures the PagerDuty connection with an ingest-scoped API key that PagerDuty sends as an Authorization Bearer header on every delivery. PagerDuty reaches Tares through the webhook connector and a vendor setup, because Tares has no connector of its own for PagerDuty today. The header is needed when Tares has auth on, and the key only lets the sender post events to your ingest URL.

How it works

Each connection type holds its credential in a different place:

• Polled sources. Tares pulls from the upstream, so the source holds the upstream credential. The Prometheus connector, for example, takes bearer_token, or username and password.
• Pushed sources. The sender posts to a Tares ingest URL, so the sender carries the credential. That is either an ingest-scoped API key in an Authorization header, or a signature that Tares verifies.

PagerDuty is a pushed source that uses the header. A v3 webhook subscription posts each incident event as JSON to the ingest URL of a webhook source. On an instance with auth on, the delivery carries Authorization: Bearer followed by an ingest-scoped API key, and that header is the credential. On an instance with auth off, no header is needed.

Do not set signature_scheme on this source. PagerDuty signs its deliveries with a header whose value starts with v1=, and the hmac_sha256 scheme in Tares does not accept that format. The Bearer header does the authentication instead.

The Alertmanager receiver works the same way. It is also a pushed source, and it sends the same kind of ingest-scoped key when auth is on. Once the connection is secured, the incidents join the rest of your monitoring described in AI SRE.

Set it up

1. In Tares, create a webhook source for PagerDuty. In the console, go to Sources → Add source → webhook. Creating a push source in the console mints an ingest-scoped API key for it and shows it once, so copy it then. Also copy the ingest endpoint from the source page.
2. Declare labels from the PagerDuty payload. Use one-level dotted fields such as event.event_type. Set a fixed event_type, because event_type_field reads top-level fields only and the PagerDuty event type sits one level down.

- name: pagerduty-incidents
  connector: webhook
  config:
    event_type: pagerduty
    labels:
      - { name: event_type, field: event.event_type, primary: true }
      - { name: resource_type, field: event.resource_type }

3. In PagerDuty, create a v3 webhook subscription, in the console or with the Webhook Subscriptions API. Set its delivery URL to the Tares ingest URL.
4. When Tares has auth on, add a custom header to the delivery method: Authorization with the value Bearer followed by the ingest-scoped API key from step 1.
5. Choose the events, such as incident.triggered and incident.resolved, and the scope: account, team or service.
6. Check the source. Post a test payload to the ingest URL, then open Recent events. Fields shows which payload fields arrive and how often.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"action": "deploy", "sender": "ci", "pipeline": "web"}'

On a secured instance, a request without the key is not accepted, so send the same Authorization Bearer header that PagerDuty sends when you test.

What you get

• Only senders that hold your ingest-scoped API key can post incidents to the PagerDuty source on a secured instance.
• The key is scoped to ingestion, so it posts events and nothing else.
• Every incident delivery is stored lossless as an event, with the labels you declared, so you can correlate incidents with your other sources.
• The credential model is the same across your sources: pulled sources keep their upstream credential in the source, and pushed sources check what the sender presents.

When something else fits

• If you want a signature as the credential, use a vendor that signs in a scheme Tares accepts. PagerDuty’s v1= signature is not one of them, so the Bearer header stays the credential here.
• If the system you are connecting is Prometheus, use a polled source. Tares then holds the Prometheus credentials itself and nothing needs to be configured on the sender.

Related