How do I test PagerDuty incidents about a cron job that stopped before a real incident?

Tares lets you test PagerDuty incidents about a stopped cron job by posting a sample payload to the source’s ingest URL, then checking Recent events and Fields on the source page. PagerDuty reaches Tares through the webhook connector and a PagerDuty setup, and Tares has no connector of its own for PagerDuty today. Because test_source on a push source only returns the ingest endpoint, the sample post is what checks your label mapping.

How it works

• PagerDuty v3 webhook subscriptions deliver JSON shaped like {"event": {"id": "...", "event_type": "incident.triggered", "resource_type": "incident", ...}} to the Tares ingest URL.
• A webhook source stores the whole payload lossless and maps the fields you declare as labels.
• Label fields can be one-level dotted paths such as event.event_type. The event type field setting reads top-level fields only, so you set a fixed event_type instead.
• Key the source on a service label with a const value that names the cron job, and mark it primary. Every incident then carries that service as its key, and incident types do not collapse into one key.
• Keep the PagerDuty event type in a separate, non-primary label named incident_event. Do not name a label event_type, because the event type column shadows it in trigger filters.
• When Tares has auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. Do not set signature_scheme for PagerDuty. Its signed header carries a v1= prefix that hmac_sha256 does not accept, so the Bearer header is the credential.

Set it up

1. Create the source under Sources → Add source → webhook, or ask a connected agent to create a webhook source for your PagerDuty incidents. Use this config, replacing the const value with a name for your cron job:

- name: pagerduty-cron
  connector: webhook
  config:
    event_type: pagerduty_incident
    labels:
      - { name: service, const: nightly-backup-cron, primary: true }
      - { name: incident_event, field: event.event_type }
      - { name: resource_type, field: event.resource_type }

2. Copy the ingest endpoint from the source page. Calling test_source returns only that endpoint, so it does not check the mapping.
3. Post a sample incident.triggered payload that looks like what PagerDuty sends:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"event": {"id": "TEST1", "event_type": "incident.triggered", "resource_type": "incident", "occurred_at": "2024-01-01T00:00:00Z"}}'

4. If Tares has auth on, send the request with an Authorization: Bearer header holding an ingest-scoped API key. Without auth, no header is needed.
5. Open the source page. Under Recent events, confirm the event arrived with the fixed event type pagerduty_incident. Under Fields, confirm event.event_type and event.resource_type show up and fill the incident_event and resource_type labels.
6. Post the same payload with incident.resolved as the event type and check that incident_event changes while service stays nightly-backup-cron.
7. In PagerDuty, create a v3 webhook subscription whose delivery URL is the ingest URL. Add the Bearer header to the delivery method when Tares has auth on. Choose incident.triggered and incident.resolved, and pick the scope: account, team or service. Then trigger a test incident on the service that watches the cron job and confirm it lands the same way as your sample.

What you get

• You know the ingest URL, auth and label mapping work before a real cron job stops.
• Triggered and resolved incidents both show in Recent events under one stable service key.
• Fields shows which payload fields arrive and how often, so you can add more labels before the real incident.
• The whole PagerDuty payload is stored lossless, so nothing is lost if you add a label later.
• You can build triggers on the incident_event label, for example to act on incident.triggered only. You start from a mapping you have already seen work.

When something else fits

• If your cron job alerts come from Prometheus rules routed through Alertmanager, the Alertmanager connector fans each alert out into its own event and has its own test batch to post.
• If you want the full picture of how Tares handles alert sources, start from AI SRE.

Related