What should an AI SRE read next to PagerDuty incidents for a cron job that stopped?

Tares puts the PagerDuty incident for a stopped cron job on one timeline with the same service’s metrics, logs and recent deploys. An AI SRE reads that timeline and cites the evidence in its diagnosis. PagerDuty reaches Tares through the webhook connector and a vendor setup, and Tares has no connector of its own for PagerDuty today.

How it works

The incident is where the investigation starts. The evidence that explains it sits next to it, keyed by the same service label. This follows the pattern in the AI SRE guide.

For a cron job that stopped, put these four things on one service timeline:

• The incident. PagerDuty posts a v3 webhook delivery to a Tares webhook source. Each delivery lands as an event, so the triggered and resolved states are both on the timeline.
• Metrics. The prometheus connector polls the series you pick for that service. For a cron job, that means whatever you export about its runs.
• Logs. The docker_logs connector tails the job’s container. The otlp connector takes OpenTelemetry logs instead. The last lines before the job went quiet are usually the most useful.
• Recent changes. A second webhook source records deploys and config changes. A schedule edit or a new image just before the last run is a common cause.

All four sources carry the same service label, so Tares merges them on one clock. The agent reads that timeline, for example with read and a service name over MCP. Each claim in its diagnosis can then point at a metric, a log line or a change event.

A cron job can also report on itself. The webhook connector accepts a POST from anything that can send an HTTP request, so the job can post a short line on each run and its last run shows up beside the incident.

Set it up

1. In PagerDuty, create a v3 webhook subscription, in the console or with the Webhook Subscriptions API. Set its delivery URL to the Tares ingest URL. Choose the events, such as incident.triggered and incident.resolved. Choose the service scope, so one subscription covers the cron job’s service.
2. When Tares has auth on, add a custom header to the delivery method. It is an Authorization Bearer header holding an ingest-scoped API key. Leave signature settings off for this source, because PagerDuty signs in a form that hmac_sha256 does not accept. The Bearer header is the credential.
3. Create the webhook source for the incidents. PagerDuty’s payload nests the event under event, and labels read one-level dotted fields such as event.event_type. Use a fixed event_type, and give the source a constant service label so the incident lands on the right timeline. The service name here is an example.

- name: pagerduty-incidents
  connector: webhook
  config:
    event_type: pagerduty_incident
    labels:
      - { name: service, const: nightly-report, primary: true }
      - { name: incident_event, field: event.event_type }

4. Check that your metrics and log sources use the same service value, for example nightly-report. Without a shared label there is no shared timeline.
5. Add a second webhook source for changes, with a primary service label read from the payload and event_type: deploy. Then add one line to your deploy pipeline:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

6. Open Recent events and Fields on each source page to confirm events arrive and the labels carry values. Then ask your connected client: “Use tares: what happened to nightly-report in the last 15 minutes?”

What you get

• One timeline for the cron job’s service. The PagerDuty incident, the metrics, the logs and the deploy events sit on the same clock.
• A diagnosis that cites evidence, such as an incident at one time, the last log line a minute earlier and a config change shortly before. You can check each claim against the event it points to.
• Both incident states. The resolved delivery lands next to the triggered one, so the agent sees whether the job came back.
• A finding written back to the timeline. The next investigation of the same service starts from the previous conclusion.
• A path to automation. You can attach a trigger and an agent to the same sources, so the first look runs without anyone asking.

When something else fits

• If your alerting already runs in Prometheus or Alertmanager, use the prometheus_alerts or alertmanager connectors. They carry the alert fields without a vendor webhook setup.
• If the PagerDuty fields you need for labels sit more than one level deep, such as the service name inside the incident data, scope the subscription to one service and use a constant label, as above.
• If you only need to know that PagerDuty paged, the incident event alone is enough. The extra sources matter once you want the cause.

Related