Which labels on Datadog monitors about a cron job that stopped tie it to the right service?

Tares ties a stopped-cron Datadog monitor to the right service through one label that every source carries, usually service, marked primary: true so it becomes the key. Datadog reaches Tares through the webhook connector and a Datadog webhook setup, and you put the same service value in the alert payload. The alert, metrics, logs and recent deploys then land on one entity timeline.

How it works

Every event in Tares carries labels, and the label marked primary: true is the key. An entity is a (label, value) pair, such as (service, nightly-report), and a read returns that entity’s events from every source merged in time order. Selecting by service is an index lookup, not a text scan. See Concepts for the full model.

For a cron job that stopped, one label does the work:

• service names the job or the service that owns it. Give every source that can speak about the job the same label name and the same value, and mark it primary.
• env is an optional second label, for example a constant prod. A read is a strict AND over the labels you give, so adding it narrows the timeline to one environment.

Datadog has no connector of its own in Tares today. It reaches Tares through the webhook connector and a Datadog setup. You write the payload JSON yourself in the Datadog Webhooks integration, so the shape is yours. Include a flat service field in it, because labels read the fields your payload carries. Keep the payload flat, since event_type_field and text_template read top-level fields only.

The other sources carry the same label in their own way:

• Prometheus metrics read service from the metric labels.
• Docker logs can set service as a constant per container.
• A deploy webhook reads service from the payload’s service field.

That shared label is what puts the alert, the metrics, the logs and the latest change on one timeline.

Set it up

1. Choose the key. Open each source’s Fields tab in the console and check its coverage, meaning how many sampled events carry the label. Pick a label that is reliably present, normally service.
2. In Tares, create a webhook source for the Datadog monitors. Declare service as a label and mark it primary. A constant env label is optional.

- name: ci-events
  connector: webhook
  config:
    labels:
      - { name: service, field: service, primary: true }
      - { name: env, const: prod }

Use your own source name in place of ci-events. Copy the ingest endpoint from the source page. 3. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the payload as JSON from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, and include a flat service field holding the job’s service name. 4. If Tares has auth on, add custom headers as JSON in the Datadog webhook, with an Authorization Bearer header holding an ingest-scoped API key. On an instance without auth you need no header. 5. In the monitor message for the stopped cron job, put @webhook- followed by your webhook’s name, so the monitor calls it. 6. Give the metrics, logs and deploy sources the same service label, marked primary, with the same value the Datadog payload uses. 7. Post a sample payload and check the result. Replace the placeholder with your source’s ingest key:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"service": "nightly-report"}'

Then read Recent events and Fields on the source page. service should appear with full coverage. 8. Open Explore, pick the entity (service, nightly-report) and confirm the alert sits next to the other sources on one clock.

What you get

• One timeline per service. When the Datadog monitor fires for the stopped cron job, you read that job’s alert, metrics, logs and recent deploys together, merged in time order.
• A stored key_value on every event, taken from the primary label, so lookups by service are fast.
• The original Datadog payload stored unmodified, so $EVENT_MSG and $LINK stay available next to the labels.
• A trigger that can watch the sources together. Its key_field defaults to the primary label of the first source, so firings are grouped per service. A dispatch carries that service’s timeline to the agent it wakes.
• A foundation for the loop in Build an AI SRE: the agent’s finding is written back onto the same entity, so the next investigation of that job starts from the previous conclusion.

When something else fits

• If a source cannot carry the same service value, for example a metric whose labels name the job differently, correlate on a different label that every source does carry and mark that one primary. The label name matters less than being present on all sources with the same values.
• If your alerts come from Prometheus rules, the prometheus_alerts or alertmanager connectors read the alert directly, and no Datadog webhook is needed.

Related