Which labels on Prometheus alerts about a stopped cron job tie it to the right service?

Tares ties a Prometheus alert about a stopped cron job to its service through a service label that you mark as the key (primary: true) on the alerts source. Use the same service label on your metrics, logs and deploy sources, and the alert and its evidence land on one entity timeline. The labels severity and state are always present, so they narrow an alert but never identify the service.

How it works

Each alert that the Prometheus alerts connector reads becomes an event. The alert name is the event_type, the summary is the text, and the alert’s own labels become Tares labels. Tares also adds a state label with the value firing, pending or resolved.

The concepts page defines an entity as a (label, value) pair, such as (service, checkout). The label marked primary: true is the key, and the event’s key_value is set from it. A read for { service: "checkout" } returns that service’s events from every source, merged in time order. Selecting by a label is a lookup, not a text scan.

For a stopped cron job, the label that matters is the one your alerting rule puts on the alert to name the owning service. The connector can only carry what the alert has.

• severity and state are present on every alert. Tares proposes them for you.
• service and namespace are dynamic labels. They exist while the alert is firing, so you add them from the source’s Fields tab after real alerts have been ingested.
• If no label is marked primary, the connector’s default key is used. Set the key yourself so it is the same across sources.

Set it up

1. Add the alerts source under Sources → Add source → Prometheus alerts, or ask a connected agent to ingest the firing alerts from your Prometheus. Enter the url and run Discover.
2. Make sure the alerting rule for the cron job carries a service label in Prometheus. Let the alert fire once so Tares ingests it.
3. Open the source’s Fields tab. It shows each label’s coverage, meaning how many sampled events carry it, and its top values. Pick service if its coverage is reliable.
4. Edit the source so service is the primary label and the guaranteed labels stay alongside it:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    labels:
      - { name: service, field: labels.service, primary: true }
      - { name: severity, field: labels.severity }
      - { name: state, field: state }

5. Give every other source the same service label as its key, so they all share one entity:
   • Metrics from the prometheus connector read service from the metric labels.
   • Logs from docker_logs can use a constant, for example { name: service, const: api-server, primary: true }.
   • Deploys and config changes from a webhook source read service from the payload’s service field.
6. Open Explore, pick your service, and check that the alert, the metrics, the logs and the latest deploy sit on one clock. Use the same service value to build a trigger that watches these sources keyed by service, as the AI SRE guide does.

The label values must match across sources. The alert’s service value has to equal the value your logs and deploys carry, such as checkout in all of them.

What you get

• One timeline per service. When the cron job’s alert fires, the metrics, logs and recent deploys for that service sit next to it, merged in time order.
• A resolved event on the same timeline when the alert clears. Each alert fires once, and Tares re-emits only on a state change.
• A key that triggers can group by. A trigger watching the alerts, metrics and logs sources, keyed by service, fires one dispatch per service, carrying that service’s timeline.
• Findings that land in the right place. A Tares agent that diagnoses the alert writes its finding with the same label, so the next read of that service starts from the earlier conclusion.
• A way to check your choice. The Fields tab shows whether service is carried by enough alerts before you rely on it as the key.

When something else fits

• If your alerts reach Tares through Alertmanager rather than by polling Prometheus, use the alertmanager connector. Keep the same service key there so the timeline stays aligned.
• If a service label is not on the alert and you cannot add it in the Prometheus rule, pick another label that all your sources share, such as namespace, and key every source on that instead.

Related