Which labels on Alertmanager alert groups about a cron job that stopped tie it to the right service?

Tares ties a stopped cron job’s alerts to its service through one label that every source carries, such as service, marked as the key. Alertmanager alert labels become Tares labels, so you set key_label on the alertmanager source to that label, and the alert lands on the same entity timeline as your metrics, logs and recent changes. Without a setting, Tares keys on namespace first, then service, job and instance.

How it works

Alertmanager sends a batch of alerts to the ingest URL of an alertmanager source. Tares splits the batch into one event per alert. Each event has the alert name as its event_type, the summary as its text, and the alert’s own labels as Tares labels, plus a synthetic status of firing or resolved. The full alert, including its fingerprint, stays in the payload.

The label marked as the key sets the event’s key_value. An entity is a pair of a label and a value, such as service and checkout. A read for { service: "checkout" } returns every event with that pair, merged in time order across sources. This is why one shared label matters more than which label you pick.

On the alertmanager source, Tares derives the key from the first of these labels it finds: namespace, then service, then job, then instance. If your cron alerts carry both namespace and service, the default keys on namespace, which groups by namespace instead of by service. Set key_label to service to key on the service.

See Concepts for how labels, keys and entities fit together.

Set it up

1. Pick the label. Choose one that your alert groups, metrics, logs and deploy events all carry for the same service, usually service. Open the source’s Fields tab in the console, or call GET /api/sources/<name>/fields. It shows how many sampled events carry each label and its top values, so you can confirm the label is reliably present.
2. Create the alertmanager source in the console under Sources, Add source, Alertmanager alerts, or ask a connected agent to create it. Set its key label:

- name: cron-alerts
  connector: alertmanager
  config:
    key_label: service

3. Add the receiver to alertmanager.yml with the ingest URL from the source page, then reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

Set send_resolved: true so the timeline shows when the cron job’s alert clears. On a secured instance, the receiver sends a Bearer Authorization header holding an ingest-scoped API key. Tares needs it only when auth is on. 4. Mark the same label as the key on every other source. Declare it with primary: true:

labels:
  - { name: service, field: service, primary: true }
  - { name: env, const: prod }

For a source whose payload or metric labels hold the service under another path, point field at that path. A fixed service can use const. 5. Send a test batch and check Recent events:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

6. Check Fields on the alertmanager source. It lists the alert labels, so you can promote the one you chose and see its coverage.

If you also use a trigger, key_field names the label that identifies the entity across its sources. By default it is the primary label of the first source.

What you get

• One entity timeline per service. When a cron job’s alert fires, a read on that service returns the alert, the metrics, the logs and the latest deploys together on one clock.
• Firing and resolved events for the same alert, so you can see when the job’s alert started and when it cleared.
• A key you chose on purpose, not one Tares picked from namespace, service, job or instance.
• Context for agents. A trigger that watches these sources wakes an agent with the entity’s correlated timeline, and the agent can write a finding back onto it. The AI SRE guide walks through that loop end to end.

The Alertmanager connector page has the full receiver setup and the configuration field.

When something else fits

• If you want Tares to poll Prometheus for fired alerts instead of receiving them from Alertmanager, use the prometheus_alerts connector. You lose Alertmanager’s grouping and silencing, and delivery is not real time.
• If the alerts carry no label that matches your other sources, no key setting can join them. Add that label in your alert rules first, or use a const label on the source for a single known service.

Related