Which labels on Prometheus alerts about crash loops tie them to the right service?

Tares ties a Prometheus crash loop alert to the right service through one label, usually service, that you mark as the key on every source. The alert, the metrics, the logs and recent changes then land on one entity timeline. Pick the label by checking which one your crash loop alerts actually carry in the source’s Fields tab.

How it works

Every event carries labels, and the label marked primary: true is the key. An entity is a label and value pair, such as service=checkout. A read of that entity returns its events from every source, merged in time order.

For the Prometheus alerts connector:

• Each active alert becomes one event. The event_type is the alert name and the text is its summary.
• The alert’s own labels become Tares labels, plus a synthetic state of firing, pending or resolved.
• Discover only proposes severity and state, because those are always present. Labels such as namespace and service exist only while an alert is firing, so you add them after real alerts have been ingested.
• State is tracked across polls. A crash loop alert fires once, and a resolved event lands when it clears.

The label that ties the alert to the right service is the one every source carries for that service. Use the same name and the same values everywhere, for example service. If the alert’s label is called something else, declare a Tares label named service that reads from it. The name on the Tares side is what must match across sources. Marking it primary sets key_value on each event, and a trigger keys on the primary label of its first source by default.

The Fields tab reports each label’s coverage, meaning how many sampled events carry it, and its top values. Use coverage to confirm your crash loop alerts carry the label reliably before you make it the key. See Concepts for how labels, keys and entities fit together.

Set it up

1. Create the alerts source and let some crash loop alerts be ingested. In the console go to Sources → Add source → Prometheus alerts, enter the url and click Discover. You can also ask a connected agent to ingest the firing alerts from your Prometheus.
2. Open the source and check Recent events for your crash loop alerts. Open Fields and look at coverage and top values for the label that names the service. If it is missing on most alerts, fix the alert rule so it carries the label, or choose a label that is present.
3. Declare the label and mark it as the key. This is the shape from the connector page, with a service label read from the alert’s labels and set as primary:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: service, field: labels.service, primary: true }
      - { name: severity, field: labels.severity }
      - { name: state, field: state }

4. Give the other sources the same label name. Metrics read it from the metric labels, and container logs can use a constant:

labels:
  - { name: service, field: metric.service, primary: true }

labels:
  - { name: service, const: api-server, primary: true }

5. Add recent changes the same way. Create a webhook source with a primary service label read from the payload’s service field, and post a deploy event on every release. A deploy that lands just before a crash loop then sits right next to it on the timeline.
6. Open Explore, pick the service, and confirm that the alert, metrics, logs and deploys appear together on one clock. The AI SRE guide walks through this setup end to end, including a trigger keyed by service that wakes an agent when an alert event arrives.

What you get

• One timeline per service. When a crash loop alert fires, you see the metrics, the container logs and the latest deploy for that same service next to it.
• No duplicate noise. Each alert fires once, and the resolved event marks when the crash loop cleared.
• A reliable key. Because you checked coverage in the Fields tab, the alert does not drop onto an empty or wrong entity.
• A ready input for triggers and agents. A trigger keyed by service carries the entity’s timeline across its sources, so the agent that wakes starts from the correlated evidence.
• Findings on the same entity. An agent’s conclusion is written back with the same label, so the next investigation of that service starts from it.

When something else fits

• If your alerts come from Alertmanager rather than Prometheus’s own alert list, use the alertmanager connector and apply the same labeling idea there.
• If your crash loop alerts carry no service label at all, add the label in the alert rule first. Tares can only key on labels the alert actually has.

Related