What should Prometheus alerts about crash loops include for an AI SRE?

Tares works best with crash loop alerts that carry a service name, a severity, a clear alert name and a summary. Tares turns each alert into an event whose type is the alert name and whose text is its summary, and it keeps the alert’s labels as Tares labels. The prometheus_alerts connector adds a state of firing, pending or resolved, so the agent can tie the alert to the evidence on the same timeline.

How it works

The Prometheus alerts connector polls the alerts that your Prometheus rules have already fired. Prometheus decides what counts as a crash loop. Tares ingests the result, correlates it and can dispatch it to an agent. Tares needs no Alertmanager and no PromQL for this.

Each active alert becomes one event:

• event_type is the alert name, so a descriptive name for your crash loop rule tells the agent what kind of failure it is looking at.
• text is the alert’s summary, so a short, specific summary is the first thing the agent reads.
• The alert’s labels become Tares labels, plus a state label that is firing, pending or resolved.

Tares tracks state across polls. An alert fires once and re-emits only when its state changes. When the alert clears, Tares emits a resolved event, so the agent can see that a crash loop started and later stopped.

In the AI SRE setup, sources that share the same service label are merged into one timeline per service, with metrics, logs and alerts on one clock. That is where a crash loop alert sits next to the evidence the agent uses to explain it.

Set it up

1. Give your crash loop rule these labels and annotations in Prometheus:
   • a severity label, such as critical or warning. Tares proposes this label automatically and can filter on it.
   • a service label that matches the service label on your metrics and log sources.
   • a namespace label if you run more than one environment.
   • a summary that says what is wrong and where. The summary becomes the event text.
   • an alert name that says what happened. The alert name becomes the event type.
2. Add a source in the console under Sources → Add source → Prometheus alerts. Enter the url, add authentication if your Prometheus needs it, and click Discover. Discovery lists your alerting rules and the severities present. You can also ask a connected agent: “Use tares to ingest the firing alerts from the Prometheus at localhost:9090.”
3. Choose the severities to ingest. Leave the field blank to take all of them. Turn on include pending if you want alerts that are still waiting out their for: duration.
4. Start with the always-present labels:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

4. Wait until a real crash loop alert has been ingested. An alert’s dynamic labels, such as namespace and service, exist only while it is firing, so Discover cannot propose them in advance. Open the source and look at the Fields tab. Promote service and namespace to labels there so you can correlate on them.
5. Open Recent events to check that you see one event per active alert, and resolved events as alerts clear.

What you get

• An event per crash loop alert, named after the alert, with your summary as its text.
• A severity label and a state label on every alert, so the agent can tell a firing critical alert from a resolved warning.
• The service and namespace labels you promoted, so you can correlate on them. With a shared service label, the alert appears on the same timeline as your metrics and logs.
• A resolved event when the alert clears, so the agent knows the crash loop ended.
• One firing per state change instead of a copy of the alert on every poll, so the agent is not flooded with duplicates.
• A trigger can wake an agent when an alert event arrives, as the incident trigger does in the AI SRE walkthrough. The agent reads the correlated timeline and writes its finding back, where the next investigation finds it.

The more specific your summary and labels are, the easier it is for the agent to tie the alert to concrete evidence lines.

When something else fits

If you already route alerts through Alertmanager and want Alertmanager’s grouping, use the alertmanager connector instead. Tares polls Prometheus alerts, so the source needs a reachable Prometheus server with alerting rules configured. If your signal is a raw metric rather than a fired alert, the prometheus connector reads metrics by name or label.

Related