What should alerts from Prometheus about high latency include for an AI SRE?

Tares works best when a high latency alert from Prometheus carries a service label, a severity label and a summary annotation. Tares turns each active alert into an event on that service’s timeline, with the alert name as the event type, the summary as the text and the labels as Tares labels. The agent then reads the alert next to the metrics and logs for the same service.

How it works

The prometheus_alerts connector polls Prometheus’s own alerts endpoint, so Prometheus keeps deciding what counts as an alert. Tares ingests what fired and correlates it. Each active alert becomes one event, and a resolved event follows when it clears.

Here is how the parts of a high latency alert map to what the agent sees:

• Alert name. It becomes the event type, for example HighLatency. Pick a name that says what is slow.
• Summary annotation. It becomes the event text. Write one plain sentence that names the symptom, such as which latency is high and for which service.
• Severity label. Tares always carries severity, so the agent and your triggers can tell a warning from a critical alert. You can also ingest only some severities, such as critical,warning.
• State. Tares adds a state label of firing, pending or resolved. You do not set it in the rule.
• Service label. Put a service label on the alert rule. It is the key that ties the alert to the metrics and logs of the same service, so all three sources land on one timeline per service.

State is tracked across polls. An alert fires once and re-emits only when its state changes, so the agent sees a clean sequence instead of a copy every tick. Because the events of a service share one clock, the agent can line up when the alert fired against when the latency metric moved and what the logs show. That is how it ties the alert to specific evidence lines.

Set it up

1. In your Prometheus alerting rule for high latency, set a service label, a severity label and a summary annotation. Use the same service name your metric and log sources use.
2. Add the source. In the console go to Sources, Add source, Prometheus alerts, enter the Prometheus url and click Discover. You can also ask a connected agent to ingest the firing alerts from your Prometheus. Discovery lists your configured alerting rules and the severities present.
3. Use this source definition. severity and state are the labels Tares can propose up front, because they are always present:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

4. Let a real alert arrive, then open the source and check Fields. An alert’s dynamic labels, such as namespace and service, only exist while it is firing, so they show up once alerts have been ingested. Promote service from Fields so the alert correlates with your other sources.
5. Optionally turn on include pending to also ingest alerts still waiting out their for: duration. That gives the agent an earlier signal, at the cost of alerts that may never fire.
6. Open Recent events to confirm one event per active alert. Then build a trigger over the alert source and your metric and log sources, keyed by service, and subscribe an agent to it.

The Prometheus alerts connector page has the full list of options. The AI SRE guide shows the whole loop, including the demo HighLatency rule and a latency scenario you can trigger to watch it run.

What you get

• An alert event on the right service’s timeline, labeled with its severity and state.
• A readable summary as the event text, so the agent knows the symptom before it opens any other evidence.
• One correlated view where the alert sits next to the latency metric and the request logs for the same service.
• A resolved event when the alert clears, so the agent knows whether the problem is still live.
• A trigger that can wake an agent when an alert event arrives, with the evidence already in one place. In the AI SRE guide, the agent writes a finding with what broke, since when, the likely cause tied to evidence lines and a next action.

The more consistent your labels, the better this works. A service label that matches your other sources is the single most useful thing an alert can carry.

When something else fits

• If your alerts already route through Alertmanager, the alertmanager connector takes them by webhook instead of polling Prometheus.
• If you want to threshold the latency metric itself rather than react to a fired alert, use the prometheus connector and a trigger on the typed fields.
• If you keep your alerts in a system with no connector of its own, send them through the webhook connector.

Related