How do I route alerts from Prometheus about high latency to an AI SRE?

Tares polls the alerts your Prometheus rules have fired, and a trigger that filters on the high latency alert name wakes your AI SRE only for that symptom. Other alerts still land on the timeline but do not wake the agent. The trigger is named for the symptom, so each firing and the agent run it starts are easy to trace. Prometheus keeps deciding what counts as an alert, and Tares does the correlating and the waking.

How it works

• The prometheus_alerts connector polls Prometheus’s own alerts endpoint, every 30 seconds in the example below. It needs no Alertmanager and no PromQL.
• Each active alert becomes one event. The event type is the alert name, the text is the alert’s summary, and its labels become Tares labels, plus a state of firing, pending or resolved.
• Tares tracks state across polls. A latency alert fires once and emits a resolved event when it clears, not one copy per tick.
• A trigger watches the source and filters on event_type. An alert named HighLatency matches, and HighErrorRate or DependencyDown do not. The matching event wakes the agent you subscribe to the trigger.
• The agent reads the correlated timeline for that service, so it sees the HighLatency event next to the metrics and logs around it. For the full picture of the pieces, see Build an AI SRE.

Set it up

1. Start Tares and open the console.

   tares up

2. Add the alerts source under Sources, Add source, Prometheus alerts. Enter the Prometheus url and run Discover, which lists your alerting rules and the severities present. Or ask a connected agent to ingest the firing alerts from your Prometheus. The result looks like this:

   - name: alerts
     connector: prometheus_alerts
     poll: 30s
     config:
       url: http://localhost:9090
       severities: critical,warning   # blank = all
       labels:
         - { name: severity, field: labels.severity, primary: true }
         - { name: state, field: state }

   Use severities to narrow by severity at the source. Alerts still pending are ingested only if you turn on include_pending. See Prometheus alerts for every field and for adding per-alert labels such as service from the Fields tab once alerts have arrived.

3. Add a trigger that names the symptom and matches only the latency alert. Replace HighLatency with the alert name from your own rules.

   - name: high-latency
     sources: [alerts]
     filters:
       - { field: event_type, op: eq, value: HighLatency }
       - { field: state, op: eq, value: firing }
     condition: { aggregate: count, predicate: "> 0", window: 1m }
     cooldown: 5m

   The state filter keeps the resolved event from waking the agent again. The cooldown stops a flapping alert from waking it over and over.

4. Choose who gets woken by this trigger:

   • A Tares agent subscribed to high-latency, whose prompt you write for latency, for example to check the slowest dependency first.
   • Your own agent service over a webhook, which receives the timeline.
   • A Slack channel, so the firing lands where your team looks.

5. Verify. Open the source and check Recent events for your latency alert. When it fires, the trigger’s page lists the firing under Firings.

What you get

• Only latency alerts wake the agent. Error-rate and dependency alerts stay on the timeline as context but start no run.
• The agent starts from the HighLatency event, which carries the alert name, its summary and its labels, with the correlated evidence for that service alongside it.
• The trigger is named high-latency, so the firings and the runs they start read as one thread you can follow.
• One firing per episode, and a resolved event when Prometheus clears the alert.
• Prometheus keeps owning alerting. Tares does not re-threshold anything, it ingests what fired.
• To cover a second symptom, add another trigger with its own alert name and its own agent.

When something else fits

• If you route alerts through Alertmanager and want it to push them, use the Alertmanager connector instead of polling. The same trigger filter on the alert name works.
• If the latency alerts come from a tool Tares has no connector for, send them to a webhook source and filter on the field that carries the alert name.
• If you want to wake on a threshold on the metric itself rather than on a fired alert, build the trigger over a metrics source and set the threshold there.

Related