What should an AI SRE read next to alerts from Prometheus for 5xx errors on an API?

Tares puts the Prometheus alert for 5xx errors on the same per-service timeline as that API’s metrics, logs and recent deploys, so an AI SRE reads all of it in one pass. The alert tells the agent where to look. The 5xx rate, the error logs and the last change tell it why. Every claim in the diagnosis can then cite a line of evidence.

How it works

The prometheus_alerts connector polls Prometheus’s own alerts endpoint, by default every 30 seconds. Prometheus keeps deciding what counts as an alert. Tares does not re-threshold anything. It ingests what fired and correlates it.

Each active alert becomes one event:

• event_type is the alert name, for example HighErrorRate.
• text is the alert’s summary.
• The alert’s labels become Tares labels, plus a state of firing, pending or resolved.

Tares tracks state across polls. An alert fires once, re-emits only when its state changes, and produces a resolved event when it clears. The agent does not see a copy of the alert on every tick.

The alert sits on a timeline keyed by a shared label, usually service. Sources that carry the same service label merge onto one clock. For a 5xx alert on an API, an AI SRE should read these next, in this order:

• The 5xx rate metric from the prometheus connector, to see when the rate started climbing and how steep it is.
• The service’s request logs, from the Docker logs connector or OTLP, to see which errors the failing requests produced.
• Recent deploys and config changes, from a webhook source with event_type: deploy, because most real root causes are a change. Seeing a deploy at 14:02 and errors from 14:04 is the strongest lead.
• Earlier findings on the same service, so the agent starts from the previous conclusion instead of from scratch.

The AI SRE guide builds this whole chain, including the trigger that wakes an agent when the alert event arrives.

Set it up

1. Add the alerts source. In the console go to Sources → Add source → Prometheus alerts, enter the Prometheus url (plus any auth) and click Discover. It lists your alerting rules and the severities present. Optionally filter by severity, and toggle include pending if you want alerts that are still waiting out their for: duration. You can also ask a connected agent: “Use tares to ingest the firing alerts from the Prometheus at localhost:9090.”

2. Key the alerts by service. Discover proposes only the labels guaranteed to be present, severity and state. The service label exists only while an alert is firing, so wait until real alerts have been ingested, then open the Fields tab and add service as a label. Mark service as the primary label and keep severity as a plain label, so the alerts share the service key with the other sources instead of being keyed by severity.

   - name: alerts
     connector: prometheus_alerts
     poll: 30s
     config:
       url: http://localhost:9090
       severities: critical,warning   # blank = all
       labels:
         - { name: service, field: labels.service, primary: true }
         - { name: severity, field: labels.severity }
         - { name: state, field: state }

3. Give the evidence sources the same key. Your metrics, logs and deploy webhook each need a service label as their primary label. Then the 5xx rate, the error logs, the deploy events and the alert land on one timeline for that API.

4. Verify. Open the alerts source. Recent events shows one event per active alert, and resolved events as they clear. Fields shows the labels the alerts carry. Then open Explore, pick the API’s service value, and check that the alert appears beside its metrics, logs and deploys.

5. Start Tares locally if you have not already, then read the timeline from an agent or let a trigger wake one.

   tares up

What you get

• A diagnosis that cites evidence. The agent ties its conclusion to specific lines: the alert, the metric that tripped it, the error logs and the deploy before it.
• One read instead of four. The agent gets one correlated timeline per service and does not query a metrics store, a log system, a deploy tool and an alert feed separately.
• No duplicate noise. Each alert fires once and resolves once, so the agent reads a clean sequence of state changes.
• A loop that improves. The agent writes its finding back to the timeline, so the next investigation of that API starts from the last conclusion.
• Alerting stays where it is. Prometheus keeps owning the rules, and Tares correlates what they fire.

When something else fits

• If you route alerts through Alertmanager and want to keep that path, use the alertmanager connector, which receives them at a Tares ingest URL. The same evidence reading applies.
• If you want to alert on the typed metric fields directly, for example an error-rate predicate for 5xx storms, put a trigger over the metrics source. The Prometheus alerts page covers the alert path only.

Related