What should Prometheus alerts about 5xx errors on an API include for an AI SRE?

Tares works best when a Prometheus 5xx alert carries a service label, a severity label, a clear alert name and a summary annotation. With those, the agent can tie the alert to the metrics and logs for the same service. The Prometheus alerts connector turns each alert into an event, so what you put on the rule is what the agent reads.

How it works

The prometheus_alerts connector polls Prometheus’s own alerts endpoint, so Prometheus keeps deciding what counts as an alert. Tares ingests what fired, correlates it and can dispatch it to an agent. Each active alert becomes one event, and the fields of your rule map onto it like this:

• Alert name becomes the event type. Name the rule for the symptom, for example HighErrorRate, so the event says what broke.
• Summary annotation becomes the event text. This is the sentence the agent reads first, so say what is failing, for example that the API 5xx rate is high.
• Labels become Tares labels. The service label is the one that matters most. When your metrics and logs sources carry the same service label, Tares merges them with the alert into one timeline per service.
• Severity label lets you filter on critical and warning, and lets an agent or trigger judge urgency.
• State is added by Tares as firing, pending or resolved. Each alert fires once and re-emits only when its state changes. A resolved event lands when it clears.

On the timeline the alert sits on one clock beside the error logs and the metric that tripped it. The agent can see when the 5xx rate climbed relative to the alert, and tie its conclusion to specific evidence lines. The AI SRE guide shows this chain with the HighErrorRate alert.

Set it up

1. In your Prometheus alerting rule for 5xx errors, make sure the alert carries a service label, a severity label and a summary annotation. Give the rule an alert name that describes the symptom.

2. Start Tares locally:

   tares up

3. In the console, go to Sources → Add source → Prometheus alerts. Enter the Prometheus url, add authentication only if your Prometheus needs it, and click Discover. It lists your alerting rules and the severities present. Optionally filter by severity, and turn on include pending to see alerts that are still waiting out their for: duration.

4. Declare the labels. Discover proposes only severity and state, because they are always present. Add service yourself, as in this source:

   - name: alerts
     connector: prometheus_alerts
     poll: 30s
     config:
       url: http://localhost:9090
       severities: critical,warning   # blank = all
       labels:
         - { name: service, field: labels.service, primary: true }
         - { name: severity, field: labels.severity }
         - { name: state, field: state }

5. Open the source and check Recent events for one event per active alert. Then open Fields to see the labels your alerts carry. An alert’s dynamic labels, such as namespace and service, exist only while it is firing. If service is missing from the list, trigger or wait for a real alert and promote the label from Fields.

6. Key your metrics and logs sources by the same service label, so the alert and its evidence share one timeline.

You can also ask a connected agent to ingest the firing alerts from your Prometheus. It calls discover_source, then create_source.

What you get

• One event per 5xx alert, named after the rule and carrying your summary as its text.
• Alerts, metrics and logs for the same service on one timeline, so the agent starts from correlated evidence instead of from the alert alone.
• A resolved event when the alert clears, so the agent can tell an active incident from a past one.
• Severity filtering at the source, so noisy info alerts stay out if you only want critical,warning.
• No change to how you alert today. Prometheus keeps owning the rules, and nothing is re-thresholded in Tares.

When something else fits

• If your alerts already route through Alertmanager and you want to keep that path, use the alertmanager connector, which receives what Alertmanager sends to a webhook receiver. The AI SRE guide lists both options.
• If you want a trigger on the raw 5xx rate rather than on a fired alert, threshold the typed fields from the prometheus metrics source.

Related