What should an AI SRE read next to Alertmanager alert groups for 5xx errors on an API?

Next to Alertmanager alert groups, an AI SRE on Tares reads the same service’s 5xx metrics, request logs and recent deploys from one correlated timeline. Every source carries the same service label, so the alert, the error-rate metric, the failing requests and the last change sit on one clock. The agent reads that timeline and cites specific evidence lines in its diagnosis instead of reasoning from the alert alone.

How it works

An Alertmanager alert says that something is wrong. It does not say why. For a 5xx alert on an API, Tares gives the agent four kinds of evidence together:

• The alert. The alertmanager connector turns every alert Alertmanager routes into one event. The event type is the alert name, the text is its summary, and the labels become Tares labels plus a status of firing or resolved. The full alert, including its fingerprint, stays in the payload. Alertmanager has already grouped and deduped, so you keep its grouping, silencing and resolved behavior.
• Metrics. A prometheus source for the same service: the 5xx rate, p99 latency, DB connections and dependency health. This shows when the errors started and how steep the climb is.
• Logs. The API’s request logs, through docker_logs per container or through OTLP from your OpenTelemetry pipeline. These show which requests fail.
• Recent changes. A webhook source that your deploy pipeline posts to on every release, with event_type set to deploy. A second one with event_type set to config covers config changes. Most real root causes are a change, so this is often the most useful line on the timeline.

The join happens on the service label. The alertmanager connector derives its entity key from a label, checking namespace, then service, then job, then instance, or a primary label you declare. Make sure that key is the same value your metrics, logs and deploy sources use. Then they merge into one timeline per service, and a trigger or an MCP client reads that timeline in a single call.

Set it up

1. Create the alert source. In the console go to Sources, then Add source, then Alertmanager alerts. Name it and click Create. You can also ask a connected agent to create an Alertmanager source and give you the webhook URL.
2. Add the receiver to your alertmanager.yml with the ingest URL from the source page, then reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If Tares has auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. Without auth, the receiver above is enough. Keep send_resolved: true so the timeline shows when the 5xx alert clears.
4. Check the keying. Open Fields on the alert source and confirm the alert labels include the service value. Promote service to the key, or set key_label in the advanced options, so it matches your other sources.
5. Add the evidence beside it: a prometheus source with your 5xx and latency queries, a log source for the API (docker_logs or OTLP), and a webhook source for deploys with a primary service label. Then add one line to your deploy pipeline:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

6. Open Explore and pick the service. Alerts, metrics, logs and deploys now appear merged on one clock. To investigate interactively, run tares mcp and ask your client what happened to the service in the last 15 minutes.

The AI SRE guide walks through the full wiring, including the trigger that wakes an agent when an alert arrives. The Alertmanager connector page covers the receiver and the key options.

What you get

• A diagnosis that points at evidence: the alert, the metric that tripped it, the error logs and the deploy that landed minutes earlier, such as “deploy at 14:02, errors from 14:04”.
• One read per service instead of separate queries against each system.
• Alertmanager keeps owning alerting. Tares records what it pushes and correlates it with your other sources. Nothing is re-thresholded.
• Resolved events on the same timeline, so the agent sees the end of the incident as well as the start.
• Findings written back to the service’s timeline, so the next investigation starts from the previous conclusion.

When something else fits

• If you do not run Alertmanager and want Prometheus rules polled directly, use the prometheus_alerts connector instead.
• If your API is not keyed by a shared service label, fix the keying first. Without it the alert and the evidence land on different timelines.

Related