What should an AI SRE read next to Alertmanager alert groups for high latency?

Tares puts the same service’s metrics, logs and recent deploys on one timeline beside Alertmanager alerts, so the diagnosis cites evidence. For a high latency alert group, an AI SRE reads that service’s latency and error metrics, its request logs and its latest deploy or config events. Key every source by the same service label and an agent gets all of it in one read.

How it works

An Alertmanager alert is a signal that something crossed a line. It does not say why. Tares records each alert as an event and lines it up with the other sources that carry the same service label.

• The alert. Alertmanager has already evaluated, grouped and deduped. Its webhook receiver posts a batch, and Tares splits the alerts array into one event per alert. The event_type is the alert name, the text is its summary, and the labels become Tares labels plus a status of firing or resolved. The full alert, including its fingerprint, stays in the payload.
• Metrics. A prometheus source brings the numbers behind the alert, such as p99 latency, the 5xx rate, database connections and dependency health.
• Logs. A docker_logs source, or an OTLP pipeline, brings the request logs for the same service, so slow requests and errors sit on the same clock as the alert.
• Recent changes. A webhook source with a deploy event type, and a second one for config changes, shows what shipped just before latency rose. Most real root causes are a change, so this is the highest-value addition.

The entity key for Alertmanager events is derived from a label, in the order namespace, service, job, instance, or from a label you declare as primary. When every source resolves to the same service value, Tares builds one timeline per service. The agent reads that timeline and can tie its conclusion to specific evidence lines. This is the pattern in the AI SRE guide.

Set it up

1. Create an Alertmanager source in the console under Sources → Add source → Alertmanager alerts, or ask a connected agent to create one and give you the webhook URL. The source page shows a receiver block with your ingest URL filled in. Add it to alertmanager.yml and reload Alertmanager. The Alertmanager connector page has the full details.

   receivers:
     - name: tares
       webhook_configs:
         - url: http://localhost:8787/ingest/<ingest_key>
           send_resolved: true
   route:
     receiver: tares

2. If Tares runs with auth on, Alertmanager sends a Bearer Authorization header holding an ingest-scoped API key. Without auth, the receiver needs nothing more.

3. Send a test batch and check Recent events:

   curl -X POST http://localhost:8787/ingest/<ingest_key> \
     -H 'Content-Type: application/json' \
     -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

4. Open Fields on the source and promote namespace or service so the alert correlates with your other sources. If your alerts use a different label, set key_label to it.

5. Add the evidence sources, each with the same primary service label:

   • a prometheus source for latency, error rate and dependency metrics, reading service from the metric labels;
   • a docker_logs or otlp source for the service’s logs;
   • a webhook source for deploys, with a primary service label read from the payload and a deploy event type. Add a second one for config changes. Post to its ingest URL from your CD pipeline on every deploy.

6. Add a trigger over those sources, keyed by service. You can keep reacting to fired alerts, or threshold a typed field directly, for example max(p99_ms) > 1000 for latency. Choose who wakes: a Tares agent, your own agent over a webhook, or a Slack channel.

7. To investigate on demand, start the MCP endpoint and point your client at it:

   tares mcp

   Then ask: what happened to this service in the last 15 minutes? The client reads the correlated timeline for that service.

What you get

• One timeline per service, with the latency alert, the p99 and error metrics, the request logs and the latest deploy on the same clock.
• A diagnosis that points at evidence, for example a deploy that landed minutes before latency climbed, instead of a guess from the alert text alone.
• Alertmanager keeps its grouping, silencing and resolved handling. Tares records what it pushes, and the resolved event lands on the timeline when the alert clears.
• A finding written back onto the service’s timeline. The next investigation, whether from a trigger or from your MCP client, starts from that conclusion and not from scratch.
• Freedom in who investigates: a Tares agent with a prompt you edit, your own agent service, or you through a connected client.

When something else fits

• If you do not need Alertmanager’s grouping and silencing and only want Tares to read the alerts your Prometheus rules fire, the prometheus_alerts connector polls them instead.
• If the latency question is purely about numbers with no alert involved, a prometheus source and a trigger on a typed field such as p99 are enough.

Related