What should an AI SRE read next to alerts from Prometheus for high latency?

Tares puts the HighLatency alert on one timeline with the same service’s metrics, logs and recent deploys, so an AI SRE reads all of it together. The alert tells the agent where to look. The p99 and error metrics, the request logs and the latest change tell it why. Every claim in the diagnosis can then cite an evidence line.

How it works

The prometheus_alerts connector polls Prometheus for the alerts its own rules have fired. Each alert becomes an event. The event_type is the alert name, the text is its summary, and the alert’s labels become Tares labels plus a state of firing, pending or resolved. Each alert fires once and emits a resolved event when it clears, so the timeline does not fill with repeats.

The alert is one source. Tares merges it with the other sources that carry the same service label into one timeline per service. For a latency alert, the AI SRE reads these next:

• Metrics for the same service. The p99, the 5xx rate, database connections and dependency health, from a prometheus source. These show whether latency rose alone or with errors and a struggling dependency.
• Logs for the same service. Request logs from a docker_logs source or from OTLP. They show which requests were slow and what they logged.
• Recent changes. Deploy and config events from a webhook source. Most real root causes are a change, so a deploy that landed minutes before the latency rise is often the answer.
• Earlier findings. Conclusions that agents wrote back to the timeline, so a new investigation starts from the last one.

A trigger watching these sources can wake an agent when the alert event arrives. The agent reads the correlated timeline, names the likely cause, ties it to specific evidence lines and writes a finding back. An MCP client can ask the same question on demand, for example read with the service name. Prometheus keeps owning alerting. Tares ingests what fired and correlates it with the evidence.

Set it up

1. Start Tares and open the console.

   tares up

2. Add the alerts source under Sources, Add source, Prometheus alerts. Enter the Prometheus url and click Discover to list your alerting rules and severities. The same source as YAML:

   - name: alerts
     connector: prometheus_alerts
     poll: 30s
     config:
       url: http://localhost:9090
       severities: critical,warning   # blank = all
       labels:
         - { name: severity, field: labels.severity, primary: true }
         - { name: state, field: state }

3. Once real alerts have been ingested, open the Fields tab and promote service (or namespace) to a label. Discover only proposes severity and state, because an alert’s other labels exist only while it fires. This service label is what joins the alert to everything else.

4. Add a metrics source with the prometheus connector, and a logs source with docker_logs or OTLP. Give each the same service label as the alerts source.

5. Add a webhook source with a primary service label, event_type: deploy and a text template such as deploy {commit} by {author}: {message}. Creating it gives you a stable ingest URL. Add one line to your CD pipeline, run on every deploy:

   curl -X POST https://<your-tares>/ingest/<ingest_key> \
     -H 'content-type: application/json' \
     -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

   A second webhook source with event_type: config covers config changes the same way.

6. Create a trigger over these sources, keyed by service. You can react to the alert event as the demo does, or threshold a typed field directly, such as max(p99_ms) > 1000 for latency. Choose who gets woken: a Tares agent, your own agent service over a webhook, or a Slack channel.

7. Open Explore, pick the service and check that alerts, metrics, logs and deploys sit on one clock. The full walkthrough, including a demo with a HighLatency rule and a latency scenario you can trigger, is in Build an AI SRE. Field details for the alerts source are on the Prometheus alerts connector page.

What you get

• A diagnosis that cites evidence. The agent sees the alert, the p99 curve, the slow request logs and the deploy in one read, and can point at the lines that support its conclusion.
• Context before anyone looks. A trigger wakes the agent when the alert event arrives, so a finding with the likely cause and a next action can be on the timeline before you open the console.
• A clear start and end. Each alert fires once, and a resolved event lands when it clears, so you can see how the latency episode lined up with a change.
• A loop that improves. The finding is written back to the service’s timeline. The next investigation, by an agent or by you through MCP, starts from that conclusion.

When something else fits

• If your alerts already route through Alertmanager, the alertmanager connector ingests them with a webhook receiver instead. The same service label still joins them to metrics, logs and deploys.
• If Prometheus alerts are all you have and no other source carries service, the agent has only the alert text to go on. Add metrics and a deploy source first, then expect evidence-backed diagnoses.

Related