What should an AI SRE read next to alerts from Prometheus for error spikes?

Tares gives an AI SRE the same service’s metrics, logs and recent deploys on one timeline next to the Prometheus alert. When an error spike fires, the agent cites that evidence instead of reasoning from the alert alone. Key every source by the same service label and the agent reads one correlated timeline.

How it works

The prometheus_alerts connector polls Prometheus’s own alerts endpoint every 30 seconds by default. Each alert becomes one event, with the alert name as the event type, its summary as the text, its labels as Tares labels and a state of firing, pending or resolved. An alert fires once and emits a resolved event when it clears. Prometheus keeps deciding what counts as an alert, and Tares correlates it.

For an error spike, put these signals beside the alert, all keyed by the same service label:

• Metrics. The 5xx rate, p99 latency, database connections and dependency health, from the prometheus connector.
• Logs. The service’s request logs, from the docker_logs connector or from OTLP.
• Recent changes. Deploys and config changes, posted to a webhook source. Most real root causes are a change, so this is the highest-value addition.

Because every source carries service, Tares builds one timeline per service. The agent sees a line like “deploy landed 14:02, errors started 14:04” next to the alert, the error logs and the metric that tripped it. A trigger can wake an agent when an alert event arrives, or you can read the same timeline on demand from an MCP client. The agent then writes its finding back onto the timeline, so the next investigation starts from the previous conclusion.

Set it up

1. Add the alert source. In the console, go to Sources, Add source, Prometheus alerts, enter the url and click Discover. The equivalent YAML is:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

2. Promote the per-alert labels. Discover proposes only severity and state, because an alert’s dynamic labels exist only while it fires. Once real alerts are ingested, open the Fields tab and add service (and namespace if you use it) so the alert joins the service’s timeline.
3. Add the metrics and logs sources for the same service, with service as a label on each. Read service from the metric labels, and use one container or your OpenTelemetry pipeline for logs.
4. Add a webhook source with a primary service label, event_type: deploy and a text template like deploy {commit} by {author}: {message}. Creating it gives you a stable ingest URL. Add one line to your deploy pipeline:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

5. Verify. Open the alert source: Recent events shows one event per active alert. Then open Explore, pick the service, and confirm that metrics, logs, the alert and the deploy sit on one clock.
6. Choose who investigates. Point a trigger at a Tares agent, your own agent service over a webhook, or a Slack channel. To read interactively, run tares mcp and ask your client what happened to the service in the last 15 minutes.

The full walkthrough, including a demo that causes an error spike on purpose, is in Build an AI SRE. Connector options are in Prometheus alerts.

What you get

• A diagnosis that cites evidence: the alert, the error logs, the metric that crossed its threshold and the deploy that preceded it, all on one timeline.
• One alert per state change instead of a copy every poll, plus a resolved event when the spike clears.
• Prometheus stays your alerting system. Nothing is re-thresholded in Tares.
• Findings written back to the timeline, so each investigation starts from the last conclusion instead of from scratch.
• The same evidence for both modes: a trigger that wakes an agent on a firing alert, and an MCP client you query on demand.

When something else fits

If your alerts come from Alertmanager routing rather than from Prometheus rules directly, the alertmanager connector receives them at a Tares ingest URL and keeps the rest of the setup the same. If you only want to react to a metric crossing a limit, you can write a trigger over the typed metric fields, such as an error-rate predicate for 5xx storms, instead of waiting for an alert. If the spike has no related deploy or log source yet, start with the alert and metrics and add the others as you go. The agent works from whatever shares the service label.

Related