What should an AI SRE read next to alerts from Prometheus for a disk filling up?

Tares puts the disk alert on one timeline with the same service’s metrics, logs and recent deploys or config changes, so an AI SRE reads all of them together. The alert from the prometheus_alerts connector is the starting point, and the correlated evidence beside it is what the diagnosis cites. Every source carries the same service label, so one read returns the whole picture for the service whose disk is filling.

How it works

The prometheus_alerts connector polls Prometheus’s own alerts every 30 seconds by default. Prometheus keeps deciding what counts as an alert. Tares ingests what fired and correlates it.

Each alert becomes one event:

• event_type is the alert name.
• text is the alert’s summary.
• The alert’s labels become Tares labels, plus a state of firing, pending or resolved.

Tares tracks state across polls. A disk alert fires once, re-emits only when its state changes, and produces a resolved event when it clears. You get no repeated copy on every tick.

The alert is the start of the investigation. For an AI SRE, the next things to read are the other signals that share the service’s service label, merged on one clock, as described in Build an AI SRE:

• Metrics from the prometheus connector, for the same service. You choose them by name or label, with no PromQL needed. They show how the disk metrics moved before and after the alert.
• Logs from docker_logs for a container, or from OTLP if you run an OpenTelemetry pipeline. They show what the service was doing while the disk filled.
• Deploys and config changes from webhook sources. Most real root causes are a change, so a deploy event landing just before the alert is often the lead.
• Earlier findings that agents wrote onto the same timeline, so a new investigation starts from the previous conclusion.

Because the alert, the metrics, the logs and the changes sit on one timeline, the agent can tie its conclusion to specific evidence lines.

Set it up

1. Create the alerts source. In the console go to Sources, Add source, Prometheus alerts, enter the url and choose Discover to see your alerting rules and severities. Or use this config:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

2. Promote the service label. Discover proposes only the labels that are always present, severity and state. An alert’s own labels, such as namespace or service, exist only while it fires. Once a disk alert has been ingested, open the Fields tab and promote service so you can correlate on it.
3. Give the metrics and log sources the same service label. Use prometheus for the service’s disk metrics and docker_logs or OTLP for its logs. Matching labels are what put them on the same timeline as the alert.
4. Add the change signal. Create a webhook source with a primary service label, event_type: deploy and a text template such as deploy {commit} by {author}: {message}. Then send one request from your CD pipeline on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

5. Connect an MCP client so you can read the same evidence yourself. Start the endpoint with tares mcp, then point your client at it:

tares mcp

6. Verify. Open the alerts source: Recent events shows one event per active alert, and Fields shows the labels the alerts carry. Then open Explore, pick the service, and check that alert, metric, log and deploy events appear together.
7. Add a trigger over these sources, keyed by service, that fires when an alert event arrives, and choose who wakes: a Tares agent, your own agent service over a webhook, or a Slack channel.

What you get

• A disk alert that arrives with context. The agent sees the alert, the service’s metrics, its logs and its latest deploys in one read, instead of the alert alone.
• A diagnosis that cites evidence. The agent can point to the log lines, the metric movement and the change that line up with the alert.
• A single firing per alert, plus a resolved event on the timeline when the disk pressure clears.
• A written-back finding. The agent records what broke, since when, the likely cause tied to evidence lines, and a next action. The next investigation, yours or an agent’s, starts from that conclusion instead of from scratch.
• The same view for you. Your MCP client reads the same correlated timeline when you ask what happened to the service in the last 15 minutes.

When something else fits

• If you already route alerts through Alertmanager and want one event per routed alert, the alertmanager connector takes them from a webhook receiver instead. Your alerts then stay where they are and Tares ingests what fires.
• If you only need to know that the disk alert fired, and not why, the alerts source on its own is enough. Add metrics, logs and deploy events when you want the agent to explain the cause.

Related