What should an AI SRE read next to Alertmanager alert groups for a disk filling up?

Tares puts the same service’s metrics, logs and recent changes on one timeline next to the Alertmanager alert, so an AI SRE reads all of them together when a disk fills up. Key every source by the same label, usually service, and the agent’s diagnosis cites evidence lines instead of repeating the alert text. The alert says a disk is filling. The metrics show how fast, the logs show what is writing, and a deploy or config event shows what changed.

How it works

Alertmanager has already evaluated, grouped and deduped the alert. Tares records what Alertmanager pushes and correlates it with your other sources.

• The alert. Alertmanager POSTs a batch. Tares fans the alerts array out into one event each. event_type is the alert name, text is the summary, the alert labels become Tares labels, and a synthetic status is firing or resolved. The full alert, including fingerprint, stays in the payload.
• The entity key. Tares derives it from a label in this order: namespace, service, job, instance. You can declare a primary label or set key_label to choose another.
• The evidence beside it. Every source that carries the same key lands on the same timeline, on one clock. For a disk alert, read three things next to the alert:
  • Metrics for the disk you care about, from the prometheus connector, which lets you pick metrics by name or label with no PromQL.
  • Logs from the same service, from docker_logs or OTLP.
  • Recent changes, such as deploys and config changes, from a webhook source with a primary service label.

The change source matters most. Most real root causes are a change, so a deploy event landing shortly before the disk started growing is often the clue that explains it.

An agent reads the merged timeline through the MCP read surface, or a trigger wakes it with the timeline already attached. It writes its finding back onto the same timeline, so the next investigation starts from that conclusion. See Build an AI SRE for the full walkthrough.

Set it up

1. Start Tares and open the console.

   tares up

2. Create the alert source. In Sources → Add source → Alertmanager alerts, name it and click Create. The source page shows a receiver block with the ingest URL filled in. Add it to your alertmanager.yml and reload Alertmanager. The Alertmanager connector page has the details.

   receivers:
     - name: tares
       webhook_configs:
         - url: http://localhost:8787/ingest/<ingest_key>
           send_resolved: true
   route:
     receiver: tares

   If your Tares has auth on, the receiver sends a Bearer Authorization header holding an ingest-scoped API key. Without auth, you need no key.

3. Add the evidence sources and keep them on the same key. Create a prometheus source for the disk metrics, with service read from the metric labels. Create a docker_logs or otlp source for the service’s logs. Create a webhook source for deploys, with a primary service label and event_type set to deploy. A second webhook source with event_type set to config covers config changes the same way.

4. Send a test alert and check Recent events.

   curl -X POST http://localhost:8787/ingest/<ingest_key> \
     -H 'Content-Type: application/json' \
     -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

   Fields shows the alert labels, so you can promote namespace or service to correlate on.

5. Connect your MCP client so you can investigate the same evidence on demand.

   tares mcp

   Then ask your client what happened to the service in the last 15 minutes. It answers from the correlated read.

6. Add a trigger over the sources, keyed by the same label, and choose who it wakes: a Tares agent, your own agent service over a webhook, or a Slack channel.

What you get

• One timeline per service where the disk alert, the disk metrics, the service’s logs and its recent deploys sit side by side on one clock.
• A diagnosis that ties the failure to specific evidence lines, such as “deploy landed, then the alert fired,” instead of restating the alert.
• Resolved events on the same timeline. With send_resolved: true, the firing to resolved change is recorded, so you can see whether a fix worked.
• Findings written back to the timeline, so the next investigation starts from the previous conclusion.
• Alerting stays in Alertmanager. Tares ingests what fires, correlates it and handles the waking, and does not re-threshold anything.

When something else fits

• If you do not need Alertmanager’s grouping, silencing and real-time delivery, the prometheus_alerts connector polls the alerts that Prometheus rules fire and needs no receiver change.
• If you only want to check one thing by hand, query the metric directly. The timeline pays off when you want metrics, logs and changes read together with cited evidence.

Related