What should an AI SRE read next to Alertmanager alert groups for failed deploys?

Tares puts the failing service’s metrics, logs and recent deploys on one timeline beside its Alertmanager alerts, so an AI SRE reads all of them together. Key every source by the same service label. The agent then cites specific evidence lines, such as a deploy at 14:02 and errors from 14:04, instead of only restating the alert. The alert tells the agent where to look, and the neighboring events tell it why.

How it works

Alertmanager has already evaluated, grouped and deduped the alerts, and with send_resolved it tracks firing to resolved. Tares records what Alertmanager pushes and correlates it with your other sources.

When Alertmanager posts a batch, Tares fans the alerts array out into one event per alert:

• event_type is the alert name.
• The text is the alert’s summary.
• The alert labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert, including fingerprint, stays in the payload.

The entity key comes from a label, checked in the order namespace, service, job, instance, or from a label you declare primary. The optional key_label field lets you pick the alert label to key on.

Every source that carries the same service label lands on the same timeline. For a failed deploy, read these next to the alert:

• Metrics. A prometheus source shows the 5xx rate, latency and dependency health around the alert.
• Logs. A docker_logs or OTLP source shows the error lines the service wrote.
• Deploys and config changes. A webhook source with event_type: deploy records every release. A second one with event_type: config covers config changes. Most real root causes are a change, so this is the highest-value source to add.

A trigger watching these sources wakes an agent when an alert event arrives. The agent reads the correlated timeline, ties the failure to evidence lines and writes a finding back. The next investigation starts from that finding. The AI SRE guide builds this end to end.

Set it up

1. Create the Alertmanager source. In the console go to Sources → Add source → Alertmanager alerts, name it and click Create. Or ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” The source page shows a receiver block with the ingest URL filled in.

2. Add the receiver to your alertmanager.yml and reload Alertmanager. Keep send_resolved: true so resolved alerts reach the timeline.

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If your Tares instance has auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. Without auth, the receiver above is enough.

4. Open Fields on the source page and promote namespace or service to a label, so alerts key on the same service as your other sources.

5. Add the change signal. Create a webhook source with a primary service label read from the payload’s service field, event_type: deploy, and a text template like deploy {commit} by {author}: {message}. Then send one POST from your CD pipeline on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

6. Add a metrics source with prometheus and a log source with docker_logs or OTLP, both keyed by the same service label. Keep your alerting in Alertmanager.

7. Open Explore, pick the service, and check that alerts, metrics, logs and deploys share one clock. What the agent gets shows the exact text the agent reads.

To test the alert path before a real failure, send a sample batch to the ingest endpoint and check Recent events. The Alertmanager page has the test batch to copy.

What you get

• One timeline per service with the alert, the metrics that tripped it, the error logs and the deploy that landed just before.
• Diagnoses that cite evidence lines. An agent can say the deploy landed at 14:02 and errors began at 14:04, because both events sit on the same clock.
• Alertmanager’s grouping, silencing and resolved behavior stay as they are. Tares records what Alertmanager pushes and does not re-threshold it.
• Real-time delivery, with one event per alert and the fingerprint kept in the payload.
• A pull path through an MCP client, which calls read with the service name and answers from the correlated read.
• A push path through a trigger that wakes an agent, which writes a finding back to the timeline for the next investigation.

When something else fits

• If you do not run Alertmanager and want Prometheus rules polled directly, the prometheus_alerts connector reads fired alerts without a receiver. The Alertmanager connector is the better fit when you want Alertmanager’s grouping, silencing and real-time delivery.
• If your deploys are not announced to Tares, the timeline shows the alert and its metrics and logs but no change events. Add the deploy webhook first.

Related