What should an AI SRE read next to Alertmanager alert groups for crash loops?

Tares puts the same service’s metrics, logs and recent deploys on one timeline beside Alertmanager alerts, so an AI SRE diagnosing a crash loop cites evidence. Alertmanager tells you which service is unhealthy. The metrics, the container logs and the latest deploy or config change tell you why. Give every source the same service label and your agent reads all of it in one call.

How it works

The Alertmanager connector is a push receiver. You point an Alertmanager webhook_configs URL at the source’s ingest endpoint. Alertmanager has already evaluated, grouped and deduped the alerts, and Tares records what it pushes.

When Alertmanager posts a batch, Tares fans the alerts array out into one event per alert:

• event_type is the alert name.
• text is the alert’s summary.
• The alert’s labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert, including its fingerprint, stays in the payload.

The entity key comes from a label automatically, in the order namespace, service, job, instance. You can set key_label to choose another alert label.

The alert is only the start. For a crash loop, add three more kinds of evidence, all keyed by the same service label so they land on the same timeline:

• Metrics. The prometheus connector polls the metrics you pick by name or label from a Prometheus server, with no PromQL needed.
• Logs. The docker_logs connector tails a running container’s logs. OTLP works if you run an OpenTelemetry pipeline.
• Recent changes. A webhook source with event_type: deploy records each deploy your pipeline posts. A second one with event_type: config covers config changes. Most real root causes are a change, so this is the highest-value source to add.

The AI SRE guide shows the whole loop. A trigger fires when an alert event arrives, an agent wakes and reads the correlated timeline, and it writes a finding back that ties the failure to specific evidence lines.

Set it up

1. Create the alert source. Ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” Or open Sources → Add source → Alertmanager alerts in the console, name it and click Create. The source page shows a copy-paste receiver block with your ingest URL filled in.

2. Add the receiver to your alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

Keep send_resolved: true so the timeline shows when the crash loop clears. If your Tares has auth on (tares up --auth), Alertmanager sends a Bearer Authorization header holding an ingest-scoped API key. Without auth you need nothing extra.

3. Check the source. Send a test batch, then look at Recent events. Fields lists the alert labels, so you can promote namespace or service as the label to correlate on.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

4. Add the logs for the crashing service, with the same service label:

- name: logs
  connector: docker_logs
  poll: 5s
  config:
    container: tares-demo-api-server   # the container name (docker ps)
    labels:
      - { name: service, const: api-server, primary: true }

5. Add a metrics source with the prometheus connector, choosing the metrics for that service and reading service from the metric labels.

6. Record deploys. Create a webhook source with a primary service label and event_type: deploy, then add one line to your CD pipeline:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

7. Connect your agent with tares mcp, then ask it what happened to the service in the last 15 minutes. It answers from the correlated read.

What you get

• One timeline per service, with the Alertmanager alert, the metrics, the container logs and the latest deploy on the same clock.
• Diagnoses that cite evidence. The agent can say “deploy landed at 14:02, restarts and errors began at 14:04” and point to the lines behind it.
• Alertmanager’s grouping, silencing and resolved handling stay where they are. Tares records what Alertmanager sends, one event per alert.
• A resolved event on the timeline when the alert clears, so the agent sees the full firing-to-resolved span.
• A loop that learns. The agent writes its finding back to the timeline, so the next investigation starts from the previous conclusion.

When something else fits

• If you do not run Alertmanager, the prometheus_alerts connector polls the alerts your Prometheus rules fire, every 30 seconds by default. You give up Alertmanager’s grouping and real-time delivery.
• If your logs come from an OpenTelemetry pipeline rather than containers on the host Tares can reach, use OTLP instead of docker_logs.

Related