What should an AI SRE read next to Alertmanager alert groups for error spikes?

Tares puts the same service’s metrics, logs and recent deploys on one timeline beside the Alertmanager alerts, so an AI SRE reads them together. The alert says an error spike started. The metrics show its size, the logs show the failing requests, and the deploy events show what changed just before. Every source shares one label, such as service, so the diagnosis cites evidence lines instead of the alert alone.

How it works

The Alertmanager connector is a push receiver. Alertmanager has already evaluated, grouped and deduped the alerts. It POSTs a batch, and Tares splits the alerts array into one event per alert. Each event has the alert name as event_type, the summary as text, the alert labels as Tares labels and a synthetic status of firing or resolved. The full alert, including fingerprint, stays in the payload.

Tares keys each event to an entity from a label, checked in this order: namespace, service, job, instance. You can override it with key_label or a declared primary label. Any other source that carries the same label lands on the same timeline. For an error spike, read these next to the alert:

• Metrics. The 5xx rate, p99 latency, database connections and dependency health, from a prometheus source. They show how large the spike is and when it started.
• Logs. The failing requests, from a docker_logs source for a container or an OTLP source for your OpenTelemetry pipeline.
• Deploys and config changes. A webhook source that your CD pipeline posts to on every deploy. Most real root causes are a change, so this is the highest-value addition.

A trigger watching these sources wakes an agent when an alert event arrives. The agent reads the correlated timeline and writes a finding back onto it. The next investigation starts from that conclusion. This is the pattern in Build an AI SRE.

Set it up

1. Create the Alertmanager source. In the console go to Sources → Add source → Alertmanager alerts, name it and click Create. Or ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” The source page shows a receiver block with your ingest URL filled in.

2. Add the receiver to alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If Tares has auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. Without auth, the URL is enough. The source page shows the auth block for a secured instance.

4. Check the alert shape. Send a test batch to the ingest URL, then look at Recent events. The Fields view lists the alert labels, so you can promote namespace or service as the key.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

5. Add the metrics and log sources. Use a prometheus source with your own PromQL and a docker_logs or OTLP source. Read the service label from the metric labels and the log source, so it matches the key on your alerts.

6. Add a webhook source for deploys. Give it a primary service label read from the payload’s service field, event_type: deploy and a text template like deploy {commit} by {author}: {message}. Then add one line to your CD pipeline, run on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

A second webhook source with event_type: config covers config changes the same way.

7. Add a trigger over these sources, keyed by service, that fires when an alert event arrives. Subscribe a Tares agent, your own agent over a webhook or a Slack channel to it.

What you get

• One timeline per service with the Alertmanager alert, the metric that tripped it, the error logs and the deploy that landed minutes before, all on one clock.
• A diagnosis that ties the failure to specific evidence lines, such as “deploy landed 14:02, errors started 14:04”, instead of restating the alert.
• Alertmanager keeps owning grouping, silencing and resolved tracking. With send_resolved on, the resolved event lands on the same timeline, so the agent sees when the spike ended.
• A finding written back to the timeline, so the next investigation, from an agent or from you over MCP, starts from the previous conclusion.

When something else fits

• If you do not need Alertmanager’s grouping, silencing or real-time delivery, the prometheus_alerts connector polls the alerts that Prometheus rules fire, every 30 seconds by default.
• If you only want to chat with the data on demand, connect an MCP client with tares mcp and ask about the service. A trigger is what wakes an agent without anyone asking.

Related