What should an AI SRE read next to Alertmanager alert groups for slow database queries?

Tares puts the same service’s metrics, logs and recent deploys on one timeline beside your Alertmanager alerts, so an AI SRE reads all of it together. The alert says a query is slow. The metrics show when it started, the logs show what the service did, and the deploy or config events show what changed just before. Key every source by the same service label and the agent cites those lines in its diagnosis.

How it works

Alertmanager has already evaluated, grouped and deduplicated the alerts. When you point a webhook receiver at a Tares alertmanager source, each alert in a batch becomes one event. The event type is the alert name, the text is its summary, and its labels become Tares labels plus a synthetic status of firing or resolved. The full alert, including its fingerprint, stays in the payload.

The entity key comes from a label, checked in this order: namespace, service, job, instance. If you declare a primary label, that one wins. Any other source that carries the same label lands on the same timeline. For a slow-query alert, read these next to it:

• Metrics for the same service. A prometheus source brings in the series you pick, such as request latency, 5xx rate and database connections. They show when the slowdown began and whether connections were running out.
• Logs for the same service. A docker_logs source tails the container’s request logs. An OTLP source does the same through your OpenTelemetry pipeline. They show which requests were slow and what errors came with them.
• Deploys and config changes. A webhook source that your pipeline posts to on every deploy puts the change on the timeline. A second webhook source can carry config changes. Most root causes are a change, so this is the highest-value addition.

All of it merges on one clock per service. The AI SRE guide builds this end to end. An agent reads the timeline through MCP, or a trigger wakes an agent when an alert event arrives.

Set it up

1. Create the Alertmanager source. In the console go to Sources → Add source → Alertmanager alerts, or ask a connected agent to create an Alertmanager source and give you the webhook URL. The source page shows a receiver block with the ingest URL filled in.

2. Add the receiver to alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

Keep send_resolved: true so the timeline records when the alert clears. If your Tares instance has auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. Without auth you don’t need it.

3. Check Fields on the source and promote namespace or service as the label to correlate on. If you want to pick a different alert label, set key_label.

4. Add the metrics source with the same service label. This example reads the service label from the metric labels:

- name: metrics
  connector: prometheus
  poll: 10s
  config:
    url: http://localhost:9090
    queries:
      - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
        by_name: true
    labels:
      - { name: service, field: metric.service, primary: true }
      - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

5. Add the logs source, either docker_logs or OTLP, labeled with the same service. Then create a webhook source for deploys with a primary service label and event_type: deploy. Post to its ingest URL from your CD pipeline:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

6. Add a trigger over these sources, keyed by service, that fires when an alert event arrives. Choose who it wakes: a Tares agent, your own agent over a webhook, or a Slack channel.

What you get

• One timeline per service. The alert group, the latency and connection metrics, the request logs and the deploy events sit on the same clock. You see “deploy landed at 14:02, slow queries from 14:04” in one read.
• A diagnosis that cites evidence. The agent ties its conclusion to specific lines, such as the alert, the metric that moved, the log errors and the change that preceded them.
• Alerting stays where it is. Alertmanager keeps owning grouping, silencing and resolved handling. Tares records what it sends and correlates it with your other sources, so nothing is re-thresholded.
• Findings that carry over. The agent writes its finding back onto the service’s timeline. The next investigation, interactive or triggered, starts from that conclusion instead of from scratch.
• Interactive reads too. Connect an MCP client and ask what happened to the service in the last 15 minutes. It answers from the same correlated read.

When something else fits

• If you don’t need Alertmanager’s grouping, silencing and real-time delivery, the prometheus_alerts connector polls the alerts that Prometheus rules fire. The Alertmanager connector page explains when each fits.
• If the slow queries live in a table you can read, the postgres connector brings in new and changed rows on a cursor column. Use it next to the metrics and logs, not in place of them.

Related