What should an AI SRE read next to alerts from Prometheus for slow database queries?

Tares lets an AI SRE read the slow-query alert together with the same service’s metrics, logs and recent deploys on one timeline. The prometheus_alerts connector brings in the alert, and sources for metrics, logs and changes that share a service label sit beside it. The diagnosis can then cite specific evidence lines instead of repeating the alert text.

How it works

The prometheus_alerts connector polls Prometheus’s own alerts endpoint every 30 seconds by default. Prometheus keeps owning the alerting rule, for example one for slow database queries. Tares ingests what fired, and nothing is re-thresholded.

Each active alert becomes one event. Its event_type is the alert name, its text is the summary, and its labels become Tares labels plus a state of firing, pending or resolved. State is tracked across polls, so an alert fires once and a resolved event follows when it clears.

The alert is only the start. When other sources carry the same service label, Tares builds one timeline per service. The AI SRE reads these next to the alert:

• Metrics. A prometheus source for the service, such as database connections, latency and dependency health, picked by name or label. These show whether the slowdown matches a rise in load or in connections.
• Logs. A docker_logs source for the service’s container, or OTLP from your OpenTelemetry pipeline. These show the requests and errors around the moment the alert fired.
• Recent changes. A webhook source that your deploy pipeline posts to on every deploy, with event_type: deploy. Most real root causes are a change, so a deploy landing shortly before the alert is often the lead.

If you route alerts through Alertmanager, use the alertmanager connector instead. Alertmanager sends the alerts it routes to a webhook receiver pointed at a Tares ingest URL.

Set it up

1. Install Tares and start it locally, then open the console.

tares up

2. Add the alert source. In Sources → Add source → Prometheus alerts, enter the Prometheus url and click Discover. It lists your alerting rules and the severities present. You can also ask a connected agent to ingest the firing alerts from your Prometheus. The source looks like this:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

3. Let a few alerts arrive, then open the source’s Fields tab. An alert’s dynamic labels, such as namespace and service, only exist while it is firing. Promote service there so the alert can be correlated with other sources.
4. Add the evidence sources and key all of them by the same service label: a prometheus source for the database and service metrics, a docker_logs or OTLP source for logs, and a webhook source for deploys and config changes with a primary service label.
5. Open Explore and pick the service. Metrics, logs, deploys and the alert appear merged on one clock. What the agent gets shows the exact text an agent reads.
6. Add a trigger over the sources, keyed by service, that fires when an alert event arrives, and subscribe an agent to it. The AI SRE guide walks through the full chain, including a Tares agent whose prompt asks it to tie the failure to specific evidence lines. Connect an MCP client the same way to investigate on demand.

The Prometheus alerts page lists every configuration field, including severities and include_pending. Set include_pending to also ingest alerts that are still waiting out their for: duration.

What you get

• A diagnosis tied to evidence. The agent sees the slow-query alert, the metrics that moved, the log lines around it and the deploy that landed beforehand, all in time order.
• One firing per alert. The trigger wakes the agent on the alert event, and a resolved event marks the end of the incident on the timeline.
• A finding written back to the service’s timeline. The next investigation, whether by an agent or by you through an MCP client, starts from that conclusion instead of from scratch.
• Alerting that stays where it is. Prometheus decides what counts as slow, and Tares correlates and dispatches.

When something else fits

• If you route alerts through Alertmanager, the alertmanager connector takes what it sends, one event per alert, at a Tares ingest URL.
• If you only want to be told when queries are slow, Prometheus alerting alone is enough. The timeline helps when someone or something has to explain why.

Related