How do I wake an AI SRE once when alerts from Prometheus report slow database queries?

Tares wakes your AI SRE from a trigger on the prometheus_alerts source, and a per-entity cooldown sets the minimum gap between firings for the same service. The source emits each alert once and a resolved event when it clears, not a copy on every poll. You define the trigger in YAML and subscribe your agent to it.

How it works

Tares polls Prometheus’s own alerts endpoint, so Prometheus decides what counts as an alert and Tares correlates it. This is how the pieces fit together:

• The source emits one event per alert. State is tracked across polls, so an alert is re-emitted only on a state change. Each event has event_type set to the alert name, text set to its summary, its labels, and a state of firing, pending or resolved.
• Resolved events are recorded. When an alert disappears, the source emits a resolved event. It appears under Recent events on the source next to the firing events, so the timeline shows when the slow queries started and when they cleared.
• The trigger evaluates per entity. With group_by: [key_value], each service is evaluated on its own, and key_field names the label that identifies the service.
• The cooldown spaces out firings. cooldown is the minimum gap between firings for the same entity. It stops a flapping condition from firing repeatedly for one service.
• The agent gets the timeline. When the trigger fires, Tares POSTs the entity’s correlated timeline over the context_window to each subscriber, so the agent starts with context. This is the same flow your AI SRE setup uses for any symptom.

Set it up

1. Add a prometheus_alerts source. Discover proposes only the labels that are always present, severity and state, so add service from the Fields tab after real alerts have arrived (see the Prometheus alerts connector page). The sample below declares service with the field labels.service and marks it primary, with severity as a plain label.

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: service, field: labels.service, primary: true }
      - { name: severity, field: labels.severity }
      - { name: state, field: state }

2. Define the trigger from the triggers guide. Replace the alert name with the name of your slow-query rule, since the alert name becomes the event_type.

triggers:
  - name: slow_db_queries
    sources: [alerts]
    key_field: service
    filters:
      - { field: event_type, op: eq, value: PostgresSlowQueries }
      - { field: state, op: eq, value: firing }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: slow_db_queries
      context_window: 15m
    cooldown: 30m

3. Pick a cooldown that matches how long you want to hold off further firings for the same service. The 30m above is a starting value.
4. Wake an agent. A Tares agent is woken when a project wires it to the trigger and you turn it on there. An external agent registers a webhook over MCP:

Use tares to subscribe to the slow_db_queries trigger and deliver to https://my-agent.example.com/hook.

5. Make your endpoint acknowledge with a 2xx. Delivery is at-least-once, so dedupe on dispatch_id. 5xx responses and transport errors are retried with backoff.
6. Check the result. Open the source to see firing and resolved events under Recent events, and open the trigger’s page to see each firing and its delivery status.

What you get

• A trigger that evaluates each service separately, so slow queries on one database do not mix with another service’s alerts.
• A cooldown that keeps a flapping alert from firing repeatedly for the same service within the gap you set.
• A correlated timeline in each dispatch, covering the 15 minutes around the firing, so the agent can start investigating without a follow-up call.
• A source timeline that shows resolved events, so you can see when an alert cleared.
• A logged record of every firing, with per-subscriber delivery status, on the trigger’s page and in the console under Firings.

When something else fits

• If you want Tares to look at the alerts on a clock instead of on a condition, give the trigger a schedule such as condition: {every: 10m}. It fires once per interval and hands over a summary of the window, as described in the triggers guide.
• If your alerts reach you through Alertmanager rather than a Prometheus server Tares can poll, use the alertmanager connector, which receives them at an ingest URL.

Related