How do I route alerts from Prometheus about failed deploys to an AI SRE?

Tares polls the alerts your Prometheus rules have fired, and a trigger that filters on the failed-deploy alert name wakes your AI SRE only for that symptom. Add a prometheus_alerts source, then a trigger on that source with an event_type filter, and subscribe your agent to the trigger. Every other alert stays on the timeline for correlation but does not wake the agent.

How it works

• The prometheus_alerts connector polls Prometheus’s own alerts endpoint, every 30 seconds by default. Prometheus decides what counts as an alert, and Tares needs no Alertmanager and no PromQL.
• Each active alert becomes one event. Its event_type is the alert name, its text is the alert summary, and its labels become Tares labels plus a state of firing, pending or resolved.
• Tares tracks state across polls. An alert fires once, re-emits only when its state changes, and produces a resolved event when it clears.
• A trigger watches the source. Its filters narrow the stream to one symptom, in this case the alert name of your failed-deploy rule. Its condition and cooldown decide when the match wakes the agent and how often.
• The agent subscribed to the trigger wakes and reads the correlated timeline for the service. The AI SRE guide shows the full chain of source, trigger, agent and finding.

Prometheus keeps owning alerting. Tares ingests what fired, correlates it with your other evidence, and handles the waking. Nothing is re-thresholded.

Set it up

1. Make sure your Prometheus has an alerting rule for failed deploys and note its alert name. This example uses DeployFailed. Replace it with your own rule name.
2. Add the source. In the console go to Sources → Add source → Prometheus alerts, enter the url and click Discover. You can also ask a connected agent: “Use tares to ingest the firing alerts from the Prometheus at localhost:9090.” The resulting source looks like this:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

3. Once real alerts have been ingested, open the source’s Fields tab and promote a per-alert label such as service or namespace. This lets Tares put the alert on the right service’s timeline next to your metrics and logs.
4. Add a trigger on that source. Name it after the symptom and filter on the alert name. The condition below wakes the agent when at least one matching alert arrives within 5 minutes, and the cooldown keeps repeats quiet:

- name: failed-deploys
  sources: [alerts]
  filters:
    - { field: event_type, op: eq, value: DeployFailed }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 5m

5. Subscribe your agent to the failed-deploys trigger. A Tares agent needs only a prompt, so tell it that this trigger means a deploy failed and ask it to tie the failure to specific evidence. To wake your own service instead, use a webhook delivery.
6. Open the source and check Recent events. You should see one event per active alert, with event_type equal to the alert name. Resolved alerts appear there too, with state set to resolved.

To narrow the stream before it reaches any trigger, set severities on the source, for example critical. To also catch alerts still waiting out their for: duration, turn on include_pending.

What you get

• Only failed-deploy alerts wake the agent. Other alerts from the same Prometheus stay on the timeline as context.
• The trigger is named for the symptom, so a firing in the console’s trigger page and Firings list tells you at a glance what woke the agent.
• Each alert fires once, and a resolved event follows when it clears. Your agent is not woken on every poll.
• The agent reads one correlated timeline per service: the alert, the metrics and the logs on one clock. It writes its finding back where the next investigation finds it.
• You can add a deploy signal from your pipeline. Create a webhook source with event_type set to deploy and a primary service label, and post one request per deploy. The agent then sees when the deploy landed relative to the alert.

When something else fits

• If your alerts go through Alertmanager rather than being read from Prometheus, use the alertmanager connector, which receives them at a Tares ingest URL. The same trigger shape applies.
• If you want every alert to wake the agent regardless of symptom, drop the event_type filter and let the trigger watch the whole source, as the demo incident trigger in Prometheus alerts and the AI SRE guide does.

Related