Skip to Content
Tares

How do I route alerts from Prometheus to an AI SRE?

Tares routes Prometheus alerts to an AI SRE with the prometheus_alerts connector, which polls the alerts your Prometheus rules have already fired. A trigger on those alert events wakes an agent with one correlated timeline of metrics, logs and alerts for the affected service. You keep alerting in Prometheus, and no Alertmanager and no PromQL are needed.

How it works

The prometheus_alerts connector polls the /api/v1/alerts endpoint of your Prometheus server, every 30 seconds in the example below. Prometheus decides what counts as an alert. Tares ingests it, correlates it and can dispatch it to an agent.

  • One event per active alert. Each event has event_type set to the alert name and text set to its summary. The alert’s labels become Tares labels, plus a synthetic state of firing, pending or resolved.
  • State is tracked across polls. An alert fires once and re-emits only when its state changes. When it clears, you get a resolved event, not a fresh copy every tick.
  • One timeline per service. If your alert source, metrics source and log source all carry the same service label, Tares merges them on one clock. This is the read your agent works from.
  • A trigger does the waking. A trigger watches the sources, keyed by service, and fires when an alert event arrives (sum(alert_active) > 0 over 1m, per service). It wakes the agent you chose with the correlated timeline already attached. The AI SRE guide walks through this chain end to end.

The division of labour is deliberate. Prometheus keeps owning alerting, and Tares does not re-threshold anything.

Set it up

  1. Start Tares and open the console at http://127.0.0.1:8787.

    tares up

  2. Add the alert source. In the console, go to Sources → Add source → Prometheus alerts, enter the url (plus bearer_token, or username and password, if your Prometheus needs auth) and click Discover. Discover lists your configured alerting rules and the severities present. Filter by severity, where blank means all, and turn on include pending if you want alerts still waiting out their for: duration. You can also ask a connected agent: “Use tares to ingest the firing alerts from the Prometheus at localhost:9090.” The resulting source looks like this:

    - name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

    You can also POST /api/sources with the same object.

  3. Verify the source. Open it and check Recent events, which shows one event per active alert and resolved events as they clear. Open Fields to see the labels your alerts carry.

  4. Promote the labels you correlate on. Discover proposes only the labels guaranteed to exist, severity and state. Per-alert labels such as namespace and service exist only while an alert is firing, so add them from the Fields tab once real alerts have been ingested. Use the same service label on your metrics and log sources so everything lands on one timeline.

  5. Create a trigger over those sources, keyed by service, that fires when an alert event arrives.

  6. Pick who gets woken, per trigger:

    • A Tares agent, defined by a prompt you edit for your stack, with no infrastructure to run.
    • Your own agent service over a webhook. Tares POSTs the timeline to your endpoint, so the investigation starts with zero reads.
    • A Slack channel, so every firing and finding lands where your team already looks.

  7. Optionally connect an MCP client so you can ask the same question interactively. Start the endpoint in a second shell:

    tares mcp

The Prometheus alerts connector page has the full configuration table.

What you get

  • A wake-up on every fired alert. When Prometheus fires a rule, the trigger fires and your agent starts working. Nothing polls for it and nobody has to page it.
  • Evidence already attached. The agent starts from one timeline for the service, with the alert next to the metric that tripped it and the logs around it.
  • A conclusion that stays. The agent writes a finding back onto the service’s timeline. The next investigation, yours or an agent’s, starts from that finding instead of from scratch.
  • Clean lifecycle. Alerts fire once, and a resolved event lands when they clear, so the timeline shows when an incident started and ended.
  • No change to your alerting. Your Prometheus rules stay exactly where they are.

When something else fits

  • You route alerts through Alertmanager. Use the alertmanager connector. It receives the alerts Alertmanager sends to a webhook receiver pointed at a Tares ingest URL, so Alertmanager keeps its grouping and routing.
  • Your alerts come from another vendor. Datadog, Sentry, PagerDuty, Grafana Alerting, CloudWatch and Opsgenie alerts reach Tares through the webhook connector and a vendor-side setup. Tares has no connector of its own for those vendors today. The webhook source stores the payload lossless and maps the fields you declare as labels.
  • You want to alert on raw metrics. The prometheus connector reads metrics you pick by name or label, with no PromQL needed. You can then threshold the typed fields in a trigger, for example max(p99_ms) > 1000. If your rules already express what matters, the alerts connector keeps one place for alert logic.
  • You have no Prometheus rules yet. Define alerting rules in Prometheus first. This connector reads only alerts that your rules have fired.

Still stuck? Get help

Last updated on
Quickstart