How do I route alerts from Prometheus about error spikes to an AI SRE?

In Tares, ingest your Prometheus alerts with the prometheus_alerts connector, then add a trigger that filters to the error spike alert and wakes your agent only for it. Other alerts still land on the timeline but do not wake anything. Name the trigger after the symptom so each firing is tied to error spikes.

How it works

The prometheus_alerts connector polls Prometheus’s own alerts endpoint, by default every 30 seconds. Prometheus keeps deciding what counts as an alert. Tares only ingests what its rules have fired.

• Each active alert becomes one event. The event_type is the alert name, the text is the alert summary, and the alert labels become Tares labels, plus a state label of firing, pending or resolved.
• State is tracked across polls, so an alert fires once and emits a resolved event when it clears. You do not get one copy per tick.
• The source ingests every alert it is allowed to see. The severities key narrows by severity, not by symptom.
• A trigger narrows by symptom. It watches the alerts source, filters on event_type, and wakes the agent subscribed to it only when an event matches.

Because the event_type is the alert name, the filter is one line: match the name of your error spike rule. In the demo from the AI SRE guide, that rule is HighErrorRate. Use the alert name from your own rule.

Set it up

1. Start Tares if it is not running.

tares up

2. Add the alerts source. In the console go to Sources → Add source → Prometheus alerts, enter your Prometheus url and run Discover. Or use this definition, which also works with POST /api/sources. Keep severities blank to take every severity, or list the ones you want.

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

3. Open the source and check Recent events. You should see one event per active alert. Open Fields to confirm the labels your alerts carry, such as namespace or service, and promote them if you want to correlate on them. If your Prometheus needs auth, set bearer_token, or username and password.

4. Add a trigger that names the symptom and matches only the error spike alert while it is firing. The filter on state keeps the resolved event from waking the agent.

- name: error-spike
  sources: [alerts]
  filters:
    - { field: event_type, op: eq, value: HighErrorRate }
    - { field: state, op: eq, value: firing }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 5m

The condition fires when at least one matching alert arrives within 5 minutes. The cooldown stops repeat firings from waking the agent again right away. To match more than one alert name for the same symptom, use op in with a list of values, as a trigger on a list of repos does.

5. Subscribe your agent to the trigger. A Tares agent subscribes to the trigger directly, and its prompt can tell it that it was woken for an error spike. You can also deliver to your own agent service over a webhook, or to a Slack channel. The AI SRE guide shows each option.

6. Test it. Cause an error spike in a test system, wait for the alert to fire and for the next poll, then open the trigger’s page. The firing appears there under the error-spike name.

What you get

• Only error spike alerts wake the agent. Latency, dependency and other alerts still arrive on the timeline as evidence, but they do not start a run.
• A resolved event does not start a second investigation, because the trigger requires state firing.
• The trigger name says which symptom woke the agent. Firings are listed on that trigger’s page under its name, so you can see how often error spikes fire and what each one woke.
• The agent reads one correlated timeline, with the alert next to the metrics and logs that carry the same labels, and not only the alert on its own.
• Prometheus keeps owning alerting. Tares does not re-threshold anything. It ingests what fired, correlates it and handles the waking.

For another symptom, add another trigger on the same alerts source, with its own name and its own alert name in the filter. One source can feed several triggers.

When something else fits

• If your alerts already pass through Alertmanager, the alertmanager connector takes them from a webhook receiver instead of polling Prometheus. The same trigger approach applies.
• If you want the agent woken for every alert, skip the event_type filter and keep a condition on any alert event. This is how the demo in the AI SRE guide works.
• If the signal does not come from Prometheus rules, such as deploys or CI events, use the webhook connector and filter the same way on its event type.
• If you want a threshold on the raw metric, such as an error rate predicate, use the prometheus connector and put the condition in the trigger. Use the alerts connector when you already trust your Prometheus rules and want to keep them where they are.
• Details for this connector, including all configuration keys and discovery, are on the Prometheus alerts page.

Related