How do I wake an AI SRE once when Alertmanager alert groups report a disk filling up?

Tares wakes your AI SRE once per disk when you send Alertmanager alerts to an alertmanager source and put a trigger with a per-entity cooldown on top. A repeating alert inside the cooldown window starts no second investigation. With send_resolved on, resolved alerts still land on the timeline as events with status resolved. This is one pattern in AI SRE.

How it works

• Alertmanager groups and dedupes alerts, then POSTs a batch to the ingest URL of your alertmanager source. Tares turns each alert in the batch into one event. The event_type is the alert name, text is the summary, the alert labels become Tares labels, and a synthetic status label is firing or resolved. The full alert, including its fingerprint, stays in the payload. See Alertmanager.
• The source derives an entity key from a label, trying namespace, then service, job, then instance. For a disk alert, set key_label to the label that names the disk’s host, such as instance, so each host is its own entity.
• A trigger watches the source and checks its condition as events arrive. With group_by: [key_value] it evaluates per entity. The cooldown is the minimum gap between firings for the same entity, so Alertmanager’s repeated notifications for one full disk fire one investigation, and a second disk still fires on its own.
• When the trigger fires, Tares POSTs the entity’s correlated timeline to each subscriber, or starts the Tares agents the project wires to that trigger.
• Resolved alerts arrive only when send_resolved is true in the receiver. They are stored as events with status resolved, so the timeline shows when the disk recovered.

Set it up

1. Create the source. Ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” Or use Sources → Add source → Alertmanager alerts in the console. To key on the host, set key_label:

- name: alerts
  connector: alertmanager
  config:
    key_label: instance

2. Add the receiver to your alertmanager.yml with the ingest URL, keep send_resolved: true, and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

The sender uses a Bearer Authorization header holding an ingest-scoped API key, needed only when Tares has auth on. The source page shows the receiver block with the auth part filled in for you.

3. Add a trigger on the project’s page, under Triggers → Add trigger, or in the catalog YAML. Use the alert name from your own Alertmanager rule in place of DiskFillingUp:

triggers:
  - name: disk_filling_up
    sources: [alerts]
    key_field: instance
    filters:
      - { field: event_type, op: eq, value: DiskFillingUp }
      - { field: status, op: eq, value: firing }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: disk_filling_up
      context_window: 15m
    cooldown: 1h

4. Wake an agent. A Tares agent is woken by the trigger in a project that wires it to that trigger, once you turn it on there. An external agent registers a webhook with the subscribe tool: “Use tares to subscribe to the disk_filling_up trigger and deliver to https://my-agent.example.com/hook.” Your endpoint must be reachable from the Tares server and should answer with a 2xx. Deliveries are at-least-once, so dedupe on dispatch_id.

5. Check the result. Send a test batch to the ingest endpoint, look at Recent events, and open Fields to confirm the label you key on is there. Every firing is logged under Firings, even with zero subscribers.

What you get

• One investigation per disk per cooldown window, however often Alertmanager re-notifies. Pick a cooldown that matches how long you want to wait before a still-full disk may wake the agent again.
• A dispatch that carries the entity’s correlated timeline over context_window, so the agent starts with the alert, the logs and the deploys around it and makes no follow-up call.
• Resolved events on the source with status resolved, so you can see in Recent events when the disk recovered and compare it with the firing history.
• A record of every firing and every delivery outcome on the trigger’s page, so you can check that a repeat was held back by the cooldown.

The status filter in the example keeps resolved events from counting toward the trigger, and filters also narrow what is handed over. If you want resolved alerts in the dispatch payload too, remove that filter and use the alert name filter alone. A resolved event after the cooldown has passed can then fire the trigger.

When something else fits

• If you want Prometheus rules polled directly and do not need Alertmanager’s grouping, silencing or resolved handling, use the prometheus_alerts connector instead.
• If you want a periodic look at the sources and not a reaction to one alert, give the trigger a schedule such as condition: {every: 10m}, as described in Watch with triggers.

Related