How do I wake my own agent service when an incident starts?

Tares wakes your own agent service through a trigger with a webhook subscription. When the trigger fires, Tares POSTs the entity’s correlated timeline to your endpoint, so the investigation starts with zero reads. You subscribe your URL to the trigger and answer each delivery with a 2xx.

How it works

A trigger watches one or more sources and checks its condition as events arrive. When the condition holds, Tares POSTs one JSON body to every subscribed webhook.

The body carries the incident context, so your agent does not call back to fetch it:

• trigger, kind and key name what fired and for which entity, for example a service.
• fired_at gives the time of the firing.
• payload is the entity’s correlated timeline across the trigger’s sources over context_window: the deploy, the logs, the metrics around the spike.
• dispatch_id identifies the delivery.
• run_ids lists the Tares agent runs the same firing started. It is empty when none started.

Your endpoint can be anything that accepts a POST and starts your agent with the body as context: a small FastAPI or Express route, a serverless function, or a workflow-engine webhook. It must be reachable from the Tares server.

Delivery follows three rules:

• At-least-once. A firing can arrive more than once, so dedupe on dispatch_id.
• Acknowledge with a 2xx. Tares retries 5xx responses and transport errors with backoff. It records a 4xx as a failed delivery and does not retry it.
• Every firing is logged, even with zero subscribers.

The cooldown on the trigger sets the minimum gap between firings for the same entity, so a flapping condition does not wake your agent over and over.

Set it up

1. Define the trigger. Add it on the project’s page, under Triggers → Add trigger, or in the catalog YAML. At least one watched source must produce a numeric field to aggregate.

triggers:
  - name: error_spike
    project: checkout watch   # optional; the Default project when left out
    sources: [checkout_metrics, checkout_logs]
    key_field: service        # optional; default: the first source's primary label
    filters:                  # optional; only these events count
      - { field: env, op: eq, value: prod }
    condition:
      aggregate: max          # count | sum | avg | max | min | any
      field: rate_5xx
      predicate: "> 1.0"      # fire when max(rate_5xx) over the window exceeds 1.0
      window: 1m
      group_by: [key_value]   # evaluated per entity (per service)
    emit:
      kind: error_spike
      context_window: 15m     # how much timeline to hand the agent
    cooldown: 5m              # minimum gap between firings for the same entity

Set context_window to how much timeline your agent needs. Set group_by: [key_value] so each service fires on its own.

2. Build your endpoint. Accept a POST with a JSON body, start your agent with the body as context, and return a 2xx once you have taken it. Dedupe on dispatch_id.

3. Subscribe your URL. From an MCP client, ask your agent to register the webhook with the subscribe tool:

   Use tares to subscribe to the error_spike trigger and deliver to https://my-agent.example.com/hook.

   subscribe(trigger, url) returns a subscription id. You can also register with POST /subscribe. The trigger’s page then lists every subscriber with its delivery counts.

4. Check a delivery. When the trigger fires, your endpoint receives a body like this:

{
  "dispatch_id": "9f2c81d4…",
  "trigger": "error_spike",
  "project": "uc_1a2b3c",
  "projects": ["uc_1a2b3c"],
  "kind": "error_spike",
  "key": "api-server",
  "fired_at": "2026-07-17T09:14:03+00:00",
  "payload": "…the entity's correlated timeline, ready to hand to the model…",
  "run_ids": ["run_4be1c0d2a9f3"]
}

Each dispatch has its own page in the console with the delivery outcomes and the exact payload that was sent.

To receive every trigger of a project at once, an outside agent can join the project instead of subscribing trigger by trigger. The Build an AI SRE guide walks the whole chain on a demo system, from sources to a woken agent.

What you get

• Your agent starts with the deploy, logs and metrics for the incident already in hand, so it needs no follow-up reads before it forms a first hypothesis.
• One correlated timeline per entity, such as a service, across every source the trigger watches.
• Retries on 5xx responses and transport errors, with delivery status per subscriber.
• A full record: firings appear under Firings, on the trigger’s own page, and on the project’s Activity, each with the payload that was sent.
• Control over noise through cooldown and per-entity evaluation.
• A way to pick different incident shapes by changing the aggregate and field, for example max(p99_ms) > 1000 for latency or max(rate_5xx) > 1.0 for an error storm.

Your agent can still read from Tares over MCP when it wants more, and it can write its conclusion back so the next investigation starts from it. The AI SRE guide shows that loop.

When something else fits

If you do not want to run any infrastructure, use a Tares agent instead. It is woken by the trigger in a project that wires it to that trigger, and its whole definition is a prompt. Pick this when a prompt covers your first look at an incident.

If your team wants every firing and finding where it already looks, deliver to a Slack channel. That fits human triage more than agent wake-up.

If you want to investigate on demand rather than be woken, connect an MCP client and ask it to read the service’s timeline. That is the pull path, and it works alongside triggers.

If you want a check on a clock instead of on a condition, give the trigger a schedule with condition: {every: 10m}. It fires once per interval and hands over a summary of the window. See the triggers guide for the details.

Related