Watching

Two agents share the work.

The watcher checks every service on a schedule. It compares what each one sends with what is usual, and raises what looks wrong.
The investigator looks into what the watcher raises. It reads the service’s whole timeline, its runbook and what earlier investigations found, and writes the incident note.

Set them up with Guided setup, as in Get started.

Settings

Change both under Settings in the console.

Group	Setting	What it does
Watching	What we call what is watched	The name used in messages and on the home page.
Watching	How often it checks	From every minute to once a day.
Watching	Model, budget, when the budget resets	The watcher’s model and spending limit, monthly or never reset.
Watching	What it compares	The labels it groups events by before it decides what is unusual, such as `service`.
Investigating	Model, budget, when the budget resets	The same, for the investigator.
Investigating	The wait between investigations	How long before the same trouble is looked into again.
Notes	The Slack channel	Where incident notes go.
Notes	The channel for services nobody watches	Where Tares says a service is no longer watched.

To change the watcher’s wording, open its setup from Settings, Watching.

When nobody is watching

A service is watched while a watcher covers it and can run. When none can, the home page says so, why, and since when, and Tares posts it to Slack once, and again when watching starts. The reasons:

no watcher is set up yet
the project is paused, or the watcher is turned off
the budget is used up
the daily limit of runs is reached
no model is set up
the schedule has stopped

When the budget ran out, the home page offers to raise it, which turns the watcher back on.

Worth raising

Each note asks Was this worth raising?, in Slack and on the incident’s page. Answer yes or no. Each watcher’s hit rate shows under Settings, Watching, and the watcher reads its recent “not worth it” cases on every check, so it raises fewer of them.