How do I stop an agent alert from notifying me again and again?

Rius stops repeat notifications for a firing as soon as you acknowledge it, and an alert’s Renotify setting at off sends only one notification per firing. To make the alert fire less often, narrow its rule with filters, a higher threshold or a different window. All three are on the alert’s page under Alerts.

How it works

A firing is one occurrence of an alert’s condition. It opens the first time the condition matches and stays open while it keeps matching. What repeats, and when, depends on three things:

• Renotify. Each alert has a Renotify setting. Off, the default, sends one notification per firing. Every N minutes, from 5 to 1440, repeats the notification on that interval while the firing is open and nobody has acknowledged it. Each repeat appears as a row in the firing’s delivery list, and an open firing with renotify on shows when the next reminder goes out.
• Acknowledging. Acknowledging a firing records your name and time and stops any further notifications for it. It does not close the firing. The firing still resolves on its own when the condition clears.
• New firings. Once the condition has stopped matching for two hours, the firing resolves. If the same condition comes back later, that is a new, unacknowledged firing, and it notifies again. Saving a change to an alert also resolves its open firings and starts the rule afresh, so a condition that still holds opens a new firing and notifies once more.

So repeated messages come from one of two causes: Renotify is set to repeat, or the condition keeps clearing and returning, which produces a stream of separate firings.

Set it up

1. Check the Renotify setting. Open the alert’s details page and look at Renotify. If it repeats every N minutes and you do not want reminders, set it to off. Creating and editing alerts is an admin action. Saving the change resolves the open firings, and a condition that still holds opens one new firing under the new setting.

2. Acknowledge the firing. Click Ack on the firing’s row in the Firings page, or acknowledge it from its own page. Members can do this too. Use Unacked only to list the firings nobody has handled yet.

3. Narrow the rule so it fires less. Open the alert and edit its rule. For a custom alert, these changes reduce firings:

   • Add a filter so only the spans you care about feed the rule. Filters apply before aggregation and cover Service, Agent, Step, Model, Span kind, Operation, Session and Instance.
   • Raise the threshold, so the condition matches less often.
   • Add a second rule. Rules combine with AND only, so every rule must hold for the alert to fire.
   • Adjust the window, which can be anywhere from 5 minutes to 24 hours and is shared by all rules.

   A rule that fires on a noisy cost average might become:

FILTERS   Service = checkout-agent          (optional, applied before aggregation)
RULES     avg(Cost) > $1.00
   AND    count(Errors) > 10
WINDOW    30 minutes                        (shared by all rules)

4. Tune a pre-defined alert. For Consistent Error Pattern, Swallowed Error or Tool Loop, open the alert’s details page, change its thresholds, or turn it off. Swallowed Error’s errored-span threshold is set only through the MCP server.
5. Pause everything when you need silence. An admin can turn off the Alerting switch at the top of the Alerts page. It pauses detection and notifications for every alert without changing each alert’s own state, and turning it back on restores them as configured. To stop one alert, disable it. Its open firings stay as they are until you enable it again.

What you get

• One notification per firing when Renotify is off, or reminders only until someone acknowledges.
• A firing that records who acknowledged it and when. Acknowledged firings stay in the list, dimmed, so the history remains auditable.
• Fewer firings from a rule that matches only the spans, models or services you care about.
• A count of open, unacknowledged firings on the Alerts entry in the sidebar, so you can see what still needs a look without relying on repeated messages.
• Firings that record the observed value of every rule against its threshold, so you can see which condition tripped and set the threshold accordingly.

When something else fits

If the alert is firing because something is really wrong, fix the cause or acknowledge the firing rather than loosening the rule. If you want quiet for a planned migration or a noisy incident, the workspace-wide Alerting switch is simpler than editing rules one by one. If you want to be told when ingestion stops, a Traces count < 1 rule over a generous window does that, so keep a rule like it.

Related