How do I stop a flood of Sentry issue alerts from waking an AI SRE again and again?

Tares lets you put a cooldown, filters and a key on the trigger that watches your Sentry source, so a burst of issue alerts wakes your AI SRE once per entity instead of once per alert. Sentry reaches Tares through the webhook connector and a vendor setup, and Tares has no connector of its own for Sentry today. The trigger counts events, filters out the ones you do not care about, and waits out its cooldown before it fires again for the same entity.

How it works

Sentry posts each issue alert to the ingest URL of a Tares webhook source. The signature Sentry adds to each delivery is the authentication. Tares stores the payload lossless, and a trigger watches the source as events arrive. Three trigger settings keep a storm from waking your agent repeatedly:

• Filters narrow which events count. Each filter is a field, an op and a value. The field can be a label on the source or one of event_type, source, text and key_value. For Sentry, set event_type_field to action, which is a top-level field, so you can filter on the action, for example only created.
• Key (key_field) names the entity across the trigger’s sources. With group_by: [key_value], the condition is evaluated per entity, so each entity fires independently.
• Cooldown is the minimum gap between firings for the same entity. A flapping condition cannot fire again for that entity until the gap has passed.

The AI SRE guide shows how the woken agent uses the trigger’s dispatch. Each firing carries the entity’s correlated timeline, so one wake-up gives the agent the context it needs.

Set it up

1. In Sentry, add an integration under Settings, Developer Settings. Set its Webhook URL to the Tares ingest URL. Subscribe it to the resources you want, such as issues, and turn on Alert Action to use it from alert rules.
2. Copy the integration Client Secret. Create a webhook source in Tares with signature_scheme set to hmac_sha256, signature_header set to Sentry-Hook-Signature and signing_secret set to the Client Secret. The signing options are described in the release notes. Declare a label for the field you want to key on and mark it primary.

- name: sentry-issues
  connector: webhook
  config:
    event_type_field: action
    signature_scheme: hmac_sha256
    signature_header: Sentry-Hook-Signature
    signing_secret: <client-secret>
    labels:
      - { name: installation, field: installation.uuid, primary: true }

3. Open Fields on the source page and check which payload fields arrive and how often. Pick a reliable field for the key. If a field in your payloads names the entity you want to page on, add a label for it and make it the primary label.
4. Add a trigger that filters to the actions you care about, evaluates per entity and sets a cooldown:

triggers:
  - name: sentry_issue_storm
    sources: [sentry-issues]
    key_field: installation
    filters:
      - { field: event_type, op: eq, value: created }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    cooldown: 30m

5. In the project that uses the trigger, turn on the agent you want woken, or subscribe an external agent with the subscribe tool. Then send a test alert from Sentry and check Recent events on the source and Firings on the trigger.

With installation as the key, all issue alerts from one Sentry integration share one entity and one cooldown. Choose a different label as the key when you want separate entities to fire on their own.

What you get

• One wake-up per entity per cooldown period, not one per Sentry alert.
• Only the actions you filter for count, so other Sentry events do not wake the agent.
• Entities fire independently once you key the trigger on a label that names them.
• Every firing is logged on the trigger’s page and the project’s Activity, even with zero subscribers, so you can see how many storms the cooldown absorbed.
• Delivery is at-least-once, so your agent should dedupe on dispatch_id.
• Every Sentry payload stays stored lossless on the source, so the timeline handed to the agent keeps the original detail.

When something else fits

• To look at Sentry on a clock instead of on a condition, give the trigger a schedule such as condition: {every: 10m}. It fires once per interval with a summary of the window.
• If you want the agent to see every single alert, leave the cooldown short and rely on the correlated timeline instead.

Related