Which Grafana Loki logs does an AI SRE search when Sentry issue alerts fire?

Tares gives the AI SRE the Loki logs of the alerting service, laid out on one timeline next to the Sentry event, so it reads the minutes around the first event instead of searching everything. Sentry reaches Tares through the webhook connector and a Sentry setup, and Loki is polled with a LogQL query you choose. A trigger wakes the agent when a Sentry alert lands, and the agent reads that one service’s timeline.

How it works

Tares puts every source that carries the same label, such as service, on one timeline with one clock. The Build an AI SRE guide describes this per-service correlation, and it is what narrows the search.

• Loki side. The loki source polls a LogQL query every 5 seconds by default, including on Grafana Cloud. The query is where you scope the logs. Point one source at one service’s logs and label it with that service. It also takes match and drop options to keep or discard lines.
• Sentry side. Sentry reaches Tares through the webhook connector and a Sentry integration. Tares has no connector of its own for Sentry today. Sentry POSTs each delivery to the source’s ingest URL, and Tares stores the payload lossless.
• Shared label. Give the Sentry source and the Loki source the same service label. The Sentry event and the service’s log lines then sit on one timeline.
• Wake and read. A trigger over the Sentry source fires when an alert event arrives. It wakes an agent that reads the correlated timeline for that one service, not the whole log store. A connected client does the same read with read and a service, for example the last 15 minutes of api-server.

The agent’s whole definition is a prompt. To anchor the search, tell it to start at the first Sentry event and work through the log lines around it.

Set it up

1. Start Tares and install it if you have not.

tares up

2. In Sentry, add an integration under Settings, Developer Settings. Set its Webhook URL to the Tares ingest URL, subscribe it to issues or errors, and turn on Alert Action so alert rules can use it.
3. Create a webhook source for the service. The Sentry payload has no service field, so give the label a fixed value. Use one source per service and point that service’s Sentry integration at its ingest URL. Copy the integration Client Secret into signing_secret.

- name: sentry-checkout
  connector: webhook
  config:
    event_type_field: action
    signature_scheme: hmac_sha256
    signature_header: Sentry-Hook-Signature
    signing_secret: <client-secret>
    labels:
      - { name: service, const: checkout, primary: true }

The signature is the credential for this source, so it needs no Tares API key. The signing options are described in the release notes, not on the webhook connector page.

4. Add a loki source with your Loki URL and a LogQL query for that same service. Label it service with the same value, checkout. Add match or drop if you want to trim lines.
5. Add a trigger over the Sentry source, keyed by service.

triggers:
  - name: sentry-issue
    sources: [sentry-checkout]
    key_field: service
    condition: {aggregate: count, predicate: "> 0", window: 5m}
    cooldown: 30m

6. Create a Tares agent on the trigger. In its prompt, ask it to start at the first Sentry event for the service and read the Loki lines around it. Open Explore, pick the service, and check that the Sentry event and the log lines appear together.

What you get

• The agent starts from the alerting service’s logs and the Sentry event on one clock, so it checks what happened around the first event.
• Other services’ logs stay out of the read unless they share the same label.
• The agent writes its conclusion back as a finding on that service’s timeline. The next investigation, yours or another agent’s, starts from it.
• Your Sentry alert rules keep owning alerting. Tares ingests what fired, correlates it with the logs and handles the waking.

When something else fits

If Sentry is noisy and you want a cheap first look before a full investigation, use the pattern in Watch, then escalate, where a triage agent hands one entity to a main agent. If you want to browse Loki by hand, Grafana’s own tools do that directly, while Tares is for having the agent read the correlated evidence.

Related