How do I line up Sentry issue alerts with Prometheus metrics on one timeline?

Tares puts Sentry issue alerts and Prometheus metrics on one timeline when both sources carry the same service label. Sentry reaches Tares through the webhook connector and a Sentry integration setup, and Prometheus uses the prometheus connector. Open Explore, pick the service, and the alert and the metrics that led to it appear merged on one clock.

How it works

Tares joins events on a shared key. Every source declares labels, and one label is marked primary. When a Prometheus source and a Sentry source both use service as the primary label with the same value, their events land on the same entity, and Explore shows them together.

• Metrics: the Prometheus connector polls the metrics you pick by name or label. Each series becomes one metric event per poll tick, and the numeric value is stored as a number-typed value label. You write no PromQL.
• Sentry alerts: Tares has no connector of its own for Sentry today. Sentry reaches Tares through the webhook connector and a setup on the Sentry side. Sentry posts each alert to the source’s ingest URL, and Tares stores the payload lossless.
• Authentication: Sentry signs its deliveries. The signature is the credential, so the Tares source needs no API key. You give the source the signature scheme hmac_sha256, the header Sentry-Hook-Signature and the integration’s Client Secret as the signing secret.
• Flat fields only: event_type_field and text_template read top-level payload fields. action is top level, so it works as the event type field.

This is the same per-service correlation described in the AI SRE guide: the alert shows what fired, and the metric shows how the service got there.

Set it up

1. Start Tares and open the console.

   tares up

2. Add the Prometheus source. In Sources → Add source → Prometheus, enter the server URL, pick the metrics you want and confirm. The console generates a config like this. Make sure the primary label is service.

   - name: metrics
     connector: prometheus
     poll: 10s
     config:
       url: http://localhost:9090
       queries:
         - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
           by_name: true
       labels:
         - { name: service, field: metric.service, primary: true }
         - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

3. Add the Sentry source. In Sources → Add source → webhook, declare a primary service label. Set the value with const to the service this Sentry integration covers, so it matches the Prometheus value. Add the signing options, then save and copy the ingest endpoint from the source page.

   - name: sentry-alerts
     connector: webhook
     config:
       event_type_field: action
       text_template: '{action}'
       signature_scheme: hmac_sha256
       signature_header: Sentry-Hook-Signature
       signing_secret: YOUR_CLIENT_SECRET
       labels:
         - { name: service, const: api-server, primary: true }

4. In Sentry, go to Settings, Developer Settings and add an integration. Set its Webhook URL to the Tares ingest endpoint.

5. Subscribe the integration to the resources you want, such as issues or errors, and turn on Alert Action so alert rules can use it.

6. Copy the integration Client Secret into the Tares source as the signing secret, replacing the placeholder above.

7. Trigger an issue alert in Sentry. Open the Sentry source, check Recent events, and use Fields to confirm action arrives.

8. Open Explore and pick the service. The Sentry alert and the Prometheus metric events for that service now sit on one timeline.

What you get

• One timeline per service, with the Sentry issue alert next to the metric events that preceded it.
• Metric values stored as numbers, so a trigger can aggregate them with max, avg or sum.
• The full Sentry payload stored lossless, so detail beyond your labels stays readable.
• A correlated read that a connected agent can use, so it starts from the alert and the metrics together instead of querying two tools.
• Signed deliveries from Sentry that Tares accepts without a Tares API key on the source.

When something else fits

• If your alerts come from Prometheus rules rather than Sentry, use the prometheus_alerts or alertmanager connector instead of a webhook source.
• If you only need Sentry events and no metrics, a webhook source alone is enough.
• If the metric you need is a derived rate or a p99, use a computed series, an advanced queries row with an explicit event_type and field.

Related