How do I secure the connection that carries Grafana alert rules into Tares?

Tares secures the Grafana connection with an ingest-scoped API key that Grafana sends as a Bearer Authorization header on its webhook contact point. The key is needed only when Tares has auth on. Grafana Alerting reaches Tares through the webhook connector and a Grafana-side setup, and Tares has no connector of its own for Grafana Alerting today. That makes the connection a pushed one, so the credential lives on the Grafana side.

How it works

Each connection type holds its credential in a different place:

• Pushed source (Grafana): Grafana posts to the source’s ingest URL, so Grafana carries the credential. On a secured Tares instance, the webhook contact point sends an ingest-scoped API key as Authorization: Bearer followed by the key. When Tares has auth on, a request without a valid key is not accepted.
• Polled source: Tares pulls from the upstream and holds its own credential for it. The Prometheus connectors, for example, take bearer_token, or username and password, in their config. Grafana alert rules do not use this path, because Grafana pushes them.
• Signed deliveries: some vendors sign each delivery, and for those the signature is the credential. Grafana Alerting authenticates with the header, so you do not set a signature scheme on this source.

On the Tares side the Grafana source is a webhook source. One Grafana notification arrives as one event, with the alerts list kept inside the lossless payload. Top-level fields such as status, receiver and title work for labels and for text_template.

Set it up

1. In Tares, create a webhook source for Grafana alerts, in the console under Sources, Add source, webhook, or by asking a connected agent. Declare labels for the top-level fields you want to correlate on and mark one as primary.

- name: grafana-alerts
  connector: webhook
  config:
    text_template: '{status}: {title}'
    labels:
      - { name: receiver, field: receiver, primary: true }
      - { name: status, field: status }

2. Copy the ingest URL from the source page. If your instance has auth on, creating a push source in the console mints an ingest-scoped API key for it and shows it once. Copy it then.
3. In Grafana, add a Webhook contact point. Set the URL to the Tares ingest URL and the method to POST.
4. When Tares has auth on, set the contact point’s Authorization header scheme to Bearer and the credentials to the ingest-scoped API key.
5. Route alerts to the contact point from a notification policy.
6. Send a test payload to the ingest URL and check Recent events. Fields shows which payload fields arrive, so you can confirm your labels match.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"action": "deploy", "sender": "ci", "pipeline": "web"}'

The curl call above is the shape of a test post. On a secured instance, it also needs the Bearer header with the key. Swap in a Grafana-shaped body with status, receiver and title to see your labels fill in.

What you get

• Only senders that hold the ingest-scoped key can post Grafana alerts into your secured Tares instance.
• A key scoped to ingest does one job, so the contact point holds no wider access to Tares.
• Each Grafana notification is stored lossless as one event, with the full alerts list in the payload and your chosen top-level fields as labels.
• The alerts correlate with your other sources, such as metrics, logs and deploys, through the labels you declare. For the wider picture, see AI SRE.

When something else fits

• If you want one event per alert instead of one per notification, create a separate alertmanager source. Grafana’s default payload is Alertmanager compatible, so the same payload works there. That source has its own ingest URL and takes the same ingest-scoped API key when Tares has auth on.
• If you already run Prometheus and want its fired alerts without any sender configuration, a polled alerts source holds the Prometheus credentials on the Tares side instead.

Related