What should Grafana alert rules about a disk filling up include for an AI SRE?

Tares gets the most from a disk-filling Grafana alert when each rule carries a service name, a severity and a summary, plus the start time Grafana adds. Tares receives the notification through the webhook connector and a Grafana contact point, and keeps the full payload, so your agent can read those fields and tie the alert to metrics, logs and deploys for that service. Set the fields once on the rule and every firing arrives complete.

How it works

Grafana Alerting reaches Tares through the webhook connector and a Grafana vendor setup. Tares has no connector of its own for Grafana today. A Webhook contact point in Grafana POSTs each notification to the ingest URL of a Tares webhook source.

The notification arrives as one event. The alerts list sits inside the lossless payload, and each alert in it carries these fields:

• status, which is firing or resolved.
• labels, the labels set on the rule.
• annotations, the annotations set on the rule.
• startsAt and endsAt, the start and end times.
• fingerprint, which identifies the alert.

The notification also has top-level fields: receiver, status, title, state and message. Top-level fields work for labels and for text_template on the Tares source. The payload is stored lossless, so the rule’s own labels and annotations stay readable by your agent in the event.

If you want one event per alert and a separate ingest URL, you can send the same Grafana payload to an alertmanager source instead, because the default Grafana payload is Alertmanager compatible.

Set it up

1. Write the disk rule so that it carries what the agent needs. Put a service name and a severity in the rule’s labels. Put a short summary in the annotations. Name the host or mount in the summary, for example that the data volume on the database host is 90 percent full. Grafana adds startsAt itself, so you do not set the start time.
2. In Tares, create a webhook source. Open Sources, Add source, webhook, or ask a connected agent to create it. Copy the ingest endpoint from the source page. Map the top-level title and message into the event text and keep status as a label:

- name: grafana-disk
  connector: webhook
  config:
    event_type: disk_alert
    text_template: '{title}: {message}'
    labels:
      - { name: status, field: status, primary: true }

3. In Grafana, add a Webhook contact point. Set its URL to the Tares ingest URL and its method to POST.
4. If Tares has auth on, set the contact point’s Authorization header scheme to Bearer and the credentials to an ingest-scoped API key. Without auth on, you need no key.
5. Route the disk alerts to that contact point from a notification policy.
6. Trigger a test notification, then open the source page. Recent events shows the event, and Fields shows which payload fields arrive and how often. The webhook connector page covers these checks in full.

The label mapping above uses only top-level fields. The service name from your rule travels inside the alerts list of the payload, where your agent can read it.

What you get

• A disk alert that arrives as an event with a readable line, such as the title and message, and a status of firing or resolved.
• The service name, severity and summary from your rule kept in the payload, so the agent knows which service is affected, how urgent it is and what the rule saw.
• The start time from startsAt, so the agent can line the alert up against deploys, config changes and log growth around that moment.
• A resolved notification that lands in the same stream when the disk recovers, so the agent can see when the problem ended.
• The full payload kept lossless, so you can add more rule labels or annotations later without changing the Tares source.

The AI SRE guide shows how to key your sources by service, so a trigger can wake an agent with one correlated timeline and write its finding back.

When something else fits

• If you want one event per alert rather than one per notification, point a separate alertmanager source at the same Grafana payload.
• If you already run Prometheus alerting rules, the prometheus_alerts or alertmanager connectors take the alerts there, and you can keep Grafana out of the path.

Related