What should Alertmanager alert groups about a disk filling up include for an AI SRE?

Tares turns each alert in an Alertmanager group into one event, so a disk-filling alert should carry a service label, a severity label, a summary annotation and a start time. Tares uses the alert name as the event type, the summary as the event text and the labels as Tares labels. The service label ties the alert to the metrics and logs for the same service, so your AI SRE agent starts from evidence instead of a bare page.

How it works

Alertmanager POSTs a batch to the ingest URL of an alertmanager source. Tares splits the alerts array into one event per alert:

• event_type is the alert name, so name the alert for the symptom, for example one that says a disk is filling up.
• text is the alert’s summary annotation. This is the line the agent reads first, so make it state what is filling and where.
• The alert’s labels become Tares labels, plus a status label that is firing or resolved.
• The full alert, including fingerprint and startsAt, stays in the payload.

The entity key, which decides whose timeline the event lands on, comes from a label. Tares looks for namespace, then service, then job, then instance, or it uses a label you declare as primary. You can also set key_label on the source to pick the label yourself.

So these are the four things worth sending:

1. A service name. Put a service label on the alert. Use the same value your metrics and log sources use, so the alert, the disk metric and the application logs share one timeline. In the AI SRE guide every source is keyed by the same service label for this reason.
2. A severity. A severity label such as critical or warning tells the agent, and any trigger you write, how urgent the alert is.
3. A summary. A summary annotation becomes the event text. Write it as a sentence with the symptom and the place.
4. A start time. Alertmanager sends startsAt with each alert. Tares keeps it in the payload, so the agent can compare when the disk alert started with when errors or deploys appeared.

If the host or instance matters for a disk, leave that label on the alert too. Every label you route is kept as a Tares label, and instance is the last fallback for the entity key.

Set it up

1. Create the source. In the console go to Sources → Add source → Alertmanager alerts, name it and click Create. Or ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” The source page shows a receiver block with the ingest URL filled in.
2. Add the receiver to your alertmanager.yml and reload Alertmanager. Set send_resolved: true so the agent also sees when the disk alert clears.

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If Tares runs with auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. The source page shows the auth block for you when the instance requires a token. Without auth on, the key is not needed.
4. In the Prometheus rule that fires on the disk, set the service and severity labels and write a summary annotation. Alertmanager groups the alerts and sends them to the tares receiver, and Tares reads those labels and that annotation from each alert.
5. Send a test batch and check Recent events. The sample below uses a Postgres alert, so swap in your own alert name and summary when you test your disk rule.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

6. Open Fields on the source. It lists the alert labels, so you can promote service to the label you correlate on. The Alertmanager connector page has the full reference.

What you get

• One event per alert, so a group with several disks produces several events, each with its own labels and summary.
• A disk alert that lands on the same per-service timeline as your metrics and logs, because the service label matches.
• A firing event and later a resolved event, so the agent sees whether the disk problem is still open.
• Readable event text from your summary annotation, and the complete alert, including fingerprint and startsAt, in the payload.
• Alertmanager keeps its grouping, deduping and silencing. Tares records what Alertmanager pushes and correlates it with your other sources, so you do not re-threshold anything.

When something else fits

• If you do not run Alertmanager and Prometheus rules fire the alerts directly, the prometheus_alerts connector polls them instead. Alertmanager gives you its grouping, silencing and resolved handling with real-time delivery.
• If you want the disk metric values themselves on the timeline, add a prometheus source for them next to the alerts, keyed by the same service label.

Related