What should Prometheus alerts about a disk filling up include for an AI SRE?

Tares works best when a Prometheus disk alert carries a service label, a severity label and a summary. Tares turns each alert into one event with the alert name as its type, the summary as its text and the labels as Tares labels, plus a state of firing, pending or resolved. That puts the alert on the service’s timeline next to its metrics and logs, so the agent can tie it to evidence.

How it works

The prometheus_alerts connector polls the alerts endpoint of your Prometheus server every 30 seconds by default. Prometheus decides what counts as an alert. Tares ingests what the rule fired, correlates it and can dispatch it to an agent. Alertmanager is not required.

Each active alert becomes one event:

• event_type is the alert name, so a clear rule name tells the agent what kind of problem it is looking at.
• text is the alert summary, so a short summary that names the disk or mount gives the agent a readable line.
• The alert labels become Tares labels, plus a synthetic state.

Tares tracks state across polls. A disk alert fires once, re-emits only when its state changes, and produces a resolved event when it clears. The agent sees when the problem appeared and when it ended, not a copy of the alert on every tick.

The label that matters most is service. When your sources share a service label, Tares builds one timeline per service. A disk alert that carries the service name lands on the same timeline as that service’s metrics and logs. The AI SRE guide builds this setup end to end.

Set it up

1. In the Prometheus alerting rule for the disk, set a severity label, a service label with the name of the service that owns the disk, and a summary annotation that says what is filling up. Keep the alert name specific.
2. Install and start Tares if you have not yet. The install is uv tool install tares or brew install glassflow/tap/tares, then:

tares up

3. In the console, open Sources → Add source → Prometheus alerts. Enter the Prometheus url and any auth, then click Discover. It lists your alerting rules and the severities present.
4. Optionally filter by severity, for example critical,warning, or leave it blank for all. Turn on include pending if you want alerts that are still waiting out their for: duration. The source looks like this:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

5. Discover proposes only the labels that are always present, severity and state. An alert’s own labels, such as service and namespace, exist only while it is firing. Once the disk alert has fired and been ingested, open the Fields tab of the source and add service as a label.
6. Check the result. Recent events shows one event per active alert and a resolved event as each clears. Fields shows the labels your alerts carry.

You can also ask a connected agent: use Tares to ingest the firing alerts from the Prometheus at localhost:9090. The agent runs discovery and creates the source for you.

What you get

• A disk alert that sits on the right service’s timeline, merged with its metrics and logs on one clock.
• An event whose type is the alert name and whose text is your summary, so the agent reads a plain description of the problem.
• A severity label you can filter on, both at the source and in triggers.
• A state on every event, so the agent can tell a firing disk alert from a resolved one.
• One event when the alert first fires and one when it resolves, instead of a repeat on every poll.

The alert can then wake an agent through a trigger, and the agent starts from the correlated timeline instead of from the bare alert. The full connector reference is on the Prometheus alerts page.

When something else fits

• If your alerts already route through Alertmanager and you prefer to keep that path, the alertmanager connector receives them at a Tares ingest URL instead.
• If you want the raw disk usage numbers on the timeline, not only the alert, add the prometheus connector for those metrics alongside this source.

Related