How do I line up alerts from Prometheus with Prometheus metrics on one timeline?

In Tares, you ingest your Prometheus metrics and your Prometheus alerts as two sources that carry the same service label. Tares then merges both on one clock for that service in Explore. The alert shows what fired, and the metric shows how it got there.

How it works

Two connectors read from the same Prometheus server, and the shared service label joins them.

• The Prometheus metrics connector turns each series you pick into a metric event on every poll tick. You choose metrics by name or by a label they carry, and you write no PromQL. The value is stored as a number-typed value label.
• The Prometheus alerts connector polls Prometheus’s own alert list, the alerts your alerting rules have already fired. It makes one event per active alert and a resolved event when the alert clears. It needs no Alertmanager.
• Alert state is tracked across polls, so each alert fires once and re-emits only on a state change. You do not get one copy per tick.
• Each alert event has the alert name as its event_type, its summary as text, its labels as Tares labels, and a state of firing, pending or resolved.

Because both sources are keyed by the same service label, Tares builds one timeline per service. The AI SRE guide builds exactly this with a metrics source, a logs source and an alerts source, all keyed by service.

Set it up

1. Start Tares and open the console at http://127.0.0.1:8787.

   tares up

2. Add the metrics source. Go to Sources → Add source → Prometheus, enter the url and any auth, and click Discover. Pick metrics by name (a metric name or a prefix_* pattern) or by label. Tares proposes the entity key and labels from a sample. You can also ask a connected agent: “Use tares to ingest the pg_* metrics from the Prometheus at localhost:9090.” The generated config looks like this:

   - name: metrics
     connector: prometheus
     poll: 10s
     config:
       url: http://localhost:9090
       queries:
         - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
           by_name: true
       labels:
         - { name: service, field: metric.service, primary: true }
         - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

3. Add the alerts source. Go to Sources → Add source → Prometheus alerts, enter the same url, and click Discover. It lists your alerting rules and their severities. Optionally filter by severity (blank means all) and toggle include pending.

   - name: alerts
     connector: prometheus_alerts
     poll: 30s
     config:
       url: http://localhost:9090
       severities: critical,warning   # blank = all
       labels:
         - { name: severity, field: labels.severity, primary: true }
         - { name: state, field: state }

4. Give the alerts the same key as the metrics. Discover only proposes the labels every alert has, severity and state. An alert’s dynamic labels such as namespace or service exist only while it fires. Once real alerts have been ingested, open the source’s Fields tab and promote service so it matches the label on your metrics source.

5. Open Explore and pick the service. The metric events and the alert events appear merged on one clock.

You can also create either source with POST /api/sources using the same object.

What you get

• One timeline per service, with the metric points and the alert events interleaved in time order.
• The alert that fired and the metric that tripped it side by side, so you see the climb before the alert and the resolved event when it clears.
• A numeric value on every metric event, so a trigger over the same service can aggregate it with max, avg or sum.
• Prometheus keeps owning alerting. Tares ingests what fired, correlates it with the evidence and can dispatch it to an agent. Nothing is re-thresholded.
• A quick check that it works. In each source, Recent events shows one event per series per poll for metrics, and one event per active alert for alerts. Fields shows the labels each event carries, including metric.service and the numeric value.

The same read is what a connected agent works from, so an investigation starts from the correlated timeline rather than from separate queries.

When something else fits

• If your alerts already route through Alertmanager and you prefer to keep that path, use the Alertmanager connector instead of polling Prometheus for alerts. It receives the alerts through a webhook receiver pointed at a Tares ingest URL. The metrics source works the same way beside it.
• If you want a derived value rather than a raw series, such as a rate or a p99, a queries row on the metrics source can hold a single PromQL expression with an explicit event_type and field. The pick-metrics flow covers raw ingestion without any PromQL.
• If a root cause is usually a change, add deploys and config changes as a webhook source keyed by the same service label, so they land on the same timeline. The AI SRE guide shows how.

Related