What should Alertmanager alert groups about high latency include for an AI SRE?

Tares works best when each Alertmanager high latency alert carries a service label, a severity label, a summary annotation and a start time. Tares turns every alert in a group into its own event, with the alert name as the event type, the summary as the text and the labels kept as labels. The service label places the alert on the same timeline as your metrics and logs, so an AI SRE can tie the latency alert to evidence in one read.

How it works

Alertmanager sends a batch to your Tares ingest URL. Tares splits the alerts array into one event per alert:

• event_type is the alert name, for example HighLatency.
• text is the alert’s summary annotation.
• The alert’s labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert is kept in the payload, including fingerprint and startsAt.

Tares picks the entity key from a label, checking namespace, then service, then job, then instance, or from a label you declare as primary. That key decides which timeline the alert lands on. When the alert shares its service label with your Prometheus metrics and log sources, the agent reads the p99 curve, the request logs and the alert together. The AI SRE guide builds this per-service timeline end to end.

For a high latency group, send these fields:

• alertname: a stable name such as HighLatency. It becomes the event type, so triggers and agents can recognise it.
• service: the same value your metric and log sources use. It is the most important label, because it links the alert to evidence.
• severity: critical or warning, so the agent can judge urgency.
• summary annotation: one sentence that says what is slow and where, for example the service and the latency it crossed. This becomes the event text the agent reads first.
• start time: Alertmanager sends startsAt with each alert, and Tares keeps it in the payload. The agent can compare it with deploys and errors on the same timeline.
• resolved notices: set send_resolved: true so the timeline shows when latency recovered.

Set it up

1. Create the source. Ask a connected agent to create an Alertmanager source and give you the webhook URL, or use Sources → Add source → Alertmanager alerts in the console. The source page shows a receiver block with the ingest URL filled in.
2. Add the receiver to alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If Tares runs with auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. The source page shows the matching block. Without auth you need no header.
4. Put the fields in your latency alert rule. Give it a service label, a severity label and a summary annotation. Group by service in your route so each group stays about one service.
5. Send a test batch shaped like a latency alert and check Recent events:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"HighLatency","severity":"warning","service":"checkout"},"annotations":{"summary":"checkout p99 latency is high"},"startsAt":"2024-01-01T00:00:00Z"}]}'

6. Open Fields to see the alert labels. If an alert carries both namespace and service, the default key is namespace. Set key_label to the service label, or promote it, so the alert keys on the service. Details are on the Alertmanager connector page.
7. Add a trigger over the alert source and your metric source, keyed by the same service label. You can wake an agent when an alert event arrives, or threshold a typed field directly, such as max(p99_ms) > 1000 for latency.

What you get

• One event per alert, so a group of several services shows up as separate entries, each on its own service timeline.
• An alert that sits beside the metrics and logs for the same service, so the agent reads one correlated timeline instead of hopping between tools.
• A readable event text from your summary, and an event type you can filter triggers on.
• A firing and resolved history, so the agent sees how long latency lasted and when it recovered.
• Alertmanager keeps owning grouping, deduping and silencing. Tares records what it pushes and correlates it.

When something else fits

• If you do not run Alertmanager and want Prometheus rules to be polled directly, the prometheus_alerts connector fits better. You lose Alertmanager’s grouping, silencing and real-time delivery.
• If you want latency to trip a trigger without waiting for an alert, threshold the metric from the prometheus connector, and keep the alert as supporting evidence.
• If your alerts come from a vendor that is not Alertmanager, send them through the webhook connector instead.

Related