Which labels on Alertmanager alert groups about 5xx errors on an API tie it to the right service?

Tares ties a 5xx alert from Alertmanager to its service through the alert’s own labels, and service is the label to use as the key. Set it as the key on the Alertmanager source and give your metrics, logs and deploy sources the same service value. The alert, the metrics, the logs and recent changes then land on one entity timeline, for example (service, api-server).

How it works

Alertmanager posts a batch of alerts to the ingest URL of your Tares source. Tares splits the batch into one event per alert. Each event has these properties:

• event_type is the alert name.
• text is the alert summary.
• The alert’s labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert, including its fingerprint, stays in the payload.

The label marked as the key sets the event’s key_value. For the Alertmanager source Tares derives the key from the first of these labels it finds: namespace, then service, then job, then instance. You can override that with the advanced key_label setting, which names the alert label to key on.

This order matters for 5xx alerts. If your alert carries both namespace and service, the default key is namespace. Your metrics and logs are probably keyed by service, so set key_label to service and the alert joins them.

An entity is a label and value pair, such as service and api-server. A read for that pair returns every matching event from every source, merged in time order. The selector is a strict AND, so adding a label such as an environment narrows the result. See Concepts for labels, keys and entities in full.

Set it up

1. Create the source. In the console go to Sources, Add source, Alertmanager alerts, name it and click Create. The source page shows a receiver block with your ingest URL filled in. A connected agent can do the same when you ask it to create an Alertmanager source and give you the webhook URL.
2. Add the receiver to alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

If Tares has auth on, the receiver also sends a Bearer Authorization header holding an ingest-scoped API key. Without auth the key is not needed. 3. Make sure the alert rules for your 5xx errors put a service label on the alert, with the same value your other sources use for that API, such as api-server. 4. Set key_label to service in the source’s advanced settings, so the alert is keyed on service and not on namespace. 5. Send a test batch and check Recent events:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

6. Open the source’s Fields tab. It lists the alert labels with their coverage and top values, so you can confirm that service is present on the alerts you care about.
7. Give the other sources the same key. For example, a deploy webhook source reads service from its payload and marks it as the primary label:

labels:
  - { name: service, field: service, primary: true }
  - { name: env, const: prod }

The AI SRE guide builds this end to end, with metrics, logs, alerts and deploys all keyed by service. The Alertmanager connector page has the full receiver details.

What you get

• One timeline per service. A read of service equal to api-server returns the 5xx alert, the error logs, the metric that crossed its threshold and the latest deploy, all on one clock.
• Firing and resolved states on the timeline. With send_resolved on, a resolved event lands next to the firing one.
• A reliable key. The Fields tab shows how many sampled events carry service, so you pick a label that is actually present.
• Triggers and agents that work per service. A trigger groups by entity and wakes an agent with that service’s correlated timeline. The agent can write its finding back onto the same timeline, so the next investigation starts from it.
• Alertmanager keeps its role. It still groups, dedupes and silences. Tares records what it pushes and correlates it with your other sources.

When something else fits

• If you want Tares to read the alerts that Prometheus rules fire, without Alertmanager’s grouping, silencing and real-time delivery, use the prometheus_alerts connector instead.
• If your alerts have no service label at all, keep the default key (namespace, job or instance) and use that same label on your other sources, or add a service label in your alert rules.

Related