What do I put in alertmanager.yml so Alertmanager sends its alerts to Tares?

Tares receives Alertmanager alerts when you add a receiver with a webhook_configs url pointing at your source’s ingest URL, send_resolved: true, and a route that uses it. Create an Alertmanager source in Tares, paste the receiver block into alertmanager.yml and reload Alertmanager. Each alert Alertmanager routes then becomes an event in Tares. The source page in the console shows the block with your URL already filled in.

How it works

The Alertmanager connector is a push receiver. Alertmanager has already evaluated, grouped and deduplicated your alerts, so Tares records what it sends and correlates it with your other sources. This is part of the wider AI SRE workflow.

• Receiver. You add a receiver named tares with a webhook_configs entry. Its url is the ingest endpoint of your Tares source, which is http://localhost:8787 followed by the path /ingest/ and your source’s ingest key on a local Tares.
• Resolved alerts. With send_resolved set to true, Alertmanager also sends a notification when an alert clears, so Tares tracks firing and then resolved.
• Route. The route block names the receiver, so Alertmanager actually sends alerts to it.
• Batches. Alertmanager posts a batch of alerts in one request. Tares splits the alerts array into one event per alert.

Each event is shaped like this:

• event_type is the alert name.
• text is the alert summary.
• The alert labels become Tares labels, plus a synthetic status label that is either firing or resolved.
• The full alert, including its fingerprint, stays in the payload.

The entity key comes from a label automatically: namespace, then service, then job, then instance, or a label you declare as primary. The optional key_label field lets you choose a different alert label to key on.

Set it up

1. Create the source. In the console, go to Sources, then Add source, then Alertmanager alerts. Name it and click Create. Or ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” The agent creates the source and returns the ingest URL.
2. Copy the receiver block. The source page shows a copy-paste alertmanager.yml receiver block with the ingest URL already filled in. When your instance requires a token, the page also shows the auth block.
3. Add the receiver and the route to your alertmanager.yml. Replace the placeholder with your source’s ingest key if you write it by hand:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

4. Add authentication if Tares has auth on. Alertmanager then sends a Bearer Authorization header that holds an ingest-scoped API key, set in the http_config of the webhook_configs entry. The console block includes it for you. If auth is off on your instance, skip this step.
5. Reload Alertmanager so it picks up the new receiver and route.
6. Send a test batch to the ingest endpoint:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

7. Open the source page and check Recent events. The test alert appears as an event named PostgresDown. Then open Fields to see the alert labels. You can promote namespace or service there to correlate on.

What you get

• Every alert Alertmanager routes to the receiver arrives in Tares as its own event, in real time.
• Firing and resolved states are both recorded through the status label, so you can see when an alert started and when it cleared.
• Alertmanager’s grouping, silencing and deduplication still apply, because Tares records what Alertmanager decides to send.
• Alert labels are searchable and usable for correlation with your logs, metrics and commits.
• The full alert, including its fingerprint, is kept in the event payload.
• A copy-paste block in the console, so you do not assemble the URL by hand.

When something else fits

If you want Tares to read fired alerts straight from a Prometheus server on a schedule, use the prometheus_alerts connector. It does not give you Alertmanager’s grouping, silencing and resolved handling, or real-time delivery. If your alerts come from a vendor other than Alertmanager, use the webhook connector instead.

Related