What should Alertmanager alert groups about slow database queries include for an AI SRE?

Tares records each alert in an Alertmanager group as its own event, so slow database query alerts should carry a service label, a severity label, a summary and a start time. The service label ties the alert to the same service’s metrics and logs. The summary becomes the event text, and the start time and status show when the problem began and whether it resolved.

How it works

Alertmanager POSTs a batch to your Tares ingest URL. Tares splits the alerts array into one event per alert, so each field you send has a place to land:

• Alert name. The alertname label becomes the event type. Name the alert for what it detects, for example SlowDatabaseQueries, so the event reads clearly on the timeline.
• Summary. The summary annotation becomes the event text. Write one plain sentence that says what is slow.
• Service. Alert labels become Tares labels. Tares picks the entity key from namespace, then service, then job, then instance, or from a label you declare as primary. A service label that matches the label on your metric and log sources puts the alert on the same per-service timeline as that evidence, which is what the AI SRE guide builds on.
• Severity. A severity label arrives as a Tares label like any other, so triggers and agents can read it.
• Start time. startsAt is part of the alert and is kept in the payload.
• Status. Tares adds a status label, firing or resolved. With send_resolved: true you see both states.

The full alert, including its fingerprint, stays in the payload, so nothing you send is lost.

If an alert carries both namespace and service, the default key is namespace. To key on service instead, set the advanced key_label field on the source to service.

Set it up

1. Create the source. Ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” Or open Sources → Add source → Alertmanager alerts, name it and click Create. The source page shows a receiver block with your ingest URL filled in.
2. Add the receiver to your alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If Tares runs with auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. The source page shows the auth block for you. Without auth, you need nothing more.
4. In the alert rule for slow queries, set the labels service and severity, and a summary annotation. Alertmanager supplies startsAt itself when it delivers the alert.
5. Send a test batch shaped like a slow query alert:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"SlowDatabaseQueries","severity":"warning","service":"checkout"},"annotations":{"summary":"Database queries are slow"},"startsAt":"2024-01-01T00:00:00Z"}]}'

6. Open Recent events to see the event, then open Fields to see the alert labels. Promote service there so the alert correlates with your other sources. The Alertmanager connector page lists every field.

What you get

• Each slow query alert is one event with a clear type, a readable summary and its labels, instead of a single opaque group.
• The alert sits on the service’s timeline next to metrics and logs on one clock, so an agent can tie it to evidence.
• The status label shows whether the alert is still firing or has resolved.
• A trigger can fire on the alert event and wake an agent, which reads the correlated timeline and records its conclusion for the next investigation.
• Alertmanager keeps its grouping, deduplication and silencing. Tares records what Alertmanager sends and does not re-threshold anything.

When something else fits

• If you do not run Alertmanager, the prometheus_alerts connector polls the alerts your Prometheus rules have fired, every 30 seconds by default. Alertmanager is the better fit when you want its grouping, silencing, resolved handling and real-time delivery.
• If the alert comes from a tool that is not Alertmanager, send it through the webhook connector and keep the payload flat, with the same service, severity, summary and start time fields.

Related