How do I use the Codex CLI to investigate Alertmanager alert groups with Tares?

Tares lets you connect the Codex CLI over MCP and ask about the service that alerted, so Codex reads the Alertmanager alert together with the logs and metrics around it. You add the Tares MCP endpoint to Codex with one command, point an Alertmanager webhook receiver at a Tares ingest URL, and label the alerts with the same service as your other sources. Codex then answers from one correlated, time-ordered timeline.

How it works

Alertmanager evaluates, groups, dedupes and tracks firing to resolved. Tares records what it pushes. Each batch that Alertmanager posts is fanned out into one event per alert in the alerts array:

• event_type is the alert name.
• text is the alert summary.
• The alert labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert, including its fingerprint, stays in the payload.

Tares derives the entity key for an alert from a label, trying namespace first, then service, job and instance. To key your alerts on the service, set key_label to service on the Alertmanager source. The service value on the alert must match the service value of your logs and metrics so the alert sits next to them.

Codex connects to the Tares MCP endpoint. When you ask about a service, it calls read with a selector such as service set to api-server, and gets metrics, logs, alerts and findings merged on one clock. See Connecting agents for every client and the full tool list, and the Alertmanager connector page for the receiver details.

Set it up

1. Start Tares locally, then start the MCP endpoint in a second terminal. The daemon alone does not serve /mcp.

   tares up
   tares mcp

2. Add the Tares MCP endpoint to Codex. Keep --url before the URL, because a bare URL registers a stdio server.

   codex mcp add tares --url http://localhost:8788/mcp

   For a secured instance or Tares Cloud, Codex reads the credential from an environment variable. Use a scoped API key with the read scope, and replace the URL with your own endpoint. Tares Cloud shows it in your console under Connect.

   export TARES_API_KEY=<api-key>
   codex mcp add tares --url https://<your-host>/mcp \
     --bearer-token-env-var TARES_API_KEY

3. Create the Alertmanager source under Sources, Add source, Alertmanager alerts, or ask Codex: “Use tares to create an Alertmanager source and give me the webhook URL.” Set key_label to service on the source so alerts are keyed on the service:

   - name: alerts
     connector: alertmanager
     config:
       key_label: service

4. Add the receiver to your alertmanager.yml with the ingest URL from the source page, then reload Alertmanager. Make sure your alerts carry a service label whose value matches the service value of your logs and metrics.

   receivers:
     - name: tares
       webhook_configs:
         - url: http://localhost:8787/ingest/<ingest_key>
           send_resolved: true
   route:
     receiver: tares

   The sender uses a Bearer Authorization header holding an ingest-scoped API key, needed only when Tares has auth on. The source page shows the auth block for the receiver when your instance requires a token.

5. Send a test alert with service set to api-server, then check Recent events:

   curl -X POST http://localhost:8787/ingest/<ingest_key> \
     -H 'Content-Type: application/json' \
     -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","service":"api-server"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

6. Verify Codex is connected. Ask: “Use tares: what are you ingesting right now?” Codex calls catalog_list and lists your sources, triggers and projects. Then ask about the service that alerted:

   Use tares: what happened to api-server in the last 15 minutes?

   In the console, Reads lists each read as it happens, tagged client = mcp.

What you get

• Codex answers from one timeline per service, with the Alertmanager alert next to the logs and metrics for that service, instead of reading the alert alone.
• Each alert in a group is its own event, with its name, summary, labels, status and full payload, so Codex can tell what fired and what resolved.
• Findings that agents recorded earlier are part of the read, so a later investigation starts from the previous conclusion.
• Every read shows up in the console under Reads.

The AI SRE guide shows how to add a trigger that wakes an agent when an alert event arrives, so the first look is already on the timeline when you open Codex.

When something else fits

• If you want Prometheus itself to be the alert source, without Alertmanager’s grouping and silencing, use the prometheus_alerts connector instead.
• If you prefer a chat inside the console, use Ask, which has the same read tools and needs no external client.
• If the alert should wake an agent without you asking, use a trigger as described in the AI SRE guide rather than investigating by hand.

Related