How do I use Claude Code to investigate Grafana alert rules with Tares?

Tares lets Claude Code investigate a Grafana alert by reading the alert together with the logs, metrics and findings around it. Grafana sends alerts to a Tares webhook source, and Claude Code connects to the Tares MCP endpoint. You then ask about the service that alerted, and the agent answers from one correlated timeline. Tares has no connector of its own for Grafana today, so the alert reaches Tares through the webhook connector and a Grafana contact point.

How it works

• Grafana sends the alert. A Webhook contact point posts each notification to the ingest URL of a Tares webhook source. The default payload carries receiver, status, title, message and an alerts list with each alert’s labels and annotations.
• Tares stores it whole. The notification arrives as one event, and the alerts list stays inside the lossless payload. Top-level fields such as status, receiver and title work for labels and for text_template.
• Claude Code reads it back. Over MCP, the read tool returns a time-ordered timeline across all sources that match a label selector, so the alert sits next to the other evidence for the same entity. catalog_list and catalog_describe show what is ingested.
• Findings close the loop. An agent can save its conclusion with record_finding, and the next investigation, yours or an agent’s, starts from it.

The webhook connector page covers the source in detail, and AI SRE shows the whole pattern with metrics, logs and alerts keyed by one service label.

Set it up

1. Start Tares and its MCP endpoint. Run each command in its own terminal. A local Tares does not serve /mcp until the MCP server runs.

   tares up
   tares mcp

2. Create a webhook source for Grafana. In the console, go to Sources, Add source, webhook. Declare labels from top-level payload fields and mark one as primary. Save, then copy the ingest URL from the source page.

   - name: grafana-alerts
     connector: webhook
     config:
       text_template: '{title}: {status}'
       labels:
         - { name: receiver, field: receiver, primary: true }
         - { name: status, field: status }
         - { name: title, field: title }

3. Point Grafana at it. In Grafana, add a Webhook contact point with the Tares ingest URL and the POST method. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key. Then route alerts to the contact point from a notification policy.

4. Connect Claude Code. Run this in your terminal. The agents page also covers the Tares plugin, which adds session capture on top of the read-back.

   claude mcp add --transport http tares http://localhost:8788/mcp

   On a secured remote instance, pass a read-scoped API key as an Authorization Bearer header, as shown on the agents page.

5. Verify the connection. Ask Claude Code:

   Use tares: what are you ingesting right now?

   It calls catalog_list and lists your sources, including grafana-alerts.

6. Investigate the alert. Name the service that alerted and ask for the story around it:

   Use tares: what happened to api-server in the last 15 minutes?

   Claude Code calls read with a selector such as {"service": "api-server"} and answers from the merged timeline.

The selector matches labels, so the alert joins a service’s timeline when its source declares the same label name as your metric and log sources. Labels on the Grafana webhook source come from top-level fields. If you want one event per alert instead of one per notification, create an alertmanager source with its own ingest URL. It can take the same Grafana payload, because the default payload is Alertmanager compatible.

What you get

• Claude Code answers from one correlated read of alerts, logs, metrics and findings, so you do not paste alert text into a chat or switch between tools.
• The full Grafana notification is kept lossless, including every alert’s labels, annotations and timestamps, so the agent can quote the alert exactly.
• The setup is the same for every Grafana alert rule, because the contact point and notification policy decide which rules reach Tares.
• The agent can record what it concluded as a finding, so the next investigation starts from it instead of from scratch.
• The same MCP connection works for other questions about your systems, such as what changed before the alert fired.

When something else fits

• If Prometheus rules own your alerts, the prometheus_alerts connector (polled) or the alertmanager connector (pushed) ingests them directly. No Grafana contact point is needed.
• If you want an agent woken the moment an alert arrives, instead of asking Claude Code yourself, add a trigger on the source and let a Tares agent take the first look. The AI SRE guide walks through that.

Related