What should an AI SRE read next to CloudWatch alarms for slow database queries?

Tares lets an AI SRE read the same service’s metrics, logs and recent deploys on one timeline next to a CloudWatch alarm, so the diagnosis cites evidence. The alarm says that queries are slow. The metrics show when and how much, the logs show which requests suffered, and a deploy or config event shows what changed just before. CloudWatch alarms reach Tares through the webhook connector and an EventBridge setup, and every other source shares the alarm’s service label.

How it works

Tares has no connector of its own for CloudWatch today. The alarm arrives through the webhook connector and a vendor setup in EventBridge, which posts the alarm state change as a JSON body to a Tares ingest URL. The connector stores the payload lossless and parses it whatever the Content-Type.

A slow-query alarm is a starting point. Tares correlates sources that carry the same label, usually service, into one timeline per service. Next to the alarm event, an AI SRE reads:

• Metrics for the service, such as request latency (p99) and database connections, polled from Prometheus.
• Logs for the service, from a container’s logs or from your OpenTelemetry pipeline over OTLP.
• Recent changes, meaning deploys and config changes that your CD pipeline posts to a webhook source.

The agent reads that merged timeline, ties its conclusion to specific evidence lines, and writes the finding back onto the timeline. The next investigation starts from that conclusion. The full pattern is in Build an AI SRE.

Set it up

1. Send the alarm to Tares. In EventBridge, create an API destination whose endpoint is the Tares ingest URL (it must be public HTTPS) and whose method is POST. Give it a connection with API key authorization, header name Authorization and the value Bearer followed by an ingest-scoped API key. The key value only matters when Tares has auth on. Then create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as its target. Alarm actions that publish to SNS do not fit, because an SNS HTTPS subscription cannot send a custom header and its confirmation message would never be confirmed.

2. Create a webhook source for the alarm. Give it the same service label your other sources use. The detail-type field is top level, so it can set the event type. The alarm name is a one-level dotted field. The alarm state is two levels deep, so flatten it with an input transformer on the rule target if you want it as a label or in a text template.

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: service, const: checkout, primary: true }
      - { name: alarm, field: detail.alarmName }

3. Add the service’s metrics. Create a prometheus source for the database and latency metrics you care about, and read service from the metric labels. Add a docker_logs or otlp source for the service’s logs with the same service label.

4. Add the recent changes. Create a second webhook source for deploys, with a primary service label, event_type: deploy and a text template.

- name: deploys
  connector: webhook
  config:
    event_type: deploy
    text_template: 'deploy {commit} by {author}: {message}'
    labels:
      - { name: service, field: service, primary: true }

5. Post a deploy from your pipeline. Add one line to your CD pipeline, run on every deploy. A second webhook source with event_type: config covers config changes the same way.

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

6. Check the timeline. Open the source page and read Recent events and Fields to confirm the alarm fields arrive. Then open Explore for the service. Alarm, metrics, logs and deploys sit on one clock. Add a trigger over these sources and wake an agent, as Build an AI SRE describes.

What you get

• A diagnosis that cites evidence. When the slow-query alarm fires, the agent sees whether latency climbed before or after a deploy, whether connections ran out, and which log lines carried the errors.
• A change next to the symptom. A line such as a deploy that raised the database pool size lands on the timeline beside the alarm, so the likely cause is visible without hunting through other tools.
• One read instead of many. The agent works from a single correlated timeline for the service instead of querying CloudWatch, your metrics store and your logs one by one.
• A finding written back. The conclusion stays on the timeline, so the next investigation, yours or an agent’s, starts from it.
• Your alerting stays where it is. CloudWatch keeps owning the alarm, and Tares ingests what fired and correlates it.

When something else fits

• If you only need the alarm forwarded to a chat channel, with no correlation, CloudWatch and EventBridge alone are enough and Tares adds little.
• If your metrics and logs live somewhere you cannot poll, receive or post from, the timeline will hold the alarm and little else, so connect at least one more source first. You can pull JSON from an API with the http_poll connector.

Related