How do I add JSON API endpoints to the evidence an AI SRE reads when CloudWatch alarms fire?

Tares lets you add an http_poll source for your JSON API next to the webhook source that receives CloudWatch alarms, so both sets of events are stored in Tares. CloudWatch alarms reach Tares through the webhook connector and an EventBridge setup, and Tares has no connector of its own for CloudWatch today. The http_poll connector calls your API on a schedule and stores one event per item, such as the customers, tenants or services a record names.

How it works

Two sources do two jobs.

• Alarms come in through the webhook connector. An EventBridge rule matches CloudWatch alarm state changes and sends each one to an API destination whose endpoint is the Tares ingest URL. The webhook connector stores the JSON payload lossless, and the labels you declare map payload fields such as detail.alarmName into the event.
• API data comes in through http_poll. Tares calls your endpoint itself, with no script on your side. It stores one event per item in the reply, or one event if the reply is a single object. The http_poll connector makes nested values reachable by dotted name in labels, trigger filters and trigger conditions.

The API source runs on its own poll schedule. It does not fire when an alarm fires, so the events it stores reflect the API as of the most recent poll. Set the interval to match how fresh you need that data. The connector refuses anything under 10 seconds.

If the evidence you want is rows in your own database rather than an API, the Postgres connector reads new and changed rows of a table on a cursor column in the same way. For the overall approach to incident investigation, see AI SRE.

Set it up

1. Create a webhook source for the alarms. Declare a label per field you want to correlate on and mark one primary. detail-type and account are top-level fields. detail.alarmName is a one-level dotted field, so it works as a label.

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: alarm, field: detail.alarmName, primary: true }
      - { name: account, field: account }

2. Copy the ingest URL from the source page. In EventBridge, create an API destination whose endpoint is that URL (it must be public HTTPS) and whose method is POST.
3. Give the destination a connection. EventBridge requires one on every API destination. Use API key authorization with header name Authorization and the value Bearer followed by an ingest-scoped API key. The key only matters when Tares has auth on.
4. Create a rule on the default event bus that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change, and set the API destination as its target. detail.state.value is two levels deep, so flatten it with an input transformer on the rule target if you want it as a label or in a text template.
5. Add the JSON API as an http_poll source. In the console, go to Sources, Add source, HTTP API (poll), fill in the request and press Discover. Tares calls the API once, shows the reply and fills in what becomes an event. You can also declare it directly:

- name: affected-customers
  connector: http_poll
  poll: 5m
  config:
    url: https://api.example.com/v1/readings
    credential: abc123
    credential_header: X-API-Key
    payload_key: data
    id_field: id
    labels:
      - { name: customer, field: customer_id, primary: true }

payload_key points at the part of the reply that holds the items. id_field is the deduplication key, so an item seen in recent polls is not stored again. credential is stored as a secret and sent in the header named by credential_header. 6. Verify both sources. Open each source page. Recent events shows what arrived, and Fields lists every field seen with its coverage, so you can confirm your keys are reliably present.

CloudWatch alarm actions that publish to SNS do not fit this route. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed. Use EventBridge as above.

What you get

• Alarm events in Tares from CloudWatch, each stored as the full payload with the labels you declared.
• A second stream of events from your own API, one per item, so rows and responses that show who or what is affected are stored in Tares next to the alarms.
• Nested fields from either source, in labels, trigger filters and trigger conditions, by dotted name.
• Built-in handling of API limits. The connector backs off on a 429 or any 5xx, and the wait doubles from the poll interval up to an hour. A Retry-After value on a 429 is used. Any other 4xx shows up as a config error on the source’s health.

When something else fits

• If the data you want sits in a database table rather than behind an API, use the Postgres connector, which reads new and changed rows on a cursor column.
• If the system can call Tares itself, send its events to a webhook source instead of polling it.
• If you need data fetched at the exact moment an alarm fires, a poll will not do that, because http_poll runs on its own schedule.

Related