How do I secure the connection that carries CloudWatch alarms into Tares?

Tares secures CloudWatch alarms with an ingest-scoped API key sent as a Bearer Authorization header on every delivery from EventBridge. Tares has no connector of its own for CloudWatch today, so alarms reach Tares through the webhook connector and an EventBridge setup. The key matters when Tares has auth on, and the endpoint must be public HTTPS.

How it works

Credentials depend on which way data moves. This is how it works for each connection type in AI SRE:

• Polled sources. Tares pulls from the upstream, so the source holds the credential. A Prometheus source, for example, takes a bearer_token, or a username and password, in its config.
• Pushed sources. The sender calls Tares, so the sender carries the credential. On a secured Tares that is an ingest-scoped API key sent as Authorization: Bearer followed by the key. Senders that sign their deliveries can use a signature instead.
• CloudWatch alarms. They are pushed. CloudWatch alarms flow through Amazon EventBridge: a rule matches the alarm state change event and sends it to an API destination, which POSTs the event as JSON to a webhook source’s ingest URL.

EventBridge requires a connection on every API destination. That connection is where the credential lives on the sending side. You configure it with API key authorization, header name Authorization, and a value of Bearer followed by your ingest-scoped API key. EventBridge adds that header to every delivery. The key value only matters when Tares has auth on.

The ingest URL must be public HTTPS, because EventBridge calls it from AWS. The webhook connector parses the JSON body whatever the Content-Type.

The Alertmanager connector follows the same pattern for a different sender. It also takes an ingest-scoped API key in an Authorization header when the instance is secured.

Set it up

1. In Tares, create a webhook source and declare labels for the alarm fields you want. When you create a push source in the console on a secured instance, Tares mints an ingest-scoped API key for it and shows it once, so copy it then. Copy the ingest endpoint from the source page as well.

- name: cloudwatch-alarms
  connector: webhook
  config:
    event_type_field: detail-type
    labels:
      - { name: alarm, field: detail.alarmName, primary: true }

detail-type is a top-level field, so it works as the event type field. detail.alarmName is one level deep, so it works as a dotted label.

2. In EventBridge, create an API destination. Set its endpoint to the Tares ingest URL and its method to POST.

3. Create the connection for that API destination. Choose API key authorization, set the header name to Authorization, and set the value to Bearer followed by the ingest-scoped API key from step 1.

4. On the default event bus, create a rule that matches source aws.cloudwatch and detail-type CloudWatch Alarm State Change. Set the API destination as the rule target.

5. If you want detail.state.value as a label or in text_template, flatten it with an input transformer on the rule target. It sits two levels deep, and labels read one-level dotted fields.

6. Test the source by posting a sample payload to the ingest endpoint, then check Recent events and Fields on the source page. The curl below is the webhook connector’s example. On a secured instance, add the same Bearer Authorization header to it.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"action": "deploy", "sender": "ci", "pipeline": "web"}'

What you get

• Each alarm state change that EventBridge delivers lands in Tares as an event on your webhook source, with the full payload stored lossless.
• The EventBridge connection holds the Bearer header, so every delivery carries the credential without extra work on the rule.
• You use the same ingest-scoped API key approach as other header-based push senders, so one rule of thumb covers your pushed sources.
• The alarm name, source and detail-type are available as labels and event type, so you can correlate alarms with your other sources.
• You scope the key to ingest, so it is meant for sending events to Tares.

When something else fits

• If your alarms publish through SNS, this route does not fit. An SNS HTTPS subscription cannot send a custom header, and its SubscriptionConfirmation message would never be confirmed, so use the EventBridge API destination route above.
• If you already run Prometheus or Alertmanager, use the Alertmanager connector or a polled Prometheus source instead. Those keep the credential in the way described for their connection type.

Related