What runbook should an AI SRE follow for high latency?

Tares lets you write one high latency runbook, attach it to your service by its labels with the reference connector, and have it surfaced every time the agent correlates on that service. The runbook is a document you own, in md, txt, json or csv. It is tied to labels such as service and environment, so it comes back with the rest of the timeline for that entity. A read time window never drops it.

How it works

The reference connector holds documents as declarative material, not as a stream. Each attachment carries its own name, format, content and labels, and each one becomes a single event.

• Attached by labels. You tag the runbook with the labels of the entity it describes, for example service=tares and environment=production. Those labels become real Tares labels you can read and trigger on.
• Always surfaced. When the agent correlates on that entity, the document comes back in the timeline regardless of the read time window. A runbook does not age out of a 15-minute read.
• The config is the data. There is no poll interval and no entity key. Editing the source re-materializes it.

The labels block declares the label names, so reads can correlate on them. Each attachment supplies the values.

What goes into the runbook is up to you. A useful high latency runbook is short and written for your own service, for example:

• which dashboards and metrics show the latency you care about
• which dependencies to check first, such as the database or an upstream service
• what recent changes, such as deploys, to look at
• who to escalate to and when

Tares does not ship a built-in high latency runbook. It makes sure yours is in front of the agent for the right service.

Set it up

1. Write the runbook as a md, txt, json or csv file. Name it so it is easy to spot, such as high-latency.md.
2. In Tares, open Sources → Add source → Reference documents.
3. Upload the file. Tares reads it into text.
4. Tag it with the labels of the service it covers, for example service=tares and environment=production.
5. Click Create. To add more runbooks later, re-open the source and use Upload more.

You can also declare the source in YAML, or send the same object with POST /api/sources:

- name: runbooks
  connector: reference
  config:
    attachments:
      - name: high-latency.md
        format: md
        content: |
          1. Check the latency dashboard for the service
          2. Check the database and upstream dependencies
          3. Check recent deploys
          4. Escalate if latency stays high
        labels: { service: tares, environment: production }
    labels:
      - { name: service, field: service }
      - { name: environment, field: environment }

The name and content fields are required for each attachment. format is one of json, csv, md or txt, and labels is an object of entity labels.

To check it, open the source. Recent events shows one event per document, and Fields shows the labels they carry. Then read the entity the runbook describes by its service label. The runbook comes back in the timeline, even outside the read window. The full field list is on the reference connector page.

What you get

• One high latency runbook per service and environment, kept as a document you edit in place.
• A runbook that is always surfaced when the agent correlates on that service, whatever the read window.
• Labels you can read and trigger on, because the labels you tag each document with become real Tares labels.
• A simple check: one event per document in Recent events, with the labels shown under Fields.

This is the building block for the wider AI SRE setup. Your event sources feed the timeline, and the reference connector keeps your own guidance next to them for the same labels.

When something else fits

• If you want data that changes over time, such as metrics, logs or alerts, use an event connector. The reference connector is for documents that stay put, not for a stream.
• If you need a different runbook for each environment, tag each document with its own environment label rather than writing one document for everything.

Related