What runbook should an AI SRE follow for error spikes?

Tares lets you attach your error spikes runbook to a service with the reference connector, so the AI SRE always sees it when it correlates on that service. You write the runbook as a md, txt, json or csv document and tag it with the service’s labels. The document comes back in every read of that entity, even outside the read time window.

How it works

The reference connector holds documents you declare, not a stream of events. Each document is attached to an entity by its labels, such as service=tares and environment=production.

• One event per document. Each attachment becomes one always-surfaced event.
• Always surfaced. When the agent correlates on the entity, the runbook is included regardless of the read time window. A runbook does not age out of a 15-minute read.
• No polling. The config is the data. Editing the source re-materializes it, so there is no poll interval and no entity key.
• Real labels. The labels you tag each document with become Tares labels you can read and trigger on.

So the runbook for error spikes is a document you own. Tares does not ship one. It makes sure the document you write sits next to the live events for the same service.

Set it up

1. Write your error spikes runbook as a short md file. Put the steps you want the agent to follow in order, for example how to find out what changed, how to roll back and how to confirm recovery.
2. In Tares, go to Sources → Add source → Reference documents.
3. Upload the file. json, csv, md and txt files are read into text.
4. Tag the document with the labels of the service it describes, for example service=tares and environment=production.
5. Click Create. To add more documents later, re-open the source and click Upload more.

If you prefer config, declare the source with the same shape. This example is the reference source from the connector page. Replace the name, the content and the label values with your own error spikes runbook and service.

- name: runbooks
  connector: reference
  config:
    attachments:
      - name: restart.md
        format: md
        content: |
          1. kubectl rollout restart deploy/tares
          2. watch the pod come healthy
        labels: { service: tares, environment: production }
    labels:
      - { name: service, field: service }
      - { name: environment, field: environment }

The labels list at the bottom declares the label names, so reads can correlate on them. Each attachment supplies the values. Each attachment needs a name and content. format is one of json, csv, md or txt, and labels is the object of entity labels.

You can also send the same object to POST /api/sources.

6. Verify it. Open the source. Recent events shows one event per document and Fields shows the labels they carry. Read the entity the document describes, here the service tares, and the runbook comes back in the timeline, even outside the read window.

What you get

• The agent sees your error spikes runbook every time it correlates on the tagged service, next to the live events for that service.
• The runbook stays visible whatever time window the read uses.
• One document per service and environment. Use different label values to give each service its own steps.
• A runbook you can extend. Upload more documents to the same source, such as schemas or project notes, and tag them the same way.
• Labels you can trigger on, because the document labels are real Tares labels.

When something else fits

The reference connector is for material that does not change with each incident. If you need a stream of events instead, such as metrics, logs or alerts, use an event connector and keep the runbook in the reference source alongside it.

If the runbook is long or covers many unrelated services, split it into several documents and tag each with the labels of the service it describes. That way each read surfaces the steps for the entity the agent is looking at.

Related