What runbook should an AI SRE follow for 5xx errors on an API?

Tares lets you attach one 5xx runbook to your API by its labels with the reference connector, so an AI SRE gets it every time it correlates on that service. You write the steps once as a markdown document and tag it with labels such as service and environment. Tares then returns it with the events for that entity, even outside the read time window.

How it works

The reference connector holds documents (json, csv, md or txt) that describe an entity. It is declarative reference material, not a stream, so there is no poll interval and no entity key. Correlation is label-native.

• Each document is one event. Every attachment becomes one always-surfaced event that carries the labels you gave it.
• Labels are the link. Tag the runbook with the labels of the service it describes, for example service and environment. When the agent reads that entity by those labels, the runbook comes back in the timeline.
• It does not age out. A runbook is surfaced regardless of the read time window, so a 15-minute read of a 5xx spike still includes it.
• Editing re-materializes it. Change the source and Tares rebuilds the event, so the next read sees your updated steps.

The labels you tag documents with become real Tares labels. You can read them and trigger on them like labels from any other source. For where this fits in an incident flow, see AI SRE.

Set it up

1. Write the runbook. Keep it short and ordered, so an agent can follow it step by step. A 5xx runbook for an API usually covers scope, recent changes, dependencies, the fix and the record. You own the content, so adapt the steps to your service.
2. Add the source. In Tares, go to Sources → Add source → Reference documents. Upload the markdown file, tag it with the labels of the API (for example service=api and environment=production), and select Create. To extend it later, re-open the source and use Upload more.
3. Or declare it in config. The same source as YAML:

- name: runbooks
  connector: reference
  config:
    attachments:
      - name: api-5xx.md
        format: md
        content: |
          1. Confirm scope: which routes return 500, 502, 503 or 504, and since when.
          2. Check what changed: recent deploys, config changes and merged pull requests for this service.
          3. Check dependencies: the database, the cache and upstream services.
          4. If a deploy caused it, roll back. If one instance is unhealthy, restart it.
          5. Record what you found and what you did.
        labels: { service: api, environment: production }
    labels:
      - { name: service, field: service }
      - { name: environment, field: environment }

You can also send the same object with POST /api/sources. 4. Declare the label names. The labels list under config declares the label names so reads can correlate on them. Each attachment supplies the values. Use the same label names and values your other sources use for this API, so the runbook lines up with its events. 5. Verify. Open the source. Recent events shows one event per document, and Fields shows the labels they carry. Then read the entity by its service label and confirm the runbook appears in the timeline, even outside the read window.

The full field list is on the reference connector page.

What you get

• A runbook the agent always sees. Whenever the AI SRE correlates on your API, the 5xx steps arrive with the logs, metrics and alerts for that service.
• One place to maintain it. Edit the source and every later read gets the new steps.
• Runbooks per service and environment. Use different labels to give production and staging their own documents, or give each service its own.
• Labels you can use elsewhere. Because the document labels are real Tares labels, you can read and trigger on them like any other label.
• More than one document. Attach a schema, a dependency map or an escalation note next to the runbook, each tagged to the same entity.

When something else fits

The reference connector fits steady, written guidance. If you need live data about the 5xx errors, such as error logs, metrics or alerts, bring those in through event connectors and let the runbook sit beside them. If the guidance is a one-off note from an agent’s own work, an agent’s memory is a better place than a declared document.

Related