How do I send Tares agent and Ask model calls through an internal LLM gateway such as LiteLLM?

Tares sends agent and Ask model calls through your internal LLM gateway once you set the gateway URL and token on the Anthropic model provider. You can do it in the console under Settings, Model providers, or with ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN in the environment. Both ways cover every model call, for trigger-woken agents and for Ask alike. The gateway decides which model answers.

How it works

By default, Tares agents and Ask talk to api.anthropic.com. If your organisation routes all model traffic through a gateway such as LiteLLM, Portkey, Helicone or a proxy in front of Bedrock or Vertex, you point Tares at it with one of two settings:

• Console. Edit the Anthropic entry under Settings > Model providers. Put the gateway’s URL in the base URL field and the gateway token or your key in the key field. The change applies from the next model call, with no restart.
• Environment. Set ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN, the same variables Claude Code uses. taresd reads them when it starts. The old name TARES_ANTHROPIC_BASE is still read for the base URL.

Values stored in the console win over the environment. The console values are read on every call, while the environment values are read once at startup.

The credential goes out in one of two ways. A gateway token (stored, or ANTHROPIC_AUTH_TOKEN) is sent as a bearer header. A key (stored, or ANTHROPIC_API_KEY) is sent as the Anthropic key header, which gateways accept too. Tares picks the first one it finds in this order: a stored gateway token, a stored key, the environment’s token, then the environment’s key.

Set it up

1. Get the gateway’s URL and a token from your gateway team. The gateway must serve the Anthropic Messages format.

2. Choose where to set it.

   Option A: the console. This is the path on a hosted cell, where the environment is not yours to set.

   • Open Settings > Model providers.
   • Edit the Anthropic entry.
   • Enter the gateway URL as its base URL and the token (or your key) in the key field.
   • Save. The next model call goes through the gateway.

   On a hosted cell the platform key stops being used, exactly as when you store your own key.

   Option B: the environment. Set the two variables, then start Tares:

   ANTHROPIC_BASE_URL=https://llm-gateway.internal   # serves the Anthropic Messages format
   ANTHROPIC_AUTH_TOKEN=<token the gateway issued>    # sent as Authorization: Bearer
   tares up

   For a container or compose deployment, set the same variables on the taresd service. See Deployment for the full list of environment variables and which process reads each one.

3. Run an Ask question or wake an agent, then check your gateway’s logs for the request.

4. To go back, use Stop using the gateway in the console. Tares returns to the environment’s gateway if the deployment set one, and to Anthropic otherwise.

If you set both the console entry and the environment variables, the console entry is the one in use. Clear it when you want the environment to apply.

What you get

• Every agent and Ask model call goes through your gateway, so your usual routing, logging and access rules apply to Tares too.
• You can change the gateway in the console without restarting taresd.
• The gateway decides what answers. Tares sends a Claude model id and accepts any well-formed reply. If your gateway maps that id to another model, that is your choice as the operator. The agent prompts and the guided setup are written and tested against Claude.
• The spend meter prices runs at Claude rates, from Tares’s own price table, by the model name in the reply. Behind a gateway, treat the gateway’s bill as the true cost. A reply that names a model Tares does not price shows as an uncosted run.

To see what the agents do once they run, read the AI SRE guide.

When something else fits

• If you run LiteLLM and want its own aliases and cost figures, add it as an OpenAI-compatible model provider instead. The picker then offers the aliases your key may use, and LiteLLM’s own cost figure lands on each run.
• If you do not use a gateway, leave the defaults, and Tares calls api.anthropic.com with your ANTHROPIC_API_KEY.

Related