Set up LLM access

Any code you run in Domino that calls a Large Language Model (LLM) needs an endpoint to call and credentials to call it with. That applies whether the caller is an agent, an App, a Job, or exploratory code in a Workspace. Decide how your code reaches its models before you start building. Two decisions shape LLM access:

Where the model runs: at an external provider, or on Domino compute as a Domino-hosted endpoint.
How your code reaches it: through LLM Gateway 2.0, or by calling the model directly.

Route calls through LLM Gateway 2.0 whenever your deployment runs it. The gateway fronts both external providers and Domino-hosted endpoints behind one base URL, so your code uses the same client, the same Domino token, and a model alias no matter where the model runs. Provider credentials stay in the gateway instead of in your Project, and every call is subject to the gateway’s access grants, budgets, guardrails, and cost tracking. Call a model directly only when the gateway isn’t deployed yet.

	Connect to an external provider	Host a model in Domino
Where inference runs	Outside Domino, at the provider	On Domino compute, on hardware you choose
Who manages the model	The provider handles hosting, scaling, and versioning	You do, through a Domino endpoint
Reach for it when	You want a frontier model and can send prompts outside the cluster	External providers are off the table, or you need data residency, cost control, or your own fine-tuned model
What you configure	An admin registers the provider in LLM Gateway 2.0 and publishes an alias. Without the gateway, you install the provider’s SDK and supply an API key.	Registration, endpoint, GPU hardware tier, and scaling. An admin can then publish the endpoint as a gateway alias.

Which model location fits is a question about your constraints, not a default. Teams that already have access to a managed LLM service often start there, since it’s the shorter path. Teams that can’t send prompts to a third party, whether because of data residency rules, a policy against external model providers, or an air-gapped deployment, host in Domino from the outset and get the same workflow. Most teams end up using both. Either way, the decision stays reversible. Through the gateway, an admin can change the provider or model behind an alias without any change to caller code. Without the gateway, standardize on an OpenAI-compatible client: Domino-hosted endpoints expose an OpenAI-compatible API, so you can move between an external provider and a Domino-hosted model by changing the base URL and model name.

Option A: Connect to an external LLM provider

Most teams already have access to a managed LLM service. Domino runs your code, and the LLM inference happens outside Domino, at the provider.

Call the provider through LLM Gateway 2.0

An admin registers the provider in LLM Gateway 2.0 once and publishes its models as aliases. The gateway encrypts the provider credentials at rest, so nobody needs a provider API key in a Project, a Workspace, or code. Supported providers are OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, Google Vertex AI, OpenRouter, and any endpoint that speaks the OpenAI API. From a Workspace, Job, or App, get a token from the local sidecar and point an OpenAI-compatible client at the gateway:

import requests
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider

# Domino issues a short-lived token from the local sidecar; no provider key needed
token = requests.get("http://localhost:8899/access-token").text
model = OpenAIChatModel(
    "<alias-name>",
    provider=OpenAIProvider(
        base_url="https://<your-domino-host>/apps/<gateway-app-id>/v1",
        api_key=token,
    ),
)
agent = Agent(model)

result = agent.run_sync("Hello")
print(result.output)

Sidecar tokens expire quickly, so fetch a fresh token before each request in long-running code. See LLM Gateway 2.0 for every endpoint the gateway exposes, and Control access, spend, and usage for who can call each alias.

Domino distributes LLM Gateway 2.0 as a binary that an admin deploys as a Domino App, and plans to natively integrate it into a future release. Contact your Domino field representative for access. LLM Gateway 2.0 is the successor to the legacy AI Gateway, which is no longer available in Domino Cloud.

Call the provider directly

If your deployment doesn’t run LLM Gateway 2.0 yet, your code can call the provider’s API directly. Every Project that calls the provider then holds its own copy of the API key, and calls aren’t covered by gateway budgets, guardrails, or usage tracking.

Provider	SDK / package	Typical environment variable
OpenAI	`openai`	`OPENAI_API_KEY`
Anthropic	`anthropic`	`ANTHROPIC_API_KEY`
AWS Bedrock	`boto3`	`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
Azure OpenAI	`openai`	`AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`
Google Vertex AI	`google-cloud-aiplatform`	Service account credentials

Install the provider's SDK

Add the package to your Environment’s Dockerfile instructions (for example, RUN pip install openai), or install it in your Workspace.

Store API keys as Domino environment variables

In your Project settings, add environment variables for your credentials. Domino injects them into Workspaces and Jobs automatically, and your code reads them at runtime through os.environ. Never put keys in code or version control.

Call the provider's SDK as you would locally

Read the key from os.environ and pass it to the provider’s client, for example OpenAIProvider(api_key=os.environ["OPENAI_API_KEY"]). Nothing about the call is Domino-specific. When the gateway becomes available, switch the client’s base URL and model to a gateway alias and remove the key from the Project.

Option B: Host a model in Domino

Domino can host a model as an endpoint on your own compute. Teams take this route when an external provider isn’t permitted at all, and when they need data residency, cost control, or a model they fine-tuned themselves. The endpoint exposes an OpenAI-compatible API. Host an LLM has the full setup guide, including:

Registering a model from Hugging Face or a custom checkpoint.
Deploying it as a Domino endpoint.
Configuring GPU hardware and scaling.

Publish the endpoint through LLM Gateway 2.0

Once the endpoint is running, have an admin publish it as an alias in LLM Gateway 2.0. The gateway reaches Domino-hosted endpoints through its built-in Domino Platform provider, which needs no credentials. Callers then use the same base URL and token for your hosted model as for external providers, and the gateway applies grants and budgets, guardrails, and cost tracking to it. A hosted model can also serve as a fallback for an external alias. Without the gateway, callers use the endpoint URL from the endpoint’s Calling tab directly, and the endpoint’s own access controls decide who can call it.

Mix approaches

Many production systems use both an external frontier model for primary reasoning and a Domino-hosted model for specialized tasks such as a fine-tuned classifier or embedding model. LLM Gateway 2.0 puts both behind one base URL, one token, and one usage dashboard. If you’re building an agent, the @add_tracing instrumentation described in Develop agentic systems captures traces from all LLM calls regardless of where the model is hosted.

You can swap models during experimentation. In Domino, model switching is a config change, not an infrastructure change. You can update a YAML file to move between external and Domino-hosted models, or point your code at a different gateway alias, making it easy to evaluate cost, reliability, and performance as you iterate.

LLM Gateway 2.0: route calls to external providers and Domino-hosted endpoints through one base URL, with access grants, budgets, guardrails, and cost tracking.
Host an LLM: register a model and deploy it as a Domino endpoint, including the access controls on that endpoint.
Agents: build, evaluate, and deploy an agentic system on top of these endpoints.

Core Concepts

Features

Set up LLM access

Option A: Connect to an external LLM provider

Call the provider through LLM Gateway 2.0

Call the provider directly

Option B: Host a model in Domino

Publish the endpoint through LLM Gateway 2.0

Mix approaches

​Option A: Connect to an external LLM provider

​Call the provider through LLM Gateway 2.0

​Call the provider directly

​Option B: Host a model in Domino

​Publish the endpoint through LLM Gateway 2.0

​Mix approaches

​Related

Option A: Connect to an external LLM provider

Call the provider through LLM Gateway 2.0

Call the provider directly

Option B: Host a model in Domino

Publish the endpoint through LLM Gateway 2.0

Mix approaches

Related