Set up LLM access

Any code you run in Domino that calls a Large Language Model (LLM) needs an endpoint to call and credentials to call it with. That applies whether the caller is an agent, an App, a Job, or exploratory code in a Workspace. Decide how your code reaches its models before you start building. Domino supports two approaches, and most teams end up using both:

	Connect to an external provider	Host a model in Domino
Where inference runs	Outside Domino, at the provider	On Domino compute, on hardware you choose
Who manages the model	The provider handles hosting, scaling, and versioning	You do, through a Domino endpoint
Reach for it when	You want a frontier model and can send prompts outside the cluster	External providers are off the table, or you need data residency, cost control, or your own fine-tuned model
What you configure	The provider’s SDK and an API key	Registration, endpoint, GPU hardware tier, and scaling

Which one fits is a question about your constraints, not a default. Teams that already have access to a managed LLM service often start there, since it’s the shorter path. Teams that can’t send prompts to a third party, whether because of data residency rules, a policy against external model providers, or an air-gapped deployment, host in Domino from the outset and get the same workflow. Both are first-class paths. Either way, the decision stays reversible: if you standardize on an OpenAI-compatible client, you can move between an external provider and a Domino-hosted model later without changing how your code calls it, since Domino-hosted endpoints expose an OpenAI-compatible API.

Option A: Connect to an external LLM provider

Most teams already have access to a managed LLM service. Your code calls the provider’s API directly. Domino runs your code, and the LLM inference happens outside Domino. Common providers include:

Provider	SDK / package	Typical environment variable
OpenAI	`openai`	`OPENAI_API_KEY`
Anthropic	`anthropic`	`ANTHROPIC_API_KEY`
AWS Bedrock	`boto3`	`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
Azure OpenAI	`openai`	`AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`
Google Vertex AI	`google-cloud-aiplatform`	Service account credentials

How to configure access in Domino

Install the provider's SDK

Add the package to your Environment’s Dockerfile instructions (for example, RUN pip install openai), or install it in your Workspace.

Store API keys as Domino environment variables

In your Project settings, add environment variables for your credentials. Domino injects them into Workspaces and Jobs automatically, and your code reads them at runtime through os.environ. This keeps secrets out of your code and out of version control.

Call the provider's SDK as you would locally

Read the key from os.environ and pass it to the provider’s client, as in the example below. Nothing about the call is Domino-specific; the provider handles model hosting, scaling, and versioning.

import os
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider

# API key is read from a Domino environment variable
model = OpenAIChatModel(
    "gpt-5.4-mini",
    provider=OpenAIProvider(api_key=os.environ["OPENAI_API_KEY"]),
)
agent = Agent(model)

result = agent.run_sync("Hello")
print(result.output)

Deployments where an administrator has explicitly enabled the legacy AI Gateway can route external provider calls through it instead, which centralizes credentials and adds an audit trail of every LLM interaction. It is not enabled by default.

Option B: Host a model in Domino

Domino can host a model as an endpoint on your own compute. Teams take this route when an external provider isn’t permitted at all, and when they need data residency, cost control, or a model they fine-tuned themselves. Your code calls the Domino-hosted endpoint the same way it would call any other API, using an OpenAI-compatible interface. Host an LLM has the full setup guide, including:

Registering a model from Hugging Face or a custom checkpoint.
Deploying it as a Domino endpoint.
Configuring GPU hardware and scaling.

Mix approaches

Many production systems use both an external frontier model for primary reasoning and a Domino-hosted model for specialized tasks such as a fine-tuned classifier or embedding model. If you’re building an agent, the @add_tracing instrumentation described in Develop agentic systems captures traces from all LLM calls regardless of where the model is hosted.

You can swap models during experimentation. In Domino, model switching is a config change, not an infrastructure change. You can update a YAML file to move between external and Domino-hosted models, making it easy to evaluate cost, reliability, and performance as you iterate.

Host an LLM: register a model and deploy it as a Domino endpoint, including the access controls on that endpoint.
AI Gateway (legacy): centralized external provider access, where a deployment has it enabled, including per-endpoint permissions for users and organizations.
Agents: build, evaluate, and deploy an agentic system on top of these endpoints.

Core Concepts

Features

Set up LLM access

Option A: Connect to an external LLM provider

How to configure access in Domino

Option B: Host a model in Domino

Mix approaches

​Option A: Connect to an external LLM provider

​How to configure access in Domino

​Option B: Host a model in Domino

​Mix approaches

​Related

Option A: Connect to an external LLM provider

How to configure access in Domino

Option B: Host a model in Domino

Mix approaches

Related