> ## Documentation Index
> Fetch the complete documentation index at: https://docs.domino.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Set up LLM access

> How a Large Language Model becomes callable from your code in Domino, and who is allowed to call it: connect to an external provider, host your own model, or mix both.

Any code you run in Domino that calls a Large Language Model (LLM) needs an endpoint to call and credentials to call it with. That applies whether the caller is an [agent](/cloud/platform-capabilities/features/agents), an [App](/cloud/platform-capabilities/features/apps), a [Job](/cloud/platform-capabilities/core-concepts/jobs), or exploratory code in a [Workspace](/cloud/platform-capabilities/core-concepts/workspaces). Decide how your code reaches its models before you start building.

```mermaid theme={null}
flowchart LR
  code["Your code<br/>(agent, App, Job, Workspace)"]
  ext["External provider<br/>OpenAI, Anthropic, Bedrock,<br/>Azure OpenAI, Vertex AI"]
  dom["Domino-hosted endpoint<br/>on your compute"]

  code -->|"API key in an env var"| ext
  code -->|"OpenAI-compatible call"| dom
```

Domino supports two approaches, and most teams end up using both:

|                       | Connect to an external provider                                    | Host a model in Domino                                                                                       |
| :-------------------- | :----------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------- |
| Where inference runs  | Outside Domino, at the provider                                    | On Domino compute, on hardware you choose                                                                    |
| Who manages the model | The provider handles hosting, scaling, and versioning              | You do, through a Domino endpoint                                                                            |
| Reach for it when     | You want a frontier model and can send prompts outside the cluster | External providers are off the table, or you need data residency, cost control, or your own fine-tuned model |
| What you configure    | The provider's SDK and an API key                                  | Registration, endpoint, GPU hardware tier, and scaling                                                       |

Which one fits is a question about your constraints, not a default. Teams that already have access to a managed LLM service often start there, since it's the shorter path. Teams that can't send prompts to a third party, whether because of data residency rules, a policy against external model providers, or an [air-gapped deployment](/cloud/platform-overview/deployment-options), host in Domino from the outset and get the same workflow. Both are first-class paths.

Either way, the decision stays reversible: if you standardize on an OpenAI-compatible client, you can move between an external provider and a Domino-hosted model later without changing how your code calls it, since Domino-hosted endpoints expose an OpenAI-compatible API.

## Option A: Connect to an external LLM provider

Most teams already have access to a managed LLM service. Your code calls the provider's API directly. Domino runs your code, and the LLM inference happens outside Domino.

Common providers include:

| Provider         | SDK / package             | Typical environment variable                    |
| ---------------- | ------------------------- | ----------------------------------------------- |
| OpenAI           | `openai`                  | `OPENAI_API_KEY`                                |
| Anthropic        | `anthropic`               | `ANTHROPIC_API_KEY`                             |
| AWS Bedrock      | `boto3`                   | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`    |
| Azure OpenAI     | `openai`                  | `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT` |
| Google Vertex AI | `google-cloud-aiplatform` | Service account credentials                     |

### How to configure access in Domino

<Steps>
  <Step title="Install the provider's SDK">
    Add the package to your [Environment's](/cloud/platform-capabilities/core-concepts/compute-environments) Dockerfile instructions (for example, `RUN pip install openai`), or install it in your Workspace.
  </Step>

  <Step title="Store API keys as Domino environment variables">
    In your Project settings, add environment variables for your credentials. Domino injects them into Workspaces and Jobs automatically, and your code reads them at runtime through `os.environ`. This keeps secrets out of your code and out of version control.
  </Step>

  <Step title="Call the provider's SDK as you would locally">
    Read the key from `os.environ` and pass it to the provider's client, as in the example below. Nothing about the call is Domino-specific; the provider handles model hosting, scaling, and versioning.
  </Step>
</Steps>

```python theme={null}
import os
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider

# API key is read from a Domino environment variable
model = OpenAIChatModel(
    "gpt-5.4-mini",
    provider=OpenAIProvider(api_key=os.environ["OPENAI_API_KEY"]),
)
agent = Agent(model)

result = agent.run_sync("Hello")
print(result.output)
```

<Note>
  The legacy AI Gateway is no longer available in Domino Cloud. [LLM Gateway 2.0](/cloud/platform-capabilities/features/llms/llm-gateway) is its successor, with usage tracking, cost controls, guardrails, and broader model and vendor coverage. Domino distributes it today as a binary deployed as a Domino app and plans to natively integrate it into a future release. Contact your Domino field team member or representative for access.
</Note>

## Option B: Host a model in Domino

Domino can host a model as an endpoint on your own compute. Teams take this route when an external provider isn't permitted at all, and when they need data residency, cost control, or a model they fine-tuned themselves. Your code calls the Domino-hosted endpoint the same way it would call any other API, using an OpenAI-compatible interface.

[Host an LLM](/cloud/platform-capabilities/features/llms/host-an-llm) has the full setup guide, including:

* Registering a model from Hugging Face or a custom checkpoint.

* Deploying it as a Domino endpoint.

* Configuring GPU hardware and scaling.

## Mix approaches

Many production systems use both an external **frontier model** for primary reasoning and a **Domino-hosted model** for specialized tasks such as a fine-tuned classifier or embedding model.

If you're building an agent, the `@add_tracing` instrumentation described in [Develop agentic systems](/cloud/platform-capabilities/features/agents/develop) captures traces from all LLM calls regardless of where the model is hosted.

<Tip>
  You can swap models during experimentation. In Domino, model switching is a config change, not an infrastructure change. You can update a YAML file to move between external and Domino-hosted models, making it easy to evaluate cost, reliability, and performance as you iterate.
</Tip>

## Related

* [Host an LLM](/cloud/platform-capabilities/features/llms/host-an-llm): register a model and deploy it as a Domino endpoint, including the access controls on that endpoint.

* [LLM Gateway 2.0](/cloud/platform-capabilities/features/llms/llm-gateway): the migration path for teams that were using the legacy AI Gateway before it was removed.

* [Agents](/cloud/platform-capabilities/features/agents): build, evaluate, and deploy an agentic system on top of these endpoints.


## Related topics

- [Set up LLM access](/cloud/platform-capabilities/features/llms/index.md)
- [LLM Gateway 2.0](/cloud/platform-capabilities/features/llms/llm-gateway.md)
- [Host an LLM](/6.3/platform-capabilities/features/llms/host-an-llm.md)
- [Develop agentic systems](/cloud/platform-capabilities/features/agents/develop.md)