> ## Documentation Index
> Fetch the complete documentation index at: https://docs.domino.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM Gateway 2.0

> The legacy AI Gateway has been removed from Domino Cloud. LLM Gateway 2.0 is its successor, distributed today as a Domino-deployed app and planned for native integration in a future release.

<Warning>
  **The legacy AI Gateway has been removed from Domino Cloud.** The **Gateway LLMs** UI, the AI Gateway endpoint APIs, and the endpoint audit trail are no longer available. Any code that called AI Gateway endpoints through the MLflow Deployment Client, or the AI Gateway routes on the Domino Platform API, stops working after your deployment upgrades. This page is retained so existing links resolve.
</Warning>

LLM Gateway 2.0 is the successor to the legacy AI Gateway. It centralizes external provider access and adds capabilities the legacy gateway did not have:

* Usage tracking across users, Projects, and models.
* Cost controls, including per-team and per-model budgets.
* Guardrails for prompt and response inspection.
* Broader model and vendor coverage.

Domino distributes LLM Gateway 2.0 today as a binary that is deployed as a Domino app, and plans to natively integrate it into a future Domino release. Contact your Domino field team member or representative for access.

## Other replacement paths

If LLM Gateway 2.0 is more than you need, or if you're routing calls before you get access, pick one of these direct paths from Workspaces, Jobs, Apps, and agents.

<Columns cols={2}>
  <Card title="Call an external provider directly" href="/cloud/platform-capabilities/features/llms#option-a-connect-to-an-external-llm-provider">
    Install the provider's SDK, store the API key as a Domino environment variable, and call the provider from your code. Works with OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, and Google Vertex AI.
  </Card>

  <Card title="Host a model in Domino" href="/cloud/platform-capabilities/features/llms/host-an-llm">
    Register a model from Hugging Face or a custom checkpoint and deploy it as a Domino endpoint on your own compute. Callers use an OpenAI-compatible API.
  </Card>
</Columns>

## Migrate off the legacy AI Gateway

<Steps>
  <Step title="Identify code that used AI Gateway endpoints">
    Search your Projects for calls to `mlflow.deployments.get_deploy_client("databricks")` or `predict()` / `get_endpoint()` / `list_endpoints()` against a Domino AI Gateway target. Any such calls fail after the upgrade.
  </Step>

  <Step title="Pick a replacement path">
    Ask your Domino field representative about LLM Gateway 2.0 if you want a centralized gateway with usage tracking, cost controls, and guardrails. Otherwise, use the provider's SDK directly (Option A on [Set up LLM access](/cloud/platform-capabilities/features/llms)), or host the model in Domino (Option B).
  </Step>

  <Step title="Move credentials off AI Gateway's central vault">
    Provider credentials that AI Gateway held in Domino's central vault are no longer accessible through the AI Gateway UI or API. Store the credentials as Project environment variables, or as inputs to your LLM Gateway 2.0 configuration, and read them from `os.environ` in your code.
  </Step>

  <Step title="Update code and rerun">
    Replace the MLflow Deployment Client call with a call through LLM Gateway 2.0, a direct provider SDK call, or a call to your Domino-hosted endpoint. Rerun affected Jobs, Apps, and agents to confirm they work end to end.
  </Step>
</Steps>

## Related

* [Set up LLM access](/cloud/platform-capabilities/features/llms): decide between an external provider and a Domino-hosted model.

* [Host an LLM](/cloud/platform-capabilities/features/llms/host-an-llm): full setup guide for a Domino-hosted endpoint.


## Related topics

- [Set up LLM access](/cloud/platform-capabilities/features/llms/index.md)
- [Clinical Data Explorer](/cloud/platform-capabilities/features/extensions/data-visualization.md)
- [Reuse your coding agent subscription across workspaces](/cloud/platform-capabilities/features/coding-assistants/reuse-credentials.md)
- [Glossary](/cloud/reference/glossary.md)