Skip to main content
The legacy AI Gateway has been removed from Domino Cloud. The Gateway LLMs UI, the AI Gateway endpoint APIs, and the endpoint audit trail are no longer available. Any code that called AI Gateway endpoints through the MLflow Deployment Client, or the AI Gateway routes on the Domino Platform API, stops working after your deployment upgrades. This page is retained so existing links resolve.
LLM Gateway 2.0 is the successor to the legacy AI Gateway. It centralizes external provider access and adds capabilities the legacy gateway did not have:
  • Usage tracking across users, Projects, and models.
  • Cost controls, including per-team and per-model budgets.
  • Guardrails for prompt and response inspection.
  • Broader model and vendor coverage.
Domino distributes LLM Gateway 2.0 today as a binary that is deployed as a Domino app, and plans to natively integrate it into a future Domino release. Contact your Domino field team member or representative for access.

​
Other replacement paths

If LLM Gateway 2.0 is more than you need, or if you’re routing calls before you get access, pick one of these direct paths from Workspaces, Jobs, Apps, and agents.

Call an external provider directly

Install the provider’s SDK, store the API key as a Domino environment variable, and call the provider from your code. Works with OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, and Google Vertex AI.

Host a model in Domino

Register a model from Hugging Face or a custom checkpoint and deploy it as a Domino endpoint on your own compute. Callers use an OpenAI-compatible API.

​
Migrate off the legacy AI Gateway

1

Identify code that used AI Gateway endpoints

Search your Projects for calls to mlflow.deployments.get_deploy_client("databricks") or predict() / get_endpoint() / list_endpoints() against a Domino AI Gateway target. Any such calls fail after the upgrade.
2

Pick a replacement path

Ask your Domino field representative about LLM Gateway 2.0 if you want a centralized gateway with usage tracking, cost controls, and guardrails. Otherwise, use the provider’s SDK directly (Option A on Set up LLM access), or host the model in Domino (Option B).
3

Move credentials off AI Gateway's central vault

Provider credentials that AI Gateway held in Domino’s central vault are no longer accessible through the AI Gateway UI or API. Store the credentials as Project environment variables, or as inputs to your LLM Gateway 2.0 configuration, and read them from os.environ in your code.
4

Update code and rerun

Replace the MLflow Deployment Client call with a call through LLM Gateway 2.0, a direct provider SDK call, or a call to your Domino-hosted endpoint. Rerun affected Jobs, Apps, and agents to confirm they work end to end.
  • Set up LLM access: decide between an external provider and a Domino-hosted model.
  • Host an LLM: full setup guide for a Domino-hosted endpoint.

Related topics

Set up LLM accessClinical Data ExplorerReuse your coding agent subscription across workspacesGlossary