LLM Gateway 2.0

The legacy AI Gateway has been removed from Domino Cloud. The Gateway LLMs UI, the AI Gateway endpoint APIs, and the endpoint audit trail are no longer available. Any code that called AI Gateway endpoints through the MLflow Deployment Client, or the AI Gateway routes on the Domino Platform API, stops working after your deployment upgrades. This page is retained so existing links resolve.

LLM Gateway 2.0 sits between your code and the large language models (LLMs) it calls. Callers send requests to one base URL with a model alias. The gateway checks who is calling, applies budgets and guardrails, logs the cost, and forwards the request to the provider behind that alias. LLM Gateway 2.0 replaces the legacy AI Gateway. Domino distributes it as a binary that you deploy as a Domino App, and plans native integration in a future release. Contact your Domino field representative for access.

What the gateway routes

An admin registers providers, then publishes models from those providers as aliases. Callers pass the alias name as model. You can change the provider or model behind an alias without changing caller code. Supported providers:

Models hosted as Domino endpoints, through the built-in Domino Platform provider. No credentials needed.
OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, Google Vertex AI, and OpenRouter.
Any endpoint that speaks the OpenAI API.
System One decision models, such as TypeSafe’s Jev. See Call decision models.

The gateway encrypts provider credentials at rest in its Project Dataset.

What the gateway adds

Capability	What it does
Access grants	Decide who can call each alias: a user, an organization, a service account, all users, or callers bringing their own provider key.
Budgets	Cap spend per alias and per user, organization, or service account, on a daily, weekly, or monthly period.
Usage and cost	Record the estimated cost of every request and break it down by model, user, organization, or your own tags.
Guardrails	Screen prompts and responses with regex rules or an external LLM evaluator, then deny, redact, or log.
Fallback routing	Retry a failed or over-budget request on the next alias in a chain.
Claude Code	Run Claude Code against any alias through an Anthropic Messages endpoint.
Batch	Submit OpenAI and Anthropic batch jobs, with the vendor’s batch discount where the vendor supports it.

Endpoints

The base URL is https://<your-domino-host>/apps/<gateway-app-id>.

Path	Protocol	Use it for
`/v1/chat/completions`	OpenAI Chat Completions	Most chat and agent traffic. The only path that supports fallback chains.
`/v1/responses`	OpenAI Responses	Clients built on the Responses API
`/v1/embeddings`	OpenAI Embeddings	Embedding models
`/anthropic/v1/messages`	Anthropic Messages	Claude Code and the Anthropic software development kit (SDK)
`/v1/systemone`	System One	Decision models
`/v1/files`, `/v1/batches`	OpenAI Batch	Batch jobs through the OpenAI SDK
`/anthropic/v1/messages/batches`	Anthropic Batch	Batch jobs through the Anthropic SDK

Make your first call

From a Workspace, Job, or App, get a token from the local sidecar and point the OpenAI SDK at the gateway:

import requests
from openai import OpenAI

token = requests.get("http://localhost:8899/access-token").text
client = OpenAI(
    base_url="https://<your-domino-host>/apps/<gateway-app-id>/v1",
    api_key=token,
)

response = client.chat.completions.create(
    model="<alias-name>",
    messages=[{"role": "user", "content": "Summarize this incident report."}],
)

Sidecar tokens expire quickly. In long-running code, fetch a fresh token before each request. From outside Domino, use a Domino API key instead. To attribute spend to your own dimensions, add X-LLM-Tag-<key>: <value> headers to any /v1/* request, for example X-LLM-Tag-team: fraud. Tags show up as filters and breakdowns in the usage dashboard.

Deploy and operate LLM Gateway 2.0

Publish the gateway as an App, upgrade it, and manage backups, logs, and settings.

Control access, spend, and usage

Set grants, pricing, and budgets for each alias, and track usage and cost.

Use Claude Code with the gateway

Point Claude Code at any alias from a Workspace or from outside Domino.

Call decision models

Get typed, calibrated answers from System One decision models.

Other replacement paths

If LLM Gateway 2.0 is more than you need, or if you’re routing calls before you get access, pick one of these direct paths from Workspaces, Jobs, Apps, and agents.

Call an external provider directly

Install the provider’s SDK, store the API key as a Domino environment variable, and call the provider from your code. Works with OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, and Google Vertex AI.

Host a model in Domino

Register a model from Hugging Face or a custom checkpoint and deploy it as a Domino endpoint on your own compute. Callers use an OpenAI-compatible API.

Migrate off the legacy AI Gateway

Identify code that used AI Gateway endpoints

Search your Projects for calls to mlflow.deployments.get_deploy_client("databricks") or predict() / get_endpoint() / list_endpoints() against a Domino AI Gateway target. Any such calls fail after the upgrade.

Pick a replacement path

Ask your Domino field representative about LLM Gateway 2.0 if you want a centralized gateway with usage tracking, cost controls, and guardrails. Otherwise, use the provider’s SDK directly (Option A on Set up LLM access), or host the model in Domino (Option B).

Move credentials off AI Gateway's central vault

Provider credentials that AI Gateway held in Domino’s central vault are no longer accessible through the AI Gateway UI or API. Store the credentials as Project environment variables, or as inputs to your LLM Gateway 2.0 configuration, and read them from os.environ in your code.

Update code and rerun

Replace the MLflow Deployment Client call with a call through LLM Gateway 2.0, a direct provider SDK call, or a call to your Domino-hosted endpoint. Rerun affected Jobs, Apps, and agents to confirm they work end to end.

Set up LLM access: decide where your models run and how your code reaches them through the gateway.
Host an LLM: full setup guide for a Domino-hosted endpoint.

Core Concepts

Features

LLM Gateway 2.0

What the gateway routes

What the gateway adds

Endpoints

Make your first call

Deploy and operate LLM Gateway 2.0

Control access, spend, and usage

Use Claude Code with the gateway

Call decision models

Other replacement paths

Call an external provider directly

Host a model in Domino

Migrate off the legacy AI Gateway

​What the gateway routes

​What the gateway adds

​Endpoints

​Make your first call

Deploy and operate LLM Gateway 2.0

Control access, spend, and usage

Use Claude Code with the gateway

Call decision models

​Other replacement paths

Call an external provider directly

Host a model in Domino

​Migrate off the legacy AI Gateway

​Related

What the gateway routes

What the gateway adds

Endpoints

Make your first call

Other replacement paths

Migrate off the legacy AI Gateway

Related