> ## Documentation Index
> Fetch the complete documentation index at: https://docs.domino.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploy and operate LLM Gateway 2.0

> Deploy LLM Gateway 2.0 as a Domino App, upgrade it, and keep it running with backups, log purge, warehouse mirroring, health checks, and environment variables.

An admin deploys LLM Gateway 2.0 once, as an App in its own Project, then keeps it healthy from the gateway **Settings** page. The release is a self-contained Linux x86\_64 binary, so the runtime container doesn't need Python or a `pip install` at startup.

## Before you start

* The App owner must be a Domino admin. The gateway reads organization memberships and role assignments with the owner's credentials. With a non-admin owner, organization grants never match and admin detection fails.
* Use a dedicated Project, and restrict its Dataset to the admin owner. The Dataset holds the gateway database, the key that encrypts provider credentials, and any backups. Anyone with Dataset access can read them.
* Use a hardware tier with at least 2 CPU cores and 8 GB of RAM.
* Use a Compute Environment based on Ubuntu 22.04 or later. The binary needs glibc 2.35.

## Deploy the App

<Steps>
  <Step title="Get and verify the release">
    Get the `llm-gateway-X.Y.Z-linux-x86_64.tar.gz` release from your Domino field representative. Verify it against its `.sha256` file with `sha256sum -c`.
  </Step>

  <Step title="Unpack the release in the Project">
    Upload the tarball to the Project's Files, then unpack it from a Workspace terminal:

    ```bash theme={null}
    tar -xzf llm-gateway-*-linux-x86_64.tar.gz
    ```

    This creates `./llm-gateway-X.Y.Z/`. Don't rename the folder: `app.sh` finds it by name.
  </Step>

  <Step title="Configure the App">
    In the Project's App tab, configure the App:

    | Setting | Value |
    | :- | :- |
    | Run command | `app.sh` |
    | Hardware tier | 2+ CPU, 8+ GB RAM |
    | Enable deep linking | On. The gateway identifies callers from the signed JSON Web Token (JWT) that deep linking sends. |
    | Allow App to act for viewers in Domino | Off. The gateway needs to know who the viewer is, not act as them. |
    | View permissions | Your choice. Access grants control who can call models, independent of who can see the UI. |
  </Step>

  <Step title="Publish the App">
    Click **Publish**. The first line in the App log reads `[app.sh] launching binary: ./llm-gateway-X.Y.Z/llm-gateway`.
  </Step>

  <Step title="Complete first boot">
    Open the App URL as the owner. On first boot, the gateway creates its database and encryption key in the Project Dataset and adds the Domino Platform provider.
  </Step>

  <Step title="Add providers and aliases">
    Add providers and aliases from the UI, then [grant access](/cloud/admin/platform-configuration/llm-gateway/access-and-spend#grants) to them.
  </Step>
</Steps>

## Upgrade

Unpack the new release next to the old folder, then restart the App. `app.sh` starts the newest version it finds. After you confirm the upgrade, delete the old folder. The database, encryption key, and scheduled Jobs live in the Dataset, so they carry over.

## Troubleshoot

| Problem | Fix |
| :- | :- |
| Log shows `no binary found, falling back to source mode` | The release folder was renamed or unpacked somewhere else. Unpack it into the Project root under its original name. |
| Binary fails to load | The Compute Environment is older than Ubuntu 22.04. Switch to a newer base image. |
| Organization grants never apply | The App owner isn't a Domino admin. Republish the App under an admin. |

## Operate the gateway

The gateway **Settings** page holds 3 opt-in features that keep the gateway database small and recoverable: backups, log purge, and warehouse mirroring. Each runs as a Domino scheduled Job that calls back into the App.

### Back up the database

Schedule daily, weekly, monthly, or quarterly backups, or click **Run now**. Backups go to `<dataset>/llm_gateway/backups/` by default and are standard SQLite files. The gateway prunes older backups according to the retention setting.

To restore, stop the App, replace `llm_gateway.db` in the Dataset with a backup, and restart. Provider credentials in the backup stay encrypted, and the `.gateway_key` file next to the database decrypts them.

### Purge old logs

Delete usage and audit logs older than a set horizon. 6 months is a good default. The gateway won't purge inside your longest active budget period, and can refuse to purge without a backup from the last 7 days. Use **Dry run** to see row counts first.

### Mirror logs to a warehouse

Mirror usage logs, audit logs, or both to a Domino Data Source: Snowflake, Postgres, Redshift, BigQuery, MySQL, ClickHouse, or MariaDB.

<Steps>
  <Step title="Pick the Data Source and tables">
    In Settings, pick the Data Source and table names, then click **Test connection**.
  </Step>

  <Step title="Create the table">
    Click **Create table**.
  </Step>

  <Step title="Turn on mirroring">
    Set each log type to `mirror`. Rows arrive 5 to 15 seconds after the request.
  </Step>

  <Step title="Copy history">
    Click **Backfill now** to copy existing logs. It's safe to rerun.
  </Step>
</Steps>

Once mirroring is on, admins can switch the Usage and Logs pages between local data and the warehouse. Budgets always read local data, because that's what the gateway enforces. With purge and mirroring both on, purge deletes only rows that have been mirrored.

### Monitor health

`GET /health` returns `{"status": "ok"}`. `GET /health?verbose=1` adds database, backup age, and mirror status. When the mirror queue backs up or the last backup is older than twice its interval, the gateway writes a warning to the audit log.

## Environment variables

Set these under **App Configuration > Environment**. A standard deployment needs none of them.

| Variable | What it does | Default |
| :- | :- | :- |
| `LLM_GATEWAY_DB` | Path to the SQLite database | Project Dataset |
| `LLM_GATEWAY_KEY_FILE` | Path to the credential encryption key. Keep it next to the database. | Project Dataset |
| `LLM_GATEWAY_ENCRYPTION_KEY` | The encryption key value, for injection from a secrets manager | Unset |
| `LLM_GATEWAY_SIGNING_KEY` | RSA private key for signing per-user tokens. Disables automatic rotation. | Unset |
| `LLM_GATEWAY_KEY_ROTATION_HOURS` | Signing key rotation interval. `0` disables rotation. | `24` |
| `LLM_GATEWAY_MAX_THINKING_BUDGET_TOKENS` | Cap on extended-thinking tokens per request on the Anthropic endpoint | Unset |
| `LLM_GATEWAY_DB_POOL_SIZE` | Database connections kept open. Raise this first under heavy load. | `150` |
| `DOMINO_USER_ORGS_CACHE_TTL_SECONDS` | How long the gateway caches organization memberships, in seconds | `300` |
| `DOMINO_USER_ROLES_CACHE_TTL_SECONDS` | How long the gateway caches Domino admin roles, in seconds | `300` |

<Warning>
  Don't change the encryption key or delete `.gateway_key` without re-encrypting the database. Every stored provider credential becomes unreadable and has to be entered again.
</Warning>

## Related

* [LLM Gateway 2.0](/cloud/platform-capabilities/features/llms/llm-gateway): what the gateway routes, its endpoints, and how callers make their first request.
* [Control access, spend, and usage](/cloud/admin/platform-configuration/llm-gateway/access-and-spend): grant callers access to aliases, set budgets, and track cost.
* [Configure guardrails and fallback routing](/cloud/admin/platform-configuration/llm-gateway/guardrails-and-fallback): screen requests and route around failures.


## Related topics

- [LLM Gateway 2.0](/cloud/platform-capabilities/features/llms/llm-gateway/index.md)
- [Control access, spend, and usage](/cloud/admin/platform-configuration/llm-gateway/access-and-spend.md)
- [Configure Domino](/cloud/admin/platform-configuration/index.md)
- [Set up LLM access](/cloud/platform-capabilities/features/llms/index.md)