# Customize Environments
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/index

Customize Domino Environments by managing revisions, packages, Workspace tools, external images, DNS access, and custom images for models and Apps.

[Manage Compute Environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments)\
Learn how to work with Domino Environments.

[Add packages to Environments](/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments)\
Add a package to the Environment used in a Job, Workspace, or App.

[Work with Pluggable Workspaces](/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces)\
Leverage a variety of Workspaces based on specific languages, development tools, and software.

[Use external images in Domino Environments](/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images)\
Use pre-built images from external catalogs.

[Deploy a custom image](/cloud/platform-capabilities/core-concepts/compute-environments/deploy-a-custom-image)\
Use Dockerfile instructions to enable custom images to publish models or host web applications.

[Access additional domains and hostnames](/cloud/platform-capabilities/core-concepts/compute-environments/access-additional-domains-and-hostnames)\
Use custom DNS name servers to handle internal or private domain names and URLs.


# Best Practices for Domino Environments
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/1-best-practices-domino-environments

Manage Domino Environments with clear names, controlled build propagation, smaller images, cached bases, and limited dependency files.

## Domino environments

Cumbersome Environment definitions can significantly impact startup times for executions that use that Environment.

### Use clear naming conventions and descriptions

**Reason**: A common reason environments get duplicated is that users are unsure if an existing environment meets their needs. Using clear and descriptive names for all environments makes it easier for new users and admins to understand the catalog. This clarity helps make Domino easier to work with and maintain.

**Solution**: Keep things clear for everyone by using these tips:

* Use clear, descriptive names for your environments.

* Enforce a standard of quality in naming and Dockerfile commenting.

* Make sure you don’t make a copy of an environment without renaming it.

### Use build propagation to keep environments updated

**Reason**: Prevents users from having to manually update environments downstream.

**Solution**: Automatically update environments by:

* Environments that need to be auto-updated are set to **Active (auto-rebuilds)**.

* Testing newly built revisions after updating (ensure **Auto-assign next new revision as Active** is off).

* Once you’re happy with the latest revision, set it as **Active**.

<Note>
  You should assess the downstream impact in the [Dependency Tree tab](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments#view-accessible-environments) prior to making changes, especially if they trigger a lot of rebuilds.
</Note>

### Limit global environments

**Reason**: Focusing on a few global environments with broad uses is more beneficial than creating many specialized ones.

**Solution**: Follow these ideas for limiting global environments:

* Use the default compute environment as the base.

* Create a global environment and image that meets 80-90% of the community’s requirements.

### Create one environment per team to avoid sprawl

**Reason**: When a user asks for a new environment with specific features, consider adding those to an existing global environment instead of making a new environment. Having fewer global environments makes it easier to maintain them.

**Solution**: Here are some ideas to keep sprawl to a minimum:

* Individual users’ environments should be based on a primary global environment.

* Environments should always be based on the latest standard base image.

### Audit and archive environments regularly

**Reason**: Avoid sprawl to maintain control of your environments and prevent slow executions.

**Solution**: Follow these tips to create a culture of tidiness around environment management:

* Share responsibility for environment management.

* Clean up old or poorly maintained environments.

* Be assertive about pruning unnecessary environments.

* Team leads/SMEs are responsible for keeping images up-to-date.

* Have few global environments.

* If several users need to install something larger than 1G, it may be time to update your base image.

* Use Organizations to manage access to restricted environments.

## Building images

These are our best practices for building more efficient Docker images.

### Reduce the Environment image size

**Reason**: If the environment image needed for a Workspace, Job, or other task is not on the compute node where it runs, it must first be sent over the network from the image registry in a compressed format. After the transfer, the image is then decompressed on the compute node. Transferring and decompressing the image slows startup time, and the delay increases as the image size grows.

**Solution**: Follow these tips to minimize the Environment image size:

* Choose the smallest base image that suits your needs, either an existing Domino Environment or a custom image.

* Install only essential software via the Dockerfile.

* Consider creating several smaller, special-purpose Environments rather than one larger, general-purpose Environment.

### Limit dependency file usage

**Reason**: Every package listed in a dependency file, like `requirements.txt`, is installed via a network transfer for every execution. This causes startup latency and is vulnerable to network failures.

**Solution**: Follow these tips to minimize usage of dependency files:

* Move packages from dependency files to the Environment’s Dockerfile.

* Packages in the Dockerfile are installed just once during Environment creation.

* Eliminate rarely used packages from dependency files. On the rare occasions they are needed, install them manually in the running execution or as part of your execution’s code.

### Cache global environments on the executor machine image

**Reason**: If users create environments with base images that differ significantly from what is cached on the machine image, it can take a long time to pull the necessary images when launching executors.

**Solution**: To improve the startup time for your users:

* Cache your global environments in the executor template machine image so that each new executor starts with the cached base Docker image for any environment.

## Next steps

* [Domino Standard Environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) describes the pre-configured environments that come with Domino.

* [Create a Domino Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/3-create-an-environment) using an existing environment or start from a custom setup.

* [Edit an Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition) to enable tools, run scripts, or manage environment variables.

* [Set default Environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/6-set-a-default-environment) establishes a starting point for all new projects or an environment for a specific project.


# Domino Standard Environments
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments

Choose Domino Core, Standard, Cluster, or GPU Environments for Workspaces, Jobs, Apps, endpoints, and distributed compute workloads.

Every Domino installation includes the default Domino Core Environment (DCE) and the Domino Standard Environment (DSE) setup. These environments come with pre-configured data science tools and libraries for Domino users. They can be used for any Domino execution, such as a Workspace, Job, App, or Domino endpoint.

With each major release, Domino updates its core environments with new libraries and packages. You can still update your core environments with custom libraries and packages. Your Domino administrator can guide you through that process if needed.

DCEs are smaller and more straightforward, making them suitable only for the worker nodes in the cluster. DSEs cannot be used for worker nodes in a cluster.

## Domino core environment (DCE)

Domino provides a security-focused, smaller-footprint image called the Domino Core Environment (DCE). It includes only the essential packages needed to operate within Domino.

This can serve as a base to help you create a Domino-compatible environment from scratch, add only the packages you want to it, and speed up the process of building the environment and initiating executions.

The Domino Core Environment supports Jupyter and JupyterLab but lacks some packages available in the Domino Standard Environment (DSE).

Domino recommends the DCE for custom package installations on a base image because its smaller size speeds up builds and minimizes conflicts.

## Domino standard environment (DSE)

The Domino Standard Environment (DSE) is available with standard libraries and packages. This environment has a much larger image size but many common packages for data scientists. It can handle common data science workflows out of the box and includes the most common Python and R packages.

### FUSE environment option

The DSE offers a version that includes FileSystem in Userspace (FUSE) binaries, enabling support for Goofys and SSHFS. You can add these commands to your Dockerfile to enable FUSE functionality:

```dockerfile theme={null}
USER root

# Goofys
ADD https://github.com/kahing/goofys/releases/download/v0.24.0/goofys  /usr/bin/
RUN chmod a+x /usr/bin/goofys

# SSHFS
RUN apt-get update && apt-get install -y sshfs &&
    sed -i "s/^#user_allow_other/user_allow_other/" /etc/fuse.conf

USER ubuntu
```

## Domino cluster environments

Domino Cluster Environments work similarly to Domino Standard Environments (DSEs) but include extra libraries for specific cluster types. They offer the same workspace tools and general packages as DSEs to support data science tasks.

Each Domino environment is designed specifically for the corresponding compute cluster environments. They are most effective when used in a distributed compute cluster alongside a Domino Compute Cluster Environment.

You can use these environments with any Domino execution, such as a Workspace or Job. Here are a few things to remember about clusters in Domino:

* All environments must maintain identical Python versions across the cluster.

* A cluster won’t work correctly if the worker nodes are not using the appropriate Domino Cluster Environment for cluster workers.

* A compatible Domino Compute Environment for Job or Workspace is also needed for proper functionality.

The [Compute Environment catalog](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/8-compute-environment-catalog) has a complete list of pre-built compute environments for you to use with Domino.

| Type | Description |
| - | - |
| **Domino Spark Environment** | This includes Scala and Spark, along with the typical features of DSE. [Work with your Spark cluster](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/work-with-your-cluster) has complete instructions on creating simple Spark clusters for use with Domino Spark environments. |
| **Domino Ray Environment** | This combines Ray features with standard DSE tools. [Create a base Ray cluster environment](/cloud/platform-capabilities/features/compute-clusters-hpc/ray/configure-prerequisites) has complete instructions on creating simple Ray clusters for use with Domino Ray environments. |
| **Domino Dask Environment** | This environment includes Dask on top of the typical DSE functionality. [Work with clusters](/cloud/platform-capabilities/features/compute-clusters-hpc/dask/work-with-your-cluster) has complete instructions on creating simple Dask clusters for use with Domino Dask environments. |
| **Domino MPI Environment** | Provision and orchestrate an Open MPI cluster directly on the infrastructure backing your Domino deployment. [Distributed GPUs with Open MPI](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi) has complete instructions on enabling and configuring Open MPI on your environment. |

## Domino GPU environments

The Domino GPU Environment includes CUDA support and common packages for taking advantage of GPUs. Domino recommends using an Environment with explicit GPU support when using GPU hardware tiers.

The [Compute Environment catalog](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/8-compute-environment-catalog) has a complete list of pre-built compute environments for you to use with Domino.

## Next steps

* [Create a new Domino Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/3-create-an-environment) to meet your language and package needs.

* [Set default Environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/6-set-a-default-environment) establishes a starting point for all new projects or an environment for a specific project.

* The [Compute Environment catalog](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/8-compute-environment-catalog) has information on accessing Domino Environment images.

* [Our Best Practices](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/1-best-practices-domino-environments) guide you in setting up and maintaining your Domino environments.


# Create a Domino Environment
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/3-create-an-environment

Create or duplicate a Domino Environment from a core, existing, or custom image, control revisions and visibility, and declare CPU architectures.

This guide shows you three ways to create a new Domino Environment, whether you want to make a core Domino environment, base it on an existing environment, or start from a custom setup.

## Tour the new environment window

<img alt="Create a new Environment" />

1. **Name**: The name you give your environment.

2. **Description (Optional)**: Use this to record details about your environment.

3. **Base Environment / Image**: These are the options for a base environment/image:

   1. **Start from an existing Environment**: Base your new environment on an existing Domino environment.

   2. **Active (auto-rebuilds)**: For an existing environment, keep this selected, or choose a specific revision from the menu.

   3. **Start from a custom base image**: Input a URI to build a custom environment.

4. **Supported Clusters**: Lists the currently supported types of clusters.

5. **Visibility**: Choose a visibility level: Private, Available to an Organization, or Globally Accessible (admin only).

6. **Revision Behavior**: Auto-assigns the latest revision as active.

7. **Customize before building**: Use this to add custom features to your environment.

8. **Create Environment**: Validates and creates your new environment.

## Create environment based on Domino core environment

Domino comes with the [Domino Core Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) installed, but you can create a new Domino Environment using the available environment images.

1. Go to **Govern** > **Environments** and click **+ Create Environment**.

2. Fill in the following fields:

   1. **Name**: Name your environment according to the naming convention of your organization.

   2. **Description (Optional)**: Enter a meaningful description of the environment. For example, you could add details like the OS, Python version, and whether the environment uses Jupyter, JupyterLab, VS Code, etc.

3. Choose **Start from an existing Environment** for your **Base Environment / Image**.

   1. The global default for your environment is pre-selected.

   2. Scroll until you can select the **Domino Core Environment**.

   3. **Active (auto-rebuilds)**: keep this selected, or choose a specific revision from the menu.

4. Choose an option for any **Supported Clusters**.

5. Decide on your environment’s **Visibility** level: **Private**, **Available to an Organization**, or **Globally Accessible** (admin only).

6. Click **Create Environment**. The **Revisions** tab for the environment opens.

To customize the environment to enable tools, run scripts, or manage environment variables, click **Customize before Building** and use the [Customize your Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition) guide.

You can also click **Edit Definition** to customize the environment later.

## Create environment based on existing environment

You can create a new Domino Environment by duplicating existing environments, or by choosing a base environment from the **Start from existing environment** menu.

After building or editing an environment, use the **Auto-assign next new revision as Active** toggle in the **Revisions** tab to control which revision is active. When enabled, the latest successful build becomes **Active** automatically. When disabled, the environment stays on the current **Active** revision, even as new ones are built.

<img alt="Auto-assign next new revision as Active" />

Suppose your environment updates automatically due to a subscription, but you have locked its **Active** revision to a specific version. In that case, you see a notice when the base environment’s **Active** revision changes. This reminds you to consider updating your environment’s revision.

You can keep a chosen version of your environment as **Active** while still receiving updates on the base environment. This lets you test the new version before connecting your Jobs, Apps, and Model Endpoints to it.

Please note that archived images are not rebuilt automatically.

### Create environment by duplication

To create a new environment by duplicating an existing environment:

1. Go to **Govern** > **Environments**.

2. Choose the environment you want to copy and click the three dots on the right.

3. Choose **Duplicate Environment** from the menu.

4. Rename it by clicking the edit icon and then entering when done.

5. You can edit other fields quickly to customize them:

   1. **Edit Description**: Click **Update** when you are done with the description.

   2. **Edit Visibility**: Click **Update Visibility** when finished.

6. Click **Customize before building** or **Create Environment**.

Use the [Customize your Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition) article for guidance to enable tools, run scripts, or manage environment variables.

### Create environment from existing environment

When you create a new environment based on an existing one or change the settings of an environment, you have two options for building it:

* You can create it based on a specific version of the base environment. You can change this on a later revision of your own.

* You can set it to update automatically by linking it to changes in the base environment’s **Active** revision. If the base environment changes its **Active** revision, your environment automatically builds a new revision based off of it.

To create a new environment by using an existing environment as the base:

1. Go to **Govern** > **Environments**.

2. Select **Create Environment**.

3. Choose the environment you want to use as the base from the menu.

   1. **Active (auto-rebuilds)**: keep this selected, or choose a specific revision from the menu.

4. Name your new environment.

5. **Description (Optional)**: Enter a description of the environment.

6. Choose a **Visibility** level for your environment.

7. Click **Create Environment**. The **Revisions** tab for the environment opens.

Use the [Customize your Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition) article for guidance to enable tools, run scripts, or manage environment variables.

## Create environment based on custom images

Choose your [custom Docker or container image](/cloud/platform-overview/domino-overview) before creating a new environment and copy the URI.

1. Go to **Govern** > **Environments** and click **+Create Environment**.

2. Fill in the following fields:

   1. **Name**: Name your environment according to the naming convention of your organization.

   2. **Description (Optional)**: Enter a meaningful description of the environment. For example, you could add details like the OS, Python version, and whether the environment uses Jupyter, JupyterLab, VS Code, etc.

3. Choose **Start from a custom base image** for your **Base Environment / Image**.

   1. **FROM**: Paste the custom Docker or container image URI here.

   2. **Optional**: Review the information about the **Automatically make compatible** field.

4. Choose an option for any **Supported Clusters**.

5. Decide on your environment’s **Visibility** level: **Private**, **Available to an Organization**, or **Globally Accessible** (admin only).

6. Click **Create Environment**. The **Revisions** tab for the environment opens.

To customize the environment to enable tools, run scripts, or manage environment variables, click **Customize before Building** and use the [Customize your Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition) guide.

You can also click **Edit Definition** to customize the environment later.

## Set supported CPU architectures

Each environment revision declares the CPU architectures it supports: `amd64`, `arm64`, or both. When an execution starts, Domino checks that the environment supports the CPU architecture of the selected [hardware tier](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers) and rejects the execution if it doesn't. For on-demand clusters, Domino runs the same check against the cluster's head and worker hardware tiers.

If a revision doesn't declare architectures, it inherits the architectures declared on its base environment. If neither declares architectures, the revision supports `amd64`, so existing environments keep running on their current hardware tiers.

Declare architectures with the v2 environments API:

* `POST /api/environments/v2/environments` creates an environment. Set the `architectures` array on the `initialRevision` object.
* `POST /api/environments/v2/environments/{environmentId}/revisions` creates a revision of an existing environment. Set the `architectures` array at the top level of the request body.

Domino returns a `400` response when the `architectures` array contains a value other than `amd64` or `arm64`. In v2 request bodies, the `dockerfileInstructions`, `preSetupScript`, `postSetupScript`, `preRunScript`, and `postRunScript` fields are nested under a `buildConfiguration` object. The beta environments endpoints are deprecated and don't support architectures.

Example request:

```python theme={null}
import requests
import json

url = "https://your-domino-url/api/environments/v2/environments"

payload = json.dumps({
  "name": "multi-architecture-environment",
  "visibility": "private",
  "addBaseDependencies": True,
  "initialRevision": {
    "image": "quay.io/example/base-image:latest",
    "architectures": ["amd64", "arm64"]
  }
})
headers = {
  'X-Domino-Api-Key': 'YOUR_API_KEY',
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)
```

To read the architectures a revision supports, get the environment with `GET /api/environments/v1/environments/{environmentId}` and check the `architectures` array on each revision.

## Archive an environment

You can archive environments as long as they are not the default environment for a project. Archiving an environment does not affect reproducibility; executions based on it still run.

1. Go to **Govern** > **Environments**.

2. Choose an environment from the list.

3. There are two ways to archive an environment quickly:

   1. You can click the 3 dots within the list and choose **Archive Environment**.

   2. Choose **Archive Environment** from the menu. You can also archive from any tab within the **Environment** details.

## Next steps

* [Customize your Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition) to enable tools, run scripts, or manage environment variables.

* [Set default Environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/6-set-a-default-environment) establishes a starting point for all new projects or an environment for a specific project.

* [Our Best Practices](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/1-best-practices-domino-environments) guides you in setting up and maintaining your Domino environments.

* [Use AI coding assistants](/cloud/platform-capabilities/features/coding-assistants) to enhance your coding in a Domino Environment.


# Use Environment Revision Controls in Domino
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/4-use-environment-revision-controls

Control environment inheritance with revision selection, Active revision pinning, automatic activation, and chained build propagation.

Domino enables you to manage environment inheritance and revision lifecycles with features such as build propagation, revision pinning, and revision selection. These tools help ensure the automatic propagation of environment updates, allowing for safe testing of changes, and giving you control over which revisions are used across your workflows.

This approach improves environment lifecycle management in two key ways:

* **Better governance and compliance**, since environment updates can be cascaded through derived environments.

* **More reliable automation**, with the ability to test updated environments before production workloads adopt them.

Together, these capabilities make it easier to scale, standardize, and maintain reliable environments across teams and projects.

## Prerequisites

* You must have permission to create or edit compute environments.

* You should be familiar with how base and derived environments work in Domino.

## Step 1: Select a base revision (revision selection)

When creating or editing an environment:

1. Go to **Environments** > **Create Environment** or open an existing one.

2. Under **Base Environment**, select a base environment.

3. In the **Revision** dropdown, you can:

   1. Choose a specific revision of the base, or

   2. Select **Active - auto-rebuild** to subscribe to the base’s current Active revision.

      Choosing **Active - auto-rebuild** enables build propagation. Any change to the Active revision of the base environment triggers an automatic rebuild of this environment.

## Step 2: Manage automatic activation of revisions (revision pinning)

You can control whether the latest revision becomes Active automatically or if activation must be manual.

**When creating a new environment**\
In the setup form, check or uncheck **Auto-assign latest revision as Active**:

* **Checked:** New revisions are automatically set to Active.

* **Unchecked:** You must manually activate revisions (that is, the Active revision is pinned).

**For existing environments**\
If using build propagation, uncheck this option to avoid automatically activating new, untested revisions:

1. Go to the **Revisions** tab.

2. Uncheck **Auto-assign latest revision as Active** to pin the current Active revision.

3. Manually promote future revisions when ready.

## Step 3: Use build propagation

If you selected **Active - auto-rebuild** in Step 1:

* Any time the base environment’s Active revision changes, a new revision of your environment is built automatically.

* If the environment has **Auto-assign latest revision as Active** toggled **on**, this new revision becomes Active.

* If the environment has **Auto-assign latest revision as Active** toggled **off** (that is, the Active revision is pinned), the new revision must be activated manually.

If the environment has **Auto-assign latest revision as Active** toggled **on**, the new auto-rebuilt revision is automatically set to Active and consequently propagates builds to any of its subscribed environments. This way you can achieve chained propagation across multiple levels of environment inheritance.

## Next steps

* [Customize an Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition) to enable tools, run scripts, or manage environment variables.

* [Set a default Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/6-set-a-default-environment) to address particular package or IDE requirements.

* Use [AI coding assistants](/cloud/platform-capabilities/features/coding-assistants) in a Domino Environment.

* [Manage Environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables) to help configure how the system runs and manage secrets or settings across different areas.


# Customize your Environment
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition

Customize an Environment with Dockerfile instructions, Workspace tools, setup and run scripts, variables, VPN networking, and revision notes.

You can customize an environment to enable tools, run scripts, or manage environment variables. You may need to be an administrator to edit an existing environment.

* Click **Customize before Building** if you just created an Environment and want to enable tools, run scripts, or manage environment variables.

* To edit an existing Environment, go to **Govern** > **Environments** in the navigation pane and click the name of an Environment. Then click **Edit Definition**.

## Order of script execution

Domino runs setup scripts before and after running the `requirements.txt` file. The scripts within an environment are executed in the following order:

* First, it executes **pre-setup scripts**.

* Next, it installs packages from the `requirements.txt` file.

* Then Domino runs **post-setup** and **pre-run scripts**.

* **Post-run scripts** are executed after your Workspace, Job, or App is stopped.

However, no scripts are run before or after tasks in a distributed compute cluster environment, such as Spark, Ray, Dask, or MPI.

## Define your environment

Before you add items to these scripts and the `requirements.txt` file, think about whether similar instructions should be included directly in the environment’s Dockerfile.

This can make it faster to start workspaces, jobs, and other tasks using the environment because the necessary work is already included in the Docker image.

<img alt="Domino Environment Overview Tab" />

You can start by going to **Govern** > **Environments** and selecting the environment you want to define from the list. Then click **Edit Definition**.

On the **Overview** tab, you can update the **Description** or change the environment’s Visibility. You won’t be able to change the **Environment Base** or **Supported Cluster Settings**.

### Dockerfile instructions

Enter your Dockerfile layers. Some things to remember:

* Exclude the `FROM` statement.

* Commands can be run as root or other user available in the base image.

* Docker can’t access your project files or user environment variables when building.

* There is a limit of 127 layers.

[Docker’s official site](https://docs.docker.com/engine/reference/builder/) has more detailed information about Docker.

### Pluggable Workspace tools

Enter the interactive tools needed by any Project that uses this Environment. [Add Workspace IDEs](/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/add-workspace-ides) has more information about interactive tools.

### Run setup scripts

Enter the `bash` code that is executed at runtime. Domino executes these commands at runtime and runs them as root.

* **Pre Run Script**: Enter commands to execute before the Python packages in your Project’s `requirements.txt` are installed.

* **Post Run Script**: Enter commands to execute after your Workspace/Job/App is stopped.

### Environment variables

Set the **Variable Names** and their **Values**.

### Advanced

In the advanced section, enter the `bash` code executed at a specified step in your experiment’s lifecycle.

* **Pre Run Script**: Enter commands to execute before installing Python packages in your Project’s `requirements.txt`.

* **Post Run Script**: Enter commands to execute after the Python packages in your Project’s `requirements.txt` are installed.

* **Enable VPN Networking**: Select this if a VPN is to be used for that environment.

### Revision history

Enter a short description of the changes to the Environment.

## Next steps

* [Our Best Practices](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/1-best-practices-domino-environments) guides you in setting up and maintaining your Domino environments.

* [Use Domino’s environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables) to store keys and credentials or to inject values into your runs.

* [Create a Domino Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/3-create-an-environment) using an existing environment or start from a custom setup.

* [Set default Environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/6-set-a-default-environment) establishes a starting point for all new projects or an environment for a specific project.


# Set a default Environment
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/6-set-a-default-environment

Set a Project's default Compute Environment so new Jobs and Workspaces start with the required packages, tools, and IDE configuration.

You can set a project-specific environment as the default for your project. This might address particular package or IDE requirements. The project owner can choose the default environment for that project.

When you launch a job or create a new workspace, Domino automatically populates the fields with the compute environment defaults from the project’s settings.

## Set default environment for a project

Adjust the project’s Compute Environment settings to replace the global default with one tailored to your project’s specific needs:

1. Go to **Develop** > **Projects**.

2. Choose the project that you want to set a default environment for.

3. In the navigation pane, click **Settings** > **Hardware & Environment**.

4. From **Compute Environment**, select the environment you want to use as the default for your project.

5. Click **Manage Environment** to see your environment’s details.

## Next steps

* [Our Best Practices](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/1-best-practices-domino-environments) guides you in setting up and maintaining your Domino environments.

* [Create a Domino Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/3-create-an-environment) using an existing environment or start from a custom setup.

* [Edit an Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition) to enable tools, run scripts, or manage environment variables.

* [Use AI coding assistants](/cloud/platform-capabilities/features/coding-assistants) to enhance your coding in a Domino Environment.


# Compute Environment Catalog
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/8-compute-environment-catalog

Find registry paths for Domino Core, Standard, Minimal, GPU, and cluster environments, plus partner images for MATLAB, NVIDIA, Intel, and Stata.

Domino is an open platform for data science which integrates various languages, IDEs, data sources, and tools in one place.

You can add various environments to any Domino installation by visiting quay.io. Download the image from the URL link to the repository. Our documentation provides instructions for setting up these environments.

You can also use Domino to automatically adjust Docker images so they work with Domino Workspaces and Jobs. You can find more information, including the requirements for these images, in [Use External Images in Domino Environments](/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images).

<table>
  <thead>
    <tr>
      <th>Partner</th>
      <th>Product/Version</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>
        <img alt="Domino logo" />
      </td>

      <td><div><a href="/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments#domino-standard-environment-dse">Domino Standard Environment (DSE)</a></div><div><code>quay.io/domino/domino-standard-environment:develop.1197facc</code></div></td>
    </tr>

    <tr>
      <td>
        <img alt="Domino logo" />
      </td>

      <td><div><a href="/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments#domino-core-environment-dce">Domino Core Environment (DCE)</a></div><div><code>quay.io/domino/domino-core-environment:develop.1197facc</code></div></td>
    </tr>

    <tr>
      <td>
        <img alt="Domino logo" />
      </td>

      <td><div><a href="/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments">Domino Minimal Environment (DME)</a></div><div><code>quay.io/domino/domino-minimal-environment:develop.1197facc</code></div></td>
    </tr>

    <tr>
      <td>
        <img alt="Domino logo" />
      </td>

      <td><div><a href="/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments#domino-gpu-environments">Domino GPU Environment (DGE)</a></div><div><code>quay.io/domino/domino-gpu-environment:develop.1197facc</code></div></td>
    </tr>

    <tr>
      <td>
        <img alt="Domino logo" />
      </td>

      <td><div><a href="/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments#domino-cluster-environments">Domino Spark Environment</a></div><div><code>quay.io/domino/domino-spark-environment:develop.1197facc</code></div><div><code>quay.io/domino/spark-cluster-environment:develop.1197facc</code></div></td>
    </tr>

    <tr>
      <td>
        <img alt="Domino logo" />
      </td>

      <td><div><a href="/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments#domino-cluster-environments">Domino Dask Environment</a></div><div><code>quay.io/domino/domino-dask-environment:develop.1197facc</code></div><div><code>quay.io/domino/dask-cluster-environment:develop.1197facc</code></div></td>
    </tr>

    <tr>
      <td>
        <img alt="Domino logo" />
      </td>

      <td><div><a href="/cloud/platform-capabilities/features/compute-clusters-hpc/mpi">On-Demand Open MPI</a></div><div><code>quay.io/domino/mpi-cluster-environment:develop.1197facc</code></div><div><code>quay.io/domino/domino-mpi-environment:develop.1197facc</code></div></td>
    </tr>

    <tr>
      <td>
        <img alt="Domino logo" />
      </td>

      <td><div><a href="/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments#domino-standard-environment-dse">Domino Standard Environment without Coding Assistants</a></div><div><code>quay.io/domino/domino-standard-environment-no-assistants:develop.1197facc</code></div></td>
    </tr>

    <tr>
      <td>
        <img alt="Intel AI logo" />
      </td>

      <td><div><a href="/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/intel-ai-kit">Intel AI Analytics Toolkit (AI Kit)</a></div><div><code>intel/oneapi-aikit</code></div></td>
    </tr>

    <tr>
      <td>
        <img alt="MATLAB logo" />
      </td>

      <td><div><a href="/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/matlab">MATLAB 2021a</a></div><div><code>quay.io/domino/matlab:R2021a</code></div></td>
    </tr>

    <tr>
      <td>
        <img alt="MATLAB logo" />
      </td>

      <td><div><a href="/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/matlab">MATLAB 2020b</a></div><div><code>quay.io/domino/matlab:R2020b</code></div></td>
    </tr>

    <tr>
      <td>
        <img alt="MATLAB logo" />
      </td>

      <td><div><a href="/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/matlab">MATLAB 2020a</a></div><div><code>quay.io/domino/matlab:R2020a</code></div></td>
    </tr>

    <tr>
      <td>
        <img alt="MATLAB logo" />
      </td>

      <td><div><a href="/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/matlab">MATLAB 2019b</a></div><div><code>quay.io/domino/matlab:R2019b</code></div></td>
    </tr>

    <tr>
      <td>
        <img alt="NVIDIA NGC logo" />
      </td>

      <td><div><a href="/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/nvidia-ngc">Container images from the NVIDIA NGC Catalog</a></div></td>
    </tr>

    <tr>
      <td>
        <img alt="Domino logo" />
      </td>

      <td><div><a href="/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments#domino-cluster-environments">Domino Ray Environment</a></div><div><code>quay.io/domino/domino-ray-environment:develop.1197facc</code></div><div><code>quay.io/domino/ray-cluster-environment:develop.1197facc</code></div><div><code>quay.io/domino/ray-cluster-gpu-environment:develop.1197facc</code></div></td>
    </tr>

    <tr>
      <td>
        <img alt="Snowflake logo" />
      </td>

      <td><div><a href="/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-snowflake">Snowflake Snowpark</a></div><div><code>quay.io/domino/snowflake:snowpark-latest</code></div></td>
    </tr>

    <tr>
      <td>
        <img alt="Stata logo" />
      </td>

      <td><div><a href="/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/stata">Stata 17</a></div><div><code>quay.io/domino/stata:17</code></div></td>
    </tr>
  </tbody>
</table>


# Manage Compute Environments
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/index

Manage Docker-based Compute Environments, inspect revisions and dependency trees, control access, and choose images for each execution.

Domino Environments, or Compute Environments, help teams manage their projects better. They improve processes in MLOps, generative AI, and data science workflows.

In Domino, environments are defined by Docker images that create isolated containers for each execution unit including Workspaces, Runs, Apps, and Models. This setup allows professionals to experiment and customize their work in a contained, versioned, and shareable space, boosting reproducibility and collaboration.

## Basics of Domino environments

You can create environments, use those provided by Domino, or modify existing ones. You might choose to create or alter an environment if:

* You must install a package for Python, R, Octave, or some other software dependency.

* You want to cache a package or library so that it’s immediately available when you start a run.

* You manage an organization and want to create a default Environment for all projects.

Domino has two main types of Compute Environments: the Domino Standard Environment (DSE) and the Domino Minimal Environment (DME). The DSE offers a complete set of libraries and packages, while the DME is lighter and includes fewer packages.

Domino also provides Cluster Environments, which work like DSEs but are designed for specific types of clusters. The environments are smaller and simpler, suitable only for worker nodes in a cluster, while DSEs cannot be used for worker nodes.

## View accessible environments

You can view the environments that you can access by clicking **Govern** > **Environments** in the navigation bar.

In the **Environments Overview** page, you can see a list of:

* Global Environments.

* Specific environments used by Projects you [collaborate](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects) on.

* Environments shared with [organizations](/cloud/platform-capabilities/features/collaboration/manage-organizations) you belong to.

To switch from the default *table* view to the *tree* view, use the selector on the top right, below the **+ Create Environment** button. The tree view allows you to see hierarchical relationships between environments, collapse and expand nodes with children, view environment metadata, and filter by visibility.

<img alt="Tree view of Environments" />

To see more details about an environment, click that environment’s name.

<img alt="Environment Overview Pane" />

<table>
  <thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td><p><strong>Overview</strong></p></td>
      <td><p>The Overview tab lists all metadata about your environment or image. If your environment is built on top of another, you may need to click through to the parent environment before seeing the underlying Docker image. Click <a href="/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition">Edit Definition</a> if you need to make changes to your environment.</p></td>
    </tr>

    <tr>
      <td><p><strong>Revisions</strong></p></td>
      <td><p>All revisions of your compute environment are listed here, along with their build status, timestamp, and Docker image URI. Each saved change increments the revision number by one. By default, the latest successfully built revision is automatically set as <strong>Active</strong>. If you’d like to keep a specific revision active instead, you can disable the <strong>Auto-assign next new revision as Active</strong> toggle to lock your chosen revision in place. Domino rebuilds the environment and pushes the resulting image to the local Docker registry. By default, the latest successfully built revision becomes the Active revision.</p> <p>The list also indicates whether a revision was built automatically via build propagation, and includes metadata about its parent revision: specifically, which revision it was derived from and whether that parent is currently Active. This context helps track inheritance chains and understand how updates cascade across environments.</p></td>
    </tr>

    <tr>
      <td><p><strong>Revision details</strong></p></td>
      <td><p>Click the three vertical dots for additional options like viewing build logs, downloading an image source project, or canceling builds.</p></td>
    </tr>

    <tr>
      <td><p><strong>Projects</strong></p></td>
      <td><p>These are listed after being assigned to an environment. You can view details of the project by clicking its name.</p></td>
    </tr>

    <tr>
      <td><p><strong>Endpoints</strong></p></td>
      <td><p>Shows you a list of endpoints for that environment, including the status of the endpoint.</p></td>
    </tr>

    <tr>
      <td><p><strong>Dependency Tree</strong></p></td>
      <td><p>The Dependency Tree view is similar to the global tree view, except that it only displays the subtree of the selected environment, with it as its only root node. By default, only the descendants that would receive automatic downstream rebuilds are shown. The <strong>Include non-auto-rebuilding</strong> toggle allows you to see all descendants.</p></td>
    </tr>
  </tbody>
</table>

## Next steps

* [Our Best Practices](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/1-best-practices-domino-environments) guides you in setting up and maintaining your Domino environments.

* [Domino Standard Environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) describes the pre-configured environments that come with Domino.

* [Create a Domino Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/3-create-an-environment) using an existing environment or start from a custom setup.

* [Use Environment Revision Controls in Domino](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/4-use-environment-revision-controls) to manage environment inheritance and revision lifecycles.

* [Customize your Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition) to enable tools, run scripts, or manage environment variables.

* [Set default Environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/6-set-a-default-environment) establishes a starting point for all new projects or an environment for a specific project.

* [Compute Environment Catalog](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/8-compute-environment-catalog) provides images to add various environments to any Domino installation.

* [Manage environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables) to configure how the system runs and manage secrets or settings across different areas.


# Connect to DataRobot
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/access-external-data-directly/connect-to-datarobot

Connect directly to DataRobot by installing a versioned Python client in a Domino Environment and confirming network connectivity.

This topic describes how to connect directly to DataRobot from Domino. You must have network connectivity between DataRobot and your Domino deployment.

Use the following commands to install the DataRobot Python client in the Dockerfile instructions section of your environment:

```dockerfile theme={null}
RUN pip install "datarobot==<version>"
```

## Next steps

* [Access external data directly](/cloud/platform-capabilities/core-concepts/data/access-external-data-directly): the general procedure, including where to store credentials.

* Compare direct connections with [Domino Data Source connectors](/cloud/platform-capabilities/core-concepts/data/data-source-connectors) for supported data stores.


# Connect to Impala
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/access-external-data-directly/connect-to-impala

Connect directly to Apache Impala with Cloudera's ODBC driver and pyodbc, store Kerberos settings as variables, and configure SSL when needed.

This topic describes how to connect directly to <a href="http://impala.apache.org/">Apache Impala</a> from Domino. You must have network connectivity between Impala and your Domino deployment.

<Warning>
  Domino does not officially support this method. We provide this information as a courtesy.
</Warning>

## Use Impala ODBC connector for Cloudera Enterprise with pyodbc

Domino recommends using the <a href="https://www.cloudera.com/downloads/connectors/impala/odbc/2-6-0.html">Impala ODBC Connector for Cloudera Enterprise</a> in concert with the <a href="https://pypi.org/project/pyodbc/">pyodbc</a> library for interacting with Impala from Python.

### Environment setup

1. Visit the Cloudera downloads page to download the Impala ODBC Connector for Cloudera Enterprise to your local machine. For default Domino images of Ubuntu 16.04, download the 64-bit Debian package. Keep track of where you save this file, as you need it in a later step.

   <img alt="Cloudera ODBC Connector" />

2. Create a new *public* project in your Domino instance to host the driver files for use in Domino environments.

   <img alt="Create Project modal with Visibility set to Public" />

3. In the new project, click **browse for files** and select the driver file you downloaded earlier to queue it for upload. Click **Upload** to add it to the project.

   <img alt="Upload the driver" />

4. After the driver file has been added to your project files, click the gear next to it in the files list, then right-click **Download** and click **Copy link address**. Save this address as you need it when setting up your environment.

   <img alt="Copy the link address" />

5. Add the following Dockerfile instructions to install the driver and pyodbc in your environment, pasting the URL you copied earlier where indicated on line 5.

   ```dockerfile theme={null}
   USER root

   # download the driver from your project
   RUN mkdir /ref_files
   RUN
   cd /ref_files &&
   wget --no-check-certificate [paste-download-url-from-previous-step-here] &&
   gzip -d clouderaimpalaodbc_2.6.0.1000-2_amd64.deb.gz

   # install the driver
   RUN gdebi /ref_files/clouderaimpalaodbc_2.6.0.1000-2_amd64.deb --n

   # update odbc.ini file for impala driver
   RUN
   echo "

   [Cloudera ODBC Driver for Impala]

   Driver=/opt/cloudera/impalaodbc/lib/64/libclouderaimpalaodbc64.so

   KrbFQDN=_HOST

   KrbServiceName=impala
   " >> /etc/odbcinst.ini

   # set up impala libraries
   RUN export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/cloudera/impalaodbc/lib/64
   RUN ldd /opt/cloudera/impalaodbc/lib/64/libclouderaimpalaodbc64.so

   # install pyodbc
   RUN pip install pyodbc

   USER ubuntu
   ```

### Credential setup

Set up the following Domino environment variables to store secure information about your Impala connection.

* `IMPALA_HOST`

Hostname where your Impala service is running. Make sure your Impala service and network firewall are configured to accept connections from Domino.

* `IMPALA_PORT`

The port your Impala service is configured to accept connections on.

* `IMPALA_KERB_HOST`

Hostname of your Kerberos authentication service.

* `IMPALA_KERB_REALM`

The name of the Kerberos realm used by the Impala service.

See [Store Project credentials](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables) to learn more about Domino environment variables.

### Usage

See the <a href="https://github.com/mkleehammer/pyodbc/wiki">pyodbc documentation</a> for detailed information about how to use the package to interact with a database. The following are some examples for how to set up a connection.

```python theme={null}
import pyodbc
import os

# fetch values from environment variables
hostname = os.environ['IMPALA_HOST']
service_port = os.environ['IMPALA_PORT']
kerb_host = os.environ['IMPALA_KERB_HOST']
kerb_realm = os.environ['IMPALA_KERB_REALM']

# create connection object
conn = pyodbc.connect('Host=hostname;'
                     +'DRIVER={Cloudera ODBC Driver for Impala};'
                     +'PORT=service_port;'
                     +'KrbRealm=kerb_realm;'
                     +'KrbFQDN=kerb_host;'
                     +'KrbServiceName=impala;'
                     +'AUTHMECH=1',autocommit=True)

# if you see:
# 'Error! Filename not specified'
# while querying Impala using the connection object,
# add the following configuration line:
#
# conn.setencoding(encoding='utf-8', ctype=pyodbc.SQL_CHAR)

# if your Impala uses SSL, add SSL=1 to the connection string
# conn = pyodbc.connect('Host=hostname;'
#                      +'DRIVER={Cloudera ODBC Driver for Impala};'
#                      +'PORT=service_port;'
#                      +'KrbRealm=kerb_realm;'
#                      +'KrbFQDN=kerb_host;'
#                      +'KrbServiceName=impala;'
#                      +'AUTHMECH=1;'
#                      +'SSL=1;'
#                      +'AllowSelfSignedServerCert=1', autocommit=True)
```

## Next steps

* [Access external data directly](/cloud/platform-capabilities/core-concepts/data/access-external-data-directly): the general procedure, including where to store credentials.

* Compare direct connections with [Domino Data Source connectors](/cloud/platform-capabilities/core-concepts/data/data-source-connectors) for supported data stores.


# Access external data directly
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/access-external-data-directly/index

Connect Domino code directly to APIs, databases, and network file stores by installing clients, storing credentials, and fetching data.

The best way to connect to an external data store from Domino is to use a Domino connector to [create a Domino data source](/cloud/platform-capabilities/core-concepts/data/data-source-connectors). However, you also have the option to connect to an external data store directly from a Domino application. Because Domino can run any user-produced code, you can always execute code that connects to any data source you wish. As long as the Domino executor machines have network connectivity to your data, Domino runs the code.

<Frame>
  <img alt="Direct access to external data." />
</Frame>

Common use cases for directly accessing external data include pulling data from APIs, accessing a database without going through a Domino connector, or accessing a network-based file store without going through an External Data Volume (EDV).

The general procedure is as follows:

1. Install the package for the data store in your environment or in your workspace session.

   See [Add packages to environments](/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments).

2. Set up the credentials for the data store.

   Be sure to store your credentials securely and use environment variables to access them in your code. See [Store Project credentials](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables).

3. Fetch data from the data store.

   The method for fetching data depends on the data store.

The following data sources are not supported by a Domino Data Source connector, but you can still connect directly to these data sources. Domino does not provide support for any issues related to direct connections.

* [DataRobot](/cloud/platform-capabilities/core-concepts/data/access-external-data-directly/connect-to-datarobot)

* [Impala](/cloud/platform-capabilities/core-concepts/data/access-external-data-directly/connect-to-impala)


# Connect to Amazon S3
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-amazon-s3

Create and authenticate an Amazon S3 Data Source, or connect with AWS credentials through public URLs, the AWS CLI, boto3, or R's aws.s3 package.

This topic describes how to connect to [Amazon Simple Storage Service (S3)](https://aws.amazon.com/s3/) from Domino. You must have network connectivity between S3 and your Domino deployment.

The easiest way to connect to an Amazon S3 instance from Domino is to create a Domino Data Source as described below.

<Note>
  Administrators can create a Data Source for [tabular S3 with AWS Glue](https://docs.aws.amazon.com/athena/latest/ug/data-sources-glue.html). See the [Admin Guide](/cloud/admin/data-administration/external-data#create-a-starburst-powered-data-source) for details.
</Note>

## Create an Amazon S3 Data Source

1. From the navigation pane, click **Data > Data Sources**.

2. Click **Create a Data Source**.

3. In the New Data Source window, from **Select Data Store**, select **Amazon S3**.

4. Enter the **Bucket**.

5. Optional: Enter a **Subfolder Path**, such as `myFolder` or `data/tests`, to restrict the Data Source to only list data from and write to the specified folder.

6. Enter the **Region**.

7. Enter the **Data Source Name**.

8. Optional: Enter a **Description** to explain the purpose of the Data Source to others.

9. Click **Next**.

10. Enter the credentials to authenticate to S3.

    By default, Domino supports basic authentication; the Domino secret store backed by HashiCorp Vault securely stores the credentials. If your administrator enabled it, [IAM credential propagation](/cloud/admin/data-administration/external-data#iam-authentication-for-aws-data-sources) might be available.

    <Note>
      IAM-authenticated connections can be used only to execute jobs and workspaces. Other execution types, such as scheduled jobs and Domino endpoints, require basic authentication.
    </Note>

11. Click **Test Credentials**.

12. If the Data Source authenticates, click **Next**.

13. Select who can view and use the Data Source in projects.

14. Click **Finish Setup**.

## Alternate way to connect to an Amazon S3 Data Source

<Warning>
  This section describes an alternate method to connect to . Domino does not officially support this method.
</Warning>

Use one of the following methods to authenticate with S3 from Domino.

Both follow the common naming convention of environment variables for AWS packages so you don’t have to explicitly reference credentials in your code.

### Set up credentials

1. Use a short-lived credential file obtained through Domino’s AWS Credential Propagation feature.

   After your administrator configures this feature, Domino automatically populates any Run or Job with your AWS credentials file. These credentials are periodically refreshed throughout the Workspace duration to ensure they don’t expire.

   Following common AWS conventions, you see an environment variable `AWS_SHARED_CREDENTIALS_FILE` which contains the location of your credential files which are stored at `/var/lib/domino/home/.aws/credentials`.

   Learn more about [using a credential file with the AWS SDK](https://aws.amazon.com/blogs/security/a-new-and-standardized-way-to-manage-credentials-in-the-aws-sdks/).

2. Store your AWS access keys securely as environment variables.

   To connect to the S3 buckets to which your AWS account has access, enter your [AWS Access Key and AWS Secret Key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) to the AWS CLI. By default, AWS utilities [look for these in your environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-environment.html).

   Set the following as Domino environment variables on your user account:

   * `AWS_ACCESS_KEY_ID`

   * `AWS_SECRET_ACCESS_KEY`

   See [Store Project credentials](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables) to learn more about Domino environment variables.

### Get a file from an S3-hosted public path

1. If you have files in S3 that are set to allow public read access, use [Wget](https://en.wikipedia.org/wiki/Wget) from the OS shell of a Domino executor to fetch the files. The request for those files looks similar to the following:

   ```bash theme={null}
   wget https://s3-<region>.amazonaws.com/<bucket-name>/<filename>
   ```

   This method is simple but doesn’t require authentication or authorization. Do not use this method with sensitive data.

### AWS CLI

1. Use the [AWS CLI](https://docs.aws.amazon.com/cli/latest/reference/index.html#cli-aws) for a secure method to read S3 from the OS shell of a Domino executor. Making the AWS CLI work from your executor, install it in your environment and enter your credentials.

2. Get the AWS CLI as a Python package from [pip](https://pypi.org/project/pip/).

3. Use the following Dockerfile instruction to install the CLI and automatically add it to your system PATH. You must have [pip](https://pypi.org/project/pip/) installed.

   ```dockerfile theme={null}
   USER root
   RUN pip install awscli --upgrade
   USER ubuntu
   ```

4. After your Domino environment and credentials are set up, fetch the contents of an S3 bucket to your current directory by running:

   ```bash theme={null}
   aws s3 sync s3://<bucket-name> .
   ```

5. If you are using an AWS credential file with multiple profiles, you might need to specify the profile. (The `default` profile is used if none is specified.)

   ```bash theme={null}
   aws s3 sync s3://<bucket-name> . --profile <profile name>
   ```

   See the [official AWS CLI documentation](https://docs.aws.amazon.com/cli/latest/reference/s3/index.html) on S3 for more commands and options.

### Python and boto3

1. To interact with AWS services from Python, Domino recommends [boto3](https://aws.amazon.com/sdk-for-python/).

2. If you’re using a [Domino Standard Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments), boto3 is already installed. If you want to add boto3 to an environment, use the following Dockerfile instructions.

   This instruction assumes you already have [pip](https://pypi.org/project/pip/) installed.

   ```dockerfile theme={null}
   USER root
   RUN pip install boto3
   USER ubuntu
   ```

3. To interact with S3 from boto3, see [the official boto3 documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html). The following is an example of downloading a file where:

   * You have set up your credentials as instructed above

   * Your account has access to an S3 bucket named `my_bucket`

   * The bucket contains an object named `some_data.csv`

     ```python theme={null}
     import boto3
     import io
     import pandas as pd

     # Create a new S3 client
     client = boto3.client('s3')

     # Download some_data.csv from my_bucket and write to ./some_data.csv locally
     file = client.download_file('my_bucket', 'some_data.csv', './some_data.csv')
     ```

     Alternatively, for users using a credential file, use the following code:

     ```python theme={null}
     import boto3

     # Specify your profile if your credential file contains multiple profiles
     session = boto3.Session(profile_name='<profile name>')

     # Specify your bucket name
     users_bucket = session.resource('s3').Bucket('my_bucket')

     # 'list' bucket should succeed
     for obj in users_bucket.objects.all():
        print(obj.key)

     # Download a file
     users_bucket.download_file('some_data.csv', './some_data.csv')
     ```

     This code does not provide credentials as arguments to the client constructor, since it assumes:

   * Credentials are automatically populated at `/var/lib/domino/home/.aws/credentials` as specified in the environment variable `AWS_SHARED_CREDENTIALS_FILE`

   * You have already set up credentials in the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables.

After running the previous code, a local copy of *some\_data.csv* exists in the same directory as your Python script or notebook. You can load the data into a [pandas](https://pandas.pydata.org/) dataframe.

```python theme={null}
df = pd.read_csv('some_data.csv')
```

See [Use Datasets and Snapshots](/cloud/platform-capabilities/core-concepts/data/datasets/use-datasets-and-snapshots) to learn how to make files available to code in Domino.

### R and aws.s3

1. To connect to S3 from R, Domino recommends the [cloudyr project’s](https://cloudyr.github.io/) package called [aws.s3](https://github.com/cloudyr/aws.s3).

2. If you’re using a [Domino Standard Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments), aws.s3 is already installed. To add aws.s3 to an environment, use the following Dockerfile instructions.

   ```dockerfile theme={null}
   USER root

   RUN R -e 'install.packages(c("httr","xml2"), repos="https://cran.r-project.org")'
   RUN R -e 'install.packages("aws.s3", repos = c("cloudyr" = "http://cloudyr.github.io/drat"))'

   USER ubuntu
   ```

3. For basic instructions about using aws.s3 see the [package README](https://github.com/cloudyr/aws.s3/blob/master/README.md). The following is an example of downloading a file where:

   * You have set up the environment variables with credentials for your AWS account

   * Your account has access to an S3 bucket named `my_bucket`

   * The bucket contains an object named `some_data.csv`

   ```r theme={null}
   # Load the package
   library("aws.s3")

   # If you are using a credential file with multiple profiles. Otherwise, this can be excluded.
   Sys.setenv("AWS_PROFILE" = "<AWS profile>")

   # Download some_data.csv from my_bucket and write to ./some_data.csv locally
   save_object("some_data.csv", file = "./some_data.csv", bucket = "my_bucket")
   ```

4. After running the previous code, a local copy of *some\_data.csv* exists in the same directory as your R script or notebook. Read from that local file to work with the data it contains.

   ```r theme={null}
   myData <- read.csv(file="./some_data.csv", header=TRUE, sep=",")
   View(myData)
   ```

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to Amazon S3 Tabular with AWS Glue
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-amazon-s3-with-aws-glue

Use an administrator-created Amazon S3 Tabular Data Source with AWS Glue in a Project, then access and share it through standard Data Source workflows.

Administrators can create a Data Source for [tabular S3 with AWS Glue](https://docs.aws.amazon.com/athena/latest/ug/data-sources-glue.html). See the [Admin Guide](/cloud/admin/data-administration/external-data#create-a-starburst-powered-data-source) for details.

After your administrator has created the Data Source, you can [work with it in your project](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources) as usual.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to Azure Blob Storage
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-azure-blob-storage

Create an Azure Blob Storage Data Source with an account, container, access key, and access permissions, then test its credentials in Domino.

Azure Blob Storage stores massive amounts of unstructured object data, such as text or binary data. Domino Data Sources is the easiest way to connect to Azure Blob Storage from Domino.

## Create an Azure Blob Storage Data Source

1. From the navigation pane, click **Data > Data Sources**.

2. Click **Create a Data Source**.

3. In the New Data Source window, from **Select Data Store**, select **Azure Blob Storage**.

4. Enter unique names for the account, container, and Data Source. Optionally, enter a **Description** to explain the purpose of the Data Source to others.

5. Enter your Azure Blob Storage [Access Key](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal#view-account-access-keys). The Domino secret store backed by HashiCorp Vault securely stores the credentials.

6. Click **Test Credentials** to validate authentication.

7. Select who can view and use the Data Source in Projects.

8. Click **Finish Setup**.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to Azure Data Lake Storage
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-azure-data-lake-storage

Create an Azure Data Lake Storage Data Source with an account, container, and access key, then test credentials and set who can use it.

Azure Data Lake Storage (ADLS) is a cloud-based data-warehouse. This topic describes how to connect to [account access keys](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal#view-account-access-keys) from Domino. You must have network connectivity between Azure Data Lake Storage and your Domino deployment.

The easiest way to connect to a Azure Data Lake Storage instance from Domino is to create a Domino Data Source as described below.

## Create an ADLS Data Source

1. From the navigation pane, click **Data > Data Sources**.

2. Click **Create a Data Source**.

3. In the New Data Source window, from **Select Data Store**, select **ADLS** for **Azure Data Lake Store**.

4. Enter the unique storage account name in **Account Name**.

5. Enter the name of the **Container**.

6. Enter the **Data Source Name**.

7. Optional: Enter a **Description** to explain the purpose of the Data Source to others.

8. Click **Next**.

9. Enter your [Access Key](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal#view-account-access-keys) to connect to Azure Data Lake Storage. The Domino secret store backed by HashiCorp Vault securely stores the credentials.

10. Click **Test Credentials**.

11. If the Data Source authenticates, click **Next**.

12. Select who can view and use the Data Source in projects.

13. Click **Finish Setup**.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to BigQuery
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-bigquery

Create a BigQuery Data Source with a project ID and JSON private key, or use an unsupported direct Python connection with a service account.

Domino can connect to and query any common database, including Google BigQuery. The easiest way to connect to a [Google BigQuery](https://cloud.google.com/bigquery/) instance from Domino is to create a Domino Data Source as described below.

You must have network connectivity between BigQuery and your Domino deployment.

## Create a BigQuery Data Source

1. From the navigation pane, click **Data > Data Sources**.

2. Click **Create a Data Source**.

3. In the New Data Source window, from **Select Data Store**, select **Google BigQuery**.

4. Optional: Enter the unique identifier for your project in **GCP Project ID**.

5. Enter the **Data Source Name**.

6. Optional: Enter a **Description** to explain the purpose of the Data Source to others.

7. Click **Next**.

8. Copy your **Private Key (JSON format)**. See [creating a service account](https://cloud.google.com/docs/authentication/getting-started#creating_a_service_account) for instructions about creating a service account and downloading the JSON credentials file. You must copy the entire content of the file. The Domino secret store backed by HashiCorp Vault securely stores the credentials.

9. Click **Test Credentials**.

10. When your Data Source authenticates, click **Next**.

11. Enter the users and organizations who can view and use the Data Source in projects.

12. Click **Finish Setup**.

    <img alt="The BigQuery setup" />

## Alternate way to connect to a Google BigQuery Data Source

You can use the [BigQuery API](https://cloud.google.com/bigquery/docs/reference/libraries-overview) to query any Dataset and table to which you have access, whether it’s public or private, by authenticating with your Google Cloud private key.

<Warning>
  This section describes an alternate method to connect to . Domino does not officially support this method.
</Warning>

1. Go to Google’s Service Accounts page. Select a previous project or create a new project.

   <img alt="Select a project" />

   If you selected **Create**, the New Project page opens:

   <img alt="Create a new project" />

2. **Create a Service account** for your project.

   <img alt="Create a service account" />

3. Define the access that the Service account must have to BigQuery. See [Google’s Access Control documentation](https://cloud.google.com/bigquery/docs/access-control) for more information.

   <img alt="Define the access" />

4. Confirm that your Service account has been created.

   <img alt="New service account has been created" />

5. On the Service Accounts page, create a new key.

   <img alt="Create a new key" />

6. Download the JSON key and keep it in a safe place. You use this key later to programmatically authenticate to Google.

   <img alt="Obtain the JSON key" />

7. To enable the BigQuery API, click the **Google APIs** logo.

   <img alt="Click the Google APIs logo" />

8. In the Library page, select the **Big Query API**.

   <img alt="Select BigQuery API" />

9. If it is not enabled, click **Enable**.

   <img alt="Enable the BigQuery API" />

### Activate your credentials from Domino

Google Cloud uses the [Google Cloud SDK](https://cloud.google.com/sdk/) to activate your credentials. This is already installed in the Domino Default environment.

Execute the following bash command:

```shell theme={null}
/home/ubuntu/google-cloud-sdk/bin/gcloud auth activate-service-account <service account name> --key-file <key file path>
```

For example:

```shell theme={null}
/home/ubuntu/google-cloud-sdk/bin/gcloud auth activate-service-account big-query-example@example-big-query-170823.iam.gserviceaccount.com --key-file key.json
```

You can use a custom Domino compute environment and enter this command in Domino pre-setup script to activate the credentials before each run. Otherwise, you can execute them in workspace sessions. See [how to store your credentials securely](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables).

### Authenticate and query using Python

You need the `gcloud` and `oauth2client==1.4.12` Python packages. Use the following package to install them in your custom Domino compute environment or in your workspace session.

```shell theme={null}
pip install --user gcloud oauth2client==1.4.12
```

Use the following code to authenticate your Google credentials and query a public BigQuery table:

```python theme={null}
from oauth2client.client import GoogleCredentials
from googleapiclient.discovery import build

# Grab the application's default credentials from the environment.
credentials = GoogleCredentials.get_application_default()

# Construct the service object for interacting with the BigQuery API.
bigquery_service = build('bigquery', 'v2', credentials=credentials)

query_request = bigquery_service.jobs()
query_data = {
    'query': (
    'SELECT TOP(corpus, 10) as title, '
    'COUNT(*) as unique_words '
    'FROM [publicdata:samples.shakespeare];')
}

query_response = query_request.query(
    projectId='example-big-query-170823', # Substitute your ProjectId
    body=query_data).execute()

print('Query Results:')
for row in query_response['rows']:
    print(' '.join(field['v'] for field in row['f']))
```

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to ClickHouse
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-clickhouse

Use an administrator-created ClickHouse Data Source in a Domino Project, then access its data and share the Data Source with collaborators.

Your [administrator must create a Data Source for this data store](/cloud/admin/data-administration/external-data#add-starburst-jdbc-data-entities) for you to use.

After your administrator has created the Data Source, you can work with it in your project as usual.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to Databricks
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-databricks

Connect to a Databricks cluster or SQL warehouse through a Data Source, or configure the Databricks SDK and authentication in a Workspace.

Learn how to connect to a [Databricks Cluster](https://docs.databricks.com/en/clusters/index.html) or [Databricks SQL Warehouse](https://docs.databricks.com/en/sql/admin/create-sql-warehouse.html) from Domino.

## Create a Databricks Data Source

1. From the navigation pane, click **Data > Data Sources**.

2. Click **Create a Data Source**.

3. In the New Data Source window, from **Select Data Store**, select **Databricks**.

4. Enter the server **Host**, **Port**, and cluster/warehouse **HTTP Path**. Details on where to find these can be found on the [Databricks Documentation](https://docs.databricks.com/en/integrations/compute-details.html). The host should not contain the protocol. For example, a valid host format is "name.cloud.databricks.com".

   1. Optional: Enter the initial Catalog and/or Schema to use for this session. If you include schema, you may need to include the corresponding catalog so it can be identifiable.

5. Enter the Data Source **Name** and a **Description** to explain the purpose of the Data Source to others.

6. Enter your **Personal Access Token** to connect to Databricks. The Domino secret store backed by HashiCorp Vault securely stores the credentials.

   <Note>
     If this Personal Access Token expires after you created the Data Source, you can generate a new Personal Access Token on Databricks and edit this field with the new value.
   </Note>

7. Click **Test Credentials** to validate the authentication.

8. Select who can view and use the Data Source in projects.

9. Click **Finish Setup**.

If your users have Domino permissions to use the Data Source and enter their credentials, they can now use the Domino Data API to retrieve data with the connector.

See [Retrieve data](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources) for more information.

## Connect using the Databricks SDK

<Note>
  Domino recommends that you connect to Databricks using Data Sources to take advantage of flexible access controls, the [Domino Data API](/cloud/reference/api/data_api), and receive technical support. However, you can also connect using the [Databricks SDK](https://docs.databricks.com/en/dev-tools/sdk-python.html) if your business requires it. This SDK allows for Databricks workspace and compute management among other data operations.
</Note>

### Prepare your Workspace environment

* Ensure you have a workspace environment that uses a Python compatible [workspace IDE](/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/add-workspace-ides) like Jupyter Notebook.

* Add Python 3.8 or above and the Databricks SDK package "databricks-sdk" to the environment. For more information, please see [Add Packages to Environments](/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments).

<Note>
  You may want to use the [Domino Standard Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) base image as a starting environment, which is compatible with Python workspace IDEs and contains a Python version above 3.8.
</Note>

### Databricks authentication

1. Set up a [Databricks configuration profile](https://docs.databricks.com/en/dev-tools/auth.html#config-profiles).

2. Add the ".databrickscfg" file mentioned in the link above to your project.

3. Set the DATABRICKS\_CONFIG\_PROFILE [environment variable](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables) to be the name of the custom configuration profile you want to use.

4. Launch your workspace.

5. Add the following lines at the top of your code file:

```python theme={null}
from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
```

Alternatively, set environment variables for your workspace using your Databricks workspace host, token, and any other relevant configurations. Pass them in as parameters when calling the WorkspaceClient function:

```python theme={null}
from databricks.sdk import WorkspaceClient

w = WorkspaceClient(
  host  = MY_HOST_VARIABLE,
  token = MY_TOKEN_VARIABLE
)
```

The Databricks SDK is ready to be used in your workspace. For more information, please see the [Databricks SDK documentation](https://databricks-sdk-py.readthedocs.io/en/latest/).

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to Druid
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-druid

Use an administrator-created Druid Data Source in a Domino Project, then access its data and share the Data Source with collaborators.

Your [administrator must create a Data Source for this data store](/cloud/admin/data-administration/external-data#add-starburst-jdbc-data-entities) for you to use.

After your administrator has created the Data Source, you can work with it in your project as usual.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to Google Cloud Storage
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-gcs

Create a Google Cloud Storage Data Source for a bucket with a service account private key, then test credentials and assign access.

This topic describes how to connect to [Google Cloud Storage (GCS)](https://cloud.google.com/storage/) from Domino. You must have network connectivity between GCS and your Domino deployment.

The easiest way to connect to a GCS instance from Domino is to create a Domino Data Source as described below.

## Create a Google Cloud Storage Data Source

1. From the navigation pane, click **Data > Data Sources**.

2. Click **Create a Data Source**.

3. In the New Data Source window, from **Select Data Store**, select **GCS** for **Google Cloud Storage**.

4. Enter the name of the **Bucket**.

5. Enter the **Data Source Name**.

   <Note>
     Although Google Cloud allows storage bucket names that include underscores, Domino only allows lowercase letters, numbers, periods, and hyphens.
   </Note>

6. Optional: Enter a **Description** to explain the purpose of the Data Source to others.

7. Click **Next**.

8. Copy the **Private Key (JSON format)**. See [creating a service account](https://cloud.google.com/docs/authentication/getting-started#creating_a_service_account) for instructions about creating a service account and downloading the JSON credentials file. You must copy the entire content of the file. The Domino secret store backed by HashiCorp Vault securely stores the credentials.

9. Click **Test Credentials**.

10. If the Data Source authenticates, click **Next**.

11. Select who can view and use the Data Source in projects.

12. Click **Finish Setup**.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to Generic S3
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-generic-s3

Create a Generic S3 Data Source with a bucket, optional subfolder, host, and access keys, then test credentials and assign access.

This topic describes how to connect to a generic S3 service (such as [MiniO](https://min.io/)) from Domino. You must have network connectivity between your S3 service and your Domino deployment.

The easiest way to connect to a generic S3 service from Domino is to create a Domino Data Source as described below.

## Create a Generic S3 Data Source

1. From the navigation pane, click **Data > Data Sources**.

2. Click **Create a Data Source**.

3. In the New Data Source window, from **Select Data Store**, select **Generic S3**.

4. Enter the **Bucket**.

5. Optional: Enter a **Subfolder Path**, such as `myFolder` or `data/tests`, to restrict the Data Source to only list data from and write to the specified folder.

6. Enter the **Host**.

7. Enter the **Data Source Name**.

8. Optional: Enter a **Description** to explain the purpose of the Data Source to others.

9. Click **Next**.

10. Enter the **Access Key ID** and **Secret Access Key** to connect to Generic S3. The Domino secret store backed by HashiCorp Vault securely stores the credentials.

11. Click **Test Credentials**.

12. If the Data Source authenticates, click **Next**.

13. Select who can view and use the Data Source in projects.

14. Click **Finish Setup**.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to Greenplum
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-greenplum

Use an administrator-created Greenplum Data Source in a Domino Project, then access its data and share the Data Source with collaborators.

Your [administrator must create a Data Source for this data store](/cloud/admin/data-administration/external-data#add-starburst-jdbc-data-entities) for you to use.

After your administrator has created the Data Source, you can work with it in your project as usual.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to IBM Db2
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-ibm-db2

Connect to IBM Db2 through an administrator-created Data Source, or set up unsupported direct access from Python with ibm_db and R with ibmdbR.

You can connect to IBM db2 using a Data Source or directly.

## IBM db2 Data Source

Your [administrator must create a Data Source for this data store](/cloud/admin/data-administration/external-data#create-a-starburst-powered-data-source) for you to use.

After your administrator has created the Data Source, you can work with it in your project as usual.

## Direct connection

Connect directly to [IBM db2](https://www.ibm.com/products/db2) from Domino. You must have network connectivity between IBM db2 and your Domino deployment.

<Warning>
  Domino does not officially support this method. We provide this information as a courtesy.
</Warning>

## Python and ibm\_DB2

Domino recommends the [ibm\_db](https://github.com/ibmdb/python-ibmdb) package for interacting with DB2 databases from Python.

### Environment setup

Use the following Dockerfile instruction to add [ibm\_db](https://pypi.org/project/ibm_db/) to your environment.

This instruction assumes you already have [pip](https://pypi.org/project/pip/) installed.

```dockerfile theme={null}
RUN pip install ibm_db ibm_db_sa
```

### Credential setup

Set the following Domino environment variables to store secure information about your DB2 connection.

* `db_username`

* `db_password`

See [Store Project credentials](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables) to learn more about Domino environment variables.

### Usage

See [Python support for IBM Db2 for LUW, IBM Informix and IBM Db2 for z/OS](https://github.com/ibmdb/python-ibmdb) for detailed information about how to use the package. The following is an example for connecting to DB2 with ibm\_db where:

* You have set up the environment variables with the `db_username` and `db_password`.

* You’ve replaced `my.host.name` with the host name for your machine.

```python theme={null}
import ibm_db
import ibm_db_dbi
import pandas as pd

hostname = 'my.host.name'
port = 50001
username = os.environ['db_username']
password = os.environ['db_password']

def query_db(sql):
 ibm_db_conn = ibm_db.connect("DATABASE=IBMPROD;HOSTNAME={};PORT={};PROTOCOL=TCPIP;UID={};PWD={};".format(hostname, port, username, password), "", "")
 conn = ibm_db_dbi.Connection(ibm_db_conn)

 df = pd.read_sql_query(sql, conn)
 ibm_db.close(ibm_db_conn)
 return df

sql_cmd = """

SELECT
 *
FROM
 table

"""

df_cmd = query_db(sql_cmd)

df_cmd
```

See [Connecting to an IBM database server in Python](https://www.ibm.com/docs/en/SSEPGG_9.7.0/com.ibm.swg.im.dbclient.python.doc/doc/t0054368.html).

## R and ibmdbR

Domino recommends the [imbdbR](https://www.ibm.com/support/knowledgecenter/en/SS6NHC/com.ibm.swg.im.dashdb.doc/connecting/connect_connecting_rstudio.html) library for interacting with DB2 databases from R.

### Environment setup

Use the following Dockerfile instruction to add ibmdbR to your environment.

```dockerfile theme={null}
RUN R -e 'install.packages("ibmdbR")'
```

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to IBM Netezza
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-ibm-netezza

Connect to IBM Netezza through an administrator-created Data Source, or set up unsupported direct access with nzpy for Python or RJDBC for R.

You can connect to IBM Netezza using a Data Source or directly.

## IBM Netezza Data Source

Your [administrator must create a Data Source for this data store](/cloud/admin/data-administration/external-data#create-a-starburst-powered-data-source) for you to use.

After your administrator has created the Data Source, you can work with it in your project as usual.

## Direct connection

Connect directly to [IBM Netezza](https://www.ibm.com/products/netezza) from Domino. You must have network connectivity between Netezza and your Domino deployment.

<Warning>
  Domino does not officially support this method. We provide this information as a courtesy.
</Warning>

## Python and nzpy

Domino recommends the [nzpy](https://pypi.org/project/nzpy/) package for interacting with Netezza from Python.

### Environment setup

Use the following Dockerfile instruction to add [nzpy](https://pypi.org/project/nzpy/) to your environment.

This instruction assumes you already have [pip](https://pypi.org/project/pip/) installed.

```dockerfile theme={null}
RUN pip install nzpy
```

### Credential setup

You must set the following Domino environment variables to store secure information about your Netezza connection

* `db_username`

* `db_password`

See [Store Project credentials](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables) to learn more about Domino environment variables.

### Usage

See the [nzpy docs](https://github.com/IBM/nzpy) for detailed information about how to use the package. The following is an example for connecting to Netezza with nzpy where:

* You have set up environment variables noted above with the `db_username` and `db_password`

* You’ve replaced `my.host.name` with the host name for your machine

* You’ve replaced `my.database.name` with the database name for your machine

```python theme={null}
import nzpy
import os
import pandas as pd

hostname = 'my.host.name'
database_name = 'my.database.name'
port = 5480
username = os.environ['db_username']
password = os.environ['db_password']

def query_db(sql):
   conn = nzpy.connect(user=username, password=password, host=hostname, port=port, database=database_name)
   with conn.cursor() as cursor:
      cursor.execute(sql)
      df = cursor.fetchall()
      return df

sql_cmd = """

SELECT
 *
FROM
 table

"""

df_cmd = query_db(sql_cmd)

df_cmd
```

## R and RJDBC

Domino recommends the RJDBC package and nzjdbc jar for interacting with Netezza from R.

### Environment setup

Use the following Dockerfile instruction to add RJDBC to your environment.

```dockerfile theme={null}
RUN R -e 'install.packages("RJDBC")'
```

### Credential setup

You must set the following Domino environment variables to store secure information about your Netezza connection

* `db_username`

* `db_password`

See [Store Project credentials](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables) to learn more about Domino environment variables.

### Usage

Test if the `nzjdbc` jar is working as expected by running the below in a workspace terminal

`java -jar <path.to.the.nzjdbc.jar> -t -h my.host.name -p 5480 -u <username> -db <database_name>`

The following is a simple example for connecting to Netezza with RJDBC and `nzjdbc` jar where:

* `nzjdbc` jar is present in the project files (`/mnt/nzjdbc3`)

* You have set up environment variables noted above with the `db_username` and `db_password`

* You’ve replaced `my.host.name` with the host name for your machine

* You’ve replaced `my.database.name` with the database name for your machine

```R theme={null}
library(RJDBC)
drv <- RJDBC::JDBC(driverClass = "org.netezza.Driver", classPath = "/mnt/nzjdbc3.jar")
conn <- dbConnect(drv, 'jdbc:netezza://my.host.name:5480/my.database.name;logLevel=2',Sys.getenv('db_username'), Sys.getenv('db_password'))
dbListTables(conn)
```

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to Ignite
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-ignite

Use an administrator-created Ignite Data Source in a Domino Project, then access its data and share the Data Source with collaborators.

Your [administrator must create a Data Source for this data store](/cloud/admin/data-administration/external-data#add-starburst-jdbc-data-entities) for you to use.

After your administrator has created the Data Source, you can work with it in your project as usual.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to MariaDB
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-mariadb

Use an administrator-created MariaDB Data Source in a Domino Project, then access its data and share the Data Source with collaborators.

Your [administrator must create a Data Source for this data store](/cloud/admin/data-administration/external-data#add-starburst-jdbc-data-entities) for you to use.

After your administrator has created the Data Source, you can work with it in your project as usual.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to MongoDB
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-mongodb

Use an administrator-created MongoDB Data Source in a Project, query it with SQL using UNNEST for nested objects, and share it with collaborators.

Your administrator must create a Data Source for [MongoDB](https://www.mongodb.com/) that you can access. See [Connect to MongoDB](/cloud/admin/data-administration/external-data#create-a-starburst-powered-data-source) in the [Admin Guide](/cloud/admin).

After your administrator has created the Data Source, you can [work with it in your project](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources) as usual.

<Note>
  Domino data sources do not support querying nested objects. The workaround is to [`UNNEST`](https://count.co/sql-resources/bigquery-standard-sql/unnest) the object in the SQL query.

  The following is an example `UNNEST` query:

  ```python theme={null}
  res = ds.query("""
  select account_id, t1
  from sample_analytics.transactions
  cross join unnest (transactions)
  as t(t1, t2, t3, t4, t5, t6)
  """)
  ```
</Note>

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to MSSQL
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-mssql

Create a Microsoft SQL Server Data Source with host, database, and credentials, or use unsupported direct connections from Python or R.

This topic describes how to connect to a [Microsoft SQL Server](https://www.microsoft.com/en-us/sql-server/sql-server-2016) (MSSQL) from Domino. You must have network connectivity between Microsoft SQL Server and your Domino deployment.

The easiest way to connect to MSSQL from Domino is to create a Domino Data Source as described below.

## Create a MSSQL Data Source

1. From the navigation pane, click **Data > Data Sources**.

2. Click **Create a Data Source**.

3. In the New Data Source window, from **Select Data Store**, select **SQL Server**.

4. Enter the **Host**. Valid values are `<host string>:<port>` or `<host string>`.

5. Enter the **Port**.

6. Enter the name of the **Database**.

7. Enter the **Data Source Name**.

8. Optional: Enter a **Description** to explain the purpose of the Data Source to others.

9. Click **Next**.

10. Enter the **Username** and **Password** to connect to SQL Server. The Domino secret store backed by HashiCorp Vault securely stores the credentials.

11. Click **Test Credentials**.

12. If the Data Source authenticates, click **Next**.

13. Select who can view and use the Data Source in projects.

14. Click **Finish Setup**.

## Alternate way to connect to an MSSQL Data Source

<Warning>
  This section describes an alternate method to connect to . Domino does not officially support this method.
</Warning>

1. To interact with MSSQL databases from Python, Domino recommends the [pymssql](http://www.pymssql.org/) package.

2. Use the following Dockerfile instruction to [install pymssql](http://www.pymssql.org/intro.html) in your environment.

   This instruction assumes you already have [pip](https://pypi.org/project/pip/) installed.

   ```dockerfile theme={null}
   RUN pip install pymssql
   ```

3. You must set up the following Domino environment variables to store secure information about your MSSQL connection.

   * `DB_SERVER`

   * `DB_USERNAME`

   * `DB_PASSWORD`

     See [Store Project credentials](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables) to learn more about Domino environment variables.

     See the [pymssql documentation](http://www.pymssql.org/pymssql_examples.html) for detailed information about how to use the package. The following is an example of connecting to MSSQL with Python where the following is true:

   * You have set up environment variables noted previously.

   * The server hosts a database named `myData` with a table named `addresses`.

     ```python theme={null}
     from os import getenv
     import pymssql

     server = getenv("DB_SERVER")
     user = getenv("DB_USERNAME")
     password = getenv("DB_PASSWORD")

     conn = pymssql.connect(server, user, password, "myData")
     cursor = conn.cursor()

     cursor.execute('SELECT * FROM addresses')
     row = cursor.fetchone()
     while row:
         print("ID=%d, Name=%s" % (row[0], row[1]))
         row = cursor.fetchone()

     conn.close()
     ```

### R and RODBC to MSSQL

1. To interact with MSSQL databases from R, Domino recommends the [RODBC](https://cran.r-project.org/web/packages/odbc/index.html) library. You can use an alternative package if you’d like.

2. Use the following Dockerfile instructions to add the MSSQL drivers to your Ubuntu 16.04 environment.

   ```dockerfile theme={null}
   USER root
   RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
   RUN curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
   RUN apt-get update
   RUN ACCEPT_EULA=Y apt-get install msodbcsql17
   USER ubuntu
   ```

3. See Microsoft’s documentation for the Dockerfile instructions to add the MSSQL drivers to your Domino-supported Ubuntu environment.

4. See the [RStudio RODBC documentation](https://db.rstudio.com/databases/microsoft-sql-server/) for information about how to use the package.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to MySQL
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-mysql

Create a MySQL Data Source with basic or IAM authentication, or use unsupported direct connections with mysql-connector-python or RMySQL.

This topic describes how to connect to [MySQL](https://www.mysql.com/) from Domino. You must have network connectivity between MySQL and your Domino deployment.

The easiest way to connect to MySQL from Domino is to create a Domino Data Source as described below.

## Create a MySQL tabular Data Source

1. From the navigation pane, click **Data > Data Sources**.

2. Click **Create a Data Source**.

3. In the New Data Source window, from **Select Data Store**, select **MySQL**.

4. Enter the **Host**. Valid values are `<host string>:<port>` or `<host string>`.

   If no port is specified, the default is 3306.

5. Enter the name of the **Database**.

6. Enter the **Data Source Name**.

7. Optional: Enter a **Description** to explain the purpose of the Data Source to others.

8. Click **Next**.

9. Specify the credentials for authenticating to MySQL.

   Basic authentication is supported by default. IAM credential propagation might also be available if your administrator has [enabled it](/cloud/admin/data-administration/external-data#iam-authentication-for-aws-data-sources).

   <Note>
     IAM-authenticated connections can be used only for executing jobs and workspaces. Other execution types, such as scheduled jobs and Domino endpoints, require basic authentication.
   </Note>

10. If you are using basic authentication, click **Test Credentials** to validate your username and password.

11. Click **Next** (or **Skip for Now** to configure authentication later).

12. Select whether **Everyone** can access this Data Source or just **Specific users or organizations**.

13. Select who can view and use the Data Source in projects.

14. Click **Finish Setup**.

## Alternate way to connect to a MySQL Data Source

<Warning>
  This section describes an alternate method to connect to . Domino does not officially support this method.
</Warning>

1. Domino recommends the [mysql-connector-python](https://dev.mysql.com/doc/connector-python/en/) library to interact with MySQL databases from Python.

2. Use the following Dockerfile instruction to [install mysql-connector-python](https://dev.mysql.com/doc/connector-python/en/) in your environment. You must have [pip](https://pypi.org/project/pip/) installed.

   ```dockerfile theme={null}
   USER root
   RUN pip install mysql-connector-python
   USER ubuntu
   ```

3. You must set up the Domino environment variables to store secure information about your MySQL connection.

   * `MYSQL_HOST`

     Hostname where your MySQL service is running. Make sure your MySQL service and network firewall are configured to accept connections from Domino.

   * `MYSQL_USER`

     The MySQL user you want to authenticate as.

   * `MYSQL_PASSWORD`

     The password for the user chosen previously.

     See [Store Project credentials](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables) to learn more about Domino environment variables.

4. See the [mysql-connector-python documentation](https://dev.mysql.com/doc/connector-python/en/) for information about how to use the package. The following is an example to connect to MySQL with `mysql-connector-python` where:

   * You have set up environment variables with the host, user, and password.

   * Your user has access to a database named `db1` in the target MySQL instance.

   * The `db1` database contains a table called `employees`.

     ```python theme={null}
     from mysql.connector import (connection)
     import os

     # fetch values from environment variables and set the target database
     hostname = os.environ['MYSQL_HOST']
     username = os.environ['MYSQL_USER']
     password = os.environ['MYSQL_PASSWORD']
     dbname = 'db1'

     # establish connection to db1 database in your mysql service
     cnx = connection.MySQLConnection(user=username,
                                      password=password,
                                      host=hostname,
                                      database=dbname)

     # create cursor for passing queries to database
     cursor = cnx.cursor()

     # define query
     query = ("SELECT * FROM employees")

     # execute query
     cursor.execute(query)

     # print results
     for row in cursor:
       print(row)

     # close connection
     cnx.close()
     ```

### R and RMySQL

1. To interact with MySQL services from R, Domino recommends the [RMySQL](https://www.rdocumentation.org/packages/RMySQL/versions/0.10.16) library.

2. Use the following Dockerfile instructions to add RMySQL to your environment.

   ```dockerfile theme={null}
   RUN sudo apt-get install -y libmariadb-client-lgpl-dev
   RUN R -e 'install.packages("RMySQL")'
   ```

3. Set the following Domino environment variables to store secure information about your MySQL connection.

   * `MYSQL_HOST`

     Hostname where your MySQL service is running. Make sure your MySQL service and network firewall are configured to accept connections from Domino.

   * `MYSQL_USER`

     The MySQL user you want to authenticate as.

   * `MYSQL_PASSWORD`

     The password for the user chosen previously.

     See [Store Project credentials](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables) to learn more about Domino environment variables.

4. See the [RMySQL documentation](https://www.rdocumentation.org/packages/RMySQL/versions/0.10.16) for information about how to use the package. The following is an example for connecting to MySQL with RMySQL where:

   * You have set up environment variables with the host, user, and password.

   * Your user has access to a database named `db1` in the target MySQL instance.

   * The database contains a table named `employees`.

     ```r theme={null}
     # load the library
     library(RMySQL)

     # fetch values from environment variables and set the target database
     hostname <- Sys.getenv['MYSQL_HOST']
     username <-  Sys.getenv['MYSQL_USER']
     password <- Sys.getenv['MYSQL_PASSWORD']
     database <- 'db1'

     # set up a driver and use it to create a connection to your database
     con <- dbConnect(RMySQL::MySQL(), host = hostname,
      user = username, password = password, dbname = database)

     # run a query and load the response into a dataframe
     df_mysql <- dbGetQuery(con, "SELECT * FROM employees")

     # close your connection when finished
     dbDisconnect(con)
     ```

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to Oracle Database
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-oracle

Create an Oracle Data Source with host, port, database, and credentials, or use unsupported direct Python and R connections with Oracle Instant Client.

This topic describes how to connect to [Oracle Database](https://www.oracle.com/database/) from Domino. You must have network connectivity between Oracle and your Domino deployment.

The easiest way to connect to Oracle from Domino is to create a Domino Data Source as described below.

## Create an Oracle Data Source

1. From the navigation pane, click **Data > Data Sources**.

2. Click **Create a Data Source**.

3. In the New Data Source window, from **Select Data Store**, select **Oracle**.

4. Enter the **Host**, **Port**, and name of the **Database**.

5. Enter the **Data Source Name**.

6. Optional: Enter a **Description** to explain the purpose of the Data Source to others.

7. Click **Next**.

8. Enter the **Username** and **Password** to connect to Oracle. The Domino secret store backed by HashiCorp Vault securely stores the credentials.

9. Click **Test Credentials**.

10. If the Data Source authenticates, click **Next**.

11. Select who can view and use the Data Source in projects.

12. Click **Finish Setup**.

## Alternate way to connect to Oracle

<Warning>
  This section describes an alternate method to connect to . Domino does not officially support this method.
</Warning>

1. Before you can connect to Oracle, you must install the following client software your environment:

   * The basic package: `instantclient-basic-linux.x64-<oracle-version>dbru.zip`

   * The SDK package: `instantclient-sdk-linux.x64-<oracle-version>dbru.zip`

     You must download these files from the [Instant Client Downloads page](https://www.oracle.com/technetwork/topics/linuxx86-64soft-092277.html).

     1. Use your Oracle customer login.

     2. Host an internal mirror of the files somewhere accessible to your Domino hosts.

     3. Install the software manually. In the example environments in this topic, these files are retrieved from a private S3 bucket with [Wget](https://en.wikipedia.org/wiki/Wget). You must make them available in a similar manner for your Domino deployment.

### Python and cx\_Oracle

1. To interact with Oracle databases from Python, Domino recommends the [cx\_Oracle](https://oracle.github.io/python-cx_Oracle/) library.

2. Use the following Dockerfile instruction to install the Oracle client drivers and cx\_Oracle in your environment.

   <Note>
     You cannot copy and paste this Dockerfile directly, because you must set up your own internal host of the Oracle clients and modify the `wget` step shown here to retrieve them.
   </Note>

   ```dockerfile theme={null}
   USER root

   RUN \
       wget https://download.oracle.com/otn_software/linux/instantclient/2350000/instantclient-sdk-linux.x64-23.5.0.24.07.zip -O /home/ubuntu/instantclient-sdk-linux.x64-23.5.0.24.07.zip && \
       cd /home/ubuntu && \
       unzip instantclient-sdk-linux.x64-23.5.0.24.07.zip && \
       mv instantclient_23_5 /usr/local/lib && \
       rm instantclient-sdk-linux.x64-23.5.0.24.07.zip && \
       apt-get install -y libaio1

   RUN \
       echo 'export OCI_LIB=/usr/local/lib/instantclient_23_5' \
             >> /home/ubuntu/.domino-defaults && \
       echo 'export OCI_INC=/usr/local/lib/instantclient_23_5/sdk/include' \
             >> /home/ubuntu/.domino-defaults && \
       echo 'export LD_LIBRARY_PATH=/usr/local/lib/instantclient_23_5:$LD_LIBRARY_PATH' \
             >> /home/ubuntu/.domino-defaults

   RUN \
       cd /usr/local/lib/instantclient_23_5 && \
       ln -sf libclntsh.so.23.5 libclntsh.so && \
       chown -R ubuntu:ubuntu /usr/local/lib/instantclient_23_5

   RUN \
       echo '/usr/local/lib/instantclient_23_5' \
             > /etc/ld.so.conf.d/oracle-instantclient.conf && \
       ldconfig -v

   RUN pip install cx_Oracle --upgrade

   USER ubuntu
   ```

3. Set the following Domino environment variables to store secure information about your Oracle connection.

   * `ORACLE_HOST`

     Hostname where your database is running. Make sure your Oracle host and network firewall are configured to accept connections from Domino.

   * `ORACLE_SERVICE`

     The service name of the Oracle service running on the target host.

   * `ORACLE_USER`

     The Oracle user you want to authenticate as.

   * `ORACLE_PASSWORD`

     Password for the user specified previously.

     See [Store Project credentials](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables) to learn more about Domino environment variables.

4. See the [cx\_Oracle documentation](https://cx-oracle.readthedocs.io/en/latest/index.html) for information about how to use the package. The following is an example to connect to Oracle with cx\_Oracle where:

   * You have set up environment variables with the hostname, service name, username, and password.

   * Your user has access to a database named `houses` in the target Oracle instance.

   ```python theme={null}
   from __future__ import print_function
   import cx_Oracle
   import os

   # fetch values from environment variables and set the target database
   hostname = os.environ['ORACLE_HOST']
   service  = os.environ['ORACLE_SERVICE']
   username = os.environ['ORACLE_USER']
   password = os.environ['ORACLE_PASSWORD']
   connection_string = hostname + "/" + service

   # Connect as user "hr" with password "welcome" to the "oraclepdb" service running on this computer.
   connection = cx_Oracle.connect(username, password, connection_string)

   cursor = connection.cursor()
   cursor.execute("""
       SELECT address
       FROM houses
       WHERE zip = 90210""")
   for address in cursor:
       print("Address:", address)
   ```

### R and ROracle

1. To interact with Oracle databases from R, Domino recommends the [ROracle](https://cran.r-project.org/web/packages/ROracle/index.html) library.

2. Use the following Dockerfile instruction to install the Oracle client drivers and RODBC in your environment.

   <Note>
     You cannot copy and paste this Dockerfile directly because you must set up your own internal host of the Oracle clients and modify the `wget` step shown here to retrieve them.
   </Note>

   ```dockerfile theme={null}
   USER root

   RUN \
       wget https://download.oracle.com/otn_software/linux/instantclient/2350000/instantclient-sdk-linux.x64-23.5.0.24.07.zip -O /home/ubuntu/instantclient-sdk-linux.x64-23.5.0.24.07.zip && \
       cd /home/ubuntu && \
       unzip instantclient-sdk-linux.x64-23.5.0.24.07.zip && \
       mv instantclient_23_5 /usr/local/lib && \
       rm instantclient-sdk-linux.x64-23.5.0.24.07.zip && \
       apt-get install -y libaio1

   RUN \
       echo 'export OCI_LIB=/usr/local/lib/instantclient_23_5' \
             >> /home/ubuntu/.domino-defaults && \
       echo 'export OCI_INC=/usr/local/lib/instantclient_23_5/sdk/include' \
             >> /home/ubuntu/.domino-defaults && \
       echo 'export LD_LIBRARY_PATH=/usr/local/lib/instantclient_23_5:$LD_LIBRARY_PATH' \
             >> /home/ubuntu/.domino-defaults

   RUN \
       cd /usr/local/lib/instantclient_23_5 && \
       ln -s libclntsh.so.23.5 libclntsh.so && \
       chown -R ubuntu:ubuntu /usr/local/lib/instantclient_23_5

   RUN \
       echo '/usr/local/lib/instantclient_23_5' \
             > /etc/ld.so.conf.d/oracle-instantclient.conf && \
       ldconfig -v

   RUN \
       cd /home/ubuntu && \
       wget https://cran.r-project.org/src/contrib/ROracle_1.3-1.tar.gz && \
       R CMD INSTALL --configure-args='--with-oci-inc=/usr/local/lib/instantclient_23_5/sdk/include --with-oci-lib=/usr/local/lib/instantclient_23_5' ROracle_1.3-1.tar.gz

   USER ubuntu
   ```

3. Set up the following Domino environment variables to store secure information about your Oracle connection.

   * `ORACLE_HOST`

     Hostname where your database is running. Make sure your Oracle host and network firewall are configured to accept connections from Domino.

   * `ORACLE_SERVICE`

     The service name of the Oracle service running on the target host.

   * `ORACLE_USER`

     The Oracle user you want to authenticate as.

   * `ORACLE_PASSWORD`

     Password for the user specified previously.

     See [Store Project credentials](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables) to learn more about Domino environment variables.

4. See the [ROracle documentation](https://www.rdocumentation.org/packages/ROracle/versions/1.3-1/topics/dbConnect-methods) for usage details.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to Palantir Foundry
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-palantir

Access Palantir Foundry through an administrator-created Data Source or configure a user token, dataset path, and Palantir SDK in Domino.

Learn how to connect to [Palantir Foundry](https://www.palantir.com/platforms/foundry/) from Domino. You must have network connectivity between Palantir and your Domino deployment.

## Create a Palantir Foundry Data Source

The easiest and most secure way to connect to Palantir from Domino is to have your administrator [create a Data Source for Palantir Foundry](/cloud/admin/data-administration/external-data#create-a-starburst-powered-data-source) for users to access.

After your administrator has created the Data Source, you can [use the Data Source](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources) as usual.

<Note>
  You must specify the dataset path, obtained from the [Dataset Preview](https://www.palantir.com/docs/foundry/data-integration/dataset-preview/index.html) in Palantir, in your query as shown in this example:

  ```python theme={null}
  res =  ds.query('SELECT * FROM "/namespace/project"."/namespace/project/path/to/your/dataset"')
  ```
</Note>

## Connect using individual user credentials

You can also choose to use individual user credentials to connect to Palantir. However, you should contact your Palantir admin to ensure this method is compliant with your organization’s governance policies.

To connect to Palantir Foundry using this method, you must have the following:

* A Palantir Foundry account.

* An access token.

* A path to the Dataset.

### Create an access token

1. Sign in to Palantir Foundry.

2. In the Foundry sidebar, click **Account** and select **Settings**.

3. Under **User settings**, click **Tokens**.

   <img alt="Palantir UI on the Tokens page" />

4. Click **Create token**.

5. Copy the user token and save the token string. You can store it as an environment variable in Domino.

### Copy the Dataset path

1. In the Palantir Foundry application, go to the **Project** containing the dataset and select the dataset.

2. In the **About** section, copy the **Location** attribute.

   <Tip>
     You can also copy and use the **RID** field as a permalink to this dataset.
   </Tip>

3. Save the location path or RID string to store as an environment variable in Domino.

### Add environment variables to Domino

You can store your Palantir connection information securely in Domino as environment variables in your user or project settings.

1. Sign in to Domino.

2. Click **User** or **Project** settings.

3. Click **Environment Variables**.

4. Add the `PALANTIR_TOKEN` environment variable with the value of the user token you created in Foundry.

5. Add the `PALANTIR_HOSTNAME` environment variable with the value of your Foundry instance hostname.

See [Store Project credentials](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables) to learn more about Domino environment variables.

### Add the Palantir-SDK package to your environment

Some Domino Workspace Environments have the `palantir-sdk` package installed by default. You can also add Dockerfile commands to the environment definition to add packages:

```shell theme={null}
RUN pip install -user palantir-sdk
```

You can also create a cell at the top of a Jupyter Notebook and run the following:

```shell theme={null}
pip install palantir-sdk
```

### Run your code

You can now access your Palantir Foundry data through Python, from a Jupyter notebook or VS Code.

<img alt="Jupyter notebook accessing a Palantir dataset and using the Palantir SDK" />

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to Pinecone
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-pinecone

Create a Pinecone Data Source with its API key, install the Python client and Domino Data SDK, and configure the client in a Workspace.

Domino’s Pinecone vector database Data Source connector enables easy, secure access to vectorized content stored in [Pinecone](https://www.pinecone.io/).

## Create a Pinecone Data Source

You need the following information about your Pinecone database:

* Optional: Pinecone environment name

* Optional: Pinecone project ID

* [Pinecone API key](https://docs.pinecone.io/docs/quickstart#2-get-your-api-key)

<Note>
  Pinecone environment name and Pinecone project ID are required if the data source is accessed through Python pinecone-client\<3.0.0.
</Note>

Follow these steps to find the Pinecone project ID:

1. Log into Pinecone and select the index you want to use:

   <img alt="Select Pinecone Index" />

2. Your project ID is the seven-character string (`3r4nvmg` in this example):

   <img alt="Pinecone ID Location" />

   To create a Pinecone Data Source:

3. From the navigation pane, click **Data > Data Sources**.

4. Click **Create a Data Source**.

5. In the New Data Source window, from **Select Data Store**, select **Pinecone**.

6. Provide the **Pinecone environment name**, **Pinecone project ID**, and [Pinecone API Key](https://docs.pinecone.io/docs/quickstart#2-get-your-api-key) for your Pinecone database.

7. Create a **Data Source Name** and an optional **Description**.

Domino stores your Pinecone API Key, along with other secrets, in a [secure secret store backed by HashiCorp Vault](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables), so you can have confidence that your secrets are safe. You can test the API key before finalizing your Data Source:

1. Click **Test Credentials**.

2. Confirm that the Data Source authenticates.

3. Select who can view and use the Data Source in Projects.

## Use the Pinecone Data Source

To use your Pinecone Data Source, you must install the prerequisite Python libraries in the compute Environment you want to run Pinecone commands in and configure the Pinecone client using a Domino-specific configuration.

### Install the Pinecone Python library and Domino Data SDK

To use the Pinecone Python library in Domino, you must install the following libraries in your compute Environment. Domino standard environments have these libraries installed by default. If you are using a customized environment, [Edit your compute Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition) and add the following lines to the Dockerfile instructions:

<Warning>
  You must replace the placeholder `{x.y.z}` with the compatible version of your Domino deployment. Refer to [all released packages](https://pypi.org/project/dominodatalab-data/#history).
</Warning>

```dockerfile theme={null}
USER root

RUN python -m pip install pinecone-client

RUN python -m pip install dominodatalab-data=={x.y.z}

USER ubuntu
```

### Configure the Pinecone Python client

To connect to your Pinecone service in a Domino execution, you must initialize your Pinecone connection using a Domino-specific Environment. To find the configuration code snippets:

1. [Add the Pinecone Data Source to your Project](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources#add-an-existing-data-source-to-a-project).

2. In your Workspace, go to **Data** > **Data Sources** > **Code snippet** > **Python**.

3. Copy the code snippet, paste it into your code, and modify it as needed.

   * The code snippet for pinecone-client\<3.0.0:

     ```python theme={null}
     import os
     import pinecone
     from domino_data.vectordb import DominoPineconeConfiguration

     datasource_name = "pinecone"
     conf = DominoPineconeConfiguration(datasource=datasource_name)
     # The pinecone API key should be provided when creating the Domino Data Source and persisted securely.
     # This api_key variable is only used to satisfy the native Pinecone Python client initialization where
     # api_key is a mandatory non-empty field.
     api_key = os.environ.get("DOMINO_VECTOR_DB_METADATA", datasource_name)

     pinecone.init(
         api_key=api_key,
         environment="domino",
         openapi_config=conf)

     print(pinecone.list_indexes())

     # Replace the place holder {{index_name}} below with the index name.
     index = pinecone.Index("{{index_name}}")
     index.describe_index_stats()
     ```

   * The code snippet for pinecone-client>=3.0.0:

     ```python theme={null}
     from domino_data.vectordb import domino_pinecone3x_init_params, domino_pinecone3x_index_params
     from pinecone import Pinecone

     datasource_name = "pinecone"
     pc = Pinecone(**domino_pinecone3x_init_params(datasource_name))
     print(pc.list_indexes())

     # Replace the place holder {{index_name}} below with the index name.
     index_name = "{{index_name}}"
     index = pc.Index(**domino_pinecone3x_index_params(datasource_name, index_name))
     print(index.describe_index_stats())

     # More Pinecone Python library-supported operations can be found at https://docs.pinecone.io/docs/python-client
     # gRPC client is currently not supported.
     ```

Once your Pinecone connection is initialized in the Domino execution, you can use the [Pinecone Python client](https://docs.pinecone.io/docs/python-client) as usual.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to PostgreSQL
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-postgres

Create a PostgreSQL Data Source with basic or IAM authentication, or use unsupported direct connections with psycopg2 or RPostgreSQL.

This topic describes how to connect to [PostgreSQL](https://www.postgresql.org/) from Domino. You must have network connectivity between PostgreSQL and your Domino deployment.

The easiest way to connect to PostgreSQL from Domino is to create a Domino Data Source as described below.

## Create a Postgres Data Source

1. From the navigation pane, click **Data > Data Sources**.

2. Click **Create a Data Source**.

3. In the New Data Source window, from **Select Data Store**, select **PostgreSQL**.

4. Enter the **Host**, **Port**, and **Database** name.

5. Enter the **Data Source Name**.

6. Optional: Enter a **Description** to explain the purpose of the Data Source to others.

7. Click **Next**.

8. Specify the credentials for authenticating to Postgres.

   Basic authentication is supported by default. IAM credential propagation might also be available if your administrator has [enabled it](/cloud/admin/data-administration/external-data#iam-authentication-for-aws-data-sources).

   <Note>
     IAM-authenticated connections can be used only for executing jobs and workspaces. Other execution types, such as scheduled jobs and Domino endpoints, require basic authentication.
   </Note>

9. If you are using basic authentication, click **Test Credentials** to validate your username and password.

10. Click **Next** (or **Skip for Now** to configure authentication later).

11. Enter the users and organizations who can view and use the Data Source in projects.

12. Click **Finish Setup**.

## Alternate way to connect to a Postgres Data Source

<Warning>
  This section describes an alternate method to connect to . Domino does not officially support this method.
</Warning>

This instruction assumes you already have [pip](https://pypi.org/project/pip/) installed.

1. For Python and psycopg2, Domino recommends the [psycopg2](https://pypi.org/project/psycopg2/) library to interact with PostgreSQL databases from Python.

2. Use the following Dockerfile instruction to [install psycopg2](https://www.psycopg.org/docs/install.html) in your environment.

   ```dockerfile theme={null}
   RUN pip install psycopg2
   ```

3. Set the following as Domino environment variables to store secure information about your PostgreSQL connection.

   * `POSTGRES_HOST`

     Hostname where your DB is running. Make sure your PostgreSQL DB and network firewall are configured to accept connections from Domino.

   * `POSTGRES_USER`

     The PostgreSQL user you want to authenticate as.

   * `POSTGRES_PASSWORD`

     The password for the user chosen previously.

     See [Store Project credentials](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables) to learn more about Domino environment variables.

4. See the [psycopg2 documentation](https://www.psycopg.org/docs/) for detailed information about how to use the package. The following is an example to connect to PostgreSQL with `psycopg2` where:

   * You have set up environment variables with the hostname, username, and password.

   * Your user has access to a database named `db1` in the target PostgreSQL instance.

   * The database contains a table named `metrics`.

     ```python theme={null}
     import psycopg2
     import os

     # fetch values from environment variables and set the target database
     hostname = os.environ['POSTGRES_HOST']
     username = os.environ['POSTGRES_USER']
     password = os.environ['POSTGRES_PASSWORD']
     dbname = 'db1'

     # set up a connection object with parameters for your database
     conn = psycopg2.connect(
       host=hostname,
       port=5432,
       user=username,
       password=password,
       database=dbname, )

     # create a cursor in your connection
     cur = conn.cursor()

     # execute a query on the metrics table and store the response
     cur.execute("SELECT * FROM metrics;")
     results = cur.fetchall()

     # display the contents of the response
     print(results)
     ```

     The results object created in the previous example is a Python array of entries from the queried table.

### R and RPostgreSQL

1. To connect to R and RPostgreSQL, Domino recommends the [RPostgreSQL](https://www.rdocumentation.org/packages/RPostgreSQL/versions/0.6-2) library to interact with PostgreSQL databases from R.

2. Use the following Dockerfile instruction to add RPostgreSQL to your environment.

   ```dockerfile theme={null}
   RUN R -e 'install.packages("RPostgreSQL")'
   ```

3. Set up the Domino environment variables to store secure information about your PostgreSQL connection.

   * `POSTGRES_HOST`

     Hostname where your DB is running. Make sure your PostgreSQL DB and network firewall are configured to accept connections from Domino.

   * `POSTGRES_USER`

     The PostgreSQL user you want to authenticate as.

   * `POSTGRES_PASSWORD`

     The password for the user chosen previously.

     See [Store Project credentials](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables) to learn more about Domino environment variables.

4. See the [RPostgreSQL documentation](https://www.rdocumentation.org/packages/RPostgreSQL/versions/0.6-2) for information about how to use the package. The following is an example for connecting to PostgreSQL with RPostgreSQL where:

   * You have set up environment variables with the hostname, username, and password.

   * Your user has access to a database named `db1` in the target PostgreSQL instance.

   * The database contains a table named `` metrics` ``.

     ```r theme={null}
     # load the library
     library(RPostgreSQL)

     # fetch values from environment variables and set the target database
     hostname <- Sys.getenv['POSTGRES_HOST']
     username <-  Sys.getenv['POSTGRES_USER']
     password <- Sys.getenv['POSTGRES_PASSWORD']
     database <- 'db1'

     # set up a driver and use it to create a connection to your database
     drv <- dbDriver("PostgreSQL")
     conn <- dbConnect(
       drv,
       host=hostname,
       port=5432,
       user=username,
       password=password,
       dbname=database )

     # run a query and load the response into a dataframe
     df_postgres <- dbGetQuery(conn, "SELECT * from metrics;")

     # close your connection when finished
     dbDisconnect(conn)
     ```

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to Qdrant
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-qdrant

Create a Qdrant Data Source with its host, port, and API key, install qdrant-client, and configure the Python client in a Workspace.

Domino’s Qdrant vector database Data Source connector enables easy, secure access to vectorized content stored in [Qdrant](https://qdrant.tech/).

## Create a Qdrant Data Source

You need the following information about your Qdrant database:

* Host

* Port

* [Qdrant API key](https://qdrant.tech/documentation/cloud/authentication/#create-api-keys)

To create a Qdrant Data Source:

1. From the navigation pane, click **Data > Data Sources**.

2. Click **Create a Data Source**.

3. In the New Data Source window, from **Select Data Store**, select **Qdrant**.

4. Provide the Qdrant **Host** name, **Port**, and [Qdrant API Key](https://qdrant.tech/documentation/cloud/authentication/#create-api-keys) for your Qdrant database.

5. Specify a **Data Source Name** and an optional **Description**.

Domino stores your Qdrant API Key, along with other secrets, in a [secure secret store backed by HashiCorp Vault](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables), so you can have confidence that your secrets are safe. You can test the API key before finalizing your Data Source:

1. Click **Test Credentials**.

2. Confirm that the Data Source authenticates.

3. Select who can view and use the Data Source in Projects.

## Use the Qdrant Data Source

To use your Qdrant Data Source, you must install the prerequisite Python libraries in the compute Environment you want to run Qdrant commands in and configure the Qdrant client using a Domino-specific configuration.

### Install the Qdrant Python library

To install the Qdrant Python library in your compute Environment by [edit your compute Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition) and add the following lines to the Dockerfile instructions:

```dockerfile theme={null}
USER root

RUN pip install --user qdrant-client

USER ubuntu
```

### Configure the Qdrant Python client

To connect to your Qdrant service in a Domino execution, you must initialize your Qdrant connection using a Domino-specific Environment. To find the configuration code snippets:

1. [Add the Qdrant Data Source to your Project](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources#add-an-existing-data-source-to-a-project).

2. In your Workspace, go to **Data** > **Data Sources** > **Code snippet** > **Python**.

3. Copy and paste the following code snippet into your code, and modify it as needed.

```python theme={null}
import os

from qdrant_client import QdrantClient

# Create the Qdrant client
qdrant = QdrantClient(
    url=os.environ.get("DOMINO_DATA_API_GATEWAY", "http://127.0.0.1:8766"),
    metadata={"X-Domino-Datasource": "qdrant"},
)

# List all collections
print(qdrant.get_collections())
```

Once your Qdrant connection is initialized in the Domino execution, you can use the [Qdrant Python client](https://python-client.qdrant.tech/) normally.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to Redshift
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-redshift

Create an Amazon Redshift Data Source with basic or IAM authentication, or use unsupported direct connections with psycopg2 or RPostgreSQL.

This topic describes how to connect to [Amazon Redshift](https://aws.amazon.com/pm/redshift/) from Domino. You must have network connectivity between Redshift and your Domino deployment.

The easiest way to connect to Redshift from Domino is to create a Domino Data Source as described below.

## Create a Redshift Data Source

1. From the navigation pane, click **Data > Data Sources**.

2. Click **Create a Data Source**.

3. In the New Data Source window, from **Select Data Store**, select **Amazon Redshift**.

4. Enter the **Host**, **Port**, and name of the **Database**.

5. Enter the **Data Source Name**.

6. Optional: Enter a **Description** to explain the purpose of the Data Source to others.

7. Click **Next**.

8. Specify the credentials for authenticating to Redshift.

   Basic authentication is supported by default. IAM credential propagation might also be available if your administrator has [enabled it](/cloud/admin/data-administration/external-data#iam-authentication-for-aws-data-sources).

   <Note>
     IAM-authenticated connections can be used only for executing jobs and workspaces. Other execution types, such as scheduled jobs and Domino endpoints, require basic authentication.
   </Note>

9. If you are using basic authentication, click **Test Credentials** to validate your username and password.

10. Select who can view and use the Data Source in projects.

11. Click **Finish Setup**.

## Alternate way to connect to a Redshift Data Source

<Warning>
  This section describes an alternate method to connect to . Domino does not officially support this method.
</Warning>

### Prerequisites

* Domino recommends storing your database username and password as [environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables) in your project. This lets you access them at runtime without including them in your code.

### Python

To establish a connection to Redshift with the `psycopg2` library:

```python theme={null}
import psycopg2
import os

HOST = os.environ['REDSHIFT_HOST']
PORT = 5439 # redshift default
USER = os.environ['REDSHIFT_USER']
PASSWORD = os.environ['REDSHIFT_PASSWD']
DATABASE = 'mydatabase'

def db_connection():
    conn = psycopg2.connect(
        host=HOST,
        port=PORT,
        user=USER,
        password=PASSWORD,
        database=DATABASE,
    )
    return conn

example_query = "SELECT * FROM my_table LIMIT 5"

conn = db_connection()
try:
    cursor = conn.cursor()
    cursor.execute(example_query)
    results = cursor.fetchall() # careful, the results could be huge
    conn.commit()
    print results
finally:
    conn.close()

# using pandas
import pandas as pd
conn = db_connection()
try:
    df = pd.read_sql(example_query, conn)
    df.to_csv('results/outfile.csv', index=False)
finally:
    conn.close()
```

### R

To establish a connection to Redshift with the RPostgreSQL library:

```r theme={null}
install.packages("RPostgreSQL")
library(RPostgreSQL)

redshift_host <- Sys.getenv("REDSHIFT_HOST")
redshift_port <- "5439"
redshift_user <- Sys.getenv("REDSHIFT_USER")
redshift_password <- Sys.getenv("REDSHIFT_PASSWORD")
redshift_db <- "mydatabase"

drv <- dbDriver("PostgreSQL")
conn <- dbConnect(
    drv,
    host=redshift_host,
    port=redshift_port,
    user=redshift_user,
    password=redshift_password,
    dbname=redshift_db)

tryCatch({
    example_query <- "SELECT * FROM my_table LIMIT 5"
    results <- dbGetQuery(conn, example_query)
    print(results)
}, finally = {
    dbDisconnect(conn)
})
```

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to SAP HANA 2
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-saphana

Use an administrator-created SAP HANA 2 Data Source in a Domino Project, then access its data and share the Data Source with collaborators.

Your [administrator must create a Data Source for this data store](/cloud/admin/data-administration/external-data#create-a-starburst-powered-data-source) for you to use.

After your administrator has created the Data Source, you can work with it in your project as usual.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to SingleStore (MemSQL)
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-singlestore

Use an administrator-created SingleStore Data Source in a Domino Project, then access its data and share the Data Source with collaborators.

Your [administrator must create a Data Source for this data store](/cloud/admin/data-administration/external-data#add-starburst-jdbc-data-entities) for you to use.

After your administrator has created the Data Source, you can work with it in your project as usual.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to Snowflake
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-snowflake

Create a Snowflake Data Source with database, schema, warehouse, and role options, then authenticate with basic credentials or OAuth.

This topic describes how to connect to [Snowflake](https://www.snowflake.com/) from Domino.

The easiest way to connect to Snowflake from Domino is to create a Domino Data Source as described below.

## Prerequisites

You must have network connectivity between Snowflake and your Domino deployment.

To use Snowflake code integrations, such as [Snowpark](https://docs.snowflake.com/en/developer-guide/snowpark/index.html), you must agree to the [Snowflake third party terms](https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-packages.html#getting-started). To agree to these terms, you must have a Snowflake account with the [ORGADMIN](https://docs.snowflake.com/en/user-guide/organizations-gs.html#enabling-the-orgadmin-role-for-an-account) role. If you don’t have access to a Snowflake account with the ORGADMIN role, [submit a Snowflake support ticket](https://community.snowflake.com/s/article/How-To-Submit-a-Support-Case-in-Snowflake-Lodge).

## Create a Snowflake Data Source

Domino recommends that you use a [Domino Data Source](/cloud/platform-capabilities/core-concepts/data/data-source-connectors) to connect to a Snowflake instance from Domino.

1. From the navigation pane, click **Data > Data Sources**.

2. Click **Create a Data Source**.

3. In the New Data Source window, from **Select Data Store**, select **Snowflake**.

   **Account Name**\
   If the Domino deployment and Snowflake Data Source, are in the same region, enter the **Account Name** as `<account name>`. However, if the Domino deployment and Snowflake Data Source are in different regions, enter the **Account Name** as `<account name>.<region>`. For example, `abc.us-east`. (For Azure, these might take the form of `def.east-us-2.azure`.)

   **Optional: Database**\
   The name of the Snowflake database that contains the data.

   **Optional: Schema**\
   The name of the active schema for the session.

   **Optional: Warehouse**\
   The name of all the compute resource clusters that provide the resources in Snowflake.

   **Optional: Role**\
   The role that has privileges to the Data Source.

   **Data Source Name**\
   The name that identifies the Data Source.

   **Optional: Description**\
   The purpose for the Data Source.

4. Click **Next**.

5. Specify the credentials for authenticating to Snowflake.

   * **Basic authentication** is supported by default. The Domino secret store backed by HashiCorp Vault securely stores the credentials.

   * **OAuth authentication** is available if your administrator has [enabled this option](/cloud/admin/data-administration/external-data#oauth-for-snowflake).

     <Note>
       OAuth-authenticated connections can be used for any execution type *except* Domino endpoints.
     </Note>

6. If you are using basic authentication, click **Test Credentials** to validate your username and password.

7. Click **Next** (or **Skip for Now** to configure authentication later).

8. Select who can view and use the Data Source in projects.

9. Click **Finish Setup**.

If your users have Domino permissions to use the Data Source and enter their credentials, they can now use the Domino Data API to retrieve data with the connector.

See [Retrieve Data](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources) for more information.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources#share-a-data-source) with your collaborators.


# Connect to Starburst/Trino
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-starburst-trino

Create a Starburst or Trino Data Source with host, port, catalog, schema, and credentials, then run queries against connected data stores.

Use the Starburst/[Trino](https://trino.io/docs/current/) Data Source to connect to an existing Starburst/Trino cluster. Once connected, you can use the Data Source to:

* Execute queries against the Starburst/Trino cluster using the data stores it’s connected to.

* Run Starburst/Trino federated queries supported by your Starburst/Trino cluster.

See the [list of Trino connectors](https://trino.io/docs/current/connector.html) Trino connectors for supported data stores.

You must have network connectivity between the Starburst/Trino cluster and your Domino deployment.

<Note>
  In a hybrid deployment, Trino Data Sources are only available in executions that use a hardware tier in the local [data plane](/cloud/platform-capabilities/core-concepts/data-plane).
</Note>

## Create a Trino Data Source

Domino recommends that you use a [Domino Data Source](/cloud/platform-capabilities/core-concepts/data/data-source-connectors) to connect to a Trino instance.

1. From the navigation pane, click **Data > Data Sources**.

2. Click **Create a Data Source**.

3. In the New Data Source window, from **Select Data Store**, select **Trino**.

4. Enter the details about your Data Source and click **Next**:

   **Host**\
   The URL, hostname, or IP address of your Trino host.

   **Optional: Networking Proxy**\
   The URL (host and port) of your networking proxy that points to your Trino cluster, if applicable.

   **Port**\
   The port number for the connection, such as 443.

   **Catalog**\
   The name of the Trino catalog where your data is located. For example, if you plan to use the table `hive.test_data.test` then the catalog name is `hive`.

   **Optional: Schema**\
   The name of the Trino schema to use. For example, if you plan to use the table `hive.test_data.test` then the schema name is `test_data`.

   **Data Source Name**\
   The name that identifies the Data Source.

   **Optional: Data Source Description**\
   The purpose for the Data Source.

5. Specify the credentials for authenticating to Trino.

   Basic authentication is supported by default. The Domino secret store backed by HashiCorp Vault securely stores the credentials.

6. Click **Test Credentials**.

7. If the Data Source authenticates, click **Next**.

8. Select who can view and use the Data Source in projects.

9. Click **Finish Setup**.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to Synapse
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-synapse

Use an administrator-created Synapse Data Source in a Domino Project, then access its data and share the Data Source with collaborators.

Your [administrator must create a Data Source for this data store](/cloud/admin/data-administration/external-data#add-starburst-jdbc-data-entities) for you to use.

After your administrator has created the Data Source, you can work with it in your project as usual.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Connect to Teradata
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-teradata

Use an administrator-created Teradata Data Source, or use an unsupported direct connection with the Teradata SQL Driver for Python and environment variables.

This topic describes how to connect to [Teradata](https://www.teradata.com/) from Domino.

Teradata Vantage is a multicloud data platform providing access to data lakes, data warehouses, and analytics. You must have network connectivity between Teradata and your Domino deployment.

The easiest way to connect to Teradata from Domino is to create a Domino Data Source as described below.

## Create a Teradata Data Source

Administrators can create a Data Source for Teradata that Domino users can access. Domino recommends this method. See the [Admin Guide](/cloud/admin/data-administration/external-data#create-a-starburst-powered-data-source) for details.

After your administrator has created the Data Source, you can [query it](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

<Note>
  You must specify the database name in your query, as in this example where `td_demo` is the database name and `yellow_cab` is the table name:

  ```python theme={null}
  res = ds.query("select * from td_demo.yellow_cab")
  ```
</Note>

## Alternate way to connect to a Teradata Data Source

<Warning>
  Domino does not officially support this method. We provide this information as a courtesy.
</Warning>

Domino recommends the [Teradata SQL Driver for Python](https://pypi.org/project/teradatasql/).

### Environment setup

The Teradata SQL driver for Python comes pre-installed in several Domino environments. If your environment does not have it installed by default, you can use the following Dockerfile instructions to add it to your environment:

```dockerfile theme={null}
RUN pip install teradatasql
```

### Credential setup

Set the following Domino environment variables to store secure information about your Teradata connection.

* `TERADATA_USER`

* `TERADATA_PASSWORD`

* `TERADATA_HOST`

See [Store Project credentials](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables) to learn more about Domino environment variables.

### Usage

See the [Teradata SQL Driver for Python documentation](https://github.com/Teradata/python-driver) for information about how to use the package. The following is an example.

```python theme={null}
import teradatasql
import os

con = teradatasql.connect(
    user=os.environ['TERADATA_USER'],
    password=os.environ['TERADATA_PASSWORD'],
    host=os.environ['TERADATA_HOST']
    )
cur = con.cursor()
try:
    cur.execute ('{fn teradata_nativesql}Driver version {fn teradata_driver_version}  Database version {fn teradata_database_version}')
    print (cur.fetchone () [0])
finally:
    cur.close()
con.close()
```

## Additional resources for Teradata SQL driver

**Python**:

* [Official documentation](https://docs.teradata.com/r/I5P9sy9O8T4Bsv1ndwGMbg/y3UhMbfkXZVQkYdQlf92lA)

* [Repository](https://github.com/Teradata/python-driver)

* [Package](https://pypi.org/project/teradatasql/)

**R**:

* [Official documentation](https://docs.teradata.com/r/I5P9sy9O8T4Bsv1ndwGMbg/dVYM0P9c1un4po_WBJCFYg)

* [Repository](https://github.com/Teradata/r-driver)

* [Packages](https://github.com/Teradata/r-driver#Installation)

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources#share-a-data-source) with your collaborators.


# Connect to Vertica
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-vertica

Use an administrator-created Vertica Data Source in a Domino Project, then access its data and share the Data Source with collaborators.

Your [administrator must create a Data Source for this data store](/cloud/admin/data-administration/external-data#add-starburst-jdbc-data-entities) for you to use.

After your administrator has created the Data Source, you can work with it in your project as usual.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Access Data Sources from SAS workspaces
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/data-source-from-sas

Access Microsoft SQL Server, Oracle, and Databricks Data Sources from SAS Workspaces through ODBC, then troubleshoot common connection errors.

You can access [Microsoft SQL Server](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-mssql), [Oracle](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-oracle), and [Databricks](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-databricks) Data Sources from your [SAS Workspaces](/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/sas-workspace). Domino uses the SAS/ACCESS interface to ODBC to access these Data Sources.

## Scope limitations

* Domino only supports [Microsoft SQL Server](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-mssql), [Oracle](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-oracle), and [Databricks](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-databricks) Data Sources through the SAS/ACCESS interface to ODBC.

* Parameterized queries aren’t supported.

## Prerequisites

* Your Data Sources must be created before you launch your Workspace. Restart your Workspace to see new Data Sources.

* The environment must have the Arrow Flight SQL ODBC driver installed.

  To install the Arrow Flight SQL ODBC driver, add the following lines to your Dockerfile.

  ```dockerfile theme={null}
  USER root

  # ODBC Driver for Arrow Flight SQL

  RUN curl -s -o arrow-flight-sql-odbc-driver.rpm https://download.dremio.com/arrow-flight-sql-odbc-driver/0.9.1/arrow-flight-sql-odbc-driver-0.9.1.168-1.x86_64.rpm

  RUN dnf clean packages
  RUN dnf config-manager --add-repo https://repo.almalinux.org/almalinux/8/BaseOS/x86_64/os/
  RUN dnf install -y --nogpgcheck --repo "repo.almalinux.org_almalinux_8_BaseOS_x86_64_os_" libnsl
  RUN sudo dnf -y --nogpgcheck localinstall arrow-flight-sql-odbc-driver.rpm

  RUN dnf clean all && rm arrow-flight-sql-odbc-driver.rpm

  # Configure SAS odbc manager
  RUN echo "export ODBCSYSINI=/etc" >> /opt/sas/viya/config/etc/workspaceserver/default/workspaceserver_usermods.sh && \
      echo "export ODBCINI=odbc.ini" >> /opt/sas/viya/config/etc/workspaceserver/default/workspaceserver_usermods.sh && \
      echo "export ODBCINSTINI=odbcinst.ini" >> /opt/sas/viya/config/etc/workspaceserver/default/workspaceserver_usermods.sh && \
      echo "export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/lib64" >> /opt/sas/viya/config/etc/workspaceserver/default/workspaceserver_usermods.sh

  USER domino
  ```

## Code snippet

The following SAS snippet shows you how to access Domino Data Sources using the SAS/ACCESS interface to ODBC and SQL pass-through to connect and send statements directly to the DBMS. Enter your Domino Data Source name in the `datasrc` field.

```sas theme={null}
options set=EASYSOFT_UNICODE=YES;

proc sql;
connect to odbc as mycon(datasrc="sqlserver");

select *
   from connection to mycon
      (select top 10 * from yellow_cab);

disconnect from mycon;
quit;
```

## Troubleshooting

Troubleshoot SAS errors by viewing the error message and stack trace in the SAS Studio UI log.

Here are some common errors and solutions:

* If a Data Source is created after launching the Workspace, it won’t generate an ODBC configuration causing the error, "Data source name not found and no default driver specified". To resolve the error, restart the Workspace.

* The error "Flight returned internal error" indicates issues with the SQL statement or the use of an invalid SQL dialect.

## Next steps

* After connecting to your Data Source, learn how to [Use Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources).

* [Share this Data Source](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) with your collaborators.


# Domino Data Sources
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/index

Compare Domino Data Source connectors with direct connections, identify supported stores, and choose an access method for external data.

Domino can connect to your external data sources, such as databases, data warehouses, and data lakes.

Domino automatically creates an [audit trail](/cloud/admin/operations/audit-logs/monitor-data-source-logs) for Domino Data Sources to let you audit and track all Data Source-related activities.

<Note>
  [Remote data planes](/cloud/platform-capabilities/core-concepts/data-plane) are needed to use External Data Volumes in Domino Cloud.
</Note>

## Domino Data Source vs. direct connections

There are two ways to connect your external data:

* **Domino Data Source connectors** are available for some types of data sources. Data Source connectors are the simplest way to connect to supported data sources. It takes just a few clicks to create a Data Source, and you can re-use it in any project. Administrators can [pre-configure a Data Source](/cloud/admin/data-administration/external-data#create-a-data-source) with the connection details and share it, so users supply only their credentials instead of guessing at hostnames or port numbers.

* **Direct connections** are available for all types of data sources. This approach is useful if you want to use a data source that is not supported by a Domino connector, or if you want to use a different version of the connector than the one that is installed on the Domino server. However, direct connections require more configuration than Domino connectors and are not officially supported.

<Note>
  In a [Domino Nexus deployment](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture), Data Sources can be accessed on both the `Local` and remote [data planes](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace), with the exception of Starburst Trino which can only be accessed on the Local data plane. Data sources may not be usable in every data plane due to network restrictions.
</Note>

The table below compares the two methods for accessing external data sources:

<table>
  <thead>
    <tr>
      <th>Domino Data Source</th>
      <th>Direct connection</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td><p>Fetch data from the Data Source into a data frame using SQL.</p></td>
      <td><p>Fetch data using native library functions.</p></td>
    </tr>

    <tr>
      <td><p>Optimized for read-heavy workflows.</p> <p>Data sources support synchronous read/write operations. However, Data Source connectors don’t return metadata about the results of a write operation, such as the number of rows affected.</p></td>
      <td><p>Write-heavy workflows.</p> <p>Direct connections can support asynchronous read/write operations</p></td>
    </tr>

    <tr>
      <td><p>Easy to set up and use.</p> <p>Once configured a few simple steps or programmatically, you can embed the Data Sources into your notebooks and fetch data into your data frames through SQL.</p></td>
      <td><p>Use of specialized library functions is a priority.</p> <p>This can be important in non-traditional data sources or in data lakes, where you might rely on specialized functions to process different shapes of data to embed them into your CI/CD pipelines.</p></td>
    </tr>

    <tr>
      <td><p>Sharing is a priority.</p> <p>Domino Data Sources are easy to share with other users across Domino projects. They also support service accounts that authenticate all users, in addition to individual accounts where users provide their own credentials.</p></td>
      <td><p>Sharing is not a priority.</p> <p>You must set up connectivity to your Data Sources on your own and explicitly share them with other users.</p></td>
    </tr>

    <tr>
      <td><p>Easy security configuration.</p> <p>Domino Data Sources allow for secure safe-keeping of individual and service account credentials in HashiCorp Vault; there is no risk of those secrets being shared with or leaked to other users.</p></td>
      <td><p>Secure connectivity must be established through custom solutions using Kubernetes secrets or through integration with a centralized vault/safe-keeping mechanism.</p></td>
    </tr>

    <tr>
      <td><p>Portability is not a priority.</p> <p>Domino Data Sources do not prioritize support for integrated workflows/pipelines that pull/process data outside of Domino. Although you can call the <a href="/cloud/reference/api/data_api">Domino Data API</a> from outside of Domino, it does not offer the same benefits and safeguards that native libraries provide when talking directly to services outside of Domino.</p></td>
      <td><p>Portability is a priority.</p> <p>If you need to reuse your code outside of Domino or you want broad functionality when communicating with your data services from outside of Domino, using native libraries might be preferable.</p></td>
    </tr>
  </tbody>
</table>

## Domino connectors

Domino comes with connectors for many external data sources. See the full list of supported connectors in the left navigation under Domino Data Sources.

### Starburst JDBC Data Source connectors

Only Domino administrators can create [Data Sources powered by Starburst JDBC](/cloud/admin/data-administration/external-data#create-a-starburst-powered-data-source). These extend Domino’s connectivity to data stores such as [ClickHouse](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-clickhouse), [Druid](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-druid), [Greenplum](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-greenplum), [Ignite](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-ignite), [MariaDB](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-mariadb), [SAP HANA 2](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-saphana), [SingleStore](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-singlestore), [Synapse](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-synapse), [Vertica](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-vertica), and [tabular S3 with AWS Glue](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-amazon-s3-with-aws-glue). After your administrator has created the Data Source, [add it to your Project](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources) and [share it](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) like any other Data Source.

For the full list of data stores compatible with Starburst JDBC drivers, refer to [Starburst’s documentation](https://docs.starburst.io/latest/connector.html).

## Connect directly to an external data source

The following data sources are not supported by a Domino Data Source connector, but you can still connect directly to these data sources. Domino does not provide support for any issues related to direct connections.

* [DataRobot](/cloud/platform-capabilities/core-concepts/data/access-external-data-directly/connect-to-datarobot)

* [Impala](/cloud/platform-capabilities/core-concepts/data/access-external-data-directly/connect-to-impala)

## Next steps

[Use Data Sources.](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources)


# Use Data Sources
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources

Create or add Data Sources to Projects, get Python and R code snippets, and use the Domino Data API to query, read, write, and export data.

Data sources have a global scope in a Domino deployment and are accessible to anyone with the appropriate permissions in any project.

## Connect to a Data Source

Some data stores require additional steps. The [Data Source connector](/cloud/platform-capabilities/core-concepts/data/data-source-connectors) page has more details about specific connections.

<Note>
  Data Sources can be set up to use either service account credentials or individual user credentials. Verify the credentials you’ll need with your administrator before connecting to data sources.
</Note>

These steps show you how to connect to a Data Store:

1. From the navigation pane, click **Data** > **Data Sources**.

2. Click **Create a Data Source**.

3. Choose an option from the **Select Data Store** dropdown.

4. Enter credentials for the Data Source and choose from the options.

5. Click **Finish Setup**.

## Add an existing Data Source to a project

You can add data sources to a project in two ways. First, you can add a data source directly to the project’s data page. Second, a data source can be added automatically when used in the code within the project.

You can add a data source to a project if you have access to that data source and it is set up. This step is not required, but it helps you see which data sources are used in your projects.

If you don’t add a Data Source to a project, you can still use it in your code if you have permission to access it.

1. In your project, go to **Data** > **Data Sources** > **Add a Data Source**.

2. Select an existing Data Source from the list.

3. Click **Add to Project**.

## Use the Domino Data API

After a Data Source is properly configured, use the [Domino Data API](/cloud/reference/api/data_api) to retrieve data without installing drivers or Data Source-specific libraries.

The auto-generated code snippets provided in your workspace are based on the Domino Data API, which supports tabular and file-based Data Sources. The API supports Python and R.

The Data API comes pre-packaged in the Domino Standard Environment (DSE). You can [install the Data API](/cloud/reference/api/data_api/install_the_data_api) in custom environments if needed.

The Data API’s Data Source client uses environment variables available in the workspace to automatically authenticate your identity. You can override this behavior using [custom authentication](/cloud/reference/api/data_api/datasource_usecases/custom_auth).

### Get code snippets

Domino creates code snippets to help you access data sources for your project using the Domino Data API. Code snippets are available for Python and R and customized for tabular and file-based Data Sources. The Data Source must be [added to your project](#add-an-existing-data-source-to-a-project) to enable snippets.

Here’s how to get a code snippet that you can copy and paste into your workspace:

1. In your workspace, go to **Data** > **Data Sources**.

2. Click the copy icon to display language options.

   <img alt="Code snippet button for Python or R" />

3. Select **Python** or **R** to copy the code snippet in the desired language.

4. Paste the copied snippet into your own code, and modify it as needed.

<Note>
  Domino data sources do not support querying nested objects. The workaround is to [`UNNEST`](https://count.co/sql-resources/bigquery-standard-sql/unnest) the object in the SQL query.

  The following is an example `UNNEST` query:

  ```python theme={null}
  res = ds.query("""
  select account_id, t1
  from sample_analytics.transactions
  cross join unnest (transactions)
  as t(t1, t2, t3, t4, t5, t6)
  """)
  ```
</Note>

### Query a tabular store

You can query a tabular store with the following code, assuming a data source named `redshift-test` is configured with valid credentials for the current user:

```python theme={null}
from domino.data_sources import DataSourceClient

# instantiate a client and fetch the datasource instance
redshift = DataSourceClient().get_datasource("redshift-test")

query = """
     SELECT
         firstname,
         lastname,
         age
     FROM
         employees
     LIMIT 1000
 """

 # res is a simple wrapper of the query result
 res = redshift.query(query)
 # to_pandas() loads the result into a pandas dataframe
 df = res.to_pandas()
 # check the first 10 rows
 df.head(10)
```

```r theme={null}
library(DominoDataR)
client <- DominoDataR::datasource_client()

query <- "
  SELECT
    firstname,
    lastname,
    age
  FROM
    employees
  LIMIT 1000"

# table is a https://arrow.apache.org/docs/r/reference/Table-class.html
table <- DominoDataR::query(client, "redshift-test", query)
#> Table
#> 1000 rows x 3 columns
#> $firstname <string not null>
#> $lastname <string not null>
#> $age <int32 not null>
```

### Read/write to an object store

#### List

Get the data source from the client:

```python theme={null}
from domino.data_sources import DataSourceClient

s3_dev = DataSourceClient().get_datasource("s3-dev")
```

```r theme={null}
library(DominoDataR)
client <- DominoDataR::datasource_client()
```

You can list objects available in the data source. You can also specify a prefix:

```python theme={null}
objects = s3_dev.list_objects()

objects_under_path = s3_dev.list_objects("path_prefix")
```

```r theme={null}
keys <- DominoDataR::list_keys(client, "s3-dev")

keys_under_path <- DominoDataR::list_keys(client, "s3-dev", "path_prefix")
```

By default the number of returned objects is limited by the underlying data source. You can specify how many keys you want as an optional parameter:

```python theme={null}
objects = s3_dev.list_objects(page_size = 1500)
```

```r theme={null}
keys = DominoDataR::list_keys(client, "s3-dev", page_size = 1500)
```

#### Read

You can get object content, without having to create object entities, by using the data source API and specifying the `Object` key name:

```python theme={null}
# Get content as binary
content = s3_dev.get("key")

# Download content to file
s3_dev.download_file("key", "./path/to/local/file")

# Download content to file-like object
f = io.BytesIO()
s3_dev.download_fileobj("key", f)
```

```r theme={null}
# Get content as raw vector
content <- DominoDataR::get_object(client, "s3-dev", "key")

# Get content as character vector
content <- DominoDataR::get_object(client, "s3-dev", "key", as = "text")

# Download content to file
DominoDataR::save_object(client, "s3-dev", "key", "./path/to/local/file")
```

You can also get the data source entity content from an object entity (Python only):

```python theme={null}
# Key object
my_key = s3_dev.Object("key")

# Get content as binary
content = my_key.get()

# Download content to file
my_key.download_file("./path/to/local/file")

# Download content to file-like object
f = io.BytesIO()
my_key.download_fileobj(f)
```

#### Write

Similar to the read/get APIs, you can also write data to a specific object key. From the data source:

```python theme={null}
# Put binary content to given object key
s3_dev.put("key", b"content")

# Upload file content to specified object key
s3_dev.upload_file("key", "./path/to/local/file")

# Upload file-like content to specified object key
f = io.BytesIO(b"content")
s3_dev.upload_fileobj("key", f)
```

```r theme={null}
# Put raw or character vector to object key
DominoDataR::put_object(client, "s3-dev", "key", what = "content")

# Upload file content to specified object key
DominoDataR::upload_object(client, "s3-dev", "key", file = "./path/to/local/file")
```

You can also write from the object entity (Python only).

```python theme={null}
# Key object
my_key = s3_dev.Object("key")

# Put content as binary
my_key.put(b"content")

# Upload content from file
my_key.upload_file("./path/to/local/file")

# Upload content from file-like object
f = io.BytesIO()
my_key.upload_fileobj(f)
```

### Write to a local file

#### Parquet

Because Domino uses [PyArrow](https://arrow.apache.org/docs/python/) to serialize and transport data, the query result is easily written to a local parquet file. You can also use pandas as shown in the CSV example.

```python theme={null}
redshift = DataSourceClient().get_datasource("redshift-test")

res = redshift.query("SELECT * FROM wines LIMIT 1000")

# to_parquet() accepts a path or file-like object
# the whole result is loaded and written once
res.to_parquet("./wines_1000.parquet")
```

```r theme={null}
client <- DominoDataR::datasource_client()
table <- DominoDataR::query(client, "redshift-test", "SELECT* FROM wines LIMIT 1000")

# We can use https://arrow.apache.org/docs/r/reference/write_parquet.html since we leverage the arrow library
arrow::write_parquet(table, "./wines_1000.parquet")
```

#### CSV

Because serializing to a CSV is lossy, Domino recommends using the [Pandas.to\_csv](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_csv.html#pandas-dataframe-to-csv) API so you can leverage its multiple options.

```python theme={null}
redshift = DataSourceClient().get_datasource("redshift-test")

res = redshift.query("SELECT * FROM wines LIMIT 1000")

# See Pandas.to_csv documentation for all options
csv_options = {header: True, quotechar: "'"}

res.to_pandas().to_csv("./wines_1000.csv", **csv_options)
```

```r theme={null}
client <- DominoDataR::datasource_client()
table <- DominoDataR::query(client, "redshift-test", "SELECT* FROM wines LIMIT 1000")

# We can use https://arrow.apache.org/docs/r/reference/write_csv_arrow.html since we leverage the arrow library
arrow::write_csv_arrow(table, "./wines_1000.csv")
```

Your identity is verified automatically to make sure you have the right permissions to use the Domino Data API. The system tries to use a [Domino JWT token](/cloud/reference/api/domino-api-authentication). If that’s not available, it uses a [user API key](/cloud/reference/api/domino-api-authentication) instead.

* In a [Domino Nexus deployment](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture), Data Sources can be accessed on both the `Local` and remote [data planes](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace), with the exception of Starburst Trino which can only be accessed on the Local data plane. Data sources may not be usable in every data plane due to network restrictions.

* Connectivity issues may originate anywhere between your Domino deployment and the external data store. Consult your administrator to verify that the Data Source is accessible from your Domino deployment.

## Next steps

* [Create training sets](/cloud/reference/api/data_api/trainingsets_usecases) using the Domino Data API.

* [Run or schedule a job](/cloud/platform-capabilities/core-concepts/jobs) to train your model or update your model’s predictions using the latest data.

* Learn more about [developing](/cloud/platform-capabilities/features/development) and [deploying models](/cloud/platform-capabilities/features/models) in Domino.

* Use [model monitoring](/cloud/platform-capabilities/features/monitoring) to [detect data drift](/cloud/platform-capabilities/features/monitoring/2-data-drift-and-model-quality).


# Vector databases
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/data-source-connectors/vector-databases

Choose a vector database, generate and index embeddings, configure API access, and retrieve similar data for retrieval augmented generation or recommendations.

Vector databases store vector embeddings, which represent data in high-dimensional space, and support the fast similarity searches behind applications such as retrieval augmented generation (RAG), recommender systems, and media recognition. This page outlines how to index, load, and retrieve data from vector databases in Domino.

Vector databases provide:

* Efficient similarity searches: Quickly find the most similar items in large datasets, which is essential for applications like RAG and personalized recommendation systems.

* Scalable infrastructure: Handle large volumes of data and complex query patterns without sacrificing performance.

* Enhanced data handling: Store and manage high-dimensional data effectively, optimizing storage and retrieval operations.

## Set up vector databases

Set up vector databases in Domino with the following steps:

### Step 1: Choose a vector database

Select a vector database that best fits your needs. Domino supports popular vector databases like Pinecone, Qdrant, and others, offering a range of features tailored to different requirements.

### Step 2: Index data

* Prepare your data: Organize your data into a suitable format for vectorization. This might involve preprocessing steps like normalization or tokenization, depending on the nature of your data.

* Generate embeddings: Use a machine learning model to convert your prepared data into vector embeddings. Domino can facilitate this process through its scalable compute resources and integration with machine learning frameworks.

* Load data into the database: Upload the generated embeddings into your chosen vector database. Domino’s job scheduler can automate this process, ensuring data is indexed efficiently and regularly updated as needed.

### Step 3: Integrate a vector database

* Configure access: Set up connectivity between Domino and the vector database, ensuring secure and reliable data transfer.

* Implement API calls: Use Domino’s APIs to query the vector database directly from your applications or during analytical workflows.

### Step 4: Retrieve data

* Execute queries: Perform queries to retrieve data based on similarity or other criteria, integrating these operations within your Domino Workspaces or Runs.

* Utilize results: Use the retrieved data to enhance your AI applications, whether it’s generating responses in a RAG setup, recommending products, or recognizing images or videos.

## Use cases

* RAG: Enhance chatbots and other generative AI applications by providing contextually relevant data at runtime.

* Recommender systems: Improve recommendation accuracy by matching user profiles with products or content based on similarity in vector space.

* Media recognition: Quickly identify and classify media files by comparing embeddings generated from audio or video content.

## Next steps

Find out how to connect to [Pinecone](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-pinecone) and [Qdrant](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-qdrant).


# Create and manage Datasets
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets

Create Domino Datasets through the UI, SDK, or API, then upload, download, rename, delete, share, and refresh their files with scheduled Jobs.

Create Datasets, control who can access them, and manage their files, snapshots, and settings.

A Dataset always reflects the most recent version of the data. For reproducibility, you can create read-only snapshots of your Dataset at any moment in time. Snapshots are associated with the Dataset they version.

## Create a new Dataset

If the Dataset you need is not yet in Domino, you can create it in your Project:

<Tabs>
  <Tab title="In the UI">
    1. In your Project, go to **Data** > **Domino Datasets** > **Create New Dataset**.

    2. Enter a **Dataset Name** and **Description**, then click **Next**.

    3. Enter the users or organizations to give them permission to the Dataset.

       <Tip>
         To give all Project members access to the Dataset, click the **Add all project members** link.
       </Tip>

    4. Specify the user or organization's [role](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-datasets).

    5. Click **Add**.

    6. Repeat steps 3-5 as needed.

       <Tip>
         In the **Current Permissions** area, you can modify the [role](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-datasets) as needed, or click the trash icon to delete permissions.
       </Tip>

    7. Click **Finish**.
  </Tab>

  <Tab title="In `python-domino`">
    You can use the [python-domino library](/cloud/reference/python-sdk/python-domino) to create a Dataset using `datasets_create`:

    ```python theme={null}
    datasets_create(dataset_name, dataset_description)
    ```

    * `dataset_name`: Name of the new Dataset. The name must be unique.
    * `dataset_description`: Description of the Dataset.
  </Tab>

  <Tab title="In the REST API">
    Endpoint:

    ```
    /api/datasetrw/v1/datasets
    ```

    Body:

    ```json theme={null}
    {
        "name": "<new-dataset-name>",
        "description": "<dataset-description>",
        "projectId": "<project-id>",
        "grants": []
    }
    ```

    Example:

    Example request:

    ```python theme={null}
    import requests
    import json

    url = "test14995.test-api.tech/api/datasetrw/v1/datasets"

    payload = json.dumps({
      "name": "test-dataset4",
      "description": "A new dataset",
      "projectId": "649cb3b0dab75c31b6e9daf6",
      "grants": []
    })
    headers = {
      'X-Domino-Api-Key': 'YOUR_API_KEY',
      'Content-Type': 'application/json'
    }

    response = requests.request("POST", url, headers=headers, data=payload)

    print(response.text)
    ```

    Example response:

    ```json theme={null}
    {
        "dataset": {
            "id": "649cb4dedeb75c31b6e7dafc",
            "name": "test-dataset4",
            "description": "A new dataset",
            "projectId": "649cb3b0dab75c31b6e9daf6",
            "createdAt": "2023-06-28T22:31:58.042Z",
            "snapshotIds": [
                "649cb4dedeb75c31b6e7dafb"
            ],
            "tags": {}
        },
        "metadata": {
            "requestId": "956ef589-8f4d-4c6d-b7f3-1c28d0201248",
            "notices": []
        }
    }
    ```
  </Tab>
</Tabs>

### Create a new Dataset from a snapshot

Snapshots are read-only. To iterate on one, create a new Dataset from it, modify the new Dataset, and take a new snapshot. See [Create a new Dataset from a snapshot](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots#create-a-new-dataset-from-a-snapshot) for the UI, CLI, and API steps.

### Upload files to a Dataset

Use the Domino UI to upload up to 50 GB or 50,000 individual files. For larger uploads, use the Domino CLI for your upload.

<Tabs>
  <Tab title="In the UI">
    1. In your Project, go to **Data** > **Domino Datasets**.

    2. Open the Dataset to which you want to add files, then click **Upload files**:

       <img alt="Upload files to a Dataset" />

       You can browse your local filesystem or drag and drop files to upload.

       <Tip>
         To preserve the filesystem structure of your uploads, use the drag-and-drop option. You can pause and resume the upload as needed.
       </Tip>

       <Note>
         Uploading to a folder can fail if you do not have permissions to write to it. For example, a folder created from a Workspace might not be writable through the UI without appropriate changes to permissions.
       </Note>
  </Tab>

  <Tab title="In the CLI">
    Upload all the files in a folder from your local machine to an existing dataset.

    <Warning>
      This command overwrites existing files unless you use the `‑‑fileUploadSetting` option.
    </Warning>

    Syntax:

    ```shell theme={null}
    domino [--fileUploadSetting Ignore|Overwrite|Rename] [--targetRelativePath <destination path>] <project-owner>/<project-name>/<dataset-name> <folder path>
    ```

    The optional `fileUploadSetting` flag handles path collisions:

    * `Ignore`: If a file already exists in the Dataset, ignore the new file.

    * `Overwrite`: If a file already exists in the Dataset, overwrite the existing file with the new file.

    * `Rename`: If a file already exists in the Dataset, append `_1` to the uploaded filename. For example, if `/Users/myUser/data/file.txt` already exists then the newly uploaded file becomes `/Users/myUser/data/file_1.txt`.

    The optional `targetRelativePath` flag enables uploading to a subdirectory in the Dataset:

    * `<destination path>`: This is the path, relative to the root directory of the Dataset, where files are uploaded. Note that other users with access to the Dataset can alter the file structure, which may require you to modify the destination path accordingly.

      If `targetRelativePath` is not specified, files are uploaded to the root directory of the Dataset.

    <Note>
      `fileUploadSetting`, `targetRelativePath`, and their values are case-sensitive.
    </Note>

    Example:

    ```shell theme={null}
    domino upload-dataset --fileUploadSetting Overwrite --targetRelativePath Experiments/Run1/Analysis jsmith/global-predictions/global-data /Users/myUser/data
    ```

    If you don't have the CLI installed, see [Install the Domino Command Line (CLI)](/cloud/reference/cli/install-the-cli) for instructions.
  </Tab>
</Tabs>

### Download files from Datasets

Use the Domino UI or CLI to download files and folders from your Projects.

<Tabs>
  <Tab title="In the UI">
    1. In your Project, go to **Data** > **Domino Datasets**.

    2. Open the Dataset you want to download files from.

    3. Navigate to the directory where your files are located.

    4. Either:

       * Click the vertical dots next to the file or folder you want to download. Click **Download**. If you are downloading a folder, the downloaded file is a ZIP or TAR archive, which can be toggled via the `com.cerebro.domino.dataset.batchDownloadArchiveFormat` configuration record key. Otherwise, the file downloads directly.
       * Click the checkboxes to the left of the files and directories you want to download, and then click **Download Selected Items**. If you are downloading a folder or multiple files, the downloaded file is a ZIP file (default) or TAR archive, which can be toggled via the `com.cerebro.domino.dataset.batchDownloadArchiveFormat` configuration record key. Otherwise, the file downloads directly.
  </Tab>

  <Tab title="In the CLI">
    `download` downloads the latest copy of your files from the cloud into your current project folder. If you have made changes that conflict with changes in the cloud, you see both versions of the conflicting file side-by-side.

    There are two reasons files in the cloud might change: first, your collaborators on a Project might make changes; second, you might have executed a run that produced new output files.

    Example

    ```shell theme={null}
    domino download
    ```

    To download the output files of the given run, please see the [Download files with the CLI](/cloud/reference/cli/download-files-with-the-cli) page.

    If you don't have the CLI installed, see [Install the Domino Command Line (CLI)](/cloud/reference/cli/install-the-cli) for instructions.
  </Tab>
</Tabs>

## Modify Datasets

You can always modify the contents of a Dataset or rename the Dataset.

<Tip>
  Always create a snapshot before modifying the contents of a Dataset so that you can always return to the previous version of the data.
</Tip>

### Rename a Dataset

You can change the name of a Dataset.

1. In the navigation pane, click **Data**.

2. Click **Domino Datasets**.

3. Click the Dataset to rename.

4. Go to **More Actions > Rename Dataset**.

5. Enter a **New Name** and click **Rename**.

### Delete a Dataset

If you no longer need the entire Dataset, you can mark it for deletion. When you mark a Dataset for deletion, it removes the Dataset and its associated snapshots from the originating Project and from all projects that it was shared with. Domino executions cannot use the Dataset. A Domino administrator must perform the final deletion.

1. In the navigation pane, click **Data**.

2. Click **Domino Datasets**.

3. Click the name of the Dataset to delete.

4. Go to **More Actions** > Click **Delete Dataset**.

5. Click **Delete Dataset** to confirm that you want to mark the Dataset for deletion. Your administrator must permanently [delete the Dataset](/cloud/admin/data-administration/datasets#delete-datasets-and-snapshots).

### Add or remove files

You can add or delete files in a Dataset using the Domino UI. With the CLI, you can add all the files in a folder to a Dataset.

<Tabs>
  <Tab title="In the UI">
    In the navigation pane, click **Data**, then click the name of the Dataset to change.

    * To add files, click **Upload files**.

    * To delete files, select the files to delete, then click **Delete Selected Items**.

    * To rename the Dataset, click **Rename Dataset**, enter the new name, then click **Rename**.

    Before deleting a file with a special character like a backslash () in its name, you need to rename it first. You can use a tilde (\~) or colon (:) anywhere in a filename, except at the beginning. If the file that you want to delete has a tilde or colon at the beginning of its name, rename it.
  </Tab>

  <Tab title="In the CLI">
    To add all the files in a folder, use the same `domino upload-dataset` command and options described in [Upload files to a Dataset](#upload-files-to-a-dataset).
  </Tab>
</Tabs>

### Rename files and folders

You can change the name of the latest version of a file or folder in a Dataset. Domino also does not rename files or folders in snapshots.

<Warning>
  You must update references to the original file or folder. If you don't, your Project might not work. For example, you might see inconsistencies in text files and documentation.
</Warning>

1. Go to a Project that uses a Dataset.

2. In the navigation pane, click **Data**.

3. To rename the file or folder, go to the end of the row and click the three vertical dots.

4. Click **Rename**.

5. In the Rename window, enter the **New Name** and click **Rename**.

## Schedule Jobs to update a Dataset

If you have data in an external source from which you want to periodically fetch and load into Domino, you can set up [scheduled jobs](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job) to write to Datasets.

Suppose you have data stored in an [external Data Source](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/2-connect-a-data-source) that is periodically updated. If you wanted to fetch the latest state of that file once a week and load it into a Domino Dataset, you could set up a scheduled Run:

1. Create a Dataset to store the data from the external source.

2. Write a script that fetches the data and writes it to the Dataset.

3. Create a scheduled Job to run your script with the new Dataset configuration.

   The following is a detailed example showing how to fetch a large, dynamic data file from a [private S3 bucket](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-generic-s3) with a scheduled Run once a week.

4. Create a Dataset to hold the file. This example shows the Dataset named `fetched-from-s3`.

   <img alt="Best practices to fetch Datasets from S3" />

   For this example, assume the S3 bucket is named `my_bucket` and the file you want is named `some_data.csv`. You can set up your script like this:

   fetch-data.py

   ```shell theme={null}
   import boto3
   import io

   # Create new S3 client
   client = boto3.client('s3')

   # Download some_data.csv from my_bucket and write to latest-S3 output mount
   file = client.download_file('my_bucket',
                               'some_data.csv',
                               '/domino/datasets/fetched-from-s3/some_data.csv')
   ```

5. Set up a [scheduled Job](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job) that executes this script once a week with the correct Dataset configuration.

## Next steps

* [Find a Dataset's path](/cloud/platform-capabilities/core-concepts/data/datasets/use-datasets-and-snapshots)

* [Create and manage Dataset snapshots](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots)

* [Manage large Datasets](/cloud/platform-capabilities/core-concepts/data/datasets/manage-large-data)

* [Share a Dataset](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-datasets)


# Work with Domino Datasets
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/datasets/index

Use Domino Datasets as versioned filesystem storage, share and mount them across Projects, and preserve reproducible data with snapshots.

Domino Datasets provide high-performance, versioned, and structured filesystem storage in Domino. You can use Datasets to build multiple curated collections of data in one project and share them with your collaborators to use in their projects. Likewise, you can mount Datasets from other projects in your own project if they are shared with you. You can modify the contents of a Dataset through the Domino application or through executions.

A Dataset always reflects the most recent version of the data. For reproducibility, you can create read-only snapshots of your Dataset at any moment in time. Snapshots are useful when you need to ensure that the data you’re working on does not change.

Datasets mount into Workspaces, Jobs, Launchers, and Apps, and your code reads and writes them as ordinary directories: `/domino/datasets/local/<name>` in DFS Projects, `/mnt/data/<name>` in Git-based Projects (see [Use Datasets and snapshots](/cloud/platform-capabilities/core-concepts/data/datasets/use-datasets-and-snapshots) for the full path scheme). Snapshots count toward your storage, so delete the ones you no longer need. Sharing is zero-copy: every Project mounts the same files rather than receiving a copy, so a Dataset [shared read-only](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-datasets) works as a controlled distribution mechanism for curated data products.

The topics in this section show you how to use Datasets in your projects.

[Create and manage Datasets](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets)\
Find shared Datasets and make them part of your project.

[Use Datasets and Snapshots](/cloud/platform-capabilities/core-concepts/data/datasets/use-datasets-and-snapshots)\
Use the Dataset’s path to access the Dataset from your code.

[Version data with Snapshots](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots)\
Create a read-only, immutable version of the Dataset. Add tags to create a friendly, stable path to snapshots that hold the desired state for your data.

[Work with local data](/cloud/platform-capabilities/core-concepts/data/datasets/local-data)\
Store data in a Domino Dataset and reserve your project files for the scripts and documents that you want to work with locally.

[Manage large Datasets](/cloud/platform-capabilities/core-concepts/data/datasets/manage-large-data)\
Use Domino Datasets or Project data compression to handle large data.


# Work with local data
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/datasets/local-data

Move large data files from Project files into Domino Datasets to speed local CLI download and sync operations while keeping code paths current.

If you use the [Domino CLI](/cloud/reference/cli/install-the-cli) to work with projects on your local machine, you might find that storing large data files slows your [download](/cloud/reference/cli/cli-reference#download-the-latest-version-of-your-files) and [sync](/cloud/reference/cli/cli-reference#synchronize-the-files-on-your-computer-with-the-server) operations, and fills up a lot of your local disk storage. To prevent this, store data in a Domino Dataset, and reserve your project files for the scripts and documents that you want to work with locally.

<Note>
  In a [Domino Nexus deployment](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture), CLI access to Datasets is only available in the `Local` [data plane](/cloud/platform-capabilities/core-concepts/data-plane).
</Note>

To simplify your local workflow:

1. [Create a Dataset](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets) in your project, and write your large data files to it.

2. After the files have been written to the Dataset, remove them from your project files.

3. [Fetch](/cloud/reference/cli/cli-reference#get-a-project-from-the-server-onto-your-computer-for-the-first-time) a clean, lighter-weight copy of your project.

4. Update your code to reference your data files in their new location, at:

   `/domino/datasets/local/<dataset-name>/`

5. When everything is working properly, delete copies of the project that have the large data files in them from your local machine.

## Next steps

* [Get the path to a snapshot](/cloud/platform-capabilities/core-concepts/data/datasets/use-datasets-and-snapshots)

* Learn [how Domino handles large Datasets](/cloud/platform-capabilities/core-concepts/data/datasets/manage-large-data)


# How Domino handles large Datasets
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/datasets/manage-large-data

Choose Domino Datasets for large static data or Project data compression for dynamic data, then monitor storage quotas and usage.

When you start an execution, Domino copies your project files to the executor that is hosting the execution. After every execution in Domino, by default, Domino tries to write all files in the working directory back to the project as a new revision. When working with large volumes of data, this presents two potential problems:

* The number of files that are written back to the project might exceed the configurable limit. By default, the file limit for Domino project files is 10,000 files.

* The time required for the write process is proportional to the size of the data. It can take a long time if the size of the data is large.

The following table shows the recommended solutions for these problems. They are described in further detail after the table.

| Case | Data size | # of files | Static / Dynamic | Solution |
| - | - | - | - | - |
| Large volume of **static** data | Unlimited | Unlimited | Static | Domino Datasets |
| Large volume of **dynamic** data | Up to 300 GB | Unlimited | Dynamic | Project Data Compression |

"Unlimited" means Domino sets no cap: the ceiling is whatever storage backs Datasets in your deployment, so check that provider's documentation for the figure that applies to you. Cost still grows with the data and snapshots you keep, so delete what you no longer need. Beyond a few terabytes, consider keeping bulk data in object storage such as Amazon S3 through a [Data Source connector](/cloud/platform-capabilities/core-concepts/data/data-source-connectors). Administrators can also cap per-user storage, as described in [Dataset quotas and limits](#dataset-quotas-and-limits).

## Domino Datasets

When working on image recognition or image classification deep learning projects, you often need a training Dataset of thousands of images. The total Dataset size can easily become tens of GB. For these types of projects, the initial training also uses a static Dataset. The data is not constantly being changed or updated. Furthermore, the actual data that is used is normally processed into a single large tensor.

Store your processed data in a Domino Dataset. Datasets can be mounted by other Domino projects, where they are attached as a read-only network filesystem to that project’s runs and workspaces.

## Project data compression

Sometimes, you must work with logs as raw text files. Typically, new log files are constantly being updated, so your Dataset is dynamic. You might encounter both problems described previously at the same time:

1. The number of files are over the 10k limit.

2. There are long times to prepare and sync data.

Domino recommends that you store these files in a compressed format. If you need the files to be in an uncompressed format during your Run, you can use Domino Compute Environments to prepare the files. In the pre-run script, you can uncompress your files:

```shell theme={null}
tar -xvzf many_files_compressed.tar.gz
```

Then in the post-run script, you can re-compress the directory:

```shell theme={null}
tar -cvzf many_files_compressed.tar.gz /path/to/directory-or-file
```

If your compressed file is still large, the time to prepare and sync might still be long, depending on how large your compressed file is. Consider storing these files in a [Domino Dataset](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets) to minimize the time to copy.

## Dataset quotas and limits

Admins can [set quotas and limits on Dataset storage](/cloud/admin/data-administration/datasets#set-limits-on-dataset-usage) in configuration records. Contact your admin to configure quotas.

As you approach your quota limit, you receive notifications, emails, and warnings on Dataset pages.

Your Dataset storage usage is the sum of the size of all active Datasets that you own. If a Dataset has multiple owners, then the size of that Dataset counts towards the storage usage of each owner. For more information, see [Dataset roles](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-datasets#dataset-roles).

## Next steps

* [Get the path to a snapshot](/cloud/platform-capabilities/core-concepts/data/datasets/use-datasets-and-snapshots)

* Learn about [Dataset best practices](/cloud/platform-capabilities/core-concepts/data/datasets/local-data)


# Use Datasets and snapshots
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/datasets/use-datasets-and-snapshots

Mount shared Domino Datasets, preview file formats, choose snapshots for executions, and use the correct paths in Domino File System or Git-based Projects.

Domino Datasets provide easy-to-use, fast, and organized storage for files in the Domino platform. You can create different sets of data within a project and share them with your teammates.

If you mount a Dataset, others can use it in their own projects. If someone shares their Datasets with you, you can also mount them for use in your own project.

## Mount a Dataset

To mount a Dataset, you’ll need the appropriate [Dataset role](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-datasets#dataset-roles) and you must be [an owner or contributor](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects) on the Project to which you want to mount the Dataset.

1. Launch Domino and open the **Project** that you want to work with.

2. From the left navigation pane, click **Data** > **Datasets**.

3. On the **Domino Datasets** page, click the **Mount Shared Dataset** button.

4. Choose the Dataset that you want to mount from the **Dataset Name** menu, then click **Mount Dataset**.

The Dataset that you selected should now appear in the **Shared Datasets** list. You can find the directory path for that Dataset under **Name**.

<Note>
  Datasets (both imported and standard) are mounted as read-write (RW) or read-only (RO) based on the role of the user running the execution.
</Note>

## Unmount a shared Dataset

Unmounting a shared Dataset does not remove it from existing executions until the execution completes. However, the Dataset is not available for new executions in this Project.

To unmount a Dataset, you’ll need the appropriate [Dataset role](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-datasets#dataset-roles) and you must be [an owner or contributor](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects) on the Project from which you want to unmount the Dataset.

1. Open the **Project** that you want to unmount the Dataset from.

2. From the left navigation pane, click **Data** > **Datasets**.

3. Look for the Dataset you want to unmount in the **Domino Dataset** list.

4. At the end of its row, click the three vertical dots and click **Unmount**.

## Render a Dataset

Domino supports tabulated previews for a number of file formats to streamline your data analysis and workflows. This minimizes downloading files and artifacts from Domino, while keeping sensitive data inside the Domino platform.

As a prerequisite for viewing in a tabulated format, make sure that the **Define.xml** file and its associated **style sheet (.xsl file)** are in the same directory. There are a couple of ways to view these:

* From within a **Domino Dataset**: Double-click the **Define.xml** file to render it in a readable format.

* From within Domino artifacts:

  * View **Styled XML**: click **Show Rendered XML** to render it in a readable format.

These are the supported file types for rendering:

<table>
  <thead>
    <tr>
      <th>File formats</th>
      <th>File types</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td><p><strong>Microsoft Office</strong></p></td>
      <td><ul> <li><p>.doc, .docx, .ppt, .pptx, .xls, .xlsx</p></li> </ul></td>
    </tr>

    <tr>
      <td><p><strong>OpenDocument</strong></p></td>
      <td><ul> <li><p>.odg, .odp, .ods, .odt</p></li> </ul></td>
    </tr>

    <tr>
      <td><p><strong>Other</strong></p></td>
      <td><ul> <li><p>.csv, .pdf, .rtf, .xml</p></li> </ul></td>
    </tr>

    <tr>
      <td><p><strong>SAS</strong></p></td>
      <td><ul> <li><p>.sas7bdat</p></li> </ul></td>
    </tr>
  </tbody>
</table>

## Choose which Datasets and snapshots to mount

By default, every Dataset attached to a Project mounts into your executions with all of its snapshots at their versioned paths (see [Understand Dataset paths](#understand-dataset-paths) below). You can pin any one of them to a single snapshot instead, for the length of that execution.

Do it on the **Data Plane & Hardware** tab when you launch a Workspace or Job, or when you define a scheduled Job. Domino lists each mounted Dataset and NetApp Volume there in a table with **Storage** and **Snapshot** columns, and the **Snapshot** column is where you pick. A pinned snapshot mounts read-only at the path the live Dataset normally uses, for example `/domino/datasets/local/<name>`, so your existing code paths keep working.

[Choose a snapshot per asset at launch](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots#choose-a-snapshot-per-asset-at-launch) covers every option, how long the selection lasts, and what changes about the asset's other mounts.

## Find a Dataset path to access it in your code

To access a Dataset or a snapshot in your code, you need to know its path.

There are several ways to find the path to a Dataset or any of its snapshots in Domino.

* From a Workspace:

  * [Before launching a Workspace](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace): from the left navigation, click **Workspaces** > **Settings**.

  * In a [running Workspace](/cloud/platform-capabilities/core-concepts/workspaces/manage-workspaces): from the left navigation, click **Data** > **Datasets**.

* From a Job:

  * When [launching a Job](/cloud/platform-capabilities/core-concepts/jobs/create-and-run-jobs): from the left navigation, click **Jobs**, choose the job from the list, then click **Details** > **Data**.

  * In a running Job, click **Details** > **Data**.

## Understand Dataset paths

Domino executions (workspaces, jobs, apps, and launchers) automatically make Datasets and their associated snapshots from a Project available.

The following configuration demonstrates the conventions followed by a predefined path:

* Dataset called `clapton` (local to the project)

  * Snapshot 1 (tagged with `tag1`)

  * Snapshot 2 (not tagged)

* Dataset called `mingus` (local to project)

  * Snapshot 1 (tagged with `tag2`)

  * Snapshot 2 (not tagged)

* Dataset called `ella` (shared from another project)

  * Snapshot 1 (tagged with `tag3`)

  * Snapshot 2 (not tagged)

* Dataset called `davis` (shared from another project)

  * Snapshot 1 (tagged with `tag4`)

  * Snapshot 2 (not tagged)

### Dataset paths in DFS projects

The mounting paths in DFS projects behave according to the following scheme:

**Dataset in the project:**

* `/domino/datasets/local/<name>` - mounted as RW

* `/domino/datasets/local/snapshots/<name>/<tag>` - mounted as RO

* `/domino/datasets/local/snapshots/<name>/1` - mounted as RO

* `/domino/datasets/local/snapshots/<name>/2` - mounted as RO

**Dataset imported from a different project:**

* `/domino/datasets/<name>` - mounted as RO

* `/domino/datasets/snapshots/<name>/<tag>` - mounted as RO

* `/domino/datasets/snapshots/<name>/1` - mounted as RO

* `/domino/datasets/snapshots/<name>/2` - mounted as RO

For a Domino File System (DFS) project, Datasets and snapshots are available in the following hierarchy:

```console theme={null}
/domino
   |--/datasets
      |--/local               <== Local datasets and snapshots
         |--/clapton          <== Read-write dataset for owner and editor, read-only for reader
         |--/mingus           <== Read-write dataset for owner and editor, read-only for reader
         |--/snapshots        <== Snapshot folder organized by dataset
            |--/clapton       <== Read-write for owner and editor, read-only for reader
               |--/tag1          <== Mounted under latest tag
               |--/1             <== Always mounted under the snapshot number
               |--/2
            |--/mingus
               |--/tag2
               |--/1
               |--/2
      |--/ella                <== Read-write shared dataset for owner and editor, Read-only for reader
      |--/davis               <== Read-write shared dataset for owner and editor, Read-only for reader
      |--/snapshots           <== Shared datasets snapshots organized by dataset
         |--/ella             <== Read-write for owner and editor, read-only for reader
            |--/tag3          <== Mounted under latest tag
            |--/1             <== Always mounted under the snapshot number
            |--/2
         |--/davis
            |--/tag4
            |--/1
            |--/2
```

### Dataset paths in Git-based projects

The mounting paths in Git-based projects behave according to the following scheme:

**Dataset in the project:**

* `/mnt/data/<name>` - mounted as RW

* `/mnt/data/snapshots/<name>/<tag>` - mounted as RO

* `/mnt/data/snapshots/<name>/1` - mounted as RO

* `/mnt/data/snapshots/<name>/2` - mounted as RO

**Dataset imported from a different project:**

* `/mnt/imported/data/<name>` - mounted as RO

* `/mnt/imported/data/snapshots/<name>/<tag>` - mounted as RO

* `/mnt/imported/data/snapshots/<name>/1` - mounted as RO

* `/mnt/imported/data/snapshots/<name>/2` - mounted as RO

For Git-based Projects, the Datasets and snapshots are available in the following hierarchy:

```console theme={null}
/mnt
   |--/data                  <== Local datasets and snapshots
     |--/clapton             <== Read-write dataset for owner and editor, read-only for reader
     |--/mingus              <== Read-write dataset for owner and editor, read-only for reader
     |--/snapshots           <== Snapshot folder organized by dataset
        |--/clapton          <== Read-write for owner and editor, read-only for reader
           |--/tag1          <== Mounted under latest tag
           |--/1             <== Always mounted under the snapshot number
           |--/2
        |--/mingus
           |--/tag2
           |--/1
           |--/2
   |--/imported
     |--/data
        |--/ella             <== Read-write shared dataset for owner and editor, read-only for reader
        |--/davis            <== Read-write shared dataset for owner and editor, read-only for reader
        |--/snapshots        <== Shared dataset snapshots organized by dataset
           |--/ella          <== Read-write for owner and editor, read-only for reader
              |--/tag3       <== Mounted under latest tag
              |--/1          <== Always mounted under the snapshot number
              |--/2
           |--/davis
              |--/tag4
              |--/1
              |--/2
```

## Next steps

* [Create and manage Dataset Snapshots](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots) for data reproducibility.

* Learn about [Dataset best practices](/cloud/platform-capabilities/core-concepts/data/datasets/local-data).


# Version data with Snapshots
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots

Create, tag, mount, pin, download, and delete snapshots for Domino Datasets and NetApp Volumes, or build a new Dataset from a snapshot.

Domino lets you create snapshots of both NetApp Volumes and Domino Datasets to support reproducibility, version control, and collaboration. A snapshot is a read-only, immutable record of your data at a specific point in time. You can tag snapshots, download them, or use them to create versions. You can also add commit messages to NetApp Volumes only.

When you want to reproduce a training experiment, you can version a NetApp Volume or Domino Dataset so that you can return to a specific version used in the past.

To do this with Domino:

* Create a snapshot to create versions of a NetApp Volume or Domino Dataset.

* Use a naming convention and a folder hierarchy to organize data your way in the read/write portions of a NetApp Volume or Dataset.

## Snapshotting: NetApp Volumes vs. Domino Datasets

| Feature | NetApp Volumes | Domino Datasets |
| - | - | - |
| **Snapshot speed** | Nearly instant, regardless of size (\~3 sec for 100+ GB) | Time scales with size (seconds to hours) |
| **Storage efficiency** | Snapshots initially consume no extra space (redirect-on-write) | Snapshots duplicate physical data at snapshot time |
| **Mounting in executions** | Mounted like datasets; supports tag-based mounting | Mounted like volumes; supports tag-based mounting |
| **Tag support** | Yes | Yes |
| **Commit messages** | Yes | No - not supported |
| **Writable after snapshot** | No - read-only; cannot clone snapshot into a new volume | Yes - can create a **new** Dataset from a snapshot; original snapshot remains read-only |

Use **NetApp Volumes** when you need fast, large-scale snapshotting or integration with external storage systems.

Use **Datasets** if you don't have NetApp Volumes set up, and you want to clone or branch datasets more flexibly within Domino.

## Create a snapshot

You can take a snapshot from the overview page of a NetApp Volume or Dataset in the Domino UI. A snapshot captures the current state of the data in a read-only, versioned format.

**Caution:** You should never modify files while a snapshot is in progress.

<Tabs>
  <Tab title="In the UI">
    1. Go to the **NetApp Volumes** or **Datasets** tab in your project.

    2. Select the NetApp Volume or Dataset you want to snapshot.

    3. Click **Take Snapshot** and choose one of the following:

       * **Include all files**: snapshots the entire contents.

       * **Include only selected files** allows a partial snapshot.

    4. Optionally, add a commit message (NetApp Volumes) or a tag name for the snapshot. You can add multiple tags for NetApp Volumes.

    5. Click **Confirm** to begin.
  </Tab>

  <Tab title="In the CLI (Datasets only)">
    Syntax:

    ```shell theme={null}
    domino create-snapshot <project-owner>/<project-name>/<dataset-name>
    ```
  </Tab>

  <Tab title="In the REST API">
    Domino NetApp Volume Apis

    /snapshots/\{id} in the [Domino NetApp Volume APIs documentation](/cloud/reference/api/domino-volumes-api) has details on how to do this.

    Endpoint:

    ```
    /api/datasetrw/v1/datasets/<dataset-id>/snapshots
    ```

    Body:

    ```json theme={null}
    {
      "relativeFilePaths": [
        "string"
      ]
    }
    ```

    Example: Snapshot all files in the Dataset

    Request:

    ```python theme={null}
    import requests
    import json

    url = "test14995.test-api.tech/api/datasetrw/v1/datasets/649cb4dedeb75c31b6e7dafc/snapshots"

    payload = json.dumps({
      "relativeFilePaths": [
        "/*"
      ]
    })
    headers = {
      'X-Domino-Api-Key': 'YOUR_API_KEY',
      'Content-Type': 'application/json'
    }

    response = requests.request("POST", url, headers=headers, data=payload)

    print(response.text)
    ```

    Response:

    ```json theme={null}
    {
        "snapshot": {
            "id": "64935f78867ba25a6903c6c4",
            "datasetId": "6488daa88548661d66844e23",
            "creatorId": "64876299e4941004222e2b93",
            "createdAt": "2023-06-21T20:37:12.95Z",
            "status": "pending"
        },
        "metadata": {
            "requestId": "adc8ecc5-9887-46ef-a38c-5569311bf4cd",
            "notices": []
        }
    }
    ```
  </Tab>
</Tabs>

## Commit messages (NetApp Volumes only)

A commit message is a short, optional description attached to a snapshot. This description is also editable. It's useful for recording the purpose, contents, or context of the snapshot, especially when creating many versions over time.

Examples:

* "Cleaned version with NA values removed"

* "Post-labeling data for training 3.2"

* "Before joining with demographic.csv"

Commit messages help:

* Track what changed

* Clarify the purpose without editing the Dataset or Volume description

* Improve transparency in team workflows

## Add tags to a snapshot

Tags are human-readable labels that provide a stable, friendly path to a specific snapshot when you mount it in an execution. They let you reference a named snapshot in code or across projects, even if the tag later moves to a different snapshot.

* Tags can be created or removed by NetApp Volume or Dataset owners or editors.

* Tags can be reassigned to newer snapshots as needed.

* Only the **most recently added tag** is used for automatic mounting in executions.

To add or remove a tag:

1. Go to the **Snapshots** section for the Dataset or Volume.

2. Select the snapshot.

3. Click **+Tag Snapshot**, enter a name, and click **Add**.

4. To remove a tag, click the **X** next to the tag name.

## Snapshot mounting in executions

If a snapshot has a tag, Domino automatically mounts it in any execution that uses the NetApp Volume or Dataset. If a snapshot has multiple tags, only the most recently added one is mounted.

This mount path is in addition to the default mount, and it depends on the data and project types:

NetApp Volumes

* **Git-based projects** - `/mnt/netapp-volumes/snapshot-tags/<name>/<tag-name>`

* **Regular projects** - `/domino/netapp-volumes/snapshot-tags/<name>/<tag-name>`

Datasets

* **Git-based projects** - `/mnt/data/snapshots/<name>/<tag-name>`

* **Regular projects** - `/domino/datasets/local/snapshots/<name>/<tag-name>`

Datasets imported from another project use a different prefix. [Understand Dataset paths](/cloud/platform-capabilities/core-concepts/data/datasets/use-datasets-and-snapshots#understand-dataset-paths) lists every form.

Pinning an asset to one snapshot changes this: a pinned Dataset gets no tag mount, while a pinned NetApp Volume keeps its tag mounts. [Choose a snapshot per asset at launch](#choose-a-snapshot-per-asset-at-launch) has the details.

## Choose a snapshot per asset at launch

When you launch a Workspace or Job, or define a scheduled Job, you can pin each mounted NetApp Volume or Dataset to a single snapshot. Pin a snapshot to work against known-good data instead of the live version, without moving tags.

1. Start a new Workspace or Job from your Project, or create a scheduled Job.

2. Open the **Data Plane & Hardware** tab. Domino lists each mounted NetApp Volume and Dataset in a table with **Storage** and **Snapshot** columns.

3. In the **Snapshot** column for an asset, choose one of:

   * **All snapshots**, the default: mounts the asset the standard way, with its live data, every snapshot under its own snapshot number, and any tag mounts.

   * A **specific snapshot**, listed as `Snapshot #<version> - <date>`: mounts that one version, read-only.

4. Repeat for any other mounted assets, then launch or save.

The picker has no tag option. Domino mounts tagged snapshots automatically, as described in [Snapshot mounting in executions](#snapshot-mounting-in-executions), except for a Dataset you pin.

A pinned snapshot takes over the path the asset normally mounts at, read-only, so there is no separate `snapshots/` path to read it from. Pinning also changes which other mounts the asset gets:

| Asset | Pinned snapshot mounts at | What else changes |
| :- | :- | :- |
| Dataset | `/domino/datasets/local/<name>`, or `/mnt/data/<name>` in a Git-based project, the same path the live data normally uses | Domino collapses the version and tag paths into it, so nothing mounts under `snapshots/<name>/`, neither the numbered versions nor the tag |
| NetApp Volume | `/domino/netapp-volumes/<name>`, or `/mnt/netapp-volumes/<name>` in a Git-based project | The read/write mount and the `snapshots/<name>` directory are disabled. Tag mounts under `snapshot-tags/<name>/<tag-name>` stay |

How long the selection lasts depends on what you launch:

| Execution type | The selection applies to |
| :- | :- |
| Workspace | Every restart of that Workspace, until you change it |
| One-off Job | That run only |
| Scheduled Job | Every run the schedule triggers |

## Create a new Dataset from a snapshot

You can create as many snapshots as you need, but existing snapshots are read-only.

**Note:** You can't create a new NetApp Volume directly from a snapshot, but you can copy a volume for use in Project Templates, which includes its existing snapshots.

You can create a new Dataset from a snapshot, modify it, and take a new snapshot:

<Tabs>
  <Tab title="In the UI">
    1. Go to the existing snapshot.

    2. Click **Copy to New Dataset**.

    3. Complete the fields as needed.

    4. Click **Upload files** to add files to the Dataset.

    5. Click **Take Snapshot** > **Include all files**.
  </Tab>

  <Tab title="In the CLI">
    Syntax:

    ```shell theme={null}
    domino create-dataset-from-snapshot <project-owner>/<project-name>/<dataset-name> <snapshot-number> <new-dataset-name>
    ```
  </Tab>

  <Tab title="In the API">
    Endpoint:

    ```
    /api/datasetrw/v1/datasets
    ```

    Body:

    ```json theme={null}
    {
        "name": "test-dataset5",
        "description": "A new dataset based on a snapshot",
        "grants": [],
        "snapshotId": "649cbe38fe7c5443a9cb1b62"
    }
    ```

    Example: Create a new Dataset from a snapshot

    Example request:

    ```python theme={null}
    import requests
    import json

    url = "test14995.test-api.tech/api/datasetrw/v1/datasets"

    payload = json.dumps({
      "name": "test-dataset5",
      "description": "A new dataset based on a snapshot",
      "grants": [],
      "snapshotId": "649cbe38fa7c5443a7cb1b62"
    })
    headers = {
      'X-Domino-Api-Key': 'YOUR_API_KEY',
      'Content-Type': 'application/json'
    }

    response = requests.request("POST", url, headers=headers, data=payload)

    print(response.text)
    ```

    Example response:

    ```json theme={null}
    {
        "dataset": {
            "id": "649cb689deb75c31b6e7dafe",
            "name": "test-dataset5",
            "description": "A new dataset based on a snapshot",
            "projectId": "649cb2b0deb75c31b6e7daf6",
            "createdAt": "2023-06-28T22:39:05.788Z",
            "snapshotIds": [
                "649cb689deb75c31b6e7dafd"
            ],
            "tags": {}
        },
        "metadata": {
            "requestId": "3d87b111-8052-4965-9044-0531ba49a93a",
            "notices": []
        }
    }
    ```
  </Tab>
</Tabs>

## Download a snapshot

You can download a snapshot:

1. Open the overview page for the Dataset or Volume.

2. In the **Snapshots** list, select the snapshot to download.

3. Click **Download Snapshot** to begin downloading.

   * If the selected snapshot contains a folder or multiple files, it downloads as a `.zip` file (default) or `.tar` archive, depending on your system setting.

<img alt="Download Dataset snapshot button" />

## Delete a snapshot

Deletion behavior depends on the volume type and configuration.

### Delete a NetApp Volume snapshot

Admins can directly delete snapshots for both NetApp Volumes and Datasets from the Admin UI. [Delete a NetApp Volume](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/add-remove-netapp-volumes) has information about deleting NetApp Volumes.

### Delete a Dataset snapshot

Marked snapshots are no longer mounted in executions and are flagged for admin removal. You can restore a snapshot if it is marked for deletion. They are not fully deleted until removed by a Domino admin.

1. Open the NetApp Volume or Dataset overview page.

2. Go to the **Snapshots** section.

3. Select the snapshot to delete.

4. Click **Mark Snapshot for Deletion**, then confirm.

## Next steps

* [Get the path to a snapshot](/cloud/platform-capabilities/core-concepts/data/datasets/use-datasets-and-snapshots)

* Learn about [Dataset best practices](/cloud/platform-capabilities/core-concepts/data/datasets/local-data)


# External Data Volumes (EDVs)
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/external-volumes/index

Use External Data Volumes to mount Network File Systems, AWS EFS, or Windows Share storage in Projects and understand execution support and limitations.

External Data Volumes (EDVs) are volumes from network-attached storage systems that are mounted to the Domino system. Domino can register EDVs and attach them to projects. When an EDV is attached to a project, Domino mounts its file system when running code in that project (such as in a workspace or a job).

Once an EDV is [registered by a Domino admin](/cloud/admin/data-administration/external-data/external-data-volumes#register-external-data-volumes), Domino automatically mounts it and you can [add it to your project](/cloud/platform-capabilities/core-concepts/data/external-volumes/use-a-mounted-volume).

<img alt="EDV mounted in Domino" />

<Note>
  [Remote data planes](/cloud/platform-capabilities/core-concepts/data-plane) are needed to use External Data Volumes in Domino Cloud.
</Note>

## Supported EDV types

Domino supports the following data volumes:

* Network File Systems (generic NFS and AWS EFS)

* Windows Share (SMB)

## EDV support in Domino executions

External Data Volumes are supported in the following Domino executions:

* [Jobs](/cloud/platform-capabilities/core-concepts/jobs) (including [Scheduled Jobs](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job))

* [Workspaces](/cloud/platform-capabilities/core-concepts/workspaces)

* [Apps](/cloud/platform-capabilities/features/apps)

* [Launchers](/cloud/platform-capabilities/features/launchers)

* [On-demand clusters](/cloud/platform-capabilities/features/compute-clusters-hpc)

## Limitations

* Domino endpoints do not support External Data Volumes.

* External Data Volume actions are not exposed by the Domino Platform API or Domino CLI.

* Domino does not track snapshots or versions for EDV data. For reproducible data, use [Dataset snapshots](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots).

* SMB is disabled by default. See [Set up Kubernetes PV and PVC](/cloud/admin/data-administration/external-data/external-data-volumes#set-up-the-kubernetes-pv-and-pvc) for more details.

## Next steps

[Add EDVs to Projects.](/cloud/platform-capabilities/core-concepts/data/external-volumes/use-a-mounted-volume)


# Add EDVs to Projects
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/external-volumes/use-a-mounted-volume

Add shared EDVs to Projects, choose which volumes mount in executions, use their mount paths, and remove volumes without affecting other Projects.

If a Domino admin has registered an external data volume (EDV) and shared it with you, you can add it to any of your projects. When you remove an EDV from a project, it remains mounted and available to other projects in Domino.

<Note>
  [Remote data planes](/cloud/platform-capabilities/core-concepts/data-plane) are needed to use EDVs in Domino Cloud.
</Note>

## Add an EDV to a Project

1. In your project, go to **Data** and click **External data volumes**.

   <img alt="Navigate to External data volumes" />

2. Click **Add External Volume**.

3. Select an EDV from the list.

4. Click **Add**.

If you cannot view, search for, or select External data volumes, they are not registered in your deployment or you do not have access privileges. Contact your Domino administrator for assistance.

If your volume is successfully mounted, it is listed in your project at **Data** > **External data volumes**. The table indicates which [data planes](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace) have access to this EDV:

<img alt="List of the mounted external data volumes" />

## Remove an EDV from a Project

1. In your project, go to **Data** > **External data volumes**.

2. Click the three vertical dots at the end of the row for the volume in the table, then click **Remove**.

   <img alt="Where to click to remove an external data volume" />

## Use an EDV in a Project

By default, EDVs that are mounted to your project (and that you have access to) are also automatically mounted in supported executions. To access a volume in an execution, reference the mount path of the volumes.

Choose to mount specific EDVs for your execution. The following example shows specific EDVs being selected in the Workspace startup modal.

<img alt="Selectable external data volumes" />

<Tip>
  Each EDV used by an execution must be successfully bound to the execution during startup. If any binding issues occur, the execution does not start. Additionally, each EDV may introduce latency to the startup process. To optimize startup time and avoid unnecessary startup failures, select only the necessary EDVs for your task.
</Tip>

### View all mounted volumes in a Project

In your project, go to **Data** > **External data volumes**.

Each volume’s properties are shown in the table.

* **Name**: An alias for the volume, set by your Domino administrator.

* **Type**: The type of volume. Domino supports NFS, AWS EFS, and Windows Share (SMB).

* **Description**: A description of the volume, set by your Domino administrator.

* **Mount Path**: The mount path of the volume: `/domino/edv/name-of-volume`. Use this mount path when using the volume in a [Job, Workspace, or other supported Domino execution](/cloud/platform-capabilities/core-concepts/data/external-volumes#edv-support-in-domino-executions).

* **Data Plane**: In [Domino Nexus deployments](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture), the [data plane](/cloud/platform-capabilities/core-concepts/data-plane) where the volume is mounted.

<Warning>
  If a volume is greyed out, you do not have privileges to use it. Your project can also contain volumes that are mounted, but not listed. You see a message in the application for both these situations. If you need access to these volumes, contact your Domino administrator.
</Warning>

## Next steps

* Learn about [EDV security and sharing](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-edv).


# Data
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/index

Where your data lives when you work in Domino, and how to choose between Data Source connectors, Domino Datasets, NetApp Volumes, and External Data Volumes.

Domino separates your data from your code. Your code lives in Project files, which Domino versions on every change. Your data lives outside those files, so your commit history stays about code, and the same data can serve any Project that needs it. Domino gives you several ways to reach data, and the right one depends on where the data already lives and how you want to work with it.

You don't have to move data into Domino to use it. You can query databases, warehouses, and object stores where they already run, mount existing NFS or SMB file shares, plug in enterprise NetApp storage, or store data inside Domino when you want managed, shareable storage with built-in snapshots backed by the platform.

The Data section covers all of these methods, in the order they appear in the sidebar:

* **[Domino Datasets](/cloud/platform-capabilities/core-concepts/data/datasets)**: Managed filesystem storage inside Domino, shareable across Projects and versioned with snapshots. Every Project starts with its own Dataset, named after the Project, which is the fastest way to write data alongside your code without polluting your Project's file history.
* **[NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes)**: NetApp ONTAP enterprise storage mounted directly into Domino workloads. For organizations that run ONTAP, NetApp Volumes match what Datasets do and add enterprise-grade storage features. See [Datasets vs. NetApp Volumes](#datasets-vs-netapp-volumes) for how to choose.
* **[External Data Volumes (EDVs)](/cloud/platform-capabilities/core-concepts/data/external-volumes)**: Existing Network File System (NFS) or Server Message Block (SMB) file shares that an administrator registers with Domino and mounts into your executions.
* **[Access external data directly](/cloud/platform-capabilities/core-concepts/data/access-external-data-directly)**: Reach any data service from your code using the same libraries you'd use locally, when Domino doesn't have a managed connector for it.
* **[Data sharing and security](/cloud/platform-capabilities/core-concepts/data/sharing-and-security)**: How access control works for each method.
* **[Domino Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors)**: Managed connections to popular external databases, warehouses, and object stores that you query from code, without hand-managing drivers or credentials.

[Project artifacts](/cloud/platform-capabilities/core-concepts/projects/git-based-projects#save-and-pull-artifacts) round out the picture: a version-controlled folder inside each Project, sized for small outputs such as charts and serialized models, and the one place where "data" is intentionally tied to a single Project's history.

### Data across the platform

Data connections in Domino are platform-level objects, not Project contents: the same Data Source, Dataset, NetApp Volume, or EDV can be [attached to every Project](/cloud/platform-capabilities/core-concepts/projects) that needs it, with its own permissions, and each Project's executions mount or query the attached data. Project artifacts are the exception that lives inside a Project.

Connected data then serves the whole lifecycle, not just development. [Workspaces](/cloud/platform-capabilities/core-concepts/workspaces), [Jobs](/cloud/platform-capabilities/core-concepts/jobs), [Flows](/cloud/platform-capabilities/features/flows), and [on-demand compute clusters](/cloud/platform-capabilities/features/compute-clusters-hpc) read and write it during development and training. Published [Apps](/cloud/platform-capabilities/features/apps), [Agents](/cloud/platform-capabilities/features/agents), and [model APIs](/cloud/platform-capabilities/features/model-deployment) query it at serving time. [Model Monitoring](/cloud/platform-capabilities/features/monitoring) connects to your data sources on its own to compare live predictions against ground truth, and its drift and quality alerts drive retraining. [Governance](/cloud/platform-capabilities/features/governance) closes the accountability loop: Dataset files can serve as evidence in governance bundles, and Data Source activity is recorded in the [Audit Trail](/cloud/platform-capabilities/features/governance/audit-trail).

```mermaid placement="top-right" theme={null}
flowchart LR
    ext[("Your external data:<br/>databases, warehouses,<br/>file shares")]
    subgraph domino ["The Domino platform"]
        subgraph shared ["Platform-level data, shared across Projects"]
            ds["Data Source<br/>connectors"]
            edv["External Data Volumes"]
            nv["NetApp Volumes"]
            dsets[("Domino Datasets:<br/>Domino-managed storage")]
        end
        proj["Projects:<br/>attach the data they need<br/>(artifacts live here)"]
        exec["Executions:<br/>Workspaces, Jobs, Apps,<br/>Agents, model APIs"]
    end
    ext --> ds
    ext --> edv
    ext --> nv
    ds --> proj
    edv -->|attached to| proj
    nv -->|attached to| proj
    dsets --> proj
    proj -->|mounted and queried by| exec
```

## Choose a method

Two defaults cover most work:

* **Your data lives in an external store**: Use a Data Source connector. Your administrator can pre-configure the connection details centrally, so you add only your credentials: no guessing at hostnames or port numbers. Credentials are stored securely, and the connection is reusable in any Project. If Domino has no connector for your store, [connect directly](/cloud/platform-capabilities/core-concepts/data/access-external-data-directly) with the same library you would use locally.
* **You want managed storage inside Domino**: Use a Domino Dataset, or a NetApp Volume if your organization runs ONTAP. Both mount into executions as filesystem paths, both support snapshots for reproducibility, and both are shareable across Projects. See [Datasets vs. NetApp Volumes](#datasets-vs-netapp-volumes) below.

The other methods fit specific situations: use an External Data Volume when IT already maintains an NFS or SMB file share you need, and Project artifacts for small outputs that belong with your code.

### Datasets vs. NetApp Volumes

Datasets and NetApp Volumes cover the same ground. Anything Datasets do, NetApp Volumes generally do at least as well and often better, but they aren't strictly "Domino-managed" the way Datasets are: an administrator needs a NetApp ONTAP account configured and connected first. Once that's done, users work with NetApp Volumes the same way they work with Datasets, from the same **Data** menu.

Use **Datasets** when:

* You want storage that works out of the box, with no infrastructure prerequisites.
* You are creating a data artifact scoped to a Project (Datasets originate inside a Project, though you can then share them anywhere).
* You want to branch your data: you can create a new Dataset from a Dataset snapshot, where a NetApp Volume snapshot can't be cloned into a new NetApp Volume.

Use **NetApp Volumes** when:

* Your organization already runs, or is willing to set up, NetApp ONTAP.
* You need multi-terabyte, high-performance shared storage with enterprise backup, snapshot retention, and disaster recovery.
* You want snapshots that don't consume additional storage capacity and near-instant snapshot creation.
* You want data to be a global entity in Domino rather than a Project-owned artifact.

## Compare the methods

| Method | Best for | Where the data lives | Typical size | Access control |
| - | - | - | - | - |
| [Domino Datasets](/cloud/platform-capabilities/core-concepts/data/datasets) | Versioned training data and pipeline outputs, out of the box | Domino-managed storage | Gigabytes to a few terabytes | [Dataset roles](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-datasets) |
| [NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes) | Multi-terabyte shared storage with enterprise backup and efficient snapshots | Your NetApp ONTAP filesystems | Multi-terabyte and beyond | [NetApp Volume roles](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/create-netapp-volumes#add-users-or-organizations-to-netapp-volumes) |
| [External Data Volumes](/cloud/platform-capabilities/core-concepts/data/external-volumes) | NFS or SMB file shares IT already maintains | Your NFS or SMB storage | Whatever your NFS or SMB storage provides | [Assigned users and groups](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-edv) |
| [Data Source connectors](/cloud/platform-capabilities/core-concepts/data/data-source-connectors) | Querying existing databases, warehouses, and object stores | Your external data service | What you pull into the executor | [Users, groups, or a service account](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources) |
| [Project artifacts](/cloud/platform-capabilities/core-concepts/projects/git-based-projects#save-and-pull-artifacts) | Small outputs that belong with code | The Project's artifact storage | About 10 GB in total | [Project collaborators](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#collaborator-permissions) |

### Sizing and cost

Domino sets no size limit on Datasets, NetApp Volumes, or External Data Volumes: the ceiling comes from whatever storage backs them, so check that provider's documentation for the figure that applies to you. For NetApp Volumes it depends on the ONTAP volume type, and a single traditional FlexVol supports up to 300 TiB. Project files, where artifacts live, are the exception, and Domino applies its own configurable limits there. See [Files, artifacts, and results](/cloud/platform-capabilities/core-concepts/projects/files-artifacts-and-results).

What grows is cost, not a Domino quota. Storage cost tracks the bytes you keep, and every Dataset snapshot duplicates the data it captures, so delete what you no longer need. NetApp's redirect-on-write snapshots consume no extra space to start, which keeps the cost of versioning low. When cost matters more than filesystem access, keep bulk data in object storage such as Amazon S3 and reach it through a Data Source connector.

Administrators can cap what any one person stores. See [Dataset quotas and limits](/cloud/platform-capabilities/core-concepts/data/datasets/manage-large-data#dataset-quotas-and-limits).

## Data reproducibility

When your work must be repeatable, use snapshots: immutable, versioned copies of a Dataset or NetApp Volume that your code can target by version number or tag. Data Source connectors and EDVs read live data, so Domino can't guarantee that the same query returns the same result later. If the external system supports its own snapshots or time travel, you can use those features from your code, or copy query results into a Dataset or NetApp Volume and snapshot them.

## Share your data securely

Data in Domino is reusable across Projects with full control over who can access it. Datasets, NetApp Volumes, Data Sources, and EDVs each carry their own permissions, so sharing a Project does not automatically expose its data. Dataset and NetApp Volume sharing is zero-copy: every Project mounts the same files rather than receiving a copy, so a read-only shared Dataset or NetApp Volume works as a controlled way to distribute a curated data product across teams. See [Data sharing and security](/cloud/platform-capabilities/core-concepts/data/sharing-and-security) for how to grant and restrict access for each method.

## Work with data

* [Datasets](/cloud/platform-capabilities/core-concepts/data/datasets): Create, use, and snapshot Domino Datasets.
* [NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes): Mount ONTAP volumes into Domino workloads.
* [External Data Volumes](/cloud/platform-capabilities/core-concepts/data/external-volumes): Mount existing network file shares.
* [Access external data directly](/cloud/platform-capabilities/core-concepts/data/access-external-data-directly): Reach any other data service with its native library.
* [Data sharing and security](/cloud/platform-capabilities/core-concepts/data/sharing-and-security): Control who can read and edit your data.
* [Domino Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors): Connect to supported databases, warehouses, and object stores.


# Add or remove NetApp Volumes on Projects
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/netapp-volumes/add-remove-netapp-volumes

Attach NetApp Volumes to Projects, Workspaces, and Jobs, choose read-only or read-write access, and detach or delete volumes safely.

NetApp Volumes are global entities in Domino, similar to Datasets but not scoped to a single Project. You attach a NetApp Volume to any Project that needs it, and the Volume mounts into that Project's executions the same way a Dataset does. Multiple users on multiple Projects can work against the same NetApp Volume in parallel, which makes NetApp Volumes a good fit for shared team datasets and enterprise-scale storage.

You'll need a role that allows you to see or work with a NetApp Volume to attach it to a Project. If you have the **Reader** role, you can still attach the NetApp Volume, but you can only view the data.

## Add a NetApp Volume to a Project

1. Close out any Projects and go to the main Domino screen.

2. From the toolbar, click **Data** > **NetApp Volumes**.

3. Find the volume you want to add in the NetApp Volumes list.

4. Click the three dots to the right and select **Add to project**.

5. Use the menu to search for and select the Project.

6. Click **Confirm**.

If you want to add the same NetApp Volume to multiple Projects, repeat these steps.

## Remove a NetApp Volume from a Project

Remove a NetApp Volume from a Project when the Volume has failed or you no longer need it there. Removing a Volume from a Project only detaches it: the Volume itself and its data are unaffected, and other Projects that use it keep working.

1. Open the **Project** with the NetApp Volume you want to remove.

2. From the left panel, click **Data** > **NetApp Volumes**.

3. Find the volume in the NetApp Volumes list.

4. Click the three dots to the right and select **Remove from project**.

## Delete a NetApp Volume

Deleting a NetApp Volume permanently loses the data. Only delete a Volume when you're sure it's no longer needed anywhere.

You can delete your own NetApp Volumes if your Domino administrator has enabled that ability for your role. If not, you can mark a NetApp Volume for deletion by submitting a deletion request to your Domino administrator.

1. From the toolbar, click **Data** > **NetApp Volumes**.

2. Find the volume in the NetApp Volumes list.

3. Click the three dots to the right and:

   1. Select **Delete Volume**.

   2. Or select **Mark for deletion**.

4. Read the confirmation message:

   1. If you agree, click **Delete** or **Mark for deletion**.

   2. To abort the action, click **Cancel**.

## Attach a NetApp Volume to a Workspace or Job

Attaching a NetApp Volume to a Workspace, Job, App, or Launcher gives that execution direct access to its files. You select which NetApp Volumes to attach when you launch the execution:

* Users with the **Volume Owner** or **Editor** role may attach the NetApp Volumes assigned to the Project to Workspaces or Jobs within that Project in **read-write** mode.
* Users with the **Reader** role may attach the same NetApp Volumes in **read-only** mode.
* Apps mount as **read-write** or **read-only** based on the permissions of the App owner.

By default an attached NetApp Volume mounts with its live data and all of its snapshots. On the **Data Plane & Hardware** tab you can pin any mounted NetApp Volume to a single snapshot instead, the same way you can [pin a Dataset](/cloud/platform-capabilities/core-concepts/data/datasets/use-datasets-and-snapshots#choose-which-datasets-and-snapshots-to-mount).

## Mount paths

NetApp Volumes mount into executions at file paths that mirror the Dataset scheme. Because NetApp Volumes are global, there's no local vs. imported distinction:

* **DFS Projects**: `/domino/netapp-volumes/<name>` for the read-write NetApp Volume, `/domino/netapp-volumes/snapshots/<name>` for its snapshots, and `/domino/netapp-volumes/snapshot-tags/<name>/<tag>` for a tagged one.
* **Git-based Projects**: `/mnt/netapp-volumes/<name>` for the read-write NetApp Volume, `/mnt/netapp-volumes/snapshots/<name>` for its snapshots, and `/mnt/netapp-volumes/snapshot-tags/<name>/<tag>` for a tagged one.

For the equivalent Dataset paths and the underlying directory layout, see [Understand Dataset paths](/cloud/platform-capabilities/core-concepts/data/datasets/use-datasets-and-snapshots#understand-dataset-paths). For how tagged and numbered snapshots are mounted, see [Snapshot mounting in executions](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots#snapshot-mounting-in-executions).

## Next steps

* [Create NetApp Volumes from Domino or a project](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/create-netapp-volumes) teaches you how to create and configure new NetApp Volumes from Domino or within a specific project.

* [View or edit details of a NetApp Volume](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/view-and-edit-netapp-volumes) teaches you about viewing lists of NetApp Volumes, and editing the descriptions and permissions of them (only if you are the **Owner** or **Editor** of that NetApp Volume).


# Create NetApp Volumes from Domino or a project
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/netapp-volumes/create-netapp-volumes

Create NetApp Volumes globally or within a Project, set capacity and filesystem details, and assign Reader, Editor, or Owner roles.

NetApp Volumes are created within an existing filesystem. You can create NetApp Volumes within a specific project or from globally accessible data.

<img alt="Create NetApp Volume" />

## Create a NetApp Volume from Domino

When you create a new volume from Domino’s home page, you must manually [add the volume to a project](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/add-remove-netapp-volumes).

1. Open **Domino** to create a NetApp Volume.

2. From the toolbar, click **Data** > **NetApp Volumes**.

3. Click **Add NetApp Volume** > **Create Volume**.

4. Enter the following details in the **Create NetApp Volume** window:

   1. **Name:** Enter a name for the volume. Names must contain only letters, numbers, underscores, and hyphens.

   2. **Description:** Enter a brief description of the volume.

   3. **Data Plane:** Choose a data plane from the menu.

   4. **NetApp Filesystem:** Choose a filesystem from the menu.

   5. **Capacity:** Specify the maximum capacity of the volume.

5. Click **Next**.

6. [Add users or organizations](#add-users-or-organizations-to-netapp-volumes) to your volume and assign their roles if needed.

7. Click **Finish**.

## Create a NetApp Volume from a project

When you create a new volume within a project, it is automatically added to that project.

1. Open the **Project** where you want to create a NetApp Volume.

2. From the left panel, click **Data** > **NetApp Volumes**.

3. Click **Add NetApp Volume** > **Create Volume**.

4. Enter the following details in the **Create NetApp Volume** window:

   1. **Name:** Enter a name for the volume. Names must contain only letters, numbers, underscores, and hyphens.

   2. **Description:** Enter a brief description of the volume.

   3. **Data Plane:** Choose a data plane from the menu.

   4. **NetApp Filesystem:** Choose a filesystem from the menu.

   5. **Capacity:** Specify the maximum capacity of the volume.

5. Click **Next**.

6. [Add users or organizations](#add-users-or-organizations-to-netapp-volumes) to your volume and assign their roles if needed.

7. Click **Finish**.

## Add users or organizations to NetApp Volumes

There are different Roles preconfigured for use with your NetApp Volumes. If you are added to a volume and your assigned role is:

* **Reader:** You can view files and snapshots, and manage NetApp Volumes in projects as read-only.

* **Editor:** You can manage NetApp volumes by updating them, mounting them, marking them for deletion, or restoring them. You can also create and delete snapshots and manage users.

* **Owner:** You have all the capabilities of an Editor and the ability to update Volume grants.

<img alt="Set User Roles for NetApp Volume" />

To add users and set permissions for individuals or organizations, on the **Volume Permissions** screen:

1. Type a few characters to select a user or organization from the menu.

2. Select a **Role** for that user from the menu and click **Add**.

3. Repeat steps 1-2 to add as many users or organizations as needed.

4. Once you have added users, click **Finish**.

## Manage users on a NetApp Volume

You can add, remove, or update the roles of users later if needed:

1. Open the **Project** with the NetApp Volume where you want to manage user roles.

2. From the left panel, click **Data** > **NetApp Volumes**.

3. Find the volume in the NetApp Volumes list.

4. Click the three dots to the right and select **Edit permissions**.

5. You can manage users by:

   1. Removing users by clicking the delete icon.

   2. Adding new users and assigning them roles.

   3. Changing user roles as needed.

6. Once finished, click **Save permissions**.

## Next steps

* [Add or remove NetApp Volumes from a project](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/add-remove-netapp-volumes) shows you how to use the different roles that are preconfigured for use with your NetApp Volumes.

* [View or edit details of a NetApp Volume](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/view-and-edit-netapp-volumes) teaches you about viewing lists of NetApp Volumes, and editing the descriptions and permissions of them (only if you are the **Owner** or **Editor** of that NetApp Volume).

* [Request deletion of a NetApp Volume](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/add-remove-netapp-volumes#delete-a-netapp-volume) shows you why and how to request that an admin deletes a NetApp Volume.


# Work with NetApp Volumes
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/netapp-volumes/index

Use NetApp Volumes for shared, large-scale ONTAP storage, take near-instant snapshots, and manage access, attachments, and deletion.

Domino Volumes for NetApp ONTAP (NetApp Volumes) offer a native integration with NetApp’s enterprise-grade storage system. Mount storage from NetApp ONTAP filesystems directly into Domino workloads.

NetApp Volumes cover nearly the same ground as [Domino Datasets](/cloud/platform-capabilities/core-concepts/data/datasets), and the Domino Data team recommends them wherever your organization runs ONTAP. Their [snapshots](https://docs.netapp.com/us-en/ontap/concepts/snapshot-copies-concept.html) are the main reason:

* **Near-instant, regardless of size**: a snapshot of 100 GB or more completes in about 3 seconds, where a Dataset snapshot takes anywhere from seconds to hours as the data grows.

* **No extra storage to start**: ONTAP uses redirect-on-write, so a snapshot initially consumes no extra space. A Dataset snapshot duplicates the data it captures at the moment you take it.

Cloning is the one thing Datasets do that NetApp Volumes don't: you can create a new Dataset from a Dataset snapshot, but you can't clone a NetApp Volume snapshot into a new NetApp Volume.

<Frame>
  <img alt="Domino Volumes for NetApp ONTAP." />
</Frame>

NetApp Volumes suit multi-terabyte and larger datasets that need high availability and enterprise backup and recovery. They are not as quick to start with as a Dataset: they require NetApp ONTAP infrastructure, and an administrator registers an ONTAP Filesystem in Domino first. Once that is done, you work with them from the same **Data** menu as Datasets. They aren't version-controlled by Domino, and there are limits on the number of snapshots per NetApp Volume and NetApp Volumes per filesystem.

Domino sets no size limit on a NetApp Volume. The ceiling comes from ONTAP and depends on the volume type: a single traditional FlexVol supports up to 300 TiB. Check [NetApp's ONTAP documentation](https://docs.netapp.com/us-en/ontap/volumes/index.html) for the limits that apply to your filesystem.

A NetApp ONTAP Filesystem connects a Data Plane and Kubernetes storage class to a named storage container for NetApp Volumes. The ONTAP Filesystem offers:

* Clear visibility into where NetApp Volumes are stored.

* Support for multiple accounts and Data Planes, which is essential given the per-account limits on NetApp Volume creation.

* Improved deletion flow by allowing NetApp Volumes to be moved to cold storage, instead of being permanently deleted.

This helps enforce retention policies and reduces the risk of accidental data loss.

## What you can do

[Create NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/create-netapp-volumes)\
Create and configure new NetApp Volumes from Domino or the context of a specific project.

* Set up the NetApp Volume name, description, and access roles.

* Choose snapshot and synchronization options.

[Manage NetApp Volume access in Projects](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/add-remove-netapp-volumes)\
Add or remove existing NetApp Volumes from any project:

* Attach NetApp Volumes to enable data access within the project.

* Detach NetApp Volumes when no longer needed or to restrict access.

[Understand NetApp Volume roles](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/view-and-edit-netapp-volumes)\
Learn about the built-in role types and their permissions:

* **Owner**: Full control. Reads and writes data, edits the description, takes and permanently deletes snapshots, changes who has access and their role, and requests deletion of the volume.

* **Editor**: Reads and writes data on the volume, edits the description, and takes or permanently deletes snapshots. An Editor cannot change who has access to the volume or change other users' roles; only an Owner can do that.

* **Reader**: Reads data on the volume and views its snapshots. Cannot write data, edit the description, or manage access. A Reader can only mark a snapshot for deletion, not delete it outright.

[Version data with Snapshots](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots)\
Domino lets you create snapshots of both NetApp Volumes and Domino Datasets to support reproducibility, version control, and collaboration.

[View and edit NetApp Volume details](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/view-and-edit-netapp-volumes)\
Access your list of NetApp Volumes where you have permissions:

* View NetApp Volume metadata and configuration.

* Edit description and permissions (Owner or Editor only).

[Request NetApp Volume deletion](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/add-remove-netapp-volumes#delete-a-netapp-volume)\
If a NetApp Volume is no longer needed:

* Delete the NetApp Volume or submit a deletion request to your Domino administrator.

* Ensure the NetApp Volume is no longer attached to active projects or in use by users.


# View and edit NetApp Volumes
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/netapp-volumes/view-and-edit-netapp-volumes

View NetApp Volume metadata, Projects, snapshots, and mount paths, then rename volumes, edit descriptions, and manage or delete snapshots.

You can view a list of NetApp Volumes where you have **Owner**, **Editor**, or **Reader** roles. You can edit the description and permissions if you are the **Owner** or **Editor** of a NetApp Volume.

## View a list of usable NetApp Volumes

The **Data** menu in Domino lets you view a list of NetApp Volumes that are usable for your workspace.

1. Close any projects and go to the main Domino screen.

2. From the toolbar, click **Data** > **NetApp Volumes**.

3. You’ll see a screen with a list of available NetApp Volumes.

The information shown in this list view for each NetApp Volume includes its **Name**, **Owner**, and **Size**, as well as:

* **Projects**: Displays a list of projects that the NetApp Volume is associated with.

* **Snapshots**: This shows the number of snapshots taken of the NetApp Volume. Clicking on a number shows a list of these snapshots.

* **Data Plane**: This refers to the data plane to which the NetApp Volume is connected.

You can also view more granular details for individual NetApp Volumes.

## View details of a NetApp Volume

Here is how you view the details of NetApp Volumes from a project:

1. Open the **Project** with the NetApp Volume you want to view.

2. From the left panel, click **Data** > **NetApp Volumes**.

3. Find the volume in the NetApp Volumes list and click it.

The details page shows the volume name with a snapshot selector next to it, a **Files** tab that lists the volume's files, and a details sidebar. On the **Files** tab, you can search the file list, choose which columns to show, upload files, and download or delete selected files from the **Actions** menu. The sidebar includes:

* **Description**: A brief description of the NetApp Volume’s contents or purpose. If you have the **Editor** or **Owner** role, you can edit the volume description.

* **Data Plane**: This refers to the data plane to which the NetApp Volume is connected.

* **Storage name**: The name of the storage file system that backs the NetApp Volume.

* **Storage path**: The location that the data is written to and read from.

* **Max snapshots**: The maximum number of snapshots the volume can keep.

* **Tags** and **Properties**: The [tags and Property values](/cloud/platform-capabilities/core-concepts/knowledge-management) assigned to the volume.

## Edit details of a NetApp Volume

You can rename the volume, edit the description, and change permissions. Changing the volume name alters its mount path, which can cause errors in scripts that rely on a specific volume name. If you are the **Owner** or **Editor** of a NetApp Volume, edit details by:

1. From the toolbar, click **Data** > **NetApp Volumes**.

2. Find the volume in the NetApp Volumes list.

3. Click the name of the volume you want to edit.

4. In the details sidebar, click the **Description** edit icon, update the text, and save your change.

5. Rename the NetApp Volume by clicking the three dots and choosing **Rename Volume**.

   1. Click **Confirm** to accept the change or **Cancel** to exit.

6. Change permissions by clicking **Share** at the top of the details page.

## Take a snapshot of your data

You can snapshot your data without incurring duplication. To do so, you’ll need either **Owner** or **Editor** permissions to snapshot all files in a NetApp Volume.

1. From the toolbar, click **Data** > **NetApp Volumes**.

2. Find the volume in the NetApp Volumes list.

3. Click the name of the volume you want to snapshot.

4. On the **Files** tab, click **Take Snapshot**. The button is unavailable when the volume has reached its maximum number of snapshots.

5. Expand the snapshot selector next to the volume name, labeled **Latest data** by default, to view a list of snapshots for that volume.

## Delete a snapshot of a NetApp Volume

Deleting a snapshot is *permanent*. You need permission to delete the NetApp Volume to delete its snapshots; without it, the **Delete Snapshot** button is unavailable.

1. From the toolbar, click **Data** > **NetApp Volumes**.

2. Find the volume in the NetApp Volumes list.

3. Click the name of the volume you want to view.

4. In the snapshot selector next to the volume name, select the snapshot you want to delete. Domino shows a read-only view of the snapshot.

5. On the **Files** tab, click **Delete Snapshot**.

6. At the prompt, choose **Delete** to delete the snapshot or **Cancel** to exit.

## Next steps

* [Create NetApp Volumes from Domino or a project](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/create-netapp-volumes) teaches you how to create and configure new NetApp Volumes from Domino or within a specific project.

* [Add or remove NetApp Volumes from a project](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/add-remove-netapp-volumes) shows you how to use the different roles that are preconfigured for use with your NetApp Volumes.

* [Request deletion of a NetApp Volume](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/add-remove-netapp-volumes#delete-a-netapp-volume) shows you why and how to request that an admin deletes a NetApp Volume.


# Share data securely
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/sharing-and-security/index

Compare access controls for Domino Datasets, NetApp Volumes, Data Sources, and EDVs, then choose roles, credentials, and visibility.

Data in Domino is reusable across Projects, and each data method carries its own permissions: sharing a Project does not automatically expose its data. The four data methods use related but distinct access models:

* **Datasets** use [Dataset roles](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-datasets): the Dataset owner grants roles (Reader, Editor, Owner) to users and organizations, and the role decides who can view, edit, or manage the Dataset. Datasets originate inside a Project but can then be shared with any Project or user in Domino.
* **NetApp Volumes** use [role-based permissions similar to Datasets](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/create-netapp-volumes#add-users-or-organizations-to-netapp-volumes): the Volume owner grants Reader, Editor, or Owner roles to users and organizations. NetApp Volumes differ from Datasets in that they are global from the start (they aren't scoped to a single "home" Project), so their permissions travel with the Volume, not with any one Project.
* **Data Sources** use [credentials and visibility](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources): access depends on how the Data Source authenticates (individual credentials or a shared service account) and whether it's shared with everyone or with specific users and organizations.
* **External Data Volumes (EDVs)** are [administrator-controlled](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-edv): an administrator decides which users and groups can access each volume.


# Share Data Sources securely
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources

Configure service account or individual credentials for Data Sources, control sharing, and understand Data API and direct-connection security.

Domino data sources use Domino connectors to connect to external data stores. Data source permissions determine who is authorized to access a data source.

Depending on their permissions and credentials, users can modify data in the external data store from Domino. By understanding how data source credentials and sharing work, you can ensure that your collaborators have the appropriate kind of access to your data sources.

## Configure Data Source credentials

There are two methods of initializing credentials when creating a Domino Data Source: service account, and individual credentials. These two methods are defined below:

* Service account

  With this approach, anyone with permissions to access the Domino Data Source uses the same credentials (those of the service account). Only a Domino admin can create a data source that uses a service account, at **Admin** > **Data** > **Data Sources** > **Create a Data Source**.

  <Warning>
    If the service account has write access to the data store, then anyone with access to the data source can modify data in the external data store from Domino. Make sure the service account has permissions that are appropriate for all of its Domino users.
  </Warning>

* Individual credentials

  Each user who accesses the data source must set up their own credentials. Once their credentials are set up in Domino, they do not need to re-enter them when accessing the data source.

  <Warning>
    If a user’s individual credentials give them write access to the data store, then they can modify data in the external data store from Domino.
  </Warning>

## Share a Data Source

The Data Source owner or a Domino admin can grant access to the Data Source for specific users or organizations. Admin can also make it available to everyone in the Domino deployment.

When multiple users collaborate on a project, a data source used by one user might not be properly configured for another. Domino proactively surfaces such problems both from the project’s Data page as well as from the Data tab in Domino Workspaces.

### Change a Data Source’s permissions

To edit the share permissions for a data source, go to **Data** > **Data Sources**, select your data source, and click **Update Permissions**.

* To limit access to this data source, select **Specific users or organizations** and add or remove users or organizations as appropriate.

* Admin can select **Everyone** to share this data source with all users in the deployment.

Note that you cannot change the credential method (service account or individual credentials) for an existing data source. To change the credential method, you must create a new data source. If you do this, consider deleting the old data source and notifying users that they should use the new one.

## Data API authentication and authorization

The [Domino Data API](/cloud/reference/api/data_api) is used to access Domino data sources. When you use the Data API from a Domino execution, your user identity is verified automatically to enforce Domino permissions. The library attempts to use a [Domino JWT token](/cloud/reference/api/domino-api-authentication), or, if not available, a [user API key](/cloud/reference/api/domino-api-authentication).

The following is a summary of the user identity that is used for access to a data source based on the Domino execution type:

* **Workspaces and Jobs**: The user who started the execution.

* **Launchers**: The user who started the launcher regardless of who created the launcher.

* **Domino Apps**: The user who published the app regardless of who is accessing the app.

* **Domino endpoint** - No user identity.

For Domino endpoints and other advanced use cases that require establishing a different user identity, you inject an API key into an execution with an environment variable. Then, you can use it explicitly when retrieving a data source.

For detailed information, see [Authentication](/cloud/reference/api/data_api/authentication) in the API documentation.

## Security for direct connections to external data stores

When directly accessing an external data store, it is up to you to configure the appropriate authentication and access control.

In cases of directly accessing common AWS data stores (such as S3, Redshift, or Postgres) within an AWS deployment, Domino can propagate an IAM token down through your query.

## Next steps

* [Create data sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors)

* [Work with data sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources)


# Share Datasets securely
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-datasets

Share Domino Datasets with users or organizations, assign Owner, Editor, or Reader roles, and curate large Project data for consumers.

Dataset owners can grant access to Domino users and groups. Only authorized users can import and access a dataset. Additionally, the actions that a user can perform on a dataset are determined by their Domino role.

## Set permissions to share a Dataset

You can change the permissions on a dataset to do the following:

* Add users or organizations.

* Update roles for existing users or organizations.

* Delete users or organizations.

### Change who can access a Dataset

1. In the navigation pane, click **Data**.

2. Click **Domino Datasets**.

3. In the **Your Domino Datasets** area, go to the end of the row for the dataset whose permissions you want to change.

4. Click the three vertical dots and click **Edit Permissions**.

5. Enter the users or organizations to give them permission to the dataset.

   <Note>
     If you want to give all project members access to the dataset, click the **Add all project members** link.
   </Note>

6. Specify the user or organization’s role. Dataset roles are independent of Domino’s global user roles; see below for details about dataset roles.

7. Click **Add**.

   <Note>
     In the **Current Permissions** area, you can modify the role as needed, or click the trash icon to delete permissions.
   </Note>

8. Click **Save Permissions**.

   <Note>
     You must restart executions to pick up permission changes.
   </Note>

## Dataset roles

Datasets have three roles: Owner, Editor, and Reader. These are independent of Domino’s global [user roles](/cloud/admin/identity-and-access/manage-users/roles). Roles determine the permissions that users have when they work with a dataset.

The table below describes the roles and access.

<Note>
  The **Relates to Project Role** column is only useful if you are migrating from versions of Domino earlier than 5.4 where datasets were integrated with projects.
</Note>

<table>
  <thead>
    <tr>
      <th>Role</th>
      <th>Relates to Project Role</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td><p>Owner</p></td>
      <td><p>Dataset Author</p></td>
      <td><ul> <li><p>Upload and view files, update metadata.</p></li> <li><p>Take, update, and view snapshots.</p></li> <li><p>Mark datasets and snapshots for deletion.</p></li> <li><p>Restore datasets and snapshots for deletion.</p></li> <li><p>Copy a snapshot to a new dataset.</p></li> <li><p>Download files and folders.</p></li> <li><p>Mount a dataset as read-write to <a href="/cloud/platform-capabilities/core-concepts/jobs">executions</a>.</p></li> <li><p>Edit the dataset’s permissions.</p></li> </ul></td>
    </tr>

    <tr>
      <td><p>Editor</p></td>
      <td><p>Project Owner, Project Contributor</p></td>
      <td><ul> <li><p>Upload and view files, update metadata.</p></li> <li><p>Take, update, and view snapshots.</p></li> <li><p>Mark datasets and snapshots for deletion.</p></li> <li><p>Restore datasets and snapshots for deletion.</p></li> <li><p>Copy a snapshot to a new dataset.</p></li> <li><p>Download files and folders.</p></li> <li><p>Mount a dataset as read-write to <a href="/cloud/platform-capabilities/core-concepts/jobs">executions</a>.</p></li> </ul></td>
    </tr>

    <tr>
      <td><p>Reader</p></td>
      <td><p>ResultsConsumer, ProjectImporter, LauncherUser</p></td>
      <td><ul> <li><p>View files.</p></li> <li><p>View snapshots.</p></li> <li><p>Copy a snapshot to a new dataset.</p></li> <li><p>Download files and folders.</p></li> <li><p>Mount a dataset as read-only to <a href="/cloud/platform-capabilities/core-concepts/jobs">executions</a>.</p></li> </ul></td>
    </tr>
  </tbody>
</table>

## Shareable data in your Project

If you [export and import project content](/cloud/platform-capabilities/core-concepts/projects/manage-projects#export-and-import-project-content) to share data with other members of your team, the consumers of your project will receive the entire contents of your project files in their Runs and Workspaces. That works well if your project is small, simple, and narrowly scoped.

However, for large projects that produce many data artifacts, you might want to expose them to your consumers in smaller, curated subsets. You can do this with Domino datasets.

Consider a project with a small folder full of code and nine folders with various kinds of output data, labeled `experiment1` through `experiment9`. Each data folder is larger than 10 GB, and the whole project is 100 GB. It would be impractical to ask your data consumers to import this project, but you also don’t want to separate the data from the code that produced it by moving the data to a different project.

You can organize the data into datasets, with one dataset for each type of data in which your consumers are interested. In this example, suppose you have two colleagues who want to consume your data. One of them is only interested in the data from the `experiment1` folder, and the other is only interested in the data from `experiment9`.

You can [create](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets) and write two datasets with scripts like the following, where it’s assumed you have named the datasets `experiment1-data` and `experiment9-data`.

`experiment1-populate-dataset.sh`

```shell theme={null}
cp -R $DOMINO_WORKING_DIR/experiment1/. /domino/datasets/experiment1-data/
```

`experiment9-populate-dataset.sh`

```shell theme={null}
cp -R $DOMINO_WORKING_DIR/experiment9/. /domino/datasets/experiment9-data/
```

Your consumers can then [mount](/cloud/platform-capabilities/core-concepts/data/datasets/use-datasets-and-snapshots) only the datasets in which they are interested.

If you are working with data at this scale, write it to datasets whenever you produce it, instead of storing it in your project files.

## Next steps

* [Create datasets](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets)

* [Work with datasets](/cloud/platform-capabilities/core-concepts/data/datasets)


# Share EDVs securely
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-edv

Control EDV access with read-only mounts and Volume Access settings, and verify which administrator-managed volumes you can add to a Project.

This topic explains how security and sharing with external data volumes (EDVs). EDVs can only be defined by Domino admins. Users can mount EDVs in their projects. Domino does not propagate the user’s identity down into the underlying network-attached file system.

## EDV security settings

When an admin defines an EDV in Domino, they can specify two important security settings:

* **Mount as read-only**

  Selecting this option means that users cannot modify the data in the EDV through Domino. This is the recommended setting in most cases.

* **Volume Access**

  This setting specifies which Domino users or groups are allowed to access the EDV. This can be everyone in the Domino deployment, or only specific users and organizations.

## Share or get access to an EDV

Only an admin can modify the **Volume Access** setting for an EDV. If an EDV is not available to all of the users who need to access it, contact your Domino admin.

To see whether an EDV is available to you, go to **Data** > **External Data Volumes** > **Add External Volume** in your project and check whether the EDV you need is listed.

## Next steps

* Learn more about [EDV support in Domino executions](/cloud/platform-capabilities/core-concepts/data/external-volumes#edv-support-in-domino-executions).

* [Work with EDVs](/cloud/platform-capabilities/core-concepts/data/external-volumes) in your projects.


# Create and run Jobs
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/jobs/create-and-run-jobs

Launch and configure Domino Jobs through the UI, CLI, Python SDK, or API, then tag, rename, stop, schedule, and re-run them.

Learn about the many flexible ways that you can launch Domino Jobs depending on your needs. Domino [Jobs](/cloud/platform-capabilities/core-concepts/jobs) provide a structured approach to run [code files](/cloud/platform-capabilities/core-concepts/jobs/create-custom-jobs) organizing the necessary hardware, compute environment, data resources, and security protocols to create a highly reproducible execution environment.

## Run a Job in the UI

You can start a Job interactively from multiple locations in the Domino UI.

### Jobs Dashboard in a Project

To run a Job from the Job dashboard in your Project:

1. In your project, navigate to the **Jobs** section. The Jobs dashboard shows all Jobs for your Project, sorted into tabs by Job status.

   <Note>
     You can also use the Quick Action menu to create a Job. To do this, click the + next to Jobs in the navigation pane.
   </Note>

2. Click **Run** to start a new Job.

3. In the Start Run window, configure the Job.

4. Enter the **Command or File Name** of the script you want Domino to execute, followed by any arguments for the script.

   <Tip>
     By default, all non-zero exit codes are reported as Domino Job failures. If you’d prefer to change this behavior, the best approach is by mapping your code’s exit status to a conventional exit code. In most environments (environments that run POSIX-compliant shells), this can be done using the snippet below. It uses the command exit 2 as an example to exit with a non-conventional exit status that should be treated as success:

     ```bash theme={null}
     bash -c 'exit 2 || [ %? -eq 2 ]'
     ```
   </Tip>

5. Choose the [Compute Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments) for the Job and its specific version. By default, the Job uses the latest version of your environment.

   <Tip>
     Domino recommends that you use the revision that was set as active by the Project owner. If you select another revision, you can see a **Not Recommended** warning.
   </Tip>

   <Note>
     [Restricted Projects](/cloud/admin/platform-configuration/manage-compute-environments/restricted-environments) may limit your choice of environments.
   </Note>

6. If your administrator has enabled [volume provisioning recommendations](/cloud/admin/infrastructure-and-compute/manage-compute-resources/manage-persistent-volumes#enable-volume-provisioning-recommendations), you can select a volume size automatically recommended by Domino, instead of the default volume size.

   <Note>
     In a [Git-based Project](/cloud/platform-capabilities/core-concepts/projects/git-based-projects), the first Job you launch always uses the volume size [configured in your Project settings](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace#select-a-volume-size). Subsequent Job launches receive volume provisioning recommendations.
   </Note>

7. Click **Next** and select the [Hardware Tier](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers) to use for this Job.

   In a [Domino Nexus deployment](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture), hardware tiers in the dropdown are grouped by [data plane](/cloud/platform-capabilities/core-concepts/data-plane), such as `GCP` below. The `Local` data plane corresponds to running the execution in the Domino control plane cluster.

   <img alt="Launch a Job and select a hardware tier" />

8. If necessary, attach a compute cluster to your execution. To learn more about clusters, see the following:

   * [Spark on Domino](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/index)

   * [On-demand Ray](/cloud/platform-capabilities/features/compute-clusters-hpc/ray/index)

   * [On-demand Dask](/cloud/platform-capabilities/features/compute-clusters-hpc/dask/index)

   * [On-demand Open MPI](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/index)

9. Domino shows you what data is mounted in your execution based on the selected data plane. Each mounted NetApp Volume or Dataset has a **Snapshot** column where you can pin it to a single snapshot for this run. [Choose a snapshot per asset at launch](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots#choose-a-snapshot-per-asset-at-launch) covers the options.

   <Tip>
     If you want to use a data mount that is not available, note which data planes it is available in, then return to the first step in the modal to select a hardware tier in the corresponding data plane.
   </Tip>

10. You can also configure your Job to create [Dataset Snapshots](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots) to mount upon execution completion. Created snapshots are shown under the details tab for a Job.

<img alt="Data in selected data plane" />

<Note>
  This feature requires `AllowDatasetSnapshotsOnExecutionCompletion` to be enabled. See the [create a snapshot](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots) guide for more details.
</Note>

### Project files page

You can start a Job directly from the Project **Code** page.

1. Go to your Project > **Code** > Find the file you want to run.

2. Click the gear icon > **Run**.

3. Go to the file you want to run and click the gear icon at the end of the row in the files list, then click **Run**.

4. Use the window to configure the Run. This window has additional options to set up recurring scheduled Jobs and select different hardware tiers.

   **Title**\
   In this field, enter a name for the Job. This name is the label for the Job in the Jobs dashboard.

   **Parameters**\
   If there are any arguments or parameters you’d like to pass to your script, enter them here.

   **Hardware tier**\
   This dropdown list lets you set the hardware tier used by the Job.

   **Schedule to repeat**\
   In this section, specify if you want the run to execute only once, or to repeat on a schedule.

   **Publish after complete**\
   Check this if you want Domino to republish a [Domino endpoint](/cloud/platform-capabilities/features/model-deployment) once the Job completes successfully.

### Launcher UI

A [Launcher](/cloud/platform-capabilities/features/launchers) is a web form that rests on top of a script that Domino can execute in a Job. You can use a Launcher to pass arguments to your script from UI fields in a customized web form.

## Run a Job from the CLI

From your workstation, you can start Jobs through the [Domino CLI](/cloud/reference/cli). You must first [install the Domino CLI](/cloud/reference/cli/install-the-cli) on your system.

After you have logged in and opened a Project, you can start Jobs with:

```bash theme={null}
`domino run <filename>`
```

Examples:

```bash theme={null}
`domino run calculateStats.r`
`domino run runModel.m model1 .05`
`domino run --wait runModel.m model1 .05`
`domino run --direct "pip freeze | grep pandas"`
```

### Domino Python SDK

You can use the [Domino Python SDK](/cloud/reference/python-sdk/python-domino) to start and monitor Jobs in Domino either from a tool like a Jupyter Notebook or VS Code or from within another Domino Job to chain together multiple Jobs in a more complex workflow.

1. Write code to connect to Domino and start a Run using the following example:

```python theme={null}
import os

from domino import Domino

domino = Domino(
    "marks/quick-start",
    api_key = os.environ["DOMINO_USER_API_KEY"],
    host = os.environ["DOMINO_API_HOST"],
)

# Blocking: this will start the run and wait for the run to finish before returning the status of the run
domino_run = domino.runs_start_blocking(
    ["main.py", "a", "b", "9"], title = "Started from Python API example code"
)
print(domino_run)

# Non-blocking: this will start the run and print the Run ID and check the status of the run *once*
# domino_run = domino.runs_start(["main.py", "a", "b", "9"],
#                                title = "Started from Python API example code")
# print(domino_run)
# run_status = domino.runs_status(domino_run.get("runId"))
# print(run_status)
```

<Note>
  When using the Python SDK from within a Domino Workspace or Job, the environment variables needed for authentication are automatically configured. If you would like to set up your workstation outside Domino, learn more about how to [programmatically access Domino](/cloud/reference/api/domino-api-authentication)
</Note>

## Run a Job from the Domino API

You can use the Domino API to start a Job with a POST request.

For more information about how to send valid configuration data, see [`startJob` in the Domino Platform API reference](/api-reference/jobs/start-a-job).

<Note>
  The `commitId` field pins a commit in the Project's internal Domino File System (DFS) history, never a commit on the main Git repository. In a [Git-based Project](/cloud/platform-capabilities/core-concepts/projects/git-based-projects), the DFS still stores artifacts, so `startJob` accepts a `commitId` that exists in the Project's DFS history and rejects any other value with HTTP 400. To pin the main repository to a specific commit, pass `mainRepoGitRef` with `refType: "commitId"` and the SHA in `value`.
</Note>

To list Jobs for a Project from the API, see [List Jobs programmatically](/cloud/platform-capabilities/core-concepts/jobs/monitor-jobs#list-jobs-programmatically) on the Monitor Jobs page.

### Resolve Job defaults before you start a Job

Call `POST /api/jobs/v1/projects/{projectId}/job-defaults` to see the exact configuration a Job in a Project would run with, without starting anything. Use it to show or record a Job's effective configuration, or to fill in a `startJob` request from the Project's defaults.

Every field in the request body is optional. Each field you send is applied as an override, and Domino resolves everything you leave out from the Project's defaults. Send an empty object to resolve every value from the Project. Resolution is read-only and leaves no trace on the Project, and it requires only the `ViewJobs` permission.

The response reports the fully resolved hardware tier, environment and revision, Git ref, Dataset and NetApp Volume snapshots, external volumes, and volume size. Seven fields use the same names as the `startJob` request body, so you can copy each resolved value straight into `POST /api/jobs/v1/jobs`: `hardwareTier`, `environmentId`, `environmentRevisionSpec`, `mainRepoGitRef`, `computeCluster`, `externalVolumeMountIds`, and `netAppVolumeIds`. The `commitId` field shares its name too, but carries over only for Projects that store files in DFS (see the note above). The remaining fields (`datasetSnapshots`, `netAppVolumeSnapshots`, `volumeSizeGiB`, `resolvedMainRepoGitRef`, and `resolvedImportedGitRepoRefs`) are reported for inspection and have no counterpart on the `startJob` request, so copy individual fields rather than the whole object.

### Python and R Domino API bindings

The Domino API is also available in language-specific wrappers.

Use the [dominodatalab Python library](/cloud/reference/python-sdk/python-wrapper-for-domino-api) and [domino R package](/cloud/reference/r-package/domino-r-package) to call the Domino API in your preferred language.

## Schedule a Job

Use Domino to [schedule Jobs](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job) in advance, and set them to execute on a regular schedule. These can be useful when you have a data source that is updated regularly.

## Organize and manage

Jobs Data science is an experimental science - it is common for a Project to require many Jobs before you’ve achieved your goals. Domino allows you to tag and rename Jobs to help you and your team track and manage Project work history effectively.

### Tag Jobs

Tagged Jobs facilitates better search and discoverability in the [Jobs dashboard](/cloud/platform-capabilities/core-concepts/jobs/monitor-jobs).

It’s best practice to start tagging Jobs from the start of a Project. For your tagging strategy, think about groups of Jobs you and collaborators would like to view in the Jobs dashboard.

For example, tagging your LLM pre-training Jobs differently than the instruction fine-tuning Jobs might provide for better search and visualization.

1. From the navigation pane, click **Jobs**.

2. Select the Jobs you want to tag, then click the Tag icon.

### Rename Jobs

Rename Jobs to clarify their purpose or highlight the importance of a particular Job.

1. From the navigation pane, click **Jobs**.

2. Click the **Edit** button next to the Job name.

### Stop a Job

1. From the navigation pane, click **Jobs**.

2. Go to the **Active** tab.

3. Select the checkboxes for the Jobs to stop and click the Stop icon.

### Re-run a Job

To re-run a Job:

1. From the **Jobs** page, mark the check box of the Run that you want.

2. Select either **Re-Run with Original Version** or **Re-Run with the Current Version**.

   Original and Current are referencing all inputs to the Job *except* the command input argument string. See the table below that outlines the behavior of the two re-run modes in both Git-based projects (GBP) and Domino File System (DFS) projects.

#### Git-based Projects

| | Command input arguments | Code in primary Git repository | Compute environment | Artifacts in DFS | Imported Git repositories |
| - | - | - | - | - | - |
| Re-Run with Original Version | Original | Original | Original | Original | Original |
| Re-Run with Current Version | Original | Current | Current | Current | Current |

#### Domino File System Projects

| | Command input arguments | Compute environment | Files in DFS | Imported Git repositories |
| - | - | - | - | - |
| Re-Run with Original Version | Original | Original | Original | Original |
| Re-Run with Current Version | Original | Current | Current | Current |

<Note>
  * When working in a GBP project, re-running a Job using **Re-Run with Current Version** only works as expected on an *original Job*. If you re-run a Job that was itself a re-run, the platform does not fetch the latest commit from the branch. It simply uses the commit ID from the re-run Job itself. This is because re-runs are tied to a specific commit, not the branch state.

  * In both DFS and GBP projects, the **Re-Run with Current Version** button only fetches the latest Git commits for your main or imported repositories if the original Job was run from a branch (like the default or any named branch). If the Job was originally run from a specific commit or tag, the re-run does not pull the latest Git commit.

  * In [restricted Projects](/cloud/admin/platform-configuration/manage-compute-environments/restricted-environments), Job re-runs use the current restricted revision of the environment. This revision may be different from the revision that was used to run the Job previously.
</Note>

## Next steps

Learn how to [schedule Jobs](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job) and [view Job results](/cloud/platform-capabilities/core-concepts/jobs/monitor-jobs).


# Create custom Job types
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/jobs/create-custom-jobs

Review native Job file handlers and invocation commands, then add custom handlers to Compute Environments through Dockerfile instructions.

Learn about Domino’s natively supported [Job](/cloud/platform-capabilities/core-concepts/jobs) types and languages, as well as how to create custom Job types to support your specific needs.

Domino has native support for running a wide variety of different types of Jobs, both in a single container and using on-demand, auto-scaling clusters. Additionally, Domino lets you customize the type of programming languages and applications that can be used in Domino Jobs to achieve nearly any type of computational workload. This flexibility lets you do things like large-scale data preparation, distributed GPU deep learning model training, and more.

## Natively supported Job types

The following lists the supported file types and the commands that Domino executes when it runs the file type:

| Language | Filetype/Keyword | Invocation |
| - | - | - |
| Python | `py` | `python -u` |
| R | `r` | `R --no-save --no-environ --no-site-file --no-restore --max-ppsize=100000` |
| Markdown | `rmd`/`rhtml` | `R -e require('knitr'); knit('file_name')` |
| Bash | `sh` | `bash file_name` |
| Matlab | `m`/`mlx`/`mexa64` | `matlab –sd $(dirname file_name) -batch` |
| Perl | `pl` | `perl file_name` |
| Julia | `jl` | `julia file_name` |
| Python notebook | `ipynb` | `ipython nbconvert -to notebook --execute file_name --output file_name` |
| SAS | `sas` | `sas file_name -stdio` |
| Stata | `do` | `stata -b do` |

## Custom Job types

Domino supports executing nearly any type of user application code to achieve a wide variety of computational scenarios by [editing an Environment definition](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition) to support your needs.

<Note>
  This feature is disabled by default. To use this feature, please reach out to Domino Support.
</Note>

### What is a file handler?

When you run a Job, Domino automatically handles the execution of the file. For natively supported file types, Domino already has commands it runs based on the file extension. To execute a file extension that isn’t natively supported, or to run a different command for a file extension that is in the list, you can use a file handler.

A file handler is a bash script with the name of the file extension (without the `.sh` extension), located in `/opt/domino/handlers/{extension}`.

### File handler examples

The file handler lives in the Environment’s file system, so the execution of the Job depends on the Environment in which you run the Job.

* The file handler `/opt/domino/handlers/jpeg` targets files with the `.jpeg` extension.

* The file handler `/opt/domino/handlers/py` overrides the default behavior for `.py` files.

<Tip>
  Always include instructions to install the necessary software and dependencies to execute the file type. For example, if you create a file handler to add support for Node.js, include a line to install `nodejs`:

  ```shell theme={null}
  RUN apt-get -y install nodejs
  ```
</Tip>

### Add a file handler through the Domino UI

To add a file handler through the Domino UI, [edit an Environment definition](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition).

1. Click **Environments** in the Domino sidebar.

2. In the **Dockerfile instructions** field, create a directory for the custom file handler:

   ```shell theme={null}
   RUN mkdir -p /opt/domino/bin/file-handlers
   ```

3. Add the commands you want to execute when you invoke the file. Save the custom commands to a file with the same name as the file extension you want to target:

   ```shell theme={null}
   RUN echo "#!/bin/bash" >> /opt/domino/bin/file-handlers/<target-file-extension>
   RUN echo "<Line 1 of bash script>" >> /opt/domino/bin/file-handlers/<target-file-extension>
   RUN echo "<Line 2 of bash script>" >> /opt/domino/bin/file-handlers/<target-file-extension>
   ...
   ```

### Add a file handler through the terminal

To add a file handler through the terminal, you must build a Docker image and upload it to an external repository. You can use the Docker image to [create an Environment](/cloud/admin/platform-configuration/manage-compute-environments/create-envs-from-external-custom-images) that contains your file handler.

1. Create a directory for the custom file handler:

   ```shell theme={null}
   mkdir -p /opt/domino/bin/file-handlers
   ```

2. Add the commands you want to execute when you invoke the file. Save the custom commands to a file with the same name as the file extension you want to target:

   ```shell theme={null}
   RUN echo "#!/bin/bash" >> /opt/domino/bin/file-handlers/<target-file-extension>
   RUN echo "<Line 1 of bash script>" >> /opt/domino/bin/file-handlers/<target-file-extension>
   RUN echo "<Line 2 of bash script>" >> /opt/domino/bin/file-handlers/<target-file-extension>
   ...
   ```

3. Save a copy of this file to a Docker image:

   ```shell theme={null}
   FROM {DOCKER_BASE_IMAGE}
   RUN mkdir -p /opt/domino/bin/file-handlers
   COPY <target-file-extension> /opt/domino/bin/file-handlers/
   ```

4. Build the Docker image and upload it to an external repository associated with your Project.

   ```shell theme={null}
   docker build -t "repository/domino-env-with-executable-target" .
   docker push "repository/domino-env-with-executable-target"
   ```

### Share a custom file handler

When you add custom file handlers to an Environment, the file handlers are associated with that Environment’s Docker image. Each Environment has its own Docker image. You can only have one Docker image per Environment.

If you must use your custom file handler in another Environment, you have several alternatives:

1. If the file handler is in the Dockerfile instructions field, you can copy and paste the handler’s commands into the same field in a different Environment.

2. If the file handler is in a Docker image in an [external repository](/cloud/admin/platform-configuration/manage-compute-environments/create-envs-from-external-custom-images):

   1. Use the Docker image as a base to build Docker images for any Environment that needs your custom file handler.

   2. Upload the file handler itself to an external repository. You can reference this file from multiple Docker images.

## Next step

Learn about all the ways you can [launch a Job](/cloud/platform-capabilities/core-concepts/jobs/create-and-run-jobs).


# Customize Job result views
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/jobs/customize-job-results

Add custom Job metrics with dominostats.json, filter result files with .dominoresults or .dominoignore, and configure result emails.

Domino offers a solution for capturing and visualizing Job metrics. While it’s recommended to use Domino’s MLflow implementation for this purpose, using Domino directly remains a popular choice. Users have the flexibility to log metrics in both places if they prefer.

1. Click **Jobs** from the Project menu.

2. Click the funnel icon above the table of Jobs to customize the columns in the Jobs dashboard.

## Add columns to the Jobs Dashboard with dominostats.json

To include metrics in the Jobs dashboard that Domino doesn’t provide by default, you can define custom columns. This can be done by using the `dominostats.json` file, letting you tailor the dashboard to better understand and analyze your Job performance.

1. Write a file named `dominostats.json` to the root of your project directory.

2. Use keys in `dominostats.json` to identify the outputs you’re interested in and add the corresponding values.

3. If Domino detects that `dominostats.json` has been written to the Project root by a Job, it parses the values and shows them as columns on the [Jobs dashboard](/cloud/platform-capabilities/core-concepts/jobs).

<Note>
  Domino automatically deletes `dominostats.json` before each execution.
</Note>

The following is sample R code that writes three key/value pairs to `dominostats.json`:

```r theme={null}
diagnostics = list("R^2" = 0.99, "p-value" = 0.05, "sse" = 10.49)
library(jsonlite)
fileConn<-file("dominostats.json")
writeLines(toJSON(diagnostics), fileConn)
close(fileConn)
```

The following is the same data written to `dominostats.json` in Python:

```python theme={null}
import json
with open('dominostats.json', 'w') as f:
    f.write(json.dumps({"R^2": 0.99, "p-value": 0.05, "sse": 10.49}))
```

The resulting `dominostats.json` from these code examples looks like this:

```json theme={null}
{
  "sse": 10.49,
  "R^2": 0.99,
  "p-value": 0.05
}
```

## Use .dominoresults and .dominoignore to render specific files in the Jobs dashboard

Use the `.dominoresults` and `.dominoignore` files to control which files Domino renders in the Jobs results view. This can be useful to limit your results if many files are changed in your execution, or to exclude unnecessary log files generated during the execution.

<Tip>
  R language exceptions can be especially large, which can impact sync times. So you may want to exclude the R debug outputs, depending on your use cases.
</Tip>

### Include or exclude single files in the results dashboard

To include or exclude only specific files, your `.dominoresults` or `.dominoignore` file must list the relative path to those, each on a new line. The following example excludes or includes `histogram.pdf` and `output.txt` in the top-level directory of the Project.

```shell theme={null}
histogram.pdf
output.txt
```

### Use patterns with wildcard characters

Domino can use wildcard patterns in `.dominoresults` and `.dominoignore`. This way you can specify groups of files to show. The following example limits the files to PDF documents in the top-level directory, all PNG files in a directory images and text files arbitrarily deep in sub-directories of the Project.

```shell theme={null}
*.pdf
results/*.png
**/*.txt
```

### Configure Job result views

Run results can also be configured to control things like the maximum number of files Domino renders for comparisons and diffs.

For more information, please reach out to Domino Support.

## View custom metrics as a time series over time

You can see how your custom keys change over time.

1. Click **Jobs Timeline** to expand a line chart of `dominostats.json` values over time.

   This chart shows all Jobs listed in the current dashboard. To filter to a specific set of related Jobs, [tag](/cloud/platform-capabilities/core-concepts/jobs) the Jobs to create a separate dashboard view.

The x-axis ticks on the timeline represent individual Jobs, and the y-axis represents the values for the statistics in those Jobs.

1. Hold your cursor over the chart to see individual data points as tooltips.

2. Click a point to open the details for the Job that produced that value.

3. Click and drag on the chart to zoom in.

4. Click each stat in the legend to toggle its line on and off.

## Email notifications

By default, Domino notifies you by email about *failed* Project executions. You and your Project collaborators can also receive notifications about *successful* executions.

On successful executions, you receive an email with the last few lines of `stdout`, and up to ten results files from your execution. Domino detects which files were added or changed during your execution, and captures those as the execution’s results.

### Edit notification preferences

1. In the navigation pane, click **Projects**.

2. Go to your Project and click **Settings**.

3. Click **Access and Sharing** and go to **Collaborations and permissions**.

4. Set the **Notifications preference** for users.

## Send custom emails

You can create fully customizable Job email notifications with any formatting and results you want.

For [Domino File System (DFS) projects](/cloud/platform-capabilities/core-concepts/projects/dfs-projects), create a file named `email.html` in the *root of your project folder* as part of your run.

For [Git-based projects](/cloud/platform-capabilities/core-concepts/projects/git-based-projects), create a file named `email.html` in the `/mnt/artifacts/` directory as part of your run.

The HTML in `email.html` can be used as the body and subject of the email sent on success.

### Tips and tricks

* To include images, reference the path to the image from the root of the folder. The image can be anywhere in your Project. For example, to include an image in `plots/plot1.png`, write `<img src="plots/plot1.png">`.

* Put all CSS styles in inline `style` attributes. Most email clients ignore `<style>` blocks in HTML emails.

* You can render a Jupyter Notebook to HTML and name it `emails.html` to send notebooks.

* Use tables for complex layouts. Most email clients ignore CSS positioning.

* Include the `<head>` and `<title>` tags at the start of the HTML file to customize the subject. For example: `<head><title>Custom Subject</title></head>` creates an email with the subject "\[Domino] Custom Subject".

* To explicitly define the files that are sent with your success email, create a file named `.dominoresults` and write a filename per line in the `.dominoresults` file.

  <Warning>
    If you create a `.dominoresults` file, you must list `email.html` in this file to ensure that your custom email is used. If you don’t, the default results email is generated.
  </Warning>

* For Git-based projects, write Job results to the `/mnt/artifacts/results/` directory. This allows the files to be included as attachments to the email and appear as results in the Job UI. See [Save Artifacts to the Domino File System](/cloud/platform-capabilities/core-concepts/projects/git-based-projects#save-and-pull-artifacts) for more info.

<Tabs>
  <Tab title="R">
    Example of an R script and the resulting email:

    ```r theme={null}
    # generate and save plot
    png("pressure.png")
    plot(pressure)
    dev.off()

    # generate HTML in a string
    html_string <- paste0("
    <head>
      <title>",Sys.Date()," - Pressure report","</title>
    </head>
    <body>
      <h2>Exponential pressure growth! </h2>
      <h3>",Sys.time(),"</h3>
      <img src='pressure.png' />
      <p>Caption goes here</p>
    </body>
    ")

    # write string to file
    write(html_string, "email.html")
    ```
  </Tab>

  <Tab title="Python">
    Example of a Python script:

    ```python theme={null}
    import matplotlib.pyplot as plt
    import numpy as np
    x = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8])
    y = np.array([495, 620, 761.88, 899.8, 1039.93, 1270.63, 1589.04, 1851.31, 2154.92])
    plt.axis([0, 8, 0, 2200])
    plt.xlabel('year')
    plt.ylabel('balance')
    plt.plot(x, y)
    plt.savefig('results/plot.png')
    email_content = """
    <head>
      <title>Latest projections</title>
    </head>
    <body>
      <h1>Latest projections</h1>
      <div>
          <img src="results/plot.png">
      </div>
    </body>
    """
    f = open('email.html', 'w')
    f.write(email_content)
    f.close()
    ```
  </Tab>
</Tabs>


# Work with Jobs
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/jobs/index

Use Domino Jobs for reproducible batch workloads, choose UI, scheduler, CLI, or API launch options, and monitor or customize results.

Domino Jobs are designed to execute code logic in a fully reproducible manner. Whereas Domino [Workspaces](/cloud/platform-capabilities/core-concepts/workspaces) are for interactive development, Jobs are for batch/headless workloads. Ideal when you need a fully contained and complete reproducibility package.

The result is a highly reproducible execution environment capable of handling a wide range of tasks including data processing, analytics, and machine learning. Jobs can handle single, simple tasks with intricate dependencies. Complex, multi-task workflows are possible, but [Domino Flows](/cloud/platform-capabilities/features/flows) is the recommended approach for such workflows, especially when they span different execution environments.

## Flexible Job run options

Domino offers numerous ways to [run Jobs](/cloud/platform-capabilities/core-concepts/jobs/create-and-run-jobs):

* **Launch from the UI**: Execute Jobs directly from the Jobs UI for a low-code option.

* **Job scheduler**: Use the [Job scheduler](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job) to easily run Jobs on-demand or at regular intervals.

* **Domino CLI**: Execute Jobs from the [Domino CLI](/cloud/reference/cli). This is typically used for iterative local development. For example, when using an IDE that can’t run in Domino, or you don’t want to use Domino compute resources to launch a Job.

* **Domino API**: Run Jobs using the [Domino API](/cloud/reference/api/domino-open-api). Often used when integrating with an external pipeline tool to launch Jobs using triggers.

## Job reproducibility

Key information about each Job is automatically captured to ensure the reproducibility and auditability of work. When you start a Job, Domino launches a new [Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments) for your code on the target hardware tier.

You can start multiple concurrent Jobs and each Job gets its own container environment, so you can try multiple parameters and techniques in parallel.

## Monitor Jobs and view results

[Monitor Jobs](/cloud/platform-capabilities/core-concepts/jobs/monitor-jobs) for a complete picture of your Jobs at various levels, from individual Job details to the broader view of how metrics evolve. Job monitoring supports reproducibility and auditability by capturing extensive Job context, including storage logs, hardware specifications, environment details, inputs, outputs, metadata, and custom metrics.

Quickly view your results directly in the Domino UI or [customize the results views](/cloud/platform-capabilities/core-concepts/jobs/customize-job-results) including notifications, reports, and custom emails to display your results elsewhere.

## Jobs vs Workspaces

Depending on the specific needs and nature of your work, you may want to use [Workspaces](/cloud/platform-capabilities/core-concepts/workspaces) instead of Jobs:

It’s best to use a **Job** under the following circumstances:

* **Model training & large-scale computations**: If you need to train models on a large dataset or run any sort of long-running, computationally intensive tasks.

* **Batch processing**: If you have tasks you want to run in the background and/or in parallel.

* **Reproducibility**: If strict reproducibility is important, Jobs provide a guarantee that the same execution Environment can be re-created.

* **Automation**: If the task needs to be run on a regular schedule or is part of a retraining or automated workflow.

It’s best to use a **Workspace** under the following circumstances:

* **Exploratory data analysis (EDA)**: If you are in the initial stages of a data science Project and need to explore and analyze your data interactively.

* **Rapid model iterations**: If you want to iterate rapidly on model development, hyperparameter tuning, or feature engineering and need rapid feedback with the ability to make instant adjustments.

* **Code development & debugging**: If you need an Environment to write, test, and modify code in real-time and see results immediately.

## Next steps

* Learn about the many flexible ways that you can [create and run Domino Jobs](/cloud/platform-capabilities/core-concepts/jobs/create-and-run-jobs) depending on your needs.

* Learn how to [schedule Domino Jobs](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job) in advance and set them to execute on a regular cadence.

* Find out more about how you can [monitor Jobs in Domino](/cloud/platform-capabilities/core-concepts/jobs/monitor-jobs).

* Find out how to [Customize Job result views](/cloud/platform-capabilities/core-concepts/jobs/customize-job-results) to capture and visualize Job metrics.

* Get more info about the [states of a Domino Job](/cloud/platform-capabilities/core-concepts/jobs/job-states) lifecycle.

* Learn how to [create custom Job types](/cloud/platform-capabilities/core-concepts/jobs/create-custom-jobs) to support your specific needs.

* Save money while training machine learning models using Domino hardware tiers based on [AWS Spot Instances](/cloud/platform-capabilities/core-concepts/jobs/spot-instances).


# Job states
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/jobs/job-states

Trace a Domino Job from Queued through Preparing, Running, and Finishing, and distinguish successful, stopped, failed, and error outcomes.

When you start a [Job](/cloud/platform-capabilities/core-concepts/jobs) in Domino, it moves through several states during its lifecycle.

* **Queued**: The execution is waiting for a machine of your specified hardware tier to become available. If one is available, it quickly leaves this state. However, if no slots are available, it can take several minutes to start up a new machine.

* **Scheduled**: The dispatcher requested an executor to process the execution and the executor acknowledges this request, but hasn’t begun processing. An execution only remains in this state for a few seconds.

* **Pending**: A Run is placed in the pending state before it starts running in Kubernetes.

* **Preparing**: Your Project files are being copied to the executor where your code runs. The time this takes depends on the size of your data and the number of files in your Project. Files are cached whenever possible. Therefore, if you start an execution on a hardware tier that you used recently with the same Project, this can be quick, even for Projects with large files.

* **Building**: If you are using a [custom environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments), you might have to wait for the Docker image to build. This is cached whenever possible, so subsequent executions on the same hardware tier might skip this step.

* **Pulling**: When your Docker image has been saved to a network-attached repository, this state indicates that Domino is fetching the image.

* **Running**: Your code is executing. You can view the console output and resource usage on the Jobs or Workspaces dashboard.

* **Finishing**: Your execution has completed. File changes are being copied back to the Domino file store.

* **Succeeded**: Your execution has finished without error.

* **StopRequested**: The system received the request to stop your execution.

* **Stopping**: If you manually stop your execution, it enters this state while new or updated files are synced back to the Project.

* **Stopped**: The execution has been manually stopped.

* **Failed**: Your execution did not complete due to a problem with your code.

* **Error**: Some problem outside your code caused the execution to terminate.


# Monitor Jobs
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/jobs/monitor-jobs

Monitor Jobs and inspect logs, results, metadata, and metrics, list Project Jobs through the API, compare runs, and archive completed work.

Monitoring Jobs in Domino is comprehensive and flexible, giving you a complete picture of your Jobs at various levels, from individual Job details to the broader view of how metrics evolve over time. Domino’s Job monitoring supports reproducibility and auditability by capturing extensive Job context, including storage logs, hardware specifications, environment details, inputs, outputs, metadata, and custom metrics.

The Jobs dashboard provides a way to view all Jobs in a given Project. Use it to monitor ongoing Jobs, view their results, and reproduce results.

<img alt="Job details" />

## Track and log Jobs with MLflow

It’s a Domino best practice to use [MLflow to log experiment metrics](/cloud/platform-capabilities/features/development/track-monitor) and other key metadata.

Domino implements MLflow and hardens the MLflow experience by providing [role-based access controls (RBAC)](/cloud/admin/identity-and-access/manage-users) on top to limit access to sensitive information.

## View Job results and details

You can see the information about a Job and its results on the Jobs dashboard. Job results are any files created or modified during the Job run.

Domino stores versioned key Job components to ensure reproducibility.

1. From the navigation pane, click **Jobs**. Use **Filter table** entries to filter the list of Jobs.

2. Select the row for a Job to open the Job details. You can filter the table by Job title with the Search box.

   <Warning>
     Domino retains up to 15 days worth of usage details. This applies to all executions (Jobs, Workspace sessions, and web Apps).
   </Warning>

3. Click the tabs to see details such as:

   * **Logs**: Download the **Setup Output**, **User Output**, **Deployment Logs**, and **Support Bundle**. Domino Support often requests these logs to review and resolve Jobs-related issues.

   * **Results**: See the results for successful Jobs. You can add comments to Job results.

## List Jobs programmatically

To retrieve Jobs from a Project outside the UI (for example, from a Workspace, notebook, or external script), call the `GET /api/jobs/beta/jobs` endpoint. See [`getJobs` in the Domino Platform API reference](/api-reference/jobs/get-jobs-for-a-project) for the full schema.

### From a Workspace or Job

Inside a Domino Workspace or Job, call the API through the [API proxy](/cloud/reference/api/domino-api-authentication#use-the-api-proxy-for-domino-api-authentication) at `DOMINO_API_PROXY`. The proxy injects the caller's JWT, so no `Authorization` header is required. `DOMINO_PROJECT_ID` is already set for the current Project.

```python theme={null}
import os
import requests

response = requests.get(
    f"{os.environ['DOMINO_API_PROXY']}/api/jobs/beta/jobs",
    params={
        "projectId": os.environ["DOMINO_PROJECT_ID"],
        "limit": 25,
        "offset": 0,
        "statusFilter": "all",  # all, queued, running, completed, archived, active
    },
)
response.raise_for_status()

for job in response.json().get("jobs", []):
    print(job["id"], job["status"]["executionStatus"], job.get("title"))
```

### From outside Domino

From a local machine or external system, call `DOMINO_API_HOST` directly and authenticate with a [Personal Access Token](/cloud/reference/api/domino-api-authentication#authenticate-with-a-personal-access-token) (or a [Domino Service Account token](/cloud/reference/api/domino-api-authentication#authenticate-with-a-domino-service-account-token) for unattended pipelines) passed as a bearer token.

```python theme={null}
import os
import requests

response = requests.get(
    f"{os.environ['DOMINO_API_HOST']}/api/jobs/beta/jobs",
    headers={"Authorization": f"Bearer {os.environ['DOMINO_API_TOKEN']}"},
    params={
        "projectId": "<your-project-id>",
        "limit": 25,
        "offset": 0,
        "statusFilter": "all",
    },
)
response.raise_for_status()
```

To retrieve details or logs for a single Job, call `GET /api/jobs/beta/jobs/{jobId}` or `GET /api/jobs/beta/jobs/{jobId}/logs`. These are beta endpoints with known limitations and require the `ViewJobs` permission.

## Compare Jobs

To create a [Jobs comparison report](/cloud/platform-capabilities/core-concepts/workspaces/manage-workspaces), select two Jobs in the table, then click **Compare**.

If you are tracking [execution performance](/cloud/platform-capabilities/core-concepts/jobs/customize-job-results), the comparison view shows the difference between your stats.

Domino snapshots the state of all the files in the Project before the Run starts (the inputs) and snapshots the Project after the Run completes (the outputs). Any files that were added or modified between the input and outputs are considered results.

Domino highlights the lines in text files that are different. Other types of files are rendered side-by-side for visual comparison.

If Domino can’t render a file, it provides simple metadata and links to download the files.

## Archive Jobs

You can archive Jobs by selecting the checkbox next to the related job and clicking **Archive**.

## Next steps

* [Customize Job results views and notifications](/cloud/platform-capabilities/core-concepts/jobs/customize-job-results).


# Schedule Jobs
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job

Schedule recurring Domino Jobs with branches, files, data, hardware, environments, clusters, cron expressions, notifications, and endpoint updates.

You can schedule [Domino Jobs](/cloud/platform-capabilities/core-concepts/jobs) in advance and set them to execute on a regular cadence. These can be useful when you have a data source that is updated regularly.

1. Go to a project.

2. From the navigation pane, click **Jobs** > **Schedules**.

3. Click **Schedule a Job** to open the **Create a Scheduled Job** page.

4. Define the Job:

   **Scheduled Job Name**\
   Enter the name of the Job. The Jobs Dashboard lists each Job by this name.

   **Branch or Commit Id**\
   In a [Git-based project](/cloud/platform-capabilities/core-concepts/projects/git-based-projects) you can specify a branch or commit ID when launching or editing a Job.

   **File Name**\
   Enter the name of the file to execute. Include any optional arguments to pass to the file.

   **Data Plane & Hardware**\
   Select the Hardware Tier used by the Job. The same tab lists the Datasets and NetApp Volumes the Job mounts, in a table with **Storage** and **Snapshot** columns. Pin an asset to a single snapshot from the **Snapshot** column. [Choose a snapshot per asset at launch](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots#choose-a-snapshot-per-asset-at-launch) covers the options.

   **Environment**\
   Select the compute environment used by the Job.

   **Optional**: Select the Environment revision. By default, Domino uses the latest revision of your Environment when the Environment starts.

   <Note>
     Domino recommends using the revision set as the active one by the project owner. If you select a different revision, the **Not Recommended** warning is shown.
   </Note>

   You can also configure your Job to create Snapshots of Datasets it mounts upon execution completion. This feature requires `AllowDatasetSnapshotsOnExecutionCompletion` to be enabled. See the [create a snapshot](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots) guide for more details.

   A Dataset pinned to a single snapshot mounts read-only, so a completion snapshot of it captures no changes.

5. Select and define the **compute cluster**.

   The configuration options for each cluster type are explained here:

   * [Spark Cluster Settings](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/work-with-your-cluster#understand-your-cluster-settings)

   * [Ray Cluster Settings](/cloud/platform-capabilities/features/compute-clusters-hpc/ray/work-with-your-cluster#set-your-cluster-settings)

   * [Dask Cluster Settings](/cloud/platform-capabilities/features/compute-clusters-hpc/dask/work-with-your-cluster#understand-cluster-settings)

   * [MPI Cluster Settings](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/work-with-your-cluster#create-an-mpi-cluster-with-jobs)

6. Set up the **schedule**.

   <img alt="Schedule step of the Create a Scheduled Job dialog" />

   **Use custom expression**\
   Enter a custom Quartz CronTrigger expression. For example, if you want to run the Job on the fifth minute of every day, enter the following:

   ```bash theme={null}
   `0 5 * ? * *`
   ```

   <Note>
     To learn more about these expressions, see [CronTrigger expressions](http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html).
   </Note>

   **Repeat every**\
   Set the frequency at which you want the Job to repeat.

   **Run sequentially**\
   When you set a Job to run sequentially, the scheduler always waits for the last Job it started to complete before starting the next one. For example, if you set up a scheduled Job to run once per hour, and one of the Jobs launched by the scheduler takes 90 minutes to complete, the next hourly Job does not start until the previous one has finished. Otherwise, multiple Jobs from this scheduler are allowed to run simultaneously. The scheduler does not wait for the previous Job to finish if it’s still running. This mode should be used when your Job doesn’t depend on output from the previous Job.

7. Set up any additional **actions**:

   <img alt="Actions step of the Create a Scheduled Job dialog" />

   **Notify emails**\
   Enter a list of email addresses to notify when the Job is completed.

   **Update Domino endpoint**\
   If a Domino endpoint has been published from the project, the selected Domino endpoint is republished after the Job has been completed. Use this for retraining and updating a Domino endpoint regularly.

   <Note>
     In [restricted Projects](/cloud/admin/platform-configuration/manage-compute-environments/restricted-environments), scheduled Runs use the current restricted revision of the Environment selected for the Job. This revision may be different for a scheduled Run from the revision used in a previously scheduled Run.
   </Note>

## Manage scheduled Jobs

From **Jobs** > **Schedules**, you can update or remove existing schedules. These actions require the **Create Scheduled Job**, **Edit Scheduled Job**, or **Delete Scheduled Job** permission granted to Project Owners and Contributors. See [Collaborator Permissions](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects) for details.

**Edit a schedule**\
Open the scheduled Job and change any field, including the cron expression, file name, hardware tier, environment, or notification settings. Saving applies the changes to the next scheduled run.

**Delete a schedule**\
Delete a scheduled Job to stop all future runs. Historical Job runs launched by the schedule remain in the Jobs Dashboard.

Domino does not have a dedicated pause action for scheduled Jobs. To stop future runs without losing the configuration, edit the schedule and set a far-future start time, then restore it when you want runs to resume.

<Note>
  Create and manage scheduled Jobs in the UI. The [public API reference](/cloud/reference/api) does not include endpoints for scheduled Jobs. The Job endpoints in the public API start on-demand Jobs only.
</Note>

## Next steps

* [Monitor and view Job results](/cloud/platform-capabilities/core-concepts/jobs/monitor-jobs).

* Automate complex pipelines with [Apache Airflow or Kubeflow](/cloud/platform-capabilities/features/integrations).


# Run Jobs on Spot Instances
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/jobs/spot-instances

Run fault-tolerant training Jobs on AWS Spot Instances, save checkpoints under /mnt/artifacts/checkpoints/, and resume after interruptions.

Domino makes it easy to save money while training machine learning models using Domino hardware tiers based on [AWS Spot Instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html). Amazon Spot Instances give you access to unused AWS EC2 compute instances at a discount over their on-demand prices.

## Spot instance interruptions

It’s important to note that Spot Instances can be interrupted, causing Jobs to take longer to start or finish, or even fail during the Job if the Spot Instance is reclaimed. Spot Instances are best used for various fault-tolerant and flexible applications.

To mitigate the impact of losing progress of a model training Job, we recommend you use [model checkpointing](https://pytorch.org/tutorials/recipes/recipes/saving_and_loading_a_general_checkpoint.html). Domino automatically reloads the model checkpoints and mounts them to a local file when the Job is restarted, enabling the training Job to resume from the last checkpoint instead of restarting completely.

If AWS interrupts a Spot Instance, the Job may fail to start or complete. If this happens, we suggest retrying the Job after some duration (more than 30 minutes). If the issue persists, the remediation is to change the hardware tier of the Job to use a non-spot node pool.

## Use model training checkpoints

Domino Jobs run in containers, so [model checkpointing](https://pytorch.org/tutorials/recipes/recipes/saving_and_loading_a_general_checkpoint.html) files are saved under a local directory of the containers (we recommend that you save checkpoints at `/mnt/artifacts/checkpoints/`). Domino synchronizes artifacts, including model checkpoints, to external storage. Existing checkpoints in S3 are mounted to the container at the start of the Job, enabling Jobs to resume from a checkpoint. Checkpoints added to the S3 folder after the Job has started are not copied to the training container.

Use checkpoints to do the following:

* Save your model snapshots under training due to an unexpected interruption to the training Job or instance.

* Resume training the model in the future from a checkpoint.

* Analyze the model at intermediate stages of training.

* Use checkpoints with Spot Instances to save on training costs.

### Checkpointing considerations

Consider the following when using checkpoints in Domino:

* To avoid overwrites in distributed training with multiple instances, you must manually configure the checkpoint file names and paths in your training script accounting for the rank of each worker node in your cluster.

* To control the checkpointing frequency, modify your training script using the framework’s model save functions or checkpoint callbacks.


# Collaborate on Projects
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects

Invite collaborators with the right role, understand what each role can do, and discuss work with comments.

## Overview

You share a Project by adding collaborators to it. Each collaborator gets a role that determines what they can do, from read-only access to results up to full contributor rights. Comments on files, Jobs, and the activity feed keep the discussion attached to the work itself.

## How it works

Two settings control who can do what in a Project:

* **[Visibility](/cloud/platform-capabilities/core-concepts/projects/project-settings#set-project-visibility)** determines who can find the Project.
* **Collaborator roles** determine what each invited user can do. Project roles are independent of a user's role in Domino.

<Note>
  **Best practice:** Audit your collaborators regularly and keep permissions as locked down as possible. Verify that each user has the permissions their work needs and no more, and use the visibility setting to limit who can see the Project.
</Note>

## Invite collaborators

You must be a contributor or the Project owner to add collaborators.

1. In the Project, go to **Settings** > **Access & Sharing**.
2. In the **Collaborators and permissions** section, enter a username, email address, or organization name, then click **Invite**. An email address that doesn't belong to a Domino user sends an invitation to join Domino.
3. Set the collaborator's **Role**:
   * **Contributor**: read and write Project files, start runs, read and write environment variables, invite collaborators, and change the default Hardware Tier and Environment.
   * **Results Consumer**: read files and access [published Apps](/cloud/platform-capabilities/features/apps) only.
   * **Launcher User**: view and run Launchers and see their results only; no access to Project files.
   * **Project Importer**: [import the Project](/cloud/platform-capabilities/core-concepts/projects/manage-projects#export-and-import-project-content), but no other access.
   * **Owner**: the Project's owner (not assignable here). Only owners can archive the Project, change the owner, change collaborator types, share environment variables and files between Projects through import and export, or set automatic Workspace shutdown times.
4. In **Notifications preference**, choose how to [notify](/cloud/platform-capabilities/core-concepts/jobs/customize-job-results#email-notifications) the collaborator when runs complete.

## Collaborator permissions

These tables list the exact permissions for each role.

### File permissions

| Action | Project Importer | Launcher User | Results Consumer | Contributor | Owner |
| - | - | - | - | - | - |
| Read files | | | x | x | x |
| Write files | | | | x | x |
| Add an external Git repository | | | | x | x |

### Run permissions

| Action | Project Importer | Launcher User | Results Consumer | Contributor | Owner |
| - | - | - | - | - | - |
| Start a Run | | | | x | x |
| View Completed Runs | | x | x | x | x |
| Start a Workspace<sup>1</sup> | | | | x | x |
| View Workspaces | | | x | x | x |
| Schedule a Run | | | | x | x |
| View Scheduled Runs | | | x | x | x |

<sup>1</sup> Start a Workspace: A user can start a workspace they've created.

### Publishing permissions

| Action | Project Importer | Launcher User | Results Consumer | Contributor | Owner |
| - | - | - | - | - | - |
| Run a Launcher | | x | | x | x |
| View an App | | x | x | x | x |
| Publish an App | | | | x | x |
| Unpublish an App | | | | x | x |
| Invite users to an App | | | | x | x |
| Change an App hardware tier | | | | x | x |
| Publish a Domino endpoint | | | | x | x |
| Create a Launcher | | | | x | x |

### Settings permissions

| Action | Project Importer | Launcher User | Results Consumer | Contributor | Owner |
| - | - | - | - | - | - |
| Set an environment variable | | | | x | x |
| Invite a collaborator | | | | x | x |
| Manage collaborator permissions | | | | x | x |
| Change the visibility setting | | | | | x |
| Change the default environment | | | | x | x |
| Change the default hardware tier | | | | x | x |
| Change the project name | | | | | x |
| Transfer project ownership | | | | | x |
| Archive a project | | | | | x |

### Dataset permissions

See [Dataset roles](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-datasets#dataset-roles) for more information.

| Action | Project Importer | Launcher User | Results Consumer | Contributor | Owner |
| - | - | - | - | - | - |
| Create a new Dataset in a project | | | | x | x |
| Mount a shared Dataset in a project | | | | x | x |

### Import permissions

| Action | Project Importer | Launcher User | Results Consumer | Contributor | Owner |
| - | - | - | - | - | - |
| Import a project | x | | | x | x |

### Experiment permissions

| Action | Project Importer | Launcher User | Results Consumer | Contributor | Owner |
| - | - | - | - | - | - |
| Create a new experiment or experiment run | | | | x | x |
| View, list, or search experiments or runs | | | x | x | x |
| Update an experiment or run (includes logging to a run or adding tags) | | | | x | x |
| Archive an experiment or run | | | | Only the original owner | x |

## Add comments

Use comments to document decision rationale, analysis, suggestions, and next steps where the work happens:

<Tabs>
  <Tab title="Files">
    In the Project, go to **Files**, open the file, enter your comment in the **Discussion** box at the bottom, and click the send icon.
  </Tab>

  <Tab title="Jobs">
    Go to **Jobs**, open the Job, and comment on the **Comments** tab (or on the **Results** tab to discuss its results) in the right pane.
  </Tab>

  <Tab title="Job comparisons">
    On **Jobs**, select two Jobs, click the **Compare** icon in the navigation tabs, scroll to the **Discussion** box, and click **Comment**.
  </Tab>

  <Tab title="Activity feed">
    Go to **Activity** and click **Add Comment** on entries about Jobs and Workspaces. Comments by anyone also appear as feed entries.
  </Tab>
</Tabs>

Comments support [Markdown](https://www.markdownguide.org/cheat-sheet/) for formatting and [MathJax](https://www.mathjax.org/) (LaTeX notation) for mathematical symbols and equations (see the [MathJax quick reference](https://math.meta.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference)); a preview appears below the **Discussion** box.

To notify someone, type `@` followed by their username in the comment. You can notify the Project's collaborators. An [organization](/cloud/platform-capabilities/features/collaboration/manage-organizations) added as a collaborator grants that access to every member, so you can notify any of them.

<Tip>
  * Check **Notify me when others add collaborators to projects I own** under **Collaboration and Sharing** in **Account Settings**, and encourage your collaborators to do the same.
  * In a **Searchable** Project, use descriptive keywords so others can find your work through [search](/cloud/platform-capabilities/features/collaboration/domino-search).
</Tip>

## View Project activity

The **Activity** feed (under **Overview** in the Project menu) records the Project's history: Jobs and Workspaces started, and comments on Jobs, Workspaces, and files. Filter the feed by type, and download it as a PDF. Select two completed runs in the feed to [compare them](/cloud/platform-capabilities/core-concepts/reproducibility/compare#compare-project-states-from-the-activity-feed).

A read-only Status badge next to the Project name shows the Project's current status.

## View event notifications

Administrators send notifications about Domino events such as planned maintenance or reminders to save your work. The **Notifications** panel is at the bottom right of the Domino home page and refreshes every 30 seconds. Click **View All** to see the past 30 days, filter by priority, and click **Mark all as read** to acknowledge them; messages sort chronologically with critical unread first, then default unread, then read and expired. Expired messages are tagged EXPIRED and disabled, and deleted messages disappear from your list.

## Related

* [Project settings](/cloud/platform-capabilities/core-concepts/projects/project-settings): visibility and environment variables.
* [Manage Projects](/cloud/platform-capabilities/core-concepts/projects/manage-projects): copies, ownership transfer, and sharing content between Projects.


# Create Projects
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/projects/create-projects

Start a Domino Project from scratch, from a template, or from existing work, and set up the Git credentials that Project creation needs.

## Overview

You can create a Project from scratch, from a Project template, or from existing work by copying another Project. Starting from existing work is often fastest: [search your Domino deployment](/cloud/platform-capabilities/features/collaboration/domino-search) for Projects you can build on, and check the tabs on your **Project's** home page for Projects where you are a collaborator and Projects that Domino recommends.

## How it works

* When you create the Project, you [choose where the code lives](/cloud/platform-capabilities/core-concepts/projects/index#project-types). Point Domino at your own Git provider for a Git-based Project; this is the usual choice.
* Git-based Projects with private repositories need Git credentials stored in your Domino account so Domino can authenticate with your Git provider; public repositories don't. Domino supports HTTPS and SSH connections for both.
* Project templates are supported for Git-based Projects only.

<Note>
  No Git provider? Choose **Domino (local hosting)** to store code in Domino's internal repository. See [Domino File System Projects](/cloud/platform-capabilities/core-concepts/projects/dfs-projects).
</Note>

## Add Git credentials

If you are connecting a private repository, want to write commits back to the remote, or use SSH, add Git credentials to your Domino account first. Domino stores SSH private keys, personal access tokens (PATs), and usernames with passwords.

<Tabs>
  <Tab title="SSH key">
    Use an SSH private key when the repository URI has the form:

    ```
    <user>@<domain>:<username>/<repository>.git
    ```

    Set up SSH access with your Git service first: [GitHub](https://help.github.com/articles/connecting-to-github-with-ssh/), [GitLab](https://docs.gitlab.com/ee/user/ssh.html), or [Bitbucket](https://confluence.atlassian.com/bitbucket/set-up-an-ssh-key-728138079.html).

    1. Copy your entire private key, including the `-----BEGIN OPENSSH PRIVATE KEY-----` and `-----END OPENSSH PRIVATE KEY-----` header and footer. If you used the standard name and location, the key is in `~/.ssh/id_rsa`.
    2. In Domino, go to **Account** > **Account Settings** > **Git Credentials** and click **+ Add Git Credentials**.
    3. Choose the Git service provider, name the credential, and select **Private SSH Key** as the access type.
    4. Paste the private key. If the key requires a passphrase, enter it.
    5. Click **Add Credentials**.

    <Note>
      If your GitHub organization requires single sign-on (SSO), you must [authorize the SSH key for SSO](https://help.github.com/articles/authorizing-an-ssh-key-for-use-with-a-saml-single-sign-on-organization/) before Domino can access private repositories.
    </Note>
  </Tab>

  <Tab title="Personal access token">
    Use a PAT when the repository URI has the form:

    ```
    https://<domain>/<user>/<repository>.git
    ```

    Create the token with your Git service first:

    * [GitHub PATs](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/): grant at least the `repo` and `read:user` scopes. To create a repository from Domino, also grant `api` access. Organizations that require SSO must [authorize the PAT for SSO](https://help.github.com/articles/authorizing-an-ssh-key-for-use-with-a-saml-single-sign-on-organization/).
    * [GitLab PATs](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html): grant at least the `read_api` and `write_repo` scopes. To create a repository from Domino, also grant `api` access.
    * Bitbucket Cloud over HTTPS requires a [Bitbucket app password](https://confluence.atlassian.com/bitbucket/app-passwords-828781300.html) instead of a PAT.
    * Bitbucket Data Center requires a [personal access token](https://confluence.atlassian.com/bitbucketserver/personal-access-tokens-939515499.html) with at least project and repository read and write permissions. Provide your Bitbucket Data Center username alongside the token; Data Center rejects the token-only auth pattern used by Bitbucket Cloud.

    Then add the token to Domino:

    1. Go to **Account** > **Account Settings** > **Git Credentials** and click **+ Add Git Credentials**.
    2. Choose the Git service provider, name the credential, and select **Personal access token (PAT)** as the access type.
    3. Paste the token and click **Add Credentials**.
  </Tab>

  <Tab title="Username and password">
    For other Git service providers, including AWS CodeCommit and Azure DevOps, Domino supports authentication with an app username and password.
  </Tab>
</Tabs>

## Create a Project

<img alt="Create a new Project" />

1. **Start the Project wizard**:\
   In the navigation pane, click **Develop** > **Projects**, then click **Create Project**.

2. **Name and configure the Project**:\
   Enter a Project name, select a Project template or **None (start from scratch)**, set the Project's **Visibility**, and click **Next**.

3. **Choose where the code lives**:\
   Select your Git provider for a Git-based Project, or **Domino (local hosting)** for a DFS Project.

4. **Connect the repository (Git-based only)**:\
   Select your credentials; click **+ Add credentials** if none are listed. Then point Domino at the repository: with SSH credentials, enter the Git repository URL; with PAT credentials, you can also have Domino create the repository or pick one from your provider.

5. **Create the Project**:\
   Click **Create**. The Project is ready for you to add files and start work.

<Warning>
  If the repository contains files larger than 2 GB, Domino still creates the Git-based Project, but workspace setup can fail. In that case, consider a DFS Project. Use [git-sizer](https://github.com/github/git-sizer) to check the total repository size and the size of individual files.
</Warning>

Project owners can set a default Git branch in [Project settings](/cloud/platform-capabilities/core-concepts/projects/project-settings). The selected branch auto-fills across the Project and can be changed anytime.

## Create a Project from a template

Project templates let you start from a working prototype instead of from scratch, which speeds up setup and spreads your organization's standards. Templates marked **official** are approved by an admin or librarian.

<Note>
  For templates stored in GitHub or GitHub Enterprise, the code repository must be marked as a [template repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-template-repository) before Projects can be created from it.
</Note>

1. On the **Projects** page, open the **Templates** tab. Filter by **Official templates** to see approved ones.
2. Click the template you want to start from.
3. Choose where to store the code:
   * **New repository**: Domino creates a repository and copies the template code into it.
   * **Existing repository (empty)**: Domino copies the template code into the repository. A `README.md` is overwritten; any other existing file makes the operation fail.
   * **Existing repository (with code)**: Domino includes only the template's Project assets and does not copy code files.

## Create a template

To turn one of your Projects into a template:

1. Open the Project and click **Create Template**. For GitHub and GitHub Enterprise, the button only appears if the Project's code repository is a [template repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-template-repository).
2. Enter a template name, an optional description, and the access level.
3. Select the assets to copy into the template, and optionally set a default billing tag, default Environment, and default Hardware Tier. Activities, reviews, Workspaces, Jobs, Experiments, and comments can't be copied into a template.
4. Choose where to store the template's code files: the provider, credentials (only PAT credentials are supported), and the repository. An existing repository must be empty; a `README.md` is overwritten, and any other file makes the operation fail.
5. Create the template.

To mark a template as official, open the **Templates** tab, hover over the template, click **Edit Details**, and check **Mark this as an official template**. Only an admin or librarian can set this flag; once a template is official, practitioners can no longer modify or delete it.

## Related

* [Manage Projects](/cloud/platform-capabilities/core-concepts/projects/manage-projects): copy an existing Project instead of creating one.
* [Project settings](/cloud/platform-capabilities/core-concepts/projects/project-settings): defaults for hardware, Environments, and access.
* [Git-based Projects](/cloud/platform-capabilities/core-concepts/projects/git-based-projects): how code syncs after you create the Project.


# Domino File System Projects
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/projects/dfs-projects

How Domino File System Projects version files, and how to upload, compare, revert, exclude, and package Project files or attach external Git repositories.

## Overview

Domino File System (DFS) Projects store your files in the DFS, Domino's traditional storage for Project assets. Domino versions your files automatically with a Git server it runs in the background. Every time you sync changes, Domino records a new Project version. Choose a DFS Project if you or your collaborators aren't comfortable with Git, or if your organization's policies require it; otherwise Domino recommends [Git-based Projects](/cloud/platform-capabilities/core-concepts/projects/git-based-projects).

## How it works

A DFS Project organizes assets into two sections in the navigation pane:

* **Code** lists your Project files, files imported from other Projects, and files in attached Git repositories. Unlike Git-based Projects, DFS Projects keep artifacts here too.
* **Data** lists the data sources used in the Project: [Domino Datasets](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets), external data volumes, and Dataset scratch spaces.

### The working directory

When you start a run, Domino loads the Project's files into the working directory, `/mnt` by default. Domino sets the `DOMINO_WORKING_DIR` environment variable for all runs; use it instead of hardcoding paths, because [importing another Project](/cloud/platform-capabilities/core-concepts/projects/manage-projects#export-and-import-project-content) changes the path to `/mnt/<project-owner-username>/<project-name>`. Dataset volumes for your home Project mount at `/domino/datasets/local/<name>`; see [Dataset paths in DFS Projects](/cloud/platform-capabilities/core-concepts/data/datasets/use-datasets-and-snapshots#dataset-paths-in-dfs-projects). For how these paths compare with a Git-based Project, see [File paths at a glance](/cloud/platform-capabilities/core-concepts/projects/files-artifacts-and-results#file-paths-at-a-glance).

Alongside your own files, the working directory holds files that Domino generates or reads:

```console theme={null}
DOMINO_WORKING_DIR/
├── ipynb_checkpoints   # Contains the auto-saved Jupyter states.
├── results             # Contains your generated results.
    └── stdout.txt      # Tail of the console output from your Run.
├── requirements.txt    # Optional: Specifies Python package dependencies.
├── .dominoresults      # Determines which files are rendered as results.
├── .dominoignore       # Contains file patterns for Domino to ignore.
├── .dominokeep         # Prevents directory removal of empty directory.
├── dominostats.json    # Values shown in the Jobs dashboard.
├── email.html          # Controls the formatting of notification emails.
├── .noLock             # Solves "too many open files" messages.
├── app.sh              # Contains app-launching code for Domino Apps.
├── domino.log          # Contains CLI logs for local CLI projects only.
└── .domino.vmoptions   # Contains proxy settings for local CLI projects only.
```

For details on individual files, see [.dominoresults](/cloud/platform-capabilities/core-concepts/projects/files-artifacts-and-results#special-files), [requirements.txt](/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments/use-requirements-txt), [.noLock](/cloud/platform-capabilities/core-concepts/projects/files-artifacts-and-results#special-files), [email.html and dominostats.json](/cloud/platform-capabilities/core-concepts/jobs/customize-job-results), and [domino.vmoptions](/cloud/reference/cli/use-the-cli-behind-a-proxy).

## Upload files

You can drag and drop files up to 550 MB into the Domino web application. Contact your Domino administrator to change this limit.

For larger files, use one of these routes:

* **Domino CLI**: [install the CLI](/cloud/reference/cli/install-the-cli), then in the local folder holding your files run `domino restore` to associate the folder with a Project (this downloads the Project's current files), and `domino sync` to upload your local files.
* **Download from a URL**: if the file is reachable over the web, save a shell script in your **Code** tab that fetches it with `wget`, and run the script once in Domino.

<Note>
  With large data (tens of GB), runs spend longer in the **Preparing** stage while files copy to the executor. For large data Projects, Domino natively integrates with Hadoop, Spark, and other big data platforms; contact [support@dominodatalab.com](mailto:support@dominodatalab.com) for more information.
</Note>

## Compare file revisions

To see what changed between versions of a file, open the file from the **Code** section and click **Compare Revisions**. Set the starting version in **Base** and the version to compare in **Target**. On the Project's **Code** page you can select among all Project revisions; when viewing an individual file, the menu lists only revisions that changed that file.

## Revert Projects and files

You can return a whole Project or a single file to an earlier version. Reverting creates a new version at the top of the history, so the version you reverted from stays accessible; this makes it cheap to recover from failed experiments.

* **Revert a Project**: In the **Code** section, open the **Commit** menu, which lists every version with the current one first. Click an older version to view it, then click **Revert Project**.
* **Revert a file**: In the **Code** section, open the file, pick an older version from the **Commit** menu, and click **Revert File**. Other files in the Project keep their current versions.

## Exclude files from sync

To keep files out of the [domino sync](/cloud/reference/cli/cli-reference#synchronize-the-files-on-your-computer-with-the-server) operation, list them in the `.dominoignore` file that Domino creates at the root of every new Project. This applies whether you sync from your local machine (CLI or [R package](/cloud/reference/r-package/domino-r-package)), at the end of a run, or during a Workspace session. Add a file name or a folder name (which excludes its contents); use `*` as a wildcard, with all paths relative to the Project root. A `.git/` directory is always ignored, even if not listed.

Two other patterns help when `.dominoignore` isn't enough:

* **Isolated results branches**: configure the Project to save results to a separate branch. The results stay accessible from the application and CLI, but `domino sync` doesn't download them and future runs don't use them as input unless you say so. See [Files, artifacts, and results](/cloud/platform-capabilities/core-concepts/projects/files-artifacts-and-results).
* **Project importing**: put your code in one Project and [import it](/cloud/platform-capabilities/core-concepts/projects/manage-projects#export-and-import-project-content) into a second Project where the large-output runs happen. You can sync the code Project without downloading the outputs, and even export the code Project as a package so it installs automatically at runtime.

## Export files as a Python or R package

If a Project's files are organized as an installable package, you can export them as one: any Project that imports it gets the package automatically installed at runtime. [Enable exports](/cloud/platform-capabilities/core-concepts/projects/manage-projects#export-and-import-project-content) for the Project, then select the language under **Code Package**.

<Tabs>
  <Tab title="Python">
    See [Packaging Python Projects](https://packaging.python.org/tutorials/packaging-projects/#generating-distribution-archives). In summary, a Python package requires:

    * A `setup.py` file containing a `setup()` function imported from `setuptools`.
    * A folder containing your Python modules and packages, usually named after the overall package.
    * Domino recommends also including a `README` file.
  </Tab>

  <Tab title="R">
    See the [official manual on writing R extensions](https://cran.r-project.org/doc/manuals/r-release/R-exts.html). In summary, an R package requires:

    * A directory called `R/` with code files, and a directory called `man/` with documentation files.
    * A `DESCRIPTION` file with one `key: value` pair per line. Required keys: Package, Version (for example `0.1`), Title, Description, Author, Maintainer (a name followed by an email address in angle brackets), and License.
    * A `NAMESPACE` file describing the package's namespace; `exportPattern( "." )` works in many cases.
  </Tab>
</Tabs>

## Import Git repositories

You can attach external Git repositories to a DFS Project. Repositories added to the Project are cloned into `/repos` in every run and Workspace you start, so your code can use their contents just like your Domino files. Domino supports HTTPS and SSH connections for both public and private repositories. For private repositories, SSH, or writing commits back, first [add Git credentials](/cloud/platform-capabilities/core-concepts/projects/create-projects#add-git-credentials) to your Domino account.

<Note>
  If you want to work primarily from a Git provider of your choice, use a [Git-based Project](/cloud/platform-capabilities/core-concepts/projects/git-based-projects) instead of importing repositories into a DFS Project.
</Note>

### Add a repository to a Project

1. Open the Project, click **Code**, then open the **Git Repositories** tab and click **Add a New Repository**.
2. Enter a directory name (the directory in `/repos` the repository clones into; defaults to the repository name) and the HTTPS or SSH URI.
3. Select what Domino checks out when it clones the repository into a run or Workspace: **Use default branch** checks out your Git service's default branch (typically `master`), or specify a branch name, tag name, commit ID, or custom Git ref.
4. Click **Add Repository**.

Your Domino working directory `/mnt` and `/repos` are siblings at the filesystem root, so scripts stored as Domino files reach repository contents through absolute paths like `/repos/<repo-name>/<file>`.

### Change a repository's default reference

To change what Domino checks out after a repository is added, send a `PUT` request to `/api/projects/v1/projects/{projectId}/repositories/{repositoryId}/default-ref`. For a Git-based Project, pass `_` as the repository ID to update its main repository. You need `Edit` permission on the Project, and runs and Workspaces started after the update check out the repository at the new reference.

```python theme={null}
import requests

project_id = 'asdf1234'
repository_id = 'qwer5678'  # Use '_' for the Project's main repository.
url = f'https://host.com/api/projects/v1/projects/{project_id}/repositories/{repository_id}/default-ref'
body = {'refType': 'branch', 'value': 'develop'}
headers = {'X-Domino-Api-Key': 'YOUR_API_KEY'}
response = requests.put(url, json=body, headers=headers)
response.raise_for_status()
```

Set `refType` to `head` to use your Git service's default branch, or to `branch`, `tag`, or `commit` with the branch name, tag name, or commit ID in `value`. The response echoes the reference that is now set.

### Pin a repository to a commit ID

Specifying a commit ID checks out that fixed point in the repository's history on every run, in a detached HEAD state; unlike a branch, it never evolves. Git tracks changes you make and you can commit them as usual, but a push from a detached HEAD fails because Git needs a branch to push to.

Domino also can't push automatically to a repository pinned this way. To publish changes, create a local branch from the detached HEAD in the Workspace command line, check it out, and run `git push -u origin`.

### Commit back to a repository

In a Workspace session, repositories added to the Project are listed in the **Session Overview** under **Git repos**, with your changes itemized file by file. To commit them, select the checkbox next to the repository and click **Full Sync**, then enter a commit message. The message attaches to the Git commits and, if Domino files also changed, to a new revision of the Project. Commits push to the default branch you chose when adding the repository. Stopping a Workspace with uncommitted repository changes prompts you the same way: select the repositories, enter a message, and click **Stop and Commit**.

If your changes conflict with the remote default branch, Domino creates a new branch from the local state and pushes that branch to the remote. Resolve the conflicts outside Domino or on the Workspace command line; the next session checks out the remote default branch again, not the pushed branch.

You can also work with the repositories directly: Jupyter and RStudio Workspaces both have command lines (Jupyter: **New** > **Terminal** from the **Files** tab; RStudio: **Tools** > **Shell…**) where standard Git commands work in `/repos`.

To audit what a run did to its repositories, expand the **Repositories** panel at the bottom of the run's **Details** tab; Domino records the checked-out commit at the start and end of the run.

### Troubleshoot repository access

* `Credentials are required for your repository`: the Git credential in Domino may have the wrong domain. Check that the credential's domain field matches your repository URL exactly (`github.com`, `bitbucket.com`, or your internal Git host).
* `The repository provided requires credentials but none were found`: confirm your SSH key or PAT is in **Git Credentials** in your account settings, and if your organization requires SSO, [authorize the key for SSO](https://help.github.com/articles/authorizing-an-ssh-key-for-use-with-a-saml-single-sign-on-organization/).
* `Invalid username or password` / `Authentication failed`: usually the same SSO authorization issue; authorize the credential for your organization.

## Related

* [Manage Projects](/cloud/platform-capabilities/core-concepts/projects/manage-projects): copy DFS Projects.
* [Files, artifacts, and results](/cloud/platform-capabilities/core-concepts/projects/files-artifacts-and-results): how Project files version and where results land.
* [Domino CLI reference](/cloud/reference/cli/cli-reference): sync commands used with DFS Projects.


# Files, artifacts, and results
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/projects/files-artifacts-and-results

How Domino versions Project files, where artifacts and execution results live, and how to bring existing data into a Project.

## Overview

Domino manages a collection of files for every Project. Your code and small supporting files are Project files; **artifacts** are files that are neither source code nor data, typically outputs like plots, charts, and serialized models; **results** are the files an execution generates or modifies when it runs. Results and artifacts suit small outputs; anything large or numerous belongs in [Datasets](/cloud/platform-capabilities/core-concepts/data/datasets), not Project files.

## How it works

You add files to a Project by uploading them from the web application, the [Domino CLI](/cloud/reference/cli/cli-reference), or the API; by creating and editing them in the web application; or by generating them from code in a Workspace or Job.

Every change to a Project's files creates a new revision of the Project. When you start an execution, Domino loads the Project's files onto the machine hosting it (the executor) and mounts them at `/mnt`. Domino tracks changes to that directory and records them as a new revision when the run completes. External Git repositories added to the Project appear under `/repos` on the executor; see [DFS Projects](/cloud/platform-capabilities/core-concepts/projects/dfs-projects#import-git-repositories).

In your code, reference **Code** files at `/mnt/code` and **Artifacts** at `/mnt/artifacts`; for example, a file under **Artifacts** called `job_output.json` is available at `/mnt/artifacts/job_output.json`.

<Tip>
  Replace `/mnt` with the `$DOMINO_WORKING_DIR` environment variable to keep your code portable, especially when Projects are [exported and imported](/cloud/platform-capabilities/core-concepts/projects/manage-projects#export-and-import-project-content).
</Tip>

Who can see Project files is controlled by the Project owner through [visibility and collaborator roles](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects). The most secure configuration is Private visibility plus collaborators with the Results Consumer role: read-only access for specific users, invisible to everyone else. Use it for Projects with the most sensitive data and results.

## File paths at a glance

The two Project types expose Project files at different paths in a Workspace or Job. Reference these locations in your code:

| Content | Git-based Project | DFS Project |
| - | - | - |
| Working directory (your code) | `/mnt/code` | `/mnt` |
| Artifacts | `/mnt/artifacts` (stored in the DFS) | Alongside your code, in the working directory |
| Project Datasets | `/mnt/data/<dataset-name>` | `/domino/datasets/local/<name>` |
| Imported or attached Git repositories | `/mnt/imported/code/<repo-name>` | `/repos/<repo-name>` |

In a DFS Project, reference the working directory through the `$DOMINO_WORKING_DIR` environment variable rather than hardcoding `/mnt`, because [importing another Project](/cloud/platform-capabilities/core-concepts/projects/manage-projects#export-and-import-project-content) changes the path to `/mnt/<project-owner>/<project-name>`. The DFS working directory also holds several [special files](#special-files) that Domino reads or generates.

When a Python script runs from an imported Project, relative paths resolve against the current working directory rather than the imported file's own location, and can fail with `FileNotFoundError: [Error 2] No such file or directory:`. Build the path from the script's own location instead. For example:

```python theme={null}
import os
file_name = os.path.join(os.path.dirname(__file__), 'your_referenced_file.dat')
```

## Special files

Domino reserves several files that control revisioning, results display, and run comparison:

* **`README`**: Every Project starts with one, shown on the **Overview** page and written in Markdown. To reference other Project files from the `README`, prepend `raw/latest` to the file's relative path (for example, `raw/latest/images/overview.jpg`).
* **`.dominoignore`**: Works like a `.gitignore`. File patterns listed here are excluded from new revisions, whether created by CLI sync, a run, or a Workspace session. Patterns can be filenames, folder names (which exclude their contents), or UNIX shell expressions with `*` wildcards, relative to the Project root. A `.git/` directory is always ignored even if unlisted. The default `.dominoignore` contains commented examples.
* **`.dominoresults`**: Controls which files appear in the results for the Project's runs. It lists patterns to *include*: with no patterns, every file a run changed appears; with patterns, only matching files do. For example, `*.pdf` and `results/*.png` on separate lines list all PDFs plus PNGs in the `results/` folder.
* **`dominostats.json`**: Read by the [compare jobs](/cloud/platform-capabilities/core-concepts/jobs/customize-job-results) feature to compare key measurables between runs. Domino deletes it at the start of each run, so it only exists in a revision if the run wrote a new version.
* **`.noLock`**: Tells Domino to skip locking your files during synchronization. Domino normally locks all Project files during sync to keep them from changing mid-sync; with thousands of files, this can hit the operating system's open-file limit and fail with a "too many open files" error. Create an empty `.noLock` file in the Project root to skip locking, but avoid changing Project files while Domino is uploading or downloading them.

## Work with artifacts

In a DFS Project, artifacts live alongside the rest of your files; save them like any other file. In a Git-based Project, artifacts live in a dedicated **Artifacts** section stored in the DFS; see [save and pull artifacts](/cloud/platform-capabilities/core-concepts/projects/git-based-projects#save-and-pull-artifacts). Your Workspaces, Jobs, and other executables read artifact files like any other data. To persist dataframes for model training with versioning and metadata, use [TrainingSets](/cloud/reference/api/data_api/trainingsets_usecases). If you manage many artifacts, the [Domino CLI](/cloud/reference/cli) is faster than the UI.

<Warning>
  Reading Project artifacts during executions can hurt performance: many events [trigger a Project file sync](/cloud/admin/data-administration/project-file-storage#how-data-is-stored-in-project-files), and running executions must wait for the sync to finish before accessing the data again. Prefer [Datasets](/cloud/platform-capabilities/core-concepts/data/datasets) for anything beyond small files.
</Warning>

### When to use Datasets instead of Project files

Domino copies Project files to every execution, which breaks down at scale: Projects hold at most 10,000 files by default, individual file transfers are capped at 8 GB by default, and transfer time grows with data size, stretching startup and shutdown of Workspaces, Jobs, Apps, and Launchers. [Datasets](/cloud/platform-capabilities/core-concepts/data/datasets) avoid all three problems: no file-count limit, no per-file size limit, and they attach to executions as networked file systems, so nothing transfers at start or finish.

Project files are also not accessible from an [on-demand compute cluster](/cloud/platform-capabilities/features/compute-clusters-hpc). Copy the files you need into the Project's default Dataset, then follow the data-access instructions for your cluster type: [Spark](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/work-with-data#use-domino-datasets), [Ray](/cloud/platform-capabilities/features/compute-clusters-hpc/ray/work-with-data#use-domino-datasets), [Dask](/cloud/platform-capabilities/features/compute-clusters-hpc/dask/work-with-data#using-domino-datasets), or [OpenMPI](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/file-sync-mpi-clusters#domino-datasets).

## Add existing data to a Project

The data you need may already be in your deployment. In the Project's **Data** section you can browse the Datasets and data sources you have permission to use across all Projects (external data volumes don't appear until you add them). Add what you need: [data sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources), [Datasets](/cloud/platform-capabilities/core-concepts/data/datasets/use-datasets-and-snapshots), or [external data volumes](/cloud/platform-capabilities/core-concepts/data/external-volumes/use-a-mounted-volume). If data you expect is missing, it may need to be [shared with you](/cloud/platform-capabilities/core-concepts/data/sharing-and-security), or added to Domino by you or your administrator; contact the Project owner or your Domino admin.

## Get execution results

View an execution's results (the files your code generated or modified) from the **Runs** tab of your Project, or the latest results on the **Results** tab.

By default, Domino saves new files a run produces into your Project folder, and they download at your next sync. If your runs generate large results you don't need every time, save them to **isolated results branches** instead: go to **Settings** > **Results** and choose whether runs save to the main line or to isolated branches. Isolated results are permanently saved and viewable in the application, but they aren't inputs to later runs or downloaded by sync unless you explicitly ask.

To download isolated results, open the execution on the **Jobs** page and click **Results** to download each file, or use the CLI's `domino download-results` command (`domino help download-results` shows the syntax).

## Monitor Project assets across the deployment

The Assets Portfolio shows the Domino endpoints, Apps, Launchers, and Scheduled Jobs in Projects you own or collaborate on. [System Administrators](/cloud/admin/identity-and-access/manage-users/roles) see assets in all Projects, and Project Managers see assets in Projects owned by users in their [organizations](/cloud/platform-capabilities/features/collaboration/manage-organizations). Open it from **Govern** > **Assets Portfolio**; pick an asset type to see its table of metrics, and use the column picker to choose columns.

## Related

* [Datasets](/cloud/platform-capabilities/core-concepts/data/datasets): scalable storage for data your executions read and write.
* [Data security and sharing](/cloud/platform-capabilities/core-concepts/data/sharing-and-security): who can see what.
* [Customize Job results](/cloud/platform-capabilities/core-concepts/jobs/customize-job-results): notification emails, stats, and run comparison.


# Git-based Projects
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/projects/git-based-projects

How Git-based Projects organize code, data, and artifacts, and how to branch, sync, and resolve conflicts in Domino.

## Overview

Git-based Projects store code at your Git provider and are the usual choice for a Project. Git workflows (branch, commit, push, pull) work natively in your Workspace, and you keep your provider's collaboration features: pull requests, branch protection, CI.

<Note>
  Domino does not support Git Large File Storage (LFS). Before using a private repository, [add Git credentials](/cloud/platform-capabilities/core-concepts/projects/create-projects#add-git-credentials) to your Domino account; public repositories don't need credentials.
</Note>

## How it works

A Git-based Project separates its assets into **Code**, **Data**, and **Artifacts**, each with its own section in the Project navigation pane:

* **Code** lists the Git repositories that hold your Project code and any imported repositories. You can browse folders, select any branch, and see the last 10 commits directly from the **Code** section (large repositories can take up to 30 seconds to load initially). The default working directory is `/mnt/code`.
* **Data** lists the Project's [Domino Datasets](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets), external data volumes, and Dataset scratch spaces.
* **Artifacts** holds results like plots, charts, and serialized models. Artifacts live in the Domino File System (DFS), not in your Git repository.

If the Project owner sets a default Git branch in [Project settings](/cloud/platform-capabilities/core-concepts/projects/project-settings), it auto-fills across the Project and can be changed anytime.

In a Workspace or Job, your code mounts at `/mnt/code`, with Project Datasets, artifacts, and imported content mounted around it. For the full path layout and how it compares with a DFS Project, see [File paths at a glance](/cloud/platform-capabilities/core-concepts/projects/files-artifacts-and-results#file-paths-at-a-glance).

Git does not support committing empty directories, so a directory with no files disappears from the repository. See [Dataset paths in Git-based Projects](/cloud/platform-capabilities/core-concepts/data/datasets/use-datasets-and-snapshots#dataset-paths-in-git-based-projects) for how Dataset mounts resolve.

<Warning>
  When a Job runs in a Git-based Project, Domino automatically syncs only the artifacts to the DFS. Code is never auto-committed: you must sync it or push it to the Git repository yourself. This is intentional and supports the code, data, and artifacts workflow.
</Warning>

## Change branches

Select a branch from the menu in your Workspace. The menu lists up to 10 local branches followed by remote branches, alphabetically; search to find more. Branches deleted at the provider keep appearing in the menu until you [sync](/cloud/platform-capabilities/core-concepts/workspaces/sync-changes-in-a-workspace) or pull your latest changes.

## Pull changes

When you pull the latest changes from the remote, Domino fetches the remote branch's content (`git fetch`), then applies your changes on top (`git rebase`). If files conflict, resolve them manually or choose **Use Remote Changes**, which discards your Workspace changes and overwrites the files with the remote versions.

## Resolve merge conflicts

Merge conflicts happen when two changes touch the same line, when one person edits a file another deleted, or when you push multiple commits. When a sync hits a conflict, the **File Conflicts** window opens with a guided resolution flow that covers both the main code repository and any imported repositories. Select **Resolve Manually** to work through the conflicts file by file, with three options per file:

* **Mark as resolved**: you've edited the file to remove the conflict markers. The file is listed under **Uncommitted changes** and pushes to the remote when you click **Continue sync**.
* **Use my changes**: overwrites the remote file with your Workspace version. Also listed under **Uncommitted changes** and pushed on **Continue sync**.
* **Use origin repo changes**: discards your Workspace changes and takes the remote file. Nothing is left to commit, but you still must click **Continue sync** to finish.

If you made multiple commits, the latest commit might overwrite changes you made during conflict resolution.

## Save and pull artifacts

Artifacts sync with the DFS, separately from your Git repository:

* To save: in your Workspace, click **File Changes**, expand **File Changes** under **Artifacts**, enter a commit message, and click **Sync to Domino**.
* To pull the latest artifacts into your Workspace: click **File Changes**, then click **Pull** under **Artifacts**.

<Warning>
  Everything in **Artifacts** is saved exclusively to the DFS. If an asset shouldn't live in the DFS, don't save it as an artifact. See [Sync changes in a Workspace](/cloud/platform-capabilities/core-concepts/workspaces/sync-changes-in-a-workspace).
</Warning>

## Related

* [Create Projects](/cloud/platform-capabilities/core-concepts/projects/create-projects): create a Git-based Project and set up credentials.
* [Manage Projects](/cloud/platform-capabilities/core-concepts/projects/manage-projects): deep and reference copies of Git-based Projects.
* [Use Git in your Workspace](/cloud/platform-capabilities/core-concepts/workspaces/use-git-in-your-workspace): day-to-day Git operations during a session.


# Projects
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/projects/index

Projects organize the code, data, artifacts, and permissions for a piece of data science work in Domino.

## Overview

A Project is Domino's unit of organization: the code, data, results, and people for one piece of work. Apps, models, and API endpoints publish from a Project, and governance bundles track the work inside it. Projects carry work from first experiment to production asset.

## How it works

### Project types

When you create a Project, you choose where its code lives:

* **Git-based Projects** keep code in your own Git provider (GitHub, GitLab, Bitbucket, or another). This is the norm: you keep your provider's branches, pull requests, branch protection, and CI, you control when work syncs, and you can change the Git reference at any time.
* **Domino File System (DFS) Projects** keep code in Domino's internal repository. Domino versions your files automatically with a Git server it runs in the background, but you don't get an external provider's collaboration features.

<Note>
  **Use a Git provider whenever you can.** Git-based Projects give you more control over code synchronization and the full workflow of your Git host. Choose Domino's internal repository only when you don't have a Git provider or your organization's policies require it.
</Note>

### What's in a Project

Every Project connects three kinds of content:

* **Code**: Each Project has a code repository, available from the **Code** section of the Project.
* **Data**: Connect [data sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors), [Domino Datasets](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets), and [external data volumes](/cloud/platform-capabilities/core-concepts/data/external-volumes) from the **Data** section. Data offerings live outside the Project so multiple Projects can share them securely.
* **Artifacts**: Files that are neither source code nor data, such as plots, reports, and serialized models. Git-based Projects have a dedicated Artifacts repository for these; DFS Projects store them alongside the code.

Keep bulk data out of the code and artifact repositories. Small data files (less than 1 GB each, or fewer than 50,000 files in total) are fine, but repositories have storage limits and large files slow down future work. Use Datasets or data sources instead; data sources are managed centrally, are shareable across Projects, and get an automatic [audit trail](/cloud/platform-capabilities/features/governance/audit-trail).

### What a Project produces

A Project is also where work ships from. These assets are created from a Project and stay tied to it, even though they serve users who may never open the Project itself:

* **[Apps](/cloud/platform-capabilities/features/apps)**: web tools built from the Project's code and published for others to use, with Domino managing infrastructure, permissions, and routing.
* **[API endpoints](/cloud/platform-capabilities/features/model-deployment)**: models from the Project deployed as REST APIs for real-time or batch predictions.
* **[Launchers](/cloud/platform-capabilities/features/launchers)** and **[Scheduled Jobs](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job)**: self-service forms and recurring runs that execute the Project's code.

[Governance](/cloud/platform-capabilities/features/governance) connects to a Project from the outside: a governance bundle originates in a Project and tracks its governable materials, such as models, Apps, API endpoints, files, and agents, against your organization's policies. Policies define what evidence and approvals each stage requires, and gates can block actions like deploying a model or an App until those approvals are complete.

```mermaid theme={null}
flowchart LR
    data[("Data sources, Datasets,<br/>external volumes")] -->|attached| project
    subgraph project ["Project"]
        code["Code"]
        artifacts["Artifacts"]
    end
    project -->|runs| exec["Workspaces and Jobs"]
    project -->|publishes| published
    subgraph published ["Published assets"]
        apps["Apps"]
        endpoints["API endpoints"]
        launchers["Launchers"]
        sched["Scheduled Jobs"]
        apps ~~~ launchers
        endpoints ~~~ sched
    end
    bundle["Governance bundle"] -.originates in.-> project
    bundle -.gates App and endpoint deployment.-> published
    sched ~~~ spacer[" "]
    style spacer fill:none,stroke:none
```

### Versioning and execution

When you change a Project's files, Domino records a new version. When you run an execution (a workspace, job, or app) from a Project, Domino loads that Project's files onto the machine that executes it, so every run starts from a known state.

## Work with Projects

* [Create Projects](/cloud/platform-capabilities/core-concepts/projects/create-projects): Start a Project from scratch, from a template, or from existing work.
* [Manage Projects](/cloud/platform-capabilities/core-concepts/projects/manage-projects): Rename, copy, transfer, export, and archive Projects.
* [Project settings](/cloud/platform-capabilities/core-concepts/projects/project-settings): Configure hardware and environment defaults, visibility, tags, credentials, and notifications.
* [Git-based Projects](/cloud/platform-capabilities/core-concepts/projects/git-based-projects): How code syncs with your Git provider and how to organize a Git-based Project.
* [Domino File System Projects](/cloud/platform-capabilities/core-concepts/projects/dfs-projects): How the Domino File System versions your files, and how to manage them.
* [Files, artifacts, and results](/cloud/platform-capabilities/core-concepts/projects/files-artifacts-and-results): Where Project files live and how to retrieve execution results.
* [Collaborate on Projects](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects): Invite collaborators, set permissions, and discuss work with comments.


# Manage Projects
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/projects/manage-projects

Rename, copy, transfer, export, and archive Domino Projects.

## Overview

Once a Project exists, you manage it through a handful of lifecycle operations: rename it, duplicate it (copy), transfer it to a new owner, share its content with other Projects, and archive it when the work is done.

## How it works

* **Copying** creates an independent duplicate. What gets copied depends on the [Project type](/cloud/platform-capabilities/core-concepts/projects/index#project-types): DFS Projects copy through the UI in one step, while Git-based Projects support deep and reference copies.
* Only the Project **owner** can rename, transfer, or archive a Project. See the [permission tables](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#collaborator-permissions).

<Note>
  **Best practice:** Before starting from scratch, copy an existing Project; it's faster and keeps setups consistent across the team. Rename the copy, and invite the original owner as a contributor for guidance.
</Note>

## Rename a Project

Renaming is quick but can't be undone, and it has consequences: the Project's endpoints are automatically unpublished and get new URLs when re-published, and local copies must be cloned again because their metadata points at the old name.

1. Go to **Settings** > **Access & Sharing** and enter a new name. Domino checks that the name isn't already in use.
2. Read the warning Domino shows. If any of the consequences are unacceptable, click **Cancel**.
3. Click **Rename Project**. You can keep working in the Project immediately.

## Copy a DFS Project

Copying a DFS Project creates a duplicate under your account with the same files and [restricted Project status](/cloud/admin/platform-configuration/manage-compute-environments/restricted-environments#restricted-projects). The copy does not include Project settings, run history, or scheduled Jobs.

1. Open the Project's **Overview** page and click the copy icon.
2. Enter a name for the new Project and click **Copy**.

## Copy a Git-based Project

Copying a Git-based Project bootstraps new work from a well-organized original. Two copy modes cover different needs:

* **Deep copy** creates an independent copy of the original repository. Use it to build a new Project from an existing one. Available from the UI or the API.
* **Reference copy** points the new Project at the same main Git repository, so changes in one Project appear in all referenced Projects. Use it when parallel Projects rely on one shared repository, a common pattern for statistical analysis in pharmaceutical work. Available only through the API.

What a copy includes:

| | Copied | Not copied |
| - | - | - |
| **Project overview** | Tags, description, and README | |
| **Project materials** | Files, artifacts, code, Datasets and imported Git repositories, imported Projects | External data sources, audit data, run history, discussion history |
| **Project settings** | Environment variables, Hardware Tier, Environment, results, exports, integrations, and visibility | Access and sharing |
| **Other** | | Published Apps, Domino endpoints, and Launchers |

To create a deep copy from the UI:

1. Open the Git-based Project and click **Copy**.
2. Enter a name for the new Project, then click **Next**.
3. Enter your Git credentials, then click **Copy**.

To create a copy from the API, POST to the `copy-project` endpoint with either a `deepCopy` or `referenceCopy` spec:

```python theme={null}
import json
import requests

project_id = 'asdf1234'
code_repo_credential_id = 'asdf123'
url = f'https://host.com/api/projects/v1/projects/{project_id}/copy-project'
body = {
    'name': 'my_new_project',
    'visibility': 'private',
    'copyDatasets': True,
    'gitCodeRepoSpec': {
        'credentialId': code_repo_credential_id,
        # Deep copy: an independent new repository.
        'deepCopy': {
            'visibility': 'private',
            'newRepoName': 'new_git_repo',
            'newRepoOwnerName': 'git_user'
        }
        # Reference copy instead: share the original repository.
        # 'referenceCopy': {'mainRepoUrl': 'https://github.com/user/repo.git'}
    }
}
headers = {
    'X-Domino-Api-Key': 'YOUR_API_KEY',
    'Content-Type': 'application/json'
}
requests.post(url, data=json.dumps(body), headers=headers)
```

Limitations and requirements:

* Supported providers: GitHub, GitHub Enterprise, GitLab, GitLab Enterprise, Bitbucket Cloud, and Bitbucket Data Center. Other providers are not supported. Your admin can enable or disable copy support for each provider independently.
* Bitbucket Data Center credentials must use a personal access token together with your Bitbucket Data Center username. Token-only credentials work for read paths but are rejected by Bitbucket Data Center for the HTTPS git push used during copy and template creation.
* SSH credentials are not supported; use a personal access token with repository read and write permissions.
* The GitHub repository you copy must be a [template repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-template-repository).
* Collaborators on the parent Project are not copied.
* Datasets are copied only through the API with `copyDatasets: True`. A Dataset being copied is locked in a "copying" state; if you see the error `One or more of the datasets in this project are already being copied elsewhere`, the new Project was not created. Wait for the in-flight copy to finish and retry the same API call.

## Forked Projects

Domino Cloud does not support forking a Project.

* Existing forks continue to work as standalone Projects.
* To move a fork's changes into the original Project, [export and import Project content](#export-and-import-project-content).
* To duplicate a DFS Project, [copy the Project](#copy-a-dfs-project).
* For branch-and-merge collaboration, use a [Git-based Project](/cloud/platform-capabilities/core-concepts/projects/git-based-projects). This keeps your Git provider's pull requests, branch protection, and CI.

## Transfer Project ownership

You can transfer any Project you own to another user or [organization](/cloud/platform-capabilities/features/collaboration/manage-organizations), for example to promote a private scratch Project into a shared team Project. Go to **Settings** > **Archive Project or Transfer Ownership**, click **Transfer Ownership**, and enter the new owner's username or organization name. The new owner receives an email (organizations don't).

<Warning>
  Any API endpoint in the Project stops serving requests after the transfer. The new owner can re-publish it, but the URL changes and your API key stops working unless you're given contributor access. Re-point external callers and coordinate around the downtime.
</Warning>

To transfer without endpoint downtime: give the new owner contributor or results consumer access, have them copy the Project and re-publish the endpoint there, point your systems at the new URL, then shut down the original endpoint and archive the original Project.

## Export and import Project content

One Project can import another Project's files, environment variables, or both. During runs, each imported Project's directory is mounted read-only at `/mnt/<username>/<project name>` and pulled in alongside the current Project's files.

<Note>
  Importing changes your own Project's path from `/mnt` to `/mnt/<username>/<project name>`. Replace any hardcoded `/mnt` paths in your code with the `$DOMINO_WORKING_DIR` environment variable.
</Note>

To export, go to **Settings** > **Exports** in the source Project and enable exports for **Files** and **Environment variables** separately, or export the files as a Python or R **Code Package**.

Nothing can be imported from a Project with no export enabled. By default the latest revision is exported; to export the revision a specific Job produced, tag that Job with the exact string `release` on the Jobs page.

To import, you need Project Importer, Contributor, or Owner permission on the source Project:

1. In the destination Project, go to **Code** and click **Other Projects**.
2. Click **Import a Project** and enter the source path as `<username>/<project-name>`, then click **Import**.

Imports are not transitive: if Project A is imported into Project B and you import B, you see only B's contents. If a script in an imported Project can't find its own files at runtime, see [File paths at a glance](/cloud/platform-capabilities/core-concepts/projects/files-artifacts-and-results#file-paths-at-a-glance).

## Archive a Project

Archive Projects you no longer need. Archiving stops all scheduled runs and runs in progress, blocks CLI file uploads and downloads for collaborators, and removes any globally discoverable published App.

Go to **Settings** > **Archive Project or Transfer Ownership** and click **Archive This Project**.

To unarchive a Project, contact your Domino representative.

## Related

* [Create Projects](/cloud/platform-capabilities/core-concepts/projects/create-projects): create from scratch or from a template.
* [Collaborate on Projects](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects): roles and permissions behind these operations.
* [Project settings](/cloud/platform-capabilities/core-concepts/projects/project-settings): visibility, defaults, and environment variables.


# Project settings
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/projects/project-settings

Configure a Project's default hardware and Environment, visibility, tags, environment variables, and notifications.

## Overview

Project settings control how executions run and who can see the Project. Set them once and every Workspace and Job starts with the right defaults. To open them, click **Settings** in the Project's navigation pane.

## How it works

* **Hardware & Environment** settings set the defaults for every execution started in the Project.
* **Access & Sharing** settings control visibility and collaborators.
* Contributors can change the default Environment and Hardware Tier and set environment variables; only the Project owner can change visibility or the Project name. See the full [permission tables](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#collaborator-permissions).

## Set the default Hardware Tier

On the **Hardware & Environment** tab, select the Hardware Tier for the Project's executors: the virtual instances or on-premises machines that run your code.

Your administrator configures which tiers are available. Pick a tier that balances performance against cloud cost or on-premises tenancy. The tier is the default for every run in the Project; a run that needs different hardware can override it.

### Workspace and Jobs volume size

Volumes are the storage space dedicated to a Workspace or Job. The default size is 10 GiB, configurable by an administrator. If your administrator has enabled recommendations, Domino suggests a volume size when you launch a Workspace, based on your Project size and previous usage. Increase the volume size if your Workspace or Job needs more storage.

<Note>
  In a Git-based Project, the first Workspace you launch always uses the [volume size from your Project settings](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace#select-a-volume-size). Later launches get volume recommendations.
</Note>

## Set the default environment

Compute Environments are the container specifications your Project runs in. Whenever an executor launches for the Project, Domino loads the Environment set on the **Hardware & Environment** tab. You can create Environments or use public ones shared in your deployment. To go further, [customize the Domino software Environment](/cloud/platform-capabilities/core-concepts/compute-environments).

<img alt="Project settings" />

<Note>
  If you use a custom Environment, reapply the same customizations after switching to another Environment.
</Note>

## Set Project visibility

On the **Access & Sharing** tab, choose who can view the Project:

* **Searchable**: Anyone can see the Project's name and description in search results, but only collaborators can see its contents.
* **Private**: Only collaborators can find or view the Project.

<Note>
  Public visibility is not available in Domino Cloud deployments.
</Note>

## Store credentials as environment variables

Don't put passwords or secret keys in your source code: anyone who can read the code can take them, and they end up in version control. Instead, store them as Project environment variables, which Domino injects when your code runs. This keeps credentials out of Git and Domino version history and lets you share source files without sharing secrets.

You must be a contributor or the Project owner to add, modify, or delete environment variables. On the **Settings** page, add key/value pairs in the **Environment variables** section. Values are passed verbatim (no escaping needed) and are limited to 64K.

<img alt="Environment variables" />

<Note>
  All collaborators in a Project can see the environment variable values, so this protects credentials from readers of your code, not from your collaborators.
</Note>

You can also [define and inject environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables) at the Environment and user level.

## Tag a Project

Tags help colleagues find Projects. Assign and remove them from the Project's overview page, drawing on the Namespaces that your admins and [librarians](/cloud/admin/identity-and-access/manage-users/roles) define centrally, so the same classification means the same thing in every Project. For the full picture, including how the tag structure is built and how to filter by tag, see [Tags](/cloud/platform-capabilities/core-concepts/knowledge-management/tags).

Project tags are separate from [model registry tags](/cloud/platform-capabilities/features/models/register-and-govern/model-registry) and [Domino Model Monitoring tags](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/9-model-monitoring-tags).

## Get notified about long-running executions

To catch Jobs and Workspaces you forgot to shut down, set email notifications in your account settings (these apply across Projects): click your username, go to **Notifications**, select **Notify me about long-running Jobs and Workspace sessions**, and pick a time threshold. The emails list your running sessions, how long they've run, when they'll shut down automatically (if applicable), and links to shut them down.

## Enforce commit messages

Admins and Project owners can make commit messages mandatory in Workspaces to enforce internal documentation practices.

## Related

* [Collaborate on Projects](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects): grant access and set collaborator roles.
* [Manage Projects](/cloud/platform-capabilities/core-concepts/projects/manage-projects): rename, transfer, or archive a Project.
* [Compute Environments](/cloud/platform-capabilities/core-concepts/compute-environments): build and manage the Environments your Projects use.


# Create a Workspace from a checkpoint
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/workspaces/create-workspace-from-checkpoint

Recreate a Workspace from a synchronized code or artifact checkpoint, review its history, and continue work on a new branch.

Checkpoints are commits that you can return to any time to review the history of your work, branch your work, or [remediate models](/cloud/platform-capabilities/features/monitoring/6-remediation/2-remediate-a-domino-endpoint) that are drifting or decaying. Checkpoints are created every time you synchronize changes to artifacts or code within a workspace.

You can preview artifacts or code from any commit to identify a checkpoint and use that to recreate a workspace. When you recreate a workspace from a previous commit, Domino creates a new branch where you can develop or train a new model.

Durable workspaces are persistent development environments that you can start, stop, and restart. Each workspace has a persistent volume that stores your files. Your changes are saved from one workspace session to the next so that you can decide when to commit changes to version control.

This feature is supported for:

* Both Domino or Git-based projects.

* Workspaces created in 5.0 and later.

Datasets and external data volumes are recreated in the new workspace and branch that you create from a checkpoint.

You can review a workspace’s commit history and then recreate a workspace from a specific checkpoint.

1. In your project, go to **Workspaces**.

2. Click **History** on the workspace you want to recreate. To determine which commit you want to use, in the **Files** column (for DFS projects) or **Artifacts** column (for git-based projects), click the commit ID to browse the code and artifacts from a checkpoint.

3. To recreate a workspace, click **Open** next to the checkpoint to use.

4. Click **Open** to start the new workspace.

To publish a Domino endpoint from the new workspace, see [Remediate a Domino endpoint](/cloud/platform-capabilities/features/monitoring/6-remediation/2-remediate-a-domino-endpoint).


# Use Workspaces
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/workspaces/index

Use Domino Workspaces for interactive development in Jupyter, RStudio, or VS Code, and manage settings, persistence, Git, history, and Apps.

A Domino workspace is an interactive session where you can conduct research, analyze data, train models, and more. Use workspaces to work in the development environment of your choice, like [Jupyter](https://jupyter.org/) notebooks, [RStudio](https://rstudio.com/), [VS Code](https://code.visualstudio.com/), and many other customizable environments.

[Adjust Workspace settings](/cloud/platform-capabilities/core-concepts/workspaces/set-up-a-workspace)\
After creating a workspace, you can change the hardware tier, environment, volume size, and so on.

[Enable package persistence in an environment](/cloud/platform-capabilities/core-concepts/workspaces/use-package-persistence)\
Package persistence helps you save time by keeping installed packages across workspace sessions.

[Use Posit Package Manager in a Workspace](/cloud/platform-capabilities/core-concepts/workspaces/use-posit-package-manager)\
Use the Posit Package Manager (PPM) integration to manage R packages more easily, especially in GxP-regulated environments.

[Streamline Local Development with SSH](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace/streamline-local-development-ssh)\
Connect your local development tools to a remote Domino workspace using SSH.

[Launch a Workspace](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace)\
Create a workspace so you can use your preferred tools in a reproducible and customizable environment. Learn how to [select or configure VS Code](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace/launch-a-vs-code-workspace) to be the workspace in an environment. You can also [set custom preferences](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace/open-rstudio-with-custom-preferences) such as themes for RStudio workspaces.

[Sync changes in a Workspace](/cloud/platform-capabilities/core-concepts/workspaces/sync-changes-in-a-workspace)\
You can synchronize changes made to files in the `/mnt` directory to a Domino project.

[Manage Workspaces](/cloud/platform-capabilities/core-concepts/workspaces/manage-workspaces)\
View, stop, resume, and delete Workspaces that you or your collaborators created. If you save work to the `mnt` directory, it is persisted when the Workspace is stopped and resumed.

[View Workspace details](/cloud/platform-capabilities/core-concepts/workspaces/view-workspace-details)\
Use the logs to see information about user actions, view current and historical Workspace resource usage, and view sessions and commits that were made in a Workspace.

[Link a GitHub account to Domino](/cloud/platform-capabilities/core-concepts/workspaces/link-github-account-to-domino)\
Link your GitHub account to Domino so you can store and manage project code in a GitHub repository.

[Use Git in your Workspace](/cloud/platform-capabilities/core-concepts/workspaces/use-git-in-your-workspace)\
Use Git-based projects or import Git repositories to use in your Workspace.

[Create a Workspace from a checkpoint](/cloud/platform-capabilities/core-concepts/workspaces/create-workspace-from-checkpoint)\
Select any commit to create a Workspace from.

[Run multiple applications in a Workspace](/cloud/platform-capabilities/core-concepts/workspaces/run-multiple-apps-in-a-workspace)\
Run multiple applications in the same Workspace session if needed.

Use the **App Preview** tab to [develop Apps in a Workspace](/cloud/platform-capabilities/features/apps/develop-and-test-an-app) in a real App runtime. Use **Publish App** to publish a version. The Draft App is linked to the Workspace: stopping the Workspace stops the preview, and deleting the Workspace deletes the Draft App.


# Launch a Workspace
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace/index

Launch a Domino Workspace with a Compute Environment, IDE, volume, Hardware Tier, data mounts, clusters, and the GPU resources your work needs.

Domino Workspaces function as a server. You can use your preferred tools in a reproducible and customizable Environment.

## Basic configuration

1. In your Project’s navigation pane, click **Workspaces**.

2. Click **+ Create New Workspace**.

<span />

3. Enter a name for your Workspace.

## Select a Compute Environment

Select an Environment for your Workspace.

You can click one of Domino’s pre-defined Environments, or create a custom Environment. See [Customize Environments](/cloud/platform-capabilities/core-concepts/compute-environments) to learn more about managing Environments.

By default, Domino uses the latest revision of your Environment. To use another revision of the Environment:

1. In the **Workspace Environment** section, click **Change** to see the revisions for the Environment.

   <img alt="Environment revisions" />

2. Click the revision of the Environment you want for your Workspace. The version you selected displays in the **Revision** field.\
   If you see the **Not Recommended** warning, then Domino recommends using the active revision set by the Project owner.

<Note>
  [Restricted Projects](/cloud/admin/platform-configuration/manage-compute-environments/restricted-environments#restricted-projects) may limit your choice of Environments.
</Note>

## Select a Workspace IDE

Select the integrated development environment (IDE) that you’ll use in your Workspace (such as Jupyter).

The selected Environment determines the available IDEs.

See [Add Workspace IDEs](/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/add-workspace-ides) to learn how to manage IDEs.

## Select a volume size

If your administrator has enabled volume provisioning recommendations, you can select a **Volume Size** recommended by Domino based on your Project size and previous usage, instead of the default volume size.

<Note>
  In a [Git-based Project](/cloud/platform-capabilities/core-concepts/projects/git-based-projects), the first Workspace you launch always uses the volume size [configured in your Project settings](/cloud/platform-capabilities/core-concepts/projects/project-settings#workspace-and-jobs-volume-size). Subsequent Workspace launches receive volume provisioning recommendations.
</Note>

## Select a Hardware Tier

Click next and select a [Hardware tier](/cloud/platform-capabilities/core-concepts/projects/project-settings#set-the-default-hardware-tier).

A hardware tier represents the compute hardware, including CPU, memory, and other resources like GPUs, used for your Run. It can be a virtual instance in a cloud services provider, or a physical machine running in your deployment’s on-premises data center. Workspace scheduling requires a server with adequate free resources; otherwise, the Workspace queues until resources free up or a new server is available. Selecting a smaller hardware tier can speed up provisioning. Opt for the smallest tier that fulfills your task’s needs.

In a hybrid deployment, hardware tiers in the dropdown are grouped by [data plane](/cloud/platform-capabilities/core-concepts/data-plane), such as `GCP europe-central2` below. The `Local` data plane corresponds to running the Workspace in the Domino control plane cluster.

<img alt="Launch a Workspace and select a hardware tier" />

If you select a GPU-enabled tier, confirm the [GPU hardware allocation](#gpu-hardware-allocation) once the Workspace starts.

### Mount data

Domino shows you what data is available to be mounted to your Workspace, based on the selected data plane.

To ensure faster startup times, only select the data that you need.

<Tip>
  * If you want to use a data mount that is not available, note which data planes it is available in, then return to the first step in the modal to select a hardware tier in the corresponding data plane.

  * When a Workspace, Job, or other execution starts, all project files, including from the main DFS repo, main Git repo (if applicable), and any imported Git repos and Domino Projects, must be transferred over the network to the Persistent Volume for the execution. Large file counts or large individual files can significantly delay the startup process. In contrast, files in a Domino Dataset or External Data Volume do not require network transfer. If your executions spend considerable time preparing files during startup, assess if files in the main DFS repo, main Git repo (if applicable), and any imported Git repos and Domino Projects must be in that repo. If not, you can relocate these files to a Domino Dataset or External Data Volume to expedite startup.
</Tip>

## Attach compute clusters

1. Click **Next** to attach a compute cluster or view additional details of the Workspace (or click **Launch** to skip those options).

2. If necessary, attach a compute cluster to your Workspace. To learn more about clusters, see the following:

   * [Spark on Domino](/cloud/platform-capabilities/features/compute-clusters-hpc/spark)

   * [On-demand Ray](/cloud/platform-capabilities/features/compute-clusters-hpc/ray)

   * [On-demand Dask](/cloud/platform-capabilities/features/compute-clusters-hpc/dask)

   * [On-demand Open MPI](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi)

     <img alt="Attach a compute cluster" />

## Launch the Workspace

1. Click **Launch**.

2. A new tab opens and the Workspace starts loading.

<Warning>
  * If you cannot access a newly launched Workspace, check your browser’s configuration settings. Your browser must allow pop-up windows from Domino.

  * By default, Domino limits each user to four Workspaces per Project. To change this setting, contact your Domino administrator.
</Warning>

## GPU hardware allocation

A GPU-enabled hardware tier allocates a fixed number of GPU devices to your Workspace, and each device has its own memory. On a tier backed by NVIDIA devices, run `nvidia-smi` in a terminal in your Workspace to see how many devices you have and how much memory each one carries:

```shell theme={null}
nvidia-smi
```

Each device gets its own row. The `Memory-Usage` column reports used memory over total memory, so an entry such as `0MiB / 16160MiB` means that device has 16160 MiB in total.

<Note>
  On a tier backed by NVIDIA Multi-Instance GPU (MIG), each device is a partition of a physical GPU rather than a whole one, with its own dedicated memory. Size your work to the memory reported for the device you were allocated, not to the physical GPU's rated capacity. Administrators select the partition through a [custom GPU resource name](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/advanced-hardware-tier-settings#use-a-custom-gpu-resource-name) such as `nvidia.com/mig-1g.5gb`, which requests a partition with 5 GB of memory. On NVIDIA Ampere GPUs (A100 and A30), `nvidia-smi` reports a MIG partition's utilization as `N/A`.
</Note>

Tiers backed by other accelerators, such as Intel Habana Gaudi, need that vendor's own tooling instead of `nvidia-smi`.

If you need more devices or more memory on each one, [change the hardware tier in your Workspace settings](/cloud/platform-capabilities/core-concepts/workspaces/set-up-a-workspace) and restart, or ask your administrator to create a tier that provides them. To see how much of the allocation your work consumes, check the [GPU utilization and GPU memory graphs](/cloud/platform-capabilities/core-concepts/workspaces/view-workspace-details#view-workspace-usage).

## Next steps

* [Open a VS Code Workspace.](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace/launch-a-vs-code-workspace)

* [Set custom preferences for RStudio Workspaces.](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace/open-rstudio-with-custom-preferences)

* [Streamline Local Development with SSH.](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace/streamline-local-development-ssh)


# Open a VS Code Workspace
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace/launch-a-vs-code-workspace

Open VS Code in a Domino Workspace, install extensions through pre-run scripts or an Environment image, or add code-server to an Environment.

The [Domino standard environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) support opening [Visual Studio Code](https://github.com/microsoft/vscode) (VS Code) in [workspaces](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace). VS Code is an open source multi-language editor maintained by Microsoft. Domino can serve the VS Code application to your browser with the power of [code-server](https://github.com/cdr/code-server) from [Coder.com](https://coder.com/).

## Install VS Code extensions

You can install VS Code extensions in the following ways:

1. Use the pre-run script.

2. Use VS Code’s extensions manager.

### Use the pre-run script

1. Go to the environment.

2. Click **Edit Environment**.

3. In **Pre Run Script**, paste the following:

   ```shell theme={null}
   code-server --install-extension <extension-name> --extensions-dir ${HOME}/.local/share/code-server/extensions
   ```

4. Click **Build**.

### Use VS Code’s extensions manager

Use the VS Code’s extensions manager to install extensions from the marketplace. You must build the extensions in your environment to make them available in every new VS Code workspace.

1. Find the extension you want to install in the [Visual Studio Marketplace](https://marketplace.visualstudio.com/). For example, install the [scala-lang](https://marketplace.visualstudio.com/items?itemName=scala-lang.scala) extension.

2. Microsoft obscures the download URL for the extension by default. Open your browser’s development tools, then click **Download extension**.

   <img alt="Download extension" />

3. Get the download URL for the extension from the request details in your browser’s development tools. It ends with `/vspackage`. Copy this URL.

   <img alt="Get vspackage download URL from browser development tools" />

4. In Domino, [create a new environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments). As the base image, use one of the VS Code-equipped Domino standard environments, listed in the [Domino standard environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments).

5. Add the following instructions to your environment’s Dockerfile. Replace the folder names and example `/vspackage` URL with the extension URL you retrieved previously. These commands download the extension, extract the required files, and add them to the appropriate folder.

   ```dockerfile theme={null}
   USER root

   RUN apt-get update
   RUN apt-get install -y bsdtar
   RUN mkdir -p /home/ubuntu/.local/share/code-server/extensions/ms-python.python-2019.3.6558
   RUN cd /home/ubuntu/.local/share/code-server/extensions/ms-python.python-2019.3.6558
   RUN curl -JL https://marketplace.visualstudio.com/_apis/public/gallery/publishers/ms-python/vsextensions/python/2019.3.6558/vspackage | bsdtar -xvf - extension

   RUN cd /home/ubuntu/.local/share/code-server/extensions/ms-python.python-2019.3.6558/extension/ && mv * ../
   RUN chown ubuntu:ubuntu /home/ubuntu/.local/share/code-server/

   USER ubuntu
   ```

6. Click **Build**. After a successful build, you can use this new environment to launch VS Code workspace sessions with the extensions already installed.

## Install VS Code

You can add VS Code to an environment that does not include it. The base environment must be 2018-05-23 or newer.

1. Add the following to your compute environment Dockerfile instructions:

   ```dockerfile theme={null}
   USER root

   # Install VS Code (we install an older version so the Python extension works)
   RUN curl -fOL https://github.com/cdr/code-server/releases/download/v3.10.2/code-server_3.10.2_amd64.deb && \
       dpkg -i code-server_3.10.2_amd64.deb

   # Add a VS Code start script
   RUN mkdir -p /opt/domino/workspaces/vscode && \
       chown -R ubuntu:ubuntu /opt/domino/workspaces/vscode && \
       echo "#!/bin/bash" >> /opt/domino/workspaces/vscode/start && \
       echo "SETTINGS_DIR=${DOMINO_WORKING_DIR}/.vscode" >> /opt/domino/workspaces/vscode/start && \
       echo "FILE=${SETTINGS_DIR}/settings.json" >> /opt/domino/workspaces/vscode/start && \
       echo "# Add a user setting file if it doesn't exist. Add in DOMINO_WORKING_DIR so it persists across sessions" >> /opt/domino/workspaces/vscode/start && \
       echo "if [ ! -f "$FILE" ]; then" >> /opt/domino/workspaces/vscode/start && \
       echo "sudo mkdir -p "${FILE%/*}"" >> /opt/domino/workspaces/vscode/start && \
       echo "sudo chown -R ubuntu:ubuntu ${SETTINGS_DIR}" >> /opt/domino/workspaces/vscode/start && \
       echo "printf "{" >> /opt/domino/workspaces/vscode/start && \
       echo "  \"extensions.autoUpdate\": false," >> /opt/domino/workspaces/vscode/start && \
       echo "  \"extensions.autoCheckUpdates\": false," >> /opt/domino/workspaces/vscode/start && \
       echo "  \"python.pythonPath\": \"$(which python)\"," >> /opt/domino/workspaces/vscode/start && \
       echo "  \"workbench.startupEditor\": \"none\"," >> /opt/domino/workspaces/vscode/start && \
       echo "  \"workbench.colorTheme\": \"Default Dark+\"" >> /opt/domino/workspaces/vscode/start && \
       echo "}" > ${FILE}" >> /opt/domino/workspaces/vscode/start && \
       echo "fi" >> /opt/domino/workspaces/vscode/start && \
       echo "code-server ${DOMINO_WORKING_DIR} --user-data-dir ${SETTINGS_DIR} --auth none --bind-addr 0.0.0.0:8888 --extensions-dir ${HOME}/.local/share/code-server/extensions --disable-telemetry" >> /opt/domino/workspaces/vscode/start && \
       chmod +x /opt/domino/workspaces/vscode/start

   USER ubuntu
   ```

2. Add the following to your compute environment’s **Pluggable Workspace Tools**:

   ```yaml theme={null}
   vscode:
      title: "vscode"
      iconUrl: "https://raw.github.com/dominodatalab/workspace-configs/develop/workspace-logos/vscode.svg?sanitize=true"
      start: [ "/opt/domino/workspaces/vscode/start" ]
      httpProxy:
         port: 8888
         requireSubdomain: false
   ```


# Set custom preferences for RStudio Workspaces
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace/open-rstudio-with-custom-preferences

Set persistent RStudio Workspace themes, key mappings, languages, and dictionaries through preference files and Compute Environment scripts.

Use the instructions below based on your RStudio version, to set custom preferences (such as themes, key mappings, languages, and dictionaries) for RStudio workspace runs. For more information on RStudio session settings, see [RStudio’s official documentation site](https://docs.posit.co/ide/server-pro/admin/rstudio_pro_sessions/customizing_session_settings.html).

## RStudio version 1.3+

RStudio versions 1.3+ use `/home/ubuntu/.config/rstudio/rstudio-prefs.json` to store user preferences. You can use a [pre run script in a custom compute environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition) to modify this file to launch RStudio with custom preferences. To learn more about the schema of `rstudio-prefs.json`, see [RStudio’s official documentation on Customizing Session Settings](https://docs.posit.co/ide/server-pro/admin/rstudio_pro_sessions/customizing_session_settings.html).

### Method 1: Write lines to settings file

If you know how to manually add configuration lines to the `rstudio-prefs.json` settings file, you can write it in the pre run script. For example:

```shell theme={null}
# Variables
DIRECTORY="/home/ubuntu/.config/rstudio/"
# Check if directory exist, else create the directory
if [ -d "${DIRECTORY}" ];
then
    # Show directory
    echo "directory ${DIRECTORY} exists"
else
    echo "The specified directory does not exist"

    echo 'Creating directory...'
    mkdir -p "${DIRECTORY}"

fi

# Check if the preference file exist, else create the preference file
if [ -f "${DIRECTORY}/rstudio-prefs.json" ]; then
    echo "RStudio-prefs json exists"

else
    # Create Rstudio prefs json
    echo "Creating RStudio Preference json file"
    touch "${DIRECTORY}/rstudio-prefs.json"
fi

# add Rstudio Preferences
echo 'Adding Rstudio Preferences'
cat <<EOF > /home/ubuntu/.config/rstudio/rstudio-prefs.json
{
    "editor_theme": "Cobalt",
    "posix_terminal_shell": "bash"
}
EOF
```

### Method 2: Copy a saved settings file

Use this method to copy the settings from an existing RStudio session. This method is useful if you aren’t familiar with the syntax of `rstudio-prefs.json`, or if you want to use preferences from an existing preference file.

1. Run a session and modify the RStudio preferences as needed.

2. Before you stop the session, use the following R code to copy the `rstudio-prefs.json` file to the root of your project directory.

   ```r theme={null}
   file.copy("/home/ubuntu/.config/rstudio/rstudio-prefs.json", ".")
   ```

3. Add the following lines to the [pre run script of your environment definition](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition) to load the preferences file (if it exists) on subsequent runs:

   ```shell theme={null}
   # After setting preferences in an RStudio session, run the following command:
   # file.copy("/home/ubuntu/.config/rstudio/rstudio-prefs.json", '.')

   DIRECTORY="/home/ubuntu/.config/rstudio/"
   SETTINGS_FILE="./rstudio-prefs.json"

   if [ -f "${SETTINGS_FILE}" ]; then
       echo "File exists"
       echo "Checking if directory exist"
       if [ -d "${DIRECTORY}" ];
       then
           # Show directory
           echo "Directory '${DIRECTORY}' exists"

           # Copy Rstudio prefs json
           echo "Copying '${SETTINGS_FILE}' to '${DIRECTORY}'"
           cp ${SETTINGS_FILE} ${DIRECTORY}
       else
           echo "The specified directory does not exist"

           echo 'Creating directory...'
           mkdir -p "${DIRECTORY}"

           # Copy Rstudio prefs json
           echo "Copying '${SETTINGS_FILE}' to '${DIRECTORY}'"
           cp ${SETTINGS_FILE} ${DIRECTORY}

       fi
   fi
   ```

## RStudio version 1.2 and lower

Older versions (1.2 and lower) of RStudio use `/home/ubuntu/.rstudio/monitored/user-settings/user-settings` to store user preferences. To launch RStudio with custom preferences, you can use the pre-setup script in a custom [compute environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments) to modify this file.

### Method 1: Write lines to settings file

If you know what to add to the settings file, you can write it in the pre-setup script. For example:

```shell theme={null}
# create the encompassing directory if it doesn't exist
mkdir -p /home/ubuntu/.rstudio/monitored/user-settings/
# write the theme to the preferences file inside the directory
echo 'uiPrefs={"theme" : "Mono Industrial"}' >> /home/ubuntu/ .rstudio/monitored/user-settings/user-settings
chown -R ubuntu:ubuntu /home/ubuntu/.rstudio 
# modifies a Domino script that would overwrite this settings file.
if [ -f .domino/launch-rstudio-server ]; then
    sed -i.bak 's# > ~/.rstudio/monitored/user-settings/user-settings# >> ~/.rstudio/monitored/user-settings/user-settings#' .domino/launch-rstudio-server 
    chown ubuntu:ubuntu .domino/launch-rstudio-server 
fi
```

### Method 2: Copy a saved settings file

Use this method to copy the settings from an existing RStudio session. This method is useful if you aren’t familiar with the syntax of `rstudio-prefs.json`, or if you want to use preferences from an existing preference file.

1. Run a session and modify the RStudio preferences as needed.

2. Before you stop the session, use the following R code to copy the `user-settings` file to the root of your project directory.

   ```r theme={null}
   file.copy("/home/ubuntu/.rstudio/monitored/user-settings/user-settings", ".")
   ```

3. Add the following lines to the pre-setup script of your environment definition to load the preferences file (if it exists) on subsequent runs:

   ```shell theme={null}
   if [ -f user-settings ]; then
       mkdir -p /home/ubuntu/.rstudio/monitored/user-settings/
       cp user-settings /home/ubuntu/.rstudio/monitored/user-settings
       sed -i.bak '/initialWorkingDirectory=/d' /home/ubuntu/.rstudio/monitored/user-settings/user-settings
       chown -R ubuntu:ubuntu /home/ubuntu/.rstudio
       if [ -f .domino/launch-rstudio-server ]; then
           sed -i.bak 's# > ~/.rstudio/monitored/user-settings/user-settings# >> ~/.rstudio/monitored/user-settings/user-settings#' .domino/launch-rstudio-server
           chown ubuntu:ubuntu .domino/launch-rstudio-server
       fi
   fi
   ```

<Note>
  The `sed` statement that deletes the `initialWorkingDirectory` variable ensures that your session starts with the correct working directory.
</Note>


# Streamline Local Development with SSH
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace/streamline-local-development-ssh

Connect VS Code, Cursor, ParaView, or PyCharm to a Domino Workspace over SSH, configure the Dom CLI, and forward ports for remote tools.

You can securely connect your local development tools to a remote Domino workspace using SSH. This allows you to interact with workspace files and computing resources directly from your local applications; improving productivity and enabling remote workflows.

You can use SSH access with any local development tool, including:

* **VS Code** to browse and edit code files directly from your local IDE.

* **ParaView** to stream rendered visualizations from a remote workspace to your desktop client.

* Use the **Remote - SSH** extension if you import your VS Code extensions into **Cursor**.

This guide covers setup and connection steps for both tools.

## Prerequisites

To use SSH with Domino workspaces, make sure you have the following:

* Your Domino workspace is based on an SSH-enabled environment. For example, `openssh-server` is installed. The following docker instructions can be used to support a custom compute environment with SSH:

  ```dockerfile theme={null}
  USER root
  RUN apt update && \
   apt-get install -y openssh-server && \
   mkdir -p /var/run/sshd && \
   chmod 0755 /var/run/sshd
  USER ubuntu
  ```

  <Note>
    If you do not see the checkbox to enable SSH in your workspace, reach out to Domino Support for assistance.
  </Note>

* `sshd` is running in the workspace. Domino automatically starts `sshd` when starting the workspace.

* You have installed the Dom CLI to authenticate and connect via SSH.

  * This Dom CLI is for authenticating SSH only, and is distinct from the [Domino Command Line Interface (CLI)](/cloud/reference/cli).

  * To install the Dom CLI:

    1. Launch a workspace. Under **Additional settings**, ensure that **Enable SSH access** is selected, then click **Launch**.

    2. Once the workspace is running, click the **Settings** icon in the left navigation pane.

    3. Under the **SSH connection** instructions, look for **Step 1**, then follow the **Dom CLI toolkit** link to install it.

* You have started a workspace and have the **Workspace ID** (available in the workspace settings pane or under **Show Details**).

* Your local machine supports SSH and the necessary development tools:

  * For **VS Code**: the **Remote - SSH** extension

  * For **ParaView**: a local installation of **ParaView Desktop**

* For most IDEs compatibility, make sure to include a Domino SSH configuration file into your local SSH configuration.

  * For Linux/Mac users, just add the line `Include ~/.domino/ssh/config` to your `~/.ssh/config` file.

## Connect with VS Code

Using VS Code’s **Remote - SSH** extension, you can access a Domino workspace directly from your local editor. This enables you to browse, edit, execute, and debug files.

Run the Domino connect command on your local machine (replace `<workspace_id>` and `<your-domino-url>` accordingly):

```shell theme={null}
dom connect <workspace_id> --domino-api-host=https://<your-domino-url> -l ubuntu
```

Follow the command’s instructions. It returns:

* An SSH command

* A path to a configuration file

Next, configure VS Code:

1. Copy the path to the config and include it in your `~/.ssh/config` file.

   1. Or you can add the `~/.domino/ssh/config` into the `Remote.SSH: ConfigFile` settings on the Remote SSH extension.

2. Open VS Code. From the sidebar, choose **Remote Explorer** > **SSH**.

3. Select your workspace from the list (as configured by the `dom connect` output).

You are now connected to your remote workspace within VS Code and can work as if the files were local.

The [VS Code documentation](https://code.visualstudio.com/docs/remote/ssh#_connect-to-a-remote-host) provides full instructions for connecting to a remote host.

## Connect with ParaView

[ParaView](https://www.paraview.org/documentation/) allows offloading rendering to a remote server and streaming visualization results to your local machine. Domino workspaces can serve as that remote backend.

1. Ensure your workspace environment has ParaView installed and SSH is enabled.

2. Start the ParaView server inside the workspace:

   1. In the environment’s pre-run script: `pvserver`

   2. Or manually with the terminal.

The default bind port is `11111`.

1. Establish an SSH connection from your local machine:

   ```shell theme={null}
   dom connect <workspace_id> --domino-api-host=https://<your-domino-url> -l ubuntu
   ```

2. Add port forwarding to expose the remote ParaView server locally. If you’ve included this config in your default SSH file, the `-F` flag is optional:

   ```shell theme={null}
   ssh -L 11111:localhost:11111 <workspace-id> -F /path/to/ssh/config
   ```

3. Connect from your local ParaView Desktop:

4. Go to **File** > **Connect** > **Add Server**.

5. Fill in the following:

   1. **Server Name**: any name

   2. **Server Type**: Client / Server

   3. **Host**: `localhost`

   4. **Port**: `11111`

6. Click **Configure** > **Startup Type: Manual** > **Save**.

7. Select the server and click **Connect**.

You are now visualizing remote data with local interactivity.

## Connect with PyCharm

Run the Domino connect command on your local machine (replace `<workspace_id>` and `<your-domino-url>` accordingly), and add the `--port` argument to specify a custom local port:

```shell theme={null}
dom connect <workspace_id> --domino-api-host=https://<your-domino-url> -l ubuntu --port 2223
```

Follow the command’s instructions. It returns:

* An SSH command

* A path to a config file. Make sure this file is included in your local SSH configuration file.

Next, configure PyCharm project:

1. Open PyCharm and create a new Project under **Remote Development** > **SSH**.

2. Set **Username** to the workspace linux user (same as the `-l` argument of `dom connect`, for example `ubuntu`).

3. Set **Host** to the `<workspace_id>`.

4. Set **Port** to the one you specified in the `--port` argument of `dom connect`, for example `2223`.

5. Follow the wizard steps to connect and open your files.

Some considerations:

* Make sure to use the same `--port` next time you wish to connect to the same workspace so that you can use the same PyCharm configuration.

* If you need to access multiple workspaces at the same time, use different `--port` values for each so you can reference them in different PyCharm configurations.

## Custom TLS certificate authority

If your Domino instance is using custom CAs in the TLS certificate signatures, make sure you also install the certificates to your local environment.

On Mac you can use `sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain your_certificate.crt`. On windows you need to import the certificate into the Trusted Root Certification Authorities store using the Certificate Manager tool.

## Next steps

* [Learn how to manage compute environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments) in Domino, whether you need to create an environment, use one provided by Domino, or modify an existing environment.

* [Enable package persistence in an environment](/cloud/platform-capabilities/core-concepts/workspaces/use-package-persistence) to save time by keeping installed packages across workspace sessions.


# Link a GitHub account to Domino
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/workspaces/link-github-account-to-domino

Create a GitHub personal access token, save it as a Domino Git credential, and verify the connection by creating a linked test repository.

Linking your GitHub account to Domino allows you to store and manage project code in a GitHub repository. This guide explains how to create a GitHub Personal access token (PAT), add it to Domino, and confirm that the integration works.

## Prerequisites

Before you begin, make sure you have:

* Access to a Domino environment.

* A Domino user account (or the ability to create one using Sign up).

* A GitHub account that you can use for the setup.

* Permissions in GitHub to create personal access tokens.

## Step 1: Log into Domino

To start, you need to sign in to your Domino environment. This is where you’ll connect your account to GitHub.

1. Go to your Domino environment URL in a browser.

2. Enter your assigned username and password.

3. If you do not have credentials, click Sign up to create an account.

## Step 2: Create a GitHub Personal Access Token (PAT)

Next, you’ll generate a personal access token in GitHub. This token acts as a secure key that Domino uses to talk to your GitHub account.

1. In a separate browser tab, log into GitHub.

2. Click your profile icon in the top-right corner and click **Settings**.

3. In the left-hand menu, select **Developer settings**.

4. Select **Personal access tokens** > **Tokens (classic)**.

5. Click **Generate new token** > **Generate new token (classic)**.

## Step 3: Configure the PAT

When you create the token, you’ll decide which permissions it should have. For a quick setup, granting all scopes is easiest. If you prefer, you can limit scopes to just repo and workflow, which are enough for Domino to work properly.

1. Add a brief description in the **Note** field.

2. Choose an expiry for the token.

3. Select the scopes you want the token to have.

4. Click **Generate token**.

5. Copy the token and save it in a secure text file on your local machine.

## Step 4: Add GitHub credentials to Domino

With your token ready, the next step is to tell Domino about it. You’ll add the token as part of your account credentials.

<img alt="Alt text" />

1. Return to Domino.

2. Click **Account** in the top-right corner and select **Account Settings**.

3. From the Account Settings panel, select **Git Credentials**.

4. Click **Add Git credentials**.

5. Enter the required fields, including the PAT you generated.

6. Click **Add credentials**.

Your GitHub PAT is now linked to your Domino account.

## Step 5: Test the integration

It’s time to make sure everything works. You’ll do this by creating a test project in Domino and linking it to a newly created GitHub repository.

1. In Domino, go to the **Home** screen.

2. Click **Create Project**.

3. Enter a project name, such as `test`.

4. In the **Code** section, select **Git Service Provider** as the code hosting service.

5. From the **Service Provider** dropdown, select **GitHub**.

   * Your GitHub credentials are automatically selected in the **Git Credentials** dropdown if you added them earlier.

6. Select the **Owner/Organization** where the new GitHub repository is created

7. Enter a **Repository Name**.

8. Choose the **Repository Visibility** setting (Public or Private).

9. Click **Create**.

## Step 6: Verify the setup

Once the project is created, Domino has created a repository in GitHub that is now linked to the project. This confirms that your credentials and token are working correctly.

If the PAT is configured properly, Domino:

* Creates a new Domino project.

* Creates a new GitHub repository.

* Links the two together.

## Troubleshooting

Sometimes the setup doesn’t go smoothly. If you run into issues, here are a few common things to check:

* **Invalid or expired token**

  * Make sure you copied the entire PAT when you generated it.

  * If the token has expired, generate a new one and update your Domino Git Credentials.

* **Missing scopes**

  * If repository creation fails, verify that your PAT includes at least the repo and workflow scopes.

  * For evaluation setups, selecting all scopes avoids scope-related errors.

* **Incorrect credentials in Domino**

  * Open **Account Settings** > **Git Credentials**.

  * Confirm that the saved username matches your GitHub account and the token is correct.

* **Network or access restrictions**

  * Verify that your Domino environment can reach [github.com](http://github.com).

  * If using an enterprise firewall or VPN, confirm that GitHub is not being blocked.

You’ve now linked your GitHub account to Domino and verified the connection by creating a test project and repository. With this setup, Domino can automatically create and manage GitHub repositories for your projects, keeping your code versioned and accessible.

## Next steps

* [Use Git-based projects](/cloud/platform-capabilities/core-concepts/workspaces/use-git-in-your-workspace) or import Git repositories to use in your Workspace.


# Manage Workspaces
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/workspaces/manage-workspaces

View, stop, resume, and delete Domino Workspaces, preserve files in /mnt, and understand persistence, notifications, and automatic deletion.

This page explains how you can view, stop, resume, and delete your Workspaces, as well as how to persist data when the Workspace is stopped and resumed.

## View Workspaces

### View Workspaces in a Project

You can navigate to a specific Workspace in your Project:

* To view the Workspaces that you **created**, go to **Workspaces** > **My Workspaces**.

* To view **all** Workspaces that have been created by you or other users, go to **Workspaces** > **All Workspaces**.

* To view **deleted** Workspaces, go to **Workspaces** > **Deleted**.

### View Workspaces in all your Projects

From the top navigation pane, click **Develop** > **Workspaces** to navigate to the global Workspaces page.

Here you’ll see all Workspaces, their status, and what project they are tied to.

## Save work in a Workspace

When you click **Save** in the IDE of your workspace, your work is saved in the workspace.

<Note>
  Saving your work does not push it to the [Domino project’s](/cloud/platform-capabilities/core-concepts/projects/dfs-projects) Domino File System (DFS). To push the changes, you must synchronize your work to Domino.
</Note>

## Stop a Workspace

You can stop a Workspace and resume it later. Frequently stopping and resuming a Workspace session is a good way to manage compute costs. However, storage costs (for example, Amazon EBS) continue to incur.

<Warning>
  Stopping a Workspace does not automatically sync your work to Domino. See [Sync Changes](/cloud/platform-capabilities/core-concepts/workspaces/sync-changes-in-a-workspace) to sync your work.
</Warning>

1. You can stop a Workspace from either within your Project or from the global Workspaces page:

   * In your Project, go to **Workspaces** > **My Workspaces**, then click **Stop**.

   * Navigate to **Develop** > **Workspaces** in the top navigation pane, click the row menu on the right of your Workspace, then click **Stop Workspace**.

2. A window opens with additional information about stopping the Workspace. Click **Stop My Workspace**.

When you stop your Workspace, the following settings persist and are available to you when you resume your workspace session:

* Files saved in the `/mnt` directory

The following settings do not persist and reload when you resume your stopped Workspace:

* Files outside the `/mnt` directory, including installed packages

* Objects in memory

* Datasets

If you’re using a [Git-based project](/cloud/platform-capabilities/core-concepts/projects/git-based-projects) and someone updates or deletes the credentials to the Git repository while your workspace session is stopped, the credentials are updated when you resume your Workspace.

## Resume a Workspace

You can restart a Workspace that you stopped previously. Saved changes are available and can be [synced](/cloud/platform-capabilities/core-concepts/workspaces/sync-changes-in-a-workspace).

1. You can start a Workspace from either within your Project or from the global Workspaces page:

   * In your Project, go to **Workspaces** > **My Workspaces**. Go to the stopped Workspace, then click **Start**.

   * Navigate to **Develop** > **Workspaces** in the top navigation pane, click the row menu on the right of your Workspace, then click **Start Workspace**.

2. The Workspace opens in a new tab.

<Note>
  When a Workspace is restarted in a [restricted project](/cloud/admin/platform-configuration/manage-compute-environments/restricted-environments#restricted-projects), Domino uses the current restricted revision of the environment. This revision may be different from the revision that was used to start the Workspace previously.
</Note>

## Delete a Workspace

<Warning>
  * Ensure that all your work is [synced to Domino](/cloud/platform-capabilities/core-concepts/workspaces/sync-changes-in-a-workspace) before deleting a Workspace. Failure to do so results in unrecoverable work.

  * When you delete a Workspace, all its snapshots are also deleted and cannot be recovered.
</Warning>

You can delete a Workspace if it’s no longer needed.

You must [stop the Workspace](#stop-a-workspace) before it can be deleted. The delete option is inactive (appears greyed out) until all persistent volumes are detached.

1. You can start a Workspace from either within your Project or from the global Workspaces page:

   * In your Project, go to **Workspaces** > **My Workspaces**. Go to the stopped Workspace, then click the red trash icon to delete the Workspace.

   * Navigate to **Develop** > **Workspaces** in the top navigation pane, click the row menu on the right of your stopped Workspace, then click **Delete**.

2. Click **Delete** to confirm that you’d like to delete the Workspace.

### Auto-delete Workspaces

If your administrator has enabled the Workspace auto-deletion feature, Domino automatically deletes Workspaces that have been stopped for 10 days plus a 30-day grace period. Your administrator can modify these periods.

Domino notifies you (in the application and through email) before your Workspace is deleted.

1. The first notification indicates that your Workspace has been marked for deletion and the grace period has started.

2. Halfway through the grace period, Domino notifies you again and indicates whether the Workspace has uncommitted changes.

3. Two days before auto-deletion, Domino notifies you again, including whether the Workspace has uncommitted changes.

4. Finally, Domino notifies you when the Workspace has been deleted.

To remove a Workspace from the auto-deletion schedule, restart it. The auto-delete period restarts the next time the Workspace is stopped.

## Next steps

* [Launch a Workspace](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace)

* [Sync changes in a Workspace](/cloud/platform-capabilities/core-concepts/workspaces/sync-changes-in-a-workspace)

* [Create a Workspace from a checkpoint](/cloud/platform-capabilities/core-concepts/workspaces/create-workspace-from-checkpoint)


# Run multiple applications in a Workspace
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/workspaces/run-multiple-apps-in-a-workspace

Run Dash, Flask, TensorBoard, VS Code, or other applications beside Jupyter in one Workspace by configuring Jupyter Server Proxy.

For security reasons, Domino workspace sessions are only accessible on one port. For example, Jupyter typically uses port 8888. When you launch a Jupyter workspace session, a Domino executor starts the Jupyter server in a [Run](/cloud/platform-capabilities/core-concepts/jobs), and opens port 8888 to serve the Jupyter application to your browser. If you used the Jupyter terminal to start another application on a different port, it would not be accessible.

<Note>
  Domino workspaces reserve port 80 for nginx and port 9000 for the executor.
</Note>

However, you might want to run multiple applications in the same workspace session. For example, you might want to:

* Edit and debug Dash or Flask apps live.

* Use Tensorboard to view progress of a live training job.

Domino supports this with [Jupyter Server Proxy](https://github.com/jupyterhub/jupyter-server-proxy) and [JupyterLab](https://jupyterlab.readthedocs.io/en/stable/).

## Prerequisites

* Python 3+

* Jupyter Server Proxy

## Install Jupyter Server Proxy

By default, [Domino standard environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) have [Jupyter Server Proxy](https://github.com/jupyterhub/jupyter-server-proxy) installed. If you are not on the recent version of the [Domino Standard Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments), use the following steps to install it in your [Domino environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments).

Add the following lines to your environment’s Dockerfile instructions.

```bash theme={null}
USER root

# Install NodeJS
# You can omit this step if your environment already has NodeJS installed
RUN curl -sL https://deb.nodesource.com/setup_16.x | bash - &&
   apt-get install nodejs -y &&
   rm -rf /var/lib/apt/lists/*

# Install jupyterlab-server-proxy (omit if already installed)
RUN if [ -x /opt/conda/bin/conda ]; then
      /opt/conda/bin/conda install -y -c conda-forge jupyter-server-proxy;
   else
      pip install --no-cache-dir jupyter-server-proxy;
   fi

# Update JupyterLab config
RUN mkdir -p /home/ubuntu/.jupyterlab/ &&
   echo "c.ServerProxy.servers = {" >> /home/ubuntu/.jupyterlab/jupyter_lab_config.py &&
   echo "    'code': {" >> /home/ubuntu/.jupyterlab/jupyter_lab_config.py &&
   # VS Code example. Replace this command with what you want to proxy to
   echo "        'command': ['code-server', '--port', '{port}', '--auth', 'none']" >> /home/ubuntu/.jupyterlab/jupyter_lab_config.py &&
   echo "    }" >> /home/ubuntu/.jupyterlab/jupyter_lab_config.py &&
   echo "}" >> /home/ubuntu/.jupyterlab/jupyter_lab_config.py

USER ubuntu
```

## Access your App

It can be accessed via a URL in the form: `https://your-domino-url/project-owner/project-name/notebookSession/run-ID/proxy/app-port/`

This URL can be generated using Domino environment variables as follows:

```bash theme={null}
echo -e "import os\nprint('https://your-domino-url/{}/{}/notebookSession/{}/proxy/8501/'.format(os.environ['DOMINO_PROJECT_OWNER'], os.environ['DOMINO_PROJECT_NAME'], os.environ['DOMINO_RUN_ID']))" | python3
```

Ensure you change the `your-domino-url` and port number to match your deployment and app.


# Adjust Workspace settings
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/workspaces/set-up-a-workspace

Change a Workspace's Hardware Tier, Compute Environment, and volume size, and configure automatic shutdown for long-running sessions.

Settings are available for individual workspaces.

## Edit Workspace settings

You can edit your workspace settings to use a different hardware tier or a different environment.

1. Go to the workspace with the settings you would like to change.

2. Click **Settings**.

3. Click **Edit Settings**.

4. Make the changes and click **Save & Restart**.

## Change the workspace volume size

Volumes represent the storage space dedicated to your workspace or job. The default volume size for all workspaces and jobs in Domino is 10GiB, configurable by an administrator.

If your administrator enabled the feature, when you open a workspace you can select a volume size that is recommended by Domino based on your project size and previous usage.

<Note>
  In [Git-based projects](/cloud/platform-capabilities/core-concepts/projects/git-based-projects), the first workspace you open always uses the volume size configured in your project settings. Subsequent workspace launches receive volume provisioning recommendations.
</Note>

You can change the size of your volume if you find that your workspace (or job) requires more storage space.

### Change the size of your Workspace volume

1. Go to your project and in the navigation pane click **Settings**.

2. In **Workspace and Jobs Volume Size**, enter the new volume size. The default minimum volume size is 4GiB and the default maximum volume size is 200GiB. Your Domino administrator can change the default minimum and maximum volume size limits.

3. Click **Update**.

   <Warning>
     Changes to the volume size are applied to new workspaces. They do not impact existing workspaces.
   </Warning>

## Configure long-running Workspaces

Domino shuts down long-running workspaces after a period, as defined by your administrator, to prevent users from starting a workspace and then forgetting to shut it down. Your administrator can also configure workload notifications to define if and when users receive notifications about long-running workspaces.

<Note>
  These options do not apply to long-running Jobs. Jobs are intended to run to completion without interruption once started. This ensures that production runs execute reliably and that Domino resources are used in the most cost-efficient manner.
</Note>

Long-running workspaces that are automatically shut down **auto-commit your work** before shutting down. No additional settings are required.

However, keep the following in mind:

* Linked Git repositories are not pushed/committed unless you’ve specifically set that up.

* Only data that has been written to disk is preserved (that is, any in-memory objects and files not yet saved to disk are lost). For example, you must save a workspace you’re editing within Jupyter in order for it to be committed when the workspace shuts down.

### Change the automatic shutdown period

You can adjust this period to make it shorter. If you make a change, it applies to all workspaces owned by you.

1. Click your account name.

2. Click **Account Settings**:

3. Click **Workspace Settings**.

   In the **Workspace Settings** section, you can see the automatic shutdown period defined by your administrator.

4. Select **Shutdown my long-running workspaces \<n> minutes/hours after startup**.

   If you leave the checkbox cleared, the time limit configured by your administrator applies.

5. Select a different time limit for long-running workspaces.

   The shutdown time is the time since the workspace was started. The workspace shuts down automatically after it has been running for this period.


# Sync changes in a Workspace
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/workspaces/sync-changes-in-a-workspace

Sync Workspace files and all changes to the Domino File System, handle detached HEAD states, and choose how to resolve or overwrite conflicts.

You can sync recent changes to your files or all changes. Domino recommends that you sync your work at least daily.

<Warning>
  If you’re using a Git-based project, the navigation pane in your workspace is different than if you were using a Domino project. See [Use Git in your workspace](/cloud/platform-capabilities/core-concepts/workspaces/use-git-in-your-workspace) to learn more about saving and syncing your work using Git in a Git-based project.
</Warning>

## Sync changes to files

When you sync changes to remote Git repositories, Domino fetches the latest content from the remote branch (`git fetch`), then commits local changes on top of the updated branch (`git rebase`), and then pushes the commits to the remote.

1. Click **File Changes** in the navigation pane of your workspace.

2. Under **Files**, expand **File Changes** to view changes.

3. Enter a commit message.

4. Click **Sync to Domino**. Changes to files in the `/mnt` directory of your workspace are synced to the Domino File System (DFS). Changes to files outside of the `/mnt` directory are not synced.

   <Warning>
     Files with names containing "?" can be saved in a workspace and synced to the DFS, but are not viewable via the Code page.
   </Warning>

5. After syncing to Domino is complete, you can view your files in the **Code** section of your project.

## Sync all changes

1. Click **File Changes** in the navigation bar of your workspace.

2. Under **All Changes**, enter a commit message.

3. Click **Sync All Changes**. Domino saves all your work to the Domino File System (DFS).

Sync in detached HEAD state

If you launch a workspace from a commit ID in a Git-based project and then try to sync or pull changes, you may see an error that it failed to commit and push repos because it’s in a detached HEAD state.

In this scenario, your workspace is in a [detached HEAD state](https://git-scm.com/docs/git-checkout#_detached_head). To pull or sync, you must [checkout a branch](https://git-scm.com/docs/git-checkout).

Different IDEs may provide different ways to checkout a branch. The following instructions work for Jupyter Notebook.

1. Click **New > Terminal**.

2. Type `git checkout -b my-new-branch-name` into the terminal. Your new branch starts from the commit ID that you launched with.

3. Re-try the sync or pull action.

<Warning>
  When saving your work, remember that changes to files outside the `/mnt` directory do not persist if you stop your workspace and resume the workspace at a later time.
</Warning>

## Force changes

When files are in conflict, you can choose to either resolve manually or force changes. **Force my changes** overwrites remote files with changes in your workspace. This means that the commit history on the remote matches the commit history in your workspace.


# Use Git in your Workspace
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/workspaces/use-git-in-your-workspace

Use Git in a Workspace through the IDE, terminal, or plugins, sync and pull repository changes, and resolve merge conflicts.

If you want to use Git workflows such as creating branches, committing code, and pushing or pulling changes in a workspace, you must use Git-based projects or imported Git repositories.

To use Git, you can:

## Use the IDE

If you prefer a user interface, or you do not have much experience using a terminal, you can use your Git service provider’s user interface (such as GitHub) with the actions in a Domino workspace’s navigation pane. Click **File Changes** to perform actions such as **Select and Sync** (if configured in your environment), **Sync to Domino**, and **Pull**.

## Use the terminal

Start a terminal in your workspace and use Git just like you would in a local environment. See the [Git cheatsheet](https://www.atlassian.com/git/tutorials/atlassian-git-cheatsheet) for common workflows.

## Use the plugin

Many IDEs integrate with Git or have plugins or extensions for Git.

Domino’s Git-based projects fully support these features. You can use them to interact with Git in Domino. See the following resources for information about how to use Git with your IDE:

* Jupyter: [How to version control Jupyter Notebooks](https://towardsdatascience.com/enhancing-data-science-workflows-mastering-version-control-for-jupyter-notebooks-b03c839e25ec/).

* RStudio: [Managing - Part 2 (GitHub and RStudio)](https://resources.rstudio.com/resources/webinars/managing-part-2-github-and-rstudio/).

* VS Code: [Using version control in VS Code](https://code.visualstudio.com/docs/editor/versioncontrol).

## Commit and push changes to your Git repository

<Note>
  In a Git-based project, the default working directory for your project’s code is `/mnt/code`.
</Note>

### Commit changes to your code and push them to your repository

1. Go to the workspace where you made changes to a file.

2. In the navigation pane, click **File Changes**.

3. In the **Code** section, expand **Uncommitted Changes** to view changes.

   The up (↑) arrow in the **Code** section indicates the number of commits that your branch is ahead of the upstream branch, while the down (↓) arrow indicates the number of commits that your branch is behind the upstream branch.

4. Enter a commit message.

5. Click **Sync to Git**. If your administrator enabled the option for you to select the changed files to commit, click **Select and Sync**.

6. If you clicked **Select and Sync**, select the files to sync.

7. Enter a commit message and click **Sync to Git**.

   Domino syncs and pushes your changes to your Git repository.

<Warning>
  If you encounter merge conflicts when you pull or push code to your repository, see [Resolve merge conflicts](/cloud/platform-capabilities/core-concepts/workspaces/use-git-in-your-workspace).
</Warning>

## Commit all changes to your Git repository

You can use Domino to commit and push all changes to the project’s Git repository. This is often referred to as syncing your changes.

1. Go to your workspace.

2. In the navigation pane, click **File Changes**.

3. In **All Changes**, enter a commit message.

4. Click **Sync All Changes**. Domino commits changes to your code and syncs them to your Git repository. It also commits changes to artifacts and syncs them to the Domino File System (DFS).

<Note>
  In a Git-based project, the default working directory for your project’s code is `/mnt/code`.
</Note>

## Pull the latest changes from your Git repository

When you collaborate with others in Git-based projects, you must pull the latest changes to your code occasionally.

1. Go to your workspace.

2. In the navigation pane, click **File Changes**.

3. In the **Code** section, click **Pull**. Domino pulls the latest changes into your workspace.

<Note>
  In a Git-based project, the default working directory for your project’s code is `/mnt/code`.
</Note>

<Warning>
  If you encounter merge conflicts when you pull or push code to your repository, see [Resolve merge conflicts](/cloud/platform-capabilities/core-concepts/workspaces/use-git-in-your-workspace).
</Warning>

## Resolve merge conflicts

You might encounter merge conflicts when you collaborate with colleagues. To resolve merge conflicts, you can use the Git service provider or a terminal.

### Resolve conflicts with the Git service provider

When merge conflicts arise, Domino creates a new branch with the conflicts in your repository and pushes them to your Git service provider (such as GitHub or Bitbucket).

1. Go to your Git service provider.

2. Use your Git service provider’s application to create a pull request between the newly created branch and your original branch.

3. Merge the pull request.

4. Return to Domino.

   <Note>
     Do **not** pull any changes.
   </Note>

5. Start a workspace terminal.

6. Use the following to reset the branch:

   ```shell theme={null}
   git reset --hard origin/name-of-branch
   ```

   where `name-of-branch` is the name of your original branch.

7. Click **Pull**. Domino pulls the resolved merge conflicts. Your branch is now free of merge conflicts.

### Resolve conflicts with a terminal

If you prefer, you can resolve merge conflicts in a workspace terminal. For information about the typical Git workflow to resolve merge conflicts, see [Git merge conflicts](https://www.atlassian.com/git/tutorials/using-branches/merge-conflicts).


# Use Package Persistence in Domino
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/workspaces/use-package-persistence/index

Keep Python and R packages and home-directory settings across Workspace sessions, and enable persistence for standard or external Environments.

Package persistence helps you save time by keeping installed packages across workspace sessions. Whether you’re working in Python or R, you no longer need to reinstall your packages every time you start or restart a workspace, as long as the environment and workspace are configured to support it.

In addition to speeding up workspace launches, package persistence contributes to:

* **Better governance and compliance**, since environment updates can be cascaded through derived environments.

* **More reliable automation**, with the ability to test updated environments before production workloads adopt them.

* **Easier troubleshooting and iteration**, as new environment revisions can be built from any existing base.

This guide explains how to enable and use package persistence, both from the environment settings and from within an active workspace.

## Enable package persistence in an environment

To use package persistence, it must first be enabled in an environment.

1. Go to the **Revisions** tab of your compute environment.

2. Create a new revision or edit an existing one.

3. Check the box for **Enable Package Persistence**.

   If unchecked, this feature is not available in workspaces using the environment.

The **Package Persistence** checkbox in the workspace UI only appears if the environment supports it.

## Use package persistence in a workspace

Once package persistence is enabled in the environment, you can use it directly in your workspace.

### Install packages

Any packages you install manually persist across workspace sessions:

| Language | Install command |
| - | - |
| Python | `pip install <package-name>` |
| R | `install.packages("<package-name>")` |

You can also use the [Posit Package Manager](https://packagemanager.posit.co/client/#/) to install R packages.

### What happens to requirements.txt

* Domino automatically runs `requirements.txt` at startup.

* Before package persistence, this reinstalled packages every time.

* With persistence enabled, already-installed packages are skipped, speeding up workspace launch.

### Control persistence on restart

When you restart a workspace that uses an environment with persistence enabled, you’ll see a Package Persistence checkbox:

* Turn it on to keep your installed packages across sessions.

* Turn it off to start clean.

## Enable package persistence for external environments

If you’re using a custom (external) environment, you’ll need to install the appropriate tools to support package persistence.

<table>
  <thead>
    <tr>
      <th>Language</th>
      <th>Tool</th>
      <th>Installation details</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td><p>Python</p></td>
      <td><p>uv</p></td>
      <td><p>Follow the <a href="https://docs.astral.sh/uv/#installation">uv installation guide</a> to add it to your base or environment image.</p></td>
    </tr>

    <tr>
      <td><p>R</p></td>
      <td><p>renv</p></td>
      <td><p>Add this line to your Dockerfile:<br /> <code>RUN R -e "install.packages('renv')"</code></p></td>
    </tr>
  </tbody>
</table>

Once installed, Domino can track and persist packages installed within workspaces using these tools.

## Home directory persistence

Domino also supports persistence of the `$HOME` directory, allowing you to retain user-specific files across workspace sessions. This is useful for preserving IDE settings, environment configs, or files related to tools like remote Ray clusters.

<Warning>
  The `rsync` package is required to set up the folder structure in workspace persistent volumes and must be installed in your environment.
</Warning>

### Enable `$HOME` directory persistence

* When launching a workspace, toggle **Home Directory Persistence** to retain the contents of the `$HOME` directory between sessions.

* This setting ensures that any config files stored in `$HOME`, such as IDE preferences, `.ssh` keys, or cluster configs, are preserved.

By enabling home directory persistence, you can keep essential configuration files and tools available across sessions. This reduces setup time and helps maintain a consistent workspace environment. This is especially useful for advanced workflows that depend on personalized settings or external integrations.

## Next steps

* [View workspace details](/cloud/platform-capabilities/core-concepts/workspaces/view-workspace-details) about user actions, view current and historical workspace resource usage, and view sessions and commits that were made in a workspace.

* [Use Git-based projects](/cloud/platform-capabilities/core-concepts/workspaces/use-git-in-your-workspace) or import Git repositories to use in your workspace.

* [Run multiple applications](/cloud/platform-capabilities/core-concepts/workspaces/run-multiple-apps-in-a-workspace) in the same workspace session if needed.


# Use Posit Package Manager in a workspace
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/workspaces/use-posit-package-manager

Search Posit Package Manager snapshots for validated R packages, generate install commands, and update the Workspace environment and renv.lock.

If you work in a Domino workspace with RStudio or VS Code, you can use the Posit Package Manager (PPM) integration to manage R packages. This is especially useful in GxP-regulated Life Sciences workflows, where reproducibility and validation are critical.

This integration helps you:

* Search for packages across PPM snapshots

* See which snapshot is configured in your `renv.lock` file

* Install packages with a single command

* Maintain reproducibility for GxP-regulated workflows

## How it works

Search for packages with or without version numbers. The integration shows which snapshots contain them and highlights your current snapshot.

Select a snapshot and copy a formatted command for R or bash. This command can install packages, update your environment, and write changes to `renv.lock`. Your admin can customize the command logic to meet GxP compliance requirements.

Paste and run the command in your workspace.

## Prerequisites

Before you use PPM integration, verify these requirements:

* A network-accessible PPM server

* Your admin has [configured a PPM server](/6.3/admin/platform-configuration/manage-compute-environments/configure-ppm-server) in Domino

* Your admin has configured the environment to include an `update.R script` which handles:

  * Package installation

  * `renv` updates

  * Snapshotting the `renv.lock` file

  * GxP compliance logic

## Step 1: Search for validated R packages in PPM

Use the PPM integration to find packages. Search by package name to see which snapshots include them.

1. [Launch a workspace](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace) using RStudio or VS Code.

2. In the left sidebar, select **Posit Package Manager**.

3. In the PPM panel, enter a comma-separated list of R packages.

   You can optionally include version constraints using `==`, `>=`, `<=`, `<`, or `>`. The panel shows snapshots that contain all the packages and meet version constraints.

4. Review the snapshot currently defined in your `renv.lock` file at the top of the panel.

## Step 2: Install packages and update your environment

After you select a snapshot, install packages and update your environment with a single command.

1. Click the **copy** button next to your selected snapshot.

2. Choose either **R** or **Bash** based on where you want to run the command (in the R console or in the terminal).

   <img alt="Choose R or Bash" />

3. Paste and run the command in your workspace. For example, if you pick `R`, it looks something like this:

   ```r theme={null}
   source("/home/ubuntu/update.R");
   manage_renv_environment(
     cran_url = "http://rstudio-service.rstudio.svc.cluster.local:80/cran/2025-04-10",
     packages = c("dplyr", "ggplot2")
   )
   ```

   This command runs a script configured by your admin. The script can install packages, update the environment, run `renv::snapshot()`, or execute GxP compliance logic.

4. Select the **refresh** icon at the top of the PPM panel.

   <img alt="Refresh to update display" />

5. Your selected snapshot now appears under **PPM Snapshots**.

### Notes

* If you launched the workspace with an existing `renv.lock` file, the panel shows the snapshot defined in that file.

* If the integration doesn’t work, verify prerequisites with your admin.

* If you use bash, reload your terminal for the snapshot to appear in your environment.

## Next steps

* Learn about [launching workspaces](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace) in Domino.

* [View workspace details](/cloud/platform-capabilities/core-concepts/workspaces/view-workspace-details) about user actions, view current and historical workspace resource usage, and view sessions and commits that were made in a workspace.


# View Workspace details
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/workspaces/view-workspace-details

Inspect Workspace user, setup, and Data Source access logs, review CPU, memory, and GPU usage, browse history, and download support bundles.

## View Workspace logs

Workspace logs contain critical information to help you track Data Source access events, troubleshoot errors, audit workspace activity, and more.

1. In your project, click **Workspaces**.

2. To view logs for a specific Workspace, click **Logs**.

   1. Click **User** to see information about actions that you have taken in the Workspace.

   2. Click **Setup** to see information about Domino and Kubernetes.

   3. Click **Data Source Access** to see the logs for all the data sources accessed by the Workspace.

## View Workspace usage

In your project, go to **Workspaces** > **Usage**.

The CPU and memory usage for your workspace opens.

If the workspace or its attached compute clusters are configured with GPUs, GPU utilization and GPU memory graphs are also shown.

## View Workspace history

You can see the sessions and commits that were made in a workspace.

1. In the navigation pane, click **Workspaces**.

2. For the workspace whose commits and sessions you want to see, click **History**. The workspace history shows information such as when the workspace was launched and stopped.

## Download a Workspace support bundle

Domino Support might ask you for a support bundle, a ZIP file with logs and reports about a Workspace execution.

1. In the navigation pane, click **Workspaces**.

2. For the workspace of interest, click **History**.

3. Under **Sessions**, select the session of interest, then click **Support** to download the support bundle.

For more details, including how to retrieve a support bundle programmatically, see [Support bundles](/help/help#support-bundles).


# Deployment options
Source: https://docs.domino.ai/cloud/platform-overview/deployment-options

The environments Domino can run in, from customer-managed infrastructure to a fully managed service, and how to choose between them.

## Overview

Some organizations need software running fully within their own infrastructure; others prefer the convenience of a managed service (SaaS). Domino deploys into a range of environments to fit either preference.

## Customer-managed infrastructure

Deploy Domino entirely within your preferred infrastructure, so all data and services stay within your network.

* **Private cloud**: Domino runs in your preferred cloud provider, including Amazon Web Services (AWS), Microsoft Azure, Google Cloud, and Oracle Cloud Infrastructure (OCI), across most regions and specialized environments such as AWS GovCloud. Domino auto-scales resources up and down, and you're responsible for the resource costs, which roll into your normal cloud provider bill.
* **On-premises**: Domino runs on a fixed set of virtual or physical servers you provide. Domino can run in air-gapped environments, including Impact Level 6 (IL6) and Impact Level 7 (IL7) environments.

## Domino Cloud

[Domino Cloud](https://domino.ai/platform/cloud) is a fully managed service (SaaS) that provides the fastest access to product updates and the least operational burden. Domino Cloud can run your workloads in your AWS and Azure regions of choice.

Domino Cloud is certified to ISO 9001:2015, with an ISO 27001:2022 information-security certification and SOC 2 Type II compliance. For certificates and audit reports, see the [Trust Center](https://trust.domino.ai/).

### Validated cloud for life sciences

Domino Cloud is available as a validated offering for life sciences organizations that require Good Practice (GxP) compliance. To learn more, see the [Quality Management System](/cloud/admin/quality-management-system).

## Hybrid SaaS with private compute

Some organizations want a managed service but have workloads that must stay within their network perimeter, for reasons such as data gravity, data security, or specialized compute. In a hybrid SaaS configuration:

* Domino's platform services and user interfaces run in Domino-managed infrastructure.
* You configure one or more private Data Planes in your own infrastructure.
* Domino securely pushes workloads to run in those private Data Planes.

For how Data Planes work and how you choose one, see [Data Plane](/cloud/platform-capabilities/core-concepts/data-plane).

<img alt="Diagram of the hybrid SaaS deployment model: Domino-managed infrastructure runs the Core Services, which push workloads down to Compute in customer-managed infrastructure across Google Cloud, Microsoft Azure, Oracle Cloud, AWS, and on-premises." />

<img alt="Diagram of the hybrid SaaS deployment model: Domino-managed infrastructure runs the Core Services, which push workloads down to Compute in customer-managed infrastructure across Google Cloud, Microsoft Azure, Oracle Cloud, AWS, and on-premises." />

## Related

* [Data Plane](/cloud/platform-capabilities/core-concepts/data-plane)
* [Architecture](/cloud/admin/architecture-and-deployment/architecture)
* [Who Domino is for](/cloud/platform-overview/who-domino-is-for)


# Domino overview
Source: https://docs.domino.ai/cloud/platform-overview/domino-overview

What Domino is, how the platform is organized, and the principles it's built on.

Domino is a platform for building, delivering, and governing AI solutions for mission-critical use cases, typically in environments with strict compliance, regulatory, or infosec requirements. The platform increases agility and development speed while providing controls, so enterprises accelerate innovation without compromising safety.

Domino is built on a flexible orchestration engine that makes it easy to run arbitrary computational workloads in secured sandboxes, while automatically capturing reproducible results of those workloads and audit trails of what they did. This core capability makes Domino ideal for use cases where humans *and agents* need governed, controlled compute environments.

Common ways that organizations use Domino:

* As a governed app platform that empowers non-expert coders to build and deliver specialized software tools within their organization.
* As an execution orchestration platform for agents running in highly sensitive environments.
* As a model development and MLOps platform for predictive ML models in sensitive environments.
* As a modern Model Risk Management platform for models in financial services.
* As a Statistical Computing Environment for biostatistics work in pharmaceutical companies.

## Platform overview

Domino acts as a central access layer and interface for your preferred tools and resources (code, data, coding assistants, IDEs, packages, and compute) to construct AI solutions. Domino makes these materials accessible through self-serve but controlled access mechanisms, so that builders get streamlined access to the resources they need, and IT organizations can secure, track, and standardize access to their preferred tools and resources.

On top of this open (tool- and infrastructure-agnostic) orchestration capability, the platform provides user experiences and workflows that streamline the lifecycle of AI work:

<img alt="Domino platform overview: the AI Factory, App & Agent Hub, and Governance Center sit on infrastructure access and multi/hybrid cloud compute orchestration, with secured connectors to tools and packages, data, code, compute, and models." />

<img alt="Domino platform overview: the AI Factory, App & Agent Hub, and Governance Center sit on infrastructure access and multi/hybrid cloud compute orchestration, with secured connectors to tools and packages, data, code, compute, and models." />

1. **The AI Factory** is a set of capabilities that empowers builders to create models, AI systems such as agents, and other computational and quantitative research work. It lets builders use their preferred programming languages, IDEs, and compute frameworks while providing tailored user experiences for [experiment management](/cloud/platform-capabilities/features/development/track-monitor), MLOps, [collaboration](/cloud/platform-capabilities/features/collaboration), and more.
2. **The App & Agent Hub** is a place to host, serve, and distribute [apps](/cloud/platform-capabilities/features/apps) and [agents](/cloud/platform-capabilities/features/agents) to a broad spectrum of users within your organization. Domino provides scalability, security, and discoverability to make your apps and agents work at enterprise scale.
3. **The Governance Center** provides ways to manage different facets of AI work, such as [defining policies](/cloud/platform-capabilities/features/governance/define-governance-policies) that enforce standards for lifecycle management, [FinOps](/cloud/platform-capabilities/features/finops) for managing costs, and an [Audit Trail](/cloud/platform-capabilities/features/governance/audit-trail) for tracking all actions taken in the platform.

## Guiding principles and core capabilities

### Integrate with your preferred tools and infrastructure

We believe different use cases demand different techniques, different builders have their own specific preferences, and the best innovation for particular methods and technologies comes from the open source and commercial ecosystem. We also know our customers have strict compliance or infosec requirements that often necessitate keeping work in their network perimeter or using specific technologies they already have.

To that end, Domino:

* Runs in your private cloud or on-premises infrastructure if necessary (we also have a managed SaaS offering, of course).
* Runs on any of the major cloud providers or in high-performance computing environments such as GPU clusters.
* Lets your builders use their preferred dev tools: IDEs, languages, packages, and [compute frameworks](/cloud/platform-capabilities/features/compute-clusters-hpc).
* Connects to your [existing data sources](/cloud/platform-capabilities/core-concepts/data), whether those are traditional databases, modern data lakes, or even legacy network file shares.

### Empower code-first builders

We've built Domino to be the best environment for builders using code to create AI and ML systems. That includes supporting best-of-breed coding assistants and IDEs in [Workspaces](/cloud/platform-capabilities/core-concepts/workspaces), letting builders use their preferred desktop development tools, and [integrating with common Git providers](/cloud/platform-capabilities/core-concepts/projects/git-based-projects).

### Governance is a foundational capability, not something bolted on

From the beginning, Domino has built in deep capabilities for [reproducibility](/cloud/platform-capabilities/core-concepts/reproducibility), auditability, and lineage, which are critical for comprehensive [governance](/cloud/platform-capabilities/features/governance).

### Standardization and reusability enable acceleration at scale

As teams grow, the biggest lever to sustain speed of innovation and development is not having faster or better models. Rather, it's having ways for people to [find and reuse work](/cloud/platform-capabilities/features/collaboration/domino-search), having standard ways of working, facilitating collaboration and context sharing, enabling quick onboarding of new hires, and mitigating key-person risk. Domino makes this possible with capabilities such as [templates](/cloud/platform-capabilities/core-concepts/projects/create-projects#create-a-project-from-a-template), search, taxonomies for organizing work, and discussion and collaboration features.

### Built for automation and agents as users

Everything in the platform is automatable via [APIs](/cloud/reference/api), both because mission-critical environments require repeatability and standardization, and because, increasingly, agents use the platform even more than humans.

### Extensibility and customizability

One size doesn't fit all, so we build Domino with customizability and configurability as first-class requirements. Through customization points and even [platform extensions](/cloud/platform-capabilities/features/extensions), customers make the platform fit for purpose in a variety of different use cases and workflows.

## Get started

* [Get started](/cloud/platform-overview/getting-started-with-domino): build and share an App while Domino manages its compute, runtime, and access.
* [Platform capabilities](/cloud/platform-capabilities/core-concepts/projects): the core concepts, starting with Projects.


# Getting started with Domino: Build a transaction review App
Source: https://docs.domino.ai/cloud/platform-overview/getting-started-with-domino/index

Build and deploy a versioned transaction review App with a Git-based Project, Domino Dataset, and authenticated model endpoint.

Use an external Git repository connected to a Domino Git-based Project. Follow [Create a Git-based Project](/cloud/platform-capabilities/core-concepts/projects/create-projects), and add Git credentials only for a private repository or Workspace pushes.

Build a simple Dash App and logistic regression model for reviewing 300 synthetic transactions. Do not use either in production.

Follow [Create a Domino Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/3-create-an-environment) and [Customize your Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition) during the Environment exercise.

## Project files

1. `generate_data.py` generates 300 transactions with synthetic routes and writes them to a Dataset.
2. `model.py` trains the model, logs the run with MLflow, saves the model, and defines the endpoint function.
3. `app.py` creates the transaction review interface, calls the endpoint, and records session decisions.
4. `app.sh` starts the App for preview and deployment.

Domino connects these four small files to durable data, reproducible compute, experiment history, an authenticated model service, and a shareable App.

## How the solution fits together

<Frame>
  ```mermaid placement="top-right" theme={null}
  flowchart LR
      git["Git repository"] --> workspace["Workspace"]
      workspace --> dataset["Domino Dataset"]
      workspace --> endpoint["Model endpoint"]
      dataset --> app["Dash App"]
      endpoint --> app
      app --> reviewer["Reviewer"]
  ```
</Frame>

## Configure the Compute Environment

Your Domino deployment must allow Datasets to mount in Apps.

A Compute Environment gives every compute stage the same software stack. The separate Hardware Tier controls its compute resources.

Create the tutorial Compute Environment:

1. Open **Govern** > **Environments** and select **Create Environment**.
2. Name the Environment **Synthetic Transaction Review** and use the Domino Standard Environment as its base.
3. Select **Customize before Building**.
4. Add these Dockerfile instructions:

<Accordion title="View the Compute Environment Dockerfile instructions">
  ```dockerfile theme={null}
  RUN pip install --no-cache-dir dash dash-mantine-components
  ```
</Accordion>

5. Create the Environment and wait for its first revision to change to **Succeeded**.
6. Use this revision and a small CPU Hardware Tier for the Workspace, endpoint, and App.

## Build the transaction review App

<Steps>
  <Step title="Generate data in a Domino Dataset">
    In the Git-based Project, create a writable Dataset named `AML`. Start a Workspace from your Git branch with the tutorial Environment revision and CPU Hardware Tier.

    The Dataset outlives the Workspace, so Jobs and Apps can use the same records.

    Add `generate_data.py` to the repository.

    <Accordion title="View generate_data.py, generate the synthetic Dataset">
      ```python generate_data.py theme={null}
      """Create a small synthetic transaction Dataset for the tutorial."""

      from __future__ import annotations

      import os
      from pathlib import Path

      import numpy as np
      import pandas as pd

      DATASET_DIR = Path(
          os.environ.get("TRANSACTION_REVIEW_DATASET", "/mnt/data/AML")
      )
      OUTPUT_PATH = DATASET_DIR / "transactions.csv"
      LOCATIONS = [
          ("Chicago", "United States"),
          ("London", "United Kingdom"),
          ("Mexico City", "Mexico"),
          ("Mumbai", "India"),
          ("Singapore", "Singapore"),
          ("Sydney", "Australia"),
          ("Toronto", "Canada"),
          ("Zurich", "Switzerland"),
      ]
      STREETS = ["Market Street", "Oak Avenue", "Park Road", "River Lane"]


      def synthetic_location(rng) -> tuple[str, str]:
          city, country = LOCATIONS[rng.integers(len(LOCATIONS))]
          address = f"{rng.integers(10, 1_000)} {rng.choice(STREETS)}, {city}"
          return address, country


      def main() -> None:
          if not DATASET_DIR.is_dir():
              raise FileNotFoundError(
                  f"Create and mount the AML Dataset at {DATASET_DIR}"
              )

          rng = np.random.default_rng(6480)
          record_count = 300
          amount = np.round(np.exp(rng.uniform(np.log(50), np.log(50_000), record_count)), 2)
          age_days = rng.integers(5, 1_500, record_count)
          recent_count = rng.integers(0, 11, record_count)
          recent_amount = np.round(amount * rng.uniform(0.5, 4.0, record_count), 2)
          channel = rng.choice(["batch", "online", "treasury"], record_count)

          probability = 1 / (
              1
              + np.exp(
                  -(
                      -4.0
                      + 0.00005 * amount
                      + 0.18 * recent_count
                      + 0.00001 * recent_amount
                      + 0.7 * (age_days < 45)
                  )
              )
          )
          needs_review = rng.binomial(1, probability)
          transaction_date = pd.Timestamp("2026-05-01") + pd.to_timedelta(
              rng.integers(0, 90, record_count), unit="D"
          )
          origins = [synthetic_location(rng) for _ in range(record_count)]
          destinations = [synthetic_location(rng) for _ in range(record_count)]

          transactions = pd.DataFrame(
              {
                  "transaction_id": [f"TX-{index:04d}" for index in range(1, record_count + 1)],
                  "transaction_date": transaction_date.strftime("%Y-%m-%d"),
                  "amount": amount,
                  "counterparty_age_days": age_days,
                  "recent_transaction_count": recent_count,
                  "recent_aggregate_amount": recent_amount,
                  "channel": channel,
                  "origin_address": [address for address, _ in origins],
                  "origin_country": [country for _, country in origins],
                  "destination_address": [address for address, _ in destinations],
                  "destination_country": [country for _, country in destinations],
                  "needs_review": needs_review,
              }
          )
          transactions.to_csv(OUTPUT_PATH, index=False)
          print(f"Wrote {len(transactions)} synthetic transactions to {OUTPUT_PATH}")


      if __name__ == "__main__":
          main()
      ```
    </Accordion>

    Commit and push the file, then run:

    ```bash theme={null}
    python generate_data.py
    ```

    The script writes 300 fixed-seed records to `/mnt/data/AML/transactions.csv`. Open the Dataset and confirm that the file exists.

    <Frame>
      <img alt="Domino Workspace showing the transaction review files, mounted AML Dataset, and terminal confirmation that 300 synthetic transactions were generated." />
    </Frame>
  </Step>

  <Step title="Train and deploy the model">
    Add `model.py` to the repository, then commit and push it.

    <Accordion title="View model.py, train and serve the model">
      ```python model.py theme={null}
      """Train and serve the tutorial transaction review model."""

      from __future__ import annotations

      import os
      import pickle
      from functools import lru_cache
      from pathlib import Path

      import mlflow
      import mlflow.sklearn
      import pandas as pd
      from sklearn.compose import ColumnTransformer
      from sklearn.linear_model import LogisticRegression
      from sklearn.metrics import roc_auc_score
      from sklearn.model_selection import train_test_split
      from sklearn.pipeline import make_pipeline
      from sklearn.preprocessing import OneHotEncoder, StandardScaler

      DATASET_DIR = Path(
          os.environ.get("TRANSACTION_REVIEW_DATASET", "/mnt/data/AML")
      )
      DATA_PATH = DATASET_DIR / "transactions.csv"
      MODEL_PATH = Path(__file__).with_name("transaction-review-model.pkl")
      TRAINED_MODEL_PATH = (
          Path(os.environ.get("DOMINO_ARTIFACTS_DIR", MODEL_PATH.parent)) / MODEL_PATH.name
      )
      NUMERIC_FEATURES = [
          "amount",
          "counterparty_age_days",
          "recent_transaction_count",
          "recent_aggregate_amount",
      ]
      FEATURES = [*NUMERIC_FEATURES, "channel"]


      @lru_cache(maxsize=1)
      def load_model():
          """Load the fitted model once per endpoint process."""
          with MODEL_PATH.open("rb") as stream:
              return pickle.load(stream)


      def train() -> None:
          transactions = pd.read_csv(DATA_PATH)
          train_data, evaluation_data = train_test_split(
              transactions,
              test_size=0.25,
              random_state=6480,
              stratify=transactions["needs_review"],
          )
          preprocessing = ColumnTransformer(
              [
                  ("numbers", StandardScaler(), NUMERIC_FEATURES),
                  ("channel", OneHotEncoder(handle_unknown="ignore"), ["channel"]),
              ]
          )
          model = make_pipeline(
              preprocessing,
              LogisticRegression(max_iter=1_000, random_state=6480),
          )
          model.fit(train_data[FEATURES], train_data["needs_review"])
          scores = model.predict_proba(evaluation_data[FEATURES])[:, 1]
          roc_auc = float(roc_auc_score(evaluation_data["needs_review"], scores))

          TRAINED_MODEL_PATH.parent.mkdir(parents=True, exist_ok=True)
          with TRAINED_MODEL_PATH.open("wb") as stream:
              pickle.dump(model, stream)
          print(f"Saved {TRAINED_MODEL_PATH} with evaluation ROC AUC {roc_auc:.3f}")

          project_id = os.environ.get("DOMINO_PROJECT_ID", "local")
          try:
              mlflow.set_experiment(f"synthetic-transaction-review-{project_id}")
              with mlflow.start_run(run_name="train-transaction-review-model"):
                  mlflow.log_param("seed", 6480)
                  mlflow.log_param("training_records", len(train_data))
                  mlflow.log_metric("roc_auc", roc_auc)
                  mlflow.sklearn.log_model(
                      model,
                      name="model",
                      input_example=train_data[FEATURES].head(3),
                  )
          except mlflow.exceptions.MlflowException as error:
              print(f"MLflow logging skipped: {error}")


      def score_transaction(
          transaction_id: str,
          amount: float,
          counterparty_age_days: int,
          recent_transaction_count: int,
          recent_aggregate_amount: float,
          channel: str,
      ) -> dict:
          model = load_model()
          record = pd.DataFrame(
              [
                  {
                      "amount": amount,
                      "counterparty_age_days": counterparty_age_days,
                      "recent_transaction_count": recent_transaction_count,
                      "recent_aggregate_amount": recent_aggregate_amount,
                      "channel": channel,
                  }
              ]
          )
          risk_score = float(model.predict_proba(record[FEATURES])[0, 1])
          priority = "high" if risk_score >= 0.65 else "medium" if risk_score >= 0.35 else "low"
          indicators = []
          if amount >= 25_000:
              indicators.append("amount is at least 25,000")
          if counterparty_age_days < 45:
              indicators.append("counterparty age is less than 45 days")
          if recent_transaction_count >= 8:
              indicators.append("recent transaction count is at least 8")
          if not indicators:
              indicators.append("no tutorial indicators crossed their review levels")
          return {
              "transaction_id": transaction_id,
              "risk_score": round(risk_score, 4),
              "review_priority": priority,
              "review_indicators": indicators,
          }


      if __name__ == "__main__":
          train()
      ```
    </Accordion>

    The Job makes training repeatable outside the Workspace. Domino retains its logs and Artifacts, and MLflow records the run.

    Start a Job with the tutorial Environment revision and CPU Hardware Tier. Use this command:

    ```bash theme={null}
    python model.py
    ```

    The Job trains one logistic regression model and writes `transaction-review-model.pkl` to its Artifacts.

    Download `transaction-review-model.pkl` from the Job's Artifacts, place it beside `model.py`, then commit and push it. In **Deployments** > **Endpoints**, create a synchronous endpoint. The latest default-branch commit packages the trained artifact with its scoring code.

    * File: `model.py`
    * Function: `score_transaction`
    * Environment: the revision used by the Workspace
    * Hardware Tier: a small CPU tier
    * Instances: 1
    * Access: require an access token

    Test the endpoint with one synthetic record:

    ```json theme={null}
    {
      "data": {
        "transaction_id": "TX-TEST-0001",
        "amount": 1250.25,
        "counterparty_age_days": 30,
        "recent_transaction_count": 4,
        "recent_aggregate_amount": 5200.75,
        "channel": "online"
      }
    }
    ```

    The response's `result` object contains `transaction_id`, `risk_score`, `review_priority`, and `review_indicators`. The endpoint lets the App request scores without loading or training the model.
  </Step>

  <Step title="Preview the App in the Workspace">
    Add `app.py` and `app.sh` to the repository.

    <AccordionGroup>
      <Accordion title="View app.py, preview the Dash App">
        ```python app.py theme={null}
        """Dash App for reviewing synthetic transactions."""

        from __future__ import annotations

        import os
        from pathlib import Path

        import dash_mantine_components as dmc
        import pandas as pd
        import plotly.graph_objects as go
        import requests
        from dash import ALL, Dash, Input, Output, State, callback, ctx, dcc, html

        dmc.pre_render_color_scheme()

        DATASET_DIR = Path(
            os.environ.get("TRANSACTION_REVIEW_DATASET", "/mnt/data/AML")
        )
        DATA_PATH = DATASET_DIR / "transactions.csv"
        TRANSACTIONS = pd.read_csv(DATA_PATH) if DATA_PATH.exists() else pd.DataFrame()
        TRANSACTION_BY_ID = (
            TRANSACTIONS.set_index("transaction_id").to_dict("index")
            if not TRANSACTIONS.empty
            else {}
        )


        def domino_path_prefix() -> str:
            prefix = os.environ.get("DOMINO_RUN_HOST_PATH", "/")
            return f"/{prefix.strip('/')}/" if prefix.strip("/") else "/"


        app = Dash(
            __name__,
            title="Synthetic Transaction Review",
            routes_pathname_prefix="/",
            requests_pathname_prefix=domino_path_prefix(),
        )


        def review_queue():
            if TRANSACTIONS.empty:
                return dmc.Alert(
                    "Run generate_data.py before starting the App.",
                    title="Dataset not found",
                    color="red",
                )
            rows = []
            queue = TRANSACTIONS.sort_values("amount", ascending=False).head(20)
            for transaction in queue.to_dict("records"):
                transaction_id = transaction["transaction_id"]
                rows.append(
                    html.Tr(
                        [
                            html.Td(transaction_id),
                            html.Td(f"${transaction['amount']:,.2f}"),
                            html.Td(transaction["channel"].capitalize()),
                            html.Td(
                                dmc.Button(
                                    "Review",
                                    id={"type": "review-transaction", "index": transaction_id},
                                    variant="light",
                                    size="compact-sm",
                                )
                            ),
                        ]
                    )
                )
            return dmc.Table(
                [
                    html.Thead(
                        html.Tr(
                            [html.Th("Transaction"), html.Th("Amount"), html.Th("Channel"), html.Th("")]
                        )
                    ),
                    html.Tbody(rows),
                ],
                striped=True,
                highlightOnHover=True,
            )


        def transaction_volume_figure(color_scheme="light"):
            template = "plotly_dark" if color_scheme == "dark" else "plotly_white"
            if TRANSACTIONS.empty:
                return go.Figure().update_layout(template=template)
            weekly_volume = (
                TRANSACTIONS.assign(
                    week=pd.to_datetime(TRANSACTIONS["transaction_date"])
                    .dt.to_period("W")
                    .dt.start_time
                )
                .groupby("week", as_index=False)["amount"]
                .sum()
            )
            figure = go.Figure(
                go.Scatter(
                    x=weekly_volume["week"],
                    y=weekly_volume["amount"],
                    fill="tozeroy",
                    hovertemplate="%{x|%b %d}<br>$%{y:,.0f}<extra></extra>",
                )
            )
            figure.update_layout(
                template=template,
                height=300,
                margin=dict(l=0, r=0, t=10, b=0),
                xaxis_title="Week",
                yaxis_title="Volume",
            )
            return figure


        def endpoint_error() -> str | None:
            missing = [
                name
                for name in ("DOMINO_ENDPOINT_URL", "DOMINO_MODEL_ACCESS_TOKEN")
                if not os.environ.get(name)
            ]
            return f"Configure these Project variables: {', '.join(missing)}" if missing else None


        def endpoint_configuration_alert():
            configuration_error = endpoint_error()
            if not configuration_error:
                return None
            return dmc.Alert(
                configuration_error,
                title="Endpoint unavailable",
                color="red",
            )


        @callback(
            Output("transaction-detail", "children"),
            Output("current-transaction", "data"),
            Output("approve-transaction", "disabled"),
            Output("flag-transaction", "disabled"),
            Input({"type": "review-transaction", "index": ALL}, "n_clicks"),
            prevent_initial_call=True,
        )
        def review_transaction(_clicks):
            transaction_id = ctx.triggered_id["index"]
            transaction = TRANSACTION_BY_ID[transaction_id]
            configuration_error = endpoint_error()
            if configuration_error:
                return (
                    dmc.Alert(configuration_error, title="Endpoint unavailable", color="red"),
                    None,
                    True,
                    True,
                )
            payload = {
                "transaction_id": transaction_id,
                "amount": transaction["amount"],
                "counterparty_age_days": transaction["counterparty_age_days"],
                "recent_transaction_count": transaction["recent_transaction_count"],
                "recent_aggregate_amount": transaction["recent_aggregate_amount"],
                "channel": transaction["channel"],
            }
            try:
                response = requests.post(
                    os.environ["DOMINO_ENDPOINT_URL"],
                    auth=(
                        os.environ["DOMINO_MODEL_ACCESS_TOKEN"],
                        os.environ["DOMINO_MODEL_ACCESS_TOKEN"],
                    ),
                    json={"data": payload},
                    timeout=15,
                )
                response.raise_for_status()
                result = response.json()["result"]
            except (requests.RequestException, KeyError, ValueError):
                return (
                    dmc.Alert(
                        "The endpoint did not return a score. Check its status and App logs.",
                        title="Scoring failed",
                        color="red",
                    ),
                    None,
                    True,
                    True,
                )
            detail = dmc.Stack(
                [
                    dmc.Group(
                        [
                            dmc.Title(transaction_id, order=3),
                            dmc.Badge(result["review_priority"], variant="light"),
                        ],
                        justify="space-between",
                    ),
                    dmc.Text(f"Risk score: {result['risk_score']:.2f}", fw=700),
                    dmc.Text(f"Amount: ${transaction['amount']:,.2f}"),
                    dmc.Text("Origin", fw=600),
                    dmc.Text(
                        f"{transaction['origin_address']} · {transaction['origin_country']}"
                    ),
                    dmc.Text("Destination", fw=600),
                    dmc.Text(
                        f"{transaction['destination_address']} · "
                        f"{transaction['destination_country']}"
                    ),
                    dmc.Text("Review indicators", fw=600),
                    html.Ul([html.Li(indicator) for indicator in result["review_indicators"]]),
                ],
                gap="sm",
            )
            return detail, transaction_id, False, False


        @callback(
            Output("transaction-decision", "children"),
            Output("review-decisions", "data"),
            Input("approve-transaction", "n_clicks"),
            Input("flag-transaction", "n_clicks"),
            Input("current-transaction", "data"),
            State("review-decisions", "data"),
            prevent_initial_call=True,
        )
        def record_decision(_approve_clicks, _flag_clicks, transaction_id, decisions):
            decisions = decisions or {}
            if not transaction_id:
                return None, decisions
            if ctx.triggered_id == "approve-transaction":
                decisions = {**decisions, transaction_id: "approved"}
            elif ctx.triggered_id == "flag-transaction":
                decisions = {**decisions, transaction_id: "flagged"}
            decision = decisions.get(transaction_id)
            if not decision:
                return None, decisions
            color = "green" if decision == "approved" else "red"
            return dmc.Alert(f"{transaction_id} {decision} for this session.", color=color), decisions


        @callback(
            Output("transaction-volume", "figure"),
            Input("color-scheme-toggle", "computedColorScheme"),
        )
        def update_chart_theme(color_scheme):
            return transaction_volume_figure(color_scheme)


        app.layout = dmc.MantineProvider(
            dmc.Container(
                dmc.Stack(
                    [
                        dcc.Store(id="current-transaction"),
                        dcc.Store(id="review-decisions", data={}),
                        dmc.Group(
                            [
                                dmc.Title("Synthetic Transaction Review", order=1),
                                dmc.Tooltip(
                                    dmc.ColorSchemeToggle(
                                        id="color-scheme-toggle",
                                        lightIcon="☀",
                                        darkIcon="☾",
                                        size="lg",
                                        **{"aria-label": "Toggle light and dark mode"},
                                    ),
                                    label="Toggle light and dark mode",
                                ),
                            ],
                            justify="space-between",
                        ),
                        endpoint_configuration_alert(),
                        dmc.Grid(
                            [
                                dmc.GridCol(
                                    dmc.Paper(
                                        dmc.Stack(
                                            [dmc.Title("Review queue", order=2), review_queue()],
                                            gap="md",
                                        ),
                                        p="lg",
                                        withBorder=True,
                                    ),
                                    span={"base": 12, "md": 8},
                                ),
                                dmc.GridCol(
                                    dmc.Stack(
                                        [
                                            dmc.Paper(
                                                dmc.Stack(
                                                    [
                                                        html.Div(
                                                            "Select Review to score a transaction.",
                                                            id="transaction-detail",
                                                        ),
                                                        html.Div(id="transaction-decision"),
                                                        dmc.Group(
                                                            [
                                                                dmc.Button(
                                                                    "Approve",
                                                                    id="approve-transaction",
                                                                    color="green",
                                                                    disabled=True,
                                                                ),
                                                                dmc.Button(
                                                                    "Flag",
                                                                    id="flag-transaction",
                                                                    color="red",
                                                                    variant="light",
                                                                    disabled=True,
                                                                ),
                                                            ]
                                                        ),
                                                    ],
                                                    gap="md",
                                                ),
                                                p="lg",
                                                withBorder=True,
                                            ),
                                            dmc.Paper(
                                                dmc.Stack(
                                                    [
                                                        dmc.Title(
                                                            "Transaction volume", order=2
                                                        ),
                                                        dcc.Graph(
                                                            id="transaction-volume",
                                                            figure=transaction_volume_figure(),
                                                            config={"displayModeBar": False},
                                                        ),
                                                    ],
                                                    gap="sm",
                                                ),
                                                p="lg",
                                                withBorder=True,
                                            ),
                                        ],
                                        gap="lg",
                                    ),
                                    span={"base": 12, "md": 4},
                                ),
                            ],
                            gutter="lg",
                        ),
                    ],
                    gap="lg",
                ),
                size="xl",
                py="xl",
            )
        )


        if __name__ == "__main__":
            app.run(host="0.0.0.0", port=8888, debug=False)
        ```
      </Accordion>

      <Accordion title="View app.sh, launch the App">
        ```bash app.sh theme={null}
        #!/usr/bin/env bash
        set -euo pipefail

        python app.py
        ```
      </Accordion>
    </AccordionGroup>

    Commit and push both files. App Preview runs that commit with the selected Environment, Hardware Tier, and Dataset mount before you publish a version.

    Create these Project environment variables. Keep their values out of Git and screenshots.

    ```text theme={null}
    DOMINO_ENDPOINT_URL
    DOMINO_MODEL_ACCESS_TOKEN
    ```

    Open **App Preview** in the Workspace and use `app.sh` as the entry point. Select the tutorial Environment revision and CPU Hardware Tier, and keep the `AML` Dataset mounted.

    Select **Review** beside a transaction to call the endpoint. Select **Approve** or **Flag** to record a decision for the browser session.

    <Frame>
      <img alt="App Preview settings beside the Synthetic Transaction Review App with a scored transaction." />
    </Frame>
  </Step>

  <Step title="Deploy the App">
    Publish the Draft App as **Synthetic Transaction Review** and confirm that it includes the `AML` Dataset mount. The immutable version pins the Git commit, entry point, Compute Environment revision, and Dataset configuration. Publishing does not deploy it.

    Select the version, configure its Hardware Tier and audience, then deploy it.

    <Frame>
      <img alt="Synthetic Transaction Review App overview showing a running version, runtime metrics, Compute Environment, and Hardware Tier." />
    </Frame>

    Open the shared URL and score one transaction. Earlier versions remain available for rollback.

    <Frame>
      <img alt="Deployed Synthetic Transaction Review App with a review queue, scored transaction details, decision buttons, and a transaction volume chart." />
    </Frame>

    Stop the Workspace, App, and endpoint when you finish.
  </Step>
</Steps>

## Related

* [Git-based Projects](/cloud/platform-capabilities/core-concepts/projects/git-based-projects)
* [Create and manage Datasets](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets)
* [Deploy Domino endpoints](/cloud/platform-capabilities/features/model-deployment)
* [Develop and test an App](/cloud/platform-capabilities/features/apps/develop-and-test-an-app)
* [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions)


# Who Domino is for
Source: https://docs.domino.ai/cloud/platform-overview/who-domino-is-for

Domino serves the needs of all the different stakeholders and participants involved in the work of data science, AI engineering, and computational and quantitative research.

## Data scientists and other builders

If you're working with code to build models, [apps](/cloud/platform-capabilities/features/apps), analyses, or [agents](/cloud/platform-capabilities/features/agents), Domino gives you easy access to self-serve infrastructure and a tightly integrated set of user experiences that rapidly accelerate your development lifecycle.

Many Domino users would call themselves data scientists, quantitative researchers, or AI/ML engineers. Or, in specific industries: biostatisticians, statistical programmers, analysts, or actuaries.

## Domain specialists, executives, and business users

Domino hosts apps and agents that people in your organization build, making it easy for you to discover, find, and use the tools that have been purpose-built to make you more productive.

## IT and infosec

If you're responsible for the technology used to build models, apps, agents, or other computational research, Domino helps you consolidate duplicative tool stacks in a single platform. That reduces total cost of ownership, simplifies operations, and lets you standardize and secure AI and analytics infrastructure, avoiding shadow IT.

Domino also gives you tools and interfaces to manage, monitor, and control [resource usage](/cloud/admin/operations/finops) and [access](/cloud/admin/identity-and-access/manage-users).

With a centralized platform, your data scientists and researchers are happier and more productive, and your infrastructure is more secure and easier to manage.

## Risk and compliance

For people in risk and audit functions, Domino lets you ensure compliance with [internal policies](/cloud/platform-capabilities/features/governance/define-governance-policies) or external regulations that govern model development, AI development, and statistical programming.


# Core concepts at a glance
Source: https://docs.domino.ai/platform-capabilities/core-concepts

The building blocks Domino provides for AI and data science work

## Ways to build

| Concept | What it is |
| :- | :- |
| <Icon icon="laptop-code" /> [Workspaces](/cloud/platform-capabilities/core-concepts/workspaces) | A self-serve interactive development environment using the IDE of your choice, designed to feel like a “laptop in the cloud.” Intended for exploratory analysis and interactive development. |
| <Icon icon="play" /> [Jobs](/cloud/platform-capabilities/core-concepts/jobs) | A batch run of a script or command, fully reproducible, for training, simulation, and data processing. Can be run manually, via APIs, or on a schedule. |
| <Icon icon="flask" /> [Experiments](/cloud/platform-capabilities/features/development/track-monitor) | Useful primarily when developing machine learning models, Experiments provide a way to track, organize, and compare different model configurations you're testing. |
| <Icon icon="diagram-project" /> [Flows](/cloud/platform-capabilities/features/flows) | Multi-step computation graphs. A good fit for data engineering tasks or MLOps tasks. |
| <Icon icon="server" /> [Compute clusters](/cloud/platform-capabilities/features/compute-clusters-hpc) | Frameworks for running highly parallelizable workloads, including Spark, Ray, Dask, and Slurm. Domino can attach clusters to Workspaces, Jobs, and other workloads and manages the necessary infrastructure, including autoscaling. |

## Use data

| Concept | What it is |
| :- | :- |
| <Icon icon="database" /> [Datasets](/cloud/platform-capabilities/core-concepts/data/datasets) | Versioned filesystem storage for large data, shared across Projects, with strong features for reproducibility. |
| <Icon icon="hard-drive" /> [NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes) | Like Datasets but backed by NetApp ONTAP storage, with near-instant snapshots regardless of size. Use them wherever your organization runs ONTAP. |
| <Icon icon="plug" /> [Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors) | Connectors to external databases, data warehouses, data lakes, and object stores such as Amazon S3. |
| <Icon icon="network-wired" /> [External Data Volumes](/cloud/platform-capabilities/core-concepts/data/external-volumes) | Network-attached storage, such as network file system (NFS) shares or Windows shares, mounted into the workloads you run. |

## Productize what you build

| Concept | What it is |
| :- | :- |
| <Icon icon="rocket" /> [Domino endpoints](/cloud/platform-capabilities/features/model-deployment) | An HTTP API wrapped around your code, meant for real-time inference tasks such as classification and scoring. |
| <Icon icon="window-maximize" /> [Apps](/cloud/platform-capabilities/features/apps) | Interactive web applications built with frameworks such as Streamlit, Dash, Shiny, Flask, or any other web framework. Domino handles routing, authentication, and scaling. |
| <Icon icon="robot" /> [Agents](/cloud/platform-capabilities/features/agents) | Programs that use a large language model (LLM) to plan and act across multiple steps. Domino hosts and serves them alongside Apps and traces them for evaluation and monitoring. |
| <Icon icon="wand-magic-sparkles" /> [Launchers](/cloud/platform-capabilities/features/launchers) | Self-serve web forms that run a predefined Job with the parameters a less technical user fills in. Useful for turning scripts into simple web interfaces. |

## Govern

| Concept | What it is |
| :- | :- |
| <Icon icon="route" /> [LLM Gateway](/cloud/platform-capabilities/features/llms/llm-gateway) | A router that lets you direct LLM calls to any underlying model, with usage tracking, cost controls, and guardrails. Contact your Domino field team for access. |
| <Icon icon="clipboard-check" /> [Governance policies](/cloud/platform-capabilities/features/governance) | Policies define the stages, required evidence, and required approvals that a work product, such as an App or model, must go through as it's built. Used to enforce standards for internal workflows and compliance policies. |
| <Icon icon="coins" /> [FinOps](/cloud/platform-capabilities/features/finops) | Manage and monitor infrastructure spend. |
| <Icon icon="clock-rotate-left" /> [Audit Trail](/cloud/platform-capabilities/features/governance/audit-trail) | A record of user activity for compliance verification, security investigations, and audit reporting. |

## Collaborate, reproduce, and manage knowledge

| Concept | What it is |
| :- | :- |
| <Icon icon="folder-open" /> [Projects](/cloud/platform-capabilities/core-concepts/projects) | A shared space for people to collaborate with access to the same set of code, data, results, and history. The primary way of organizing work in Domino. |
| <Icon icon="rotate" /> [Reproducibility](/cloud/platform-capabilities/core-concepts/reproducibility) | Domino keeps records of all the "inputs" for most deliverables and results you create in the platform: the code, data, and Compute Environment that produced a given result or Model or App. |
| <Icon icon="magnifying-glass" /> [Search](/cloud/platform-capabilities/features/collaboration/domino-search) | Domino indexes contents in the platform so you can find and reuse past work easily. |
| <Icon icon="copy" /> [Project templates](/cloud/platform-capabilities/core-concepts/projects/create-projects#create-a-project-from-a-template) | A Project pre-configured with code, data, and other materials, to be used as a starting point for new Projects to help people use best practices and avoid re-inventing the wheel. |
| <Icon icon="tags" /> [Tags and Properties](/cloud/platform-capabilities/core-concepts/knowledge-management) | Tags are controlled, hierarchical labels for categorizing and finding assets. Properties are typed custom fields that enrich them. |

## Architecture and administration

| Concept | What it is |
| :- | :- |
| <Icon icon="cubes" /> [Compute orchestration](/cloud/admin/architecture-and-deployment/architecture) | Domino runs every workload as a container on Kubernetes. A cluster autoscaler adds compute capacity as demand grows. |
| <Icon icon="cloud" /> [Hybrid and multicloud compute](/cloud/platform-capabilities/core-concepts/data-plane) | One Domino deployment can run workloads in several regions, clouds, or on-premises data centers, each a Data Plane attached to the same Control Plane. |
| <Icon icon="box" /> [Compute Environments](/cloud/platform-capabilities/core-concepts/compute-environments) | A versioned, shareable definition of the tools and packages a workload runs in, built from a Docker image, so everyone runs the same software. |
| <Icon icon="microchip" /> [Hardware Tiers](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers) | Administrator-defined compute profiles (CPU, memory, GPU) that you pick when launching a workload. Access can be limited to an organization. |


# Access additional domains and hostnames
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/access-additional-domains-and-hostnames

Add a custom DNS name server to a Compute Environment pre-setup script so workloads can resolve and access private domains and hostnames.

Use custom DNS name servers to handle internal or private domain names and URLs. This topic describes how to add a new DNS server to your [Domino Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments) to allow access to additional domains and hostnames.

On a Linux host like your Domino executor, a configuration file at `/etc/resolv.conf` controls which DNS servers are used to resolve hostnames. You can use a pre-setup script in your Domino Environment to modify this file and add custom DNS servers by following these instructions.

1. Open the Environment you want to modify, then click **Edit Definition**.

2. Scroll down to the bottom of the definition, then click to expand the **Advanced** section.

3. In the **Pre Setup Script** field, enter the following lines of Bash. Fill in the address of the custom server you want to use where indicated.

   ```bash theme={null}
   echo 'nameserver your-custom-server-address' > /tmp/resolv.conf.tmp
   cat /etc/resolv.conf >> /tmp/resolv.conf.tmp
   cat /tmp/resolv.conf.tmp > /etc/resolv.conf
   ```

4. Click **Build** at the bottom of the definition page to create a new version of the Environment.

Following a successful build, you can use this Environment to reach hosts through the added DNS name server.


# Add packages to Environments
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments/index

Add Python and R packages to Domino Environments with dependency files, Dockerfile preloading, internal mirrors, or requirements.txt.

Add packages to your Environment and manage package versions.

[Package dependency files](/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments/package-dependency-files)\
Use `requirements.txt` or `install.R` to install packages to your Environment every time you start an execution.

[Internal package hosting](/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments/set-up-internal-package-hosting)\
Use Dockerfile instructions to install internal mirrors of public package repositories.

[Preload Environment packages](/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments/preload-environment-packages)\
Use Dockerfile instructions to install packages to your Environment permanently with quick execution times.

[Use requirements.txt](/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments/use-requirements-txt)\
Specify additional dependencies, you can add a pip requirements file named `requirements.txt` to the root of your project folder. (This is for Python only.)


# Package dependency files
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments/package-dependency-files

Manage Python and R dependencies with requirements.txt and R installation scripts, including package versions, Git sources, and startup installation.

Learn how to manage packages or libraries with a dependency file.

A common term for files like `requirements.txt` in Python and `install.R` in R is a "dependency file" or "package manifest file". These files are used to specify and manage the dependencies (that is, external libraries or packages) that a Python or R project requires to run successfully. They list the specific packages and their versions that need to be installed for the Project to work as intended.

Depending on the programming language and package management system, the file formats and naming conventions may vary, but the purpose is the same: to define and document the dependencies for a Project.

No special formatting is required for Domino compatibility of a dependency file. If it works outside of Domino, it works in Domino. There are a few specifics about the location of the file which are explained below.

## When to use a package dependency file

Package dependency files can be used in a variety of scenarios. The following are a few examples.

1. Share code using Git-based source code management tools

2. Make it easier for other developers to install the correct versions of the required libraries

3. Install all the required dependencies in a single command

4. Start each Domino Workspace or Job with specific versions, or the latest available versions, of Python, R, or Git repository packages

<Warning>
  Package dependency files cause packages to install on every execution startup, slowing down the process. If you want your packages to remain permanently installed and reduce startup time, consider [preloading Environment packages](/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments/preload-environment-packages).
</Warning>

## Python dependency file - requirements.txt

If it doesn’t already exist, create a file named `requirements.txt` in the root directory of your Project.

<Note>
  In a [Git-based Project](/cloud/platform-capabilities/core-concepts/projects/git-based-projects), `requirements.txt` must be inside your Project’s Artifacts folder.
</Note>

After `requirements.txt` is in place, Domino automatically installs the Python package every time you start a Workspace or Job within the Project.

### Add Python packages

The requirements file specifies which libraries and any version requirements for them. For example:

```bash theme={null}
pandas
lxml==3.2.3
numpy>=1.7.1
```

See [pip install requirements file format](https://pip.pypa.io/en/stable/cli/pip_install/#requirements-file-format) for the syntax of the requirements file.

### Generate a requirements.txt file from an existing Environment

If you’re using pip on your local machine, the easiest way to generate the `requirements.txt` file is to run the following command in the root of your Project folder (or in the Code folder if your Project is Git-based):

```bash theme={null}
~/domino/myProject $ pip freeze > requirements.txt
```

For performance reasons, prune the file so that it includes only the libraries needed for your analysis.

If you’re working in a Jupyter Notebook, you can also use `pip` to install dependencies interactively. In a notebook cell, you can run:

```bash theme={null}
! pip install --user <package>
```

(The '!' tells the notebook to execute the cell as a shell command.)

### Install packages hosted from a Git repository

<Warning>
  This is an advanced topic.
</Warning>

Pip can install Python packages from source by cloning a public Git repository over HTTPs. See [pip install](https://pip.pypa.io/en/stable/cli/pip_install/#git) for reference. To specify this, you must add something like the following line to your `requirements.txt` file:

```shell theme={null}
-e git+https://git.yourproject.org/you/Project.git#egg=YourProject
```

or

```shell theme={null}
package-name@git+https://github.com/repository-name.git
```

The most common host of Git projects is GitHub. If the package you want to install is publicly accessible, then the previous instructions work. However, if you must install private repositories, Domino can securely integrate with GitHub to access those private repositories. See [Import Git Repositories](/cloud/platform-capabilities/core-concepts/projects/dfs-projects#import-git-repositories) for information about securely storing your GitHub credentials.

<Warning>
  Do not embed your GitHub credentials directly in the `requirements.txt` file. Instead, integrate Domino with GitHub securely by following the instructions in [Import Git Repositories](/cloud/platform-capabilities/core-concepts/projects/dfs-projects#import-git-repositories).
</Warning>

## R dependency file - `install.R`

If it doesn’t already exist, create a file named `install.R` in the root directory of your Project. Add the following line to `install.R`:

```r theme={null}
devtools::install_local('/repos/<repo_name>')
```

For example, if you want to install `ggplot2` from a Git repository, your `install.R` should look like:

<img alt="R dependency file" />

After this `install.R` is in place, enter a line at the beginning of your work within the Project to install the Project for your sessions.

```r theme={null}
source('install.R')
```


# Preload Environment packages
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments/preload-environment-packages

Preload pinned Python and R packages in an Environment with RUN, ENV, and ARG Dockerfile instructions to reduce repeated startup installs.

Domino Environments have a **Dockerfile Instructions** section. Type commands in this section to install packages directly into your Environment.

## Save package install time and improve reproducibility

Preloading packages can save time as it prevents repeated loading in Workspaces and Jobs. Installation is not required after the initial build as Domino Environments are cached. However, the Environment is still loaded with each new execution so only packages that are frequently used should be cached. Preloading too many unnecessary packages increases the size of the cached which can significantly increase load times.

Dockerfile instructions can load specific versions of packages and libraries for clarity which helps with reproducibility and collaboration. It’s a best practice to fix package versions in the Dockerfile instructions.

An alternative to caching packages in a Python project is to use a requirement.txt file. See [requirements.txt in Domino](/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments/use-requirements-txt) for more details.

## Docker command syntax in Domino

Consult the official Docker documentation to learn more about Dockerfiles:

* [Reference](http://docs.docker.com/engine/reference/builder/)

* [Best practices](http://docs.docker.com/engine/articles/dockerfile_best-practices/)

<Note>
  Domino’s Docker images must be run by the user, `ubuntu`. However, Domino recommends installing packages in your Dockerfiles as the `root` user to avoid permissions issues. To do this, add `USER root` before your Docker commands. Then, after running the commands, type `USER ubuntu`.
</Note>

Do not start your Dockerfile instructions in Domino with a `FROM` line. Domino includes the `FROM` line for you, pointing to the base image specified when setting up the Environment.

The most common Dockerfile instructions you’ll use are `RUN`, `ENV`, and `ARG`:

`RUN` commands execute lines of bash. For example:

```dockerfile theme={null}
USER root

RUN wget http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.6.tgz
RUN tar xvzf spark-1.5.1-bin-hadoop2.6.tgz
RUN mv spark-1.5.1-bin-hadoop2.6 /opt
RUN rm spark-1.5.1-bin-hadoop2.6.tgz
RUN pip install s3fs==2021.6.1 scipy==1.7.1
RUN R --no-save -e "install.packages('remotes')"

USER ubuntu
```

`ARG` commands set build-time variables, and `ENV` commands set container bash Environment variables. They are accessible from runs that use this Environment. For example:

```dockerfile theme={null}
ENV SPARK_HOME /opt/spark-1.5.1-bin-hadoop2.6
```

If you set variables in the **Environment variables** section of your definition, you can use an `ARG` statement:

```dockerfile theme={null}
ARG SPARK_HOME
```

This is available for the build step. If you want the variable to be available in the final compute Environment you must add an `ENV` statement referencing the argument name:

```dockerfile theme={null}
ENV SPARK_HOME=$SPARK_HOME
```

## Examples: Package installation

1. When you edit your Environment, click **R Package** or **Python Package** to insert a line to install packages.

2. Enter the names of the packages.

3. You can also add the commands, as in the following examples:

   * R Package Installation, example with the **devtools** package.

     ```dockerfile theme={null}
     USER root
     RUN R --no-save -e "install.packages('devtools')"
     RUN chown -R ubuntu:ubuntu /usr/local/lib/R/site-library  # ensure ubuntu user can update installed packages
     USER ubuntu
     ```

   * Python Package Installation with Pip, for example with the…​ numpy package.

     ```dockerfile theme={null}
     USER root
     RUN pip install numpy
     USER ubuntu
     ```

## Install packages that are not on a package manager

If a package is not published to a public package manager such as PyPI or CRAN, you can still preload it into your Environment.

* **Install from a Git repository.** Pip can install a package directly from a Git repository over HTTPS:

  ```dockerfile theme={null}
  USER root
  RUN pip install package-name@git+https://github.com/org/repository-name.git
  USER ubuntu
  ```

  <Warning>
    Do not embed Git credentials in the Dockerfile instructions. For packages in private repositories, install them at execution startup with a dependency file instead, using Domino's secure Git integration. See [Package dependency files](/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments/package-dependency-files#install-packages-hosted-from-a-git-repository).
  </Warning>

* **Install from a downloadable archive.** Download a wheel or source archive from a location the Environment builder can reach, then install it:

  ```dockerfile theme={null}
  USER root
  RUN wget https://example.com/packages/your-package-1.0.0-py3-none-any.whl && \
      pip install your-package-1.0.0-py3-none-any.whl && \
      rm your-package-1.0.0-py3-none-any.whl
  USER ubuntu
  ```

* **Install from an internal mirror.** If your organization hosts packages internally, point Conda, pip, or RStudio at the internal repository. See [Internal package hosting](/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments/set-up-internal-package-hosting).

## Dockerfile best practices

* Docker optimizes its build process by keeping track of commands it has run and aggressively caching the results. This means that if it sees the same set of commands as in a previous build, it assumes that it can use the cached version. A single new command invalidates the caching of all subsequent commands.

* A Docker image can have up to 127 layers, or commands. To work around this limit, you can use `&&`: to combine several commands into a single command.

  ```dockerfile theme={null}
  RUN
    wget http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.6.tgz &&
    tar xvzf spark-1.5.1-bin-hadoop2.6.tgz &&
    mv spark-1.5.1-bin-hadoop2.6 /opt &&
    rm spark-1.5.1-bin-hadoop2.6.tgz
  ```

* If you are installing multiple python packages via pip, it’s almost always best to use a single pip install command. This ensures that dependencies and package versions are properly resolved. If you install via separate commands, you may end up inadvertently overriding a package with the wrong version, due a dependency specified by a later installation. For example:

  ```dockerfile theme={null}
  RUN pip install luigi nolearn lasagne
  ```


# Set up internal package hosting
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments/set-up-internal-package-hosting

Configure Conda, pip, and RStudio to use internal package mirrors by default through Dockerfile instructions and repository settings.

Many customers maintain internal mirrors of public package repos. This is typically done while they use a whitelist to remove access to external, public repositories. This might be for security or for standardization.

You can add the following to the Dockerfile instructions in your global [Compute Environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments) to set Conda, pip, or RStudio to reference your internal mirrors by default:

```dockerfile theme={null}
RUN
   # Add $CUSTOMER mirrors to condarc config files. Also disabled auto_update since mirrors are not always up to date.
   printf "
channels:
  - $CUSTOMER_CHANNEL_1
  - $CUSTOMER_CHANNEL_1

" >> /opt/conda/.condarc &&
   printf "
allow_other_channels: False
" >> /opt/conda/.condarc &&
   printf "
ssl_verify: False
" >> /opt/conda/.condarc &&
   printf "
show_channel_urls: True
" >> /opt/conda/.condarc &&
   printf "
auto_update_conda: False
" >> /opt/conda/.condarc &&

   # Pointing R to $CUSTOMER CRAN mirror
   printf "
local({
  r <- getOption('repos')
  r['CRAN'] <- '$CRAN_MIRROR_URL'
 options(repos = r)
})
" >> /home/ubuntu/.Rprofile &&

   # Pointing pip to $CUSTOMER PyPi mirror
   printf "[global]
index-url=$CUSTOMER_PYPI_MIRROR
trusted-host=$IP_OR_FQDN" > /etc/pip.conf &&

   chown -R ubuntu:ubuntu /home/ubuntu /opt/conda/
```


# Use requirements.txt (Python only)
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments/use-requirements-txt

Place requirements.txt in a Project to install Python packages at runtime, generate pinned dependencies, and choose when to preload them instead.

Unless package persistence is enabled, Domino runs each script or interactive session in a fresh environment. In this case, any custom packages are reinstalled at the start of each execution.

To add Python dependencies to your project:

* Create a [requirements.txt](https://pip.pypa.io/en/latest/user_guide/#requirements-files) file in the root of your project directory.

* List the required packages using standard pip syntax.

Domino installs these packages automatically at runtime.

Domino provides flexibility, so if you want your changes to remain permanently installed, see [Preload Environment packages](/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments/preload-environment-packages) to create a custom environment.

Many common modules are installed by default. To get a list of pre-installed modules, include `help('modules')` at the start of your script or in a new IPython Notebook session. You can also run `domino run --direct "pip freeze"` with the Domino CLI tool.

## Improve execution startup time

Using a `requirements.txt` file allows you to specify and install Python dependencies at runtime. However, this approach comes with trade-offs:

* For each execution, Domino installs the listed packages from scratch, which can increase startup time.

To optimize performance based on your use case:

* **For Workspaces:** Enable package persistence to install packages once during initial startup. These packages are then retained between sessions, avoiding repeat installations and significantly reducing startup time for future launches.

* **For Jobs, Apps, and other executions:** Use preloaded Environment packages to build a custom environment where packages are baked into the image. This ensures packages are available immediately and persist across all executions, avoiding any runtime installation cost.

By selecting the correct installation strategy, you can strike a balance between flexibility and faster execution times.

## Add your own packages

To define additional dependencies, create a `requirements.txt` file and place it in the root of your project directory.

<Note>
  If your project is [Git-based](/cloud/platform-capabilities/core-concepts/projects/git-based-projects), the `requirements.txt` file must be located in the `Code/` folder within your project’s directory.
</Note>

The `requirements.txt` file should list all required Python packages using standard pip syntax. You can specify exact versions, minimum versions, or allow the latest versions, along with which libraries to use. For example:

```bash theme={null}
pandas
lxml==3.2.3
numpy>=1.7.1
```

Domino automatically installs the listed packages at the start of each execution, unless package persistence or preloaded environments are used to optimize startup time.

### Install packages interactively in Workspaces

If you’re working in a Jupyter Notebook, you can also install packages interactively using pip directly in a notebook cell:

```bash theme={null}
! pip install --user <package>
```

The '!' prefix runs the command in the shell from within the notebook environment.

## Generate a requirements.txt file

If you’re using pip on your local machine, the easiest way to generate the requirements.txt file is to run the following command in the root of your project folder (or in the Code folder if your project is Git-based):

```bash theme={null}
~/domino/myProject $ pip freeze > requirements.txt
```

For performance reasons, prune the file to include only the libraries needed for your analysis.

## Install packages hosted from a Git repository

<Warning>
  This is an advanced topic.
</Warning>

Pip can install Python packages from source by cloning a public Git repository over HTTPs. See [pip install](https://pip.pypa.io/en/stable/cli/pip_install/#git) for reference. To specify this, you must add something like the following line to your `requirements.txt` file:

```shell theme={null}
-e git+https://git.yourproject.org/you/Project.git#egg=YourProject
```

The most common host of Git projects is GitHub. If the package you want to install is publicly accessible, then the previous instructions work. However, if you must install private repositories, Domino can securely integrate with GitHub to access them. See [Import Git Repositories](/cloud/platform-capabilities/core-concepts/projects/dfs-projects#import-git-repositories) for information about securely storing your Github credentials.

<Warning>
  Do not embed your GitHub credentials directly in the `requirements.txt` file. Instead, integrate Domino with GitHub securely by following the instructions in [Import Git Repositories](/cloud/platform-capabilities/core-concepts/projects/dfs-projects#import-git-repositories).
</Warning>


# Deploy a custom image
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/deploy-a-custom-image

Prepare a custom image for Domino endpoints and Apps by adding Flask, uWSGI, or R packages through Environment Dockerfile instructions.

When you create an Environment, custom images can be made automatically compatible with Domino Jobs and Workspaces when creating an Environment, as explained in [Use external images in Domino Environments](/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images). This topic explains how to use additional Dockerfile instructions to enable custom images for use when publishing models or hosting web applications.

* To do this when creating the Environment, select **Customize before building**.

* To add this capability to an existing Environment, click **Edit Definition** in the top right of the Environment’s overview page.

To publish a Domino Domino endpoint, install uWSGI and Flask as follows:

```dockerfile theme={null}
# Install uWSGI pre-requisites
RUN apt update && apt install -y --no-install-recommends gcc &&
    apt-get clean && rm -rf /var/lib/apt/lists/*

# Add missing library for uWSGI
RUN conda install libpython-static==5.11
ARG LDFLAGS=-fno-lto
ENV LDFLAGS=-fno-lto

# Install Flask & uWSGI
RUN pip install --no-cache-dir Flask Flask-Compress Flask-Cors uwsgi six prometheus-client
```

<Note>
  Replace `5.11` with the specific Python version being used in your Environment.
</Note>

To publish a Domino endpoint with R, you must also add `plumber` and `future` to the Environment:

```dockerfile theme={null}
# Make sure R is installed, then add plumber and future
RUN R --no-save -e "install.packages(c('plumber', 'future'))"
```

The Environment modifications needed to host a web application in Domino depend on the framework you are using; see [App publishing](/cloud/platform-capabilities/features/apps).


# Compute Environment Variables
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/1-compute-environment-variables

Add Compute Environment variables for new revisions, declare each variable with ARG in Dockerfile instructions, and rebuild the environment.

Compute Environment variables provide user, project, and hardware information. These are set at the environment level and used for new revisions.

Environment variables help configure how the system runs and manage secrets or settings across different areas.

## Navigate to environment variables screen

There are two ways to navigate to your environment to add variables.

**Option 1 - From your project**:

1. Open your **Project** and click **Settings**.

2. Click **Manage Environment** > **Edit Definition**.

**Option 2 - From Domino Overview**:

1. Open Domino and click **Govern** > **Environments**.

2. From **Environments Overview**, scroll to locate your Environment.

3. Click the three dots and choose **Edit Definition**.

<img alt="Access variables through main" />

## Add Compute Environment variables

On the **Edit Definition** page, scroll down to the **Environment variables** section.

<img alt="Add compute environment variables" />

1. Click the plus icon and add the following:

   1. **Variable Name**: Enter the variable name. For example, API\_KEY.

   2. **Value**: Enter a value for the variable. The maximum length for a value is 64K.

2. Declare the variable in the Dockerfile instructions with an ARG statement. [Docker’s official site](https://docs.docker.com/engine/reference/builder/) has more detailed information about Docker.

3. Click **Build** after entering new variables and updating the Dockerfile instruction.

## Next steps

* [Our Best Practices](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/1-best-practices-domino-environments) guide you in setting up and maintaining your Domino environments.

* [Set up user environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/4-setup-user-environment-variables) for each user. The system injects these variables at execution time for any run initiated by that user.

* Learn to [create model variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/3-create-model-variables) to add, modify, or delete variables for your models.

* The [Default Domino Variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/5-glossary-default-variables) glossary contains details about variables automatically injected by Domino.


# Project Variables
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/2-build-project-variables

Store Project environment variables for collaborators, set names and values in Project settings, and update or delete shared credentials.

Project owners or collaborators can add, modify, or delete project level environment variables. These variables are stored securely and can hold sensitive information, such as credentials.

<img alt="Set environment variables" />

1. Open your **Project** and click **Settings**.

2. In the **Environment variables** section, add the following:

   1. **Name**: Enter the variable name. For example, `YOUR_NAME_SHARED_KEY`.

   2. **Value**: Enter a value for the variable. The maximum length for a value is 64K.

3. Click **Set Variable**.

You can send the values exactly as they are without adding escaping characters.

## Next steps

* [Our Best Practices](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/1-best-practices-domino-environments) guides you in setting up and maintaining your Domino environments.

* [Set up user environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/4-setup-user-environment-variables) for each user. The system injects these variables at execution time for any run initiated by that user.

* Learn to [create model variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/3-create-model-variables) to add, modify, or delete variables for your models.

* The [Default Domino Variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/5-glossary-default-variables) glossary contains details about variables automatically injected by Domino.


# Model Variables
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/3-create-model-variables

Create model environment variables for active endpoints, restart endpoints to apply changes, and keep model settings separate from Project variables.

Only model owners or editors can create environment variables. When you set environment variables for active Endpoints, you’ll need to restart the Endpoints so they are recognized.

In Domino 5.11, when you add a [model variable](https://docs.dominodatalab.com/en/5.11/user_guide/6ac5a1/environment-variables/), the values are pushed to all running model versions.

Project-level and user-level environment variables are not used in models and must be set separately for each model.

<img alt="Model environment variables" />

1. Open your **Model** and click **Settings**.

2. In the **Environment variables** section, add the following:

   1. **Name**: Enter the variable name. For example, `YOUR_NAME_MODEL_KEY`.

   2. **Value**: Enter a value for the variable. The maximum length for a value is 64K.

3. Click **Set Variable**.

## Next steps

* [Our Best Practices](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/1-best-practices-domino-environments) guide you in setting up and maintaining your Domino environments.

* [Set up user environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/4-setup-user-environment-variables) for each user. The system injects these variables at execution time for any run initiated by that user.

* Learn to [build project variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/2-build-project-variables) to add, modify, or delete environment variables for your projects.

* The [Default Domino Variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/5-glossary-default-variables) glossary contains details about variables automatically injected by Domino.


# User Environment Variables
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/4-setup-user-environment-variables

Manage private user environment variables in Account Settings and access injected values from Python or R across Projects and executions.

In Domino, you can create, update, or delete your private environment variables. These variables are stored securely in Vault and only you can see them. The system injects these variables at execution time for any run initiated by you.

User environment variables are automatically imported into runs for all projects. However, user-specific environment variables cannot be used in models.

<img alt="Set user environment variables" />

1. Click **Account > Account Settings** to open the Account Settings page.

2. Choose **User environment variables** from the list.

3. In the **Environment variables** section, add the following:

   1. **Name**: Enter the variable name. For example, `YOUR_NAME_ENV_KEY`.

   2. **Value**: Enter a value for the variable. The maximum length for a value is 64K.

4. Click **Set Variable**.

You can send the values exactly as they are without adding escaping characters.

## Access environment variables

Every language reads environment variables uniquely.

* [In Python](https://docs.python.org/2/library/os.html#os.environ), it might look like this:

```python theme={null}
import os
s3 = S3Client(os.environ['S3_KEY'], os.environ['S3_SECRET'])
```

Run this to retrieve an environment variable within your code for Python:

```python theme={null}
import os
os.environ['DOMINO_RUN_ID']
```

* [In R](http://stat.ethz.ch/R-manual/R-patched/library/base/html/Sys.getenv.html), it might look like this:

```r theme={null}
makeS3Client(Sys.getenv("S3_KEY"), Sys.getenv("S3_SECRET"))
```

Run this to retrieve an environment variable within your code for R:

```r theme={null}
Sys.getenv("DOMINO_RUN_ID")
```

## Next steps

* [Our Best Practices](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/1-best-practices-domino-environments) guide you in setting up and maintaining your Domino environments.

* [Set up user environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/4-setup-user-environment-variables) for each user. The system injects these variables at execution time for any run initiated by that user.

* Learn to [build project variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/2-build-project-variables) to add, modify, or delete environment variables for your projects.

* The [Default Domino Variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/5-glossary-default-variables) glossary contains details about variables automatically injected by Domino.


# Default Domino Environment Variables Glossary
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/5-glossary-default-variables

Look up Domino's default environment variables for Project, run, user, Dataset, Artifact, API, hardware, and working directory context.

Domino automatically injects the following environment variables whenever it runs your code as part of the context of your run.

Using these variables, you can create artifacts or results, such as a report mentioning the run number. Default variables are not available to use in the Model Manager.

| Variable | Description |
| - | - |
| `AWS_SHARED_CREDENTIALS_FILE` | Path to your AWS credential file to connect to additional AWS resources (for example, S3, Redshift). |
| `DOMINO_API_HOST` | Use this to use the [Domino API](/cloud/reference/api) to access another project. |
| `DOMINO_ARTIFACTS_DIR` | Directory where Artifacts files system is accessed. |
| `DOMINO_DATASETS_DIR` | Directory where Datasets are accessed. |
| `DOMINO_GROUP` | Value of the default Linux group name in the container, that is `ubuntu`. |
| `DOMINO_GROUP_ID` | Value of default Linux group ID in the container, that is `12574`. |
| `DOMINO_HARDWARE_TIER_ID` | Hardware tier the current run is executing on (new in 1.42). |
| `DOMINO_IMPORTED_ARTIFACTS_DIR` | Directory where imported Artifacts file systems are accessed. |
| `DOMINO_IMPORTED_CODE_DIR` | Directory where imported code repositories are accessed. |
| `DOMINO_IMPORTED_DATA_DIR` | Directory where imported Datasets are accessed. |
| `DOMINO_IS_GIT_BASED` | Boolean value indicating if the Project is Git-based. |
| `DOMINO_IS_LOCAL_DATA_PLANE` | Boolean value indicating if Datasets are associated with a local data plane. |
| `DOMINO_IS_WORKFLOW_JOB` | Boolean value indicating if the job was orchestrated via a Flow. |
| `DOMINO_NODE_IP` | The compute node IP. |
| `DOMINO_NODE_NAME` | Name of the compute node. |
| `DOMINO_PROJECT_ID` | Project ID of the current project. |
| `DOMINO_PROJECT_NAME` | Name of the running project. |
| `DOMINO_PROJECT_OWNER` | Username of the owner of the running project. |
| `DOMINO_RUN_ID` | Run ID of the current run. |
| `DOMINO_RUN_NUMBER` | Run number of the current run. |
| `DOMINO_STARTING_USERNAME` | Username of the user who began the run (new in 1.43). |
| `DOMINO_TOKEN_FILE` | No longer exists as an environment variable. Instead, [Use a Token for Authentication](/cloud/reference/api/domino-api-authentication). |
| `DOMINO_TRAINING_SET_PATH` | Directory where Domino Model Monitoring training sets are accessed. |
| `DOMINO_USER` | Value of the default Linux username in the container, that is `ubuntu`. |
| `DOMINO_USER_API_KEY` | This variable is sunsetting; use the [API Proxy for Domino API authentication](/cloud/reference/api/domino-api-authentication) instead. |
| `DOMINO_USER_ID` | Value of the default Linux user ID in the container, that is `12674`. |
| `DOMINO_USER_NAME` | Domino Username of the person running the Workspace/Job. |
| `DOMINO_WORKING_DIR` | Working directory for the running project. |

## Next steps

* [Our Best Practices](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/1-best-practices-domino-environments) guides you in setting up and maintaining your Domino environments.

* [Set up user environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/4-setup-user-environment-variables) for each user. The system injects these variables at execution time for any run initiated by that user.

* Learn to [build project variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/2-build-project-variables) to add, modify, or delete environment variables for your projects.


# Manage Environment Variables
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/index

Manage variables at Compute Environment, Project, model, and user levels, understand precedence, and avoid Domino's reserved names and prefixes.

Domino allows users to set environment variables at four levels: Compute Environments, Projects, Models, and User. These variables help configure how the system runs and manage secrets or settings across different areas.

* [Compute Environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/1-compute-environment-variables) provide user, project, and hardware information. These are set at the environment level and used for new revisions.

* [Project Environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/2-build-project-variables) are set at the project level and shared with all project collaborators. They help with configurations specific to the project and ensure consistency in all runs within that project.

* [Model Level variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/3-create-model-variables) are used for deployed models. They help manage settings needed for running the model, like API keys or endpoints for external services.

* [User Level variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/4-setup-user-environment-variables) are specific to each user and apply to all projects and tasks they start. They are helpful for personal settings like login details or default preferences.

You can create your own or use Domino’s environment variables to add values to your runs. These variables must start with letters and can only include letters, numbers, and underscores.

## Hierarchy of variables

You can use the same variable in different places. When Domino loads a run or workspace, it pulls environment variables from these sources in this order:

1. **Compute Environment variables**

2. **Project environment variables**

3. **User Level variables**

Here is an example of the expected run results with the same variable in multiple places:

| Place set | Run #1 | Run #2 | Run #3 |
| - | - | - | - |
| Compute Environment | A | A | A |
| Project | - | B | B |
| User Account | - | - | C |
| **Run Result** | **A** | **B** | **C** |

## Reserved variable names in Domino

Domino uses specific environment variables to configure and manage operations. These variables should not be defined at the user or project level. Here are a few generic ones:

* `USER`

* `LOGNAME`

* `MLFLOW_TRACKING_URI`

In addition to the above, any variable that starts with the following is reserved:

* `DOMINO_*`

* `KUBERNETES_*`

* `NEW_RELIC_*`

* `DATA_PLANE_*`

* `MINICONDA_*`

* `FLYTE_*`

* `HEPHAESTUS_*`

* `SELDON_*`

* `CONDA_*`

* `DISTRIBUTED_COMPUTE_*`

We recommend using a common prefix, such as YOUR\_COMPANY NAME\_, for all custom environment variables to avoid collisions with [Domino environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/5-glossary-default-variables).

## Next steps

* The [Default Domino Variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/5-glossary-default-variables) glossary contains details about variables automatically injected by Domino.

* [Set up user environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/4-setup-user-environment-variables) for each user. The system injects these variables at execution time for any run initiated by that user.

* Learn to [build project variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/2-build-project-variables) to add, modify, or delete environment variables for your projects.

* Learn to [create model variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/3-create-model-variables) to add, modify, or delete variables for your models.


# Pre-requisites for automatic custom image compatibility
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/automatic-custom-image-compatibility

Check whether a custom image includes bash, shadow-utils, required system libraries, Python, and Jupyter for automatic Domino compatibility.

The following requirements must be present in custom images for use with the automatic compatibility feature:

* `bash` 4.2+

* `shadow-utils`, including

  * `groupadd`

  * `useradd`

* `glibc` 2.23+, including:

  * `libc.so.6`

  * `libpthread.so.0`

  * `libcrypt.so.1`

  * `libutil.so.1`

  * `libdl.so.2`

  * `libm.so.6`

These requirements are typically satisfied by images based on the following off-the-shelf distributions of Linux:

* CentOS Linux 7 (or later)

* Debian GNU/Linux 10 (or later)

* Red Hat Enterprise Linux 8.2 (or later)

* Ubuntu 16.04.2 LTS (or later)

<Note>
  Alpine Linux distribution images are not officially supported. Alpine images might work if the previous requirements are met, but this has not been validated.
</Note>

The following table shows the requirements that must be present in the image for each of the default Domino Workspace Tools, as well as the requirements for Domino Jobs. If a tool isn’t working, check to see if its requirements are present in the image. You can modify the base image or add Dockerfile instructions to the Domino Environment if there are missing requirements.

| Workspace Tool | Requires Python | Requires Jupyter |
| - | - | - |
| Domino Jobs | No | No |
| VS Code | Yes | No |
| JupyterLab (kernels present) | Yes | Yes |
| JupyterLab (terminal only) | No | No |

<Note>
  * Jobs can be used without Python or Jupyter. However, if you want to run a Python script in a job, then you need Python in the custom image.

  * JupyterLab can be launched even if Python and Jupyter are not installed, but only a basic terminal is available and the JupyterLab "Launcher" is empty. To use a terminal, open JupyterLab and then click **File** > **New** > **Terminal**.
</Note>


# AWS Trainium and Inferentia silicon accelerators
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/aws-accelerators

Configure AWS Trainium and Inferentia for Domino with Neuron node groups, device plugins, Hardware Tiers, environments, and validation commands.

Domino supports the use of cost and energy-efficient AWS-designed silicon processors, [AWS Trainum](https://aws.amazon.com/machine-learning/trainium/) and [Inferentia](https://aws.amazon.com/machine-learning/inferentia/), to accelerate deep-learning model training and AI inference workloads. Use the AWS Neuron SDK to reuse existing code. Learn how to set up Trainium and Inferentia accelerators in your Domino.

Set up involves the following:

1. **Node group creation**: Create a new node group for Trainium and Inferentia instances.

2. **Device plugin configuration**: Provide hardware-specific settings.

3. **Hardware tier setup**: Enable Domino users to use Trainium and Inferentia instances for their workloads.

4. **Environment configuration**: Set up the necessary development tools and software libraries.

## Node group creation

To use AWS accelerators, create a new node group that:

* Uses one of the instance types in the following table

* Uses the [GPU-enabled AMI for your EKS version](https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html)

* Has a unique node pool label identifying its accelerator type

| Name | vCPU | Memory (GiB) | aws.amazon.com/neuron | Total Neuron Memory (GiB) |
| - | - | - | - | - |
| inf1.xlarge | 4 | 8 | 1 | 8 |
| inf1.2xlarge | 8 | 16 | 1 | 8 |
| inf1.6xlarge | 24 | 48 | 4 | 32 |
| inf1.24xlarge | 96 | 192 | 16 | 128 |
| inf2.xlarge | 4 | 16 | 1 | 32 |
| inf2.8xlarge | 32 | 128 | 1 | 32 |
| inf2.24xlarge | 96 | 384 | 6 | 192 |
| inf2.48xlarge | 192 | 768 | 12 | 384 |
| trn1.2xlarge | 8 | 32 | 1 | 32 |
| trn1.32xlarge | 128 | 512 | 16 | 512 |

For the cluster-autoscaler to scale your Neuron-based node group successfully, you must tag those autoscaling groups with the Neuron device resource template, like the `k8s.io/cluster-autoscaler/node-template/resources/aws.amazon.com/neuron` tag in the table below:

| Key | Value | Tag new instances |
| - | - | - |
| Name | inferentia-test-domino-trn1-Node | Yes |
| alpha.eksctl.io/cluster-name | inferentia-test | Yes |
| alpha.eksctl.io/eksctl-version | 0.155.0 | Yes |
| alpha.eksctl.io/nodegroup-name | domino-trn1 | Yes |
| alpha.eksctl.io/nodegroup-type | unmanaged | Yes |
| aws:cloudformation:logical-id | NodeGroup | Yes |
| aws:cloudformation:stack-id | arn:aws:cloudformation:us-west-2:873872646799:stack/eksctl-inferentia-test-n…​ | Yes |
| aws:cloudformation:stack-name | eksctl-inferentia-test-nodegroup-domino-trn1 | Yes |
| `eksctl.cluster.k8s.io/v1alpha1/cluster-name` | inferentia-test | Yes |
| eksctl.io/v1alpha2/nodegroup-name | domino-trn1 | Yes |
| `k8s.io/cluster-autoscaler/enabled` | true | Yes |
| `k8s.io/cluster-autoscaler/inferentia-test` | owned | Yes |
| `k8s.io/cluster-autoscaler/node-template/label/dominodatalab.com/node-pool` | trainium | Yes |
| `k8s.io/cluster-autoscaler/node-template/resources/aws.amazon.com/neuron` | 1 | Yes |
| kubernetes.io/cluster/inferentia-test | owned | Yes |

### Example eksctl node group config

Here’s an example eksctl node group config for Neuron-based node groups:

```shell theme={null}
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: inferentia-test
  region: us-west-2

nodeGroups:
  - name: domino-trn1
    instanceType: trn1.2xlarge
    minSize: 0
    maxSize: 3
    desiredCapacity: 1
    volumeSize: 200
    volumeType: gp3
    availabilityZones: ["us-west-2a"]
    labels:
      "dominodatalab.com/node-pool": "trainium"
    tags:
      k8s.io/cluster-autoscaler/node-template/dominodatalab.com/node-pool: trainium
      k8s.io/cluster-autoscaler/node-template/resources/aws.amazon.com/neuron: 1
    iam:
      withAddonPolicies:
        ebs: true
        efs: true
  - name: domino-inf1
    instanceType: inf1.2xlarge
    minSize: 0
    maxSize: 3
    desiredCapacity: 1
    volumeSize: 200
    volumeType: gp3
    availabilityZones: ["us-west-2a"]
    labels:
      "dominodatalab.com/node-pool": "inferentia"
    tags:
      k8s.io/cluster-autoscaler/node-template/dominodatalab.com/node-pool: inferentia
      k8s.io/cluster-autoscaler/node-template/resources/aws.amazon.com/neuron: 1
    iam:
      withAddonPolicies:
        ebs: true
        efs: true
  - name: domino-inf2
    instanceType: inf2.2xlarge
    minSize: 0
    maxSize: 3
    desiredCapacity: 1
    volumeSize: 200
    volumeType: gp3
    availabilityZones: ["us-west-2a"]
    labels:
      "dominodatalab.com/node-pool": "inferentia2"
    tags:
      k8s.io/cluster-autoscaler/node-template/dominodatalab.com/node-pool: inferentia2
      k8s.io/cluster-autoscaler/node-template/resources/aws.amazon.com/neuron: 1
    iam:
      withAddonPolicies:
        ebs: true
        efs: true
```

## Device plugin deployment

Once your nodes have joined the cluster, deploy the Neuron device plugin DaemonSet using the following specification. You must use version `2.17.3.0` or greater for Domino workloads to be correctly processed by the device plugin.

To deploy this DaemonSet:

1. Save the following specification to a file (such as `neuron-device-plugin-ds.yaml`).

2. Apply the specification with `kubectl apply -f neuron-device-plugin-ds.yaml`.

   ```shell theme={null}
   ---
   kind: ClusterRole
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     name: neuron-device-plugin
   rules:
   - apiGroups:
     - ""
     resources:
     - nodes
     verbs:
     - get
     - list
     - watch
   - apiGroups:
     - ""
     resources:
     - events
     verbs:
     - create
     - patch
   - apiGroups:
     - ""
     resources:
     - pods
     verbs:
     - update
     - patch
     - get
     - list
     - watch
   - apiGroups:
     - ""
     resources:
     - nodes/status
     verbs:
     - patch
     - update
   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: neuron-device-plugin
     namespace: kube-system
   ---
   kind: ClusterRoleBinding
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     name: neuron-device-plugin
     namespace: kube-system
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: neuron-device-plugin
   subjects:
   - kind: ServiceAccount
     name: neuron-device-plugin
     namespace: kube-system
   ---
   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: neuron-device-plugin-daemonset
     namespace: kube-system
   spec:
     revisionHistoryLimit: 10
     selector:
       matchLabels:
         name: neuron-device-plugin-ds
     template:
       metadata:
         annotations:
           scheduler.alpha.kubernetes.io/critical-pod: ""
         creationTimestamp: null
         labels:
           name: neuron-device-plugin-ds
       spec:
         affinity:
           nodeAffinity:
             requiredDuringSchedulingIgnoredDuringExecution:
               nodeSelectorTerms:
               - matchExpressions:
                 - key: node.kubernetes.io/instance-type
                   operator: In
                   values:
                   - inf1.xlarge
                   - inf1.2xlarge
                   - inf1.6xlarge
                   - inf1.24xlarge
                   - inf2.xlarge
                   - inf2.8xlarge
                   - inf2.24xlarge
                   - inf2.48xlarge
                   - trn1.2xlarge
                   - trn1.32xlarge
         containers:
         - env:
           - name: KUBECONFIG
             value: /etc/kubernetes/kubelet.conf
           - name: NODE_NAME
             valueFrom:
               fieldRef:
                 apiVersion: v1
                 fieldPath: spec.nodeName
           image: public.ecr.aws/neuron/neuron-device-plugin:2.17.3.0
           imagePullPolicy: Always
           name: k8s-neuron-device-plugin-ctr
           resources: {}
           securityContext:
             allowPrivilegeEscalation: false
             capabilities:
               drop:
               - ALL
           terminationMessagePath: /dev/termination-log
           terminationMessagePolicy: File
           volumeMounts:
           - mountPath: /var/lib/kubelet/device-plugins
             name: device-plugin
           - mountPath: /run
             name: infa-map
         dnsPolicy: ClusterFirst
         priorityClassName: system-node-critical
         restartPolicy: Always
         schedulerName: default-scheduler
         securityContext: {}
         serviceAccount: neuron-device-plugin
         serviceAccountName: neuron-device-plugin
         terminationGracePeriodSeconds: 30
         tolerations:
         - key: CriticalAddonsOnly
           operator: Exists
         - effect: NoSchedule
           key: aws.amazon.com/neuron
           operator: Exists
         volumes:
         - hostPath:
             path: /var/lib/kubelet/device-plugins
             type: ""
           name: device-plugin
         - hostPath:
             path: /run
             type: ""
           name: infa-map
     updateStrategy:
       rollingUpdate:
         maxSurge: 0
         maxUnavailable: 1
       type: RollingUpdate
   ```

3. Once the device plugin DaemonSet is deployed, run `kubectl describe node` to confirm that you see device plugin daemons running on your Neuron-based instances, and that they advertise `aws.amazon.com/neuron` resources to Kubernetes.

The following output is an example of a correctly configured Neuron-based node. Note the Neuron device plugin daemon present on the node, the advertised `aws.amazon.com/neuron` resource, and the Domino node pool label identifying the node as Trainium.

```shell theme={null}
Name:               ip-192-168-42-179.us-west-2.compute.internal
Roles:              <none>
Labels:             alpha.eksctl.io/cluster-name=inferentia-test
                    alpha.eksctl.io/instance-id=i-00549360c9911f4f1
                    alpha.eksctl.io/nodegroup-name=domino-trn1
                    beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=trn1.2xlarge
                    beta.kubernetes.io/os=linux
                    dominodatalab.com/node-pool=trainium
                    failure-domain.beta.kubernetes.io/region=us-west-2
                    failure-domain.beta.kubernetes.io/zone=us-west-2b
                    k8s.io/cloud-provider-aws=7c4bfb478ecbb2400bead13fc878a3a1
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ip-192-168-42-179.us-west-2.compute.internal
                    kubernetes.io/os=linux
                    node-lifecycle=on-demand
                    node.kubernetes.io/instance-type=trn1.2xlarge
                    topology.ebs.csi.aws.com/zone=us-west-2b
                    topology.kubernetes.io/region=us-west-2
                    topology.kubernetes.io/zone=us-west-2b

... <snip> ...

Capacity:
  aws.amazon.com/neuron:        1
  aws.amazon.com/neuroncore:    2
  aws.amazon.com/neurondevice:  1
  cpu:                          8
  ephemeral-storage:            209702892Ki
  hugepages-1Gi:                0
  hugepages-2Mi:                0
  memory:                       32338380Ki
  pods:                         58
  smarter-devices/fuse:         20
Allocatable:
  aws.amazon.com/neuron:        1
  aws.amazon.com/neuroncore:    2
  aws.amazon.com/neurondevice:  1
  cpu:                          7910m
  ephemeral-storage:            192188443124
  hugepages-1Gi:                0
  hugepages-2Mi:                0
  memory:                       31321548Ki
  pods:                         58
  smarter-devices/fuse:         20
System Info:
  Machine ID:                 ec240d0453aef36a07d5248a753946c5
  System UUID:                ec240d04-53ae-f36a-07d5-248a753946c5
  Boot ID:                    9d066b81-273b-406d-ad0f-6375adebca5d
  Kernel Version:             5.4.253-167.359.amzn2.x86_64
  OS Image:                   Amazon Linux 2
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.6.19
  Kubelet Version:            v1.26.7-eks-8ccc7ba
  Kube-Proxy Version:         v1.26.7-eks-8ccc7ba
ProviderID:                   aws:///us-west-2b/i-00549360c9911f4f1
Non-terminated Pods:          (11 in total)
  Namespace                   Name                                    CPU Requests  CPU Limits   Memory Requests  Memory Limits  Age
  ---------                   ----                                    ------------  ----------   ---------------  -------------  ---
  domino-compute              run-650b32d5eb80df0d60cec514-kgcc4      5610m (70%)   6600m (83%)  28174Mi (92%)    28174Mi (92%)  3h17m
  domino-platform             aws-ebs-csi-driver-node-lwp9g           30m (0%)      200m (2%)    80Mi (0%)        456Mi (1%)     3h10m
  domino-platform             aws-efs-csi-driver-node-kk7mb           20m (0%)      200m (2%)    40Mi (0%)        200Mi (0%)     3h10m
  domino-platform             docker-registry-cert-mgr-k67bb          0 (0%)        0 (0%)       0 (0%)           0 (0%)         3h10m
  domino-platform             fluentd-ljkhk                           200m (2%)     1 (12%)      600Mi (1%)       2Gi (6%)       3h10m
  domino-platform             image-cache-agent-7ztv6                 0 (0%)        0 (0%)       0 (0%)           0 (0%)         3h10m
  domino-platform             prometheus-node-exporter-vr9bc          0 (0%)        0 (0%)       0 (0%)           0 (0%)         3h10m
  domino-platform             smarter-device-manager-ncsr6            10m (0%)      100m (1%)    15Mi (0%)        15Mi (0%)      3h10m
  kube-system                 aws-node-q89xl                          25m (0%)      0 (0%)       0 (0%)           0 (0%)         3h10m
  kube-system                 kube-proxy-b2qpv                        100m (1%)     0 (0%)       0 (0%)           0 (0%)         3h10m
  kube-system                 neuron-device-plugin-daemonset-h8d7k    0 (0%)        0 (0%)       0 (0%)           0 (0%)         3h10m
```

## Hardware Tier setup

Next, you need to make the node group accessible to your users by creating a Domino hardware tier that does the following:

* Targets the node pool label you’ve given to your Neuron-based nodes.

* Requests a suitable amount of the node vCPU and memory, [allowing for necessary overhead](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/hardware-tier-best-practices).

* Requests a custom GPU resource with the name `aws.amazon.com/neuron`.

See the following example:

| Key | Value |
| - | - |
| Cluster Type | Kubernetes |
| ID | trainium |
| Name | Trainium |
| Cores Requested | 5.0 |
| Memory Requested (GiB) | 26.0 |
| Number of GPUs | 1 |
| Use custom GPU name | Yes |
| GPU Resource Name | aws.amazon.com/neuron |
| Cents Per Minute Per Run | 0.0 |
| Node Pool | trainium |
| Restrict to compute cluster | Options: Spark, Ray, Dask, Mpi |
| Maximum Simultaneous Executions | |
| Overprovisioning Pods | 0 |

## Environment setup

The AWS Neuron SDK is designed for use with fully integrated ML frameworks such as PyTorch. When setting up a Domino environment for a new version of Neuron or the integrated framework, read [ML framework support on AWS Neuron SDK](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/frameworks/index.html), which links the setup guide and hardware compatibility matrix for each framework.

As an example, and to facilitate testing, here’s an environment definition for adding PyTorch Neuron to the Domino 5.7 Standard Environment (`quay.io/domino/compute-environment-images:ubuntu20-py3.9-r4.3-domino5.7-standard`):

```shell theme={null}
# Configure Linux for Neuron repository updates
RUN sudo touch /etc/apt/sources.list.d/neuron.list
RUN echo "deb https://apt.repos.neuron.amazonaws.com focal main" | sudo tee -a /etc/apt/sources.list.d/neuron.list
RUN sudo wget -qO - https://apt.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB | sudo apt-key add -

# Update OS packages
RUN sudo apt-get update -y

# Install Neuron Driver
RUN sudo apt-get install aws-neuronx-dkms=2.* -y

# Install Neuron Runtime
RUN sudo apt-get install aws-neuronx-collectives=2.* -y
RUN sudo apt-get install aws-neuronx-runtime-lib=2.* -y

# Install Neuron Tools
RUN sudo apt-get install aws-neuronx-tools=2.* -y

# Add PATH
RUN export PATH=/opt/aws/neuron/bin:$PATH

# pip installs
RUN python -m pip config set global.extra-index-url https://pip.repos.neuron.amazonaws.com
RUN python -m pip install --user neuronx-cc==2.* torch-neuronx torchvision
```

## Testing Neuron devices in Domino

To test your setup, start a Jupyter workspace using a Neuron-based **hardware tier** and Neuron-enabled **Workspace Environment**.

Once your workspace has started, open a Python notebook and execute a cell with the command `!/opt/aws/neuron/bin/neuron-ls` to see mounted Neuron devices.

You can now use the Neuron framework you’ve installed to invoke the mounted accelerator.

## Next steps

Refer to the [Getting Started with Neuron guide](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/quick-start/index.html) for your chosen framework to get started.


# Use external images in Domino Environments
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/index

Use external Docker images for NVIDIA NGC, AWS accelerators, Intel AI Kit, MATLAB, and Stata, with automatic or manual compatibility.

Domino can automatically adapt most Docker images and containers This functionality makes it easy to use pre-built images from external catalogs like the GPU-optimized [NVIDIA GPU Cloud (NGC)](https://catalog.ngc.nvidia.com/containers) containers in Domino.

[Create an NVIDIA NGC Environment in Domino](/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/nvidia-ngc)\
Get the path for the NGC container and then create the image.

[Create a Stata Environment in Domino](/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/stata)\
Create the Environment needed to use Stata in Domino.

[Use AWS-designed silicon accelerators](/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/aws-accelerators)\
Use AWS-designed silicon processors to accelerate deep-learning model training and AI inference workloads.

[Create an Intel AI Kit Environment in Domino](/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/intel-ai-kit)\
Use the Intel AI Kit container image natively in Domino.

[Create a MATLAB Environment in Domino](/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/matlab)\
Set up the Environment for an interactive MATLAB® session in Domino.

[Create a Posit Workbench Environment in Domino](/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/posit-workbench)\
Set up the Environment to use Posit Workbench and the Positron IDE in Domino.

[Manually create an Environment with a pre-built image](/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/manually-create-an-environment)\
Customize images for use in Domino without using the automatic compatibility feature.

[Pre-requisites for automatic custom image compatibility](/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/automatic-custom-image-compatibility)\
Requirements for custom images that use the automatic compatibility feature.


# Create an Intel AI Kit Environment
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/intel-ai-kit

Build an Intel AI Kit Environment from an intel/oneapi-aikit image, review included oneAPI packages, and launch a Workspace with it.

The Intel OneAPI and AI Analytics Toolkit (AI Kit) provides familiar Python tools and frameworks to accelerate end-to-end data science and analytics pipelines on Intel® architecture. The components are built using oneAPI libraries for low-level compute optimizations. This toolkit maximizes performance from preprocessing through machine learning and provides interoperability for efficient model development.

## Run AI Kit in Domino

Many pre-built AI Kit Docker images run natively in Domino. See [https://hub.docker.com/r/intel/oneapi-aikit](https://hub.docker.com/r/intel/oneapi-aikit) for a list of available images on Dockerhub.

1. [Create a new Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/3-create-an-environment) from a custom image.

2. Paste or type the Docker registry path of the image that you want to use in the **FROM** field. For example, `intel/oneapi-aikit:latest`.

   The Environment includes the following packages:

   ```
   intel-aikit-getting-started
   intel-basekit-getting-started
   intel-level-zero-gpu
   intel-oneapi-advisor
   intel-oneapi-ccl-devel
   intel-oneapi-common-licensing
   intel-oneapi-common-vars
   intel-oneapi-compiler-dpcpp-cpp
   intel-oneapi-dal-devel
   intel-oneapi-dev-utilities
   intel-oneapi-dnnl-devel
   intel-oneapi-dpcpp-debugger
   intel-oneapi-ippcp-devel
   intel-oneapi-ipp-devel
   intel-oneapi-libdpstd-devel
   intel-oneapi-mkl-devel
   intel-oneapi-model-zoo
   intel-oneapi-onevpl-devel
   intel-oneapi-python
   intel-oneapi-pytorch
   intel-oneapi-tbb-devel
   intel-oneapi-tensorflow
   intel-oneapi-vtune
   intel-opencl-icd
   level-zero
   level-zero-dev
   ```

3. After the environment is built, [create a new Workspace](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace).

See the following for more information:

* [Intel AI Analytics Toolkit (AI Kit)](https://www.intel.com/content/www/us/en/developer/topic-technology/ai-pc/overview.html)

* [AI Kit containers on Dockerhub](https://hub.docker.com/r/intel/oneapi-aikit)

* [Code samples](https://github.com/oneapi-src/oneAPI-samples/tree/master/AI-and-Analytics) on GitHub


# Manually create an Environment with a pre-built image
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/manually-create-an-environment

Make a pre-built image compatible with Domino by creating its user, directories, packages, start scripts, and optional JupyterLab or VS Code tools.

<Warning>
  This is an advanced topic to assist with customizing images for use in Domino without using the automatic compatibility feature.
</Warning>

When you create an Environment with the automatic compatibility feature selected, Domino adds several Dockerfile commands so that the custom images can launch Job and Workspace containers.

At a high level, these instructions perform the following actions:

* Create a non-root Domino user.

* Install `sudo` and `curl`.

  For these images to work in Domino, you must have `Sudo` and `curl` installations. `curl` transfers data. Use `sudo` to install additional packages and libraries in the container, depending on the permissions set up by your Domino administrator.

* Add an isolated directory, `/opt/domino/`, with the tools required to run Domino Workspaces.

If you have enabled non-root executions, refer to [Create the image compatible with non-root executions](#create-the-image-compatible-with-non-root-executions).

## Create the image

1. Use the following Dockerfile instructions to replicate these steps when building a custom image.

   ```dockerfile theme={null}
   # Ensure you are injecting as the root user
   USER root

   # Add a Domino user and group ID
   ARG DOMINO_USER='domino'
   ARG DOMINO_GROUP='domino'

   # The variable names that can be used in generation of Environment scripts
   ARG ALLENV="$INSTALL_DIR,$INSTALL_BIN,$CONDA_DIR,$DOMINO_UID,$DOMINO_GID,$DOMINO_USER,$DOMINO_GROUP,$VSCODE_DIR,$VSCODE_EXT_DIR"
   RUN groupadd -g 12574 ${DOMINO_GROUP} && \
     useradd -u 12574 -g 12574 -m -N -s /bin/bash ${DOMINO_USER};

   # Install common dependencies for the compiler and setting things up
   ARG INSTALL_DIR=/opt/domino
   ARG INSTALL_BIN=${INSTALL_DIR}/bin
   ARG CONDA_DIR=${INSTALL_DIR}/conda
   RUN apt-get update && \
       apt-get -y install \
       build-essential \
       gettext-base \
       gnupg && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/* && \
     mkdir -p \
         ${INSTALL_DIR} \
         ${INSTALL_BIN} \
         ${CONDA_DIR}

   # Recommended: Add domino user account to sudoers
   RUN echo "${DOMINO_USER}    ALL=NOPASSWD: ALL" >> /etc/sudoers

   # Set to a Domino supported language to prevent unrecognized character input
   RUN echo "export LANG=${LANG}" >> /home/${DOMINO_USER}/.domino-defaults && \
       echo "export LC_ALL=${LANG}" >> /home/${DOMINO_USER}/.domino-defaults && \
       echo 'export PYTHONIOENCODING=utf-8' >> /home/${DOMINO_USER}/.domino-defaults && \
       chown -R ${DOMINO_USER}.${DOMINO_GROUP} "/home/${DOMINO_USER}"

   # Load Domino defaults
   RUN if [ -f /home/${DOMINO_USER}/.domino-defaults ]; then \
           echo "source /home/${DOMINO_USER}/.domino-defaults" >> /home/${DOMINO_USER}/.bashrc; \
       fi

   # Install Sudo and Curl
   # Note that this will only work with Debian and Ubuntu
   RUN apt update && apt install -y --no-install-recommends \
           curl \
           sudo \
       apt-get clean && \
       rm -rf /var/lib/apt/lists/*
   ```

2. If you want JupyterLab or any IDE in this workspace, define a start script for each. See [Replace Default Environment Tools](/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/configure-native-workspaces) for an example. These start scripts are stored in `/opt/domino/workspaces` by default.

3. Add a new field to your Environment’s **Pluggable Workspace Tools** for the associated tool and set the **start** field to the location of your start script.

## Create the image compatible with non-root executions

Images shouldn’t add users to sudoers, according to the non-root executions principle. It is recommended to build environment images with the packages you need, rather than relying on a root user for runtime package installation, for increased reproducibility and security in your environment.

1. Use the following Dockerfile instructions to replicate these steps when building a custom image.

   ```dockerfile theme={null}
   # Add a Domino user and group ID
   ARG DOMINO_USER='domino'
   ARG DOMINO_GROUP='domino'

   # The variable names that can be used in generation of environment scripts
   ARG ALLENV="$INSTALL_DIR,$INSTALL_BIN,$CONDA_DIR,$DOMINO_UID,$DOMINO_GID,$DOMINO_USER,$DOMINO_GROUP,$VSCODE_DIR,$VSCODE_EXT_DIR"
   RUN groupadd -g 12574 ${DOMINO_GROUP} && \
     useradd -u 12574 -g 12574 -m -N -s /bin/bash ${DOMINO_USER};

   # Install common dependencies for the compiler and setting things up
   ARG INSTALL_DIR=/opt/domino
   ARG INSTALL_BIN=${INSTALL_DIR}/bin
   ARG CONDA_DIR=${INSTALL_DIR}/conda
   RUN apt-get update && \
       apt-get -y install \
       build-essential \
       gettext-base \
       gnupg && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/* && \
     mkdir -p \
         ${INSTALL_DIR} \
         ${INSTALL_BIN} \
         ${CONDA_DIR}

   # Set to a Domino supported language to prevent unrecognized character input
   RUN echo "export LANG=${LANG}" >> /home/${DOMINO_USER}/.domino-defaults && \
       echo "export LC_ALL=${LANG}" >> /home/${DOMINO_USER}/.domino-defaults && \
       echo 'export PYTHONIOENCODING=utf-8' >> /home/${DOMINO_USER}/.domino-defaults && \
       chown -R ${DOMINO_USER}.${DOMINO_GROUP} "/home/${DOMINO_USER}"

   # Load Domino defaults
   RUN if [ -f /home/${DOMINO_USER}/.domino-defaults ]; then \
           echo "source /home/${DOMINO_USER}/.domino-defaults" >> /home/${DOMINO_USER}/.bashrc; \
       fi

   # Install Curl
   # Note that this will only work with Debian and Ubuntu
   RUN apt update && apt install -y --no-install-recommends \
           curl \
       apt-get clean && \
       rm -rf /var/lib/apt/lists/*
   ```

2. If you want JupyterLab or any IDE in this workspace, define a start script for each. See [Replace Default Environment Tools](/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/configure-native-workspaces) for an example. These start scripts are stored in `/opt/domino/workspaces` by default.

3. Add a new field to your environment’s **Pluggable Workspace Tools** for the associated tool and set the **start** field to the location of your start script.

## Optional: Enable JupyterLab

If you want to use JupyterLab in a Domino workspace that uses a custom image, use the following instructions to install with `miniconda`. The commands to install Tensorboard are included, and the comments indicate which lines to remove if you do not need Tensorboard in your image.

```dockerfile theme={null}
# Install JupyterLab and Tensorboard using Conda
# A proper combination of these versions is important!
ARG CONDA_URL=https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
ARG PYTHON_VERSION=3.9
ARG JUPYTER_VERSION=2.3

# Remove the following line if you are not installing Tensorboard:
ARG TENSORBOARD_VERSION=2.2

# You must create a start script with configurations for JupyterLab
ARG JUPYTER_SCRIPT=jupyterlab-start.sh

# Install Conda and Python
# (Download signature not available)
RUN curl -o conda-install.sh -LSsf ${CONDA_URL} && \
  /bin/bash conda-install.sh -fbp ${CONDA_DIR} && \
  ${CONDA_DIR}/bin/conda update -y --update-all && \
  ${CONDA_DIR}/bin/conda install -y python=${PYTHON_VERSION}

# Install NodeJS if installing Tensorboard
RUN ${CONDA_DIR}/bin/conda install -y nodejs && \
  mkdir -p ${INSTALL_DIR}/node && \
  ln -rs ${CONDA_DIR}/bin/node ${INSTALL_DIR}/node/ && \
  ln -rs ${CONDA_DIR}/bin/npm ${INSTALL_DIR}/node/

# Install and configure Jupyterlab and extensions
RUN ${CONDA_DIR}/bin/conda install -y jupyterlab=${JUPYTER_VERSION} && \
  ${CONDA_DIR}/bin/conda install -y tensorboard=${TENSORBOARD_VERSION} && \
  ${CONDA_DIR}/bin/conda install -y -c conda-forge jupytext && \
  ${CONDA_DIR}/bin/conda install -y nodejs && \
  ${CONDA_DIR}/bin/pip install jupyter_tensorboard && \
  PATH=${CONDA_DIR}/bin ${CONDA_DIR}/bin/jupyter labextension install jupyterlab_tensorboard

# Create Juptyerlab workspace Environment scripts
RUN envsubst "${ALLENV}" < ${JUPYTER_SCRIPT} > ${INSTALL_BIN}/${JUPYTER_SCRIPT} && \
  chmod 755 ${INSTALL_BIN}/${JUPYTER_SCRIPT}
```

## Optional: Enable VS Code

To use VS Code in a Domino workspace, add the following instructions.

```dockerfile theme={null}
# Install VS Code

#Set Versions
ARG VSCODE_VERSION=3.10.2
ARG VSCODE_URL=https://github.com/cdr/code-server/releases/download/v${VSCODE_VERSION}/code-server-${VSCODE_VERSION}-linux-amd64.tar.gz
ARG VSCODE_PYTHON_VERSION=2021.5.926500501
ARG VSCODE_PYTHON_URL=https://github.com/microsoft/vscode-python/releases/download/${VSCODE_PYTHON_VERSION}/ms-python-release.vsix

# Where it will be deployed
ARG VSCODE_DIR=${INSTALL_DIR}/vscode
ARG VSCODE_EXT_DIR=${VSCODE_DIR}/extensions

# You must create a start script with configurations for VS Code
ARG VSCODE_SCRIPT=vscode-start.sh
RUN curl -LSsf ${VSCODE_URL} | tar -xz --no-same-permissions && \
  mv -f code-server-${VSCODE_VERSION}-linux-amd64 ${VSCODE_DIR} && \
  curl -o python.vsix -LSsf ${VSCODE_PYTHON_URL} && \
  mkdir -p ${VSCODE_EXT_DIR} && \
  ${VSCODE_DIR}/bin/code-server \
      --install-extension python.vsix \
      --extensions-dir ${VSCODE_EXT_DIR}

# Create VS Code workspace Environment scripts
RUN envsubst "${ALLENV}" < ${VSCODE_SCRIPT} > ${INSTALL_BIN}/${VSCODE_SCRIPT} && \
  chmod 755 ${INSTALL_BIN}/${VSCODE_SCRIPT}
```


# MATLAB
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/matlab

Configure a licensed MATLAB Environment and Workspace, add GPU and proxy settings, persist preferences, and troubleshoot launch or network issues.

To use an interactive MATLAB® session in Domino, you must configure a Compute Environment and interactive Workspace. If your administrators have already configured such an Environment, you can use it by setting your Project to use that Environment.

## Configure a MATLAB Environment in Domino

This topic describes how to create a Compute Environment if it does not already exist. You might need help from a Domino or MATLAB admin.

## Prerequisites

**Network licensing structure for MATLAB**\
You must have a network license to configure the Compute Environment. Domino recommends a network concurrent license. A network-named license might work if the usernames specified in the license matches the usernames in Domino. If you don’t have a network concurrent license, contact your MATLAB license administrator. You can have a single license server or a set of three license servers. The following are examples of how to provide the license server information:

### Method 1

This method requires only the port and host for your license servers. For example, `27000@lic1.customer.com` (and optionally `27000@lic2.customer.com`, `27000@lic3.customer.com`). This is recommended for most cases.

### Method 2

Use this method if you have a `network.lic` file. For example:

```shell theme={null}
SERVER lic1.customer.com 000000000000 27000 SERVER lic2.customer.com 000000000010 27000 SERVER lic3.customer.com 000000000020 27000 USE_SERVER
```

This is recommended only if you have several sets of license servers (and therefore, several license files). For example, you would have separate license servers for MATLAB Parallel Server.

See [Mathworks help](https://www.mathworks.com/matlabcentral/answers/116637-what-are-the-differences-between-the-license-lic-license-dat-network-lic-and-license_info-xml-lic) for more information about the options or talk to your MATLAB license administrator.

**(Optional) Hardware Tiers with the correct GPU drivers**\
To use GPU capabilities, the hardware tier in Domino must have GPU drivers installed that can support the version of MATLAB that you want to use. Use the following to determine compatibility:

* [GPU Computing Requirements](https://www.mathworks.com/help/parallel-computing/gpu-computing-requirements.html) which states the GPU compute capability range the current MATLAB release supports and links to the requirements for earlier releases.

* [CUDA compatibility](https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver) which describes the minimum NVIDIA driver version requirement for each version of the CUDA Toolkit.

The MATLAB base images provided by Domino already have a suitable version of the CUDA Toolkit installed. You must configure the NVIDIA drivers at the Hardware Tier level. If you already have a GPU-enabled hardware tier configured, it is likely that it already has suitable NVIDIA drivers. See the [Troubleshooting section](#faq-and-troubleshooting) for instructions about how to check the driver version during a MATLAB session to verify it is configured correctly.

If you do not have the correct NVIDIA drivers configured on your GPU Hardware Tier, contact your Domino administrator and refer them to the documentation to [configure NVIDIA drivers in EKS](/cloud/admin/platform-configuration/manage-compute-environments/cache-images-in-eks/ami-operations#install-nvidia-docker-2-0-gpu-amis-only). talk to your Domino administrator. They must install and/or upgrade the NVIDIA drivers, and (if your deployment uses Amazon Web Services) update the executor AMI following a procedure similar to the one for [Creating a new executor AMI](/cloud/admin/platform-configuration/manage-compute-environments/cache-images-in-eks). If you are a Domino administrator looking for guidance on this process, contact your Domino account team or email us at [support@dominodatalab.com](mailto:support@dominodatalab.com).

## Environment setup

If a MATLAB-specific Environment does not already exist in your Domino deployment, you must create one. Read about [Environment Management](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments) if you aren’t already familiar.

### Choose a base image

When creating a new Environment, choose a base image that has MATLAB installed. The following are the most current options available from Domino.

| Version | Repo Link |
| - | - |
| MATLAB 2021a | `quay.io/domino/matlab:R2021a` |
| MATLAB 2020b | `quay.io/domino/matlab:R2020b` |
| MATLAB 2020a | `quay.io/domino/matlab:R2020a` |
| MATLAB 2019b | `quay.io/domino/matlab:R2019b` |

### Edit the Dockerfile

1. Add the following lines to your **Dockerfile Instructions**, editing the license server information to match your actual license servers.

2. Edit the `MATLAB_VERSION` variable to match the MATLAB version of the base image you chose in the previous step.

3. If you have port and host information for your license servers (recommended method):

   ```bash theme={null}
   # This MATLAB_VERSION variable is used later in persisting Add-ons and preferences
   ENV MATLAB_VERSION='R2019b'
   ENV MLM_LICENSE_FILE=27000@lic1.customer.com,27000@lic2.customer.com,27000@lic3.customer.com
   ```

   Or, if you have multiple license files and you have a `network.lic` file you can adapt this example:

   ```bash theme={null}
   USER root

   ENV MATLAB_VERSION='R2019b'
   # Here we are simply pasting the information in the license file
   # If you have the license file hosted somewhere accessible to Domino, you could instead download it to the same location via wget or similar methods
   RUN mkdir -p /usr/local/MATLAB/$MATLAB_VERSION/licenses &&
       printf "SERVER lic1.customer.com 000000000000 27000
   SERVER lic2.customer.com 000000000010 27000
   SERVER lic3.customer.com 000000000020 27000
   USE_SERVER
   " > /usr/local/MATLAB/$MATLAB_VERSION/licenses/network.lic

   USER ubuntu
   ```

### Add pluggable Workspace tools definition

1. Add the "Pluggable Properties" to the **Properties for Workspaces**.

   These are documented alongside the base image descriptions here: [https://github.com/dominodatalab/Domino\_Base\_Images/tree/master/MATLAB\_Base\_Images](https://github.com/dominodatalab/Domino_Base_Images/tree/master/MATLAB_Base_Images).

   The MATLAB base images are built on top of the [Domino Standard Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments), so you can add the Workspace configurations such as Jupyter or RStudio for those standard Environments also.

For example, if you are using the `quay.io/domino/matlab:r2019b-20200521` base image, add the following lines for the MATLAB Workspace:

```yaml theme={null}
matlab-rdp:
  title: "RDP"
  iconUrl: "https://raw.githubusercontent.com/dominodatalab/partner-resources/main/assets/workspaces/Mathworks/MATLAB/matlab-simulink.png"
  start: [ "/opt/domino/workspaces/matlab/start" ]
  httpProxy:
    port: 8080
    internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/#/?username=domino&password=domino"
    rewrite: false
    requireSubdomain: false
matlab:
  title: "MATLAB"
  iconUrl: "https://raw.githubusercontent.com/dominodatalab/partner-resources/main/assets/workspaces/Mathworks/MATLAB/matlab.png"
  start: [ "/opt/domino/workspaces/matlab-webdesktop/start" ]
  httpProxy:
    port: 8888
    internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/matlab/index.html"
    rewrite: false
    requireSubdomain: false
```

Because it is built on top of `quay.io/domino/base:Ubuntu18_DAD_Py3.6_R3.6_20190916`, you can add any of the following sections for Workspaces from that Environment:

```yaml theme={null}
jupyter:
  title: "Jupyter (Python, R, Julia)"
  iconUrl: "/assets/images/workspace-logos/Jupyter.svg"
  start: [ "/opt/domino/workspaces/jupyter/start" ]
  supportedFileExtensions: [ ".ipynb" ]
  httpProxy:
    port: 8888
    rewrite: false
    internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
    requireSubdomain: false
jupyterlab:
  title: "JupyterLab"
  iconUrl: "/assets/images/workspace-logos/jupyterlab.svg"
  start: [  "/opt/domino/workspaces/jupyterlab/start" ]
  httpProxy:
    internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
    port: 8888
    rewrite: false
    requireSubdomain: false
vscode:
  title: "vscode"
  iconUrl: "/assets/images/workspace-logos/vscode.svg"
  start: [ "/opt/domino/workspaces/vscode/start" ]
  httpProxy:
    port: 8888
    requireSubdomain: false
rstudio:
  title: "RStudio"
  iconUrl: "/assets/images/workspace-logos/Rstudio.svg"
  start: [ "/opt/domino/workspaces/rstudio/start" ]
  httpProxy:
    port: 8888
    requireSubdomain: false
```

### (Optional) Persist MATLAB preferences and add-ons between sessions

To persist your MATLAB Preferences and Add-ons between Domino sessions, you can define a **Post Run Script** to automatically copy them to your Project Files when you **Stop** and **Sync** a session, and define a **Pre Run Script** to copy them from your Project Files into the locations MATLAB recognizes at the start of subsequent sessions.

Put the following lines into your **Pre Run Script**.

```bash theme={null}
if [ -d /mnt/matlab-addons ]; then
    mkdir -p /home/ubuntu/Documents/MATLAB/SupportPackages/$MATLAB_VERSION/
    rsync -r /mnt/matlab-addons/ /home/ubuntu/Documents/MATLAB/SupportPackages/$MATLAB_VERSION
fi
if [ -d /mnt/matlab-preferences/$DOMINO_STARTING_USERNAME ]; then
    mkdir -p /home/ubuntu/.matlab/$MATLAB_VERSION/
    rsync -r /mnt/matlab-preferences/$DOMINO_STARTING_USERNAME/ /home/ubuntu/.matlab/$MATLAB_VERSION
fi
```

Put the following lines into your **Post Run Script**.

```bash theme={null}
if [ -d /home/ubuntu/Documents/MATLAB/SupportPackages/$MATLAB_VERSION ]; then
    mkdir -p /mnt/matlab-addons
    rsync -r /home/ubuntu/Documents/MATLAB/SupportPackages/$MATLAB_VERSION/ /mnt/matlab-addons
fi
if [ -d /home/ubuntu/.matlab/$MATLAB_VERSION ]; then
    mkdir -p /mnt/matlab-preferences/$DOMINO_STARTING_USERNAME
    rsync -r /home/ubuntu/.matlab/$MATLAB_VERSION/ /mnt/matlab-preferences/$DOMINO_STARTING_USERNAME
fi
```

These scripts save add-ons **per project**, so that any collaborators launching a Workspace in the same Project have the same add-ons. For preferences, however, they are saved **per user** in the Project so that each user can have a different set of preferences.

## (Optional) Generate a startup script for proxy settings

If your company uses a proxy to connect to external networks, you might already have lines like the following in many of your Compute Environment Dockerfiles:

```bash theme={null}
ENV http_proxy='http://someusername:password123@proxy.company.com:80'
ENV https_proxy='http://someusername:password123@proxy.company.com:80'
ENV HTTP_PROXY='http://someusername:password123@proxy.company.com:80'
ENV HTTPS_PROXY='http://someusername:password123@proxy.company.com:80'
```

Without setting the proxy information in MATLAB as well, you may not be able to connect to external networks from MATLAB (for example, for downloading community Add-ons). Proxy information can be set manually in MATLAB under **Preferences** > **Web**, and these are persisted across sessions if the **Pre** and **Post Run Scripts** from the previous section are included in your Environment.

Alternatively, you can define a `startup.m` file in your Project to ensure the proxy settings are correct at every launch of MATLAB. The file would look something like the following.

```matlab theme={null}
% This startup file was created from the Environment pre-run script, to set proxy settings for MATLAB
disp('Setting proxy configuration via startup.m file')
com.mathworks.mlwidgets.html.HTMLPrefs.setUseProxy(true)
com.mathworks.mlwidgets.html.HTMLPrefs.setProxyHost('proxy.company.com')
com.mathworks.mlwidgets.html.HTMLPrefs.setProxyPort('80')
com.mathworks.mlwidgets.html.HTMLPrefs.setUseProxyAuthentication(true)
com.mathworks.mlwidgets.html.HTMLPrefs.setProxyUsername('someusername')
com.mathworks.mlwidgets.html.HTMLPrefs.setProxyPassword('password123')
```

Put the following lines into your **Pre Run Script** to automatically generate such a file in any Project where it does not exist, whenever this Compute Environment is used to launch a session. (Omit the Authentication, Username, and Password lines if your proxy does not use them.)

```bash theme={null}
if [ ! -f /mnt/startup.m ]; then
    echo "% This startup file was created from the Environment pre-run script, to set proxy settings for MATLAB" > /mnt/startup.m
    echo "disp('Setting proxy configuration via startup.m file')" >> /mnt/startup.m
    echo "com.mathworks.mlwidgets.html.HTMLPrefs.setUseProxy(true)" >> /mnt/startup.m
    echo "com.mathworks.mlwidgets.html.HTMLPrefs.setProxyHost('proxy.company.com')" >> /mnt/startup.m
    echo "com.mathworks.mlwidgets.html.HTMLPrefs.setProxyPort('80')" >> /mnt/startup.m
    echo "com.mathworks.mlwidgets.html.HTMLPrefs.setUseProxyAuthentication(true)" >> /mnt/startup.m
    echo "com.mathworks.mlwidgets.html.HTMLPrefs.setProxyUsername('someusername')" >> /mnt/startup.m
    echo "com.mathworks.mlwidgets.html.HTMLPrefs.setProxyPassword('password123')" >> /mnt/startup.m
fi
```

### Build and test the Environment

Click **Build** when you have finished editing the above sections. The Revisions page for the Environment opens.

If the new revision builds successfully, you are ready to test. Go to the Project Settings for a Project where you want to use MATLAB and change the Compute Environment to use this new Environment. After you do, you see a MATLAB icon as an option for Workspaces in that Project.

See the [Troubleshooting](#faq-and-troubleshooting) section for information about known issues and frequently encountered problems if anything does not work as expected.

## FAQ and troubleshooting

### GPU functionality does not work

The most common problem with GPU functionality is not having the correct NVIDIA driver version for your Environment configured in your Hardware Tier. The following shell command should print some information about the GPU driver versions so you can verify it with the compatibility tables referenced in the `Prerequisites` section.

```matlab theme={null}
>> !nvidia-smi
Wed May 13 00:00:16 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  Off  | 00000000:00:1B.0 Off |                    0 |
| N/A   39C    P0    50W / 300W |      0MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  Off  | 00000000:00:1C.0 Off |                    0 |
| N/A   41C    P0    50W / 300W |      0MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  Off  | 00000000:00:1D.0 Off |                    0 |
| N/A   40C    P0    52W / 300W |      0MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   40C    P0    54W / 300W |      0MiB / 16160MiB |      4%      Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
```

### MATLAB Workspace fails to launch

#### License configuration problems

License issues are the most common cause of problems launching a MATLAB Workspace. To debug, start a Jupyter or RStudio Workspace (still using the MATLAB Environment) and open a terminal. Check if your license server information has been correctly entered into the `network.lic` file, in the correct part of the MATLAB directory structure (that is, matching your MATLAB version):

```bash theme={null}
$ which matlab
/usr/local/MATLAB/R2019b/bin/matlab
$ cat /usr/local/MATLAB/R2019b/licenses/network.lic
```

If that contains the license server information you expect, but there are still problems, you can use the `lmutil` utility to check that the license server is reachable and contains the correct information.

```bash theme={null}
$ lmutil lmstat -a -c 27000@lic1.customer.com
```

It should output information on the licenses being managed by that license server, including the MATLAB licenses.

### Advanced troubleshooting

The Workspace launch process for MATLAB respects several Environment variables related to logging, which may help you debug more obscure or advanced problems. You can set these as Project-level Environment variables, similar to how you would [set Environment variables for secure credential storage](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables), to enable toggling them on or off without having to edit the Compute Environment.

* `WS_LOGGING=[MATLAB]|ALL|DEBUG`. By default, only the MATLAB logs are shown.

  * `ALL` shows logs for the underlying screen management **xterm** and **xpra**.

  * `DEBUG` shows everything from `ALL`, and activates a flag to show all commands.

* `TTY_MODE=[disabled]|PRE|POST`. Runs **ttyd** before or after the Workspace. It’s a terminal that runs in a browser so you can easily inspect things.

  * `PRE` runs **ttyd** before the Workspace launch.

  * `POST` runs **ttyd** after the Workspace launch fails (if it fails).

* `START_LOGGING=[false]|true`. Increases logging verbosity in certain stages.

### MATLAB Workspace is laggy

In poor network conditions you may notice some lag in typing and general interactions with the MATLAB UI. This is to be expected in certain circumstances. Other common Domino Workspaces like Jupyter or RStudio use browser-based IDEs, while this MATLAB Environment is running a (Linux) desktop version of MATLAB in a remote desktop style interface. This results in more sensitivity to slow network conditions than a browser-based IDE.

For these situations, if you encounter them frequently, you may find it useful to install the [Domino CLI](/cloud/reference/cli/install-the-cli) on your local machine to facilitate editing files locally, then syncing them to Domino.

### Keyboard shortcuts for copy/paste do not work

MATLAB in Domino runs the Linux version of MATLAB, which uses the "Emacs" set of keyboard shortcuts by default. To enable keyboard shortcuts `Ctrl+C` and `Ctrl+V` for copy and paste, go to **Preferences** > **Keyboard** > **Shortcuts** and change this from **Emacs** to **Windows**. See the [MATLAB help](https://www.mathworks.com/matlabcentral/answers/103161-why-am-i-unable-to-use-copy-ctrl-c-and-paste-ctrl-v-keyboard-short-cuts-in-the-text-editing-mode) for more details.

### Connection error when downloading add-ons or using urlread

If you get network errors when downloading Add-ons or using `urlread` to download files, it may be due to proxy settings. See the `Proxy` section above for options for setting your proxy in MATLAB either within the current session or more globally.

### Toolbox is not available

A selection of Toolboxes is pre-installed in each MATLAB base image, but this might not include every Toolbox you have on your local installation. You can see the list of Toolboxes by running the `ver` command within MATLAB, and the list is also documented for each [base image](#choose-a-base-image). You can contact your Domino account team with any specific requests for Toolboxes to include.

In addition to having the Toolbox installed in the base image, you must have an available license for using it on your [license server](#prerequisites).

## Known issues and release notes

If your problem is not listed in this FAQ, it may be worth checking the release notes for the [base image](#choose-a-base-image) you are using. Especially if your Environment is built on an older image tag, checking the release notes of newer images may give you a quick answer about whether rebuilding on the most current image helps.

*MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See [https://www.mathworks.com/company/trust-center/trademarks.html](https://www.mathworks.com/company/trust-center/trademarks.html) for a list of additional trademarks.*


# NVIDIA NGC
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/nvidia-ngc

Add NVIDIA NGC containers for RAPIDS, PyTorch, TensorFlow, Clara Train, or MXNet as GPU-enabled Compute Environments with Jupyter tools.

## NVIDIA GPU Cloud (NGC) offers pre-built containers

They have both general-purpose and domain-specific offerings for machine learning and deep learning workloads on NVIDIA GPUs.

Domino enhances NGC containers for use in Domino and for general data science work.

**Add Domino compatibility**\
Domino automatically manages code and data versioning as part of the container lifecycle and as a result we require specific additional software in the container.

**Add Data Source drivers**\
Domino adds drivers for Data Sources such as Snowflake, Oracle, and Microsoft SQL to make it easy to connect to a Data Source without needing to manually add drivers.

**Add Workspaces**\
When an NGC container doesn’t include an interactive notebook, Domino adds Jupyter to the image so you can interactively develop in the image. When there’s a notebook already included, Domino configures the existing notebook to work.

Domino builds the latest versions of NGC containers with a preference for Ubuntu base images. If there’s an Environment that you’re looking for which is not available, request it from [ngc-request@dominodatalab.com](mailto:ngc-request@dominodatalab.com).

## Add an NGC image to Domino

1. Choose the NGC container from the list of [available containers](#available-containers).

2. Create a new Compute Environment in Domino.

3. Use the link from the container you selected as the base compute Environment (for example, `quay.io/domino/ngc-pytorch:20.12-py3`)

4. Enter the following in the **Workspace Definition** area:

```yaml theme={null}
jupyter:
 title: "Jupyter (Python, R, Julia)"
 iconUrl: "/assets/images/workspace-logos/Jupyter.svg"
 start: [ "/opt/domino/workspaces/jupyter/start" ]
 httpProxy:
   port: 8888
   rewrite: false
   internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{   requireSubdomain: false
jupyterlab:
 title: "JupyterLab"
 iconUrl: "/assets/images/workspace-logos/jupyterlab.svg"
 start: [  "/opt/domino/workspaces/jupyterlab/start" ]
 httpProxy:
   internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{   port: 8888
   rewrite: false
   requireSubdomain: false
```

## Available containers

### RAPIDS

| | [https://ngc.nvidia.com/catalog/containers/nvidia:rapidsai:rapidsai](https://ngc.nvidia.com/catalog/containers/nvidia:rapidsai:rapidsai) |
| - | - |
| Domino registry path: | `quay.io/domino/ngc-rapids:0.18-cuda11.0-runtime-ubuntu20.04-py3.8` |
| Versions: | RAPIDS 0.18, Ubuntu 18.04, CUDA 11 |
| Base Image: | `docker pull nvcr.io/nvidia/rapidsai/rapidsai:0.18-cuda11.0-base-ubuntu18.04` |
| Notes: | N/A |

### PyTorch

| | [https://ngc.nvidia.com/catalog/containers/nvidia:pytorch](https://ngc.nvidia.com/catalog/containers/nvidia:pytorch) |
| - | - |
| Domino registry path: | `quay.io/domino/ngc-pytorch:20.12-py3` |
| Versions: | Ubuntu 18.04, CUDA 11 |
| Base Image: | `docker pull nvcr.io/nvidia/pytorch:21.02-py3` |
| Notes: | N/A |

### TensorFlow

| | [https://ngc.nvidia.com/catalog/containers/nvidia:tensorflow](https://ngc.nvidia.com/catalog/containers/nvidia:tensorflow) |
| - | - |
| Domino registry path: | `quay.io/domino/ngc-tensorflow:20.12-tf1-py3` |
| Versions: | TensorFlow 21.03, Ubuntu 18.04, CUDA 11 |
| Base Image: | `docker pull nvcr.io/nvidia/tensorflow:21.03-tf1-py3` |
| Notes: | N/A |

### Clara Train

| | [https://ngc.nvidia.com/catalog/containers/nvidia:clara-train-sdk](https://ngc.nvidia.com/catalog/containers/nvidia:clara-train-sdk) |
| - | - |
| Domino registry path: | `quay.io/domino/ngc-clara-train:v3.1.01` |
| Versions: | Clara 3.1.01, Ubuntu 18.04, CUDA 11 |
| Base Image: | `docker pull nvcr.io/nvidia/clara-train-sdk:v3.1.01` |
| Notes: | N/A |

### MXNet

| | [https://ngc.nvidia.com/catalog/containers/nvidia:mxnet](https://ngc.nvidia.com/catalog/containers/nvidia:mxnet) |
| - | - |
| Domino registry path: | `quay.io/domino/ngc-mxnet:20.12-py3` |
| Versions: | MXNet 21.03, Ubuntu 18.04, CUDA 11 |
| Base Image: | `docker pull nvcr.io/nvidia/mxnet:21.03-py3` |
| Notes: | N/A |


# Posit Workbench
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/posit-workbench

Build a Domino Compute Environment from the Posit Workbench base image and launch Workspace sessions in the Positron IDE.

To use Posit Workbench, including the Positron IDE, in a Domino Workspace, configure a Compute Environment built on a Posit Workbench base image. If your administrators have already configured such an Environment, set your Project to use that Environment.

## Environment setup

If a Posit Workbench Environment does not already exist in your Domino deployment, create one. Read about [Environment management](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments) if you aren't already familiar.

### Choose a base image

When you create the Environment, use `quay.io/domino/field:rstudio-workbench` as the base image.

### Edit the Dockerfile

Add the following lines to the **Dockerfile Instructions**. They validate that the base image meets the [pre-requisites for automatic custom image compatibility](/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/automatic-custom-image-compatibility), prepare `/opt/domino` for Domino executions, set the password for the `domino` user, create a startup script that waits for Posit Workbench and tails its logs, and configure Posit Workbench to run sessions as the `domino` user.

```dockerfile theme={null}
# System-level dependency injection runs as root
USER root:root

# Validate base image pre-requisites
RUN /opt/domino/bin/pre-check.sh

# Configure /opt/domino to prepare for Domino executions
RUN /opt/domino/bin/init.sh

# Validate the environment
RUN /opt/domino/bin/validate.sh

# Set the password for the domino user
RUN echo 'domino:domino' | chpasswd

# Create a startup script that waits for Posit Workbench and tails its logs
COPY <<'EOF' /opt/domino/bin/posit-monitor.sh
#!/bin/bash
# /opt/domino/bin/posit-monitor.sh

# Wait for Posit Workbench to start
while ! pgrep -f "rserver|rstudio-server" > /dev/null; do
    echo "Waiting for Posit Workbench to start..."
    sleep 2
done

echo "Posit Workbench started. Tailing logs..."

# Tail the Posit Workbench logs (adjust the path as needed)
tail -f /var/log/rstudio/rstudio-server/*.log 2>/dev/null || \
tail -f /var/lib/rstudio-server/monitor/log/* 2>/dev/null || \
    # Fallback: just monitor the process
    while pgrep -f "rserver|rstudio-server" > /dev/null; do sleep 5; done
EOF
RUN chmod +x /opt/domino/bin/posit-monitor.sh

# Configure RStudio Server to run sessions as the domino user
RUN echo "server-user=domino" >> /etc/rstudio/rserver.conf

# Modify the existing launcher.conf to use domino as the admin group
RUN sed -i 's/^admin-group=.*/admin-group=domino/' /etc/rstudio/launcher.conf || \
    echo "admin-group=domino" >> /etc/rstudio/launcher.conf
```

### Add the pluggable Workspace tool

Enter the following in the **Pluggable Workspace Tools** field of the Environment definition:

```yaml theme={null}
posit_workbench:
  title: "Posit"
  iconUrl: "https://upload.wikimedia.org/wikipedia/commons/thumb/7/7a/Posit_logo.svg/960px-Posit_logo.svg.png"
  start: [ "/opt/domino/bin/posit-monitor.sh" ]
  httpProxy:
    port: 8787
    requireSubdomain: false
```

## Use Positron in a Workspace

<Steps>
  <Step title="Launch a Workspace">
    [Launch a Workspace](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace) that uses the Environment and select **Posit** as the IDE. Log in with the username `domino` and the password `domino`. The Dockerfile instructions above set this password. Posit Workbench opens to the **Projects** view.

    <Frame>
      <img alt="Posit Workbench Projects view with the New Session button." />
    </Frame>
  </Step>

  <Step title="Create a session">
    Select **New Session**, choose **Positron Pro**, and select **Launch**.

    <Frame>
      <img alt="New Session dialog in Posit Workbench with Positron Pro selected." />
    </Frame>
  </Step>

  <Step title="Start working">
    When the session starts, the Positron IDE opens.

    <Frame>
      <img alt="Positron IDE running in Posit Workbench with an R console started." />
    </Frame>
  </Step>
</Steps>


# Stata
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/stata

Create a licensed Stata Workspace Environment from the quay.io image, apply MP, SE, or BE credentials, and define its pluggable tool settings.

Stata® statistical software provides all the features you need for data science and inference–data manipulation, exploration, visualization, statistics, reporting, and reproducibility.

Stata has the following versions:

* Basic Edition (BE)

* Standard Edition (SE)

* Multiprocessor Edition (MP)

<Note>
  To learn more about Stata, see [Stata features](https://www.stata.com/features/), [Stata benefits](https://www.stata.com/why-use-stata/), and [compare versions of Stata](https://www.stata.com/products/which-stata-is-right-for-me/).
</Note>

## Get data from license file

To use a Stata workspace in Domino you must have a valid Stata license file. The license from Stata comes in a PDF file. You need the following data from this file:

* Licensed software version (either `MP`, `SE`, or `BE`)

* Serial number (12-digit number)

* Code (series of nine 4-character codes)

* Authorization (4-character code)

## Create a license file

1. Open the Stata Environment in Domino.

2. Right-click the background and go to **applications > shells > bash**.

3. Go to the Stata program directory:

   ```shell theme={null}
   cd /usr/local/stata
   ```

4. As superuser, run the Stata init program:

   ```shell theme={null}
   sudo ./stinit
   ```

5. Follow the prompts to enter the serial number, code, and authorization.

6. Follow the additional prompts to enter the organization name and location.

   <Note>
     These values are not validated and can be your company name and headquarters location.
   </Note>

Completing this process generates the Stata license file `stata.lic` in the `stata` directory. Copy the contents of this file to a safe location.

## Create environment

1. When you create a Domino workspace Environment for Stata, use the following custom base image:

   ```
   quay.io/domino/stata:17
   ```

2. Update the Dockerfile instructions with the following Docker commands.

   ```dockerfile theme={null}
   ENV STATA_VERSION="MP"
   RUN apply-license “<url_to_license_file>”
   ```

   Ensure the `STATA_VERSION` matches the customers' licensed version. The version must be `MP`, `SE`, or `BE`.

3. Paste the following code into the Environments pluggable workspace properties.

   ```yaml theme={null}
   stata:
       title: "Stata"
       iconUrl: "https://raw.githubusercontent.com/dominodatalab/partner-resources/main/assets/workspaces/stata/statamp.png"
       start: [ "/opt/domino/workspaces/stata/start" ]
       httpProxy:
           port: 8080
           internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/#/?username=domino&password=domino"
           rewrite: false
           requireSubdomain: false
   ```

   See the following for more resources:

   * [Stata Support Center](https://www.stata.com/support/)

   * [Stata Documentation](https://www.stata.com/features/documentation/)

   * [Stata Training](https://www.stata.com/learn/)

   * [Stata Examples and Datasets](https://www.stata.com/links/examples-and-datasets/)

   * [PyStata—Python and Stata](https://www.stata.com/python/pystata18/)


# Add a Scala kernel
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/add-a-scala-kernel

Install Scala dependencies and the Almond kernel through Environment Dockerfile instructions, then run Scala notebooks and console commands in JupyterLab.

By default the [Domino Standard Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) includes kernels for Python and Julia. However, you can add kernels to your Environment in just a few steps. In this topic, you learn how to add a Scala kernel to your Environment.

## Install dependencies

```dockerfile theme={null}
USER root

# Install scala (requires java to already be installed)
RUN echo "deb https://repo.scala-sbt.org/scalasbt/debian all main" | sudo tee /etc/apt/sources.list.d/sbt.list &&
curl -sL "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823" | sudo apt-key add &&
apt-get update && apt-get install sbt &&
rm -rf /var/lib/apt/lists/* &&
rm -Rf /tmp/*

https://xxx[USER] ubuntu

# Install scala kernel (almond)
RUN sudo curl -Lo coursier https://git.io/coursier-cli &&
    sudo chmod +x coursier &&
    ./coursier launch --fork almond:0.10.9 --scala 2.12 -- --install &&
    sudo rm -f coursier
```

After adding these commands to your Environment Dockerfile you can start Scala notebooks and run Scala commands directly from the JupyterLab console.

You can add kernels to Domino Environments like you added the scala kernel. See [https://almond.sh/docs/quick-start-install](https://almond.sh/docs/quick-start-install) for more information about how to install the Scala kernel.


# Add Workspace IDEs
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/add-workspace-ides

Add Jupyter, JupyterLab, VS Code, or RStudio to a Compute Environment by installing each IDE and defining its Pluggable Workspace Tools YAML.

Domino is interoperable with your favorite languages, development tools and software. Learn how to bring your favorite IDEs to Domino.

When you add a Workspace IDE to your Compute Environment, you can:

* Upgrade to a newer version of currently supported Domino tools such as Jupyter or RStudio.

* Add web-based tools like JupyterLab.

* Manage the standard default tool for your team or organization across all projects.

## Configure an IDE

You must set up the Environment’s Docker image installation instructions and define how Domino serves the tool.

<Note>
  Typically, an administrator or advanced user does this work. Domino recommends that you contact your administrator for help defining the tool you want to use.
</Note>

1. In your Environment, enter the instructions to install and configure a tool in the Dockerfile instructions:

### For Python version > 2.7.9

```Dockerfile theme={null}
###Remove any old workspaces
RUN
apt-get remove rstudio-server -y &&
 rm -rf /usr/local/lib/rstudio-server/rstudio-server &&
 rm -rf /var/opt/workspaces
###Setup workspaces directory and retrieve workspace configs
RUN mkdir /var/opt/workspaces
RUN cd /tmp && wget https://github.com/dominodatalab/workspace-configs/archive/2018q2-v1.9.zip && unzip 2018q2-v1.9.zip && cp -Rf workspace-configs-2018q2-v1.9/. /var/opt/workspaces &&
rm -rf /var/opt/workspaces/workspace-logos && rm -rf /tmp/workspace-configs-2018q2-v1.9
#add update .Rprofile with Domino customizations
RUN
mv /var/opt/workspaces/rstudio/.Rprofile /home/ubuntu/.Rprofile &&
chown ubuntu:ubuntu /home/ubuntu/.Rprofile
# # # #Install Rstudio from workspaces
RUN chmod +x /var/opt/workspaces/rstudio/install
RUN /var/opt/workspaces/rstudio/install
# # # # # #Install Jupyterlab from workspaces
RUN chmod +x /var/opt/workspaces/Jupyterlab/install
RUN /var/opt/workspaces/Jupyterlab/install
# # #Install Jupyter from workspaces
RUN chmod +x /var/opt/workspaces/jupyter/install
RUN /var/opt/workspaces/jupyter/install
# Clean up temporary files
RUN
 rm -Rf /var/lib/apt/lists/* &&
 rm -Rf /tmp/*
```

### For Python version \< 2.7.9

```dockerfile theme={null}
###Remove any old workspaces
RUN
apt-get remove rstudio-server -y &&
 rm -rf /usr/local/lib/rstudio-server/rstudio-server &&
 rm -rf /var/opt/workspaces

###Setup workspaces directory and retrieve workspace configs
RUN mkdir /var/opt/workspaces
RUN cd /tmp && wget https://github.com/dominodatalab/workspace-configs/archive/2018q2-v1.9.zip && unzip 2018q2-v1.9.zip && cp -Rf workspace-configs-2018q2-v1.9/. /var/opt/workspaces &&
rm -rf /var/opt/workspaces/workspace-logos && rm -rf /tmp/workspace-configs-2018q2-v1.9

#add update .Rprofile with Domino customizations
RUN
mv /var/opt/workspaces/rstudio/.Rprofile /home/ubuntu/.Rprofile &&
chown ubuntu:ubuntu /home/ubuntu/.Rprofile

# # # #Install Rstudio from workspaces
RUN chmod +x /var/opt/workspaces/rstudio/install
RUN /var/opt/workspaces/rstudio/install

# # # # # #Install Jupyterlab from workspaces (pinned to avoid working directory bug in Jupyterlab)
RUN pip install jupyterlab==0.31.12

# # #Install Jupyter from workspaces
RUN chmod +x /var/opt/workspaces/jupyter/install
RUN /var/opt/workspaces/jupyter/install
# Clean up temporary files
RUN
 rm -Rf /var/lib/apt/lists/* &&
 rm -Rf /tmp/*
```

1. Notebook properties are stored as `yaml` data mapping notebook names to their definitions. Enter this in the **Pluggable Workspace Tools** field in the Environment definition.

   For example:

```yaml theme={null}
jupyter:
  title: "Jupyter (Python, R, Julia)"
  iconUrl: "/assets/images/workspace-logos/Jupyter.svg"
  start: [ "/var/opt/workspaces/jupyter/start" ]
  supportedFileExtensions: [ ".ipynb" ]
  httpProxy:
    port: 8888
    rewrite: false
    internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
    requireSubdomain: false
jupyterlab:
  title: "JupyterLab"
  iconUrl: "/assets/images/workspace-logos/jupyterlab.svg"
  start: [  "/var/opt/workspaces/Jupyterlab/start.sh" ]
  httpProxy:
    internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
    port: 8888
    rewrite: false
    requireSubdomain: false
vscode:
  title: "vscode"
  iconUrl: "/assets/images/workspace-logos/vscode.svg"
  start: [ "/var/opt/workspaces/vscode/start" ]
  httpProxy:
    port: 8888
    requireSubdomain: false
rstudio:
  title: "RStudio"
  iconUrl: "/assets/images/workspace-logos/Rstudio.svg"
  start: [ "/var/opt/workspaces/rstudio/start" ]
  httpProxy:
    port: 8888
    requireSubdomain: false
```


# Configure native Workspaces
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/configure-native-workspaces

Replace default Workspace tools with native JupyterLab, VS Code, or other image tools by adding startup scripts and updating each start entrypoint.

<Warning>
  This is an advanced topic. Use this only if you want to change the way the default tools are run, or if you want to use Workspace Tools that were not configured by default.
</Warning>

To replace the default tools in a Domino Environment (such as JupyterLab or VS Code) with the tools in your image, you can add a startup script for each tool to your Environment. To do this, you can either:

* Add the script to the base image.

* Add the script to the Dockerfile instructions.

<Note>
  Domino recommends using JupyterLab instead of Jupyter. It is a newer offering with superior features and performance.
</Note>

See the following instructions for these tools:

## VS Code

See [Using Visual Studio Code in Domino Workspaces](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace/launch-a-vs-code-workspace).

## JupyterLab

The following example script shows how the JupyterLab script must configure access to files in the Environment, set up the working directory, and define the necessary routing.

```shell theme={null}
BASE_URL="/$DOMINO_PROJECT_OWNER/$DOMINO_PROJECT_NAME/notebookSession/$DOMINO_RUN_ID"
CONF_FILE="$HOME/.jupyter/jupyter_lab_config.py"
mkdir -p $(dirname "$CONF_FILE")

cat > "$CONF_FILE" << EOF
c = get_config()

# Lab has access to all files in the Environment
c.NotebookApp.notebook_dir='/'

# Lab starts in the domino working dir
c.NotebookApp.default_url='/lab/tree${DOMINO_WORKING_DIR}'

# Routing, networking and access
c.NotebookApp.base_url='${BASE_URL}/'
c.NotebookApp.tornado_settings={'headers': {'Content-Security-Policy': 'frame-ancestors *'}, 'static_url_prefix': '${BASE_URL}/static/'}
c.NotebookApp.token=u''
c.NotebookApp.iopub_data_rate_limit=10000000000

# The default cell execution timeout in nbconvert is 30 seconds, set it to a year
c.ExecutePreprocessor.timeout=365*24*60*60

EOF
```

## Other tools

See [Domino pluggable notebooks](/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/add-workspace-ides).

After adding the script to the Environment, modify the **Pluggable Workspace Tools** section of the Environment definition to define the entrypoint for Workspace tool startup:

1. Go to the Environment you want to edit.

2. Click **Edit Definition**.

3. Go to **Pluggable Workspace Tools**.

4. Modify the entry so that the **start** field points to your tool’s launch script.


# Work with Pluggable Workspaces
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/index

Configure Pluggable Workspaces for custom IDEs, native tools, Scala kernels, TensorBoard, and SAS through Domino Compute Environments.

Domino is interoperable with your favorite languages, development tools and software. Learn how to bring your own tools to Domino.

[Add Workspace IDEs](/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/add-workspace-ides)\
Steps to configure an IDE for your Workspace.

[Configure native Workspaces](/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/configure-native-workspaces)\
Change the way the default tools are run, or use tools not configured by default.

[Add a Scala kernel](/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/add-a-scala-kernel)\
Create Scala notebooks.

[Use TensorBoard in Jupyter Workspace](/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/use-tensorboard-in-jupyter-workspaces)\
Install and enable the Jupyter-TensorBoard server extension in your Compute Environment.

[Use SAS as a Workspace](/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/sas-workspace)\
Use SAS for Containers on Domino.


# SAS on Domino
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/sas-workspace

Build licensed SAS Analytics Pro, SAS Studio, or SAS Data Science Programming images, configure Workspace start scripts, and persist preferences.

Learn how to bring your SAS projects to Domino to unify your data science teams and provide all the benefits of the Domino platform to SAS users. Domino supports [SAS Analytics Pro, SAS Analytics Pro - Advanced Programming](https://www.sas.com/en_us/software/analytics-pro/features-list.html), [SAS Analytics for Containers](https://documentation.sas.com/doc/en/containers/9.4/n133nr0ok71e5pn1oy96124cg1iz.htm), and SAS Data Science programming-only, with support for Viya and SAS platforms.

To use a SAS Workspace, your administrator must configure the Compute Environment and Workspace for you. Once configured, you can switch your Project to the SAS-enabled Environment. With containerization, multiple SAS users can work on the same SAS code using separate Workspaces.

Contact your Domino representative for more information about configuring SAS Workspaces on Domino.

## SAS Analytics Pro

This section provides an overview of the setup process for SAS Analytics Pro on Domino (including Advanced Programming). Contact your Domino representative to learn more.

1. When you receive your license from SAS, you’ll receive a credential token to access the SAS container repository. Use the token to pull the base image for SAS Analytics Pro onto your local machine.

2. Use a Dockerfile, provided by your Domino representative, to build a Domino-compatible image.

3. Include your SAS license file, `license.jwt`, in the same location as the Dockerfile when building the Docker image.

   The license file is copied into the container and the SAS license utility runs to license the software.

4. Push your newly created Docker image to a repository that your Domino deployment can access.

   This may be a repository hosted by your organization or a private repository within the Domino cluster.

5. Once you have created the Docker image, create a new Domino Environment using the custom image path.

6. Add the pluggable properties provided by your Domino representative, and make sure the “start:” parameter points to the start script created in the Dockerfile `/opt/domino/workspaces/sas/start`.

7. After building the Environment in Domino, you can select it in the Workspace launch menu when starting a new Workspace in your Project.

## SAS Analytics for containers on Domino

Learn how to build SAS as a container. The following instructions are similar to the official [SAS instructions for containerization](https://documentation.sas.com/doc/en/containers/9.4/n08crk346hlahon17mj16pmdj8t5.htm), but with a few key differences specific to Domino. To learn about the benefits of running SAS as a container, see [SAS: Build and run a container](https://documentation.sas.com/doc/en/containers/9.4/n133nr0ok71e5pn1oy96124cg1iz.htm).

In this guide, you learn how to:

1. Get your SAS license.

2. Create a TAR file of SAS Studio.

3. Create a Docker image that includes:

   1. SAS Studio TAR file.

   2. Dockerfile with instructions to build the Docker image.

   3. Start script provided by Domino.

4. Use the Docker image to create pluggable Workspaces in Domino.

5. Optional: Persist SAS preferences and add-ons.

## Get your SAS license

To use a SAS Workspace in Domino, you must have a valid SAS license file.

1. Go to `my.sas.com` to download the license file.

2. Save the license file securely, you use the license file to build the SAS Docker image later.

## Create the SAS Studio TAR file

Bundle SAS Studio in a TAR file to run in your Docker container.

1. Run the SAS Deployment Wizard to install SAS Studio on a Linux machine with the same version and distribution as the Domino Compute Environment that you want to run the SAS container on.

2. During the installation, change the default location for the SAS Studio installation to `/usr/local/SASHome`.

3. Manually create a TAR file of SAS Studio that includes the `SASHome` directory.

   ```shell theme={null}
   tar — cvf SASHomeTar.tar /usr/local/SASHome
   ```

## Create the Dockerfile instructions

Ensure that your Dockerfile includes a command to download and apply the valid SAS license file.

The license file must be hosted on a server accessible through an HTTP request from the Domino cluster. When building the Environment image, this file must be available to the build host without authentication restrictions.

Enter the following command in your **Dockerfile Instructions**:

```dockerfile theme={null}
RUN apply-license <url_to_license_file>
```

Base64 example

```dockerfile theme={null}
ENV SAS_LICENSE=”UgbWFnbmEgYWx...pcXVhLgo=”

RUN apply-license $SAS_LICENSE
```

Project file example

In your **Pre Run Script**:

```bash theme={null}
apply-license /mnt/my-sas-license.jwt
```

## Create a start script

Create a start script in your Docker image to start SAS Studio.

The following start script is a basic example, reach out to your Domino representative for guidance on creating a start script to suit your specific needs.

```bash theme={null}
#!/usr/bin/env bash
set -o errexit -o pipefail

# Update SAS Settings to start in Domino Project directory
sed -i "s#webdms.customPathRoot=#webdms.customPathRoot=$DOMINO_WORKING_DIR#g"
    /usr/local/SASHome/SASStudioBasic/3.8/war/config/config.properties

# Set the prefix path to the SAS web service
PREFIX_PATH="${DOMINO_PROJECT_OWNER}/${DOMINO_PROJECT_NAME}/notebookSession/${DOMINO_RUN_ID}"
PREFIX_FILE="${DOMINO_PROJECT_OWNER}#${DOMINO_PROJECT_NAME}#notebookSession#${DOMINO_RUN_ID}"
sed -i "s#path="/SASStudio"#path="/${PREFIX_PATH}"#g" /usr/local/SASHome/sas/appserver/studio/conf/Catalina/localhost/SASStudio.xml
mv /usr/local/SASHome/sas/appserver/studio/conf/Catalina/localhost/SASStudio.xml
   "/usr/local/SASHome/sas/appserver/studio/conf/Catalina/localhost/${PREFIX_FILE}#SASStudio.xml"

# Start SAS Studio and keep the container alive while it runs
sudo -E bash -c "/usr/local/SASHome/sas/sasstudio.sh start && while true ; do :; sleep 60 ; done"
```

## Create the Docker image

Create the Docker image in a repository that Domino can access that includes the following:

* SAS Studio TAR file

* Domino-provided start script

* Dockerfile with instructions

## Enable SAS Workspace with pluggable Workspace properties

Paste the following code into **Pluggable Workspace Tools** to enable the SAS workspace.

```yaml theme={null}
sas:
    title: "SAS"
    iconUrl: "https://upload.wikimedia.org/wikipedia/commons/1/10/SAS_logo_horiz.svg"
    start: [ "/opt/domino/workspaces/sas/start" ]
    httpProxy:
        internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/SASStudio/"
        port: 8888
        rewrite: false
        requireSubdomain: false
```

## Optional: Persist SAS preferences and add-ons

To persist your SAS Studio preferences between Domino sessions, you can define a **Post Run Script** to automatically copy them to your Project Files when you **Stop and Sync** a session, and define a **Pre Run Script** to copy them from your Project Files into locations that SAS Studio recognizes at the start of subsequent sessions.

1. Add the following lines in the **Pre Run Script**.

   ```bash theme={null}
   # Copy preferences to Workspace
   PREFS_DIR="$DOMINO_WORKING_DIR/.sasstudio5"
   mkdir -p $PREFS_DIR
   rsync -rpq $PREFS_DIR $HOME
   ```

2. Add the following lines in the **Post Run Script**.

   ```bash theme={null}
   # Copy preferences from Workspace
   PREFS_DIR="$HOME/.sasstudio5"
   mkdir -p $PREFS_DIR
   rsync -rpq $PREFS_DIR $DOMINO_WORKING_DIR
   ```

   These scripts save add-ons per Project, so collaborators launching a Workspace in the same Project have the same add-ons. However, the scripts save preferences per user in the Project so that each user can have a different set of preferences.

## SAS Data Science Programming 3.5

To run SAS Data Science Programming 3.5 on Domino, contact your account Customer Success Manager (CSM) for SAS configuration on Domino. The following steps provide an overview of the process:

1. Build the SAS Data Science Programming 3.5 container. The Dockerfile expects a centos7 base image with the SAS 3.5 components installed (CAS, SAS Studio, Base SAS, etc.).

2. The SAS license file, `license.jwt`, must be in the same location as the Dockerfile when building the Docker image.

   The license file is copied into the container and the SAS license utility runs to license the software.

3. Push your newly created Docker image to a repository that your Domino deployment can access.

   This may be a repository hosted by your organization or a private repository within the Domino cluster.

4. Once you have created the Docker image, create a new Domino Environment using the custom image path.

5. Add the pluggable properties provided by your Domino representative, and make sure the “start:” parameter points to the start script created in the Dockerfile `/opt/domino/workspaces/sas/start`.

6. After building the Environment in Domino, you can select it in the Workspace launch menu when starting a new Workspace in your Project.

## Next steps

Learn how to [access Domino Data Sources from SAS Workspaces](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/data-source-from-sas).


# Use TensorBoard in Jupyter Workspaces
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/use-tensorboard-in-jupyter-workspaces

Install Jupyter-TensorBoard in a Compute Environment, configure Project-specific log paths, launch the extension, and troubleshoot CPU use.

[TensorBoard](https://www.tensorflow.org/programmers_guide/summaries_and_tensorboard) is a tool for visualizing TensorFlow data. TensorBoard operates by reading events files, which contain summary data that generated by TensorFlow. You can visualize your TensorFlow graph, plot quantitative metrics about graph, and show additional data that passes through the graph.

You can install and enable the [Jupyter-TensorBoard](https://github.com/lspvic/jupyter_tensorboard) server extension in your compute Environment to use TensorBoard in your Domino Jupyter.

## Set up your environment

You must create or modify an Environment to enable this extension in your Domino Workspaces. See [Environment Management](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments).

1. In the navigation bar, click **Environments**.

2. Click **Create Environment**.

3. Give the Environment a name, and then select a base image that has Python 3.8 installed. You can use an Environment as a base image if it uses this Domino standard:

   `quay.io/domino/compute-environment-images:ubuntu20-py3.9-r4.3-domino5.10-gpu` `quay.io/domino/domino-gpu-environment:develop.3d96fa12`

4. After setting the **Visibility** and entering a description, click **Create Environment**. The Environment’s Overview page opens.

5. Go to **Docker Settings** and click **Edit Dockerfile**.

6. Add the following lines to the **Dockerfile Instructions**:

   ```dockerfile theme={null}
   USER root

   RUN apt-get update
   RUN apt-get install -y --no-install-recommends libnode-dev npm nodejs
   RUN npm install -g configurable-http-proxy
   RUN pip install -U jupyter jupyter-tensorboard jupyter-server-proxy ipywidgets

   USER ubuntu
   ```

7. Add the following lines to the **Pluggable Workspace Tools** to add JupyterLab workspace:

   ```dockerfile theme={null}
   jupyterlab:
     title: "JupyterLab"
     iconUrl: "/assets/images/workspace-logos/jupyterlab.svg"
     start: [  "/opt/domino/workspaces/jupyterlab/start" ]
     httpProxy:
       internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
       port: 8888
       rewrite: false
       requireSubdomain: false
   ```

8. Add the following lines to the **Pre-run script** if the Project you plan to use in this Compute Environment is a Domino Project:

   ```dockerfile theme={null}
   #!/bin/bash
   set -e

   # Ensure required packages are installed (optional safety step)
   pip install --quiet --upgrade jupyter-server-proxy tensorboard "notebook<7"

   # Create Jupyter config directory
   CONF_DIR="${HOME}/.jupyter"
   mkdir -p "${CONF_DIR}"

   # Write valid server proxy config for TensorBoard
   cat << EOF > "${CONF_DIR}/jupyter_notebook_config.py"
   c.ServerProxy.servers = {
       "tensorboard": {
           "command": [
               "tensorboard",
               "--logdir", "/domino/datasets/local/${DOMINO_PROJECT_NAME}/tensorboard_logs",
               "--host", "0.0.0.0",
               "--port", "{port}"
           ],
           "timeout": 30,
           "launcher_entry": {
               "title": "TensorBoard"
           }
       }
   }
   EOF
   ```

9. Add the following lines to the **Pre-run script** if the Project is a Git-based Project:

   ```dockerfile theme={null}
   #!/bin/bash
   set -e

   # Ensure required packages are installed (optional safety step)
   pip install --quiet --upgrade jupyter-server-proxy tensorboard "notebook<7"

   # Create Jupyter config directory
   CONF_DIR="${HOME}/.jupyter"
   mkdir -p "${CONF_DIR}"

   # Write valid server proxy config for TensorBoard
   cat << EOF > "${CONF_DIR}/jupyter_notebook_config.py"
   c.ServerProxy.servers = {
       "tensorboard": {
           "command": [
               "tensorboard",
               "--logdir", "/mnt/data/${DOMINO_PROJECT_NAME}/tensorboard_logs",
               "--host", "0.0.0.0",
               "--port", "{port}"
           ],
           "timeout": 30,
           "launcher_entry": {
               "title": "TensorBoard"
           }
       }
   }
   EOF
   ```

10. Click **Build**.

    The **Revisions** page opens. If the new revision builds successfully, you can use this Environment.

## Use Jupyter-TensorBoard

1. Open the Project you want to use with Jupyter-Tensorboard.

2. In the navigation pane, click **Settings**.

3. From **Compute Environment**, select the Environment you created previously. A notification opens to verify that the new Environment is now set.

4. In the navigation pane, click **Workspaces**, then select **Jupyterlab** and launch a new workspace.

5. Select **Tensorboard** from the Launcher.

See the [TensorBoard README](https://github.com/tensorflow/tensorboard/blob/master/README.md) to learn how to consume TensorFlow events.

## Troubleshooting

1. If you encounter issues loading Tensorboard initially, confirm that Tensorflow is operating properly. Loading the application depends on Tensorflow.

2. By default, Domino’s standard Compute Environments have `tensorflow-gpu` installed (for example, `pip install tensorflow-gpu`). Therefore, Tensorboard and Tensorflow do not work on a CPU hardware tier. If you’d like to use Tensorboard on a CPU, make sure that CPU optimized Tensorflow is installed (for example, `pip install tensorflow`).


# Data Planes
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/data-plane

The compute locations where Domino runs your workloads, one per region or cloud, selected by the hardware tier you pick when you launch a workspace, job, app, or endpoint.

A Data Plane is a compute location where Domino runs your workloads. A single Domino deployment can have more than one Data Plane, in different regions or clouds, so you can run work close to the data it needs or inside a region or cloud that your requirements dictate.

## How it works

* **Hardware tier**: the compute configuration you pick when you launch a workload. Each hardware tier is attached to one Data Plane, so choosing a tier also chooses where the workload runs.
* **Control Plane**: the Domino services and user interface that schedule and manage work across every Data Plane.

```mermaid theme={null}
%%{init: {"flowchart": {"wrappingWidth": 400}}}%%
flowchart TB
  W["You launch a workspace,<br/>job, app, or endpoint"] --> CP["Control Plane<br/>schedules and manages work"]
  CP -->|"tier in the local<br/>Data Plane"| L
  CP -->|"tier in a remote<br/>Data Plane"| R
  subgraph DP["Data Planes"]
    L["Local<br/>namespace in the<br/>Control Plane cluster"]
    R["Remote<br/>separate Kubernetes cluster in<br/>another region, cloud, or on-premises"]
  end
  DP ~~~ spacer[" "]
  style spacer fill:none,stroke:none
```

Data might be available only in certain Data Planes, because of geographic access restrictions or the cost of moving data between data centers. Your Domino administrator configures the Data Planes and hardware tiers, and associates external data volumes (EDVs) with the Data Planes that can reach them.

## Local and remote Data Planes

* A **local Data Plane** is a namespace within the same Kubernetes cluster as the Domino Control Plane, often called the `compute` namespace.
* A **remote Data Plane** is a namespace in a separate Kubernetes cluster from the Control Plane.

[Starburst and Trino data sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-starburst-trino) are available only in the local Data Plane; every other type of data source works in both. Datasets are available in remote Data Planes that an administrator has configured for them, but [Domino CLI](/cloud/reference/cli/install-the-cli) access to Datasets works only in the local Data Plane.

For large data sets, EDVs are the primary way to access data across Data Planes, because they carry a first-class notion of data locality. Mount the EDVs that are configured for the Data Plane your execution runs in.

## Hybrid deployments

A deployment whose Data Planes span multiple clouds, regions, or on-premises infrastructure runs as Domino Nexus: remote Data Planes attached to a single Control Plane. For the architecture, see [Nexus hybrid architecture](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture).

## Use a Data Plane

Launch a [workspace](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace), [job](/cloud/platform-capabilities/core-concepts/jobs), [app](/cloud/platform-capabilities/features/apps), or [endpoint](/cloud/platform-capabilities/features/model-deployment) with a hardware tier from the Data Plane you need. In a hybrid deployment, the launch dialog groups hardware tiers by Data Plane, and the `Local` group runs the workload in the Control Plane cluster.

<Frame>
  <img alt="The Launch New Workspace dialog showing the hardware tier selector, with tiers grouped by Data Plane: CLOUD / US-EAST, CLOUD / US-WEST, CLOUD / EMEA, and ON PREM / EUROPE." />
</Frame>

You can also [mount EDVs](/cloud/platform-capabilities/core-concepts/data/external-volumes/use-a-mounted-volume) that are configured for specific Data Planes, and launch [compute clusters](/cloud/platform-capabilities/features/compute-clusters-hpc) alongside your executions.

## Related

* [External data volumes](/cloud/platform-capabilities/core-concepts/data/external-volumes/index)
* [Nexus hybrid architecture](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture)
* [Manage Data Planes](/cloud/admin/platform-configuration/manage-data-planes) for administrators
* [Deployment options](/cloud/platform-overview/deployment-options)


# Knowledge management
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/knowledge-management/index

How Tags, Properties, and search work together to make work in Domino findable and reusable.

Knowledge management is how your organization finds and reuses past work instead of reinventing it. Three capabilities carry it: [Tags](/cloud/platform-capabilities/core-concepts/knowledge-management/tags) organize assets under a controlled vocabulary, [Properties](/cloud/platform-capabilities/core-concepts/knowledge-management/properties) document open-ended information about them, and [search](/cloud/platform-capabilities/features/collaboration/domino-search) finds them.

## How it works

Classification is defined centrally and applied by everyone:

* **Tags** categorize assets with values drawn from a controlled, hierarchical list. Namespaces are the allowed keys and Tags the allowed values, so a Project can carry `business-domain:oncology` and `data-sensitivity:confidential` at once.
* **Properties** capture typed information that no fixed list can hold, such as a last-validated date or an internal model ID.
* Admins and Librarians define the tag structure and the available Properties, so classification stays consistent across teams. Everyone applies them to their own assets: Apps, Projects, Project Templates, Datasets, NetApp Volumes, Models, Environments, and Policies.
* **Search** closes the loop. Global search, entity listing filters, and the Tag Browser surface classified work across the deployment, within each user's permissions. Projects set to **Searchable** appear in results even when their contents stay restricted, so colleagues can discover that relevant work exists and request access.

```mermaid placement="top-right" theme={null}
flowchart TB
    admins["Admins and<br/>Librarians"] --> tags
    admins --> props
    subgraph classify ["Classification"]
        tags["Tags<br/>controlled,<br/>hierarchical"]
        props["Properties<br/>typed,<br/>open-ended"]
    end
    tags --> assets
    props --> assets
    assets["Apps, Projects, Project Templates,<br/>Datasets, NetApp Volumes, Models,<br/>Environments, Policies"] --> search["Search, listing filters,<br/>and the Tag Browser"]
```

## Choose between tags and properties

Tags organize entities so people can find them. Properties document information about an entity. Pick based on the shape of the value you need to capture:

| Classification | Use | Why |
| - | - | - |
| Risk tier (Low, Medium, High) | Tags | Fixed, controlled set of values |
| Data sensitivity (Public, Internal, Confidential) | Tags | Bounded, controlled vocabulary |
| Department (Marketing, Sales, Engineering) | Tags | Controlled list with predictable values |
| Last validated date | Properties | Open-ended date value, varies per asset |
| Internal model ID | Properties | Free-form identifier unique to each asset |
| Loan portfolio reference | Properties | Free-form text that varies per asset |

Use Tags when the set of allowed values is finite and known in advance: users pick from a controlled list, and parent-child relationships let you model nested categories. Use Properties when the value can't be enumerated up front. Most teams use both, Tags for organization and Properties for documentation.

## Set up tags and properties

Tags and Properties become usable once someone defines what users can pick from. Set them up in this order:

1. A CloudAdmin or Librarian builds the tag structure in **Govern** > **Tag Browser**. Create Namespaces and Tags there, or [import them from CSV](/cloud/platform-capabilities/core-concepts/knowledge-management/tags#import-and-export).
2. A CloudAdmin or Librarian defines the available [Properties](/cloud/platform-capabilities/core-concepts/knowledge-management/properties) in the same **Govern** area, choosing each property's type and the assets it applies to.
3. Asset owners apply tags and set property values from each asset's overview page.

## Who can manage tags and properties

| Action | Who can do it |
| - | - |
| Create, edit, deprecate, merge, and import Namespaces and Tags | CloudAdmin or Librarian |
| Define and change Properties | CloudAdmin or Librarian |
| Apply or remove tags on an asset | The asset's owner, and SysAdmins and CloudAdmins |
| Set or update property values on an asset | The asset's owner, and SysAdmins and CloudAdmins |
| Search and filter by tags | Any user, within their existing permissions |

To keep a classification under central control, define its allowed values centrally: use a Namespace, or a Single select or Multi select property. Only CloudAdmins and Librarians can change those values, so asset owners can only pick from the options you define.

## Work with knowledge management

<Columns>
  <Card title="Tags" href="/cloud/platform-capabilities/core-concepts/knowledge-management/tags">
    Build a tag structure, apply tags to your assets, and find tagged work.
  </Card>

  <Card title="Properties" href="/cloud/platform-capabilities/core-concepts/knowledge-management/properties">
    Define typed fields and record values on your assets.
  </Card>
</Columns>

## Related

* [Use Domino search](/cloud/platform-capabilities/features/collaboration/domino-search): find files, Projects, Jobs, and comments across the deployment.
* [Collaboration](/cloud/platform-capabilities/features/collaboration): share and discover work across teams.
* [Projects](/cloud/platform-capabilities/core-concepts/projects): the unit of organization most classified assets belong to.


# Properties
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/knowledge-management/properties

Enrich Domino assets with custom, typed information that no fixed vocabulary can hold.

Properties let you enrich entities with custom pieces of information. Where [Tags](/cloud/platform-capabilities/core-concepts/knowledge-management/tags) organize assets under a controlled vocabulary, Properties document them: a last-validated date, an internal model ID, an owning organization, a loan portfolio reference.

Properties are defined centrally, so the same field means the same thing everywhere it appears. They live in the same **Govern** area as the [Tag Browser](/cloud/platform-capabilities/core-concepts/knowledge-management/tags#manage-the-tag-structure), and defining or changing them requires the **CloudAdmin** or **Librarian** role. Everyone else fills in values on their own assets.

## How it works

Each property definition carries:

* A **name**, which identifies the property on every asset it applies to.
* A **type**, which constrains the values users can enter.
* An optional **description** explaining what the property captures.
* An optional **group**, which keeps related properties together.
* A list of **applicable assets**: Apps, Projects, Project Templates, Datasets, NetApp Volumes, Models, Environments, Policies, and Policy Versions.

Once defined, a property is available for assignment on every asset type listed for it. Properties have no hierarchy and no namespaces, and outside the select types below, no predefined set of values: any value that matches the property's type is accepted.

## Property types

| Type | Accepted value |
| - | - |
| Text | Free-form text |
| Number | A numeric value |
| Date | A single date |
| Boolean | True or false |
| User | One Domino user |
| Org | One Domino organization |
| User or Org | Either a Domino user or an organization |
| Single select | One value from a predefined list of options |
| Multi select | One or more values from a predefined list of options |

## Set property values

Set and update values from the asset's overview page. The asset's owner can do this, as can SysAdmins and CloudAdmins.

| Asset | Scope |
| - | - |
| Apps | Globally on the App, and independently on each version |
| Projects | The Project. Values appear on the Project card and are indexed by global search |
| Project Templates | Set at template creation, then carried into new Projects according to the behavior chosen per property |
| Datasets | The Dataset, shared across all of its snapshots |
| NetApp Volumes | The volume, shared across all of its snapshots |
| Models | Globally on the Model, and independently on each version |
| Environments | The Environment |
| Policies | The Policy |
| Policy Versions | The specific version of a Policy, independently of values set on the Policy itself |

On Apps, Models, and Policies, global and per-version values coexist. That lets you record stable, asset-wide information alongside values that change as you iterate.

### Carry property values into new Projects

When a Project Template is created, its creator picks one of three behaviors for each property, which decides what a Project created from the template starts with:

| Behavior | Effect on the new Project |
| - | - |
| Copy value | The value on the template is copied into the new Project |
| Empty (optional) | The property starts empty, and users can fill it in later |
| Empty (required) | Users must supply a value before the Project can be created |

Property values on the template's Datasets, NetApp Volumes, and Apps carry over to the corresponding assets in the new Project.

## Related

* [Tags](/cloud/platform-capabilities/core-concepts/knowledge-management/tags): categorize the same assets with a controlled, hierarchical vocabulary.
* [Knowledge management](/cloud/platform-capabilities/core-concepts/knowledge-management): how Tags, Properties, and search relate, and when to choose each.


# Tags
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/knowledge-management/tags

Standardize how your organization categorizes and discovers work with a shared, hierarchical tag structure.

Tags let you standardize how your organization categorizes and discovers work, so people find and reuse institutional knowledge instead of reinventing it.

Tags are organized as key-value pairs in two parts. **Namespaces** are the allowed keys, and **Tags** are the allowed values within a given Namespace. Think of a Namespace as the category or dimension you classify by, and Tags as the values that category can take. You can apply tags from several namespaces to the same asset.

<Frame>
  <img alt="Three Namespaces, Methodology, Dept, and Priority, each listing its allowed tags, assigned to a Project that carries one tag chip from each." />

  <img alt="Three Namespaces, Methodology, Dept, and Priority, each listing its allowed tags, assigned to a Project that carries one tag chip from each." />
</Frame>

Building and changing the tag structure requires the **CloudAdmin** or **Librarian** role. Once it exists, anyone can apply tags to their own assets and filter by them.

## How it works

* **Namespaces** group related tags, for example `business-domain`, `data-sensitivity`, or `project-status`. Namespace labels are alphanumeric and accept hyphens and underscores.
* **Tags** live inside a Namespace and support hierarchy through parent-child relationships. A tag is created either at the root of its Namespace or nested under an existing tag.
* **Assignment policy** is set per Namespace. When **Allow multiple tags per entity** is off, the default, an entity holds only one tag from that Namespace at a time.
* **Supported assets** are Apps, Projects, Project Templates, Datasets, NetApp Volumes, Models, Environments, and Policies. Tags appear as `namespace:tag` chips on the asset's card and are searchable as soon as they are applied.

## Apply tags

Assign and remove tags from the asset's overview page. The asset's owner can do this, as can SysAdmins and CloudAdmins. Tags come from the namespaces already defined in your tag structure; you don't create tags at assignment time.

<Frame>
  <img alt="Select tags drawer on an App overview showing the tag tree and the tags selected so far" />
</Frame>

What a tag attaches to depends on the asset:

| Asset | A tag applies to |
| - | - |
| Apps | The App |
| Projects | The Project |
| Project Templates | The template, and is inherited by Projects created from it |
| Datasets | The Dataset, shared across all of its snapshots |
| NetApp Volumes | The volume, shared across all of its snapshots |
| Models | The Model |
| Environments | The Environment |
| Policies | The Policy |

A Project created from a Project Template inherits every tag on that template. Tags on the template's Datasets, NetApp Volumes, and Apps carry over to the corresponding assets in the new Project.

<Warning>
  When a Namespace allows only one tag per entity, selecting a new tag from that Namespace replaces the tag already there.
</Warning>

<Note>
  Tags on a Model are separate from any MLflow tags on the underlying model.
</Note>

## Find tagged assets

Global search indexes tag labels and descriptions, so assets are findable by tag from Domino's main search bar without knowing the exact Namespace or tag path. Narrow a result set to one or more tags from the filter side panel beside your results, or type the tag straight into the search bar.

Each listing view for Apps, Projects, Datasets, NetApp Volumes, Models, Environments, and Policies carries its own tag filter as well. Type a Namespace name followed by a colon and the filter limits its suggestions to tags in that Namespace, which makes it fast to filter along one classification axis.

<Frame>
  <img alt="Apps and Agents listing filtered by tag, showing autocomplete suggestions for the Organism namespace" />
</Frame>

## Manage the tag structure

Go to **Govern** > **Tag Browser** to build and maintain the structure. The Tag Browser shows the full tag tree alongside the assets tagged within it, with a count on each node.

<Frame>
  <img alt="Tag Browser showing the tag tree on the left and the Apps tagged with the selected tag on the right" />
</Frame>

### Deprecate a tag

Deprecate a tag to retire a classification without breaking what already uses it. A deprecated tag stays on every entity currently assigned to it but can no longer be applied to new ones.

### Merge tags

Merging consolidates duplicate or overlapping tags. Every assignment on the source tags transfers to the target tag, and the source tags are deleted.

<Warning>
  Merging tags cannot be undone.
</Warning>

### Import and export

Import a tag structure from CSV in the Tag Browser to migrate from a spreadsheet or an external provider. Download a [sample import file](/attachments/cloud/sample-taxonomy.csv) to start from. The import preview flags any row with an error before anything is committed.

<Accordion title="CSV import file format">
  Each row defines one tag, and the file has no header row:

  ```
  namespace_label,namespace_description,tag_path,tag_description
  ```

  | Column | Required | Description |
  | - | - | - |
  | `namespace_label` | Yes | Namespace name. Created automatically if it doesn't already exist. |
  | `namespace_description` | No | Namespace description. |
  | `tag_path` | Yes | Slash-separated hierarchy. `Oncology/Phase-1` creates a `Phase-1` tag nested under `Oncology`. |
  | `tag_description` | No | Tag description. |
</Accordion>

To export, use the [Tags API](/cloud/reference/api/tags-api), which returns the full tag structure as a CSV file suitable for backup or for seeding another deployment.

## Related

* [Properties](/cloud/platform-capabilities/core-concepts/knowledge-management/properties): record typed, open-ended information on the same assets.
* [Knowledge management](/cloud/platform-capabilities/core-concepts/knowledge-management): how Tags, Properties, and search relate, and when to choose each.
* [Tags API](/cloud/reference/api/tags-api): manage namespaces, tags, and assignments programmatically.


# Compare results
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/reproducibility/compare

Compare two Jobs, file revisions, or Project states, inspect metadata and result differences, and tune file-diff rendering limits.

Domino provides comparison tools to help you analyze changes in Files, Jobs, and Project states so you can manage Project artifact changes and audit those changes more effectively.

## Compare Jobs

On the Jobs Dashboard, select exactly two Jobs by checking the checkbox in the left-hand column. Then click the icon indicating comparison, above the table.

The Job comparison view shows the following:

* Differences in summary metadata and diagnostic statistics

* Differences in the files as they were when the Job ran

* Differences in the Results of the Jobs

Certain file types are rendered side by side for visual comparison. Other types (for example, binaries) are simply listed as different.

You have the option to comment not only on the comparison itself but also on the comparison between two files.

## Compare file versions

To compare different file versions, view a file and click the Compare Revisions button. This enables side-by-side viewing of two selected versions.

## Compare Project states from the activity feed

On the Activity Feed screen, select two Workspaces or Jobs and click the compare button at the top of the page.

## Relevant Admin config settings

The following configuration records settings affect aspects of how collaboration functionality works:

| Config key | Description | Example value |
| - | - | - |
| `frontend.maxNumberOfRenderedFileDiffs` | Maximum number of files shown when comparing two Jobs. | `250` |
| `frontend.maxTotalDiffBytesToRender` | Maximum aggregate size of the files shown when comparing two Jobs. | `1024000` |

## Next steps

View [Jobs](/cloud/platform-capabilities/core-concepts/jobs/monitor-jobs), [Project files](/cloud/platform-capabilities/core-concepts/projects/files-artifacts-and-results), and [Project activity](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#view-project-activity).


# File syncing and persistence
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/reproducibility/file-syncing

Track Project files and artifacts across executions, understand Blob Store hydration and persistence, and retain access to earlier revisions.

Domino automatically tracks files in your Project. In a [DFS-backed Project](/cloud/platform-capabilities/core-concepts/projects/dfs-projects), all files in the root of your Project folder are tracked unless they are excluded by [.dominoignore](/cloud/platform-capabilities/core-concepts/projects/dfs-projects#exclude-files-from-sync) logic. When you run code, these files show up at `/mnt` in the local file system. In a [Git-backed Project](/cloud/platform-capabilities/core-concepts/projects/git-based-projects), Domino tracks files in your **artifacts** directory, which show up at `/mnt/artifacts` inside a running Workspace or executing Job.

Keeping file-based data in these folders gives you the benefit of automatic reproducibility. However, due to how Domino synchronizes these files. Domino isn’t designed for high performance if your files are more than \~10 GB in total size or more than \~100,000 individual files.

Domino stores the contents of Project files in the Domino Blob Store. The backing storage mechanism for the Blob Store varies based on the deployment infrastructure in which Domino is running:

| Deployment infrastructure | Dataset storage implementation |
| - | - |
| AWS | S3 |
| Azure | |
| GCP | |
| On-Prem or other cloud | NFS-compatible NAS |

When you first start a task in Domino that spins up a new compute resource to run your code, Domino hydrates the local file system on that compute resource, with your Project files. This happens, when:

* You start a Workspace (but not when you resume a paused Workspace)

* When you run a Job

* When a Domino endpoint or App starts (or restarts)

When a Job completes, or when you explicitly sync work within a Workspace session, Domino persists a new revision of your files to the Blob store. Note that Domino endpoints and Apps cannot persist local file system changes back to the Blob Store.

<img alt="Blob store" />

When persisting changes, Domino never destroys information. In that sense, the Blob Store is an *immutable revisioned file store*. For example, if you edit a file, Domino adds the new version but doesn’t delete the old one. Or if you delete a file, Domino notes that the latest version of your Project has it deleted, but the previous version is still accessible by reverting to a past state.

## Next steps

These topics in this section explain how you can make your workflows reproducible in Domino.

[Reproducibility use cases](/cloud/platform-capabilities/core-concepts/reproducibility/reproducibility-use-cases)\
Learn how to reproduce the results of a Job, Workspace, Model, App, or Launcher.

[Selectively revert past materials](/cloud/platform-capabilities/core-concepts/reproducibility/revert-materials)\
Selectively restore a part of a Project, such as the package library version, while keeping your latest code and data.

[Remove a file from the DRE: Permanent deletion](/cloud/platform-capabilities/core-concepts/reproducibility/permanent-deletion)\
Purge a file completely and permanently from the blob store.

[Track external data](/cloud/platform-capabilities/core-concepts/reproducibility/track-external-data)\
Materialize external data as a file in Domino to benefit from the automatic tracking that Domino provides.

[Tips for reproducibility in Domino](/cloud/platform-capabilities/core-concepts/reproducibility/reproducibility-tips)\
Tips for maximizing the power of the Domino Reproducibility Engine.


# Domino Reproducibility Engine (DRE)
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/reproducibility/index

Use the Domino Reproducibility Engine to track code, data, software, commands, and results across DFS-backed and Git-backed Projects.

The Domino Reproducibility Engine automatically tracks data science work, making it easy to reconstruct or reproduce it later.

## Reproducibility

Reproducibility in data science is critical for three reasons:

* Reproducibility is the foundation for collaboration and work reuse. Data science often involves resuming previous work (done by you or a colleague). You can only resume work if you can run the same inputs and produce the same results. That means having access to the exact code, data, and software packages as they were.

* Reproducibility is critical for auditability, review, and compliance. Especially in regulated industries, internal or external stakeholders may need to reconstruct data science work, either to validate it for production or to answer questions down the road. These tasks require reconstructing the materials used during the development process.

* Reproducibility is good science and hygiene in any research process. Much literature has been written about this, but one paper we recommend is [Ten Simple Rules for Reproducible Computational Research](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003285) in PLOS Computational Biology.

## Domino Reproducibility Engine

Because reproducibility is a complex topic, the DRE doesn’t live on a single screen or location in Domino. Rather, it is integrated deeply into the architecture of the platform and woven throughout many features.

At its core, the DRE works by automatically tracking *tuples*, which are specific versions of *input materials* and *results*. Conceptually, a tuple contains the following components:

<img alt="Tuple components" />

* **Code**: the source code used, this could also include notebooks.

* **Data**: data sources and data files used and manipulated by the code.

* **Software**: the packages, drivers, and configuration in which your code runs.

* **Command**: in the context of a batch job, the executed command.

* **Results**: any outputs produced during the execution of your code. Typically these are charts, additional data, serialized model files, and even rendered notebooks. These can also include statistics (for example, RMSE, AUC) you want to track.

In Domino, there are several ways to materialize these components. Here is how the above conceptual materials are manifested in Domino.

<table>
  <thead>
    <tr>
      <th>Concept</th>
      <th>Domino implementation</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td />

      <td><p><strong>DFS-backed Project</strong></p></td>
      <td><p><strong>Git-backed Project</strong></p></td>
    </tr>

    <tr>
      <td><p>Code</p></td>
      <td><p>Your code is in your <strong>Project files</strong> directly. In this case, the DRE keeps a space-efficient snapshot of your Project files.</p></td>
      <td><p>Your code is in one or more <strong>externally hosted Git repos</strong>. In this case, the DRE tracks the commit of each Git repo that was referenced when your code ran.</p></td>
    </tr>

    <tr>
      <td><p>Data</p></td>
      <td><p>Your code can access data that is stored inside the Domino platform, or data that lives externally. The DRE only has direct awareness of data that is stored inside the Domino Platform.</p> <p>For data stored in your Domino project directly as files or <a href="/cloud/platform-capabilities/core-concepts/projects/files-artifacts-and-results#work-with-artifacts">Artifacts</a>, the DRE automatically tracks these in a space-efficient manner.</p> <p>For data stored in <a href="/cloud/platform-capabilities/core-concepts/data/datasets">Datasets</a>, the DRE tracks which Snapshot was used by your code.</p> <p>For data in Data Sources, or data you access by connecting directly to databases or external sources, the DRE cannot automatically track it. To track this type of external data, see <a href="/cloud/platform-capabilities/core-concepts/reproducibility/track-external-data">Track external data</a>.</p></td>
    </tr>

    <tr>
      <td><p>Software</p></td>
      <td><p>Domino tracks the version of the <a href="/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments">compute Environment</a> used when your code executes. A revision of compute Environment is a revision of a Docker image along with some supplemental configuration scripts.</p></td>
    </tr>

    <tr>
      <td><p>Results</p></td>
      <td><p>Files in your Project that are created or modified when your code executes.</p></td>
      <td><p>Files in the <a href="/cloud/platform-capabilities/core-concepts/projects/files-artifacts-and-results#work-with-artifacts">Artifacts</a> section of your Project that are created or modified when your code executes.</p></td>
    </tr>

    <tr>
      <td><p>Statistics or results metadata</p></td>
      <td><p>For stats that show up in Domino’s <a href="/cloud/platform-capabilities/core-concepts/jobs/monitor-jobs">Jobs dashboard</a>, Domino extracts them from a file named <code>domino.json</code> in your Project’s Files (or Artifacts). See <a href="/cloud/platform-capabilities/core-concepts/jobs/customize-job-results">Diagnostic statistics</a> for more.</p> <p>For our newer <a href="/cloud/platform-capabilities/features/models/register-and-govern/model-registry">MLflow-based experiment tracking</a>, use the MLfLow API to record your stats from your code.</p></td>
    </tr>
  </tbody>
</table>

Each of these materials can change independently in Domino. For example, you may make multiple discrete edits to your code, re-running experiments after each one; and then later make a big change to update a package in your compute Environment.

At select operations (for example, running a Job, syncing your work in Workspace, publishing a Domino endpoint), Domino records the state of your materials at that point.

Domino automatically tracks the active revision of each component at the time you execute your code. This diagram of how the DRE keeps track of each *tuple* as components are evolving separately.

<img alt="Track tuples with DRE" />

Below is an example of the details of a Job in Domino, highlighting some of the reproducible materials. Note that the **Command** is a link that takes you to a full snapshot of the code that was executed.

<img alt="Reproducible materials" />

## Next steps

These topics in this section explain how you can make your workflows reproducible in Domino.

[Reproducibility use cases](/cloud/platform-capabilities/core-concepts/reproducibility/reproducibility-use-cases)\
Learn how to reproduce the results of a Job, Workspace, Model, App, or Launcher.

[Selectively revert past materials](/cloud/platform-capabilities/core-concepts/reproducibility/revert-materials)\
Selectively restore a part of a Project, such as the package library version, while keeping your latest code and data.

[File syncing and persistence](/cloud/platform-capabilities/core-concepts/reproducibility/file-syncing)\
Domino automatically tracks files in your Project and keeps previous versions in the blob store.

[Remove a file from the DRE: Permanent deletion](/cloud/platform-capabilities/core-concepts/reproducibility/permanent-deletion)\
Purge a file completely and permanently from the blob store.

[Track external data](/cloud/platform-capabilities/core-concepts/reproducibility/track-external-data)\
Materialize external data as a file in Domino to benefit from the automatic tracking that Domino provides.

[Compare results and Project states](/cloud/platform-capabilities/core-concepts/reproducibility/compare)\
Domino provides comparison tools to help you analyze changes in Files, Jobs, and Project states so you can manage Project artifact changes and audit those changes more effectively.

[Tips for reproducibility in Domino](/cloud/platform-capabilities/core-concepts/reproducibility/reproducibility-tips)\
Tips for maximizing the power of the Domino Reproducibility Engine.


# Remove a file from the DRE: Permanent deletion
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/reproducibility/permanent-deletion

Permanently purge every Blob Store instance with matching file contents for compliance, with help from a Domino administrator.

In rare cases, you may need to purge a file completely from the Blob Store. For example, if you discover that a file contains data that must be deleted for compliance reasons.

Contact your Domino administrator to [delete all the file instances](/cloud/admin/data-administration/project-file-storage#delete-every-instance-of-a-file) to permanently remove such a file.

<Warning>
  A full delete irrevocably removes all files that match the exact *contents* of the file you are deleting. For example, if multiple projects all contain a `data.txt` file with the exact same contents, full-deleting it in one Project full-deletes it everywhere.
</Warning>

## Next steps

These topics in this section explain how you can make your workflows reproducible in Domino.

[Reproducibility use cases](/cloud/platform-capabilities/core-concepts/reproducibility/reproducibility-use-cases)\
Learn how to reproduce the results of a Job, Workspace, Model, App, or Launcher.

[Selectively revert past materials](/cloud/platform-capabilities/core-concepts/reproducibility/revert-materials)\
Selectively restore a part of a Project, such as the package library version, while keeping your latest code and data.

[File syncing and persistence](/cloud/platform-capabilities/core-concepts/reproducibility/file-syncing)\
Domino automatically tracks files in your Project and keeps previous versions in the blob store.

[Track external data](/cloud/platform-capabilities/core-concepts/reproducibility/track-external-data)\
Materialize external data as a file in Domino to benefit from the automatic tracking that Domino provides.

[Tips for reproducibility in Domino](/cloud/platform-capabilities/core-concepts/reproducibility/reproducibility-tips)\
Tips for maximizing the power of the Domino Reproducibility Engine.


# Domino Reproducibility tips
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/reproducibility/reproducibility-tips

Strengthen reproducibility with Jobs, pinned package versions, stable file names, saved random states, and revisioned intermediate artifacts.

Learn how to make the most of the Domino Reproducibility Engine (DRE) with these tips. Since the DRE operates at a low level of abstraction like tracking Docker images and files, it offers a lot of flexibility and power.

## Jobs offer the strongest reproducibility environment

For results that have the strongest reproducibility requirements (for example, for regulatory or compliance reasons), produce these results by executing Jobs, not through Workspaces. Since a Workspace is interactive, it’s impossible to make every aspect of its state reproducible (for example, a data scientist can execute cells in a different order). Jobs are headless, meaning they run start-to-stop making them better suited for strong reproducibility.

## Explicit package versions

Use explicit version numbers when installing packages in Python or R, and consolidate package installation in one place.

It’s best to put these in your compute [Environment definition](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition). Remember, however, if Domino detects a `requirements.txt` file in your Project files, it runs `pip install` (or `conda install`) on the requirements file before it executes your code. An analogous pattern that some R practitioners use is to have a single `install.R` file in their Project, which they run before anything else.

If your package definitions are in a file like this, note that your software state is spread across both your compute Environment and your Project files. However, you may find this advantageous in some circumstances because you can use Domino’s [comparison feature](/cloud/platform-capabilities/core-concepts/projects/dfs-projects#compare-file-revisions) to compare two versions of `requirements.txt` or `install.R` to see exactly what changed.

Regardless, use explicit package versions, for example, `pip install torch==1.12.1` rather than `pip install torch`

That way, you’ll get the same results running the code in the future, even if the latest version is newer.

For instructions on installing specific versions of R packages, see this [Stackoverflow post](https://stackoverflow.com/questions/17082341/how-to-downgrade-an-r-package-by-installing-an-older-version).

## Don’t use file names to track revision

Because Domino keeps your files revisioned automatically, it’s best if you *don’t encode file names with your own versioning scheme*. For example, you don’t need to append “v1”, “v2”, “final”, etc. to your version names. In fact, it’s better if you don’t do this, because Domino treats files with the same name as changes when it presents comparisons between Jobs.

## Other tips

Because the DRE tracks all files that a Job produces, there are many clever tricks you can use to track things in files. For example:

* If you are using random numbers in your code, you can persist the seed value to a file, so that you can recreate it later if you need to reproduce the precise random behavior. In Python, you can [get the state](https://docs.python.org/3/library/random.html#random.getstate) of the random generator and pickle it to a file. In R you can use `set.seed` to set your own value (which you need to select on your own).

* Generalizing the above: R and Python both have capabilities for persisting memory state to files. In R you can [save objects](https://stat.ethz.ch/R-manual/R-devel/library/base/html/save.html), even for your entire set of variables. Python lets you [pickle objects](https://docs.python.org/3/library/pickle.html). You can use these techniques to easily create revisioned artifacts of intermediate values you’ve calculated. This may be useful to save time later, by resuming a computationally intensive process from an intermediate state instead of starting from scratch.

* You can create your own domain-specific formats with customized results you may want to track and compare over time. For example, you can create a “model summary.pdf” that has the same format between different Jobs.

## Next steps

These topics in this section explain how you can make your workflows reproducible in Domino.

[Reproducibility use cases](/cloud/platform-capabilities/core-concepts/reproducibility/reproducibility-use-cases)\
Learn how to reproduce the results of a Job, Workspace, Model, App, or Launcher.

[Selectively revert past materials](/cloud/platform-capabilities/core-concepts/reproducibility/revert-materials)\
Selectively restore a part of a Project, such as the package library version, while keeping your latest code and data.

[File syncing and persistence](/cloud/platform-capabilities/core-concepts/reproducibility/file-syncing)\
Domino automatically tracks files in your Project and keeps previous versions in the blob store.

[Remove a file from the DRE: Permanent deletion](/cloud/platform-capabilities/core-concepts/reproducibility/permanent-deletion)\
Purge a file completely and permanently from the blob store.

[Track external data](/cloud/platform-capabilities/core-concepts/reproducibility/track-external-data)\
Materialize external data as a file in Domino to benefit from the automatic tracking that Domino provides.


# Reproducibility use cases
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/reproducibility/reproducibility-use-cases

Reproduce Jobs, Workspaces, models, Apps, and Launcher results, trace artifacts to their source Jobs, and compare captured inputs and outputs.

Learn how to use the Domino Reproducibility Engine (DRE) to reproduce work across your data science workflows. In this article, you learn how to accomplish the following reproducibility scenarios:

* Reproduce Job results

* Find the Job that generated an Artifact

* Reproduce work from a Workspace

* Reproduce a deployed Model

* Reproduce an App

* Reproduce Launcher results

## Reproduce Job results

The most reliable way of creating reproducible work in Domino is with Jobs.

Domino detects file system changes made by a Job upon its completion, regardless of success or failure. Domino defines "results" as the modified or created files. In a DFS-backed project, these are visible in the Project’s **Files** section; in a git-backed Project, they are in the Project’s **Artifacts**.

This behavior applies to Jobs regardless of how they are initiated: manually through the web GUI; as a [Scheduled Job](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job); through the [CLI](/cloud/platform-capabilities/core-concepts/jobs/create-and-run-jobs#run-a-job-from-the-cli); or through the [Domino API](/cloud/platform-capabilities/core-concepts/jobs/create-and-run-jobs#run-a-job-from-the-domino-api). This is also true for executions of Launchers because [Launchers](/cloud/platform-capabilities/features/launchers) run Jobs under the hood.

The [Jobs dashboard](/cloud/platform-capabilities/core-concepts/jobs/monitor-jobs) lets you browse past Jobs. Click the row for a Job to view the Results and additional reproducibility details of that Job.

The **Results** tab on the right pane shows all files that were created or modified by the Job when it ran.

The **Details** tab shows additional materials necessary to reproduce the results: for example, the snapshot versions of any Datasets that were used, and the revision of the [compute Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments).

The Jobs Dashboard offers an even more specialized view to [compare two Jobs](/cloud/platform-capabilities/core-concepts/jobs/monitor-jobs#compare-jobs), so you can see what changed across the results and the inputs.

## Find the Job that generated an artifact

Often you may have a specific Artifact (for example, a chart or PDF) and need to find the Job that generated it. For example, to service an internal or external compliance request.

When viewing files, the revision menu at the top of the screen lets you browse past revisions. For any past revision that was produced by a Job, the revision menu provides a link to the Job. From there, you can access all the details described in the section [Reproduce a result of a Job](/cloud/platform-capabilities/core-concepts/reproducibility/reproducibility-use-cases#reproduce-job-results).

<Tip>
  When including results such as charts or figures in external materials (that is, presentations or documents outside of Domino such as PowerPoint files, Word documents, or Google documents), provide a citation that includes the Domino revision info. Domino lets you link to the specific revision of a file so you can navigate back to the version of the result with one click. You can also include a reference to the Job number that produced the result.
</Tip>

## Reproduce work from a Workspace

Unlike Jobs, [Workspaces](/cloud/platform-capabilities/core-concepts/workspaces) don’t have a completion state. Workspaces are used for interactive development. They start and stop as data scientist modify and save their work.

To integrate this mode of working with the DRE, Domino tracks the sessions and commits performed in a Workspace, even if that goes on for weeks or months.

When working in your Workspace, simply commit your files whenever you want, using the slide-out **File Changes** tab on the left.

In the Workspaces section of your Project, you can select a current or past Workspace, and click to see its **History**.

The **History** section shows the **Sessions** in your Workspace, that is, each segment of time you ran the Workspace between periods where you stopped it.

It also shows you all **Commits**, that is, each time you committed files while working in the Workspace.

Click the row for a specific session to see all the commits performed during that Session.

## Reproduce a deployed model

Whenever you [deploy a Domino endpoint](/cloud/platform-capabilities/features/model-deployment) in Domino, Domino records a new revision of that Model. On the **Versions** tab of your Domino endpoint page, you can see all published versions.

From here, hovering over the version number of any version gives you a **Show details** link that takes you to a page with details about the specific version.

The details include links to the specific revision of the compute Environment and the Project state as it was when this version was deployed.

## Reproduce an App

When you publish new versions of an [App](/cloud/platform-capabilities/features/apps), Domino keeps a record of each one. You can browse versions on the **App Versions** tab of your App page.

Click a version row to view **Details**, including revisions of the compute Environment and Project files (or Git repos, for a Git-backed project).

## Reproduce Launcher results

When you run a Launcher, Domino invokes a Job under the hood. As a result, all results of Launchers are automatically reproducible and auditable. Simply browse for the result as you would for a [Job](/cloud/platform-capabilities/core-concepts/reproducibility/reproducibility-use-cases#reproduce-job-results).

## Next steps

These topics in this section explain how you can make your workflows reproducible in Domino.

[Selectively revert past materials](/cloud/platform-capabilities/core-concepts/reproducibility/revert-materials)\
Selectively restore a part of a Project, such as the package library version, while keeping your latest code and data.

[File syncing and persistence](/cloud/platform-capabilities/core-concepts/reproducibility/file-syncing)\
Domino automatically tracks files in your Project and keeps previous versions in the blob store.

[Remove a file from the DRE: Permanent deletion](/cloud/platform-capabilities/core-concepts/reproducibility/permanent-deletion)\
Purge a file completely and permanently from the blob store.

[Track external data](/cloud/platform-capabilities/core-concepts/reproducibility/track-external-data)\
Materialize external data as a file in Domino to benefit from the automatic tracking that Domino provides.

[Tips for reproducibility in Domino](/cloud/platform-capabilities/core-concepts/reproducibility/reproducibility-tips)\
Tips for maximizing the power of the Domino Reproducibility Engine.


# Selectively revert past materials
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/reproducibility/revert-materials

Revert Project files, Compute Environment revisions, Dataset snapshots, or past Jobs independently while retaining current materials elsewhere.

Domino offers fine-grained controls to manage Project materials. This lets you revert certain Project materials to previous states without affecting others.

For example, you can revert to an older package library version while keeping your latest code and data intact. Similarly, you can run your current code using an older version of your data when needed.

Learn how to isolate and revert the following materials in a Domino Project to a previous version:

* Files

* Compute environments

* Dataset snapshots

* Job runs

## Revert file revisions in a Domino Project

When viewing a file, you can use the **Commit** menu to browse to past versions that you can revert. If you are browsing the root directory of your Project, you’ll revert all Project files.

If you are browsing a specific file, you’ll only revert that file.

## Revert Compute Environment revisions

When viewing Details of a compute Environment, the **Revisions** tab shows you all versions.

After clicking into one version of an Environment, you can make it the Active Revision, or download materials to recreate that environment outside of Domino.

## Revert Dataset snapshots

When viewing a Dataset, the **Snapshot** menu lets you browse past Snapshots.

To revert to an old version, manually create a new snapshot. Replace all files with the contents from the desired old snapshot.

## Re-run a past Job using the latest code or compute Environment

If you select a Job on the Jobs dashboard, you can re-run it.

* **Re-Run with Current Version** executes the exact command from the original Job, with the current state of your Project Files.

* **Re-Run with Original Version** uses the original command and the original Files from the selected Job, with the latest version of your compute Environment.

## Next steps

These topics in this section explain how you can make your workflows reproducible in Domino.

[Reproducibility use cases](/cloud/platform-capabilities/core-concepts/reproducibility/reproducibility-use-cases)\
Learn how to reproduce the results of a Job, Workspace, Model, App, or Launcher.

[File syncing and persistence](/cloud/platform-capabilities/core-concepts/reproducibility/file-syncing)\
Domino automatically tracks files in your Project and keeps previous versions in the blob store.

[Remove a file from the DRE: Permanent deletion](/cloud/platform-capabilities/core-concepts/reproducibility/permanent-deletion)\
Purge a file completely and permanently from the blob store.

[Track external data](/cloud/platform-capabilities/core-concepts/reproducibility/track-external-data)\
Materialize external data as a file in Domino to benefit from the automatic tracking that Domino provides.

[Tips for reproducibility in Domino](/cloud/platform-capabilities/core-concepts/reproducibility/reproducibility-tips)\
Tips for maximizing the power of the Domino Reproducibility Engine.


# Track external data
Source: https://docs.domino.ai/cloud/platform-capabilities/core-concepts/reproducibility/track-external-data

Materialize external data as Domino Datasets, DFS files, or TrainingSets so the Reproducibility Engine can track versions and lineage.

You can use the [Domino Reproducibility Engine](/cloud/platform-capabilities/core-concepts/reproducibility) to track external data in Domino for reproducibility and auditability by persisting the data in Domino when your code runs. If your code connects to an external data source (for example, a database, an API, a network drive, or a [Domino Data Source](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources)), Domino does not know about the data coming from those sources.

## Materialize in-memory data as files for the Reproducibility Engine

To track external data for reproducibility reasons, you can materialize the in-memory data as a file so that the Domino Reproducibility Engine can track it. You can materialize it using any of the following options:

* [Domino Dataset](/cloud/platform-capabilities/core-concepts/data/datasets)

* A file like a CSV or Parquet in the Domino File System (DFS)

* [Domino TrainingSet](/cloud/reference/api/data_api/trainingsets_usecases)

<Tip>
  For best performance, keep your results set under 10 GB if you are persisting files to the Domino file system.
</Tip>

### Use Datasets to track external data

Persist a data frame as a [Domino Dataset](/cloud/platform-capabilities/core-concepts/data/datasets) using the following code:

<Tabs>
  <Tab title="Python">
    To materialize a dataframe as a Dataset in Python, run the following:

    ```python theme={null}
    # Save data to the "quick-start" dataset
    csv_file_path = "/domino/datasets/local/quick-start/data.csv"
    parquet_file_path = "/domino/datasets/local/quick-start/data.parquet"

    # Write the DataFrame to a CSV file
    df.to_csv(csv_file_path, encoding='utf-8', index=False)

    # Write the DataFrame to a binary parquet format
    df.to_parquet(parquet_file_path)
    ```
  </Tab>

  <Tab title="R">
    To materialize a table as a Dataset in R, use the following code:

    ```r theme={null}
    library(arrow, warn.conflicts=FALSE)

    # Save data to the "quick-start" dataset
    csv_file_path <- "/domino/datasets/local/quick-start/data.csv"
    parquet_file_path <- "/domino/datasets/local/quick-start/data.parquet"

    # Write the table to a CSV file
    write.csv(table, file=csv_file_path, fileEncoding="UTF-8")

    # Write the table to a binary parquet format
    write_parquet(table, parquet_file_path)
    ```
  </Tab>
</Tabs>

### Store files in the DFS to track external data

Use the following code to persist a data frame as a CSV or binary file in the [Domino File System](/cloud/platform-capabilities/core-concepts/projects/dfs-projects).

<Tabs>
  <Tab title="Python">
    To materialize a data frame as a file in your working directory, run the following Python code in a notebook code cell, then [sync the file changes](/cloud/platform-capabilities/core-concepts/workspaces/sync-changes-in-a-workspace) to persist the file in the backing repository.

    ```python theme={null}
    ## Save data to the project's default working directory `mnt` as an artifact add an additional sync step
    csv_file_path = "/mnt/data.csv"
    parquet_file_path = "/mnt/data.parquet"

    # Write the DataFrame to a CSV file
    df.to_csv(csv_file_path, encoding='utf-8', index=False)

    # Write the DataFrame to a binary parquet format
    df.to_parquet(parquet_file_path)
    ```
  </Tab>

  <Tab title="R">
    To materialize a table as a file in the root of the working `/mnt/` directory, run the following R code in RStudio console, then [sync the file changes](/cloud/platform-capabilities/core-concepts/workspaces/sync-changes-in-a-workspace) to persist the file in the backing repository:

    ```r theme={null}
    library(arrow, warn.conflicts=FALSE)

    # Save data to the project's default working directory `mnt`
    csv_file_path <- "/mnt/data.csv"
    parquet_file_path <- "/mnt/data.parquet"

    # Write the table to a CSV file
    write.csv(table, file=csv_file_path, fileEncoding="UTF-8")

    # Write the table to a binary parquet format
    write_parquet(table, parquet_file_path)
    ```
  </Tab>
</Tabs>

### Use TrainingSets to track external data

To save a data frame as a TrainingSet, run the following Python code.

```python theme={null}
from domino.training_sets import TrainingSetClient

training_set_version = TrainingSetClient.create_training_set_version(
    training_set_name="my-training-set",
    df=df
)

# To retrieve the data back as Data Frame
# tsv_by_num = TrainingSetClient.get_training_set_version(
#     training_set_name="my-training-set",
#     number=1,
# )
# training_df = tsv_by_num.load_training_pandas()
```

## Data Source audit logs

For external data accessed using a [Domino Data Source connector](/cloud/platform-capabilities/core-concepts/data/data-source-connectors), you can also use the [Data Source audit log](/cloud/admin/operations/audit-logs/monitor-data-source-logs) to track Data Source activity.

Data Source logs provide a way to track user activity, which can be helpful for reproducibility and lineage. However, Domino cannot reproduce the state of your Data Source at the time of execution.

## Next steps

This section explains how to ensure workflow reproducibility in Domino:

[Reproducibility use cases](/cloud/platform-capabilities/core-concepts/reproducibility/reproducibility-use-cases)\
Learn how to reproduce the results of a Job, Workspace, Model, App, or Launcher.

[Selectively revert past materials](/cloud/platform-capabilities/core-concepts/reproducibility/revert-materials)\
Selectively restore a part of a Project, such as the package library version, while keeping your latest code and data.

[File syncing and persistence](/cloud/platform-capabilities/core-concepts/reproducibility/file-syncing)\
Domino tracks project files automatically, storing previous versions in the blob store.

[Remove a file from the DRE: Permanent deletion](/cloud/platform-capabilities/core-concepts/reproducibility/permanent-deletion)\
Purge a file completely and permanently from the blob store.

[Tips for reproducibility in Domino](/cloud/platform-capabilities/core-concepts/reproducibility/reproducibility-tips)\
Tips for maximizing the power of the Domino Reproducibility Engine.


# Agentic AI overview
Source: https://docs.domino.ai/cloud/platform-capabilities/features/agents/agentic-ai-overview

Agentic systems use LLMs, tools, memory, and traces to plan and adapt, so teams can evaluate execution paths, quality, safety, and drift.

An agentic system uses a Large Language Model (LLM) to plan, execute, and adapt across multiple steps. Domino uses traces to instrument, evaluate, deploy, and monitor these systems.

## What is an agent?

An **agent** is a program that uses an LLM to decide what actions to take. Instead of following a fixed code path, an agent operates in a loop. It **plans** a sequence of actions based on an objective, **executes** them using tools and APIs, then **reflects** on the results. If the results don’t meet the objective, it replans and retries. This plan–execute–reflect loop distinguishes an agent from a standard prompt-response interaction.

An agent is typically a framework, such as LangChain, Pydantic AI, OpenAI Agents SDK, LlamaIndex, or custom code. These frameworks wrap orchestration logic around an LLM. The LLM is often deployed separately via an API provider or a self-hosted endpoint. The framework handles the scaffolding, including prompt construction, tool dispatch, memory management, and control flow. The LLM handles reasoning and generation.

## The spectrum of agentic systems

Agentic systems exist on a spectrum:

* **Workflows**: LLMs and tools follow predefined code paths. Examples include prompt chains, retrieval-augmented generation (RAG) pipelines, and evaluator-optimizer loops. The execution graph is fixed at development time. The LLM fills in content but doesn’t choose the path.

* **Autonomous agents**: an LLM dynamically directs its own tool usage and control flow, deciding at each step what to do next. The execution graph is determined at runtime.

Most production systems fall somewhere in between. The building blocks are similar across the spectrum:

| Component | Role |
| - | - |
| **Orchestrator** | Manages the reasoning loop |
| **LLM endpoints** | Generation and decision-making |
| **Tools** | APIs, code execution, external services |
| **Memory** | Conversation history and agent state |
| **Knowledge retrieval** | External information sources the agent looks up |
| **Skills** | Named capabilities or task patterns the agent can perform |
| **Multi-agent coordination** | Specialized agents delegating to each other |

## Why agentic systems introduce new challenges

With a conventional ML model you train on a dataset, evaluate against held-out examples, and deploy a static artifact. Inputs and outputs are well-defined. Agentic systems are different in several important ways:

**Non-deterministic execution paths**\
The same input can produce different sequences of actions depending on model state, tool availability, and intermediate results. You can’t evaluate an agent the way you’d evaluate a classifier.

**Compounding errors**\
When an agent takes ten steps to complete a task, a mistake at step three can cascade. Understanding where things went wrong requires visibility into every step, not just the final output.

**Prompt and configuration sensitivity**\
Small changes to system prompts, tool descriptions, or model selection can dramatically change agent behavior. Teams need a systematic way to compare configurations, not just inspect outputs.

**Production drift**\
Agents in production encounter inputs their developers never anticipated. Without continuous monitoring, degradation goes unnoticed. In multi-agent systems, behavioral drift in one agent can cascade across the entire coordination chain.

**Safety and compliance exposure**\
Agentic systems take real actions: calling APIs, modifying data, and interacting with users. This makes safety evaluation as important as quality evaluation. Safety evaluation includes hallucination detection, toxicity scoring, and policy compliance. Traditional ML monitoring doesn’t cover these dimensions.

## How Domino addresses these challenges

Domino treats agentic systems as first-class objects in the ML lifecycle. The core abstraction is the **trace**. A trace is a structured record of an agent’s full execution path, including every LLM call, tool invocation, and decision point. It also captures token usage, latency, and cost.

Domino builds on [MLflow’s tracing infrastructure](https://mlflow.org/docs/latest/genai/tracing/index.html) and integrates it into the broader ML lifecycle: project structure, governance workflows, deployment infrastructure, and production monitoring. The same trace format captures everything from a simple RAG pipeline to a fully autonomous multi-agent system.

The result is a single thread of observability that runs from your first prototype through to production. Instead of separate tools for debugging, evaluation, and monitoring, every phase produces and consumes the same structured data.

## Next steps

* [Agents](/cloud/platform-capabilities/features/agents): See the full Domino workflow from development through to production, and how a deployed agent reaches its users.

* [Set up LLM access](/cloud/platform-capabilities/features/llms): Connect to an external provider or host your own model.

* [Develop agentic systems](/cloud/platform-capabilities/features/agents/develop): Start instrumenting your agent code.


# Deploy agentic systems
Source: https://docs.domino.ai/cloud/platform-capabilities/features/agents/deploy

Deploy an instrumented agent from Experiment Manager or the Apps view, capture production traces, and connect each deployment to its evaluation lineage.

After experimenting and identifying a promising configuration, you can deploy your agentic system to production.

Agents use the same hosting infrastructure as Domino Apps. Deploying as an agent mainly affects the dashboard and how traces and evaluations are associated.

## Instrument your production application

Your production app code must include the `@add_tracing` decorator on the function that handles user requests. This enables Domino to capture traces from live interactions once the agent is running.

A typical production agent is a web app, such as FastAPI, where each request passes through a function decorated with `@add_tracing`:

```python theme={null}
from domino.agents.tracing import add_tracing
from your_agent import create_agent

@add_tracing(name="single_question_agent", autolog_frameworks=["pydantic_ai"])
async def ask_agent(question):
    agent = create_agent()
    result = await agent.run(question)
    return result

@app.post("/chat")
async def chat(request: ChatMessage):
    result = await ask_agent(request.message)
    return {"response": result.output}
```

The [simple\_domino\_agent](https://github.com/dominodatalab/simple_domino_agent) repository has a complete production app with a chat UI in [chat\_app.py](https://github.com/dominodatalab/simple_domino_agent/blob/main/chat_app.py).

<Note>
  `DominoAgentContext` is required during [development](/cloud/platform-capabilities/features/agents/develop) to group traces into agent versions, but it is optional in production agent code. The `@add_tracing` decorator alone is sufficient to capture production traces.
</Note>

## Deploy an agent

You can deploy an agent in two ways:

<Tabs>
  <Tab title="From Experiment Manager">
    After comparing evaluation runs and identifying a configuration you want to deploy:

    1. In your project, from the left navigation, click **Experiments**.

    2. Choose the experiment with the agent configurations you’re evaluating.

    3. Select the agent version you want to launch.

    4. Click the three dots on the right to launch the run.

    5. Click the **Deploy Agent** button:

       <Frame>
         <img alt="Deploy agent button from Experiment Manager" />
       </Frame>

    6. A deployment wizard opens, similar to the Apps deployment flow. The wizard pre-fills the code commit from the selected agent version and prompts you to specify the entrypoint for your agent.
  </Tab>

  <Tab title="From the Apps view">
    Deploy directly from the Apps view in your project.

    This follows the standard [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions) workflow, with configuration options specific to agentic workloads.
  </Tab>
</Tabs>

Once deployed, production traces flow automatically into the agent’s [monitoring dashboard](/cloud/platform-capabilities/features/agents/monitor). This is the same trace data you saw during development, now captured from real user interactions.

## Next steps

* [Monitor agentic systems](/cloud/platform-capabilities/features/agents/monitor): Track production performance and run evaluations.

* [Experiment tracking and traces](/cloud/platform-capabilities/features/agents/experiment): Compare more configurations before deploying.


# Develop agentic systems
Source: https://docs.domino.ai/cloud/platform-capabilities/features/agents/develop

Instrument agent code with @add_tracing, DominoAgentContext, evaluators, and manual spans, then run it as a Job to create a deployable agent version.

Instrument your agentic system to capture traces during execution. Tracing records every call your system makes, including downstream calls from agents and tools, so you can evaluate performance and debug issues.

To develop an agentic system in Domino:

1. Instrument your evaluation script with tracing, evaluation, and configuration.

2. Write your evaluation function to score agent responses.

3. Run as a Domino Job to create an agent version.

<Note>
  As of Domino 6.2, the agent instrumentation SDK is included as part of the Domino Standard Environment.
</Note>

## Step 1: Instrument your evaluation script

Your evaluation script is the file you’ll run as a Domino Job to test your agent against sample data. It needs three things:

* The `@add_tracing` decorator on the function that calls your agent.

* A `DominoAgentContext` wrapper that groups traces into an agent version with your agent config.

* An evaluator function that scores each response.

Domino auto-instruments all [MLflow-supported frameworks](https://mlflow.org/docs/3.3.2/genai/tracing/integrations/). You can pass your framework to `autolog_frameworks`. Supported values include `langchain`, `pydantic_ai`, `openai`, `anthropic`\*, `llama_index`, and others.

*\*Note that `anthropic` may not be supported fully by auto tracing and could require manual span instrumentation. However, you can easily use anthropic models from other agent frameworks.*

If auto-tracing doesn’t capture all the steps you need, the [Advanced: Manual span instrumentation](#advanced-manual-span-instrumentation) section below has more details.

Here’s the structure of a complete evaluation script:

```python theme={null}
import csv
from domino.agents.tracing import add_tracing
from domino.agents.logging import DominoAgentContext
from your_agent import create_agent
from your_evaluation_library import AgentEvaluator

config_path = "ai_system_config.yaml"

@add_tracing(
    name="single_question_agent",
    autolog_frameworks=["pydantic_ai"],
    evaluator=judge_single_question     # explained in Step 2
)
def process_single_question(data_point):
    agent = create_agent()
    result = agent.run_sync(data_point['question'])
    return {"answer": result.output}

def main():
    with DominoAgentContext(agent_config_path=config_path) as run:
        with open("sample_questions.csv") as f:
            for row in csv.DictReader(f):
                process_single_question(row)
```

* `@add_tracing` captures every call your agent makes, including downstream LLM calls, tool invocations, and retrieval steps, as a structured trace.

* The `evaluator` parameter attaches a scoring function that runs automatically after each invocation. We’ll define that in [Step 2](#step-2-write-your-evaluation-function).

* `DominoAgentContext` groups all traces into a single agent version and logs your `agent_config_path` YAML as parameters in the Experiment Manager. This lets you track exactly which model, prompts, and settings produced each set of results.

### Aggregated metrics (optional)

Pass `aggregated_metrics` to automatically compute summary statistics across all traces in the run:

```python theme={null}
metrics = [("toxicity_score", "mean"), ("relevancy_score", "median")]

with DominoAgentContext(agent_config_path=config_path, aggregated_metrics=metrics) as run:
    ...
```

Available aggregation types: `mean`, `median`, `stdev`, `min`, `max`. If omitted, Domino automatically logs the `mean` of every metric.

## Step 2: Write your evaluation function

The `evaluator` function is called after each invocation of the traced function.

* It receives the completed span, from which you read `.inputs` (what was passed to your function) and `.outputs` (what it returned).

* It returns a dictionary of metric names to numeric values. These become the evaluation scores on that trace.

```python theme={null}
def judge_single_question(span):
    inputs = span.inputs
    output = span.outputs

    question = inputs['data_point']['question']
    answer = output['answer']

    evaluator = AgentEvaluator()
    result = evaluator.evaluate_response(query=question, agent_output=answer)

    return {
        "toxicity_score": result.toxicity_score,
        "relevancy_score": result.relevancy_score,
        "accuracy_score": result.accuracy_score,
        "overall_score": result.overall_score,
    }
```

The returned dictionary is attached to the trace and visible in the Domino UI. You can use any metric names and any number of metrics.

<Note>
  The `AgentEvaluator` class in the example above is your own code. Domino does not prescribe a specific evaluation approach. Your evaluation library can use LLM-as-judge scoring, deterministic heuristics, embedding similarity, external APIs, or any combination. The `simple_domino_agent` repository has an [example evaluation library](https://github.com/dominodatalab/simple_domino_agent/blob/main/evaluation_library.py) to use as a starting point.
</Note>

## Step 3: Run as a Domino Job

Run your evaluation script as a Domino Job. Running as a Job is what keeps lineage intact. Domino ties the agent version to a specific code commit, YAML agent config, all traces, and aggregated evaluation results. Only Job-originated runs can be deployed to production.

<Tip>
  You can run your evaluation script from a workspace to gut-check results while developing. To preserve full lineage and enable deployment from the run, run it as a Job. After pushing your file changes, you can kick off a Job directly from your workspace.
</Tip>

1. Click the **Run job** button:

   <Frame>
     <img alt="The Run job button" />
   </Frame>

2. A window opens where you can specify the script to run and the hardware tier. Select your evaluation script and confirm to launch the Job:

   <Frame>
     <img alt="Configure the Execution" />
   </Frame>

3. Your agent runs against a **dataset of sample queries** that you maintain alongside your code. Each query produces a separate trace with its own evaluation scores. Together, these traces form a complete picture of how this agent configuration performs. As you iterate on prompts, models, or tools, re-run the same dataset to compare results.

   A typical sample dataset is a CSV with questions and categories. Note that some off-topic questions are intentionally included to fully test the evaluator. For example:

   ```
   question_id,question,category
   1,What is the standard deduction for 2023?,deductions
   2,When is the tax filing deadline?,deadlines
   3,What is the distance of Andromeda from the Milky Way?,space
   4,Do I need to pay taxes on unemployment benefits?,income
   5,What is the child tax credit amount?,credits
   ...
   ```

   Your evaluation script loops over this dataset inside `DominoAgentContext`. Each call to the traced function generates one trace with evaluation scores attached:

   ```python theme={null}
   with DominoAgentContext(agent_config_path=config_path) as run:
       with open("sample_questions.csv") as f:
           for row in csv.DictReader(f):
               process_single_question(row)  # one trace per question
   ```

4. When the Job finishes, the Experiment Manager shows this run with all its traces, one per sample question, each with the evaluation scores your judge function returned. Aggregated metrics, such as mean toxicity and mean relevancy, are computed across all traces so you can compare this agent version against others at a glance.

5. After the Job completed, you can:

   * View traces in the **Agent Traces** tab in your project.

   * View the agent version in the **Experiments** tab.

   * Compare runs across different agent configurations in the **[Experiment Manager](/cloud/platform-capabilities/features/agents/experiment)**.

   * **Deploy an agent** directly from any agent version that originated from a Job.

The [simple\_domino\_agent](https://github.com/dominodatalab/simple_domino_agent) repository has a complete working example, including [dev\_eval\_simplest\_agent.py](https://github.com/dominodatalab/simple_domino_agent/blob/main/dev_eval_simplest_agent.py) and [sample\_questions.csv](https://github.com/dominodatalab/simple_domino_agent/blob/main/sample_questions.csv).

## Advanced: Manual span instrumentation

The auto-tracing integrations listed above cover the most common frameworks. In some cases your agent may include steps that auto-tracing doesn’t capture, such as custom tool calls, non-framework LLM API calls, or internal processing functions. For these, you can manually instrument spans using MLflow’s `@mlflow.trace` decorator.

Manual spans work alongside auto-traced spans. Combine `@add_tracing` on your top-level function with `@mlflow.trace` on inner functions to get typed spans with structured metadata:

```python theme={null}
import mlflow
from domino.agents.tracing import add_tracing

@mlflow.trace(span_type="TOOL", name="fetch_weather")
def fetch_weather(city):
    """This function will appear as a TOOL span in the trace."""
    return call_weather_api(city)

@mlflow.trace(span_type="LLM", name="custom_llm_call")
def call_custom_model(prompt):
    """This function will appear as an LLM span in the trace."""
    return requests.post(my_endpoint, json={"prompt": prompt}).json()

@add_tracing(name="my_agent", autolog_frameworks=["openai"])
def run_agent(question):
    weather = fetch_weather("NYC")       # manually traced as TOOL
    result = call_custom_model(question)  # manually traced as LLM
    return result
```

Available span types include `LLM`, `TOOL`, `RETRIEVER`, `CHAIN`, `AGENT`, and `EMBEDDING`. Typed spans enable filtering and specialized rendering in the Trace Explorer.

The [MLflow Manual Tracing documentation](https://mlflow.org/docs/latest/genai/tracing/app-instrumentation/manual-tracing/) has full details on manual tracing.

<Note>
  The Domino SDK is actively adding native support for span types in the `@add_tracing` decorator. Once available, you’ll be able to assign span types directly without dropping down to the MLflow API. Domino updates this section when that capability ships.
</Note>

## Next steps

* [Experiment Manager](/cloud/platform-capabilities/features/agents/experiment): Compare runs, view traces, and deploy to production.

* [Set up LLM access](/cloud/platform-capabilities/features/llms): Configure your LLM endpoints (external providers or Domino-hosted).

* [Python SDK for agents](/cloud/reference/experiment-agent-sdk/python-sdk-agents): A full API reference for `@add_tracing`, `DominoAgentContext`, and trace search.


# Experiment Manager
Source: https://docs.domino.ai/cloud/platform-capabilities/features/agents/experiment

Compare agent versions and traces in Experiment Manager, add manual annotations, export results, and deploy a Job-based configuration.

The Experiment Manager is where you compare agent configurations, review traces, and deploy the best version to production. Each time you [run an evaluation script as a Domino Job](/cloud/platform-capabilities/features/agents/develop), it creates an agent version containing traces and evaluation scores for that configuration.

## How agent versions are created

Agent versions are created when you run your evaluation script as a **Domino Job**, not from a workspace.

Each run captures:

* The **code commit** used for the Job.

* The **YAML agent configuration** (model, prompts, parameters) logged via `DominoAgentContext(agent_config_path=…​)`.

* All **traces** generated during execution, with inline evaluation scores.

* **Aggregated metrics** across all traces in the run.

Each agent version is a complete, versioned snapshot of an agent configuration and its performance. [Develop agentic systems](/cloud/platform-capabilities/features/agents/develop) has details on how to set up the instrumentation and evaluation code.

## View traces in an agent version

View and analyze traces from any agent version:

1. In your project, click **Experiments** in the left navigation.

2. Find and click the experiment you want to review.

3. Open an agent version and select the **Traces** tab.

<Frame>
  <img alt="View traces and add labels" />
</Frame>

When reviewing traces you can:

* See evaluation scores logged by your inline evaluators.

* Add metrics (float values) or labels (string values) by clicking **Add Metric/Label** for manual annotation.

* Click any metric or label cell to open the detailed trace view with the full span tree.

## Compare agent versions

Compare different agent configurations (runs) to see how changes to prompts, models, or parameters affect quality:

1. From the experiment view, select two to four runs and click **Compare**.

2. The comparison view shows aggregated metrics side by side for each run.

3. Open the **Traces comparison** view to compare individual traces across runs. This shows you exactly where one configuration outperforms another on the same inputs.

<Frame>
  <img alt="Compare runs" />
</Frame>

## Deploy from an agent version

Once you’ve identified a winning configuration, you can deploy it to production directly from the Experiment Manager.

Only agent versions that originated from a Job are eligible for deployment. Domino uses the Job’s code commit and agent config to maintain full lineage from experiment to production.

[Deploy agentic systems](/cloud/platform-capabilities/features/agents/deploy) has the step-by-step walkthrough.

## Manual annotations

Subject matter experts can manually annotate traces beyond automated evaluation scores:

* **Metrics**: Add numeric scores (float values) to any trace by clicking **Add Metric/Label**.

* **Labels**: Add string labels such as "approved", "needs review", or "hallucination" to categorize traces.

Manual annotations appear alongside automated evaluation scores in the comparison views.

## Programmatic trace analysis

You can pull trace data programmatically using the MLflow APIs for custom aggregation and analysis. This is useful when you want to do your own analytical work with the trace data beyond the built-in dashboards.

Examples include breaking down token usage, latency, and tool calls by span type, or building custom comparison visualizations in a notebook.

The [rag-agent-demo](https://github.com/dominodatalab/rag-agent-demo) repository has a notebook example that demonstrates pulling span-level metrics and visualizing them.

## Export results

Export data in two formats:

* **CSV**: Open the experiment and click the three dots in the upper right.

* **PDF report**: Select two to four runs to compare and click the three dots in the upper right.

## Next steps

* [Develop agentic systems](/cloud/platform-capabilities/features/agents/develop): Iterate on instrumentation and evaluation.

* [Deploy agentic systems](/cloud/platform-capabilities/features/agents/deploy): Production deployment details and configuration.

* [Monitor agentic systems](/cloud/platform-capabilities/features/agents/monitor): Track production performance.

* [Python SDK for agents](/cloud/reference/experiment-agent-sdk/python-sdk-agents): A full API reference for tracing, evaluation, and trace search.


# Agents
Source: https://docs.domino.ai/cloud/platform-capabilities/features/agents/index

How you build an agentic system in Domino, evaluate it against real data, and ship it to the people who use it.

An agent is a program that uses a Large Language Model (LLM) to decide what actions to take. It plans a sequence of steps toward an objective, executes them using tools and APIs, then reflects on the results and replans if they fall short. It's typically a framework such as LangChain, Pydantic AI, or the OpenAI Agents SDK wrapping orchestration logic around an LLM endpoint, or your own code doing the same.

This section covers how you build one in Domino, evaluate it against real data, and ship it to the people who use it. Domino instruments agents with **traces**, and the same trace data carries through development, evaluation, deployment, and production monitoring.

<Note>
  For AI help writing code inside a Workspace, see [coding assistants](/cloud/platform-capabilities/features/coding-assistants) instead. Those assist you as you work; agents are systems you build and deploy for others to use.
</Note>

## How an agent reaches its users

An agent runs on Domino the way an App does. You deploy it either from the [Experiment Manager](/cloud/platform-capabilities/features/agents/experiment), directly from an evaluated agent version, or from your Project's Apps view using the standard [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions) workflow. Once it's running, it appears alongside Apps under **Deployments** > **Apps & Agents**, where the people you've shared it with can find it and use it in the browser with nothing to install.

Because deployment reuses the App path, agents get the platform's hosting, permissions, and routing without extra setup. Agents also have their own [audit trail](/cloud/platform-capabilities/features/governance/audit-trail) events, covering access granted and revoked, access requests, and changes to an agent's visibility and discoverability.

## How it works in Domino

A trace is a structured record of every LLM call, tool invocation, and decision your agent makes. It captures token usage, latency, and cost. One decorator (`@add_tracing`) instruments your code, and the same trace data flows through every phase:

<Frame>
  <img alt="The agentic development workflow in Domino. What you build: an agent framework such as LangChain, Pydantic AI, or OpenAI feeds your agent logic (prompts, tools, memory, routing), which calls an LLM endpoint that is either an external API or Domino-hosted. The Domino Platform and Agents SDK underpin four phases that iterate: Develop in a Domino Workspace, Evaluate with Domino Jobs, Deploy through the Experiment Manager and the Apps flow, and Monitor in the Agent Dashboard." />

  <img alt="The agentic development workflow in Domino. What you build: an agent framework such as LangChain, Pydantic AI, or OpenAI feeds your agent logic (prompts, tools, memory, routing), which calls an LLM endpoint that is either an external API or Domino-hosted. The Domino Platform and Agents SDK underpin four phases that iterate: Develop in a Domino Workspace, Evaluate with Domino Jobs, Deploy through the Experiment Manager and the Apps flow, and Monitor in the Agent Dashboard." />
</Frame>

<Steps>
  <Step title="Set up LLM access">
    Connect to an external LLM provider (OpenAI, Anthropic, Bedrock, or Azure OpenAI) or to a Domino-hosted endpoint, and store credentials as Domino environment variables. [Set up LLM access](/cloud/platform-capabilities/features/llms) covers both approaches.
  </Step>

  <Step title="Develop in a Workspace">
    Write your agent code using any framework and add `@add_tracing` with inline evaluators to instrument it. Then prepare test data for your agent configuration. See [Develop agentic systems](/cloud/platform-capabilities/features/agents/develop).
  </Step>

  <Step title="Evaluate as a Job">
    [Run your evaluation script as a Domino Job](/cloud/platform-capabilities/features/agents/develop#step-3-run-as-a-domino-job). Each Job creates an agent version in the Experiment Manager, capturing traces with evaluation scores for that configuration. Runs aggregate evaluation results across all traces.
  </Step>

  <Step title="Compare and deploy">
    In the [Experiment Manager](/cloud/platform-capabilities/features/agents/experiment), compare runs and individual traces within runs, then [deploy](/cloud/platform-capabilities/features/agents/deploy) from any agent version that originated from a Job. Domino tracks lineage between development and production, including the commit, the agent configuration, and the performance results.
  </Step>

  <Step title="Monitor in production">
    The same `@add_tracing` instrumentation captures live user interactions as traces in the [Agent Dashboard](/cloud/platform-capabilities/features/agents/monitor). Schedule evaluation Jobs to score production traces continuously and iterate when quality drops.
  </Step>
</Steps>

## Start from a working example

Clone one of these repositories to see the whole workflow running before you instrument your own code:

<Columns>
  <Card title="simple_domino_agent" href="https://github.com/dominodatalab/simple_domino_agent">
    Minimal agent with tool calls, tracing, evaluation, and deployment.
  </Card>

  <Card title="rag-agent-demo" href="https://github.com/dominodatalab/rag-agent-demo">
    Retrieval-augmented generation (RAG) agent with ChromaDB, using the same Domino instrumentation patterns.
  </Card>

  <Card title="simple_agent_api_only" href="https://github.com/dominodatalab/simple_agent_api_only">
    REST API and agent-to-agent (A2A) example, with instructions for pairing it with a Domino-hosted agent registry and orchestrator.
  </Card>
</Columns>

## Related

* [Agentic AI overview](/cloud/platform-capabilities/features/agents/agentic-ai-overview): conceptual background on what makes a system agentic, and where a given design sits on the workflow-to-agent spectrum.
* [Set up LLM access](/cloud/platform-capabilities/features/llms): choose between an external provider and a model you host in Domino.
* [Publish Apps](/cloud/platform-capabilities/features/apps): the deployment path agents share with Apps.


# Monitor agentic systems
Source: https://docs.domino.ai/cloud/platform-capabilities/features/agents/monitor

Monitor deployed agents through performance, usage, and trace views, then schedule Jobs that score production traces without adding request latency.

After deploying your agentic system to production, monitor its performance with real user interactions. Domino automatically collects production traces and provides dashboards to track performance, usage, and quality over time.

## Monitor agent performance

Access monitoring views from your deployed agent’s details page. The details panel next to the tabs shows the deployment status, and these tabs show how your agent performs with real user interactions:

| Tab | What it shows |
| - | - |
| Performance | Evaluation metrics as visualizations alongside production traces. Review metrics over time to spot trends and identify patterns in successful versus problematic interactions. |
| Usage | User invocations and interaction tracking. |
| Monitoring | Deployment configuration and runtime metrics for the running agent. |
| Inputs | Code and data used by the selected agent version. |

The agent dashboard gives you an at-a-glance view of production performance, including evaluation metrics over time and a list of recent traces:

<Frame>
  <img alt="The agent dashboard" />
</Frame>

Click any trace to inspect the full execution path: every LLM call, tool invocation, and decision point, along with inputs, outputs, and evaluation scores:

<Frame>
  <img alt="The trace data view" />
</Frame>

## Evaluate production traces

Production evaluations run out-of-band from your real-time agent app. Rather than evaluating inside the live request path, you write a separate evaluation script that fetches production traces, scores them, and logs the results back. This script runs as a [Domino Job](/cloud/platform-capabilities/core-concepts/jobs), typically on a schedule, so your agent’s latency is never affected by evaluation overhead.

How it works:

* Your agent serves users in real time and collects traces automatically.

* A separate scheduled Job fetches those traces, runs your evaluation logic, and logs scores back to each trace.

This gives you continuous quality monitoring without impacting production performance.

### Find your agent ID and version

To fetch production traces programmatically, you need the **agent ID** and **agent version**.

Find these in the Domino UI after deploying your agent:

<Frame>
  <img alt="Use the Evaluation setup button to find the agent ID and version" />
</Frame>

### The production evaluation script

Use `search_agent_traces()` to fetch traces for a specific agent version. Then evaluate and log results back with `log_evaluation()`:

```python theme={null}
from domino.agents.tracing import search_agent_traces
from domino.agents.logging import log_evaluation

AGENT_ID = "69432f1be3cd202576bec1b1"
VERSION = "69b42b68746e4f13257c492b"

# 1. Fetch traces for this agent version
traces = search_agent_traces(
    agent_id=AGENT_ID,
    agent_version=VERSION,
)

# 2. Evaluate each trace and log results
for trace in traces.data:
    inputs = trace.spans[0].inputs
    outputs = trace.spans[0].outputs

    score = my_evaluate(inputs, outputs)

    # 3. Log evaluation results back to the trace
    log_evaluation(
        trace_id=trace.id,
        name="my_eval_metric",
        value=score,
    )
```

Schedule this script as a recurring Domino Job to continuously monitor quality as your agent handles real user requests.

The [simple\_domino\_agent](https://github.com/dominodatalab/simple_domino_agent) repository has a complete working example in [prod\_eval\_simplest\_agent.py](https://github.com/dominodatalab/simple_domino_agent/blob/main/prod_eval_simplest_agent.py).

## Iterate on configurations

When you identify issues or opportunities for improvement, relaunch your production agent’s configuration into a workspace.

[Track and monitor experiments](/cloud/platform-capabilities/features/development/track-monitor) has details on relaunching runs.

This workflow lets you:

* Reproduce the exact production configuration.

* Debug issues identified in production traces.

* Maintain clear lineage between production agents and their source experiments.

## Next steps

* [Develop agentic systems](/cloud/platform-capabilities/features/agents/develop): Iterate on your agent configuration based on production insights.

* [Experiment Manager](/cloud/platform-capabilities/features/agents/experiment): Test improvements before redeploying.

* [Python SDK for agents](/cloud/reference/experiment-agent-sdk/python-sdk-agents): A full API reference for `search_agent_traces`, `log_evaluation`, and trace data classes.


# Custom environments (advanced)
Source: https://docs.domino.ai/cloud/platform-capabilities/features/coding-assistants/custom-environments

Build a custom Compute Environment with coding agent extensions, CLI tools, authentication, persistent configuration, and MCP servers.

The Domino Standard Environment includes Copilot, Claude Code, and Codex out of the box. You can build a custom Compute Environment if your team needs to standardize on a different coding agent, pin specific versions, or add additional CLI tools.

<Note>
  The DSE can be used without coding assistants. If your organization prefers a standard environment without pre-installed coding assistants, use the `domino-standard-environment-no-assistants` image. The full image URL for the current release is available in the [Domino Compute Environment Catalog](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/8-compute-environment-catalog). Note that the image URL changes with each release, so always refer to the catalog for the latest version.
</Note>

## When to use a custom environment

* Your organization uses a coding assistant not included in the standard environment, for instance, Cody, Cursor-based agents, Continue, or Aider.

* You need to lock agent versions for compliance or reproducibility.

* You want to pre-configure API keys or authentication tokens so users don’t need to authenticate manually.

* You need to bundle additional CLI tools, MCP servers, or plugins alongside your coding agent.

## Install a VS Code extension

<Tabs>
  <Tab title="From VS Code marketplace">
    To add a VS Code extension to your environment, install it directly from the marketplace using `code-server` in your Dockerfile.

    ```dockerfile theme={null}
    # Example: Install the Continue extension for VS Code
    RUN code-server --install-extension Continue.continue
    ```

    <Tip>
      Extensions install to the user’s home directory automatically. With home directory persistence enabled, extensions installed at runtime also survive workspace restarts.
    </Tip>
  </Tab>

  <Tab title="From a .vsix file">
    For extensions that are not available in the open source marketplace, download the `.vsix` file and install it manually:

    ```dockerfile theme={null}
    # Example: Install an extension from a .vsix file
    RUN curl -o /tmp/my-extension.vsix -LSsf \
        "https://example.com/my-extension.vsix" && \
        code-server --install-extension /tmp/my-extension.vsix --force && \
        rm /tmp/my-extension.vsix
    ```
  </Tab>
</Tabs>

## Install a CLI-based agent

CLI-based coding agents work in any workspace IDE. Install them via `pip`, `npm`, or `shell` in your Dockerfile.

<Tabs>
  <Tab title="OpenCode">
    [OpenCode](https://opencode.ai/) is a terminal-native coding agent with a built-in TUI, multi-model support, and MCP integration.

    ```bash theme={null}
    # Install OpenCode CLI
    RUN curl -fsSL https://opencode.ai/install | bash
    ```
  </Tab>

  <Tab title="Aider">
    ```bash theme={null}
    # Install Aider for AI-assisted coding in the terminal
    RUN pip install aider-chat
    ```
  </Tab>
</Tabs>

## Authentication

Most coding agents require an API key or authentication token. In Domino workspaces, each user authenticates their coding agent interactively when they launch it the first time.

Credentials stored in the user’s home directory persist across workspace restarts when home directory persistence is enabled.

<Warning>
  Domino does not currently provide a secure mechanism to inject secrets via environment variables at the environment level. Do not hardcode API keys in your Dockerfile. Users should authenticate interactively on first use, or use their own API key files stored in their persistent home directory.
</Warning>

For agents that support file-based configuration, users can create a config file in their home directory once, and it is reused across workspace restarts:

```bash theme={null}
# Example: Claude Code reads ~/.claude/config.json automatically
# Users create this file once in their persistent home directory
mkdir -p ~/.claude
cat > ~/.claude/config.json << 'EOF'
{
  "apiKey": "sk-ant-...",
  "model": "claude-sonnet-4-20250514"
}
EOF
```

## Persist agent configuration across restarts

Workspace home directory persistence automatically saves files under `/home/ubuntu/`. Most coding agents store their config and history in the home directory, so they survive restarts by default.

If your agent stores configuration outside the home directory, create symlinks in your Dockerfile or startup script:

```dockerfile theme={null}
# Symlink agent config to home directory for persistence
RUN mkdir -p /home/ubuntu/.config/my-agent && \
    ln -sf /home/ubuntu/.config/my-agent /etc/my-agent/config
```

## Add MCP servers

If your coding agent supports Model Context Protocol (MCP) servers, bundle custom MCP servers in your environment to extend agent capabilities:

```bash theme={null}
# Install a custom MCP server
RUN pip install my-custom-mcp-server

# Configure Claude Code to use the MCP server
RUN mkdir -p /home/ubuntu/.claude && \
    cat > /home/ubuntu/.claude/mcp_servers.json << 'EOF'
{
  "my-server": {
    "command": "python",
    "args": ["-m", "my_custom_mcp_server"],
    "env": {}
  }
}
EOF
```

## Full example: Custom environment Dockerfile instructions

To set this up, [create a new environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/3-create-an-environment) using a base environment that has VS Code support, such as the Domino Standard Environment, then add the following to the **Dockerfile Instructions** field:

```dockerfile theme={null}
USER root

# Install Aider CLI
RUN pip install aider-chat

# Install a VS Code extension from the marketplace
RUN code-server --install-extension Continue.continue

USER ubuntu
```

<Note>
  After you’ve build your custom environment, select the environment when you launch a workspace. All coding agents and tools configured in the environment work right away.
</Note>

## Next steps

* [Troubleshoot](/cloud/platform-capabilities/features/coding-assistants/troubleshoot): Get information about known issues and workarounds for coding assistants in Domino workspaces.


# Domino Skills
Source: https://docs.domino.ai/cloud/platform-capabilities/features/coding-assistants/domino-skills

Use Domino Skills to let coding assistants manage Projects, Jobs, data, experiments, deployments, monitoring, environments, and orchestration.

Domino Skills extend coding assistants with platform-specific actions. Inside a Domino workspace, your agent automatically gains access to specific skills, including running jobs, registering models, tracking experiments, deploying apps, and more.

## How skills work

Skills load automatically when a coding assistant starts inside a Domino workspace. The agent detects the Domino project and loads available skills from the [domino-claude-plugin](https://github.com/dominodatalab/domino-claude-plugin).

You don’t need to install or configure skills manually. To use a skill, reference the skill by name when prompting your agent. For example:

```
Use the modeling assistant skill. Analyze the dataset in this project
and train a few models to predict diabetes.
```

The agent activates the skill and begins working with full access to Domino platform resources.

## Available skills

The following skills are included in the Domino Standard Environment:

### Development & modeling

| Skill | Description |
| - | - |
| **modeling-assistant** | AI-assisted model development with MCP servers |
| **experiment-tracking** | Track ML experiments using MLflow-based Experiment Manager |
| **domino-experiment-setup** | Set up MLflow experiment tracking for traditional ML |
| **environments** | Create and customize Domino Compute Environments |
| **workspaces** | Manage interactive development environments |
| **projects** | Work with Domino Projects, Git integration, and collaboration |

### Jobs & orchestration

| Skill | Description |
| - | - |
| **jobs** | Create, run, and manage Domino Jobs |
| **flows** | Orchestrate multi-step ML workflows using Domino Flows (Flyte) |
| **launchers** | Create parameterized web forms for self-service job execution |
| **distributed-computing** | Work with Spark, Ray, and Dask clusters |

### Data

| Skill | Description |
| - | - |
| **datasets** | Work with Domino Datasets for versioned storage |
| **data-connectivity** | Connect to external data sources such as S3, Azure, etc. |
| **domino-data-sdk** | Use domino-data SDK for data access and Feature Store |

### Deployment & monitoring

| Skill | Description |
| - | - |
| **model-endpoints** | Deploy and monitor model API endpoints |
| **model-monitoring** | Monitor deployed models with drift detection and alerting |
| **app-deployment** | Deploy web applications to Domino |

### GenAI & LLM

| Skill | Description |
| - | - |
| **genai-tracing** | Trace and evaluate GenAI applications and agents |
| **domino-trace-setup** | Set up GenAI tracing for agents and LLM applications |

### Applications & SDK

| Skill | Description |
| - | - |
| **domino-app-init** | Initialize new Domino-ready web apps such as Vite+React, Streamlit, Dash, and Flask |
| **domino-ui-design** | Build Domino-styled web apps matching the Domino Design System |
| **python-sdk** | Programmatically interact with Domino using python-domino SDK |
| **domino-debug-proxy** | Debug Domino proxy and routing issues for web applications |

## Use skills in practice

When you ask your coding agent to perform a Domino-specific task, the agent selects and activates the appropriate skill. For example:

* *"Train a model on the diabetes dataset and track the experiment"* activates the **modeling-assistant** and **experiment-tracking** skills.

* *"Deploy this model as an API endpoint"* activates the **model-endpoints** skill.

* *"Create a Streamlit app for this project"* activates the **domino-app-init** skill.

* *"Set up a Spark cluster and run this distributed job"* activates the **distributed-computing** skill.

[Modeling assistant walkthrough](/cloud/platform-capabilities/features/coding-assistants/modeling-assistant) has a complete example of the modeling-assistant skill in action.

## Next steps

* [Get started](/cloud/platform-capabilities/features/coding-assistants/get-started): Launch a workspace and authenticate your coding agent.

* [Modeling assistant walkthrough](/cloud/platform-capabilities/features/coding-assistants/modeling-assistant): An end-to-end demo of data analysis, model training, and comparison using skills.


# Get started with AI coding assistants
Source: https://docs.domino.ai/cloud/platform-capabilities/features/coding-assistants/get-started

Launch a Workspace in the Domino Standard Environment, authenticate Claude Code, GitHub Copilot, or Codex, and start using Domino Skills.

You can launch a Domino workspace and start using a coding assistant in under five minutes without any environment customization or manual extension installs.

## Prerequisites

Before you can start to use coding assistants in your workspace, you need:

* A Domino account with access to create workspaces.

* A subscription to at least one coding agent provider (Anthropic for Claude Code, GitHub for Copilot, or OpenAI for Codex).

## Step 1: Launch a workspace

1. Open your Domino project (or create a new one).

2. Go to **Workspaces** and click **Create New Workspace**.

3. Select the **Domino Standard Environment**, which includes all coding assistant tooling.

4. Choose your IDE, such as VS Code, JupyterLab, RStudio, or Jupyter. In VS Code, all three agents come as extensions. For other IDEs, Claude Code and Codex work as CLI tools.

5. Click **Launch**.

<Frame>
  <img alt="Select the Domino Standard Environment and VS Code when launching a new workspace" />
</Frame>

## Step 2: Authenticate your coding agent

Each coding agent requires a one-time authentication with your provider account. Below are the steps for each agent:

<Tabs>
  <Tab title="Claude Code">
    1. Click the Claude Code icon (the orange asterisk) in the VS Code sidebar to open the **Claude Code** panel.

    2. Choose your authentication method: **Claude.ai Subscription**, **Anthropic Console**, or **Vertex or Bedrock**.

       <Frame>
         <img alt="Click the Claude Code icon in the sidebar, then choose your authentication method." />
       </Frame>

    3. The panel displays a URL. Copy the URL and open the link in your browser to complete the OAuth flow.

       <Frame>
         <img alt="Copy the authentication URL and open the link in your browser to complete the login" />
       </Frame>

    4. After you’ve authenticated in the browser, return to your workspace. Claude Code is now ready to use.

    **Known issues**

    * The clickable link in the pop-up dialog does not work when you authenticate for the first time. Copy the URL manually and paste the link in a new browser tab.

    * If the Claude Code extension panel appears blank, refresh your browser. The panel renders correctly after a page reload.
  </Tab>

  <Tab title="GitHub Copilot">
    1. Open the **Chat** panel in the VS Code sidebar and type a message in the chat input.

       <Frame>
         <img alt="Open the Chat panel and type a message to trigger the sign-in flow" />
       </Frame>

    2. The **Sign in to use AI Features** dialog appears. Choose your authentication method (GitHub, Google, Apple, or GHE.com).

       <Frame>
         <img alt="Choose your authentication method to sign in to Copilot" />
       </Frame>

    3. The dialog displays your one-time device code. Click **Copy & Continue to Google** to open the authentication page in a new window, then paste the code to complete the sign-in.

       <Frame>
         <img alt="Click Copy & Continue to open the authentication page in a new window and paste your device code" />
       </Frame>

    4. Return to your workspace. Copilot is now ready to use.
  </Tab>

  <Tab title="Codex">
    <Warning>
      You must authenticate Codex through the terminal first. The VS Code extension does not work until you have completed the CLI sign-in flow.
    </Warning>

    1. Open a terminal in your workspace by selecting **Terminal** > **New Terminal** from the menu bar.

       <Frame>
         <img alt="Open a new terminal from the menu bar" />
       </Frame>

    2. Run `codex` in the terminal. The Codex CLI displays authentication options. Select **Sign in with Device Code** to authenticate via your browser.

       <Frame>
         <img alt="Run codex in the terminal and choose an authentication method" />
       </Frame>

    3. The CLI displays a URL and a one-time device code. Open the link in your browser, sign in with your OpenAI account, and enter the device code when prompted.

       <Frame>
         <img alt="Open the authentication URL in your browser and enter the one-time device code" />
       </Frame>

    4. After you’ve authenticated, **refresh** your browser to activate the Codex VS Code extension. Open the **Codex** tab in the sidebar panel to start using the extension.

       <Frame>
         <img alt="After authenticating and refreshing, open the Codex tab in the sidebar to use the VS Code extension" />
       </Frame>
  </Tab>
</Tabs>

## Step 3: Start to code

Your coding assistant is now Domino-aware and has access to [Domino Skills](/cloud/platform-capabilities/features/coding-assistants/domino-skills) for running jobs, registering models, tracking experiments, and more.

Try asking your agent to explore the project files, analyze a dataset, or train a model.

The [modeling assistant walkthrough](/cloud/platform-capabilities/features/coding-assistants/modeling-assistant) has a full example.

## Next steps

* [Domino Skills](/cloud/platform-capabilities/features/coding-assistants/domino-skills): See the full list of Domino-specific actions your agent has access to.

* [Modeling assistant walkthrough](/cloud/platform-capabilities/features/coding-assistants/modeling-assistant): An end-to-end demo of data analysis, model training, and comparison.


# Use coding assistants
Source: https://docs.domino.ai/cloud/platform-capabilities/features/coding-assistants/index

Use Claude Code, GitHub Copilot, and Codex in Domino Workspaces, with built-in extensions, CLI tools, Domino Skills, and persistent settings.

Domino Workspaces include built-in support for AI coding assistants as part of the Domino Standard Environment.

You can launch a workspace and start using coding agents right away. No setup or environment customization needed.

<Note>
  Coding assistants help you as you work: they read your Project's code and write alongside you in the IDE. To build an agentic system and deploy it for other people to use, see [Agents](/cloud/platform-capabilities/features/agents).
</Note>

<Frame>
  <img alt="Overview of coding assistants in Domino Workspaces" />
</Frame>

The [video below](https://www.youtube.com/watch?v=0mul8cit-m4) gives an overview on how to use AI coding and modeling assistants in Domino:

<iframe title="YouTube video player" />

## What’s included

The Domino Standard Environment ships with three coding agents, pre-installed and ready to use:

| Agent | VS Code extension | CLI tool | RStudio, JupyterLab, etc. |
| - | - | - | - |
| GitHub Copilot | Pre-installed | - | - |
| Claude Code | Pre-installed | Available | CLI works in any terminal |
| Codex | Pre-installed | Available | CLI works in any terminal |

In VS Code workspaces, all three agents come as pre-installed extensions. Claude Code and Codex also work as CLI tools in any workspace IDE, including the VS Code terminal.

## Domino-aware agents

Coding assistants that run in Domino detect your Domino project automatically. They interact with platform resources, including datasets, jobs, and registered models, through a [skills framework](/cloud/platform-capabilities/features/coding-assistants/domino-skills).

Your coding agent knows about your project files, datasets, and the full Domino platform. There’s no generic sandbox. The agent operates with full context of where the workspace runs and which tools are available.

<Frame>
  <img alt="Claude Code displaying the full list of Domino platform skills available to the agent" />
</Frame>

## Persistence

Workspace home directory persistence preserves your coding agent settings and chat history.

When you restart a workspace, your agents keep their configuration, authentication state, and conversation context from previous sessions.

<Tip>
  Enable home directory persistence in your workspace settings to keep your coding agent configuration across restarts.
</Tip>

## Next steps

The topics in this section show you how to use AI coding assistants in your projects.

* [Get started](/cloud/platform-capabilities/features/coding-assistants/get-started): Launch a workspace and authenticate your coding agent in under five minutes.

* [Domino Skills](/cloud/platform-capabilities/features/coding-assistants/domino-skills): Explore the skills framework and the full list of Domino-specific actions available to your agent.

* [Modeling assistant walkthrough](/cloud/platform-capabilities/features/coding-assistants/modeling-assistant): See a full end-to-end demo of how to train and compare models with the modeling assistant skill.

* [Reuse credentials](/cloud/platform-capabilities/features/coding-assistants/reuse-credentials): Reuse Claude Code, Codex, and GitHub Copilot CLI subscriptions across workspaces without authenticating again each time.

* [Custom environments (advanced)](/cloud/platform-capabilities/features/coding-assistants/custom-environments): Standardize on a different coding agent, pin versions, or add CLI tools using custom environments.

* [Troubleshoot](/cloud/platform-capabilities/features/coding-assistants/troubleshoot): Get information about known issues and workarounds for coding assistants in Domino workspaces.


# Modeling assistant walkthrough
Source: https://docs.domino.ai/cloud/platform-capabilities/features/coding-assistants/modeling-assistant

Use the modeling assistant to analyze data, generate PyTorch training code, run reproducible Jobs, compare experiment metrics, and register a model.

This walkthrough demonstrates the modeling assistant skill from start to finish. Starting from an empty project with a single dataset, the coding agent analyzes the data, generates training scripts, trains multiple models as Domino Jobs, and compares results in Experiment Manager.

<Tip>
  This walkthrough follows the [AI Coding Assistants quick intro video](https://youtu.be/44oG6nmlkVU). You can watch it alongside this guide for the full experience.
</Tip>

## Setup

This demo uses a Domino project with a single dataset for predicting diabetes. The project has no existing code or models.

1. Create a new Domino project or use an existing one with a dataset.

2. Launch a VS Code workspace using the Domino Standard Environment.

3. Open Claude Code and authenticate (as explained in [Get started](/cloud/platform-capabilities/features/coding-assistants/get-started)).

## Step 1: Activate the modeling assistant

In the Claude Code prompt, type:

```console theme={null}
Use the modeling assistant skill. Do a quick analysis of the data set
in the project and then train a few models and validate.
```

* Claude Code activates the **modeling-assistant** skill and begins by detecting the Domino environment.

* The agent identifies the project, locates the dataset, and starts the workflow.

## Step 2: Train models as Domino Jobs

The agent writes a data analysis script, generates a parameterized PyTorch model, creates a training script with MLflow tracking, and submits training as Domino Jobs. All steps run automatically.

<Frame>
  <img alt="The modeling assistant creates model architectures, submits training as Domino Jobs, and monitors progress" />
</Frame>

Key details about the generated training pipeline:

* Training runs as a **reproducible Domino Job**, not inline in the workspace.

* All parameters, including batch size, learning rate, epochs, and architecture, are logged to **Experiment Manager**.

* The agent trains **multiple models** with different configurations for comparison.

## Step 3: Review results in Experiment Manager

Experiment Manager tracks each training job with complete parameter lineage.

Navigate to **Experiments** in your project sidebar to see the runs.

The experiment view displays:

* Training loss and accuracy curves over time.

* Model parameters, including activation, batch size, dropout, epochs, and hidden layers.

* Full lineage back to the training job and code.

<Frame>
  <img alt="Experiment Manager shows training loss, accuracy curves, and all parameters for each model run" />
</Frame>

## Step 4: Compare models

After all models finish training, the agent generates a comparison grid showing both models side by side with key takeaways:

<Frame>
  <img alt="The modeling assistant compares both models and provides recommendations based on accuracy, efficiency, and parameter count" />
</Frame>

In this example, the agent trained two models:

| Metric | `small_32x16` (865 params) | `medium_64x32x16` (3,297 params) |
| - | - | - |
| Test Accuracy | 96.19% | 96.46% |
| Test Precision | 99.81% | 99.97% |
| Test Recall | 94.92% | 95.15% |
| Test F1 | 97.30% | 97.50% |

The agent concluded the small model is the better production choice given the marginal accuracy difference and significantly fewer parameters.

## Step 5: Register a model

From the **Experiments** view, register any trained model to the Domino Model Registry by clicking the **Register** button on the experiment run.

<Tip>
  All training jobs reproduce consistently. Re-run any job from the **Jobs** page to verify or iterate on results.
</Tip>

## Summary

In this walkthrough, the modeling assistant skill handled the complete workflow:

1. Detected the Domino environment and project structure.

2. Explored and analyzed the dataset.

3. Generated a parameterized PyTorch model and training script.

4. Submitted training as reproducible Domino Jobs.

5. Tracked all parameters and metrics in Experiment Manager.

6. Compared multiple model configurations with recommendations.

All lineage stays intact. The Domino platform tracks, reproduces, and audits every step from data analysis to model comparison.

## Next steps

* [Domino Skills](/cloud/platform-capabilities/features/coding-assistants/domino-skills): Explore all available skills beyond the modeling assistant.

* [Custom environments (advanced)](/cloud/platform-capabilities/features/coding-assistants/custom-environments): Add additional agents or customize your setup.


# Reuse your coding agent subscription across workspaces
Source: https://docs.domino.ai/cloud/platform-capabilities/features/coding-assistants/reuse-credentials

Reuse Claude Code, Codex, and GitHub Copilot CLI subscriptions across Workspaces by storing credentials in user environment variables and pre-run scripts.

Authenticate Claude Code, OpenAI Codex, and GitHub Copilot CLI once and reuse
the same credential across every workspace and project, without a browser
login each time a workspace starts.

## How it works

The pattern is the same for all three agents:

1. **Generate the credential once** on your laptop (or any machine with a browser).
2. **Store it as a Domino user environment variable.** In the Domino UI, go to
   **Account** > **Account Settings** > **User environment variables**.
   [User-level variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/4-setup-user-environment-variables)
   follow you across all projects and workspaces (unlike project-level
   variables), and Domino injects them into every run you start.
3. **If the CLI needs a file on disk or first-run state patched**, add a
   snippet to the Compute Environment's pre-run script (**Environments** >
   **Edit Definition** > **Run Setup Scripts**). This runs at workspace
   startup, after the user environment variables are already injected. A user
   or admin adds this to the environment once.

Claude Code and Copilot CLI only need the environment variable. Codex
additionally needs a pre-run script to materialize its credential file.

<Tip>
  These recipes reuse your existing coding agent subscription across workspaces.
  You can also use coding agents alongside the new Domino LLM Gateway 2.0 to
  route agent traffic through Domino's centrally governed model access. LLM
  Gateway 2.0 is currently in early access and distributed as a separately
  deployed app; contact your Domino field representative for details.
</Tip>

<Tabs>
  <Tab title="Claude Code">
    Claude Code uses a single long-lived OAuth token that works across many
    workspaces at once.

    1. Generate the token once on your laptop:

       ```bash theme={null}
       claude setup-token
       ```

       The command walks you through OAuth login and prints a token that starts
       with `sk-ant-oat01-...`. It is **not** saved anywhere, so copy it. The token
       is valid for roughly a year and auto-refreshes.

    2. In **Account Settings** > **User environment variables**, add:

       * Name: `CLAUDE_CODE_OAUTH_TOKEN`
       * Value: `sk-ant-oat01-...`

    3. That's it. The Claude Code settings shipped with the Domino Standard
       Environment handle first-run onboarding, so the token is picked up
       automatically in every new workspace.

    No interactive login is needed. Requires a Pro, Max, Team, or Enterprise plan.

    <Warning>
      If `ANTHROPIC_API_KEY` is also set (check project- and environment-level
      variables too), it takes precedence once approved and bills per-token to your
      API account instead of using your subscription. Remove it.
    </Warning>

    **Parallel workspaces:** Work fine; the same static token works everywhere
    simultaneously.
  </Tab>

  <Tab title="OpenAI Codex">
    Codex has no single "print a token" command. Subscription (ChatGPT) auth lives
    in a JSON credential file, so this recipe needs both a user environment
    variable and a pre-run script to write the file at workspace startup. OpenAI
    treats this as an advanced workflow. API keys are their recommended default
    for CI/CD, but those bill per-token, not against your subscription.

    1. Seed `auth.json` once on your laptop or workstation outside of Domino, or
       inside a throwaway Domino workspace.

       Configure file-backed storage in `~/.codex/config.toml`:

       ```toml theme={null}
       cli_auth_credentials_store = "file"
       ```

       Then log in:

       ```bash theme={null}
       codex login
       ```

       Confirm that `auth.json` was created. Typical locations:

       * macOS / Linux: `~/.codex/auth.json`
       * Windows: `%USERPROFILE%\.codex\auth.json` (usually `C:\Users\<you>\.codex\auth.json`)
       * Custom: wherever `$CODEX_HOME` points, if you set it

       If the file exists, you're good to continue.

    2. In **Account Settings** > **User environment variables**, add:

       * Name: `CODEX_AUTH_JSON`
       * Value: the entire contents of your `auth.json` file, pasted as-is.

    3. In the environment's pre-run script, materialize the file. A user or admin
       adds this once to the Compute Environment:

       ```bash theme={null}
       export CODEX_HOME="${CODEX_HOME:-$HOME/.codex}"
       mkdir -p "$CODEX_HOME" && chmod 700 "$CODEX_HOME"
       # only seed if missing; do NOT overwrite a refreshed file every run
       [ -f "$CODEX_HOME/auth.json" ] || printf '%s' "$CODEX_AUTH_JSON" > "$CODEX_HOME/auth.json"
       chmod 600 "$CODEX_HOME/auth.json"
       ```

    4. Run `codex` normally in the workspace. It refreshes the token bundle in
       place when the session is stale (roughly every 8 days) and writes the new
       tokens back to `auth.json`.

    #### How long the seed lasts

    `auth.json` contains two tokens with different lifetimes:

    * The **access token** (used for API calls) is a JWT valid for roughly a week.
      The CLI proactively refreshes once `last_refresh` is older than roughly 8
      days. Multiple workspaces using the same still-valid access token
      simultaneously is fine.
    * The **refresh token** is rotated on use. When any copy refreshes, the server
      issues a new refresh token and invalidates the old one. Every other copy of
      the credential (other workspaces, the `CODEX_AUTH_JSON` seed value, and even
      your laptop) then holds a dead refresh token and gets a 401 the next time
      *it* tries to refresh.

    Short-lived parallel workspaces (hours to a few days) work as expected. The
    breakage only shows up past the roughly 8-day refresh cycle, and it shows up
    as mysterious 401 responses: a long-running workspace dies after another one refreshed
    first, or a *fresh* workspace fails because the environment variable seed has
    quietly gone stale since something else in the lineage rotated the token.

    Your laptop is part of the same lineage: it keeps itself logged in
    indefinitely by rolling the refresh chain forward, which also means:

    * the seed you pasted into Domino weeks ago is probably already past its refresh, and
    * a long-running workspace that refreshes first can knock your *laptop* out of its session.

    <Warning>
      Treat `auth.json` like a password. Domino stores user environment variables
      encrypted in its secret store, which is where it belongs. Never commit it to
      a repo or project files.
    </Warning>

    **Watch out:**

    * If refresh stops working (401 responses), reseed. Run a fresh `codex login` on your
      laptop and re-paste `auth.json` into the `CODEX_AUTH_JSON` user environment
      variable.
    * The home directory of a workspace is typically ephemeral. A fresh workspace
      gets the seed from `CODEX_AUTH_JSON`, not the last refreshed file, so keep
      the seed reasonably fresh.
    * This does not apply to API-key auth or `chatgptAuthTokens` host
      integrations.

    **Parallel workspaces:** Work fine within an access-token window (roughly a week),
    breaks down across refresh cycles because the refresh token rotates.
  </Tab>

  <Tab title="GitHub Copilot CLI">
    Copilot CLI is the closest analog to Claude Code and the simplest fit for
    Domino: a static fine-grained personal access token (PAT) in a user
    environment variable, no pre-run script needed.

    1. Create a fine-grained personal access token once:

       * Resource owner: your **personal** account (not an organization).
       * Permission: **Copilot Requests** (only available on user-owned
         fine-grained PATs).
       * Repository access: as needed for your work.

       Classic PATs (`ghp_`) are **not** supported. The token looks like
       `github_pat_...`.

    2. In **Account Settings** > **User environment variables**, add:

       * Name: `COPILOT_GITHUB_TOKEN`
       * Value: `github_pat_...`

    The CLI checks these variables in order of precedence: `COPILOT_GITHUB_TOKEN`,
    then `GH_TOKEN`, then `GITHUB_TOKEN`. No browser, no per-workspace login, no
    pre-run script.

    **Watch out:**

    * An environment variable silently overrides any stored OAuth token. If
      `GH_TOKEN` is set for another tool (a common setup in Domino environments
      for Git access), the CLI may use that instead. `COPILOT_GITHUB_TOKEN` wins
      the precedence order, so set it explicitly rather than relying on
      `GH_TOKEN`.
    * Billing: smaller included models don't consume premium requests; more
      capable models draw on premium quota. Use `--model` to pick cheaper models
      for automation.
    * This is the standalone Copilot CLI, not the VS Code extension. The CLI is
      the right tool for headless workspaces.

    **Parallel workspaces:** Work fine; a static PAT works everywhere simultaneously.
  </Tab>
</Tabs>

## Quick comparison

| Agent | User environment variable | Pre-run script needed? | Parallel workspaces |
| - | - | - | - |
| **Claude Code** | `CLAUDE_CODE_OAUTH_TOKEN` from `setup-token` | No | Fine, static token |
| **Codex** | `CODEX_AUTH_JSON` = full `auth.json` contents | Yes, writes `~/.codex/auth.json` from the var | Fine for roughly a week, then refresh rotation bites |
| **Copilot CLI** | `COPILOT_GITHUB_TOKEN` = fine-grained PAT | No | Fine, static PAT |

<Tip>
  Keep any API-key environment variables (`ANTHROPIC_API_KEY`,
  `CODEX_API_KEY`, `OPENAI_API_KEY`) *out* of your user, project, and
  environment variables unless you deliberately want per-token API billing
  instead of subscription usage.
</Tip>

## Use with Domino LLM Gateway 2.0

Coding agents can be used in concert with the new Domino LLM Gateway 2.0 to
centrally govern how they call external Large Language Model providers,
alongside the subscription-reuse pattern above. LLM Gateway 2.0 is currently
in early access and distributed as a separately deployed app. Contact your
Domino field representative for details on how to route your coding agent
traffic through it.

## Next steps

* [Custom environments (advanced)](/cloud/platform-capabilities/features/coding-assistants/custom-environments):
  Bundle CLI tools, MCP servers, or a different coding agent into a Compute
  Environment.

* [User Environment Variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/4-setup-user-environment-variables):
  Reference for creating, updating, and deleting user-level variables.

* [Edit an environment definition](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/5-edit-environment-definition):
  Where pre-run scripts live in the Compute Environment definition.


# Troubleshooting
Source: https://docs.domino.ai/cloud/platform-capabilities/features/coding-assistants/troubleshoot

Resolve coding assistant authentication links, missing Domino Skills, blank Claude Code panels, and unexpected model tier selections.

This page lists known issues and workarounds for coding assistants in Domino workspaces.

**Authentication link does not work**

* **Problem:** When you authenticate a coding agent for the first time, clicking the link provided in the pop-up dialog does not open the authentication page.

* **Workaround:** Manually copy the URL displayed in the terminal output and paste it into your browser to complete the authentication flow.

**Domino Skills not available for non-Claude Code agents**

* **Problem:** Domino Skills might not work out of the box for coding agents other than Claude Code such as Copilot and Codex.

* **Details:** The skills framework is currently optimized for Claude Code through the `domino-claude-plugin`. Copilot and Codex work as general-purpose coding agents in Domino workspaces, but do not have access to Domino-specific skills like job submission, model registration, and experiment tracking.

**Claude Code extension panel appears blank**

* **Problem:** In some cases, the VS Code Claude Code extension panel renders as a blank white or empty area after opening.

* **Workaround:** Refresh your browser page. The panel renders correctly after a page reload.

**Agent defaults to unexpected model tier**

* **Problem:** When using Claude Code, the agent defaults to a model tier such as Opus, which may differ from your preferred setting.

* **Workaround:** Change the active model in the Claude Code panel. The model selector appears at the top of the Claude Code interface. Click to switch to your preferred tier like Sonnet, Opus, etc.


# Develop collaboratively
Source: https://docs.domino.ai/cloud/platform-capabilities/features/collaboration/collaborative-development

Collaborate through shared Projects and Datasets, manage permissions, and import read-only Project files and environment variables for reuse.

Domino streamlines team-based data science work with features designed to enhance collaborative efforts. Learn how to effectively add collaborators, manage permissions, and utilize modular work strategies to optimize your project workflow and drive efficient development."

Three features enable collaborative development: Project sharing, Dataset sharing, and Project imports/exports.

## Project sharing

Domino Projects are the fundamental unit of collaboration. A Project is a bundle of all the materials and the record of activities of all collaborators working on a project.

To add collaborators to your Project, go to the Project’s **Settings** > **Access & Sharing** and add people or organizations in the **Collaborators & permissions** section.

You can also change a collaborator’s role, for example, to restrict them from being able to see (but not modify) the Project’s contents.

Collaborators on a Project all see the same materials within that Project. Contributors can all modify files, run Workspaces, and Jobs, and perform other development tasks.

For more detail see [Collaborator permissions](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#collaborator-permissions).

## Dataset sharing

If you want to create some modularity in who works on different parts of a Project, you can grant access to Datasets independently from Projects. For more details see [Dataset sharing and security](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-datasets).

To summarize: some teams use this approach if they have different people working on preparing data, to be used by another part of the team for modeling or other downstream tasks.

In this scenario, you can grant access to a Dataset to the people working on the data prep, without giving them access to the downstream modeling/analysis Projects.

## Project imports/exports

Domino lets you use a Project as a component that you can link into another Project. Working this way allows you to modularize work and reduce repetition/duplication. For example, if you have a set of Project Artifacts you want to use in several different Projects, instead of copying those files, you can export a Project with the Artifacts and import it into the downstream Projects.

To import a Project, go to the **Other Projects** tab on the **Code** page in a DFS project or the **Artifacts** page in a Git-based project.

When this Project runs a Job or Workspace, the files and environment variables from the imported Project are accessible. Files are read-only.

There are more advanced options, such as controlling which version of a Project you import. For more detail see [Project imports/exports](/cloud/platform-capabilities/core-concepts/projects/manage-projects#export-and-import-project-content).

## Next steps

Learn about other ways Domino streamlines collaboration:

* [Communications and shared context](/cloud/platform-capabilities/features/collaboration/shared-context): Stay informed about what project collaborators are doing, making teamwork more streamlined and creative.

* [Share work and make it accessible](/cloud/platform-capabilities/features/collaboration/share-work): Share work and use tools to make information more accessible to less technical stakeholders.

* [Knowledge management](/cloud/platform-capabilities/core-concepts/knowledge-management) and [Domino search](/cloud/platform-capabilities/features/collaboration/domino-search): Find and reuse work in your Domino Projects.


# Use Domino search
Source: https://docs.domino.ai/cloud/platform-capabilities/features/collaboration/domino-search

Search Projects, files, Jobs, models, NetApp Volumes, and comments, filter results by Tags, and target queries with project fields.

Use Domino’s search feature to locate specific files or pieces of text across your entire deployment. Domino indexes a variety of content in the platform, such as [Projects](/cloud/platform-capabilities/core-concepts/projects), files, [Jobs](/cloud/platform-capabilities/core-concepts/jobs), [Models](/cloud/platform-capabilities/features/models), [Comments](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#add-comments), and [NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes).

Domino limits the search based on the collaboration and privacy settings for projects. For example, if you do not have read access to a project, then that project is never listed in your search results. NetApp Volume results are filtered by NetApp Volume permissions instead: only volumes your account can access appear.

One advanced feature at the intersection of search and permissions is that Projects can be set to be **Searchable**.

If a Project is set to be **Searchable**, then it appears in search results for anyone, but its contents are visible only to explicit collaborators. This is a useful feature if you want to encourage the discovery of work even when that work might be sensitive. With this setting, data scientists could know that relevant past work exists. They would just have to request access to see it. Still, that might prevent wasted effort reinventing the wheel.

<Note>
  if you have git repos attached to your Projects, Domino does not index the contents of those repos.
</Note>

## Use the search feature

1. In the top navigation pane, type your query into the **Search** bar. The search results are listed by source, including project names, files, runs, comments, and more.

When Domino searches files, it returns results found in both filenames and file contents. However, Domino only indexes the latest revisions of files, so the search results do not contain occurrences of your search term from previous versions.

## Search NetApp Volumes

Domino matches [NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes) on the volume name, description, filesystem name, owner username, associated project names, and any [taxonomy Tag](/cloud/platform-capabilities/core-concepts/knowledge-management/tags) labels, tag descriptions, and [Property](/cloud/platform-capabilities/core-concepts/knowledge-management/properties) values on the volume. A result links to the volume's overview page and shows the owner, description, associated projects, and creation date.

Access is enforced per user. NetApp Volume permissions are independent of project permissions, so access to a project does not grant access to the volumes attached to it. A volume appears in your results only when you have at least the **Reader** role on it.

## Filter results by tag

[Tags](/cloud/platform-capabilities/core-concepts/knowledge-management/tags) classify assets under a shared, controlled vocabulary, and search reads them. Narrow a result set to one or more tags from the filter side panel beside your results, or type the tag straight into the search bar.

## Advanced search options

You can use the following commands in your queries to target your search:

* `project.tag=`

* `project.description=`

* `project.name=`

* `project.user=`

* `volume.name=`

* `volume.description=`

* `volume.filesystemName=`

* `volume.ownerUsername=`

* `volume.projectName=`

## Search configuration

Four configuration records control what Domino indexes. Contact Domino Support to read or change them.

| Config key | Description | Example value |
| - | - | - |
| `search.isElasticSearchEnabled` | Whether search is enabled. | `true` |
| `search.documentTypesEnabledForSearch` | Comma-separated list of Domino entities to index for search. | project, file, data\_set, comment, model, run, environment, netapp\_volume |
| `search.fileExtensionsToIndexContent` | Comma-separated list of file extensions to index for search. | txt, out, tsv, rtf, err, log, tex, latex, README, ipynb, py |
| `search.maxIndexableFileSizeInKilobytes` | Maximum size of a file (kilobytes) to index for search. | 2048 |

## Next steps

Learn about other ways Domino streamlines collaboration:

* [Collaborative development](/cloud/platform-capabilities/features/collaboration/collaborative-development): Domino makes it easy for multiple people to work on the same project together.

* [Communications and shared context](/cloud/platform-capabilities/features/collaboration/shared-context): Stay informed about what project collaborators are doing, making teamwork more streamlined and creative.

* [Share work and make it accessible](/cloud/platform-capabilities/features/collaboration/share-work): Share work and use tools to make information more accessible to less technical stakeholders.

* [Knowledge management](/cloud/platform-capabilities/core-concepts/knowledge-management): Classify work with Tags and Properties so teams can find and reuse it.


# Collaborate in Domino
Source: https://docs.domino.ai/cloud/platform-capabilities/features/collaboration/index

Collaborate through shared Projects, Datasets, Apps, search, organizations, and reproducible work that teams can find and reuse.

As data science organizations grow, collaboration becomes critical for speeding up research and development. Many of the highest-performing data science organizations cite collaboration as part of their “secret sauce” that allows them to rapidly innovate and deliver impact from data science work.

## Facets of collaboration

“Collaboration” is a vague term that means different things to different people. Domino enables collaboration in several ways:

* [Collaborative development](/cloud/platform-capabilities/features/collaboration/collaborative-development): Domino makes it easy for multiple people to work on the same project together.

* [Communications and shared context](/cloud/platform-capabilities/features/collaboration/shared-context): Domino helps collaborators stay informed about what each other is doing, making teamwork more streamlined and creative.

* [Make work accessible](/cloud/platform-capabilities/features/collaboration/share-work): Domino makes it easy to share work and provides tools to make information more accessible to less technical stakeholders.

* [Knowledge management](/cloud/platform-capabilities/core-concepts/knowledge-management): Classify work with Tags and Properties so teams can find and reuse it.

* [Use Domino search](/cloud/platform-capabilities/features/collaboration/domino-search): Domino provides a search feature to easily locate specific files or pieces of text across your entire deployment.

* [Sharing and access](/cloud/platform-capabilities/features/collaboration/sharing-and-access): Projects, Datasets, Data Sources, External Data Volumes, Apps, and Compute Environments are each permissioned separately, so access to a Project does not carry access to the data it uses.

* [Manage organizations](/cloud/platform-capabilities/features/collaboration/manage-organizations): Domino makes it easy to give many users permission to your project at once and to add/remove collaborators from multiple projects.

## Domino Projects: The foundational element of collaboration

A Project is Domino’s fundamental unit of collaboration. A Project is a bundle of all the materials and context that collaborators use when working on a project together. This includes the code, files (for example, results or artifacts), data, software packages, and configuration, as well as the historical discussion and record of activity within the Project.

<img alt="Domino Project" />

Projects make collaboration easy through two simple characteristics:

**Single source of truth for materials across collaborators.** All the materials you use in data science projects (the code, compute environments, data, and hardware) are centralized and thus identical regardless of who is using them. As a result, Domino solves one of the common problems of collaboration: different team members have different versions of code, software, or data out of sync with each other. Gone are the problems of passing files back and forth between computers or checking versions of packages across different dev environments.

**Automatic tracking of work avoids “writing over” someone else’s work.** [Domino’s Reproducibility Engine](/cloud/platform-capabilities/core-concepts/reproducibility) ensures that work cannot be lost or permanently overwritten. Moreover, git-backed projects make it easy for collaborators to work on multiple branches concurrently, for easier organization of changes.

## Benefits of collaboration and knowledge management

These modes of collaboration drive several benefits, which often differentiate the highest-performing data science organizations.

* **Innovation and breakthroughs.** Much data science work is like scientific research in that exploratory ideas may be paused and put on the shelf. Being able to find past work about a topic, and being able to “page in” the context of past work, can accelerate the process of sparking new ideas. One data science leader we work with told us that all of the most impactful breakthroughs on his team came from someone picking up old work and looking at it in a new way.

* **Productivity.** When data scientists spend less time on mechanics of collaboration (coordinating file changes with each other, and browsing through multiple systems for prior work), they have more time for higher-value work like research and model development.

* **Less duplicate work and analytics “debt”.** A common problem in large organizations is the proliferation of duplicate work, for example, ten different teams might make a model to do the same thing. This duplication of work becomes a drag on continuous improvement, as different versions diverge in subtle ways, and improvements to one of them aren’t automatically propagated to others. By making it easier to reuse past work, Domino helps simplify and minimize your surface area of data science work, increasing maintainability and the speed of continuous improvement.

* **More flexible talent strategies.** As a centralized platform (accessible through the browser) that doesn’t require any work on local desktops or laptops, Domino makes remote teams possible, unlocking access to talent in more locations. It also makes work from home much easier. Offering these options can help attract top data scientists, who are in high demand and often have their own choice of employers.

* **More flexible project staffing models.** By making it easy to have multiple people working on a Project together, Domino enables teams to decompose work into different components that can be developed in parallel, to complete projects faster. For example, within a Project, some people could be working on feature engineering tasks while others could be working on visualization and reporting outputs, while others could be working on data prep. Domino provides a single interface to share and integrate multiple parts of a project.


# Manage organizations
Source: https://docs.domino.ai/cloud/platform-capabilities/features/collaboration/manage-organizations

Create organizations to manage membership, grant team access to Projects and Compute Environments, and establish shared ownership for production work.

You can create organizations with groups of users so you can give many users permission to your project at once. You can also easily add/remove collaborators from multiple projects.

1. In the top navigation pane, click **Account > Account settings**.

2. On the Account Settings page, click **Organization**:

The user who creates an organization is its owner. Owners manage membership on the Account page at `<Domino URL>/account`, where they can add members, remove members, and make another user an owner of the organization.

## Typical uses for organizations

### Manage Project access

The most common use for organizations in Domino is to manage project access.

Domino sees organizations and users as near-equivalents; the only difference is that an organization contains multiple users. Because of this, organizations simplify project access for groups of users.

For example, you can create an organization with members of a team to invite the team to join a project that you are working. Then, add the organization to the project. This is also useful if you want to remove the team from the project. You can also separate an existing organization into smaller groups, and then un-invite one team while retaining another organization.

You can also use organizations to provide restricted access to your projects, instead of full access. For example, to give a team read-only access to a project, add the organization to your project and set its role to Results Consumer. If there’s a group that should only be able to use Launchers, add the organization and give them Launcher User access.

Organization owners can add and remove people from their organization. Adding users to (or removing them from) organizations rather than individual projects that the organization owns saves time. When you update the organization, Domino updates the users' access to each project for you.

### Organizations for production-ready Projects

You can also use organizations to manage projects that have reached a production grade-level of quality. You can set up organizations that are specifically intended for projects that are ready to be promoted to production, or that have already been promoted. Users working with projects owned by this organization understand that these projects must be carefully documented and kept free of clutter. Changes to the project follow a coordinated review process. Executions in these projects are deliberate, intended to update data or models.

When you use organizations to manage projects promoted to production, the process can unfold in the following ways:

1. A project starts as an individual user’s project. That user invites others to this project, either individually or through an Organization. Over time other users believe this project is useful. Eventually, it is decided that it must be easier to access the project, and that it should be owned by the organization instead of the individual user.

2. A project is initially owned by an organization. This usually happens when the project is deemed to be important from the beginning, and the project can best be built up by a group. When this project is created, all users can access all its resources, and project development proceeds through shared branches or Project copies that contributors review before changes are folded back in. Execution of code is still done in the organization-owned project as a way to check in your analysis or results.

   Some Domino customers do this for models that are not in production to submit their work to be counted.

### Share Compute Environments

Organizations streamline and simplify the practice of sharing [environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments). Project owners who are members of an organization can use all of that organization’s compute environments for their own projects.

For example, when a user is a member of a corporate organization, the compute environments of that organization are available to the user for their own projects. Users can be members of multiple organizations. If a user is a member of two organizations, a corporate organization with production environments and an R\&D organization with experimental environments, that user has access to the compute environments of both organizations, as well as any global environments and environments the user already owns directly.

If this user were to be removed from an organization that they were a member of, even though several of their projects rely on that organization’s environments, the now unavailable environments would be reset to the user’s default environment. Domino sends a notification to affected users.

## Organization permissions for Projects

All members of your organization have `owner-level` access to any projects under the Organization’s account.

If you have an Organization account with username `your_org`; it has members Jane, Chris, and Rae. That means Jane, Chris, and Rae all have `owner-level` access to any the projects in `your_org`, for example, `your_org/quick-start` and `your_org/project1`.

That `owner-level` access does not extend to the organization itself. Members cannot change the organization's membership; its owners can. Some administrative roles can change the membership of any organization: see [organization actions by role](/cloud/admin/identity-and-access/manage-users/roles#organization-actions).

## Organizations as Project collaborators

If you add an organization as a collaborator to one of your projects, all members of the organization are granted [collaborator-level access](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#collaborator-permissions).


# Share work with stakeholders
Source: https://docs.domino.ai/cloud/platform-capabilities/features/collaboration/share-work

Share browser-rendered files, schedule Jobs that deliver reports, create self-service Launchers, and publish interactive Apps for stakeholders.

By hosting work centrally and making it easily accessible through a web browser, Domino simplifies the process of exposing work to less technical colleagues.

Three common formats for sharing work, in order of least interactive to most interactive, are Files, Launchers, and Apps.

## Share files

Share a presentation-mode view of a file with a colleague with the following steps:

1. Select the file you’d like to share, then click the **Copy Shareable Link** button.

2. This copies a link to view the file in a simplified UI for the viewer. This is a great way to share charts, PDFs, or other results with non-technical colleagues.

<Tip>
  Domino can render a wide variety of file types directly in the browser, including Jupyter notebooks, and Markdown files. You can send colleagues a link to an ipynb file in Domino, and they’ll be able to see the cells and rendered outputs, to view it as a report.
</Tip>

### Use scheduled Jobs to generate reports for stakeholders

You can combine Domino’s file rendering and sharing capabilities with scheduled Jobs and notifications, for a flexible way to generate reports.

You can configure scheduled Jobs to notify specific recipients when the Job finishes. You can also customize the files that get included in email notifications sent from scheduled Jobs. So you could have a scheduled Job using knitr to generate HTML or PDFs, or render an ipynb file, and email the attachment and/or a link to the resulting file.

## Launchers

[Launchers](/cloud/platform-capabilities/features/launchers) are a feature for exposing self-serve, templatized analyses to stakeholders. You can expose a simple web form UI on top of a script you’ve written. Your colleagues can specify input values using the web form, then run your script with their desired inputs and get the results.

## Apps

Domino makes it easy to publish and host [Apps](/cloud/platform-capabilities/features/apps), which can have complex, flexible interactivity and visualizations. Apps are a great way to expose rich interactive experiences to non-technical colleagues.

## Next steps

Learn about other ways Domino streamlines collaboration:

* [Collaborative development](/cloud/platform-capabilities/features/collaboration/collaborative-development): Domino makes it easy for multiple people to work on the same project together.

* [Communications and shared context](/cloud/platform-capabilities/features/collaboration/shared-context): Stay informed about what project collaborators are doing, making teamwork more streamlined and creative.

* [Knowledge management](/cloud/platform-capabilities/core-concepts/knowledge-management) and [Domino search](/cloud/platform-capabilities/features/collaboration/domino-search): Find and reuse work in your Domino Projects.


# Communicate in a shared context
Source: https://docs.domino.ai/cloud/platform-capabilities/features/collaboration/shared-context

Keep Project collaborators aligned with notification preferences, searchable comment threads, mentions, and an activity feed of prior work.

Domino helps collaborators stay informed about what others are doing, making teamwork more streamlined and creative.

Three features enable shared context: Notifications, comments, and the activity feed.

## Notifications

Collaborators on a Project can opt to get notifications when certain events in the Project take place. This helps them stay informed as their colleagues are working.

To adjust notification preferences, go to the project **Settings > Access & Sharing > Collaborators & Permissions**. From there, choose the relevant notification preferences for each collaborator.

## Comments

Collaborators on a Project can leave comments on a variety of items in the Project, such as files or Jobs. Comment threads live inside the Project and can be indexed for Search. @mentioning collaborators sends them notifications via email.

Domino’s commenting feature preserves historical context and keeps colleagues abreast of ongoing work. For more see Comments.

To comment on a file, use the text input below the file.

Comments on Jobs appear in the **Comments** tab of the Job details pane on the Jobs Dashboard.

You can also comment on Results inline in the Results tab.

See the following page for step-by-step instructions to [Add comments](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#add-comments).

## Activity feed

A Project’s activity feed makes it easy to see the history of activity on a Project. This is useful if collaborators need to catch up on developments they have missed (for example, if they were out of the office or in another time zone). Or if someone is picking up an old Project or joining a team and needs to educate themselves on the history of the work.

## Next steps

Learn about other ways Domino streamlines collaboration:

* [Collaborative development](/cloud/platform-capabilities/features/collaboration/collaborative-development): Domino makes it easy for multiple people to work on the same project together.

* [Share work and make it accessible](/cloud/platform-capabilities/features/collaboration/share-work): Share work and use tools to make information more accessible to less technical stakeholders.

* [Knowledge management](/cloud/platform-capabilities/core-concepts/knowledge-management) and [Domino search](/cloud/platform-capabilities/features/collaboration/domino-search): Find and reuse work in your Domino Projects.


# Sharing and access
Source: https://docs.domino.ai/cloud/platform-capabilities/features/collaboration/sharing-and-access

Projects, Datasets, Data Sources, External Data Volumes, Apps, and Compute Environments are each permissioned separately, so working with data inside a Project takes access to the Project and to the data.

Access in Domino stacks. To use a Dataset, Data Source, or External Data Volume (EDV) inside a Project, you need access to the Project **and** to that entity. Neither substitutes for the other: a collaborator who was never granted the Dataset cannot read it, and holding the Dataset does not let you into the Projects that use it.

```mermaid placement="top-right" theme={null}
flowchart LR
    you["You"] --> proj["Access to<br/>the Project"]
    you --> data["Access to the Dataset,<br/>Data Source, or EDV"]
    proj --> use["You can use that data<br/>inside that Project"]
    data --> use
```

That is why an invitation is sometimes not enough. When a collaborator lacks permission on a Data Source the Project uses, Domino flags it on the Project's **Data** page and on the **Data** tab in a Workspace. The fix is for the entity's owner to grant access, not for you to invite the person to the Project again.

## Who grants access to each entity

Whoever owns an entity grants access to it, except where only an administrator can. An administrator alone grants an External Data Volume, shares a Data Source with everyone in the deployment, and makes a Compute Environment globally accessible.

| Entity | Who grants access | How access is granted |
| :- | :- | :- |
| Project | The owner sets visibility; the owner or any contributor invites collaborators | A [visibility level](/cloud/platform-capabilities/core-concepts/projects/project-settings#set-project-visibility) of **Searchable**, where the name and description appear in search but the contents stay hidden, or **Private**, where only collaborators can find it at all. On top of that, named collaborators and organizations, each holding a [collaborator role](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#collaborator-permissions) |
| Dataset | The Dataset owner | An [Owner, Editor, or Reader role](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-datasets#dataset-roles) granted to users and organizations |
| Data Source | The Data Source owner, or an administrator | [Specific users or organizations](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-data-sources#share-a-data-source); an administrator can instead share it with everyone in the deployment |
| External Data Volume | An administrator only | The [Volume Access setting](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-edv#edv-security-settings): everyone in the deployment, or specific users and organizations. A separate **Mount as read-only** setting decides whether you can write to it |
| App | A Project contributor or owner | Edit rights are inherited from the parent Project. [View rights](/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security#access-controls-and-permissions) are set on the App itself: **Restricted** to named viewers and Project collaborators, optionally **Globally discoverable** so other users can request access, or **Anyone in Domino** |
| Compute Environment | Whoever creates the environment; an administrator for **Globally Accessible** ones | A [visibility level](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/3-create-an-environment) of **Private**, **Available to an Organization**, or **Globally Accessible** |

Only the Project owner can change a Project's visibility.

<Note>
  Domino does not control access to an external Git repository attached to a Project. Manage that in your Git host.
</Note>

All of this sits on top of the global role an administrator assigns you. **Practitioner**, the role every user starts with, is what lets you create Projects, run Jobs, start Workspaces, and publish Apps in the first place. See [Administer identity and access control](/cloud/admin/security-and-compliance/auth-identity-access-control) for the administrator's side: identity providers, role assignment, service accounts, and the deployment settings that change what any of this allows.

## Share with an organization

An organization is a named group of users that Domino permissions as a single unit. Grant access to an organization rather than to named individuals whenever a team, not a person, is what needs it. You grant it once, and membership does the rest.

* Add an organization as a Project collaborator and every member holds that collaborator role.
* Change the membership and Domino updates each member's Project access for you.
* Every member holds owner-level access to the Projects owned by the organization's own account, so a Project owned by an organization is open to all of its members.
* A Project owner who belongs to an organization can use that organization's Compute Environments. Losing membership resets the affected Projects to your default environment, and Domino notifies you.

An organization is owned by the user who created it. That owner adds and removes members and can make another user an owner; other members cannot change the membership, and platform administrators can change any organization's. [Manage organizations](/cloud/platform-capabilities/features/collaboration/manage-organizations) covers creating one and the patterns it supports, including holding production-grade Projects under an organization rather than a person.

## Authenticate scripts and pipelines

A script, notebook, or pipeline authenticates with its own credential instead of a browser session, and it reaches exactly what that credential's identity already reaches. A credential never widens access.

| | Personal Access Token | Domino Service Account |
| :- | :- | :- |
| Who creates it | You, from your own account settings | An administrator, through the Domino API |
| Whose identity it uses | Yours | A technical user that cannot sign in to the web interface or the CLI |
| Which roles it carries | The roles you hold. Changing your roles revokes the token instead of updating it | The roles the account held when the token was issued; later role changes do not reach tokens already issued |
| How long it lasts | However long you choose when you create it | Four months by default |
| Use it for | Your own scripts, notebooks, and Jobs | Continuous integration and continuous delivery (CI/CD) pipelines, monitoring and automation tools, and system-to-system integrations |

Create a [Personal Access Token](/cloud/reference/api/domino-api-authentication#create-a-personal-access-token) under **Account**, then **Account settings**, then **Personal Access Token**. Pick an expiration and copy the value at that moment: Domino never shows it again, and no administrator can retrieve it for you.

<Warning>
  Changing your roles revokes every Personal Access Token you hold. If an administrator adjusts them, your scripts, notebooks, and scheduled Jobs stop authenticating until you create replacement tokens.
</Warning>

<Note>
  **Best practice:** keep Personal Access Tokens for your own work and ask an administrator for a Domino Service Account for anything shared. A service account's tokens belong to no individual, so deactivating a departing employee never breaks a pipeline.
</Note>

## Related

* [Manage organizations](/cloud/platform-capabilities/features/collaboration/manage-organizations): create an organization and use it to hold Projects, access, and Compute Environments.
* [Collaborate on Projects](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects): invite collaborators and see what each Project role can do.
* [Share data securely](/cloud/platform-capabilities/core-concepts/data/sharing-and-security): the full permission model for Datasets, Data Sources, and External Data Volumes.
* [App security](/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security): view and edit rights on a published App, and the identity headers it receives.
* [Domino Service Accounts](/cloud/admin/identity-and-access/manage-users/service-accounts): what a service account can do, and what to ask an administrator for.
* [Domino API authentication](/cloud/reference/api/domino-api-authentication): create, use, and revoke a Personal Access Token.
* [Protecting tokens and API keys](/cloud/admin/security-and-compliance/protecting-tokens-api-keys): store and rotate credentials without exposing them in code.


# Configure Dask prerequisites
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/dask/configure-prerequisites

Configure Dask cluster and execution Compute Environments with compatible images, supported-cluster settings, dependencies, and Workspace tools.

Before you can start using on-demand Dask clusters on Domino, you must ensure that this functionality is enabled and properly configured on your deployment.

<Note>
  Domino on-demand Dask functionality is available starting with Domino 4.6.
</Note>

## Create a base Dask cluster environment

When using on-demand Dask in Domino you have two separate environments: one for the Dask cluster (base or worker environment) and one for the workspace/job execution (compute environment).

You can get the latest released Dask images from [compute Environment catalog](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/8-compute-environment-catalog).

To create a new base Dask cluster environment, follow the general [Environment management](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments) instructions with the following `environment_attributes`:

* **Base Image**

  Select **Custom Image** and enter an image URI that points to a deployable Dask image.

  From the options published at [compute Environment catalog](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/8-compute-environment-catalog), select the Dask cluster image available in the [cluster-environment-images repository](https://quay.io/repository/domino/dask-cluster-environment?tab=tags). Domino’s repository contains the latest Dask images curated for Domino.

  The available images include the full set of Dask components and common dependencies like Pandas and NumPy.

* **Supported Clusters**

  Select the **Domino managed Dask** option (required). This ensures that the environment is available for use when creating Dask clusters from workspaces and jobs.

* **Visibility**

  Set this attribute the same way you would for any other [compute environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments).

* **Dockerfile Instructions**

  Leave blank to use the base image provided by the Dask community.

  You can modify this section to include additional packages that might be necessary for your workloads and must be available on the Dask cluster nodes.

  See [Manage dependencies](/cloud/platform-capabilities/features/compute-clusters-hpc/dask/manage-dependencies) to learn more.

* **Pluggable Notebooks / Workspace Sessions**

  Leave this section blank as the Dask base environments are not intended to include notebook configuration.

## Prepare your Dask execution compute environment

In addition to the base Dask cluster environment, you must also configure the Dask compute environments for workspaces and/or jobs that connect to your cluster.

Domino recommends that you use the Dask base image to create a compatible workspace. See [compute Environment catalog](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/8-compute-environment-catalog).

### Customize this Workspace compute environment:

Use the image mentioned previously and add **Pluggable Workspace Tools**.

```yaml theme={null}
jupyter:
  title: "Jupyter (Python, R, Julia)"
  iconUrl: "/assets/images/workspace-logos/Jupyter.svg"
  start: [ "/opt/domino/workspaces/jupyter/start" ]
  supportedFileExtensions: [ ".ipynb" ]
  httpProxy:
    port: 8888
    rewrite: false
    internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
    requireSubdomain: false
jupyterlab:
  title: "JupyterLab"
  iconUrl: "/assets/images/workspace-logos/jupyterlab.svg"
  start: [  "/opt/domino/workspaces/jupyterlab/start" ]
  httpProxy:
    internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
    port: 8888
    rewrite: false
    requireSubdomain: false
vscode:
  title: "vscode"
  iconUrl: "/assets/images/workspace-logos/vscode.svg"
  start: [ "/opt/domino/workspaces/vscode/start" ]
  httpProxy:
    port: 8888
    requireSubdomain: false
rstudio:
  title: "RStudio"
  iconUrl: "/assets/images/workspace-logos/Rstudio.svg"
  start: [ "/opt/domino/workspaces/rstudio/start" ]
  httpProxy:
    port: 8888
    requireSubdomain: false
```


# Use on-demand Dask
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/dask/index

Provision on-demand Dask clusters for Workspaces and Jobs, then scale Python data processing, distributed training, and custom computations.

[Dask](https://dask.org) is a distributed computing library that tightly integrates with the Python ecosystem and allows for multi-core and distributed parallel execution on larger-than-memory datasets. Dask makes it simple to scale up a single machine Python workload to a multi-machine cluster with little or no changes even if the application was never developed with Dask in mind initially.

Dask offers the following:

* **Low-level scheduling and execution APIs**: Dask provides a set of APIs and facilities for scheduling and parallel execution of task graphs. This execution engine powers the high-level collections mentioned below but can also be used to develop and execute custom, user-defined distributed workloads. These low-level capabilities are an alternative to direct use of `threading` or `multiprocessing` Python libraries or other task scheduling systems like `Luigi` or `IPython parallel`.

  For additional information, see [Dask delayed](https://docs.dask.org/en/latest/delayed.html) and [Dask futures](https://docs.dask.org/en/latest/futures.html).

* **High-level distributed libraries**: Dask provides distributed equivalents for popular Python collection libraries such as NumPy arrays, Python lists, and Pandas data frames. The Dask equivalents provide API-level compatibility and can be used as drop-in replacement when one needs to work with large datasets. Additionally, Dask provides similar compatibility with scikit-learn and integration with other popular model frameworks to enable scalable training and prediction on large models and datasets.

  For additional information, see [Dask arrays](https://docs.dask.org/en/latest/array.html), [Dask dataframes](https://docs.dask.org/en/latest/dataframe.html), and [Dask ML](https://ml.dask.org/).

## Orchestrate Dask on Domino

Domino offers the ability to dynamically provision and orchestrate a Dask cluster directly on the infrastructure backing the Domino instance. This allows Domino users to get quick access to Dask without having to rely on their IT team.

When you start a [Domino workspace](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace) for interactive work or a [Domino job](/cloud/platform-capabilities/core-concepts/jobs) for batch processing, Domino creates, manages, and provides a containerized Dask cluster for execution.

See Domino’s [quick-start-dask](https://github.com/dominodatalab/domino-quickstart-dask) project.

## Suitable use cases

Domino on-demand Dask clusters are suitable for the following workloads:

**Working with large datasets**\
Dask excels at scaling up Python data analysis or transformation code where the data that needs to be processed exceeds the resources that can be provided by a single machine. With a compatible API for commonly used Python libraries, Dask is a suitable tool for Python-first data scientists who have R\&D Python code for data cleaning, manipulation, and advanced analytics which needs to be scaled to a much larger production dataset. This can be done with minimal modification and without having to switch over to a different ecosystem like Spark.

**Distributed training**\
[Dask](https://www.dask.org/) provides a simple drop-in replacement to train parallelizable scikit-learn models at scale. It’s possible to run Dask in single-node and distributed modes to enable parallelization and distribution separately and in conjunction. This approach is an excellent fit for moderate memory footprint models that are CPU/GPU-bound with many individual operations and can be parallelized beyond the limits of a single machine. For larger memory-bound workloads, there are Dask-specific ML libraries (for example, Parallel Meta-estimators and Incremental Hyperparameter Optimizers) that use algorithms; they are specifically optimized to work with the Dask scalable NumPy and DataFrame equivalents.

**Custom distributed computations**\
If the available Dask machine learning algorithms or large data representations are insufficient, use the low-level Dask scheduling APIs to build custom algorithms that can benefit from parallelism. Developers control the business logic, while Dask handles task dependencies, network communication, workload resilience, diagnostics, and support functions.

## Next steps

* Find out more about the [Validated Dask version](/cloud/platform-capabilities/features/compute-clusters-hpc/dask/validated-dask-version).

* Learn how to enable and configure the functionality on your deployment in [Configure Dask prerequisites](/cloud/platform-capabilities/features/compute-clusters-hpc/dask/configure-prerequisites).

* Learn how to [create an on-demand Dask cluster](/cloud/platform-capabilities/features/compute-clusters-hpc/dask/work-with-your-cluster) with the desired cluster settings attached to a Workspace or Job.

* Find out how you can [manage Dask dependencies](/cloud/platform-capabilities/features/compute-clusters-hpc/dask/manage-dependencies).

* Learn how to [Access data with Dask](/cloud/platform-capabilities/features/compute-clusters-hpc/dask/work-with-data).


# Manage Dask dependencies
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/dask/manage-dependencies

Add workload-specific packages and pinned versions to both Dask base and execution Compute Environments through Docker instructions.

While Dask base images come with the full set of Dask collection components and common dependencies like Pandas and NumPy, you might still have to modify your environments if you need additional packages or when you need a specific version of a given package.

Domino allows you to easily package and manage dependencies as part of your [Dask-enabled compute environments](/cloud/platform-capabilities/features/compute-clusters-hpc/dask/configure-prerequisites#create-a-base-dask-cluster-environment). This approach creates the flexibility to manage dependencies for individual projects or workloads without having to deal with the complexity of a shared cluster.

To add a new dependency, you need to add the appropriate statements in the **Docker instructions** section of the relevant Dask base and Dask execution compute environments.

For example, if you wanted to add a particular version of Dask ML, you might include the following.

```dockerfile theme={null}
### Optionally specify version if desired
RUN pip install dask-ml==1.9.0
```

There are several optional dependencies that might be required for subsets of Dask functionality. You can find more information in the [Dask documentation](https://docs.dask.org/en/latest/install.html#optional-dependencies).


# Validated Dask version
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/dask/validated-dask-version

Check the Dask version that Domino validates for on-demand clusters and understand the support boundary for other Dask releases.

For each minor release (starting with Domino 4.6), Domino performs a set of validation tests against the **most recent released version** of Dask. The purpose of the tests is to cover the integration surface between the Domino platform and the on-demand Dask clusters that the platform orchestrates. The testing is not intended to be a comprehensive test of Dask itself since Domino does not modify that software.

<Note>
  The validated and supported version of Dask is 2022.1.0.

  Other Dask versions are likely to function properly, but Domino cannot offer technical support.
</Note>


# Access data with Dask
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/dask/work-with-data

Access Domino Datasets, Amazon S3, and other external stores from ephemeral Dask clusters, including credential profiles and storage options.

When using a Domino on-demand Dask cluster, any data that is created or modified as part of the interaction needs to go into an external data store.

<Warning>
  On-demand clusters in Domino are ephemeral. Any data that is stored on cluster local storage and not externally is lost upon termination of the workload and the cluster.
</Warning>

## Using Domino Datasets

When you create a Dask cluster attached to a Domino workspace or job, any [Domino dataset](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets) accessible from the workspace or job is also accessible from all components of the cluster under the same dataset mount path. You can then access the files from your code using the same path regardless of whether your code runs on your workspace or job container or in a Dask task on the cluster.

For example, to read a file you would use the following.

```python theme={null}
import dask.dataframe as dd

df = dd.read_parquet("/mnt/data/my_dataset/large_dataset.parquet")
```

## Using S3

To access Amazon S3 (or S3 compatible object store) data with Dask, you can use any of the libraries you already use (for example, `boto3`, `s3fs`) to pull down files from S3.

For structured data, you can also read it directly into Dask dataframes of bags. You would need to specify the `s3://` as the protocol. The following is a basic example.

```python theme={null}
import dask.dataframe as dd

df = dd.read_parquet("s3://bucket/path/data-*.parquet")
df = dd.read_csv("s3://bucket/path/data-*.csv")
```

<Note>
  Dask uses `s3fs` and the underlying `boto3` for S3 access, so you need to make sure that the optional `s3fs` package is installed on your base Dask environment and your execution environment when using Dask to access S3 data.
</Note>

Additional parameters (for example, auth keys) can be passed through the `storage_options`. For full documentation of the S3 specific options (including loading data from S3 compatible services), refer to the relevant section of the [Dask documentation](https://docs.dask.org/en/latest/remote-data-services.html#amazon-s3).

### AWS credential propagation

When AWS credential propagation is enabled for your deployment, temporary AWS credentials corresponding to the roles enabled for you in your company identity provider are automatically available on all Dask workers and your execution.

The credentials are automatically refreshed and available under a profile name corresponding to each role in an AWS credential file. The location of the file is stored in the `AWS_SHARED_CREDENTIALS_FILE` environment variable, which puts in the proper search path for `s3fs` and `boto3`.

You can specify the name of the profile that corresponds to the role that you would want to use for authentication. You can do the following:

```python theme={null}
import dask.dataframe as dd

df = dd.read_parquet("s3://bucket/path/data-*.parquet",
    storage_options={
       "profile_name"="my-role-profile",
    })
```

## Using other data stores

Similar to S3, Dask can load data from Microsoft Azure Storage, Google Cloud Storage, HDFS, HTTP, NFS, and your local file system.

Detailed documentation describing the protocol to use, the required packages, and the available `storage_options` can be found in the [Remote data](https://docs.dask.org/en/latest/remote-data-services.html#remote-data) section of the Dask documentation.


# Work with clusters
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/dask/work-with-your-cluster

Create Dask clusters for Workspaces and Jobs, configure worker autoscaling and Hardware Tiers, connect clients, and use the Dask web UI.

## Create a cluster with Workspaces

<span />

To create an on-demand Dask cluster attached to a Domino Workspace, click **New Workspace** from the **Workspaces** menu. On the Launch New Workspace dialog select **Compute Cluster**. Specify the desired [cluster settings](#understand-cluster-settings) and launch your workspace. Once the workspace is up, it has access to the Dask cluster you configured.

## Create a cluster with Jobs

Similarly to workspaces, to create an on-demand Dask cluster attached to a Domino job, click **Run** from the **Jobs** menu. On the Start a Job dialog, select **Compute Cluster**. Specify the desired [cluster settings](#understand-cluster-settings) and launch your job. The job has access to the Dask cluster you configured.

As your command, you can use any Python script that interacts with your Dask cluster.

## Understand cluster settings

Domino makes it simple to specify key settings when creating a Dask cluster.

* **Min workers**

  The number of Dask node workers that make up the Dask cluster when it starts. If **Auto-scale workers** is not enabled, this is always the size of the cluster. The combined capacity of the workers is available for your workloads.

* **Max workers**

  The maximum number of Dask node workers that the cluster can reach when **Auto-scale workers** is enabled. See [cluster autoscaling](#cluster-auto-scaling) for more details.

* **Cluster size: Limit**

  The maximum number of workers that you can make available to your cluster is limited by the number of per-user executions that your Domino administrator has configured for your deployment or by the maximum simultaneous executions of the underlying hardware tier used for workers.

  In addition to the number of Dask node workers, you need 1 slot for your cluster master and 1 slot for your workspace or job.

* **Worker hardware tier**

  The amount of compute resources (CPU, GPU, and memory) made available to each Dask node worker.

* **Scheduler hardware tier**

  Same mechanics as the worker hardware tier, but applied to the resources that are available for your Dask cluster scheduler node.

  As the name suggests, the Dask scheduler is responsible for determining the worker on which a given task needs to execute. A serialized version of the task is held in the scheduler memory until it is possible to assign the task. This is not necessarily the same amount of memory needed to actually execute the task, but limiting the scheduler memory may prevent work from being scheduled for executions with a complex task graph.

  From a CPU perspective, the needs of the scheduler are likely much lower than your workers.

  By default, any hardware tier is available when selecting resources for your executor and scheduler. Domino administrators can optionally configure compute cluster [dedicated hardware tiers](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/advanced-hardware-tier-settings#restrict-to-a-compute-cluster).

* **Cluster compute environment**

  Designates your compute environment for the Dask cluster.

* **Dedicated local storage per executor**

  The amount of dedicated storage in Gigabytes (2^30 bytes) available to each Dask worker.

  The storage is automatically mounted to `/tmp`.

  The storage is automatically provisioned when the cluster is created and de-provisioned when it is shut down.

  <Warning>
    The local storage per worker should not be used for storing any data that must be available after the cluster is shut down.
  </Warning>

## Cluster auto-scaling

Cluster auto-scaling allows you to start with a small cluster, which then automatically scales up and down in response to the resource consumption of your workload. This approach utilizes resources more efficiently for bursty workloads.

By default, the cluster size increases when the average CPU utilization of your workload reaches 80%. A Domino administrator can further refine the auto-scaling settings by including memory utilization or changing the desired scaling thresholds.

Scale down happens if resource utilization is low for a period of at least 5 minutes. Note that depending on the workload that you are executing, scale down may terminate cluster nodes that contain intermediate results which would need to be recomputed.

## Connect to a cluster

When provisioning your on-demand Dask cluster, Domino sets up environment variables that hold the information needed to easily connect to your cluster.

The following snippet can be used to connect:

```python theme={null}
from dask.distributed import Client
import os
...

service_host = os.environ["DASK_SCHEDULER_SERVICE_HOST"]
service_port = os.environ["DASK_SCHEDULER_SERVICE_PORT"]
client = Client(f"{service_host}:{service_port}")

# you should now be connected to the cluster
# Dashboard link from the client object is clickable but will not route in Domino
# Use the embedded Dask Web UI tab instead
```

<Note>
  * You can optionally wait until the desired number of workers is available with `client.wait_for_workers(<desired num workers>)`.

  * If you must close your client connection, use `client.close()`. Be careful not to use `client.shutdown()`, since that terminates the cluster workers and you cannot continue working with the cluster unless you restart your execution.
</Note>

Printing the client properties in a notebook gives you a Dask dashboard link. While that link is clickable, it does not return an appropriately formatted dashboard link that works from Domino. Use the embedded Dask web UI as described below instead.

## Access the Dask web UI

Dask provides a built-in diagnostics dashboard with access to metrics, charts, and other features that helps you understand the components of the Dask cluster and the execution of your workloads.

The dashboard is extremely useful to diagnose performance issues and help improve your understanding of what is happening across your cluster.

Domino makes the Dask web UI available for active on-demand clusters attached to both workspaces and jobs.

<Note>
  In a hybrid Domino deployment, if the [data plane](/cloud/platform-capabilities/core-concepts/data-plane) is not configured for workspaces, then the user interface link is disabled. An admin can [Enable a data plane for workspaces](/cloud/admin/platform-configuration/manage-data-planes#enable-a-data-plane-for-workspaces-and-datasets).
</Note>

For a more in-depth introduction to the Dask dashboard, refer to the official [Dask documentation](https://docs.dask.org/en/latest/diagnostics-distributed.html#dashboard).

### Dask web UI from Workspaces

The Dask web UI is available from a dedicated tab in your workspace.

### Dask web UI from Jobs

The Dask web UI is also available for running jobs from the **Details** tab.

## Cluster lifecycle

On workspace or job startup, a Domino on-demand Dask cluster with the desired [cluster settings](#understand-cluster-settings) is automatically provisioned and attached to the workspace or job as soon as the cluster becomes available.

On workspace or job termination, the on-demand Dask cluster and all associated resources are automatically terminated and de-provisioned. This includes any compute resources and storage allocated for the cluster.

## Cluster network security

The on-demand Dask clusters created by Domino are not meant for sharing between multiple users. Each cluster is associated with a given workspace or a job instance. Access to the cluster and the Dask web UI is restricted only to users who can access the workspace or the job attached to it. This restriction is enforced at the networking level and the cluster is only reachable from the execution that provisioned it.


# On-demand distributed computing
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/index

Use Domino Cluster Environments to run Spark, Dask, Ray, MPI, and Slurm workloads, including remote executions through Domino Nexus.

Use Domino Cluster Environments to scale out compute-intensive workloads. Domino Cluster Environments are similar to Domino Standard Environments (DSEs), but with added libraries for specific cluster types. Just like DSEs, they include the same tools and packages, and you can use them to spin up Workspaces or Jobs. However, Domino Cluster Environments are mutually exclusive from non-cluster environments - you cannot use a Domino Cluster Environment in a non-cluster configuration.

The following clusters can be accessed with Domino:

* [Spark on Domino](/cloud/platform-capabilities/features/compute-clusters-hpc/spark)

* [On-Demand Dask](/cloud/platform-capabilities/features/compute-clusters-hpc/dask)

* [On-Demand Ray](/cloud/platform-capabilities/features/compute-clusters-hpc/ray)

* [On-Demand Open MPI](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi)

* [Slurm for HPC workloads](/cloud/platform-capabilities/features/compute-clusters-hpc/slurm)

## Clusters with Domino Nexus

[Domino Nexus](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture) has full support for Domino compute clusters in remote executions. If the selected cluster framework includes a user interface, you can only access that user interface if the [data plane](/cloud/platform-capabilities/core-concepts/data-plane) is configured for Workspaces. If the data plane is not configured for Workspaces, clusters can still be used with Jobs, but the user interface link is disabled. An admin can [Enable a data plane for Workspaces](/cloud/admin/platform-configuration/manage-data-planes#enable-a-data-plane-for-workspaces-and-datasets).


# Configure MPI prerequisites
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/configure-mpi-prerequisites

Enable on-demand MPI and build compatible cluster and execution Compute Environments with matching Open MPI versions for Workspaces and Jobs.

Before using on-demand Message Passing Interface (MPI) clusters on Domino, this functionality must be enabled and properly configured on your deployment.

## Configure MPI access on your deployment

All users can access MPI clusters by default. Your Domino Administrator may set the `ShortLived.MpiClustersEnabled` feature flag to `false` to disable the on-demand MPI functionality. The flag is `true` by default.

<Note>
  Contact [Domino Support](mailto:support@dominodatalab.com) if you are not able to access MPI On-demand functionality, as it may not be enabled for your deployment.
</Note>

## Create a base Open MPI cluster environment

Domino does not include an Open MPI compatible [compute environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) for use with the components of the cluster, by default. You can not create a cluster unless you have a compatible compute environment.

<Note>
  When using on-demand MPI in Domino, you have one environment for the MPI cluster (cluster compute environment) and one for the workspace/job execution (compute environment). These environments must have the same version of Open MPI installed.
</Note>

1. To setup base image, click **Start from a custom base image** and enter an image URI for a deployable MPI image.

   For workloads using Horovod, Tensorflow, or Pytorch, Domino recommends basing both the cluster and compute environments on an NGC image containing Open MPI. An example is [TensorFlow](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorflow). The current URI is `nvcr.io/nvidia/tensorflow:22.02-tf2-py3`. You can use other Open MPI images such as [Horovod](https://github.com/horovod/horovod). For simple, CPU-only workloads not needing ML libraries, use the pre-built Domino MPI images: `quay.io/domino/mpi:Mpi4.1.2-Py3.8.12`.

2. To enable the environment for use when creating MPI clusters from workspaces and jobs, click **MPI** in the **Supported Clusters** list.

   <Note>
     The **Automatically make compatible with Domino** checkbox is removed when you click **MPI** in the **Supported Clusters** list.
   </Note>

3. For visibility, set this attribute to the same value as any other [compute environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments).

4. To create the environment, click **Create Environment**. See [Manage dependencies](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/manage-dependencies) to add packages to the environment.

### Prepare your Open MPI execution compute environment

You must also configure the Open MPI compute environments for workspaces and jobs that connect to your cluster. For an Open MPI compute environment you can:

* Recommended: Use the same base image as the cluster environment to create a new environment.

* Enhance the **Docker Instructions** section of an existing compute environment.

#### Create a new environment with the same image as the cluster environment:

1. For your base image, click **Start from a custom base image** and enter a deployable MPI image.

   If you are using Horovod, Tensorflow, or Pytorch, Domino recommends basing both the cluster and compute environments on a suitable NGC image containing Open MPI. An example of this is [TensorFlow](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorflow). The current URI is `nvcr.io/nvidia/tensorflow:22.02-tf2-py3`. For simple, CPU-only workloads not needing ML libraries, use the pre-built Domino MPI image: `quay.io/domino/mpi:Mpi4.1.2-Py3.8.12`.

2. Check **Automatically make compatible with Domino** unless you are using a Domino image.

3. For visibility, set this attribute to the same value as any other [compute environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments).

4. To create the environment, click **Create Environment**. See [Manage dependencies](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/manage-dependencies) to add packages to the environment.

#### Add Open MPI to an existing environment

1. Open the environment and click **Edit Definition**.

2. Add the following instructions to the **Dockerfile Instructions**, editing the instructions as needed:

   ```dockerfile theme={null}
   ENV OMPI_VERSION=4.1.2
   ENV OMPI_MAJOR_VERSION=4.1
   ENV OMPI_SHA256=a400719b04375cd704d2ed063a50e42d268497a3dfede342986ab7a8d7e8dcf0

   ENV DOMINO_USER=ubuntu
   ENV DOMINO_GROUP=ubuntu

   # Create ubuntu user
   RUN if ! id 12574 &> /dev/null; then
           groupadd -g 12574 ${DOMINO_GROUP};
           useradd -u 12574 -g 12574 -m -N -s /bin/bash ${DOMINO_USER};
       fi

   WORKDIR /opt

   #NOTE:build essentials is already present in most Domino distributions.

   RUN apt-get -y update && apt-get -y install curl libdigest-sha-perl build-essential

   # see https://www.open-mpi.org/faq/?category=running#mpirun-prefix to find config options.
   # https://www.open-mpi.org/faq/?category=building#where-to-install
   RUN curl -o openmpi-${OMPI_VERSION}.tar.gz https://download.open-mpi.org/release/open-mpi/v${OMPI_MAJOR_VERSION}/openmpi-${OMPI_VERSION}.tar.gz &&
       echo "${OMPI_SHA256}  openmpi-${OMPI_VERSION}.tar.gz" | shasum -a 256 -c &&
       tar -xf openmpi-${OMPI_VERSION}.tar.gz &&
       cd openmpi-${OMPI_VERSION} &&
       ./configure
           --prefix=/opt/mpi
           --enable-mpirun-prefix-by-default &&
       make -j $(nproc) all &&
       make install

   # set the PATH vars appropriately using domino-defaults
   # https://www.open-mpi.org/faq/?category=running#run-prereqs
   RUN
       echo "export PATH=/opt/mpi/bin:$PATH" >> /home/${DOMINO_USER}/.domino-defaults &&
       if [ -z "$LD_LIBRARY_PATH" ]; then
           echo "export LD_LIBRARY_PATH=/opt/mpi/lib" >> /home/${DOMINO_USER}/.domino-defaults;
       else
           echo "export LD_LIBRARY_PATH=/opt/mpi/lib:$LD_LIBRARY_PATH" >> /home/${DOMINO_USER}/.domino-defaults;
       fi &&
       echo "export OMPI_VERSION=$OMPI_VERSION" >> /home/${DOMINO_USER}/.domino-defaults

   WORKDIR /
   ```

3. Click **Build**.


# File sync MPI clusters
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/file-sync-mpi-clusters

Sync Project code and data to MPI workers before submitting a run, use mounted Domino Datasets, and access AWS credentials across the cluster.

<Note>
  The Open MPI feature is only available with Domino versions 5.1.1 and later. Previous versions of Domino do not support this feature.
</Note>

When using Domino on-demand MPI clusters, data and code from the workspace must be synced with the worker nodes before submitting an `mpirun` run.

1. Click the **File Changes** tab on the navigation bar.

   <Note>
     The tab displays changes since the last sync operation. The `mpirun` command does not detect changes since the last sync. You must manually sync to use updated code on the cluster worker nodes. Do not execute `mpirun` commands while files are syncing.
   </Note>

2. Click **Sync** under **MPI Cluster File Sync**.

   The MPI file sync applies to the following directories:

   * `/mnt` and subdirectories including `/imported` for Domino file system projects.

   * `/mnt/code`, `/mnt/artifacts`, `/mnt/imported` and subdirectories for Git-based projects.

   * `/repos`.

<Warning>
  On-demand clusters in Domino are temporary. Any data stored on cluster local storage, and not externally, is lost when the workload and the cluster shut down.
</Warning>

## Domino Datasets

When you create an MPI cluster attached to a Domino workspace or job, any [Domino dataset](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets) accessible from the workspace or job is accessible from all components of the cluster under the same dataset mount path. Worker nodes can also write to attached read/write Domino datasets.

## AWS credential propagation

When AWS credential propagation is enabled, temporary AWS credentials are automatically available on all MPI workers and your session. These credentials correspond to the roles enabled in your company’s identity provider.

The credentials are automatically refreshed and available with a profile name corresponding to each role in the AWS credential file. The `AWS_SHARED_CREDENTIALS_FILE` environment variable contains the location of the file. This variable is in the proper search path for `s3fs` and `boto3`. See [Connect to generic S3](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-generic-s3) for more info.


# Distributed GPUs with Open MPI
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/index

Provision Open MPI clusters for distributed multi-GPU, multi-CPU, and high performance computing workloads from Domino Workspaces and Jobs.

<Note>
  The Open MPI feature is only available with Domino versions 5.1.1 and later. Previous versions of Domino do not support this feature.
</Note>

Message Passing Interface ([MPI](https://www.mpi-forum.org/)), is a communication protocol for distributed parallel computing. Domino validates the use of [Open MPI](https://www.open-mpi.org/doc/current/), a popular open source MPI distribution that is widely used in high performance computing.

Open MPI has these features:

* **Leading open source MPI distribution:** Open MPI provides low-latency and high bandwidth, gradual parallelism, and flexibility.

* **Support for machine learning in high performance environments:** MPI is the underlying communication mechanism for higher-level machine learning training libraries. MPI is often used in Horovod to train models in high-performance environments.

## Orchestrate Open MPI on Domino

Domino can dynamically provision and orchestrate an MPI cluster directly on the infrastructure backing the Domino deployment. You get quick access without needing an IT team.

Starting a [Domino workspace](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace) for interactive work or [Domino job](/cloud/platform-capabilities/core-concepts/jobs) for batch processing, Domino creates, manages, and makes available a containerized MPI cluster to your execution.

## Use cases

Domino on-demand MPI clusters are suitable for the following workloads:

**Distributed multi-GPU training**\
Open MPI is ideal for distributed multi-GPU and multi-CPU training for Tensorflow, PyTorch, Keras, or MXNet models.

**High performance computing**\
MPI clusters have lower overhead than other distributed computing systems and are highly customizable.

## Next steps

* Learn how to enable and configure the functionality on your deployment in [Configure MPI prerequisites](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/configure-mpi-prerequisites).

* Find out how to [file sync MPI clusters](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/file-sync-mpi-clusters).

* Find out more about the [Validated MPI version](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/validate-mpi-version).

* Learn how to [create an on-demand MPI cluster](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/work-with-your-cluster) with the desired cluster settings attached to a Workspace or Job.

* Find out how you can [manage MPI dependencies](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/manage-dependencies).


# Validate MPI version
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/validate-mpi-version

Check the Open MPI version Domino validates for on-demand clusters and understand the support boundary for other versions and distributions.

<Note>
  The Open MPI feature is only available with Domino versions 5.1.1 and later. Previous versions of Domino do not support this feature.
</Note>

For each minor release, Domino performs validation tests on a specific version of Open MPI. These tests cover the integration surface between the Domino platform and the on-demand MPI clusters the platform orchestrates. The testing is not a comprehensive test of Open MPI, because Domino does not modify that software.

<Note>
  The current validated and supported version of Open MPI is `4.1.2`. Other Open MPI versions might function properly, but are not supported. Domino does not validate other distributions of MPI.
</Note>


# Configure Ray prerequisites
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/ray/configure-prerequisites

Create compatible Ray cluster and execution Compute Environments from Domino-provided images, then add Workspace tools and dependencies.

Ray functionality is available with Domino-provided images. The Domino Ray Environment is built specifically for workspaces that control a Ray cluster. It includes Ray on top of the typical DSE functionality.

You can get the latest released Ray images from [compute Environment catalog](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/8-compute-environment-catalog).

## Create an environment

To set up a Ray cluster, the user needs to first create two environments, one for the Ray cluster (base or worker environment) and one for the workspace/job execution (compute environment).

To create a new base Ray cluster environment, follow the general [environment creation steps](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/3-create-an-environment) with the following `environment_attributes`:

* **Base Image**

  Select **Custom Image** and enter an image URI that points to a deployable Ray image.

  Domino recommends that you use the latest release tag for your version of Ray from the options published at [compute Environment catalog](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/8-compute-environment-catalog).

  For example, for Domino 6.0 the following cluster images are available:

  * `quay.io/domino/ray-cluster-environment:ray2.36.0-py3.10-domino6.0`

  * `quay.io/domino/ray-cluster-gpu-environment:ray2.36.0-py3.10-gpu-domino6.0`

* **Supported Clusters**

  Select **Domino managed Ray** (Required). This ensures that the environment is available for use when creating Ray clusters from workspaces and jobs.

* **Visibility**

  You can set this attribute the same way you would for any other [compute environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments) based on your desired visibility.

* **Dockerfile Instructions**

  Leave blank to use the images as provided by Ray project.

  You can modify this section to include additional packages that might be necessary for your workloads and must be available on the Ray cluster nodes.

  See [Manage dependencies](/cloud/platform-capabilities/features/compute-clusters-hpc/ray/manage-dependencies) to learn more.

* **Pluggable Notebooks / Workspace Sessions**

  Leave this section blank because the Ray base environments are not intended to include any workspace configuration.

## Prepare your Ray execution compute environment

In addition to the base Ray cluster environment, you must connect your workspace to your cluster.

Domino recommends that you use the Ray base image to create a compatible workspace. See [compute Environment catalog](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/8-compute-environment-catalog).

For example, for Domino 6.0, the available compute image is:

`quay.io/domino/domino-ray-environment:ubuntu22-py3.10-r4.4-ray2.36.0-domino6.0`

### Customize this Workspace compute environment

Use the image mentioned previously and add **Pluggable Workspace Tools**.

```yaml theme={null}
jupyter:
  title: "Jupyter (Python, R, Julia)"
  iconUrl: "/assets/images/workspace-logos/Jupyter.svg"
  start: [ "/opt/domino/workspaces/jupyter/start" ]
  supportedFileExtensions: [ ".ipynb" ]
  httpProxy:
    port: 8888
    rewrite: false
    internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
    requireSubdomain: false
jupyterlab:
  title: "JupyterLab"
  iconUrl: "/assets/images/workspace-logos/jupyterlab.svg"
  start: [  "/opt/domino/workspaces/jupyterlab/start" ]
  httpProxy:
    internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
    port: 8888
    rewrite: false
    requireSubdomain: false
vscode:
  title: "vscode"
  iconUrl: "/assets/images/workspace-logos/vscode.svg"
  start: [ "/opt/domino/workspaces/vscode/start" ]
  httpProxy:
    port: 8888
    requireSubdomain: false
rstudio:
  title: "RStudio"
  iconUrl: "/assets/images/workspace-logos/Rstudio.svg"
  start: [ "/opt/domino/workspaces/rstudio/start" ]
  httpProxy:
    port: 8888
    requireSubdomain: false
```


# Use on-demand Ray
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/ray/index

Provision on-demand Ray clusters for distributed training, hyperparameter optimization, and reinforcement learning in Workspaces and Jobs.

[Ray.io](https://ray.io) is a distributed execution framework that makes it easy to scale your single machine applications, with little or no changes, and to leverage state-of-the-art machine learning libraries.

Ray provides a set of core low-level primitives as well as a family of pre-packaged libraries that take advantage of these primitives to enable solving machine learning problems.

The following libraries come packaged with Ray:

* [Tune: Scalable Hyperparameter Tuning](https://docs.ray.io/en/latest/tune/index.html)

* [Ray Train: Distributed Training](https://docs.ray.io/en/latest/train/train.html)

* [RLlib: Industry-Grade Reinforcement Learning](https://docs.ray.io/en/latest/rllib/index.html)

* [Ray Serve: Scalable and Programmable Serving](https://docs.ray.io/en/latest/serve/index.html)

* [Ray Data: Scalable Data Processing for ML](https://docs.ray.io/en/latest/data/data.html)

Additionally, Ray has been adopted as a foundational framework by a large number of open source ML frameworks which now have community-maintained Ray integrations.

## Orchestrate Ray on Domino

Domino can dynamically provision and orchestrate a Ray cluster directly on the infrastructure backing the Domino instance. This allows Domino users to get quick access to Ray without having to rely on their IT team.

When you start a [Domino workspace](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace) for interactive work or a [Domino job](/cloud/platform-capabilities/core-concepts/jobs) for batch processing, Domino creates, manages, and makes available a containerized Ray cluster to your execution.

See Domino’s [quick-start-ray](https://github.com/dominodatalab/domino-quickstart-ray) project.

## Suitable use cases

Domino on-demand Ray clusters are suitable for the following workloads:

**Distributed multi-node training**\
Ray Train provides a lightweight mechanism for taking existing PyTorch and TensorFlow models and scaling them across multiple machines to dramatically reduce training times. Ray is suitable for both distributed CPU and GPU training.

**Hyperparameter optimization**\
Launch a distributed hyperparameter sweep with just a few lines of code and no adaptation of your existing training harness, and take advantage of a large number of advanced parameter search algorithms.

**Reinforcement learning**\
Ray, in combination with the RLlib library, allows you to take advantage of a number of built-in reinforcement learning algorithms, but also provides a general framework for developing your own.

<Note>
  While Ray offers a scalable serving capability, the ephemeral nature of the Domino Ray clusters does not make it a good fit for this use case.
</Note>

## Next steps

* Find out more about the [Validated Ray version](/cloud/platform-capabilities/features/compute-clusters-hpc/ray/validated-ray-version).

* Learn how to enable and configure the functionality on your deployment in [Configure Ray prerequisites](/cloud/platform-capabilities/features/compute-clusters-hpc/ray/configure-prerequisites).

* Learn how to [create an on-demand Ray cluster](/cloud/platform-capabilities/features/compute-clusters-hpc/ray/work-with-your-cluster) with the desired cluster settings attached to a Workspace or Job.

* Find out how you can [manage Ray dependencies](/cloud/platform-capabilities/features/compute-clusters-hpc/ray/manage-dependencies).

* Learn how to [Access data with Ray](/cloud/platform-capabilities/features/compute-clusters-hpc/ray/work-with-data).


# Manage Ray dependencies
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/ray/manage-dependencies

Install additional packages or specific versions in both Ray cluster and execution Compute Environments through Docker Instructions.

While the Ray base images (especially the `ray-ml` flavor) come with a comprehensive set of packages frequently used for machine learning, you may still need to modify your environment when you need additional packages or when you need a specific version of a given package.

Domino allows you to easily package and manage dependencies as part of your [Ray-enabled compute environments](/cloud/platform-capabilities/features/compute-clusters-hpc/ray/configure-prerequisites#create-an-environment). This approach creates the flexibility to manage dependencies for individual projects or workloads without having to deal with the complexity of a shared cluster.

To add a new dependency, you must add the appropriate statements in the **Docker Instructions** section of the relevant Ray base and Ray execution compute environments.

For example, if you wanted to add a particular version of PyTorch you might include the following.

```dockerfile theme={null}
### Optionally specify version if desired
RUN pip install torch==2.8.0 torchvision==0.23.0
```


# Validated Ray version
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/ray/validated-ray-version

Check the Ray version Domino validates for on-demand clusters and understand the technical support boundary for other Ray releases.

For each minor release (starting with Domino 4.5), Domino performs a set of validation tests against the **most recent released version** of Ray. The purpose of the tests is to cover the integration surface between the Domino platform and the on-demand Ray clusters that the platform orchestrates. The testing is not intended to be a comprehensive test of Ray itself since Domino does not modify that software.

<Note>
  The validated and supported version of Ray is `2.55.1`. Other Ray versions are likely to function properly, but Domino cannot offer technical support.
</Note>


# Access data with Ray
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/ray/work-with-data

Access Domino Datasets and Amazon S3 from ephemeral Ray clusters, and make required libraries and temporary AWS credentials available to workers.

When using a Domino on-demand Ray cluster, any data that is used, created, or modified as part of the interaction needs to go into an external data store.

<Warning>
  On-demand clusters in Domino are ephemeral. Any data that is stored on cluster local storage and not externally is lost upon termination of the workload and the cluster.
</Warning>

## Use Domino Datasets

When you create a Ray cluster attached to a Domino workspace or job, any [Domino dataset](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets) accessible from the workspace or job is also accessible from all components of the cluster under the same dataset mount path. You can then access the files from your code using the same path regardless of whether your code runs on your workspace of job container or in a Ray task on the cluster.

For example, to read a file you would use the following:

```python theme={null}
file = open("/mnt/data/my_dataset/file.csv")
```

## Using S3

To access Amazon S3 (or S3 compatible object store) data with Ray you can use any of the libraries you already use (for example, `boto3`, `s3fs`, or `pandas`). When access happens from Ray workers, the only prerequisite is to make sure that the relevant libraries and dependencies are available on both the base cluster environment and the execution environment.

### AWS credential propagation

When AWS credential propagation is enabled for your deployment, temporary AWS credentials corresponding to the roles enabled for you in your company identity provider are automatically available on all Ray worker nodes.

The credentials are automatically refreshed and available under a profile name corresponding to each role in an AWS credential file. Ray worker code can then utilize these credentials for secure access.


# Work with your Ray cluster
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/ray/work-with-your-cluster

Create Ray clusters for Workspaces and Jobs, configure autoscaling and Hardware Tiers, connect Python clients, and use the Ray web UI.

## Create a cluster with Workspaces

Create an on-demand Ray cluster attached to a Domino Workspace:

1. Go to **Workspaces** > **New Workspace**.

2. From Launch New Workspace, select the **Compute Cluster** step.

3. Specify the [cluster settings](#set-your-cluster-settings) and launch your workspace. After the workspace is up, it has access to the Ray cluster you configured.

## Create a cluster with Jobs

<span />

Create an on-demand Ray cluster attached to a Domino Job:

1. Go to **Jobs** > **Run**.

2. From Start a Job, select the **Compute Cluster** step.

3. Specify the [cluster settings](#set-your-cluster-settings) and launch your job. The job has access to the Ray cluster you configured.

You can use any Python script that interacts with your Ray cluster.

## Set your cluster settings

Set the following cluster settings:

* **Min Workers**

  The number of Ray node workers in the Ray cluster when it starts. If **Auto-scale workers** is not enabled, this is always the size of the cluster. The combined capacity of the workers is available for your workloads.

* **Max Workers**

  The maximum number of Ray node workers that the cluster can reach when **Auto-scale workers** is enabled. See [cluster autoscaling](#cluster-auto-scaling).

* **Cluster size: Limit**

  The maximum number of workers that you can make available to your cluster is limited by the number of per-user executions that your Domino administrator has configured for your deployment or by the maximum simultaneous executions of the underlying Hardware Tier used for workers.

  In addition to the number of Ray node workers, you need one slot for your cluster master and one slot for your workspace or job.

* **Worker Hardware Tier**

  The amount of compute resources (CPU, GPU, and memory) that is made available to each Ray node worker.

* **Head Hardware Tier**

  Same mechanics as the worker hardware tier, but applied to the resources that are available for your Ray cluster head node.

  The Ray head node coordinates the Ray workers, so it does not need a significant amount of CPU resources. It hosts the Ray Global Control Store. The amount of required memory depends on the complexity of your application.

  By default, any hardware tier is available when selecting resources for your worker and head. Domino administrators can optionally configure compute cluster [dedicated hardware tiers](/cloud/admin/infrastructure-and-compute/manage-compute-resources).

* **Cluster Compute Environment**

  Designates your compute environment for the Ray cluster.

* **Dedicated local storage per executor**

  The amount of dedicated storage in Gigabytes (2^30 bytes) that is available to each Ray worker.

  The storage is automatically mounted to `/tmp`.

  The storage is automatically provisioned when the cluster is created and de-provisioned when it is shut down.

  <Warning>
    The local storage per worker must not be used for storing any data that must be available after the cluster is shut down.
  </Warning>

## Cluster auto-scaling

Use cluster auto-scaling to start with a small cluster which automatically scales up and down in response to the resource consumption of your workload. This approach uses resources more efficiently for bursty workloads.

By default, the cluster size increases when the average CPU utilization of your workload reaches 80%. A Domino administrator can further refine the auto-scaling settings by including memory utilization or changing the scaling thresholds.

Scale down happens if resource utilization is low for a period of at least 5 minutes. Depending on the workload that you are executing, scale down might terminate cluster nodes that contain intermediate results which must be recomputed.

## Connect to your cluster

When provisioning your on-demand Ray cluster, Domino sets up environment variables that hold the information needed to connect to your cluster.

Use the following snippet to connect:

<Note>
  The preferred method to connect to a Ray cluster uses a modified version of `ray.init()`. For more information, including the proper way to connect to a Ray cluster depending on the version, see the relevant section of the [Ray docs](https://docs.ray.io/en/latest/index.html).
</Note>

```python theme={null}
import ray
import ray.util
import os

...

if ray.is_initialized() == False:
   service_host = os.environ["RAY_HEAD_SERVICE_HOST"]
   service_port = os.environ["RAY_HEAD_SERVICE_PORT"]
   ray.init(f"ray://{service_host}:{service_port}")
```

## Access the Ray web UI

Ray provides a built-in dashboard with access to metrics, charts, and other features that helps users understand the Ray cluster, libraries, and workloads.

Use the dashboard to:

* View cluster metrics.

* View logs, error, and exceptions across many machines in a single pane.

* View resource utilization, tasks, and logs per node and per actor.

* Kill actors and profile Ray jobs.

* See tune jobs and trial information.

Domino makes the Ray web UI available for active on-demand clusters attached to both workspaces and jobs.

<Note>
  In a hybrid Domino deployment, if the [data plane](/cloud/platform-capabilities/core-concepts/data-plane) is not configured for workspaces, then the user interface link is disabled. An admin can [Enable a data plane for workspaces](/cloud/admin/platform-configuration/manage-data-planes#enable-a-data-plane-for-workspaces-and-datasets).
</Note>

### Ray web UI from Workspaces

The Ray web UI is available from a dedicated tab in your workspace.

### Ray UI from Jobs

The Ray web UI is also available for running jobs from the **Details** tab.

## Cluster lifecycle

On workspace or job startup, a Domino on-demand Ray cluster with its [cluster settings](#set-your-cluster-settings) is automatically provisioned and attached to the workspace or job as soon as the cluster becomes available.

When the workspace or job terminates, the on-demand Ray cluster and all associated resources are automatically terminated and deprovisioned. This includes any compute resources and storage allocated for the cluster.

## Cluster network security

The on-demand Ray clusters created by Domino are not meant to be shared between users. Each cluster is associated with a given workspace or a job instance. Access to the cluster and the Ray web UI is restricted only to users who can access the workspace or the job attached to it. This restriction is enforced at the networking level and the cluster can only be reached from the execution that provisioned it.


# Connect to an Amazon EMR cluster from Domino
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/amazon-emr-cluster

Connect Domino Projects to Amazon EMR by packaging cluster binaries and configuration in a Compute Environment and enabling YARN integration.

Domino supports connecting to an [Amazon EMR](https://aws.amazon.com/emr/) cluster through the addition of cluster-specific binaries and configuration files to your [Domino environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments).

At a high level, the process is as follows:

1. Connect to the EMR Master Node and gather the required binaries and configuration files needed by Domino.

2. Create a new Domino environment that uses the uploaded files to enable connections to your cluster.

3. Enable YARN integration for the Domino projects that you want to use with the EMR cluster.

Domino supports the following types of connections to an EMR cluster:

* [FS shell](https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/bk_cloud-data-access/content/using-fs-shell.html)

* [spark-shell](https://books.japila.pl/apache-spark-internals/)

* [spark-submit](https://spark.apache.org/docs/latest/submitting-applications.html)

* [PySpark](https://spark.apache.org/docs/latest/api/python/index.html)

## Requirements

These instructions are written with the following requirements:

* Domino needs to be routable from the EMR cluster by private EC2 IP. This can be achieved by launching EMR directly into Domino’s VPC or via VPC Peering.

* Your security groups are configured to allow traffic between EMR and Domino. The Domino node security group, the EMR Master Node, and the EMR Worker Node security groups all need to allow TCP traffic between them.

## Gather and serve the required binaries and configuration files

You find the necessary files for setting up your Domino environment on the [EMR Master Node](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-master-core-task-nodes.html). To get started, [connect to your Master Node via SSH](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-connect-master-node-ssh.html).

After you are connected to the Master Node, use `vi` or the editor of your choice to create a script called `domino-emr-config-maker.sh`. Copy in the following code and save the script.

<Note>
  This script also captures Hive binaries. If you do not have Hive installed on your EMR cluster, you may need to remove the Hive-related commands from this script.
</Note>

```sh theme={null}
#!/bin/bash

rm -rf www
rm -rf /tmp/hadoop-binaries-configs

mkdir -p www
mkdir -p /tmp/hadoop-binaries-configs/configs

cp -rL /etc/hadoop /tmp/hadoop-binaries-configs/configs
cp -rL /etc/hive /tmp/hadoop-binaries-configs/configs
cp -rL /etc/spark /tmp/hadoop-binaries-configs/configs

cp -r /usr/lib/hadoop /tmp/hadoop-binaries-configs
cp -r /usr/lib/hadoop-lzo /tmp/hadoop-binaries-configs
cp -r /usr/lib/spark /tmp/hadoop-binaries-configs
cp -r /usr/share/aws /tmp/hadoop-binaries-configs
cp -r /usr/share/java /tmp/hadoop-binaries-configs

cd /tmp/hadoop-binaries-configs/configs/hadoop/conf/
sed -i '$ d'  hdfs-site.xml
echo "<property>" >> hdfs-site.xml
echo "<name>dfs.client.use.datanode.hostname</name>" >> hdfs-site.xml
echo "<value>true</value>" >> hdfs-site.xml
echo "</property>" >> hdfs-site.xml
echo "</configuration>" >> hdfs-site.xml

cd /tmp
tar -zcf hadoop-binaries-configs.tar.gz hadoop-binaries-configs
cd ~
mv /tmp/hadoop-binaries-configs.tar.gz www/
cd www
/usr/bin/python3 -m http.server
```

This script bundles together all of the required binaries and configurations and serves it via a web server on port 8000 of the Master Node. You need to open port 8000 in your cluster’s security group if you have not already.

Before moving on, note the private IP address of your EMR Master Node. This is available in your connection’s prompt.

Execute this script via the command `bash domino-emr-config-maker.sh` to begin the bundling and launch the web server. You want to leave your `ssh` connection to the Master Node open while finishing the rest of this setup.

## Create a Domino environment for connecting to EMR

1. Create a new Domino environment with the latest version of the Domino Analytics Distribution as its base image.

2. Edit this environment, and add the following code to the environment’s Dockerfile. Be sure to replace `<MASTER_NODE_PRIVATE_IP>` with the private IP address you noted earlier.

   ```dockerfile theme={null}
   ENV EMR_MASTER_PRIVATE_IP <MASTER_NODE_PRIVATE_IP>

   USER root
   RUN echo "ubuntu ALL=(ALL:ALL) NOPASSWD: ALL" >> /etc/sudoers

   RUN mkdir /tmp/domino-hadoop-downloads

   # Download the binaries and configs gzip from EMR master.
   #
   # This downloaded gzip archive should contain a configs directory with
   # hadoop, hive, and spark subdirectories directories.
   #
   # You may need to edit this depending on where you are running the web server on your EMR master.
   RUN wget -q http://$EMR_MASTER_PRIVATE_IP:8000/hadoop-binaries-configs.tar.gz -O /tmp/domino-hadoop-downloads/hadoop-binaries-configs.tar.gz &&
       tar xzf /tmp/domino-hadoop-downloads/hadoop-binaries-configs.tar.gz -C /tmp/domino-hadoop-downloads/

   RUN cp -r /tmp/domino-hadoop-downloads/hadoop-binaries-configs/configs/hadoop /etc/hadoop &&
       cp -r /tmp/domino-hadoop-downloads/hadoop-binaries-configs/configs/hive /etc/hive &&
       cp -r /tmp/domino-hadoop-downloads/hadoop-binaries-configs/configs/spark /etc/spark

   RUN mv /tmp/domino-hadoop-downloads/hadoop-binaries-configs/aws /usr/share/aws
   RUN mv /tmp/domino-hadoop-downloads/hadoop-binaries-configs/hadoop /usr/lib/hadoop
   RUN mv /tmp/domino-hadoop-downloads/hadoop-binaries-configs/hadoop-lzo /usr/lib/hadoop-lzo
   RUN mv /tmp/domino-hadoop-downloads/hadoop-binaries-configs/spark /usr/lib/spark
   RUN cp -r /tmp/domino-hadoop-downloads/hadoop-binaries-configs/java/* /usr/share/java/

   RUN
   echo 'export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64' >> /home/ubuntu/.domino-defaults &&
   echo 'export HADOOP_HOME=/usr/lib/hadoop' >> /home/ubuntu/.domino-defaults &&
   echo 'export SPARK_HOME=/usr/lib/spark' >> /home/ubuntu/.domino-defaults &&
   echo 'export PYTHONPATH=${PYTHONPATH:-}:${SPARK_HOME:-}/python/' >> /home/ubuntu/.domino-defaults &&
   echo 'export PYTHONPATH=${PYTHONPATH:-}:${SPARK_HOME:-}/python/lib/py4j-0.10.7-src.zip' >> /home/ubuntu/.domino-defaults &&
   echo 'export PATH=${PATH:-}:${SPARK_HOME:-}/bin' >> /home/ubuntu/.domino-defaults &&
   echo 'export PATH=${PATH:-}:${HADOOP_HOME:-}/bin' >> /home/ubuntu/.domino-defaults &&
   echo 'export HADOOP_CONF_DIR=/etc/hadoop/conf' >> /home/ubuntu/.domino-defaults &&
   echo 'export YARN_CONF_DIR=/etc/hadoop/conf' >> /home/ubuntu/.domino-defaults &&
   echo 'export SPARK_CONF_DIR=/etc/spark/conf' >> /home/ubuntu/.domino-defaults
   RUN mkdir -p /var/log/spark/user/ubuntu
   RUN chown ubuntu:ubuntu /var/log/spark/user/ubuntu
   RUN chmod -R 777 /usr/lib/spark/conf

   USER ubuntu
   ```

3. Build the new revision of the environment by clicking the button on the bottom of the page. You may want to follow along by viewing the Build Logs of the build, accessible from the Revisions table of the environment.

   <Note>
     If the build hangs or fails, you may need to adjust the inbound rules of your security groups as described at the start of this doc.
   </Note>

4. After the environment builds successfully, you may stop the web server on your EMR Master Node that was launched earlier and close the `ssh` connection.

## Configure a Domino Project for use with an EMR cluster

This procedure assumes that an environment with the necessary client software has been created according to the instructions above. Ask your Domino admin for access to such an environment. Note that you may need to provide Domino with additional options when setting up your project. Your Domino or AWS administrators should be able to provide you with the correct values for these options.

1. Open the Domino project you want to use with your EMR cluster, then click **Settings** from the project menu.

2. On the **Integrations** tab, click to select **YARN** integration from the Apache Spark panel.

3. Use `root` as the **Hadoop username**.

4. If your work with the cluster generates many warnings about missing Java packages, you can suppress these by adding the following to **Spark Configuration Options**.

   Key: `spark.hadoop.yarn.timeline-service.enabled`

   Value: `false`

5. After inputting your **YARN** configuration, click **Save**.

6. On the **Hardware & Environment** tab, change the project default environment to the one you built earlier with the binaries and configuration files.

You are now ready to start Runs from this project that interact with your EMR cluster.


# Connect to a Cloudera CDH5 cluster from Domino
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/cloudera-cdh5-cluster

Connect Domino Projects to Cloudera CDH5 by packaging edge-node binaries, configuration, and Java in a Compute Environment with YARN.

Domino supports connecting to a [Cloudera CDH5](https://www.cloudera.com/services-and-support/tutorials/how-to-create-a-cdp-private-cloud-base-development-cluster.html) cluster through the addition of cluster-specific binaries and configuration files to your [Domino environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments).

At a high level, the process is as follows:

1. Connect to your [CDH5 edge or gateway node](https://docs-archive.cloudera.com/documentation/enterprise/6/6.3/topics/cdh_sg_gateway_setup.html) and gather the required binaries and configuration files, then download them to your local machine.

2. Upload the gathered files into a Domino project to allow access by the Domino environment builder.

3. Create a new Domino environment that uses the uploaded files to enable connections to your cluster.

4. Enable YARN integration for the Domino projects that you want to use with the CDH5 cluster.

Domino supports the following types of connections to a CDH5 cluster:

* [FS shell](https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/bk_cloud-data-access/content/using-fs-shell.html)

* [spark2-shell](https://docs-archive.cloudera.com/documentation/spark2/2-3-x/topics/spark_running_apps.html)

* [spark2-submit](https://docs-archive.cloudera.com/documentation/spark2/2-3-x/topics/spark_running_apps.html)

* [PySpark](https://spark.apache.org/docs/latest/api/python/index.html)

* [YARN shell](https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/YarnCommands.html)

## Gather the required binaries and configuration files

You can find most of the files for setting up your Domino environment on your [CDH5 edge or gateway node](https://docs-archive.cloudera.com/documentation/enterprise/6/6.3/topics/cdh_sg_gateway_setup.html). To get started, connect to the edge node via SSH, then follow the steps below.

1. Create a directory named `hadoop-binaries-configs` at `/tmp`.

   ```shell theme={null}
   mkdir /tmp/hadoop-binaries-configs
   ```

2. Create the following subdirectories inside `/tmp/hadoop-binaries-configs/`.

   ```shell theme={null}
   mkdir /tmp/hadoop-binaries-configs/configs

   mkdir /tmp/hadoop-binaries-configs/parcels
   ```

3. Optional: If your cluster uses Kerberos authentication, create the following subdirectory in `/tmp/hadoop-binaries/configs/`.

   ```shell theme={null}
   mkdir /tmp/hadoop-binaries-configs/kerberos
   ```

   Then, copy the `krb5.conf` Kerberos configuration file from `/etc/` to `/tmp/hadoop-binaries-configs/kerberos`.

   ```shell theme={null}
   cp /etc/krb5.conf /tmp/hadoop-binaries-configs/kerberos/
   ```

4. Copy the `CDH` and `SPARK2` directories from `/opt/cloudera/parcels/` to `/tmp/hadoop-binaries-configs/parcels/`. These directories have a version number appended to their names, so complete the appropriate directory name in the commands shown below.

   ```shell theme={null}
   cp -R /opt/cloudera/parcels/CDH-<version>/ /tmp/hadoop-binaries-configs/parcels/
   cp -R /opt/cloudera/parcels/SPARK2-<version>/ /tmp/hadoop-binaries-configs/parcels/
   ```

5. Copy the `hadoop`, `hive`, `spark`, and `spark2` directories from `/etc/` to `/tmp/hadoop-binaries-configs/configs/`.

   ```shell theme={null}
   cp -R /etc/hadoop /tmp/hadoop-binaries-configs/configs/
   cp -R /etc/hive /tmp/hadoop-binaries-configs/configs/
   cp -R /etc/spark2 /tmp/hadoop-binaries-configs/configs/
   cp -R /etc/spark /tmp/hadoop-binaries-configs/configs/
   ```

6. On the edge node, run the following command to identify the version of Java running on the cluster.

   ```shell theme={null}
   java -version
   ```

   You should then [download a JDK .tar file from the Oracle downloads page](https://www.oracle.com/technetwork/java/javase/downloads/index.html) that matches that version. The filename has a pattern like the following.

   `jdk-8u211-linux-x64.tar.gz`

   Keep this JDK handy on your local machine for use in a future step.

7. Compress the `/tmp/hadoop-binaries-configs/` directory to a gzip archive.

   ```shell theme={null}
   cd /tmp

   tar -zcf hadoop-binaries-configs.tar.gz hadoop-binaries-configs
   ```

   When finished, use `SCP` to download the archive to your local machine.

8. Next, you’ll need to extract the archive on your local machine, add a `java` subdirectory, then add the JDK .tar file you downloaded earlier to the `java` subdirectory.

   ```shell theme={null}
   tar xzf hadoop-binaries-configs.tar.gz

   mkdir hadoop-binaries-configs/java

   cp jdk-8u211-linux-x64.tar.gz hadoop-binaries-configs/java/
   ```

9. When finished, your `hadoop-binaries-configs` directory should have the following structure:

   ```console theme={null}
   hadoop-binaries-configs/
     ├── configs/
           ├── hadoop/
           ├── hive/
           ├── spark/
           └── spark2/
     ├── java/
           └── jdk-8u211-linux-x64.tar.gz
     ├── parcels
           ├── CDH-version/
           └── SPARK-version/
     └── kerberos/  # optional
           └── krb5.conf
   ```

10. If your directory contains all the required files, you can now compress it to a gzip archive again in preparation for uploading to Domino in the next step.

    ```shell theme={null}
    tar -zcf hadoop-binaries-configs.tar.gz hadoop-binaries-configs
    ```

## Upload the binaries and configuration files to Domino

Use the following procedure to upload the archive you created in the previous step to a public Domino project. This makes the file available to the Domino environment builder.

1. Log in to Domino, then create a new public project.

2. Open the **Code** page for the new project, then click to **browse for files** and select the archive you created in the previous section. Then click **Upload**.

3. After the archive has been uploaded, click the gear menu next to it on the Files page, then right click **Download** and click **Copy Link Address**. Save the copied URL in your notes, as you need it in the next step.

   After you have recorded the download URL of the archive, you’re ready to build a Domino environment for connecting to your CDH5 cluster.

## Create a Domino environment for connecting to CDH5

1. Click **Environments** from the Domino main menu, then click **Create Environment**.

2. Give the environment an informative name, then choose a base environment that includes the version of Python that is installed on the nodes of your CDH5 cluster. Most Linux distributions ship with Python 2.7 by default, so you see the [Domino analytics distribution for Python 2.7](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) used as the base image in the following examples. Click **Create** when finished.

3. After creating the environment, click **Edit Definition**. Copy the following example into your Dockerfile instructions, then be sure to edit it wherever necessary with values specific to your deployment and cluster.

   In this Dockerfile, wherever you see a hyphenated instruction enclosed in angle brackets like `<paste-your-domino-download-url-here>`, be sure to replace it with the corresponding value you recorded in previous steps.

   You may also need to edit commands that follow to match downloaded filenames.

   ```dockerfile theme={null}
   USER root

   # Give user ubuntu ability to sudo as any user including root
   RUN echo "ubuntu ALL=(ALL:ALL) NOPASSWD: ALL" >> /etc/sudoers

   # Set up directories
   RUN mkdir -p /opt/cloudera/parcels &&
       mkdir /tmp/domino-hadoop-downloads &&
       mkdir /usr/java

   # Download the binaries and configs gzip you uploaded to Domino.
   # This downloaded gzip file should have the following
   # - CDH and Spark2 parcel directories in a 'parcels' sub-directory.
   # - java installation tar file in 'java' sub-directory
   # - krb5.conf in 'kerberos' sub-directory
   # - hadoop, hive, spark2 and spark config directories a 'configs' sub-directory
   RUN wget --no-check-certiticate <paste-your-domino-download-url-here> -O /tmp/domino-hadoop-downloads/hadoop-binaries-configs.tar.gz &&
       tar xzf /tmp/domino-hadoop-downloads/hadoop-binaries-configs.tar.gz -C /tmp/domino-hadoop-downloads/

   # Install kerberos client and update the kerberos configuration file
   RUN apt-get -y install krb5-user telnet &&
       cp /tmp/domino-hadoop-downloads/hadoop-binaries-configs/kerberos/krb5.conf /etc/krb5.conf

   # Install version of Java that matches hadoop cluster and update environment variables
   # Your JDK may have a different filename depending on your cluster's version of Java
   RUN tar xvf /tmp/domino-hadoop-downloads/hadoop-binaries-configs/java/jdk-8u162-linux-x64.tar -C /usr/java
   ENV JAVA_HOME=/usr/java/jdk1.8.0_162
   RUN echo "export JAVA_HOME=/usr/java/jdk1.8.0_162" >> /home/ubuntu/.domino-defaults &&
       echo "export PATH=$JAVA_HOME/bin:$PATH" >> /home/ubuntu/.domino-defaults

   # Install CDH hadoop-client binaries from cloudera ubuntu trusty repository.
   # This example shows client binaries for CDH version 5.15 here.
   # Update these commands with the CDH version that matches your cluster.
   RUN echo "deb [arch=amd64] http://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh trusty-cdh5.15.0 contrib" >> /etc/apt/sources.list.d/cloudera.list &&
       echo "deb-src http://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh trusty-cdh5.15.0 contrib" >> /etc/apt/sources.list.d/cloudera.list &&
       wget http://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh/archive.key -O /tmp/domino-hadoop-downloads/archive.key &&
       apt-key add /tmp/domino-hadoop-downloads/archive.key &&
       apt-get update &&
       apt-get -y -t trusty-cdh5.15.0 install zookeeper &&
       apt-get -y -t trusty-cdh5.15.0 install hadoop-client

   # Copy CDH and Spark2 parcels to correct directories and update symlinks
   # Note that the version strings attached to your directory names may be different than the below examples.
   RUN mv /tmp/domino-hadoop-downloads/hadoop-binaries-configs/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21 /opt/cloudera/parcels/ &&
       mv /tmp/domino-hadoop-downloads/hadoop-binaries-configs/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809 /opt/cloudera/parcels/ &&
       ln -s /opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21 /opt/cloudera/parcels/CDH &&
       ln -s /opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809 /opt/cloudera/parcels/SPARK2

   # Copy hadoop, hive and spark2 configurations
   RUN mv /etc/hadoop /tmp/domino-hadoop-downloads/hadoop-binaries-configs/configs/hadoop-etc-local.backup &&
       mv /tmp/domino-hadoop-downloads/hadoop-binaries-configs/configs/hadoop /etc/hadoop &&
       mv /tmp/domino-hadoop-downloads/hadoop-binaries-configs/configs/hive /etc/hive &&
       mv /tmp/domino-hadoop-downloads/hadoop-binaries-configs/configs/spark2 /etc/spark2 &&
       mv /tmp/domino-hadoop-downloads/hadoop-binaries-configs/configs/spark /etc/spark

   # Create alternatives for hadoop configurations. Update the extensions with the same strings as found in your edge node
   # Example: In the command 'update-alternatives --install /etc/hadoop/conf hadoop-conf /etc/hadoop/conf.cloudera.yarn 55'
   # make sure that /etc/hadoop/conf.cloudera.yarn is named the same as the corresponding file on your edge node.
   # Sometimes in the CDH5 edgenode, that is named something like /etc/hadoop/conf.cloudera.yarn_
   RUN update-alternatives --install /etc/hadoop/conf hadoop-conf /etc/hadoop/conf.cloudera.yarn 55 &&
       update-alternatives --install /etc/hive/conf hive-conf /etc/hive/conf.cloudera.hive 55 &&
       update-alternatives --install /etc/spark2/conf spark2-conf /etc/spark2/conf.cloudera.spark2_on_yarn 55 &&
       update-alternatives --install /etc/spark/conf spark-conf /etc/spark/conf.cloudera.spark_on_yarn 55

   # These instructions are for Spark2
   # Creating alternatives for Spark2 binaries, also create symlink for pyspark pointing to pyspark2
   RUN update-alternatives --install /usr/bin/spark2-shell spark2-shell /opt/cloudera/parcels/SPARK2/bin/spark2-shell 55 &&
       update-alternatives --install /usr/bin/spark2-submit spark2-submit /opt/cloudera/parcels/SPARK2/bin/spark2-submit 55 &&
       update-alternatives --install /usr/bin/pyspark2 pyspark2 /opt/cloudera/parcels/SPARK2/bin/pyspark2 55 &&
       ln -s /usr/bin/pyspark2 /usr/bin/pyspark

   # Update SPARK and HADOOP environment variables. Make sure py4j file name is correct per your edgenode
   ENV SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2
   RUN echo "export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop" >> /home/ubuntu/.domino-defaults &&
       echo "export HADOOP_CONF_DIR=/etc/hadoop/conf" >> /home/ubuntu/.domino-defaults &&
       echo "export YARN_CONF_DIR=/etc/hadoop/conf" >> /home/ubuntu/.domino-defaults &&
       echo "export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2" >> /home/ubuntu/.domino-defaults &&
       echo "export SPARK_CONF_DIR=/etc/spark2/conf" >> /home/ubuntu/.domino-defaults &&
       echo "export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip" >> /home/ubuntu/.domino-defaults

   # Change spark-defaults.conf file permission
   RUN mv /etc/spark2/conf/spark-defaults.conf /etc/spark2/ &&
       chmod 777 /etc/spark2/conf.cloudera.spark2_on_yarn

   # Copy hive-site.xml to /etc/spark2/conf to access hive tables from Spark2.
   RUN cp /etc/spark2/conf/yarn-conf/hive-site.xml /etc/spark2/conf/

   USER ubuntu
   ```

4. Scroll down to the **Pre Run Script** field and add the following lines:

   ```shell theme={null}
   cat /etc/spark2/spark-defaults.conf >> /etc/spark2/conf/spark-defaults.conf
   sed -i.bak '/spark.ui.port=0/d' /etc/spark2/conf/spark-defaults.conf
   ```

5. Scroll down and click **Advanced** to expand additional fields. Add the following line to the **Post Setup Script** field:

   ```shell theme={null}
   echo "export YARN_CONF_DIR=/etc/hadoop/conf" >> /home/ubuntu/.bashrc
   ```

6. Click **Build** when finished editing the Dockerfile instructions. If the build completes successfully, you are ready to try using the environment.

## Configure a Domino project for use with a CDH5 cluster

This procedure assumes that an environment with the necessary client software has been created according to the instructions above. Ask your Domino admin for access to such an environment.

1. Open the Domino project you want to use with your CDH5 cluster, then click **Settings** from the project menu.

2. On the Integrations tab, click to select **YARN** integration from the Apache Spark panel, then click **Save**. You do not need to edit any of the fields in this section.

3. If your cluster uses Kerberos authentication, you can [configure credentials at the user level or project level ](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/kerberos-authentication). Do so before attempting to use the environment. Note that if you followed the instructions above on creating your environment, your Kerberos configuration file has already been added to it.

4. On the **Hardware & Environment** tab, change the project default environment to the one with the cluster’s binaries and configurations files installed.

You are now ready to start Runs from this project that interact with your CDH5 cluster.


# Connect to a Hortonworks cluster from Domino
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/hortonworks-cluster

Connect Domino Projects to Hortonworks by packaging edge-node binaries, configuration, and Java in a Compute Environment, then enable YARN per Project.

Domino supports connecting to a [Hortonworks cluster](https://hortonworks.com/products/data-platforms/hdp/) through the addition of cluster-specific binaries and configuration files to your [Domino environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments).

At a high level, the process is as follows:

1. Connect to your Hortonworks cluster edge node and gather the required binaries and configuration files, then download them to your local machine.

2. Upload the gathered files into a Domino project to allow access by the Domino environment builder.

3. Create a new Domino environment that uses the uploaded files to enable connections to your cluster.

4. Enable YARN integration for the Domino projects that you want to use with the Hortonworks cluster.

Domino supports the following types of connections to a Hortonworks cluster:

* [FS shell](https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/bk_cloud-data-access/content/using-fs-shell.html)

* [spark-shell](https://books.japila.pl/apache-spark-internals/)

* [spark-submit](https://spark.apache.org/docs/latest/submitting-applications.html)

* [pyspark](https://spark.apache.org/docs/latest/api/python/index.html)

* [YARN shell](https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/YarnCommands.html)

## Gather the required binaries and configuration files

You find most of the necessary files for setting up your Domino environment on your edge node. To get started, connect to the edge node via SSH, then follow the steps below.

1. Create a directory named `hadoop-binaries-configs` at `/tmp`.

   ```shell theme={null}
   mkdir /tmp/hadoop-binaries-configs
   ```

2. Create the following subdirectory inside `/tmp/hadoop-binaries-configs/`.

   ```shell theme={null}
   mkdir /tmp/hadoop-binaries-configs/configs
   ```

3. (Optional) If your cluster uses Kerberos authentication, create the following subdirectory in `/tmp/hadoop-binaries/configs/`.

   ```shell theme={null}
   mkdir /tmp/hadoop-binaries-configs/kerberos
   ```

   Then, copy the `krb5.conf` Kerberos configuration file from `/etc/` to `/tmp/hadoop-binaries-configs/kerberos`.

   ```shell theme={null}
   cp /etc/krb5.conf /tmp/hadoop-binaries-configs/kerberos/
   ```

4. Copy the `hadoop`, `hive`, `spark`, and `spark2` directories from `/etc/` to `/tmp/hadoop-binaries-configs/configs/`.

   ```shell theme={null}
   cp -R /etc/hadoop /tmp/hadoop-binaries-configs/configs/
   cp -R /etc/hive /tmp/hadoop-binaries-configs/configs/
   cp -R /etc/spark2 /tmp/hadoop-binaries-configs/configs/
   cp -R /etc/spark /tmp/hadoop-binaries-configs/configs/
   ```

5. On the edge node, run the following command to identify the version of Java running on the cluster.

   ```shell theme={null}
   java -version
   ```

   You should then [download a JDK .tar file from the Oracle downloads page](https://www.oracle.com/technetwork/java/javase/downloads/index.html) that matches that version. The filename has a name like the following:

   ```shell theme={null}
   jdk-8u211-linux-x64.tar.gz
   ```

   Keep this JDK handy on your local machine for use in a future step.

6. Compress the `/tmp/hadoop-binaries-configs/` directory to a gzip archive.

   ```shell theme={null}
   cd /tmp

   tar -zcf hadoop-binaries-configs.tar.gz hadoop-binaries-configs
   ```

   When finished, use SCP to download the archive to your local machine.

7. Next, you’ll need to extract the archive on your local machine, add a java subdirectory, then add the JDK .tar file you downloaded earlier to the java subdirectory.

   ```shell theme={null}
   tar xzf hadoop-binaries-configs.tar.gz

   mkdir hadoop-binaries-configs/java

   cp jdk-8u211-linux-x64.tar.gz hadoop-binaries-configs/java/
   ```

8. When finished, your `hadoop-binaries-configs` directory should have the following structure:

   ```console theme={null}
   hadoop-binaries-configs/
     ├── configs/
           ├── hadoop/
           ├── hive/
           ├── spark/
           └── spark2/
     ├── java/
           └── jdk-8u211-linux-x64.tar.gz
     └── kerberos/  # optional
           └── krb5.conf
   ```

9. If your directory contains all the required files, you can now compress it to a gzip archive again in preparation for uploading to Domino in the next step.

   ```shell theme={null}
   tar -zcf hadoop-binaries-configs.tar.gz hadoop-binaries-configs
   ```

## Upload the binaries and configuration files to Domino

Use the following procedure to upload the archive you created in the previous step to a public Domino project. This makes the file available to the Domino environment builder.

1. Log in to Domino, then create a new public project.

2. Open the **Code** page for the new project, then click to **browse for files** and select the archive you created in the previous section. Then click **Upload**.

3. After the archive has been uploaded, click the gear menu next to it on the Files page, then right click **Download** and click **Copy Link Address**. Save the copied URL in your notes, as you need it in the next step.

   After you have recorded the download URL of the archive, you’re ready to build a Domino environment for connecting to your Hortonworks cluster.

## Create a Domino environment for connecting to Hortonworks

1. Click **Environments** from the Domino main menu, then click **Create Environment**.

2. Give the environment an informative name, then choose a base environment that includes the version of Python that is installed on the nodes of your Hortonworks cluster. Most Linux distributions ship with Python 2.7 by default, so you see the [Domino Analytics Distribution for Python 2.7](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) used as the base image in the following examples. Click **Create** when finished.

3. After creating the environment, click **Edit Definition**. Copy the below example into your Dockerfile Instructions, then be sure to edit it wherever necessary with values specific to your deployment and cluster.

   In this Dockerfile, wherever you see a hyphenated instruction enclosed in angle brackets like `<paste-your-domino-download-url-here>`, be sure to replace it with the corresponding value you recorded in previous steps.

   Additionally, follow the instructions in the comments carefully, as you may need to modify the commands based on versions and filenames from your system. You may also need to edit commands that follow to match downloaded filenames.

   ```dockerfile theme={null}
   USER root

   # Give user ubuntu ability to sudo as any user including root in the compute environment
   RUN echo "ubuntu ALL=(ALL:ALL) NOPASSWD: ALL" >> /etc/sudoers

   # Setup directories
   RUN mkdir /tmp/domino-hadoop-downloads &&
       mkdir /usr/jdk64

   # This downloaded gzip file should have the following
   # - java installation tar file in 'java' sub-directory
   # - krb5.conf in 'kerberos' sub-directory
   # - hadoop, hive, spark2 and spark config directories from hadoop edgenode in a 'configs' sub-directory
   # Make sure your URL is updated to reflect where you uploaded your configs.
   # You should have this saved from your preparation steps
   RUN wget --no-check-certiticate <paste-your-domino-download-url-here> -O /tmp/domino-hadoop-downloads/hadoop-binaries-configs.tar.gz &&
       tar xzf /tmp/domino-hadoop-downloads/hadoop-binaries-configs.tar.gz -C /tmp/domino-hadoop-downloads/

   # Install kerberos client and update the kerberos configuration file
   RUN apt-get update &&
       apt-get -y install krb5-user telnet &&
       cp /tmp/domino-hadoop-downloads/hadoop-binaries-configs/kerberos/krb5.conf /etc/krb5.conf

   # Install version of java that matches hadoop cluster and update environment variables
   RUN tar xvf /tmp/domino-hadoop-downloads/hadoop-binaries-configs/java/jdk-8u112-linux-x64.tar -C /usr/jdk64 &&
       ln -s /usr/jdk64/jdk1.8.0_112 /usr/jdk64/default
   ENV JAVA_HOME=/usr/jdk64/default
   RUN echo "export JAVA_HOME=/usr/jdk64/default" >> /home/ubuntu/.domino-defaults &&
       echo "export PATH=$JAVA_HOME/bin:$PATH" >> /home/ubuntu/.domino-defaults

   # Install HDP hadoop-client and spark binaries from Hortonworks Ubuntu repository.
   # Update the repo URL based for the version that matches what's running on your cluster.
   # This example shows version 2.6.5.
   RUN wget http://public-repo-1.hortonworks.com/HDP/ubuntu14/2.x/updates/2.6.5.0/hdp.list -O /etc/apt/sources.list.d/hdp.list &&
       apt-key adv --keyserver keyserver.ubuntu.com --recv 07513CAD &&
       apt-get update &&
       apt-get -y install hadoop-client spark2-python spark-python

   # Copy hadoop, hive, spark and spark2 configurations
   RUN mv /etc/hadoop /tmp/domino-hadoop-downloads/hadoop-binaries-configs/configs/hadoop-etc-local.backup &&
       mv /etc/spark /tmp/domino-hadoop-downloads/hadoop-binaries-configs/configs/spark-etc-local.backup &&
       mv /etc/spark2 /tmp/domino-hadoop-downloads/hadoop-binaries-configs/configs/spark2-etc-local.backup &&
       cp -r /tmp/domino-hadoop-downloads/hadoop-binaries-configs/configs/hadoop /etc/hadoop &&
       cp -r /tmp/domino-hadoop-downloads/hadoop-binaries-configs/configs/hive /etc/hive &&
       cp -r /tmp/domino-hadoop-downloads/hadoop-binaries-configs/configs/spark2 /etc/spark2 &&
       cp -r /tmp/domino-hadoop-downloads/hadoop-binaries-configs/configs/spark /etc/spark

   # Update symlinks to point to correct configurations
   # When you are creating these symlinks make sure that right versions are specified.
   # Example: In the command 'ln -s /etc/spark2/2.6.5.0-292/0 /etc/spark2/conf'
   # make sure that 2.6.5.0-292/0 is right version according to the edgenode and set the correct version similar to the hortonwork edgenode.
   RUN rm /etc/spark2/conf &&
       rm /etc/spark/conf &&
       rm /etc/hadoop/conf &&
       rm /etc/hive/conf &&
       ln -s /etc/spark2/2.6.5.0-292/0 /etc/spark2/conf &&
       ln -s /etc/spark/2.6.5.0-292/0 /etc/spark/conf &&
       ln -s /etc/hadoop/2.6.5.0-292/0 /etc/hadoop/conf &&
       ln -s /etc/hive/2.6.5.0-292/0 /etc/hive/conf

   # Update SPARK and HADOOP environment variables. Make sure py4j file name is correct as per your edgenode
   ENV SPARK_HOME=/usr/hdp/2.6.5.0-292/spark2
   RUN echo "export HADOOP_HOME=/usr/hdp/2.6.5.0-292/hadoop" >> /home/ubuntu/.domino-defaults &&
       echo "export HADOOP_CONF_DIR=/etc/hadoop/conf" >> /home/ubuntu/.domino-defaults &&
       echo "export YARN_CONF_DIR=/etc/hadoop/conf" >> /home/ubuntu/.domino-defaults &&
       echo "export SPARK_HOME=/usr/hdp/2.6.5.0-292/spark2" >> /home/ubuntu/.domino-defaults &&
       echo "export SPARK_CONF_DIR=/etc/spark2/conf" >> /home/ubuntu/.domino-defaults &&
       echo "export SPARK_MAJOR_VERSION=2" >> /home/ubuntu/.domino-defaults &&
       echo "export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.6-src.zip" >> /home/ubuntu/.domino-defaults

   # Backup existing spark-defautls.conf file.
   # Change spark-defaults.conf directory permission as a new spark-defaults.conf file gets created by Domino's spark integration
   RUN mv /etc/spark2/conf/spark-defaults.conf /etc/spark2/ &&
       chmod 777 /etc/spark2/2.6.5.0-292/0

   USER ubuntu
   ```

4. Scroll down to the **Pre Run Script** field and add the following lines.

   ```shell theme={null}
   cat /etc/spark2/spark-defaults.conf >> /etc/spark2/conf/spark-defaults.conf
   sed -i.bak '/spark.ui.port=0/d' /etc/spark2/conf/spark-defaults.conf
   ```

5. Scroll down and click **Advanced** to expand additional fields. Add the following line to the **Post Setup Script** field.

   ```shell theme={null}
   echo "export YARN_CONF_DIR=/etc/hadoop/conf" >> /home/ubuntu/.bashrc
   ```

6. Click **Build** when finished editing the Dockerfile instructions. If the build completes successfully, you are ready to try using the environment.

## Configure a Domino project for use with a Hortonworks cluster

This procedure assumes that an environment with the necessary client software has been created according to the instructions above. Ask your Domino admin for access to such an environment.

1. Open the Domino project you want to use with your Hortonworks cluster, then click **Settings** from the project menu.

2. On the **Integrations** tab, click to select **YARN** integration from the Apache Spark panel, then click **Save**. You do not need to edit any of the fields in this section.

3. If your cluster uses Kerberos authentication, you can [configure credentials at the user level or project level](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/kerberos-authentication). Do so before attempting to use the environment. Note that if you followed the instructions above on creating your environment, your Kerberos configuration file has already been added to it.

4. On the **Hardware & Environment** tab, change the project default environment to the one with the cluster’s binaries and configurations files installed.

You are now ready to start Runs from this project that interact with your Hortonworks cluster.


# Hadoop and Spark
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/index

Connect Domino Projects to Amazon EMR, Cloudera, MapR, or Hortonworks clusters by preparing Hadoop-enabled Compute Environments and YARN.

[Apache Hadoop](https://hadoop.apache.org/) is a collection of open source cluster computing tools that supports popular applications for data science at scale, such as [Spark](https://spark.apache.org/).

To interact with Hadoop from your Domino executors, configure your [Domino environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments) with the necessary software dependencies and credentials. Domino supports most providers of Hadoop solutions, including MapR, Cloudera, and Amazon EMR. After a Domino environment is set up to connect to your cluster, Domino projects can use the environment to work with Hadoop applications.

## Use a Hadoop-enabled environment in your Domino project

If your Domino administrators have already created an environment for connecting to a Hadoop cluster, you can follow these subsections of the setup instructions to use that environment in your Domino project.

**For users setting up projects to work with an existing environment, read these subsections:**

* [Configure a Domino project for use with a Cloudera CDH5 cluster](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/cloudera-cdh5-cluster#configure-a-domino-project-for-use-with-a-cdh5-cluster)

* [Configure a Domino project for use with a Hortonworks cluster](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/hortonworks-cluster#configure-a-domino-project-for-use-with-a-hortonworks-cluster)

* [Configure a Domino project for use with a MapR cluster](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/mapr-cluster#configure-a-domino-project-for-use-with-a-mapr-cluster)

* [Configure a Domino project for use with an Amazon EMR cluster](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/amazon-emr-cluster#configure-a-domino-project-for-use-with-an-emr-cluster)

After your project is set up to use the environment, you can execute code in your Domino Runs that connects to the cluster for Spark, HDFS, or Hive functionality.

<img alt="Use a Hadoop-enabled environment" />

## Set up Domino to connect to a new Hadoop cluster

To connect to your existing Hadoop cluster from Domino, you must create a Domino environment with the necessary dependencies installed. Some of these dependencies, including binaries and configuration files, come directly from the cluster itself. Others are external software dependencies like Java and Spark, and you need to match the version you install in the environment to the version running on the cluster.

The basic steps for setting up an environment to connect to your cluster are:

1. Gather binaries and configuration files from your cluster.

2. Gather dependencies from external sources, like Java JDKs and Spark binaries.

3. Upload all dependencies to a Domino project, to make them accessible to the Domino environment builder.

4. Author a new Domino environment that pulls from the Domino project, then installs and configures all required dependencies.

   <img alt="Configure environment dependencies" />

**For Domino admins setting up a Domino environment to connect to a new cluster, read the full provider-specific setup guides:**

* [Connect to a Cloudera CDH5 Cluster from Domino](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/cloudera-cdh5-cluster)

* [Connect to a Hortonworks cluster from Domino](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/hortonworks-cluster)

* [Connect to a MapR cluster from Domino](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/mapr-cluster)

* [Connect to an Amazon EMR cluster from Domino](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/amazon-emr-cluster)

## Additional capabilities

Domino also supports running Spark on a Domino executor in local mode, querying Hive tables with JDBC, and authenticating to clusters with Kerberos. See the following guides for more information.

* [Run Local Spark on a Domino Executor](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/local-spark-cluster)

* [Interactive PySpark notebooks](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/pyspark-in-jupyter-workspaces)

* [Kerberos Authentication](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/kerberos-authentication)


# Kerberos authentication
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/kerberos-authentication

Configure Kerberos-secured Runs with krb5.conf, keytab files, and principals at the user or Project level so Domino retrieves authentication tickets.

Domino supports Kerberos authentication, allowing users to authenticate as themselves when connecting to Kerberos-secured systems.

Users can enable Kerberos authentication at the project-level or user-level by uploading a Kerberos keytab and principal into Domino. After set up, Runs started by Kerberos-enabled users or in Kerberos-enabled projects in Domino automatically run kinit and retrieve the ticket to be able to authenticate.

## Adding your Kerberos configuration file to Domino

There are two ways to add your [krb5.conf](https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/krb5_conf.html) file to Domino.

* **Add it to your project in a folder named** `kerberos`.

* **Add it to your environment at** `/etc/krb5.conf`:

```dockerfile theme={null}
RUN echo "[libdefaults]" >> /etc/krb5.conf &&
    echo "    default_realm = [YOUR-DEFAULT-REALM]" >> /etc/krb5.conf &&
    echo "    default_tkt_enctypes = des3-hmac-sha1 des-cbc-crc" >> /etc/krb5.conf &&
    echo "    default_tgs_enctypes = des3-hmac-sha1 des-cbc-crc" >> /etc/krb5.conf &&
    echo "    dns_lookup_kdc = true" >> /etc/krb5.conf &&
    echo "    dns_lookup_realm = false" >> /etc/krb5.conf &&
    echo "" >> /etc/krb5.conf &&
    echo "[realms]" >> /etc/krb5.conf &&
    echo "    EXAMPLE.COM = {" >> /etc/krb5.conf &&
    echo "        kdc = kerberos.example.com" >> /etc/krb5.conf &&
    echo "        admin_server = kerberos.example.com" >> /etc/krb5.conf &&
    echo "    }" >> /etc/krb5.conf
```

## Adding Kerberos credentials to your user

To add a keytab and principal that is used for Runs started by your user:

1. Go to **Account Settings** > **Kerberos Integration**.

2. Select **Keytab file based authentication**, paste your keytab and principal, then click **Save**.

## Adding Kerberos credentials to your Project

To add a keytab and principal that is used for Runs started by a specific project:

1. Open the project **Settings** and click to open the **Integrations** tab.

2. In the **Kerberos** panel, click **Keytab file based authentication**, supply your keytab and principal, then click **Save**.


# Run local Spark on a Domino Executor
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/local-spark-cluster

Run Spark in local mode on a Domino Executor, use its available CPU cores for parallel tasks, and configure the Project's Spark integration.

## Use a local Spark cluster

Typically, users interested in Hadoop and Spark have data volumes and workloads that demand the power of cluster computing. However, some people use Spark for its expressive API, even if their data volumes are small or medium. Because Domino lets you run code on VM infrastructure, with up the 32 cores in AWS, you can use Domino to create a local Spark cluster and easily parallelize your tasks across all 32 cores.

## Configure Spark in local mode

To configure Spark integration in Local mode, open your project and go to **Project settings**. Under **Integrations**, choose the **Local mode** option for Apache Spark. Click **Save** to save your changes.


# Connect to a MapR cluster from Domino
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/mapr-cluster

Connect Domino Projects to MapR by packaging client files, Java, and configuration in a Compute Environment and enabling YARN integration.

Domino supports connecting to a [MapR](https://www.hpe.com/emea_middle_east/en/data-fabric-software.html) cluster through the addition of cluster-specific binaries and configuration files to your [Domino environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments).

At a high level, the process is as follows:

1. Connect to a [MapR Edge node](https://docs.ezmeral.hpe.com/datafabric-customer-managed/81/AdvancedInstallation/PlanningtheCluster-node-types.html) and gather the required binaries and configuration files, then download them to your local machine.

2. Upload the gathered files into a Domino project to allow access by the Domino environment builder.

3. Create a new Domino environment that uses the uploaded files to enable connections to your cluster.

4. Enable YARN integration for the Domino projects that you want to use with the MapR cluster.

Domino supports the following types of connections to a MapR cluster:

* [FS shell for MapRFS](https://docs.ezmeral.hpe.com/datafabric-customer-managed/81/MapROverview/c_maprfs.html)

* [spark-shell](https://books.japila.pl/apache-spark-internals/)

* [spark-submit](https://spark.apache.org/docs/latest/submitting-applications.html)

* [pyspark](https://spark.apache.org/docs/latest/api/python/index.html)

* [YARN shell](https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/YarnCommands.html)

* [Hive with Beeline](https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-Beeline%E2%80%93CommandLineShell)

## Gather the required binaries and configuration files

You find most of the necessary files for setting up your Domino environment on your [MapR Edge node](https://docs.ezmeral.hpe.com/datafabric-customer-managed/81/AdvancedInstallation/PlanningtheCluster-node-types.html). To get started, [connect to the Edge node via SSH](https://docs.ezmeral.hpe.com/datafabric-customer-managed/81/AdvancedInstallation/PreparingEachNode-connectivity.html), then follow the steps below.

1. Create a directory named `hadoop-binaries-configs` at `/tmp`.

   ```shell theme={null}
   mkdir /tmp/hadoop-binaries-configs
   ```

2. Copy `hive-site.xml` from `/opt/mapr/spark/spark-<version>/conf` to `/tmp/hadoop-binaries-configs/`. Be sure to replace the `<version>` string in the command below with the number that matches the folder name on your edge node.

   ```shell theme={null}
   cp /opt/mapr/spark-<version>/conf /tmp/hadoop-binaries-configs/
   ```

3. Copy the `ssl_truststore` from `/opt/mapr/conf` to `/tmp/hadoop-binaries-configs/`.

   ```shell theme={null}
   cp /opt/mapr/conf/ssl_truststore /tmp/hadoop-binaries-configs/
   ```

4. After you’ve copied the above files into `/tmp/hadoop-binaries-configs`, zip the directory for transfer to your local machine.

   ```shell theme={null}
   cd /tmp
   tar -zcf hadoop-binaries-configs.tar.gz hadoop-binaries-configs
   ```

   Then use `SCP` from your local machine to download the zipped archive. After transfer, extract the files to your local filesystem and keep them handy for a future step where they are uploaded to Domino.

5. On the MapR edge node, run the following command to identify the version of Java running on the cluster.

   ```shell theme={null}
   java -version
   ```

   You must then [download a JDK .tar file from the Oracle downloads page](https://www.oracle.com/technetwork/java/javase/downloads/index.html) that matches that version. The filename has a name like the following:

   ```shell theme={null}
   jdk-8u211-linux-x64.tar.gz
   ```

   Keep this JDK handy for use in a future step.

## Upload the binaries and configuration files to Domino

Use the following procedure to upload the files you retrieved in the previous step to a public Domino project. This makes the files available to the Domino environment builder.

1. Log in to Domino, then create a new public project.

2. Open the **Code** page for the new project, then click **browse for files** and select the files you downloaded from the MapR edge node, and the JDK you downloaded from Oracle. Then click **Upload**.

3. From the **Code** page of your project, click **New File**. Name the file `run-client.sh`, and in its contents you must construct an invocation of the MapR `configure.sh` script that is valid for setting up a client to connect to your cluster. A full explanation of how to invoke this script is beyond the scope of this document. Read the full [documentation on the script from MapR](https://docs.ezmeral.hpe.com/datafabric-customer-managed/81/ReferenceGuide/configure.sh.html), and consider the following example.

   `run-client.sh`

   ```shell theme={null}
   #!/bin/bash
   /opt/mapr/server/configure.sh -N <clustername> -c -secure -C <host1>:7222,<host2>:7222,<host3>:7222 -HS <historyServer>
   ```

4. After your project contains the files from the MapR edge node, the correct JDK, and a `run-client.sh` script that wraps the MapR configuration script, click the gear menu next to each of those files, then right click **Download** and click **Copy Link Address**. Save these URLs in your notes, as you need them in the next step.

   After you have recorded the download URL of the binaries and configuration files, you’re ready to build a Domino environment for connecting to MapR.

## Create a Domino environment for connecting to MapR

1. Click **Environments** from the Domino main menu, then click **Create Environment**.

2. Give the environment an informative name, then choose a base environment that includes the version of Python that is installed on the nodes of your MapR cluster. Most Linux distributions ship with Python 2.7 by default, so you see the [Domino Analytics Distribution for Python 2.7](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) used as the base image in the following examples. Click **Create** when finished.

3. After creating the environment, click **Edit Definition**. Copy the below example into your Dockerfile instructions, then be sure to edit it wherever necessary with values specific to your deployment and cluster.

   In this Dockerfile, wherever you see a hyphenated instruction enclosed in angle brackets like `<paste-your-domino-download-url-here>`, be sure to replace it with the corresponding values you recorded in previous steps. You may also need to edit commands that follow to match downloaded filenames.

   ```dockerfile theme={null}
   # Base Image: quay.io/domino/base:Ubuntu16_DAD_Py2.7_R3.4-20180727
   USER root

   # Give the ubuntu user ability to sudo as any user including root in the compute environment
   RUN echo "ubuntu ALL=(ALL:ALL) NOPASSWD: ALL" >> /etc/sudoers

   # Set up directories
   RUN mkdir /tmp/mapr-cluster-downloads &&
      mkdir /usr/jdk64

   # Create a mapr user and group
   RUN groupadd -g 5000 mapr
   RUN useradd -u 5000 -g mapr mapr
   RUN usermod -s /bin/bash mapr

   # Use the following wget commands to download the four files you added to Domino in the previous section.
   # You should have copied down the URLs to download a JDK .tar, the two files from the edge node, and the run-client.sh script you created.
   # The example below will use a JDK file named jdk-8u112-linux-x64.tar.gz. If you're using a different version or have a different filename, replace it wherever it occurs.
   RUN cd /tmp/mapr-cluster-downloads &&
      wget <paste-your-run-client-dot-sh-download-url-here> -O /tmp/mapr-cluster-downloads/run-client.sh.gz &&
      wget <paste-your-hive-site-dot-xml-download-url-here> -O /tmp/mapr-cluster-downloads/hive-site.xml.gz &&
      wget <paste-your-jdk-tar-download-url-here> -O /tmp/mapr-cluster-downloads/jdk-8u112-linux-x64.tar.gz &&
      wget <paste-your-ssl-truststore-download-url-here> -O /tmp/mapr-cluster-downloads/ssl_truststore.gz &&
      gunzip run-client.sh.gz &&
      gunzip hive-site.xml.gz &&
      gunzip jdk-8u112-linux-x64.tar.gz &&
      gunzip ssl_truststore.gz &&
      cd ~

   # Install Java from the JDK
   RUN tar xvf /tmp/mapr-cluster-downloads/jdk-8u112-linux-x64.tar -C /usr/jdk64 &&
      ln -s /usr/jdk64/jdk1.8.0_112 /usr/jdk64/default
   ENV JAVA_HOME=/usr/jdk64/default
   RUN echo "export JAVA_HOME=/usr/jdk64/default" >> /home/ubuntu/.domino-defaults &&
      echo "export PATH=$JAVA_HOME/bin:$PATH" >> /home/ubuntu/.domino-defaults

   # Install mapr-client and Spark binaries from the MapR ubuntu repository.
   # These examples are for MapR 6.1.0.
   # If you are using a different version of MapR, replace these URLs with the correct versions from http://archive.mapr.com/releases/.
   RUN echo "deb https://package.mapr.com/releases/v6.1.0/ubuntu binary trusty" >> /etc/apt/sources.list
   RUN echo "deb https://package.mapr.com/releases/MEP/MEP-6.0.0/ubuntu binary trusty" >> /etc/apt/sources.list
   RUN wget -O - https://package.mapr.com/releases/pub/maprgpg.key | sudo apt-key add -
   RUN apt-get update
   RUN apt-get -y install mapr-client mapr-spark mapr-hive

   # Copy the ssl_truststore file from the /opt/mapr/conf directory on the cluster to the /opt/mapr/conf directory on the client
   RUN cp /tmp/mapr-cluster-downloads/ssl_truststore /opt/mapr/conf/

   # Make your customized script from the previous section executable
   RUN chmod +x /tmp/mapr-cluster-downloads/run-client.sh

   # Update SPARK and HADOOP environment variables.
   # Make sure the Spark and Hadoop version numbers match what is installed on your cluster
   # The examples below show Spark 2.3.1 and Hadoop 2.7.0.
   # If you are using different versions, be sure to edit the file and directory names to match.
   # Make sure the py4j file name is correct per your edgenode.
   ENV SPARK_HOME=/opt/mapr/spark/spark-2.3.1
   RUN echo "export HADOOP_HOME=/opt/mapr/hadoop/hadoop-2.7.0" >> /home/ubuntu/.domino-defaults &&
      echo "export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop" >> /home/ubuntu/.domino-defaults &&
      echo "export YARN_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop" >> /home/ubuntu/.domino-defaults &&
      echo "export SPARK_HOME=/opt/mapr/spark/spark-2.3.1" >> /home/ubuntu/.domino-defaults &&
      echo "export SPARK_CONF_DIR=/opt/mapr/spark/spark-2.3.1/conf" >> /home/ubuntu/.domino-defaults &&
      echo "export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip" >> /home/ubuntu/.domino-defaults

   # Change spark configuration directory permission as a new spark-defaults.conf file gets created by Domino's spark integration
   RUN chmod 777 /opt/mapr/spark/spark-2.3.1/conf

   # Add symlinks for Spark binaries
   RUN ln -s /opt/mapr/spark/spark-2.3.1/bin/pyspark /usr/bin/pyspark
   RUN ln -s /opt/mapr/spark/spark-2.3.1/bin/spark-shell/usr/bin/spark-shell
   RUN ln -s /opt/mapr/spark/spark-2.3.1/bin/spark-submit /usr/bin/spark-submit

   # Update Java path for R
   RUN export LD_LIBRARY_PATH=/usr/jdk64/default/jre/lib/amd64/server && R CMD javareconf

   # Install Python and R JDBC packages
   RUN pip install jaydebeapi
   RUN R --no-save -e 'install.packages(c("RJDBC"))'

   USER ubuntu
   ```

4. Scroll down to the **Pre Run Script** field and add the following lines, being sure to match the Spark version in the directory name to the one being set up by the Dockerfile instructions.

   ```shell theme={null}
   # Configure mapr-client with your customized script.
   sudo bash /tmp/mapr-cluster-downloads/run-client.sh
   # Copy `hive-site.xml` to the spark configuration directory
   # Be sure to match the Spark version in this folder name to match what you set up above.
   cp /tmp/mapr-cluster-downloads/hive-site.xml /opt/mapr/spark/spark-2.3.1/conf
   ```

5. (Optional) If you want to store and access [MapR user tickets](https://docs.ezmeral.hpe.com/datafabric-customer-managed/81/SecurityGuide/Tickets.html) as [Domino environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables), follow these additional steps.

   1. Request a long-running MapR ticket from your cluster administrator, and copy its contents to your local machine. The ticket is formatted as:

      `<cluster-name> <token>`

   2. Add that token as a [Domino environment variable](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables) to your Domino user account with the name `USERTICKET`.

   3. Add the following lines to the bottom of the **Pre Run Script** field for the environment you edited previously:

      ```shell theme={null}
      ## Write maprticket in environment variable to a file during runtime
      echo $USERTICKET > /tmp/maprticket_12574
      chown ubuntu:ubuntu /tmp/maprticket_12574
      chmod 600 /tmp/maprticket_12574
      ```

      If you do this, every user that wants to use this environment must set up a `USERTICKET` environment variable as described in the previous step.

6. Click **Build** when finished editing the Dockerfile instructions. If the build completes successfully, you are ready to try using the environment.

## Configure a Domino Project for use with a MapR cluster

This procedure assumes that an environment with the necessary client software has been created according to the instructions above. Ask your Domino administrator for access to such an environment.

1. Open the Domino project you want to use with your MapR cluster, then click **Settings** from the project menu.

2. On the Integrations tab, click to select **YARN** integration from the Apache Spark panel, then click **Save**. You should not need to edit any of the fields in this section.

3. On the **Hardware & Environment** tab, change the project default environment to the one you built earlier with the binaries and configuration files.

You are now ready to start Runs from this project that interact with your MapR cluster.


# Use PySpark in Jupyter Workspaces
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/pyspark-in-jupyter-workspaces

Add PySpark as a pluggable Workspace tool, configure a Spark-enabled Compute Environment, enable YARN, and launch interactive Jupyter sessions.

You can configure a [Domino workspace](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace) to launch a Jupyter notebook with a connection to [your Spark cluster](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark).

This allows you to operate the cluster interactively from Jupyter with [PySpark](https://archive.apache.org/dist/spark/docs/0.9.0/python-programming-guide.html).

The instructions for configuring a PySpark workspace are below. To use them, you must have a Domino environment that meets the following prerequisites:

* The environment must use one of the [Domino Standard Environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) as its base image.

* The necessary binaries and configurations for connecting to your Spark cluster must be installed in the environment. See [the provider-specific guides](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark) for setting up the environment.

<Note>
  PySpark 2 does not support Python 3.8 or higher. Build PySpark 2 compute environments from images with Python versions before 3.8 or use PySpark 3.
</Note>

## Add a PySpark Workspace option to your Environment

1. From the Domino main menu, click **Environments**.

2. Click the name of an environment that meets the prerequisites listed previously. It must use a Domino standard base image and already have the necessary binaries and configuration files installed for connecting to your spark cluster.

3. On the environment overview page, click **Edit Definition**.

4. In the **Pluggable Workspace Tools** field, paste the following YAML configuration.

   ```yaml theme={null}
   pyspark:
      title: "PySpark"
      start: [ "/var/opt/workspaces/pyspark/start" ]
      iconUrl: "https://raw.githubusercontent.com/dominodatalab/workspace-configs/develop/workspace-logos/PySpark.png"
      httpProxy:
         port: 8888
         internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{      rewrite: false
         requireSubdomains: false
      supportedFileExtensions: [ ".ipynb" ]
   ```

   When finished, the field should look like this:

   <img alt="Pluggable workspace tools" />

5. Click **Build** to apply the changes and build a new version of the environment. Upon a successful build, the environment is ready for use.

<Note>
  If you are using an older version of a [Domino Standard Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) you might require a different **Pluggable Workspace Tool** definition for PySpark. The safest way to do this is to copy the Jupyter pluggable workspace definition for your base image (see the `` README` `` for your base image at the [Domino Analytics Distribution](https://github.com/dominodatalab/Domino_Base_Images/tree/master/Domino_Analytics_Distribution) git repo), but replace Jupyter in the **title** and **start** fields with PySpark. You can use the same **iconUrl** specified above to get the correct PySpark icon.
</Note>

## Launching PySpark Workspaces

1. Open the project you want to use a PySpark workspace in.

2. Open the project settings, then follow the [provider-specific instructions from the Hadoop and Spark overview ](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark) on **setting up a project to work with an existing Spark connection environment**. This involves enabling YARN integration in the project settings.

3. On the **Hardware & Environment** tab of the project settings, choose the environment you added a PySpark configuration to in the previous section.

4. After the previous settings are applied, you can launch a PySpark workspace from the workspaces dashboard.

   <img alt="Pluggable workspace tools" />


# Use Spark on Domino
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/spark/index

Choose between Domino-managed on-demand Spark clusters and connections to external Hadoop and Spark clusters for data processing workloads.

Apache Spark is a fast and general-purpose cluster computing system that offers a unified analytics engine for large-scale data processing and machine learning.

Domino provides flexibility on how to use Spark. You can dynamically provision an on-demand Spark cluster orchestrated by Domino or you can connect to an existing Spark cluster outside of Domino.

[Hadoop and Spark](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark)\
Domino projects can use the environment to work with Hadoop applications.

[On-demand Spark](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark)\
Use Domino to dynamically provision and orchestrate a Spark cluster directly on the infrastructure that backs the Domino instance.

[Use spot instances](/cloud/platform-capabilities/features/compute-clusters-hpc/use-spot-instances)\
Spot instances reduce compute costs by using discounted cloud capacity. They’re ideal for fault-tolerant, stateless, or short-lived workloads like batch jobs, distributed training runs, and parallel analytics tasks.


# Configure Spark prerequisites
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/configure-prerequisites

Create compatible Spark cluster and PySpark execution Compute Environments, using default or custom Hadoop client libraries and Workspace tools.

Before you can start using on-demand Spark clusters on Domino, you must enable and configure the functionality on your deployment.

## Create a base Spark cluster environment

By default, Domino provides Spark compatible images, as listed in [compute Environment catalog](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/8-compute-environment-catalog), that can be used for the components of the cluster.

When using on-demand Spark in Domino, you need one environment for the Spark cluster (base or worker environment) and one environment for the workspace/job execution (compute environment).

### Create a new base Spark cluster environment

1. Follow the instructions to [create an environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/3-create-an-environment).

2. In the **Base Image** section, select **Custom Image** and specify an image URI that points to a deployable Spark image.

   Domino recommends that you use a Domino-provided Spark image from [compute Environment catalog](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/8-compute-environment-catalog) for versions of Spark and Python.

   <Note>
     Image compatibility:

     Domino currently republishes the Spark base images from `bitnami/spark`.

     Domino’s on-demand Spark functionality has been developed and tested using [open-source Spark images from Bitnami](https://hub.docker.com/r/bitnami/spark/).
   </Note>

3. Required: In the **Supported Clusters** area, select the **Domino managed Spark** checkbox.

   This ensures that the environment is available for use when you create Spark clusters from workspaces and jobs.

4. Set the **Visibility**.

   You can set this attribute the same way you would for any other [compute environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments).

5. Leave the **Dockerfile Instructions** blank to use the Hadoop client libraries included with the image or follow the instructions to configure [custom Hadoop client libraries](#base-spark-cluster-environment-advanced-custom-hadoop-client-libraries).

   Domino-provided images are installed with all pre-requisites.

   See [Manage dependencies](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/manage-dependencies) to learn more.

6. Leave **Pluggable Notebooks / Workspace Sessions** blank as the Spark base environments are not intended to also include notebook configuration.

### Base Spark cluster environment - default Hadoop client libraries

Leave the **Docker Instructions** section blank if you want a thin base image that only contains core Spark with the default Hadoop client libraries.

### Base Spark cluster environment (advanced) - custom Hadoop client libraries

The Hadoop client libraries, pre-bundled with your Spark version, might not be appropriate for your needs. This is common if you want to use cloud object store connector improvements introduced post Hadoop 2.7.

Add the following to the **Docker Instructions** section, and adjust the Spark and Hadoop version as needed:

```dockerfile theme={null}
### Needed if using the recommended Bitnami base image
USER root

### Make sure wget is available
RUN apt-get update && apt-get install -y wget && rm -r /var/lib/apt/lists /var/cache/apt/archives

### Modify the Hadoop and Spark versions below as needed
### NOTE: The HADOOP_HOME and SPARK_HOME locations should not be modified
ENV HADOOP_VERSION=3.1.1
ENV HADOOP_HOME=/opt/bitnami/hadoop
ENV HADOOP_CONF_DIR=/opt/bitnami/hadoop/etc/hadoop
ENV SPARK_VERSION=3.2.0
ENV SPARK_HOME=/opt/bitnami/spark
ENV PATH="$PATH:$SPARK_HOME/bin:$HADOOP_HOME/bin"

### Enable access to AWS and ADLS Gen2. Can modify as needed
ENV HADOOP_OPTIONAL_TOOLS="hadoop-aws,hadoop-azure,hadoop-azure-datalake"

### Remove the pre-installed Spark since it is pre-bundled with Hadoop but preserves the Python env
WORKDIR /opt/bitnami
RUN [ -d ${SPARK_HOME}/venv ] && mv ${SPARK_HOME}/venv /opt/bitnami/temp-venv
RUN rm -rf ${SPARK_HOME}

### Install the desired Hadoop-free Spark distribution
RUN wget -q https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-without-hadoop.tgz &&
    tar -xf spark-${SPARK_VERSION}-bin-without-hadoop.tgz &&
    rm spark-${SPARK_VERSION}-bin-without-hadoop.tgz &&
    mv spark-${SPARK_VERSION}-bin-without-hadoop ${SPARK_HOME} &&
    chmod -R 777 ${SPARK_HOME}/conf

### Restore the virtual Python environment
RUN [ -d /opt/bitnami/temp-venv ] && mv /opt/bitnami/temp-venv ${SPARK_HOME}/venv

### Install the desired Hadoop libraries
RUN wget -q http://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz &&
    tar -xf hadoop-${HADOOP_VERSION}.tar.gz &&
    rm hadoop-${HADOOP_VERSION}.tar.gz &&
    mv hadoop-${HADOOP_VERSION} ${HADOOP_HOME}

### Setup the Hadoop libraries classpath
RUN echo 'export SPARK_DIST_CLASSPATH="$(hadoop classpath):'"${HADOOP_HOME}"'/share/hadoop/tools/lib/*"' >> ${SPARK_HOME}/conf/spark-env.sh
ENV LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:$HADOOP_HOME/lib/native"

### This is important to maintain compatibility with Bitnami
WORKDIR /
RUN /opt/bitnami/scripts/spark/postunpack.sh
WORKDIR ${SPARK_HOME}

USER 1001
```

## Prepare your PySpark execution compute environment

You must configure the PySpark compute environments for workspaces and/or jobs that connect to your cluster.

Domino recommends that you use the pre-built base image to create a compatible workspace from [compute Environment catalog](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/8-compute-environment-catalog).

### Customize this Workspace compute environment

Use the image mentioned previously and add **Pluggable Workspace Tools**.

```yaml theme={null}
jupyter:
  title: "Jupyter (Python, R, Julia)"
  iconUrl: "/assets/images/workspace-logos/Jupyter.svg"
  start: [ "/opt/domino/workspaces/jupyter/start" ]
  supportedFileExtensions: [ ".ipynb" ]
  httpProxy:
    port: 8888
    rewrite: false
    internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
    requireSubdomain: false
jupyterlab:
  title: "JupyterLab"
  iconUrl: "/assets/images/workspace-logos/jupyterlab.svg"
  start: [  "/opt/domino/workspaces/jupyterlab/start" ]
  httpProxy:
    internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
    port: 8888
    rewrite: false
    requireSubdomain: false
vscode:
  title: "vscode"
  iconUrl: "/assets/images/workspace-logos/vscode.svg"
  start: [ "/opt/domino/workspaces/vscode/start" ]
  httpProxy:
    port: 8888
    requireSubdomain: false
rstudio:
  title: "RStudio"
  iconUrl: "/assets/images/workspace-logos/Rstudio.svg"
  start: [ "/opt/domino/workspaces/rstudio/start" ]
  httpProxy:
    port: 8888
    requireSubdomain: false
```


# On-demand Spark
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/index

Provision containerized Spark clusters for distributed machine learning, exploratory analysis, and data transformation in Workspaces or Jobs.

You can use Domino to dynamically provision and orchestrate a Spark cluster directly on the infrastructure that backs the Domino instance. This gives you quick access to Spark without having to rely on your IT team.

## Orchestrate Spark on Domino

Domino supports fully containerized execution of Spark workloads on the Domino Kubernetes cluster. You can interact with Spark through a [Domino workspace](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace) or in batch mode through a [Domino job](/cloud/platform-capabilities/core-concepts/jobs), as well as directly with [spark-submit](https://spark.apache.org/docs/latest/submitting-applications.html).

When you start a workspace or a job that uses an on-demand cluster, Domino orchestrates a cluster in [standalone mode](http://spark.apache.org/docs/latest/spark-standalone.html). The master and workers are newly deployed containers and the driver is your Domino workspace or job.

## Suitable use cases

The Domino on-demand Spark cluster is suitable for the following workloads:

**Distributed machine learning**\
[Spark](https://spark.apache.org/) provides a simple way to parallelize compute-heavy workloads such as distributed training. Spark comes with machine learning algorithms bundled in [MLlib](https://spark.apache.org/mllib/) for this purpose.

**Interactive exploratory analysis**\
Efficiently load a large dataset in a distributed manner to explore and understand the data using familiar query techniques with [Spark SQL](https://spark.apache.org/sql/).

**Featurization and data transformation *(for experienced Spark users)***\
Sample, aggregate, relabel, or otherwise manipulate large datasets to make them more suitable for analysis or training.

<Note>
  For optimal performance, you must have a cluster with sufficient resources and you must be adept at tuning your Spark application and writing performant Spark transforms.
</Note>

## Unsuitable use cases

The following are usage patterns that are presently not suitable for on-demand Spark on Domino:

**Stream processing pipeline**\
While Spark offers a stream processing engine, the ephemeral nature of the on-demand clusters on Domino makes it not a great fit for long-lived stream processing applications.

For such cases, consider using an externally managed Spark cluster.

**Collocated Spark and HDFS**\
The Domino on-demand clusters do not come with an HDFS installation and are generally not suitable for collocating data and compute.

Data in Domino clusters is intended to reside outside the cluster (for example, object store or Domino data set). When you want to use the cluster as long-term HDFS storage, consider using an externally managed Spark cluster.

**Data pipelines with strict performance SLA**\
While Domino orchestrates Spark on Kubernetes reliably, no extensive performance tuning or optimization has been performed. The cluster configuration and default context configuration parameters might not be optimized for such workloads.

<Note>
  If you intend to explore on-demand Domino spark clusters for such workloads, perform extensive validation and tuning of your jobs.
</Note>

## Next steps

* Find out more about the [Validated Spark version](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/validated-spark-version).

* Learn how to enable and configure the functionality on your deployment in [Configure Spark prerequisites](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/configure-prerequisites).

* Learn how to [create an on-demand Spark cluster](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/work-with-your-cluster) with the desired cluster settings attached to a Workspace or Job.

* Find out how you can [manage Spark dependencies](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/manage-dependencies).

* Learn how to [Access data with Spark](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/work-with-data).


# Manage Spark dependencies
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/manage-dependencies

Package Python libraries, JARs, and pinned versions in both Spark cluster and execution Compute Environments through Docker Instructions.

In a shared Spark cluster, it can be challenging for teams to manage their dependencies (for example, Python packages or JARs). Installing every dependency that a Spark application may need before it runs and dealing with version conflicts can be complex and time-consuming.

Domino allows you to easily package and manage dependencies as part of your [Spark-enabled compute environments](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/configure-prerequisites#create-a-base-spark-cluster-environment). This approach creates the flexibility to manage dependencies for individual projects or workloads without having to deal with the complexity of a shared cluster.

To add a new dependency, add the appropriate statements in the *Docker Instructions* section of the relevant Spark and execution compute environments.

For example to add `numpy`, include the following.

```dockerfile theme={null}
### Optionally specify version if desired
RUN pip install numpy
```

## Next steps

[Access data with Spark](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/work-with-data).


# Validated Spark version
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/validated-spark-version

Check how Domino validates Apache Spark for on-demand clusters and use the release notes to find the supported framework version for each release.

For each minor release (for example, Domino 4.4 or 4.5), Domino performs a set of validation tests against the **most recently released version** of Apache Spark. The purpose of the tests is to cover the integration surface between the Domino platform and the on-demand Spark clusters that the platform orchestrates. The testing is not intended to be a comprehensive test of Spark, as Domino does not modify that software.

See the [release notes](/release-notes/cloud-release) for more information on the validated framework for each Domino version.


# Access data with Spark
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/work-with-data

Access Domino Datasets, Amazon S3, Azure Data Lake Storage, and Kerberos credentials from Spark clusters with the required Hadoop connectors.

When using a Domino on-demand Spark cluster, any data that is used, created, or modified as part of the interaction must go into an external data store.

<Note>
  On-demand Spark clusters are not intended as a permanent store of any data or collocating a big data layer such as HDFS. Any data that is not stored externally from the cluster is lost upon termination.
</Note>

## Use Domino Datasets

When you create a Spark cluster attached to a Domino workspace or job, any [Domino dataset](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets) accessible from the workspace or job is also accessible from all components of the cluster under the same dataset mount path. Data can be accessed using the following path prefix:

```shell theme={null}
file:///
```

For example, to read a file you would use the following.

```python theme={null}
rdd = sc.textFile("file:///path/to/file")
```

No additional configuration of the Spark cluster environment or the execution environment is required.

## Use S3

To enable working with data in Amazon S3 (or S3 compatible object store) you must ensure that your [base Spark cluster environment](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/configure-prerequisites#create-a-base-spark-cluster-environment) and compatible [PySpark compute environment](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/configure-prerequisites#prepare-your-pyspark-execution-compute-environment) are configured with the [Hadoop-AWS module](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html).

The environments created when [configuring prerequisites](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/configure-prerequisites) include at a minimum Hadoop 2.7.3 client libraries, which are sufficient for basic access. A number of additional commonly used features (for example, temporary credentials, SSE-KMS encryption, more efficient committers, etc.) are only available in more recent Hadoop-AWS module versions.

Consult the documentation for the relevant version to determine what may be the best fit for you.

* [Hadoop-AWS Module 2.7.3](https://hadoop.apache.org/docs/r2.7.3/hadoop-aws/tools/hadoop-aws/index.html)

* [Hadoop-AWS Module 2.8.5](https://hadoop.apache.org/docs/r2.8.5/hadoop-aws/tools/hadoop-aws/index.html)

* [Hadoop-AWS Module 2.9.2](https://hadoop.apache.org/docs/r2.9.2/hadoop-aws/tools/hadoop-aws/index.html)

* [Hadoop-AWS Module 3.1.3](https://hadoop.apache.org/docs/r3.1.3/hadoop-aws/tools/hadoop-aws/index.html)

* [Hadoop-AWS Module 3.2.1](https://hadoop.apache.org/docs/r3.2.1/hadoop-aws/tools/hadoop-aws/index.html)

For Spark 3.1.1, a good advanced option would be Hadoop 3.2.0 or later.

### S3 usage examples

Now that you have your environments properly setup, you can interact with S3. The following are several common access patterns.

**Access bucket with AWS credentials in environment variables**

```python theme={null}
import os
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

# the default configuration will pick up your credentials from environment variables
# No additional configuration is necessary

# test reading
df = spark.read.json("s3a://bucket/prefix1/prefix2/people.json")
df.show()
```

**Access bucket with SSE-KMS encryption**

<Note>
  Requires Hadoop-AWS 2.9.2+
</Note>

```python theme={null}
import os
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

# for write operations you will need the ARN of the key to use
# Note that the credentials used need to have proper access to use the key
kms_key_arn = "<your key ARN here>"

# configure the connector
# This example assumes credentials from environment variables so no need to configure
# Note: The encryption config is not needed for read only operations
hadoop_conf = spark.sparkContext._jsc.hadoopConfiguration()
hadoop_conf.set("fs.s3a.server-side-encryption-algorithm", "SSE-KMS")
hadoop_conf.set("fs.s3a.server-side-encryption.key", kms_key_arn)

# test reading
df = spark.read.json("s3a://bucket/prefix1/prefix2/people.json")
df.show()

# test writing
df.write.mode("overwrite").parquet("3a://bucket/prefix1/prefix2/write-test/output")
```

**Access a bucket with Domino assumed temporary credentials**

<Note>
  Requires Hadoop-AWS 2.9.2+

  It is important that no AWS credential variables are set in your user profile or project.
</Note>

```python theme={null}
import os
from pyspark.sql import SparkSession

try:
    spark.stop()
except:
    pass
spark = SparkSession.builder.getOrCreate()

#The name of one of the roles you are entitled to
profile_name="my-role-name-read-write"

# use boto3 for convenience to get credentials form credentials file populated by Domino
# can use any method desirable to extract the credentials
import boto3
role_creds = boto3.Session(profile_name=profile_name).get_credentials().get_frozen_credentials()

# configure the connector
# Use the TemporaryAWSCredentialsProvider
hadoop_conf = spark.sparkContext._jsc.hadoopConfiguration()
hadoop_conf.set("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider")
hadoop_conf.set("fs.s3a.access.key", role_creds.access_key)
hadoop_conf.set("fs.s3a.secret.key", role_creds.secret_key)
hadoop_conf.set("fs.s3a.session.token", role_creds.token)

# test reading
df = spark.read.json("s3a://bucket/prefix1/prefix2/people.json")
df.show()

# test writing
df.write.mode("overwrite").parquet("s3a://bucket/prefix1/prefix2/write-test/output")
```

For a full set of configuration options, see the documentation for the [Hadoop-AWS module](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html).

## Using Azure Data Lake Storage Gen2

To enable working with data in Azure Data Lake Storage (ADLS) Gen2 you need to configure your base Spark environment and your compute environment with the [Hadoop-Azure ABFS connector](https://hadoop.apache.org/docs/current/hadoop-azure/#Features_of_the_ABFS_connector.).

The ABFS connector requires Hadoop 3.2+.

To accomplish this, set `SPARK_VERSION=3.0.0` and `HADOOP_VERSION=3.2.1` when following the advanced instructions for [base Spark cluster environment](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/configure-prerequisites#prepare-your-pyspark-execution-compute-environment) and compatible [PySpark compute environment](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/configure-prerequisites#prepare-your-pyspark-execution-compute-environment).

<Note>
  You must also enable the `ENV HADOOP_OPTIONAL_TOOLS=hadoop-azure` directive in your environments.
</Note>

## Access AWS resources from a Spark cluster

<Note>
  This feature requires Hadoop 2.9.2 or higher. If you’re using an earlier version of Hadoop, you’ll need to [configure your base Spark environment](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/configure-prerequisites#base-spark-cluster-environment-advanced-custom-hadoop-client-libraries) or your [PySpark environment](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/configure-prerequisites#prepare-your-pyspark-execution-compute-environment) to use Hadoop 2.9.2+.
</Note>

You can configure on-demand Spark clusters in your Domino workspace to access AWS resources using temporary credentials issued by AWS. To do this, your Domino deployment must use single sign-on (SSO) with a trusted identity provider (IdP). The credentials can also be continuously refreshed, allowing your Spark cluster to have continuous access to AWS resources.

The specific credentials (and associated privileges) issued by AWS to your Spark cluster are based on role profiles defined in your IdP by a Domino administrator. These profiles include identity attributes used by AWS to issue appropriate temporary credentials corresponding to a role profile. The temporary credentials are then automatically distributed to your on-demand Spark cluster.

To take advantage of this feature, you can either configure your Spark context dynamically to work with profile role credentials in your code, or configure the desired profile in your project settings. Both methods are described below and you should select the option that best matches your use case.

**Configure your Spark context dynamically in your code**

This method provides you with more flexibility and is recommended if you need to frequently change role profiles. Recall that your Spark clusters must use Hadoop 2.9.2 or higher and may need to be configured accordingly prior to implementing the following code snippet.

```python theme={null}
import os
from pyspark.sql import SparkSession

try:
    spark.stop()
except:
    pass

# First, set the AWS_PROFILE environment variable to the name of the profile found in the credentials file $AWS_SHARED_CREDENTIALS_FILE.
# If you're doing this in a notebook, first stop your Spark session or context for the change to take effect.
os.environ['AWS_PROFILE'] = 'name-of-profile-to-use'

# Next, configure the Spark connector by setting up the provider type and the name of the profile to use. Be sure to replace the .appName() argument with the name of your app.
spark = SparkSession.builder
          .appName("Credential Spark Test")
          .config("spark.hadoop.fs.s3a.aws.credentials.provider", "com.amazonaws.auth.profile.ProfileCredentialsProvider")
          .config("spark.executorEnv.AWS_PROFILE", os.environ['AWS_PROFILE'])
          .getOrCreate()

# Read some data from AWS (replace with your S3 URI)
df = spark.read.json("s3a://foobar/bazbux.json")
df.show()
```

**Configure your Spark context from your Domino project settings**

You can also enable this feature by adding Spark configuration options in your Domino project settings. This method provides less flexibility and is recommended for projects that utilize one consistent role profile. To enable this feature:

1. Go to **Settings** in your Domino project.

2. Click the **Integrations** tab.

3. In the **Apache Spark mode** section, select **Domino managed on-demand cluster**.

4. In the **Spark Configuration Options** text area, add the keys and values specified below. Ensure one whitespace between the key and the value.

   ```
   spark.hadoop.fs.s3a.aws.credentials.provider com.amazonaws.auth.profile.ProfileCredentialsProvider
   spark.executorEnv.AWS_PROFILE name-of-profile-to-use
   ```

5. Go to your Domino account settings and click **User Environment Variables**. Under **Set user environment variable**, set **Name** to `AWS_PROFILE` and set **Value** to the name of the profile you’d like to use (`name-of-profile-to-use` in the previous step). Click **Set Variable**.

## Kerberos keytab propagation

When Kerberos authentication is enabled either in user settings or in project settings, the uploaded keytab is automatically distributed to all cluster containers at a well-known location.

By default, the keytab is available at: `/etc/security/keytabs/keytab`

Alternatively, a Domino administrator can change the path where the keytab is available. For more information, please reach out to Domino Support.


# Work with your Spark cluster
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/work-with-your-cluster

Create Spark clusters for Workspaces and Jobs, configure executors and autoscaling, connect with PySpark, and monitor workloads in the Spark UI.

Create an on-demand Spark cluster with the desired cluster settings attached to a Workspace or Job.

## Create an on-demand Spark cluster attached to a Domino Workspace

1. Click **New Workspace**.

2. From Launch New Workspace, select **Compute Cluster**.

3. Specify the [cluster settings](#understand-your-cluster-settings) and launch you workspace. After the workspace is running, it has access to the Spark cluster you configured.

The hardware tier for your workspace determines the compute resources available to your Spark driver process.

## Create a cluster with Jobs

1. From the **Jobs** menu, click **Run**.

2. From **Start a Job** select **Compute Cluster**.

3. Specify the [cluster settings](#understand-your-cluster-settings) and launch your job. The job has access to the Spark cluster you configured.

You can use any Python script that contains a PySpark job.

You can also use `spark-submit` to submit jobs. However, this is not recognized automatically as a Domino supported job type, so you must wrap it with a shell script unless you included a copy of `spark-submit.sh` when you [prepared your compute environment](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/configure-prerequisites#prepare-your-pyspark-execution-compute-environment).

The following is an example of a simple wrapper `my-spark-submit.sh`.

```bash theme={null}
#!/usr/bin/env bash

spark-submit $@
```

## Understand your cluster settings

Domino makes it simple to specify key settings when creating a Spark cluster.

* **Min Executors**

  Number of Executors that are available to your Spark application when the cluster starts. If **Auto-scale workers** is not enabled, this is always the size of the cluster. The combined capacity of the executors is available for your workloads.

  When you instantiate Spark context with the default settings, the `spark.executor.instances` [Spark setting](https://spark.apache.org/docs/latest/configuration.html) is set to the number specified in the previous window.

* **Max Executors**

  The maximum number of Spark executors that the cluster can reach when **Auto-scale workers** is enabled. See [Cluster autoscaling](#cluster-auto-scaling) for more details.

* **Cluster size: Limit**

  The maximum number of executors that you can make available to your cluster is limited by the number of per-user executions that your Domino administrator has configured for your deployment.

  In addition to the number of Spark application executors, you need 1 slot for your cluster master and 1 slot for your workspace or job.

* **Executor Hardware Tier**

  The amount of compute resources (CPU and memory) that is made available to Spark executors.

  When you instantiate Spark context with the default settings, the `spark.executor.cores` and `spark.executor.memory` [Spark settings](https://spark.apache.org/docs/latest/configuration.html) are set to the values corresponding to the Hardware Tier.

  For cores, the number of cores is rounded up to the nearest integer. For example, a Hardware Tier with 1.5 cores results in `spark.executor.cores` set to 2 when creating the default context.

* **Master Hardware Tier**

  Same mechanics as the Executor Hardware Tier, but applied to the resources that are available for your Spark cluster master.

  The master in the Domino configuration has a pretty limited role in Spark application scheduling, and it does not need a significant amount of resources.

  By default, any Hardware Tier is available when selecting resources for your Executor and Master. Domino administrators can optionally configure compute cluster [dedicated hardware tiers](/cloud/admin/infrastructure-and-compute/manage-compute-resources).

* **Cluster compute environment**

  Designates your environment. Projects already using this environment are allowed to continue to use it until a new project environment is set.

* **Dedicated local storage per executor**

  The amount of dedicated storage in Gigabytes (2^30 bytes) that is available to each executor.

  The storage is automatically mounted to /tmp which is the default location for `spark.local.dir` that serves as the location for RDDs, when they need to be on disk, and for shuffle data.

  The storage is automatically provisioned when the cluster is created and de-provisioned when it is shut down.

  <Warning>
    The local storage per executor should not be used for storing any data which needs to be available after the cluster is shut down.
  </Warning>

## Advanced cluster configuration topics

### Spark executors versus workers

Domino gives you the ability to have flexible resource allocation through Hardware Tiers which makes it convenient to plan your cluster resources needed for your application in terms of executors as opposed to workers. When creating on-demand Spark clusters, Domino automatically manages the relationship between Spark executors and workers (see [Spark definitions](https://spark.apache.org/docs/latest/cluster-overview.html#glossary)). By default, there is one worker for each executor.

### Resource overhead

When setting up worker and executor memory limits, Domino ensures that the worker containers have additional memory on top of the Hardware Tier memory definition to account for the recommended overhead of 10% of executor memory with a minimum of 384 MB. The usable executor memory is always based on the value from the actual Hardware Tier.

You might have to take this into account when designing Hardware Tiers to achieve optimal packing of Spark worker containers on the physical instances of the Domino Kubernetes cluster.

Because of the worker memory overhead and the approach of using 1 worker per executor, it is possible that using lots of executors with smaller Hardware Tiers may result in excessive memory overhead.

If this is undesirable, an experienced data scientist can choose a large Hardware Tier when creating the cluster and then overwrite the executor core and memory settings to achieve the desired effect. See [Adding and overwriting parameters](#add-and-overwrite-parameters).

## Cluster auto-scaling

Cluster auto-scaling allows you to start with a small cluster, which then automatically scales up and down in response to the resource consumption of your workload. This approach utilizes resources more efficiently for bursty workloads.

By default, the cluster size increases when the average CPU utilization of your workload reaches 80%. A Domino administrator can further refine the auto-scaling settings by including memory utilization or changing the desired scaling thresholds.

Scale down happens if resource utilization is low for a period of at least 5 minutes. Note that depending on the workload that you are executing, scale down may terminate cluster nodes that contain intermediate results which would need to be recomputed.

## Connect to your cluster

When provisioning your on-demand Spark cluster, Domino sets up key default cluster configuration parameters (for example, `spark.master`, `spark.driver.host`) to the appropriate cluster URLs so that creating SparkSession or SparkContext with default context properly connects the cluster to your workspace or job.

### Create SparkSession

You can create `SparkSession` using the following:

```python theme={null}
from pyspark.sql import SparkSession

spark = SparkSession
        .builder
        .appName("MyAppName")
        .getOrCreate()

 # You can examine the full config
spark.sparkContext.getConf().getAll()
```

### Create SparkContext

Alternatively, you can create `SparkContext` using the following:

```python theme={null}
from pyspark import SparkContext

conf = SparkConf().setAppName(appName).setMaster(master)
sc = SparkContext(conf=conf)
```

### Add and overwrite parameters

It is possible to add additional config parameters or overwrite any of the defaults by manipulating your context configuration.

```python theme={null}
from pyspark.sql import SparkSession

spark = SparkSession
        .builder
        .appName("MyAppName")
        .config("spark.some.config.option", "some-value")
        .getOrCreate()
```

You can also supply additional configuration settings that apply for any clusters associated with a Domino project from **Settings** > **Integrations** > **Domino managed on-demand cluster**

<Warning>
  You should not overwrite the default configuration values for `spark.master`, `spark.driver.host` since you may lose proper connectivity between your cluster and workspace or job.
</Warning>

## Access the Spark UI

Spark provides a suite of [web user interfaces](https://spark.apache.org/docs/latest/web-ui.html) that you can use to monitor the status and resource consumption of your Spark cluster.

Domino makes the Spark web UI available for active on-demand clusters attached to both workspaces and jobs.

<Note>
  In a hybrid Domino deployment, if the [data plane](/cloud/platform-capabilities/core-concepts/data-plane) is not configured for workspaces, then the user interface link is disabled. An administrator can [Enable a data plane for workspaces](/cloud/admin/platform-configuration/manage-data-planes#enable-a-data-plane-for-workspaces-and-datasets).
</Note>

### Spark UI from Workspaces

The Spark UI is available from a dedicated tab in your workspace.

### Spark UI from Jobs

The Spark UI is also available for running jobs from **Details** tab.

### Spark UI compatibility mode

With Domino 4.5 and higher, if an on-Demand Spark cluster uses Spark 3.0.1 or earlier, some links in the Spark UI do not function properly. This does not affect actual Spark processing.

For cases where you want to run an older version, you can configure Domino in Spark web UI compatibility mode using one of the following options:

* **Set for the entire Domino instance**

  A Domino administrator can set the Domino\`com.cerebro.domino.computegrid.computeCluster.spark.proxyCompatibility\` configuration records setting to `legacy`.

* **Set on a Domino Project level**

  Alternatively, this can be set at a project level from **Settings** > **Integrations** > **Domino managed on-demand cluster**

  <Warning>
    When Compatibility mode is set, the UI for Spark 3.1.1 and higher does not function properly.
  </Warning>

For more information, please reach out to Domino Support.

## Cluster lifecycle

On workspace or job startup, a Domino on-demand Spark cluster with the desired [cluster settings](#understand-your-cluster-settings) is automatically provisioned and attached to the workspace or job as soon as the cluster becomes available.

On workspace or job termination, the on-demand Spark cluster and all associated resources are automatically terminated and de-provisioned. This includes any compute resources and storage allocated for the cluster.

## Cluster network security

The on-demand Spark clusters created by Domino are not meant for sharing between multiple users. Each cluster is associated with a given workspace or a job instance. Access to the cluster and the Spark web UI is restricted only to users who can access the workspace or the job attached to it. This restriction is enforced at the networking level and the cluster is only reachable from the execution that provisioned it.


# Develop and train models
Source: https://docs.domino.ai/cloud/platform-capabilities/features/development/develop-train-models

Develop models in Workspaces, schedule training Jobs, track experiments, collaborate on results, and version code, data, and dependencies.

With Domino, you have the freedom to develop models and analytics solutions using the tools and libraries of your choice. As a result, there are many paths to development in Domino. In this overview, you learn about a model development workflow that uses interactive workspaces.

A typical data science workflow for model training iterates through these steps:

1. Import, visualize, and prepare data.

2. Explore model training approaches.

3. Review and evaluate results with collaborators.

4. Formalize your model through deployment, automation, or training on larger datasets.

<img alt="Data science workflow" />

Domino closely aligns with this workflow. Use interactive workspaces to prepare data and explore modeling approaches. Use experiment tracking to review results, then use automated jobs to automate the entire workflow.

## Launch a Workspace

Workspaces are interactive code environments used to write and iterate on code directly in your browser. By default, Domino provides Jupyter Notebooks, JupyterLab, RStudio, and VSCode workspaces. However, you can configure workspaces to use custom IDEs too.

When you [launch a workspace](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace), the configuration wizard guides you through the setup process.

Select the following settings for your workspace:

* **Compute environment** - Standard compute environments ensure consistency for repeatable results. To get started, use the [Domino Standard Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments). Eventually, you may want to [customize your environment](/cloud/platform-capabilities/core-concepts/compute-environments) to use additional libraries.

  <Note>
    Your environment can be a single machine or a [cluster of machines for distributed computing](/cloud/platform-capabilities/features/development/distributed-training#enable-and-configure-cluster-deployments).
  </Note>

* **Hardware tier** - Specify the CPU, memory, and GPU for the pods that host Domino. Your [Domino administrator defines hardware tiers](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers) for you.

* **IDE** - By default, Domino comes with popular IDEs like Jupyter Notebooks, JupyterLab, [VS Code](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace/launch-a-vs-code-workspace), and RStudio. However, you can [configure workspaces to use custom IDEs](/cloud/platform-capabilities/core-concepts/compute-environments/work-with-pluggable-workspaces/add-workspace-ides) too.

## Develop your model or solution

After you launch a Workspace, write the code to develop your solution.

Each Project is different, and one of Domino’s greatest strengths is its flexibility to accommodate the tools that you want to use while also providing platform tools like [data source connectors](/cloud/platform-capabilities/core-concepts/data/data-source-connectors) and [Dataset management](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets) to save you time and ensure consistency.

<Tip>
  Commit your code to Git frequently while you work so that you can easily revert to a previous version or share specific versions of your code with collaborators. See [Use Git in your workspace](/cloud/platform-capabilities/core-concepts/workspaces/use-git-in-your-workspace) for details.
</Tip>

## Schedule a training Job

While you develop your model in an interactive workspace, you can train it on a small dataset. To train it on a larger dataset, schedule a training job. Training jobs run outside of your development workspace, on a single machine or a distributed computing cluster, depending on the size of your data and the complexity of your model.

Jobs are also useful for re-training your model on a regular basis to ensure ongoing accuracy. For example, you can schedule a daily job that trains your model on the latest data from the previous 24 hours.

See [Schedule jobs](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job) for the steps to schedule a one-time or recurring job.

## Manage experiments and collaborate

Use [built-in MLflow tracking](/cloud/platform-capabilities/features/development/track-monitor) to track experiment runs. Use Domino’s [collaboration tools](/cloud/platform-capabilities/features/collaboration) to evaluate results and approve models for production.

## Best practices for model reproducibility

Versioning is the key to reproducibility. Follow these best practices to save time and ensure consistency across similar projects and jobs:

* Create custom environments to manage dependencies, instead of installing dependencies at the command line.

* Before modifying a dataset, create a snapshot so that you can always point to a previous version of the data. See [Create a snapshot of a dataset](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots) for instructions.

* Commit your code to Git before running any executions (even in your workspace), so that each execution is associated with a specific commit.

* For complex analysis or operations on big data, use jobs instead of workspaces.

## Next steps

After you’ve developed a model or analytics solution, Domino can help you operationalize your work for deployment or sharing.

If you’re training a model, you may want to learn how to:

* [Deploy your model](/cloud/platform-capabilities/features/models)

* [Schedule recurring training jobs](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job)

* [Scale-out training for larger datasets](/cloud/platform-capabilities/features/development/distributed-training)

If you’re performing data analytics or data engineering, you may want to learn how to:

* [Publish apps](/cloud/platform-capabilities/features/apps)

* [Save datasets](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets)


# Scale distributed workloads
Source: https://docs.domino.ai/cloud/platform-capabilities/features/development/distributed-training

Configure Spark, Ray, Dask, or MPI clusters for interactive and Job-based distributed training, with guidance for choosing each framework.

Use Domino to create on-demand Spark, Dask, Ray, or MPI compute clusters to speed up computationally intensive jobs. Execute your jobs in any cloud or on-prem cluster to preserve data locality and optimize spend.

This article contains an overview and examples for compute clusters in Domino. Learn how to do the following:

* Enable clusters in your Domino deployment.

* Use Domino to orchestrate distributed and parallel training workloads.

## Enable and configure cluster deployments

Before you use on-demand clusters, enable them in your workspace and create a base cluster image:

* [Configure Spark clusters](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/configure-prerequisites) for your Domino deployment.

* [Configure Ray clusters](/cloud/platform-capabilities/features/compute-clusters-hpc/ray/configure-prerequisites) for your Domino deployment.

* [Configure Dask clusters](/cloud/platform-capabilities/features/compute-clusters-hpc/dask/configure-prerequisites) for your Domino deployment.

* [Configure MPI clusters](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/configure-mpi-prerequisites) on your Domino deployment.

## Distributed and parallel training

Generally, there are two ways you can use compute clusters to train models in Domino:

* As the compute environment for interactive workspace such as Jupyter Notebooks (or any other IDE) running on top of the cluster.

* As a job-based compute cluster that executes a training script or job you define.

Typically, interactive workspaces are used to explore datasets and training approaches. In contrast, use the job-based method after you’ve developed a training approach and want to repeat it.

Select the cluster type to learn more. For more information on choosing a cluster type, see our blog post [Spark, Dask, and Ray: Choosing the right framework](https://www.dominodatalab.com/blog/spark-dask-ray-choosing-the-right-framework).

<Tabs>
  <Tab title="Spark">
    [Spark](https://spark.apache.org/) provides a simple way to parallelize compute-heavy workloads such as distributed training. Spark benefits iterative training algorithms or multi-threaded tasks over large data sets.

    Domino supports fully containerized executions of Spark workloads on the Domino Kubernetes cluster. You can interact with Spark through Domino in the following ways:

    * Use Spark in an [interactive workspace](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/work-with-your-cluster).
    * Use Spark in [batch mode through a Domino job](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/work-with-your-cluster).
    * Directly with [spark-submit](https://spark.apache.org/docs/latest/submitting-applications.html).

    When you start a workspace or a job that uses an on-demand cluster, Domino orchestrates a cluster in standalone mode. The master and workers are newly deployed containers, and the driver is your Domino workspace or job.

    See the [Spark quickstart project](https://github.com/dominodatalab/spark-quickstart-winequality) to walk through environment setup, project creation, and model training.

    Domino also provides access to GPU-accelerated backend compute for the Spark workers. Combined with the [RAPIDS Accelerator for Spark](https://rapids.ai/), you can enable GPU-accelerated processing on the Spark worker nodes. For more information, see the [Webinar for GPU-accelerated Spark and RAPIDS](https://github.com/dominodatalab/webinar-gpu-accelerated-spark-and-rapids).
  </Tab>

  <Tab title="Ray">
    [Ray](https://www.ray.io/) is an extensive distributed computing library of tools that reduces training time for parallel and distributed applications. You can interact with Ray through a Domino in the following ways:

    * In an [interactive workspace](/cloud/platform-capabilities/features/compute-clusters-hpc/ray/work-with-your-cluster).
    * In [batch mode through a Domino job](/cloud/platform-capabilities/features/compute-clusters-hpc/ray/work-with-your-cluster).

    Ray provides:

    * Simple primitives for building and running distributed applications.
    * Tools that allow users to parallelize single machine code with little to zero code changes.
    * Applications, libraries, and tools for deep learning.

    See the [Ray quickstart project](https://github.com/dominodatalab/domino-quickstart-ray) to walk through environment setup and multiple types of training algorithms.
  </Tab>

  <Tab title="Dask">
    [Dask](https://www.dask.org/) provides a drop-in replacement to train parallelizable scikit-learn models at scale. You can run Dask in single-node and distributed modes to enable parallelization and distribution. You can interact with Dask through a Domino in the following ways:

    * Use Dask in an [interactive workspace](/cloud/platform-capabilities/features/compute-clusters-hpc/dask/work-with-your-cluster).
    * Use Dask in [batch mode through a Domino job](/cloud/platform-capabilities/features/compute-clusters-hpc/dask/work-with-your-cluster).

    See the [Dask quickstart project](https://github.com/dominodatalab/domino-quickstart-dask) to walk through environment setup and multiple types of training algorithms.

    See the [Dask reference project](https://github.com/dominodatalab/reference-project-dask) for an example of using Dask in Domino to train a model. This project covers environment setup, data pipelines, and model training. Focus on [`dask_ml.ipynb`](https://github.com/dominodatalab/reference-project-dask/blob/main/dask_ml.ipynb).
  </Tab>

  <Tab title="MPI">
    [MPI](https://www.mpi-forum.org/) provides a communication protocol for distributed parallel computing. Open MPI is ideal for distributed multi-GPU and multi-CPU training for Tensorflow, PyTorch, Keras, or MXNet models. MPI clusters have lower overhead than other distributed computing systems and are highly customizable.

    You can interact with Dask through a Domino in the following ways:

    * Use MPI in an [interactive workspace](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/work-with-your-cluster#create-an-mpi-cluster-with-workspaces).
    * Use MPI in [batch mode through a Domino job](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/work-with-your-cluster#create-an-mpi-cluster-with-jobs).
  </Tab>
</Tabs>

## Next steps

Now that you know the concepts behind using Spark, Dask, and Ray to configure clusters for jobs, see how to [Tune Models with Ray Tune](/cloud/platform-capabilities/features/development/ray-tune).


# Develop models
Source: https://docs.domino.ai/cloud/platform-capabilities/features/development/index

Develop models with familiar libraries and IDEs, trace GenAI runs, track experiments, scale workloads, tune hyperparameters, and schedule Jobs.

This article introduces training machine learning models in Domino. See the links below to learn about Domino’s benefits and model development concepts.

## Train with your favorite libraries and IDEs

Domino is a flexible platform that’s designed to work with your favorite tools:

* [Develop and train models](/cloud/platform-capabilities/features/development/develop-train-models) with open source libraries and your favorite IDEs.

* [Capture your GenAI runs](/cloud/platform-capabilities/features/agents/develop) with tracing to see how a GenAI experiment runs step by step.

* Implement MLflow to [track and monitor experiments](/cloud/platform-capabilities/features/development/track-monitor) in R and Python.

* [Scale distributed workloads](/cloud/platform-capabilities/features/development/distributed-training) with Dask, Ray, MPI, and Spark.

* [Tune hyperparameters at scale with Ray Tune](/cloud/platform-capabilities/features/development/ray-tune) and other open source libraries.

## Manage experiments and collaborate with others

Domino helps you manage experiments and models across your organization:

* [Track and monitor experiments](/cloud/platform-capabilities/features/development/track-monitor) to see logs, outputs, and more.

* [Schedule jobs](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job) for recurring processes.

* [Develop projects with Git](/cloud/platform-capabilities/core-concepts/projects/git-based-projects) version control.

## Next steps

After you understand how to train models in Domino, learn how to [Deploy Models and Apps](/cloud/platform-capabilities/features/models).


# Tune hyperparameters with Ray Tune
Source: https://docs.domino.ai/cloud/platform-capabilities/features/development/ray-tune

Distribute hyperparameter tuning with Ray Tune, log trials to MLflow, compare metrics and parameters, and export experiment results.

Use Ray Tune and Domino to expedite hyperparameter tuning by distributing it at scale. While you have the freedom to use any library in Domino, this guide uses [Ray Tune](https://docs.ray.io/en/latest/tune/index.html) and [MLflow](https://mlflow.org/) to tune models and evaluate results.

## Prerequisites

Domino helps you manage Ray cluster infrastructure with on-demand clusters, but before you use Ray clusters, you must first [setup Ray clusters in Domino](/cloud/platform-capabilities/features/compute-clusters-hpc/ray/configure-prerequisites).

## Import and run the training script

In this example, you tune a PyTorch Lightning classifier on the MNIST dataset. You’ll use Ray Tune to run permutations on layer size, learning rate, and batch size. Import the [RayTune GitHub reference project](https://github.com/ddl-jwu/ray-tune) into Domino:

1. In the Domino navigation pane, click **Projects** > **New Project**.

2. In the **Create New Project** window, enter a name like "Ray Tune example".

3. Under **Hosted By**, select **Git Service Provider** > **GitHub**

4. For **Git Repository URL**, enter "[https://github.com/ddl-jwu/ray-tune](https://github.com/ddl-jwu/ray-tune)".

5. In the project, [launch a Ray-enabled VSCode Workspace](/cloud/platform-capabilities/features/compute-clusters-hpc/ray/work-with-your-cluster#create-a-cluster-with-workspaces) and load **train.py** to explore the training script.

6. In VSCode, open a terminal and type `python train.py` to run the script.

The following code snippet from train.py configures the RayTune parameter grid and gets the MLflow tracking URI that comes pre-configured for Domino projects:

```Python theme={null}
def tune_mnist(num_samples=4, num_epochs=5, gpus_per_trial=0, data_dir="~/data"):
    config = {
        "layer_1_size": tune.choice([16, 32, 64]),
        "layer_2_size": tune.choice([32, 64, 128]),
        "lr": tune.loguniform(1e-4, 1e-1),
        "batch_size": tune.choice([32, 64]),
        "mlflow": {
            "experiment_name": "ray-tuning",
            "tracking_uri": mlflow.get_tracking_uri()
        }
    }
```

Each permutation of the parameters produces a single run, and all runs are grouped under the same experiment. One epoch covers an entire data pass through the neural network during training. Logging is done at the epoch level. For more information on experiment tracking, see [Track and monitor experiments with MLflow](/cloud/platform-capabilities/features/development/track-monitor).

## Run the training script as a job

You can also run the training script as a job to remove the workspace creation steps. Jobs are useful automation mechanisms in Domino.

1. In the project you imported from GitHub in the previous section, click **Jobs**.

2. Configure a run with **train.py**.

This job runs the **train.py** script. An information icon on the right side under **Results** shows the experiments associated with the job. Click the link to explore them.

## Visualize and compare results

To view an individual trial, follow these steps:

1. Click the **Experiments** link to see the trials logged by a workspace or the information icon to see the trials logged by a job.

2. Click an individual trial and the **Metrics** tab to see the full-time-series metrics.

3. Look for the general trend of the metrics over time, especially with deep-learning scenarios where compute costs drive decisions. If your accuracy shows slight or no improvement, there’s no point in using more resources to go down this path.

To compare trial results:

1. Select at least two trials from the main **Experiments** page and click the compare icon.

2. Use the side-by-side comparison to discover how different parameters impact accuracy and run time.

   <Note>
     When you change the **Chart** type to **Line**, you can also examine the time series charts for the metrics side-by-side.
   </Note>

3. Click a specific trial chart line.

4. Analyze differences captured in metadata: different compute environments, users, start times, and workspaces.

## Share the experiment view

You can export the experiment view to share with others or generate reports.

From the Compare screen, export the experiment view as a PDF. This report also has a deep link you can share for direct access to the live report in Domino.

## Troubleshooting

PyTorch and Ray version 2.0.0 show compatibility issues related to auto-logging. As a workaround, use manual logging or a Ray version beyond 2.0.0.

## Next steps

Now that you know how to perform hyperparameter tuning in Domino, explore [Track and Monitor Experiments](/cloud/platform-capabilities/features/development/track-monitor) or [model deployment](/cloud/platform-capabilities/features/models).


# Track and monitor experiments
Source: https://docs.domino.ai/cloud/platform-capabilities/features/development/track-monitor

Create traditional ML and agentic system experiments with MLflow, compare runs and traces, export results, and share access through Projects.

Domino’s Experiment Manager uses [MLflow Tracking](https://mlflow.org/docs/latest/tracking/) to log experiment details, such as parameters, metrics, and artifacts. Domino stores all your experiment metadata, data, and results within the project, making it easy to reproduce your work. This allows you to:

* Monitor experiments with both automatic and manual logging.

* Compare and review experiment runs effectively.

* Collaborate securely with colleagues.

<img alt="Experiment Manager dashboard" />

## Experiment types

You can log two types of experiments in Domino:

[Traditional ML experiments](https://mlflow.org/docs/latest/ml/traditional-ml/) track parameters, metrics, and artifacts from model training workflows. When you train a model, Domino logs the configuration settings, performance metrics, and output files so you can compare different approaches.

You use traditional ML experiments when you train machine learning models with [scikit-learn](https://scikit-learn.org/stable/api/sklearn.html), TensorFlow, PyTorch, or similar frameworks.

[Agentic system experiments](/cloud/platform-capabilities/features/agents/experiment) capture traces from LLM calls, agent interactions, and prompt chains. A trace shows the sequence of calls your system makes, including inputs, outputs, and downstream tool calls. You can attach evaluations to traces to measure quality, accuracy, or safety. You use agentic system experiments when you build applications with LLMs or AI agents.

## How experiments work in Domino

When you click **Experiments** in the Domino left navigation, you’ll see an empty dashboard unless at least one run has been logged. Domino doesn’t create experiments through the UI.

**Instead**

* You create an MLFlow experiment by calling `mlflow.set_experiment()` in code.

* The experiment appears in the UI only after you log at least one run.

For example, to set a unique experiment name:

```python theme={null}
# create a new experiment
import mlflow
import os
starting_domino_user = os.environ["DOMINO_STARTING_USERNAME"]
experiment_name = f"Domino_Experiment_{starting_domino_user}"
# Replace <your_experiment_name> with the name of your experiment
mlflow.set_experiment(experiment_name="<your_experiment_name>")
```

**Tip**: Experiment names need to be unique across your Domino Deployment. A good practice is to use a unique identifier (such as `username`) as part of your experiment name.

## Step 1: Create and log an experiment run

Launch a workspace or job to run your experiment code and log results to the Experiment Manager. Jobs provide better reproducibility and version control.

1. From your project, click **Workspaces** in the left navigation.

2. Launch a workspace using any environment based on the Domino Standard Environment. These environments already have the `mlflow` package installed.

3. Open a Python script or notebook to run your experiment code.

You can log experiments using auto-logging or manual logging. Auto-logging records parameters, metrics, and artifacts automatically for supported libraries. Manual logging gives you full control over what parameters, metrics, and artifacts are recorded.

<Tabs>
  <Tab title="Auto-logging">
    This example shows you how to auto-log a [scikit-learn](https://scikit-learn.org/stable/api/sklearn.html) experiment.

    ```python theme={null}
    # import MLflow library
    import mlflow
    import os

    from sklearn.model_selection import train_test_split
    from sklearn.datasets import load_diabetes
    from sklearn.ensemble import RandomForestRegressor

    # create and set a new experiment
    starting_domino_user = os.environ["DOMINO_STARTING_USERNAME"]
    experiment_name = f"Domino_Experiment_{starting_domino_user}"
    mlflow.set_experiment(experiment_name=experiment_name)

    # enable auto-logging
    mlflow.autolog()

    # start the run
    with mlflow.start_run():
        db = load_diabetes()
        X_train, X_test, y_train, y_test = train_test_split(db.data, db.target)
        rf = RandomForestRegressor(n_estimators = 100, max_depth = 6, max_features = 3)
        rf.fit(X_train, y_train)
        rf.score(X_test, y_test)

    # end the run
    mlflow.end_run()
    ```
  </Tab>

  <Tab title="Manual logging">
    [MLflow Tracking](https://mlflow.org/docs/latest/tracking.html#automatic-logging) offers more details and examples.

    ```python theme={null}
    # import MLflow library
    import mlflow
    import os

    # create a new experiment
    starting_domino_user = os.environ["DOMINO_STARTING_USERNAME"]
    experiment_name = f"Domino_Experiment_{starting_domino_user}"
    mlflow.set_experiment(experiment_name=experiment_name)
    # start a run and log parameter,metric, and artifact
    with mlflow.start_run():
        mlflow.log_param("batch_size", 32)
        mlflow.log_metric("accuracy", 0.75)
        with open("example.txt", "w") as f:
            f.write("This is a sample artifact.")

        mlflow.log_artifact("example.txt")

    # end the run
    mlflow.end_run()
    ```
  </Tab>
</Tabs>

Once your workspace is running, you’re ready to write and execute code that creates and logs an experiment.

## Step 2: Monitor and evaluate experiments

Once you have logged at least one run in an experiment, you can view and analyze it in Domino.

1. Click **Experiments** in Domino’s left navigation pane.

2. Find and click the experiment to evaluate and show its associated runs.

3. Click a run to analyze the results in detail.

The run details show different information depending on your experiment type.

### Traditional ML experiments

When you log traditional ML experiments, Domino captures parameters, metrics, and artifacts from your model training. Here’s how to view and compare your results:

**View metrics and artifacts**\
The run details show parameters you configured, metrics that measure performance, and artifacts like model files or visualizations logged during training.

You can:

* Review parameter values to understand your configuration

* Track metrics over time to measure model performance

* Download artifacts like trained models, plots, or data files

  **Compare runs**\
  After reviewing individual runs, you can compare multiple runs to see how different configurations affect outcomes. Select up to four runs from the table view and click **Compare**.

<img alt="Compare experiments" />

You can compare runs to see how parameters affect important metrics like model accuracy or training speed.

<img alt="Compare runs" />

### Agentic system experiments

When you log agentic system experiments, Domino captures traces from LLM calls and agent interactions. Here’s how to view and compare your traces:

**View traces**\
When you open a run from an agentic system experiment, select the **Traces** tab to see the sequence of calls made during execution.

<img alt="View Traces" />

You can:

* See evaluations that were logged in code

* Add metrics (float values) or labels (string values) to traces by clicking Add Metric/Label

* Click any metric or label cell to open the detailed trace view

  **Compare traces**\
  When comparing agentic system runs, open the Traces comparison view to see how configurations perform on the same inputs.

<img alt="Compare traces" />

Review results across runs to spot patterns, compare traces and evaluations side-by-side, and click any metric or label cell to open the detailed trace view.

## Step 3: Export experiment results

You can export your single experiment results to a CSV or compare experiments and download to a PDF.

**To export single experiment results**:

1. From the **Experiments** view, open the experiment you want to export.

2. Click the three dots in the upper right and choose **Download CSV**.

**To export comparison results**:

1. From the **Experiments** view, select two to four runs and click **Compare**.

2. Click the three dots in the upper right and choose **Export as PDF**.

## Step 4: Share your projects and experiments

You can set permissions for project assets, including **MLflow logs**, on a project level. Use these methods on your projects to control access:

* [Choose the visibility](/cloud/platform-capabilities/core-concepts/projects/project-settings#set-project-visibility) for your project. This helps you control who can see your project.

  * **Searchable**: Discoverable by other Domino users.

  * **Private**: Only viewable or discoverable by your project collaborators.

* [Invite collaborators](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#invite-collaborators) and set their [permissions](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#collaborator-permissions) based on project roles. This gives you detailed control over what they can access.

## Troubleshooting

* Experiment names must be unique across all projects within the same Domino instance.

* Child runs aren’t deleted when you delete a run that has children. Delete the child runs separately.

* You can’t stop a run from the UI. To stop a run, execute `mlflow.end_run()` from your workspace or job.

* When you trigger an experiment from a workspace, the manager shows the file name. But if you are running that and editing it, it doesn’t rename the experiment automatically. After completing an experiment in a workspace, trigger a job to manage version control.

* Best practice is to give your runs a name - otherwise it automatically creates one like `gifted-slug-123`.

## Advanced functionality

You can upload large artifact files directly to blob storage without going through the MLflow proxy server.

Enable this feature in your notebook code by setting the environment variable `MLFLOW_ENABLE_PROXY_MULTIPART_UPLOAD` to `true`.

```python theme={null}
import os
os.environ['MLFLOW_ENABLE_PROXY_MULTIPART_UPLOAD'] = "true"
```

This is helpful for both `log_artifact` calls and registering large language models. It is currently supported only in AWS and GCP environments. There are two additional settings available for configuration:

* `MLFLOW_MULTIPART_UPLOAD_MINIMUM_FILE_SIZE` - the minimum file size required to initiate multipart uploads.

* `MLFLOW_MULTIPART_UPLOAD_CHUNK_SIZE` - the size of each chunk of the multipart upload. Note that a file may be divided into a maximum of 1000 chunks.

[Multipart upload for proxied artifact access](https://mlflow.org/docs/latest/self-hosting/architecture/artifact-store/#multipart-upload-for-proxied-artifact-access) in the MLflow documentation has more information on using this feature.

## Next steps

* [Build a transaction review App](/cloud/platform-overview/getting-started-with-domino) with a Workspace, Dataset, model endpoint, and App.

* [Schedule recurring training jobs](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job).

* [Scale-out training](/cloud/platform-capabilities/features/development/distributed-training) for larger datasets.

* [Monitor model endpoints](/cloud/platform-capabilities/features/model-deployment/monitor-model-endpoints) to track resource usage and optimize deployed models for production.


# Host an LLM
Source: https://docs.domino.ai/cloud/platform-capabilities/features/llms/host-an-llm

Register a large language model, deploy it with vLLM on GPU resources, call its OpenAI-compatible endpoint, and monitor latency and usage.

Domino lets you register Large Language Models (LLMs) and deploy them as hosted endpoints with optimized inference. These endpoints provide OpenAI-compatible APIs that your agentic systems and applications can call directly.

You can register models from Hugging Face or from your [experiment runs](/cloud/platform-capabilities/features/development/track-monitor), then deploy them as endpoints. You’ll need **Project Collaborator** permissions to register models and create endpoints.

## Plan your endpoint deployment

Before creating an endpoint, consider these key factors to ensure optimal performance and cost-efficiency:

### Understand your model’s requirements

Check your model’s documentation for minimum memory and compute requirements. Choose appropriate resource sizes based on requirements and expected usage patterns.

### Size resources appropriately for expected usage

**Account for concurrent users**: if you expect high throughput or multiple simultaneous requests, minimal GPU sizes may cause slowdowns. Scale up the hardware tier or consider deploying multiple endpoints.

**Balance performance against cost**: start with a tier that meets your requirements and monitor performance before scaling up.

<Note>
  GPU is required because LLM inference is compute-intensive. Always select a GPU-enabled hardware tier for your endpoint. CPU-only tiers do not provide acceptable latency for model serving. Choose a tier with enough GPU memory (VRAM) to hold your model’s weights. For example, a 7B-parameter model typically needs at least 16 GB of VRAM.
</Note>

## Step 1: Register a model

Register a model to make it available for deployment as an endpoint. Go to **Models** > **Register** to get started.

1. Choose your model source:

   1. **Hugging Face** models that you have access to, or

   2. **Experiment runs** that include a [logged MLflow model](/cloud/platform-capabilities/features/models/register-and-govern/model-registry).

2. Complete the required fields.

<Frame>
  <img alt="Register a model" />
</Frame>

## Step 2: Create an endpoint

After registering a model, you can deploy it as an endpoint.

1. From your registered model’s **Endpoints** tab, click **Create endpoint**.

2. An endpoint form appears:

   <Frame>
     <img alt="Register an agentic endpoint" />
   </Frame>

   1. Complete the endpoint configuration details.

   2. Choose a model source environment and resource size.

   3. Under **Environment**, select **Domino vLLM Environment**. (This environment is pre-configured with the vLLM runtime, which provides optimized inference and an OpenAI-compatible API out of the box.)

   4. Under **Hardware Tier**, select a GPU-enabled resource size. Match the GPU memory to your model’s requirements. Larger models need more VRAM. For example, use a tier with at least 24 GB GPU memory for 13B+ parameter models.

3. Configure access controls by adding users or organizations for access to this endpoint.

   <Note>
     Some hosted models require the following settings under **Configuration** > **Advanced Tab** > **vLLM arguments** to work with some agent frameworks:

     * `--enable-auto-tool-choice`

     * `--tool-call-parser hermes`
   </Note>

4. Click **Create endpoint**. The endpoint deploys with the vLLM runtime, which provides optimized inference performance and OpenAI-compatible APIs.

## Step 3: Get your endpoint URL and call it from an agent

1. Once your endpoint is running, navigate to the **Calling** tab on the endpoint detail page.\
   This tab provides:

   * The endpoint URL, which is the `base_url` your agent code uses.

   * A code snippet showing how to connect using the OpenAI Python SDK.

2. Copy the URL from the **Calling** tab. Inside a Domino workspace or Job, the API key is available automatically at `http://localhost:8899/access-token`.

3. Use these values to connect your agent to the Domino-hosted model.

<Frame>
  <img alt="Get endpoint URL and code snippet" />
</Frame>

### Use the endpoint from a Pydantic AI agent

Domino endpoints are OpenAI-compatible, so you can use the OpenAIProvider from Pydantic AI with a custom base\_url pointing to your endpoint:

```python theme={null}
import os
import requests
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider

# Copy the URL from the Calling tab of your endpoint
ENDPOINT_URL = os.environ["DOMINO_LLM_ENDPOINT_URL"]

# Inside Domino, the access token is available at this local URL
API_KEY = requests.get("http://localhost:8899/access-token").text

model = OpenAIChatModel(
    "your-model-name",
    provider=OpenAIProvider(base_url=ENDPOINT_URL, api_key=API_KEY),
)
agent = Agent(model)

result = agent.run_sync("Summarize the latest quarterly report.")
print(result.output)
```

<Tip>
  Rather than hard-coding the URL, store it as a [Domino environment variable](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables) such as `DOMINO_LLM_ENDPOINT_URL`. This lets you swap endpoints without changing code, which is useful when promoting from a development to a production endpoint.
</Tip>

### Use the endpoint with the OpenAI SDK directly

If you prefer calling the endpoint without a framework, use the standard OpenAI Python client:

```python theme={null}
import os
import requests
from openai import OpenAI

ENDPOINT_URL = os.environ["DOMINO_LLM_ENDPOINT_URL"]
API_KEY = requests.get("http://localhost:8899/access-token").text

client = OpenAI(base_url=ENDPOINT_URL, api_key=API_KEY)

response = client.chat.completions.create(
    model="your-model-name",
    messages=[{"role": "user", "content": "Hello"}]
)
print(response.choices[0].message.content)
```

<Note>
  OpenAI Responses API is also supported.
</Note>

## Step 4: Monitor endpoint performance

After deploying your endpoint, you can monitor its performance and usage from the endpoint detail page:

* **Overview**: Configuration details and deployment status

* **Performance**: Token usage and latency metrics over time

* **Usage**: Endpoint invocation frequency

<Frame>
  <img alt="Endpoint performance dashboard" />
</Frame>

[Monitor model endpoint performance](/cloud/platform-capabilities/features/model-deployment/monitor-model-endpoints) has more detailed information about using monitoring capabilities during model development and after deployment to make sure your models perform efficiently and reliably in production environments.

## Troubleshoot common issues

| Problem | Solution |
| - | - |
| Hugging Face model not appearing in the list | Verify you have access to the model. Some models require accepting license agreements on Hugging Face before they’re available in Domino. |
| Endpoint stuck in "Starting" status | Check that the model size is compatible with your selected hardware tier. Large models may need more GPU memory. Review endpoint logs for specific error messages. |
| Slow response times or timeouts | Monitor the **Performance** tab to identify latency patterns. If concurrent requests exceed your resource capacity, consider scaling to a larger hardware tier or deploying additional endpoints. |
| Users can’t access the endpoint | Verify users or their organizations were added to the endpoint’s access controls and that users have the required project permissions. |

## Next steps

* [Set up LLM access](/cloud/platform-capabilities/features/llms): Configure external providers or choose between hosting approaches.

* [Develop agentic systems](/cloud/platform-capabilities/features/agents/develop): Instrument your agent code to capture traces during development and attach evaluations to assess quality.

* [Monitor agentic systems](/cloud/platform-capabilities/features/agents/monitor): Track production performance.

* [Track and monitor experiments](/cloud/platform-capabilities/features/development/track-monitor): View, compare, and export experiment results for both ML and agentic systems.

* [Deploy and monitor production agents](/cloud/platform-capabilities/features/agents/deploy): Launch your best configuration to production and continue monitoring performance with real user interactions.


# Set up LLM access
Source: https://docs.domino.ai/cloud/platform-capabilities/features/llms/index

How a Large Language Model becomes callable from your code in Domino, and who is allowed to call it: connect to an external provider, host your own model, or mix both.

Any code you run in Domino that calls a Large Language Model (LLM) needs an endpoint to call and credentials to call it with. That applies whether the caller is an [agent](/cloud/platform-capabilities/features/agents), an [App](/cloud/platform-capabilities/features/apps), a [Job](/cloud/platform-capabilities/core-concepts/jobs), or exploratory code in a [Workspace](/cloud/platform-capabilities/core-concepts/workspaces). Decide how your code reaches its models before you start building.

```mermaid theme={null}
flowchart LR
  code["Your code<br/>(agent, App, Job, Workspace)"]
  ext["External provider<br/>OpenAI, Anthropic, Bedrock,<br/>Azure OpenAI, Vertex AI"]
  dom["Domino-hosted endpoint<br/>on your compute"]

  code -->|"API key in an env var"| ext
  code -->|"OpenAI-compatible call"| dom
```

Domino supports two approaches, and most teams end up using both:

| | Connect to an external provider | Host a model in Domino |
| :- | :- | :- |
| Where inference runs | Outside Domino, at the provider | On Domino compute, on hardware you choose |
| Who manages the model | The provider handles hosting, scaling, and versioning | You do, through a Domino endpoint |
| Reach for it when | You want a frontier model and can send prompts outside the cluster | External providers are off the table, or you need data residency, cost control, or your own fine-tuned model |
| What you configure | The provider's SDK and an API key | Registration, endpoint, GPU hardware tier, and scaling |

Which one fits is a question about your constraints, not a default. Teams that already have access to a managed LLM service often start there, since it's the shorter path. Teams that can't send prompts to a third party, whether because of data residency rules, a policy against external model providers, or an [air-gapped deployment](/cloud/platform-overview/deployment-options), host in Domino from the outset and get the same workflow. Both are first-class paths.

Either way, the decision stays reversible: if you standardize on an OpenAI-compatible client, you can move between an external provider and a Domino-hosted model later without changing how your code calls it, since Domino-hosted endpoints expose an OpenAI-compatible API.

## Option A: Connect to an external LLM provider

Most teams already have access to a managed LLM service. Your code calls the provider's API directly. Domino runs your code, and the LLM inference happens outside Domino.

Common providers include:

| Provider | SDK / package | Typical environment variable |
| - | - | - |
| OpenAI | `openai` | `OPENAI_API_KEY` |
| Anthropic | `anthropic` | `ANTHROPIC_API_KEY` |
| AWS Bedrock | `boto3` | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` |
| Azure OpenAI | `openai` | `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT` |
| Google Vertex AI | `google-cloud-aiplatform` | Service account credentials |

### How to configure access in Domino

<Steps>
  <Step title="Install the provider's SDK">
    Add the package to your [Environment's](/cloud/platform-capabilities/core-concepts/compute-environments) Dockerfile instructions (for example, `RUN pip install openai`), or install it in your Workspace.
  </Step>

  <Step title="Store API keys as Domino environment variables">
    In your Project settings, add environment variables for your credentials. Domino injects them into Workspaces and Jobs automatically, and your code reads them at runtime through `os.environ`. This keeps secrets out of your code and out of version control.
  </Step>

  <Step title="Call the provider's SDK as you would locally">
    Read the key from `os.environ` and pass it to the provider's client, as in the example below. Nothing about the call is Domino-specific; the provider handles model hosting, scaling, and versioning.
  </Step>
</Steps>

```python theme={null}
import os
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider

# API key is read from a Domino environment variable
model = OpenAIChatModel(
    "gpt-5.4-mini",
    provider=OpenAIProvider(api_key=os.environ["OPENAI_API_KEY"]),
)
agent = Agent(model)

result = agent.run_sync("Hello")
print(result.output)
```

<Note>
  The legacy AI Gateway is no longer available in Domino Cloud. [LLM Gateway 2.0](/cloud/platform-capabilities/features/llms/llm-gateway) is its successor, with usage tracking, cost controls, guardrails, and broader model and vendor coverage. Domino distributes it today as a binary deployed as a Domino app and plans to natively integrate it into a future release. Contact your Domino field team member or representative for access.
</Note>

## Option B: Host a model in Domino

Domino can host a model as an endpoint on your own compute. Teams take this route when an external provider isn't permitted at all, and when they need data residency, cost control, or a model they fine-tuned themselves. Your code calls the Domino-hosted endpoint the same way it would call any other API, using an OpenAI-compatible interface.

[Host an LLM](/cloud/platform-capabilities/features/llms/host-an-llm) has the full setup guide, including:

* Registering a model from Hugging Face or a custom checkpoint.

* Deploying it as a Domino endpoint.

* Configuring GPU hardware and scaling.

## Mix approaches

Many production systems use both an external **frontier model** for primary reasoning and a **Domino-hosted model** for specialized tasks such as a fine-tuned classifier or embedding model.

If you're building an agent, the `@add_tracing` instrumentation described in [Develop agentic systems](/cloud/platform-capabilities/features/agents/develop) captures traces from all LLM calls regardless of where the model is hosted.

<Tip>
  You can swap models during experimentation. In Domino, model switching is a config change, not an infrastructure change. You can update a YAML file to move between external and Domino-hosted models, making it easy to evaluate cost, reliability, and performance as you iterate.
</Tip>

## Related

* [Host an LLM](/cloud/platform-capabilities/features/llms/host-an-llm): register a model and deploy it as a Domino endpoint, including the access controls on that endpoint.

* [LLM Gateway 2.0](/cloud/platform-capabilities/features/llms/llm-gateway): the migration path for teams that were using the legacy AI Gateway before it was removed.

* [Agents](/cloud/platform-capabilities/features/agents): build, evaluate, and deploy an agentic system on top of these endpoints.


# LLM Gateway 2.0
Source: https://docs.domino.ai/cloud/platform-capabilities/features/llms/llm-gateway

The legacy AI Gateway has been removed from Domino Cloud. LLM Gateway 2.0 is its successor, distributed today as a Domino-deployed app and planned for native integration in a future release.

<Warning>
  **The legacy AI Gateway has been removed from Domino Cloud.** The **Gateway LLMs** UI, the AI Gateway endpoint APIs, and the endpoint audit trail are no longer available. Any code that called AI Gateway endpoints through the MLflow Deployment Client, or the AI Gateway routes on the Domino Platform API, stops working after your deployment upgrades. This page is retained so existing links resolve.
</Warning>

LLM Gateway 2.0 is the successor to the legacy AI Gateway. It centralizes external provider access and adds capabilities the legacy gateway did not have:

* Usage tracking across users, Projects, and models.
* Cost controls, including per-team and per-model budgets.
* Guardrails for prompt and response inspection.
* Broader model and vendor coverage.

Domino distributes LLM Gateway 2.0 today as a binary that is deployed as a Domino app, and plans to natively integrate it into a future Domino release. Contact your Domino field team member or representative for access.

## Other replacement paths

If LLM Gateway 2.0 is more than you need, or if you're routing calls before you get access, pick one of these direct paths from Workspaces, Jobs, Apps, and agents.

<Columns>
  <Card title="Call an external provider directly" href="/cloud/platform-capabilities/features/llms#option-a-connect-to-an-external-llm-provider">
    Install the provider's SDK, store the API key as a Domino environment variable, and call the provider from your code. Works with OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, and Google Vertex AI.
  </Card>

  <Card title="Host a model in Domino" href="/cloud/platform-capabilities/features/llms/host-an-llm">
    Register a model from Hugging Face or a custom checkpoint and deploy it as a Domino endpoint on your own compute. Callers use an OpenAI-compatible API.
  </Card>
</Columns>

## Migrate off the legacy AI Gateway

<Steps>
  <Step title="Identify code that used AI Gateway endpoints">
    Search your Projects for calls to `mlflow.deployments.get_deploy_client("databricks")` or `predict()` / `get_endpoint()` / `list_endpoints()` against a Domino AI Gateway target. Any such calls fail after the upgrade.
  </Step>

  <Step title="Pick a replacement path">
    Ask your Domino field representative about LLM Gateway 2.0 if you want a centralized gateway with usage tracking, cost controls, and guardrails. Otherwise, use the provider's SDK directly (Option A on [Set up LLM access](/cloud/platform-capabilities/features/llms)), or host the model in Domino (Option B).
  </Step>

  <Step title="Move credentials off AI Gateway's central vault">
    Provider credentials that AI Gateway held in Domino's central vault are no longer accessible through the AI Gateway UI or API. Store the credentials as Project environment variables, or as inputs to your LLM Gateway 2.0 configuration, and read them from `os.environ` in your code.
  </Step>

  <Step title="Update code and rerun">
    Replace the MLflow Deployment Client call with a call through LLM Gateway 2.0, a direct provider SDK call, or a call to your Domino-hosted endpoint. Rerun affected Jobs, Apps, and agents to confirm they work end to end.
  </Step>
</Steps>

## Related

* [Set up LLM access](/cloud/platform-capabilities/features/llms): decide between an external provider and a Domino-hosted model.

* [Host an LLM](/cloud/platform-capabilities/features/llms/host-an-llm): full setup guide for a Domino-hosted endpoint.


# Configuration settings for Apps
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/app-logs-and-versioning/app-configuration-settings

Configure App identity propagation, Dataset access, execution limits, and routing, including SecureIdentityPropagationToAppsEnabled and DOMINO_RUN_HOST_PATH.

Feature flags and configuration records govern how Apps handle identity propagation, dataset access, and execution limits.

The table below summarizes key settings that influence app behavior:

| Setting | Type | Purpose |
| - | - | - |
| `SecureIdentityPropagationToAppsEnabled` | Feature flag | Enables a limited-scope JSON Web Token (JWT) for identifying the requesting user. Required for deep linking and [extended identity propagation](/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security#extended-identity-propagation). |
| `com.cerebro.domino.apps.extendedIdentityPropagationToAppsEnabled` | Configuration record | Enables a full-scope JWT for identifying the requesting user. Requires `SecureIdentityPropagationToAppsEnabled`. See [extended identity propagation](/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security#extended-identity-propagation). |
| `com.cerebro.domino.dataset.mountingEnabledInApps` | Configuration record | Controls whether Datasets are mounted or accessed via API only. |
| `com.cerebro.domino.apps.maxAppsPerProject` | Configuration record | Limits how many apps can exist within a single project. The default is `10`. Users cannot create a new app in the project once this limit is reached. |
| `com.cerebro.domino.apps.maxActiveAppRunsPerProject` | Configuration record | Caps how many app sessions can run concurrently per project. The default is `4`. Users cannot start an app in the project once this limit is reached. |

An administrator can change these values for your deployment.

## Secure identity propagation and the app host path

Domino uses the `SecureIdentityPropagationToAppsEnabled` feature flag to control how requests are routed to a running app. When the flag is `true`, Domino changes the URL routing structure it uses to proxy requests.

Defaults by version:

| Domino version | Default | Notes |
| - | - | - |
| Cloud Release 90 and later | `true` | All Apps should use `DOMINO_RUN_HOST_PATH`. |
| 6.3.0 and later (self-managed) | `true` | All Apps should use `DOMINO_RUN_HOST_PATH`. |
| Earlier releases | `false` | Migrate to `DOMINO_RUN_HOST_PATH` before enabling the flag. |

### Symptoms of an incompatible app

An app that manually constructs its path prefix from `DOMINO_PROJECT_OWNER`, `DOMINO_PROJECT_NAME`, and `DOMINO_RUN_ID` produces a path that no longer matches the new routing structure. The app continues to run until it is restarted, then fails on the next start.

Common symptoms:

* Blank page, 404, or 502 after restart.
* Dash (Plotly) app fails to serve assets or callbacks.
* The app worked before the flag was enabled or before the last restart.

### Update your app to use `DOMINO_RUN_HOST_PATH`

Replace any manually constructed path prefix with the Domino-provided `DOMINO_RUN_HOST_PATH` environment variable. Domino sets this variable automatically and it always reflects the correct routing path, regardless of the flag state.

**Before** (breaks when the flag is `true`):

```python theme={null}
user    = os.environ.get("DOMINO_PROJECT_OWNER")
project = os.environ.get("DOMINO_PROJECT_NAME")
run_id  = os.environ.get("DOMINO_RUN_ID")
runurl  = f"/{user}/{project}/r/notebookSession/{run_id}/"

app = Dash(__name__, routes_pathname_prefix="/", requests_pathname_prefix=runurl)
```

**After** (recommended):

```python theme={null}
runurl = os.environ.get("DOMINO_RUN_HOST_PATH")

app = Dash(__name__, routes_pathname_prefix="/", requests_pathname_prefix=runurl)
```

Publish a new version with the updated code, then deploy that version. Verify that the page loads and that callbacks and assets resolve.

<Note>
  Any Python web framework that accepts an explicit base path or prefix (Dash, Streamlit, Flask, and others) should read the prefix from `DOMINO_RUN_HOST_PATH`.
</Note>

<Warning>
  If `SecureIdentityPropagationToAppsEnabled` is `false` and you need to allow anonymous access to Domino Apps, also set the Central Config parameter `com.cerebro.domino.launchpad.allowPublicModelProducts` to `true`.
</Warning>

## Next steps

* [Apps in Domino](/cloud/platform-capabilities/features/apps/apps-in-domino) gives an overview of how apps work within the Domino ecosystem.
* [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions) explains how publishing a version differs from deploying it, and which settings you can change without creating a version.
* Learn more about [how Apps in Domino run](/cloud/platform-capabilities/features/apps/authentication-and-identity) and what identity and permissions are used.


# Performance tuning your Apps
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/app-logs-and-versioning/app-scalability-performance

Tune App performance with Hardware Tiers, port 8888, and framework-specific concurrency settings for Flask, Dash, and Shiny workloads.

## Domino applications

The following guidelines apply to all Domino Apps, regardless of framework:

* Apps run until you explicitly stop them. If your app isn’t used anymore, stop it to minimize your compute spend.

* Unlike a job or a workspace, Domino Apps never commit (or sync) file changes from the local disk to the Domino project’s file store. To persist data, your app’s logic must write it to an external data source.

* The performance of your app depends on the design of the application.

Launching an [app in Domino](/cloud/platform-capabilities/features/apps) works the same as any other Domino run. Domino assigns hosting of your app to an executor machine in the [hardware tier](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/hardware-tier-best-practices) your App is configured to use. That executor then retrieves the default [Domino environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments) configured for your project, and creates a container based on that image. Domino then loads your project files onto the machine and executes the app file that you have selected, at which point your application is running in its container.

Depending on which hardware tier you select, the container running your application may share a host machine with other containers or run on a dedicated host. Your selection of hardware tier allows you to specify available memory and CPU. However, Domino Apps do not automatically scale horizontally to multiple hosts. Your App can scale vertically to use all available resources on the executor host machine by correctly configuring the underlying application.

The following sections describe configuring web applications to optimize scalability and performance for several popular frameworks.

## Flask and Dash (Python)

By default, Flask and Dash run single-threaded on a single process. The authors of Flask do not recommend this configuration if you are going to serve more than 10 users concurrently, or for any externally consumed applications. The [Flask documentation](https://flask.palletsprojects.com/en/3.0.x/deploying/) provides many ways to serve the application in a more scalable way.

For example, you can serve a Flask application through [gunicorn](https://gunicorn.org/). To do this in Domino, change the project’s app file. See [Gunicorn commands](https://gunicorn.org/run/) for details. You can change:

```shell theme={null}
export LC_ALL=C.UTF-8
export LANG=C.UTF-8
export FLASK_APP=app-flask.py
python -m flask run --host=0.0.0.0 --port=8888
```

to

```shell theme={null}
gunicorn -w 4 -b 0.0.0.0:8888 app-flask:app
```

This starts serving the Flask application on four processes. Apps published in Domino must serve traffic on port `8888`.

The performance and scalability of your App depend on the compute demands of your application, and the compute resources available on the host machine. If there is a command in your application that uses 100 MB RAM and 20% of a standard VM CPU, then an executor host machine with 1 core and 1 GB RAM could handle 5 concurrent users running that command without suffering reduced performance. A sixth user attempting to run the command would cause the App’s performance to suffer. There would be RAM available, but not enough CPU cycles.

## Shiny (R)

Like Python Apps, your Shiny Apps' performance depends on design of the underlying application. While multiple users can view Shiny applications in independent sessions, R is a single process language. This means that multiple users can view and interact with the App in their own isolated session, but only one can do any processing at a time regardless of the memory or CPU of the machine.

Shiny Apps typically cannot scale to more than a handful of concurrent users.

See [Make Shiny fast by doing as little work as possible](https://rstudio.com/resources/rstudioconf-2018/make-shiny-fast-by-doing-as-little-work-as-possible/).


# Autoscaling Apps
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/app-logs-and-versioning/autoscaling-apps

Configure App autoscaling with pod limits, CPU and memory targets, scaling delays, and session affinity, then monitor resource use and events.

Domino Apps can automatically scale based on resource usage to balance cost with performance. Autoscaling prevents performance loss during periods of high demand while avoiding unnecessary compute costs during low demand.

Domino uses the Kubernetes Horizontal Pod Autoscaler (HPA) v2 to manage scaling.

## How autoscaling works

You decide whether to enable autoscaling when you publish or launch an App. No additional configuration is required.

Scaling occurs without requiring you to restart or republish the App. HPA dynamically adjusts the number of Kubernetes pods for the App while it is running.

App authors can set custom autoscaling values, including maximum pods, target CPU/memory thresholds, and scale timing, at launch. These settings persist across sessions and appear on mouseover in the **Launch App** window.

### Scaling parameters

Autoscaling decisions are based on the following scaling parameters:

| Scaling parameter | Description |
| - | - |
| **Maximum Pods** | Maximum number of pods the App can scale up to. |
| **CPU % target** | Target CPU utilization across pods. |
| **Memory % target** | Target memory utilization across pods. |
| **Scale up delay** or **Scale down delay** | Delays scaling changes to prevent rapid, repeated adjustments. |
| **Enable session affinity** | Routes user requests to the same pod. |

### Session affinity

Session affinity routes all of a viewer’s requests to the same pod for the duration of their session. Without it, traffic may be distributed across multiple pods, which can disrupt apps that maintain in-memory state or require continuity across requests.

* **Enable session affinity** for frameworks like R/Shiny that rely on stateful sessions.
* Leave **session affinity** disabled for most other frameworks to improve responsiveness and scale-down efficiency.

The **Enable session affinity** option requires the `SecureIdentityPropagationToAppsEnabled` config to be set to `true`. Contact your Domino administrator to verify it’s enabled.

## Framework-specific considerations

Different frameworks benefit from different autoscaling settings. Use these guidelines to balance performance and cost:

* Rely on HPA to avoid thrashing, or rapid oscillation between scale events.
* Remember that autoscaling apps always run at least one pod.
* Scale-up typically finishes within \~20 seconds. This lets you safely select leaner, lower-cost hardware tiers than for non-autoscaling Apps.
* Use the default autoscaling settings for most frameworks.
* Select smaller hardware tiers and let the autoscaler scale up or down as needed.

<span />

### R/Shiny

R/Shiny apps have unique scaling needs because they are single-threaded. Use these settings to maximize stability and support high user counts:

* Single-threaded, so horizontal autoscaling is essential.
* **Enable session affinity** to avoid transient connectivity issues for viewers.
* Use smaller hardware tiers with a higher number of maximum pods.
* Choose a low-CPU instance type that can be aggressively scaled horizontally.
* These settings support arbitrarily high user counts.

### Other frameworks (Streamlit, Flask, Dash, etc.)

Most modern frameworks scale more efficiently. Use these guidelines to improve responsiveness and enable faster scale-down:

* Leave **session affinity** disabled for optimal performance.
* Improves responsiveness while enabling quicker scale-down.

## Monitor autoscaling

You can monitor autoscaling from the App’s detail page.

* View up to 15 days of CPU and memory utilization history across all pods.
* Review autoscaling events over time.
* Access aggregated logs from all pods in a single interleaved master log file.

## Next steps

* [Persist Data](/cloud/platform-capabilities/features/apps/persist-data-in-apps): Save and manage app data across sessions.
* [Usage and Resource Monitoring](/cloud/platform-capabilities/features/apps/app-logs-and-versioning/understanding-app-usage): Monitor app workloads.


# App logs and monitoring
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/app-logs-and-versioning/index

The logs Domino captures for each App instance, how resource consumption is reported per version, and what a support bundle contains.

Domino captures logs and resource metrics for every App instance, so you can debug a failing App, explain a slow one, and see what a version did while it was running.

## App logs

These log types are available during and after App execution:

* **App stdout/stderr**: Captures standard output, error messages, and runtime print statements.

* **Kubernetes events**: Includes pod scheduling, image pulls, and container state transitions.

* **Execution logs**: Generated by Domino's execution engine for each app run.

* **System logs**: Captures platform-level components such as executor, logjam, and frontend services.

* **NGINX & nucleus logs**: Provide visibility into routing, proxy behavior, and request handling.

* **Performance metrics (optional)**: CSV-format data that tracks system-level performance like CPU and memory.

### Logs per version and per instance

Domino organizes logs by [version](/cloud/platform-capabilities/features/apps/app-versioning-and-reproducibility). Because a version can be started many times, an instance selector on the App details page lets you read the logs from one specific run. A new version starts its logs fresh.

Resource consumption (CPU and memory) is recorded per instance. Use the instance selector to view metrics from each start of the version.

### Accessing logs

Read logs directly from the App UI or download them for deeper inspection. Domino also produces a **support bundle** that consolidates key logs and metadata for troubleshooting.

### Support bundle contents

The support bundle is a downloadable archive:

* `execution.log`, `executor.log`: App subprocess execution and engine output.

* `events.json`, `events.csv`: Timeline of internal execution events.

* `kubernetes-events.json`: Cluster and pod lifecycle messages.

* `logjam-*.log`: Runtime traces and orchestration activity.

* `nginx.log`, `nucleus-*.log`: Proxy and routing behavior.

* `execution.yaml`, `manifest.csv`: Runtime configuration and file manifest.

* `performance-metrics.csv` **(optional)**: System resource utilization metrics.

* `project.json`: Project-level settings and environment metadata.

* `run.json`: Detailed run metadata, diagnostics, and resource info.

* `run-history.json`: Sequence of lifecycle status changes.

* `saga.json`: Infrastructure orchestration details and container startup data.

## Troubleshooting tips

If your App isn't behaving as expected, these quick diagnostics can help you identify and resolve common issues:

* **No output in logs?** Check `executor.log` and `logjam-*.log` for errors.

* **Scheduling issues?** Inspect `kubernetes-events.json` for pod failures or capacity problems.

* **Unexpected behavior after a deploy?** Compare the running version's configuration against the version that last worked, then start the older version to confirm.

## Best practices

* **Use support bundles** to investigate failed or abnormal app behavior.

* **Review logs regularly**, especially after environment updates or dependency changes.

* **Compare a failing version against the last good one** by reading each version's configuration, then start the older version to confirm the difference.

[Best practices for Apps](/cloud/platform-capabilities/features/apps/best-practices-for-apps) covers the wider development and deployment practices.

## Next steps

* [App versioning and reproducibility](/cloud/platform-capabilities/features/apps/app-versioning-and-reproducibility): What a version pins, and how to start a previous one

* [Apps in Domino](/cloud/platform-capabilities/features/apps/apps-in-domino) gives an overview of how apps work within the Domino ecosystem.

* [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions): Create an immutable version, then start it for viewers

* Learn more about [how Apps in Domino run](/cloud/platform-capabilities/features/apps/authentication-and-identity) and what identity and permissions are used.


# Understand App usage
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/app-logs-and-versioning/understanding-app-usage

Track App views, status, uptime, version trends, and 15 days of CPU and memory data to investigate demand and right-size resources.

Domino Apps include built-in usage analytics to help you monitor engagement, track adoption, and spot trends over time. Everything is available directly in the UI, with no additional setup required.

<Frame>
  <img alt="Accessing the Usage Tab for Apps" />
</Frame>

## Accessing the usage tab

Open the App’s details page and select **Usage** to view engagement metrics.

The details panel stays visible next to every tab and shows the deployment state of the selected version:

| Entry | Description |
| - | - |
| `Status` | Current status: **Running**, **Suspended**, or **Stopped**. |
| `Running duration` | How long the App has been running. |
| `Created by` | The publisher of the current deployment instance for the selected version. |
| `Created on` | When the selected version was created. |
| `Data Plane` | The [Data Plane](/cloud/reference/glossary#data-plane) the App runs on. Shows **Local** for the local Data Plane, and is hidden when the Data Plane can't be resolved. |

## Total views and period comparison

This section shows how many times your App has been accessed and how that compares to the previous month.

* **Views in past 30 days**: Total number of HTTP requests to your App, for example, `263`.
* **Change over previous period**: Percentage increase or decrease compared to the prior 30-day window, for example, `+30.20%`.

## Daily views chart

This chart helps you visualize usage patterns over time.

* Each bar represents the number of App views on a specific day.
* Hover over a bar to see the exact count, like `14 views on Jun 3`.

## Version-specific filtering

Use the **Version** dropdown to filter metrics:

* **All versions** (default): Aggregated data.
* Select a specific version to view usage for that version only.
* Green bars indicate the currently deployed version.

## Interpret the data

The resource metrics chart provides insight into how your app is being used over time. You can uncover trends that inform performance tuning, capacity planning, and user engagement strategies, by analyzing usage patterns such as spikes, steady growth, or sudden drops.

Use trends in the usage data to understand how your App is being used:

* A steady increase = Growing engagement.
* Spikes = Demos, training sessions, or automation.
* Drops = Autosuspends, errors, or reduced interest.

Review logs or gather user feedback to investigate anomalies. These insights can help you improve performance, plan for growth, and boost adoption.

## Monitor resource consumption

Domino Apps include built-in resource monitoring to help you track real-time CPU and memory usage. This visibility aids in troubleshooting, performance tuning, and choosing the right hardware tier.

<Frame>
  <img alt="Monitor App resource consumption" />
</Frame>

The panel provides historical data, visual insights, and best practices for optimization.

### Access the resource consumption panel

You can monitor an App’s hardware usage (such as CPU, memory, and GPU) in the **Runtime metrics** card on the App’s **Monitoring** tab. This helps you understand performance over time, diagnose resource-related issues, or fine-tune compute settings.

Resource data is retained for the previous 15 days; if no data is available for that window, the panel displays a message indicating that no chart is visible.

### Understand the chart

The chart displays CPU and memory usage over time for the selected App instance. Use the instance selector to view metrics from another start of the same version.

* **Memory Usage (GiB)** → Purple line (left Y-axis).
* **CPU Usage (%)** → Green line (right Y-axis).
* **Time** (last 15 days) → X-axis.

Hover over any point to see timestamp, memory usage, and CPU percentage.

### Key insights

Beyond general usage patterns, the resource chart can reveal deeper operational insights about your app’s performance. Tracking these signals helps you validate resource sizing, detect inefficiencies, and identify opportunities to optimize memory or compute behavior.

Use the chart to identify these trends in resource usage:

* **Baseline usage**: Confirms App fits within allocated resources.
* **Spikes**: May occur during data loads or visualizations.
* **Gradual increases**: Can indicate memory leaks or inefficiencies.

## Data retention and external monitoring

Domino retains resource data for the **past 15 days only**.

For long-term tracking or alerting:

* Forward metrics to external monitoring systems, such as **Prometheus + Grafana**, or **Splunk**.
* Use the Domino API or a sidecar exporter to stream data.

## Best practices for resource optimization

* **Right-size early**: Scale up if usage consistently exceeds 80%.
* **Optimize hot paths**: Link resource spikes to specific features or actions.
* **Set limits and alerts**: Work with admins to configure quotas and alerts.

By monitoring and optimizing usage, you can improve stability, enhance UX, and ensure efficient use of hardware resources.

## Next steps

* [Apps in Domino](/cloud/platform-capabilities/features/apps/apps-in-domino) gives an overview of how apps work within the Domino ecosystem.
* [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions) explains how publishing a version differs from deploying it, and which settings you can change without creating a version.
* [Learn more about how Apps in Domino](/cloud/platform-capabilities/features/apps/authentication-and-identity) run and what identity and permissions are used.


# App versioning and reproducibility
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/app-versioning-and-reproducibility

How Domino creates App versions, what each version pins, and how to restore pinned inputs or roll back.

Publishing a Draft or changing version-defining content creates an App version. Starting or stopping an App does not. Each version pins its code and configuration, so you can start it again with those same inputs.

## What defines a version

A version captures the App's identity and version-defining content. Deployment settings and version metadata remain editable in place on that version.

| Setting group | Fields | Version behavior |
| :- | :- | :- |
| **App identity** | Entry point; code configuration type | Captured by every version, but fixed for the life of the App. Create another App to change either field. |
| **Version-defining content** | Source code (the Git reference and resolved commits for the main repository, Domino File System, and imported Project repositories); Compute Environment and environment revision; Data Plane; external data volume mounts; NetApp Volume mounts; Dataset mount configuration; extended identity propagation | Publishing a Draft or editing one of these fields creates a version. The fields are immutable for that version, and an edit stops the running instance. |
| **Deployment settings** | Hardware Tier; autoscaling; custom (vanity) URL; iframe rendering, deep linking, and query parameters | Updates the existing version in place and restarts the running instance. No new version is created. |
| **Version metadata** | Version description (Version Notes); Tags | Updates the existing version in place without restarting the running instance. No new version is created. |

Publishing a Draft or editing version-defining content creates a version. An edit stops the currently running instance. Starting and stopping an App never create a version.

### What changed

| Before | Now |
| :- | :- |
| Every App start allocated a new version. | Publishing a Draft or editing version-defining content creates a version. |
| The commit was resolved at start time, so a restart could pick up different code. | The commit is resolved and pinned when the version is created. |
| Editing any setting produced a new version. | Editing a deployment setting updates the version in place. |

Pinning commits at version-creation time makes the version's source code reproducible.

## Versions and instances

Each time a version starts, Domino creates an **instance** that lasts until the App stops. A version can be started many times over its life, though an App has at most one running instance at a time. Domino records start and stop events per version, visible on the App details page and in the [Audit Trail](/cloud/platform-capabilities/features/governance/audit-trail).

## Start a previous version

Select any version in the version history on the App details page and start it. Domino runs the App with that version's pinned commit and configuration, including its environment revision, Data Plane, and data configuration. Use this to restore the versioned inputs or to roll back a problematic release without reverting code. Results can differ if Dataset contents or external systems have changed.

Starting an existing version never creates a new version. Every App view shows which version is running. Start and stop are also available through the Apps API.

## Per-version history and metadata

The App details page preserves the following for each version:

| Item | Detail |
| :- | :- |
| Logs | Selectable per instance. |
| Resource consumption | Selectable per instance. |
| Start and stop events | Recorded per version. |
| Version Notes and Tags | Per-version metadata. Editable at any time without creating a new version or restarting the running instance. |
| Metadata | Details, deployment, data, files, access & sharing, and usage. |

Each new instance starts its logs and resource usage fresh.

<Note>
  Domino does not compare two versions side by side. To see what changed, read the Version Notes, or diff the pinned commits in your Git provider.
</Note>

<Note>
  Dataset contents and external systems are not part of a versioned App configuration. Only the Dataset mount configuration is versioned, so an App uses the current Dataset contents and external state whenever the version runs.
</Note>

## Related

* [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions): Create a version, then start it.
* [App logs and monitoring](/cloud/platform-capabilities/features/apps/app-logs-and-versioning): Read per-instance logs and resource consumption.
* [Develop and test an App](/cloud/platform-capabilities/features/apps/develop-and-test-an-app): Iterate in a workspace before you publish a version.
* [App structure and lifecycle](/cloud/platform-capabilities/features/apps/apps-in-domino/app-structure-lifecycle): Where versioning sits in the wider lifecycle.


# App structure and lifecycle
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/apps-in-domino/app-structure-lifecycle

What an App's settings consist of, which of them define a version, and the five stages an App moves through from creation to shutdown.

Each Domino App is defined at the project level and runs in its own isolated environment, so you can configure, start, and stop it without affecting Jobs, Workspaces, or other Apps in the same project. You control how it runs; Domino manages the orchestration.

## App structure

An App's settings fall into groups based on whether they are fixed for the App, define a version, or remain editable on a version.

| Setting group | Fields | Version behavior |
| :- | :- | :- |
| **App identity** | Entry point; code configuration type | Captured by every version, but fixed for the life of the App. Create another App to change either field. |
| **Version-defining content** | Source code (the Git reference and resolved commits for the main repository, Domino File System, and imported Project repositories); Compute Environment and environment revision; Data Plane; external data volume mounts; NetApp Volume mounts; Dataset mount configuration; extended identity propagation | Publishing a Draft or editing one of these fields creates a version. The fields are immutable for that version, and an edit stops the running instance. |
| **Deployment settings** | Hardware Tier; autoscaling; custom (vanity) URL; iframe rendering, deep linking, and query parameters | Updates the existing version in place and restarts the running instance. No new version is created. |
| **Version metadata** | Version description (Version Notes); Tags | Updates the existing version in place without restarting the running instance. No new version is created. |

The **entry point** is the App's launch file: a shell script such as `app.sh`, or a Python or R script. A typical launch file starts a process that binds to `0.0.0.0` and serves traffic on port `8888`. An App that doesn't expose a server on that address and port fails to start.

<Note>
  The entry point is fixed for the life of an App. To change it, create a new App.
</Note>

## App lifecycle

Development, publication, and deployment are separate, so each can happen independently.

<Steps>
  <Step title="Create">
    Define the App in your project, in a workspace or from an existing file. The App exists as an entity immediately, but creating it doesn't make it reachable. See [Create an App](/cloud/platform-capabilities/features/apps/create-an-app).
  </Step>

  <Step title="Develop and preview">
    Work on App code in a workspace and use Preview to run your Draft App in a real App runtime, with production networking, authentication, and environment variables, without publishing. Only the App author sees a preview, and it never collides with a deployed version. See [Develop and test an App](/cloud/platform-capabilities/features/apps/develop-and-test-an-app).
  </Step>

  <Step title="Publish">
    Promote the Draft App to a published App, either a new App or a new version of an existing one. Publishing records the configuration immutably, including the resolved commit, but doesn't start the App.
  </Step>

  <Step title="Deploy">
    Start a published version to make it available to viewers. Each start creates an instance. You can deploy the newest version or any earlier one. See [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions).
  </Step>

  <Step title="Stop">
    End the running instance and release its compute resources. The version and its history remain available.
  </Step>
</Steps>

Publishing and deploying are independent: publish several versions before deploying any of them, or redeploy an older version at any time.

### Start an App version

Starting a version makes Domino provision resources and run the version's settings:

* **Container provisioning**: Domino creates a Kubernetes pod using the version's Compute Environment and the deployment's Hardware Tier.

* **Launch file execution**: The container runs the entry point.

* **Monitoring and uptime**: Domino monitors the process to keep the App available. If the App exits or crashes, Domino may restart it automatically depending on platform settings.

* **Secure proxying**: All requests go through Domino's proxy, which authenticates users, applies access controls, logs requests, and injects headers such as `domino-username` and the route prefix.

Starting or stopping an App never creates a version. Publishing a Draft or editing version-defining content creates one. Domino records each start as an instance in the version's history.

### Autoscaling Apps

Apps can scale automatically in response to user demand.

* **Pod replicas**: Domino uses Kubernetes to add replicas of the App container when traffic increases.

* **Load balancing**: Requests are distributed across replicas to maintain performance and responsiveness.

* **Dynamic behavior**: Scaling occurs while the App is running, with replicas added or removed based on current load.

This lets Apps serve more users without manual resource changes. Autoscaling is a deployment setting, so changing it restarts the running instance rather than creating a version. [Autoscaling Apps](/cloud/platform-capabilities/features/apps/app-logs-and-versioning/autoscaling-apps) has guidance on tuning frameworks and resource allocation.

### Stop an App

Stopping an App ends the instance and releases its resources:

* **Container shutdown**: Domino terminates the Kubernetes pod and releases the assigned compute resources.

* **Data persistence**: Any unsaved file changes are lost.

Logs and resource consumption recorded during the instance stay attached to the version. Unlike Jobs and Workspaces, Apps don't automatically persist outputs, so use Datasets or external storage if you need to save results.

## Next steps

* [App versioning and reproducibility](/cloud/platform-capabilities/features/apps/app-versioning-and-reproducibility): What a version pins, and how to start a previous one

* [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions): Create an immutable version, then start it for viewers

* [App security and identity](/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security): Control access to data and resources using identity propagation and viewer-specific permissions

* [Persist data from Apps](/cloud/platform-capabilities/features/apps/persist-data-in-apps): Save results from your App to Datasets or external storage


# Apps in Domino
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/apps-in-domino/index

Turn Project code into containerized web Apps, then preview, version, deploy, secure, share, and connect them to data and platform resources.

Use Domino Apps to turn code in your project into an interactive web application. Apps run in containerized environments that provide routing, authentication, resource management, and isolation from other project workflows.

<Frame>
  <img alt="App Detail" />
</Frame>

You can use Apps to share tools built with frameworks such as Streamlit, Dash, Shiny, and Flask. Domino handles the infrastructure so you can focus on your app logic.

## Key capabilities

* **Interactive UI delivery**: Publish live dashboards, forms, or visualization tools directly from your Domino project.

* **Built-in routing and security**: Domino proxies all requests to your App and handles identity injection, access controls, and URL routing.

* **Flexible runtime configuration**: Each App version pins its own compute environment and code commit, and each deployment picks its own hardware tier. Apps run independently of Workspaces and Jobs, but you [develop and preview them from a workspace](/cloud/platform-capabilities/features/apps/develop-and-test-an-app).

* **Multiple Apps per project**: You can define and publish multiple Apps in the same project. Each App has a unique or custom URL.

## Core concepts

Domino manages Apps through four related entities.

<Frame>
  ```mermaid placement="top-right" theme={null}
  flowchart LR
    W["Workspace"] -->|"first preview"| D["Draft App"]
    D -->|"start preview"| P["Preview instance<br/>(author-only)"]
    D -->|"publish"| V
    subgraph A["App"]
      V["Version<br/>(immutable configuration)"]
    end
    V -->|"deploy"| I["Instance"]
  ```
</Frame>

| Entity | What it is |
| :- | :- |
| **App** | The top-level entity in your project. An App exists as soon as you create it, before it is deployed. A project can hold several Apps. |
| **Draft App** | A private, author-only App used for development and testing, linked to a single workspace and created the first time you use App Preview in that workspace. In the Domino UI, Draft Apps don't appear in App lists, discovery, or search. The author works with a Draft App from its linked Workspace. Previewing it doesn't change a published App's configuration or deployed instance. Preview side effects on shared Domino resources are real. One workspace can publish several Apps and several versions, because publishing never consumes the Draft App. Deleting the workspace deletes the Draft App. |
| **Version** | An immutable snapshot of the App's identity and version-defining content, including pinned source commits, Compute Environment and revision, Data Plane, mounted storage configuration, and identity propagation. Publishing a Draft or editing version-defining content creates a version; starting the App does not. Starting an older version restores those pinned inputs, but results can differ when Dataset contents or external systems change. |
| **Instance** | A single running execution of a version. Starting a version creates an instance and stopping it ends the instance. An App has at most one running instance at a time, a version can be started many times over its life, and Domino records logs and resource consumption for each instance. Instances started from a Draft App are preview instances: they run in a real App runtime but only the author can reach them. |

This separation means you can preview changes without publishing, publish a version without deploying it, and deploy any published version, including an older one, without creating a version.

## When to use Domino Apps

Domino Apps make data science insights directly accessible across the organization through simple, browser-based interfaces. They provide a simple, browser-based interface so stakeholders and operational teams can interact with models, dashboards, and tools without technical setup.

Use cases for Domino Apps include:

* **Stakeholder delivery**: Share both early prototypes and final products in a format stakeholders can use directly.

* **Operationalizing data science**: Deploy models or workflows as interactive applications.

* **Business decision support**: Provide calculators, scenario testing, or model outputs through a user-friendly UI.

* **Analytics dashboards**: Publish visualizations and monitoring tools for stakeholders or operations.

* **Data exploration**: Enable interactive exploration of datasets, models, or what-if scenarios.

Apps are best when you need to put data science in front of the business, making results easy to consume, interpret, and act on.

Apps are not intended for persistent workflows or large-scale back-end processing. For batch operations, automation, or long-running pipelines, [Create and run Jobs](/cloud/platform-capabilities/core-concepts/jobs/create-and-run-jobs) or [Scheduled Jobs](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job). For collaborative development or exploratory work, use [Workspaces](/cloud/platform-capabilities/core-concepts/workspaces).

## How Domino runs an App

When you start an App version, Domino provisions resources and runs that version's configuration:

1. Provision a container and mount project files.

2. Apply the selected compute environment and hardware tier.

3. Execute the launch file to start the web server.

4. Route traffic through a secure proxy and inject request headers.

An App's settings split into configuration fields, which define a version, and deployment settings, which you can change in place. [App structure and lifecycle](/cloud/platform-capabilities/features/apps/apps-in-domino/app-structure-lifecycle) lists both and covers the five lifecycle stages.

### Access to Project resources

When an App runs, it executes in a container with access to specific project resources. These resources define what the App can read, write, or connect to during execution.

* **Project files**: Available read/write inside the container.

* **Datasets**: Mounted with read/write access (if enabled). Access follows the App starter’s permissions, or can be filtered by the requester’s identity through the Dataset API.

* **Data sources**: Available using the App starter’s credentials. You can also apply per-requester logic if identity propagation is enabled.

* **Environment variables**: Injected from the App starter and project settings. Useful for securely passing secrets or configuration.

**Note**: File changes inside an App container are not saved to the project repository. To persist results, write to [Datasets](/cloud/platform-capabilities/core-concepts/data/datasets) or external systems.

## Feature compatibility with Apps

Apps can integrate with many of Domino’s core platform features, but not all features behave the same way in the context of an App. Use the table below to understand what is supported, what is limited, and what to watch out for when building Apps that need access to datasets, data sources, clusters, volumes, and more.

| Feature | Supported in Apps | Notes |
| - | - | - |
| Datasets | Yes | Accessible via file API; use the Dataset API + user identity for per-user control. |
| Data Sources | Yes | App inherits the starter’s permissions. |
| Compute Clusters | Partially | Requires manual configuration and uses the App starter’s permissions for access. |
| NetApp Volumes | Yes (Read/Write) | Mounted volumes are available inside the App container. |
| Environment Variables | Yes | Project-level and user-level variables from the App starter are injected at runtime. |
| Version Control (Git) | Yes | Each version pins both the Git reference and the resolved commit ID when the version is created, so the code can't shift between starts. Apps can also run from imported Git repositories, not only the main repository and Domino File System (DFS) projects. |
| App Output Persistence | No | File changes inside the App are not saved to the project. |
| External Data Volumes (EDVs) | Yes | Mounted EDVs are available inside the App container. |

## Next steps

* [Create an App](/cloud/platform-capabilities/features/apps/create-an-app): Start an App in a workspace or from an existing file

* [Develop and test an App](/cloud/platform-capabilities/features/apps/develop-and-test-an-app): Iterate on App code in a workspace and preview it in a real App runtime

* [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions): Create an immutable version, then start it for viewers

* [App versioning and reproducibility](/cloud/platform-capabilities/features/apps/app-versioning-and-reproducibility): What a version pins, and how to start a previous one

* [App structure and lifecycle](/cloud/platform-capabilities/features/apps/apps-in-domino/app-structure-lifecycle): Understand how Apps are defined, provisioned, scaled, and stopped

* [App security and identity](/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security): Control access to data and resources using identity propagation and viewer-specific permissions

* [Persist data from Apps](/cloud/platform-capabilities/features/apps/persist-data-in-apps): Save results from your App to Datasets or external storage

* [Best Practices for Apps](/cloud/platform-capabilities/features/apps/best-practices-for-apps): Tips for building reliable, maintainable, and discoverable Apps


# App security and identity
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security

Control App access and discovery, identify viewers through headers and JWT tokens, and use extended identity propagation to call Domino APIs for them.

Domino provides three levels of identity propagation for Apps, each offering different capabilities and security levels:

* **Basic identity**: Know who is viewing your app (username only)
* **Enhanced identity**: Verify user identity with tokens (username, email, user ID)
* **Extended identity**: Act on behalf of users with their full permission (access data, submit jobs, manage projects)

## Access controls and permissions

Control who can view and edit your Apps to protect sensitive data and manage collaboration.

**Edit permissions** are inherited from the project level because apps are part of projects. When you add someone as a project collaborator, they can automatically edit any apps in that project. To add editors, add them as collaborators to the project where the app resides.

**View permissions** are set at the app level and can follow one of two modes:

* **Restricted**: Only users you explicitly list and project collaborators can view. Use **Restricted** mode when your app handles sensitive data or is intended for specific team members only.
* **Anyone in Domino**: Anyone with a Domino account can view. Use **Anyone in Domino** for internal tools, dashboards, or resources that all employees should access.

### Discovery vs. viewing

Domino distinguishes between:

* **Discovering an App**: Seeing it listed in App views
* **Viewing an App**: Opening and using it

This separation lets you create a browsable app catalog where users can explore available apps and request access to ones they need, without giving them immediate access to potentially sensitive content.

### Restricted mode options

When an app’s view permissions are set to **Restricted**, additional options appear:

* **Globally discoverable**: All Domino users can find the app and request access to view it
* **Viewers**: App owners can manage the list of users who can view the app, and accept or deny requests for access

### Where to set permissions

You can set these permissions when publishing an App in the **Access and sharing** tab, or at any time after publication by selecting **Share** from the App list view.

After you set permissions, users with view access can open and interact with your app. Users without access see the app listed (if globally discoverable) with a **Request access** button.

## Request and grant access

Access requests create a self-service model for app discovery. Users can find and request access to apps they need without contacting IT or app owners directly, while owners maintain full control over who can view their apps.

### Request access to an app

If you see **Request access** next to an app in the Domino apps list view, the app is globally discoverable but you don’t have view permissions yet. To request access:

1. Find the app in the Domino Apps list view.
2. Click **Request access**.

You’ll receive a Domino notification and an email when the owner grants you access.

### Grant access to your app

When someone requests access to your app, you’ll receive a notification in Domino and an email. To grant or deny access:

1. Click the notification link or go to your app and click **Share**.
2. Review open requests.
3. Click **Accept or Deny** for each request.

## App identity and authentication

Domino apps can identify and authenticate viewers at three different levels. Each level provides different information and capabilities to help you build personalized, secure applications.

* **Basic identity** tells you who is viewing your app by passing the username in an HTTP header. Use this for loading user-specific defaults, preferences, or displaying personalized content.
* **Enhanced identity** verifies user identity with signed tokens that include username, email, and user ID. Use this when you need cryptographic proof of identity or additional user attributes beyond the username.
* **Extended identity** lets your app act on behalf of users with their full permissions. Your app can access user data, submit jobs, manage projects, and perform other authorized actions in Domino. Users must explicitly grant consent before the app receives these permissions.

### Basic identity propagation

Use basic identity when you need to know who is viewing your app. This is useful for loading user-specific defaults, preferences, or displaying personalized content. Domino passes the username in an HTTP header called `domino-username`.

If users who aren’t logged in to Domino view your apps, the `domino-username` header value is `Anonymous`.

<Note>
  Identity headers are only available in frameworks that support proxied HTTP headers. Flask and Dash support them by default. Shiny requires that you use Server Pro.
</Note>

**Example: Display the viewer’s username**<br />This Flask example shows how to get the Domino username of an app viewer:

```bash theme={null}
#!/usr/bin/env bash
export LC_ALL=C.UTF-8
export LANG=C.UTF-8
export FLASK_APP=app.py
python -m flask run --host=0.0.0.0 --port=8888
```

Here is a simple `app.py` file that renders a template named `index.html`. This app imports `request` from `flask`, which gives you access to the headers of the active HTTP request. Apps published in Domino must serve traffic on port `8888`.

```python theme={null}
import flask
from flask import request, redirect, url_for

class ReverseProxied(object):
  def __init__(self, app):
      self.app = app
  def __call__(self, environ, start_response):
      script_name = environ.get('HTTP_X_SCRIPT_NAME', '')
      if script_name:
          environ['SCRIPT_NAME'] = script_name
          path_info = environ['PATH_INFO']
          if path_info.startswith(script_name):
              environ['PATH_INFO'] = path_info[len(script_name):]
      return self.app(environ, start_response)

app = flask.Flask(__name__)
app.wsgi_app = ReverseProxied(app.wsgi_app)

# Homepage which uses a template file
@app.route('/')
def index_page():
  return flask.render_template("index.html")
```

Create a template file at `templates/index.html`:

```python theme={null}
<!DOCTYPE html>
<html>
  <body>
    <h1>Your username is {{ request.headers.get("domino-username") }}</h1>
  </body>
</html>
```

When you host this app in Domino and open it, you’ll see the username that matches the app user.

### Enhanced identity propagation

Use enhanced identity when you need verified user information beyond just the username. This approach validates the token to make sure that identity information is authentic and hasn’t been tampered with.

When the `SecureIdentityPropagationToAppsEnabled` feature flag is enabled, Domino hosts apps at a different URL structure (https\://\<domino-domain>/apps/\<app-id>/) and passes a JWT authorization token in the Authorization HTTP Header.

You can verify the token’s integrity and decode it to get the user’s username, email, and Domino userID. The token audience is scoped to a limited set of Domino endpoints, restricting what operations the app can perform.

<Note>
  Identity headers are only available in frameworks that support proxied HTTP headers. Flask and Dash support them by default. Shiny requires that you use Server Pro.
</Note>

**Configure the base path**<br />Some application servers need to know their base path to correctly handle routing, generate URLs, and serve static assets. Without proper base path configuration, your app may fail to load resources or generate broken links.

If your application server requires a base path configuration, read it from the `DOMINO_RUN_HOST_PATH` environment variable.

**Example: Verify and decode JWT token**<br />Extract the token from the `Authorization` header:

```python theme={null}
def extract_token(self, headers):
  auth_header = headers.get('Authorization', '')
  return auth_header[7:] if auth_header.startswith('Bearer ') else None
```

Verify and decode the token (using this example JWKS):

```python theme={null}
from cryptography import x509
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import serialization
from jwt import decode, get_unverified_header
import base64, json, jwt, requests

def verify_and_decode_jwt_token(token):
    keycloak_domain = "http://keycloak-http.domino-platform"
    jwks_url = f"{keycloak_domain}/auth/realms/DominoRealm/protocol/openid-connect/certs"

    jwks_dict = requests.get(jwks_url).json()
    kid = get_unverified_header(token).get('kid')

    public_key = None
    for key in jwks_dict['keys']:
        if key['kid'] == kid:
            x5c = key['x5c'][0]
            cert = x509.load_der_x509_certificate(base64.b64decode(x5c), default_backend())
            public_key = cert.public_key()
            break
    if not public_key:
        raise ValueError("No public key found for kid")

    pem = public_key.public_bytes(
        encoding=serialization.Encoding.PEM,
        format=serialization.PublicFormat.SubjectPublicKeyInfo
    )

    return jwt.decode(token, pem, algorithms=['RS256'], audience="apps")
```

Here is a decoded payload example:

```python theme={null}
{
  "sub": "66d9fc7a538b60bef7c02c9c",
  "preferred_username": "userName",
  "email": "userName@domain.com",
  "given_name": "givenName",
  "family_name": "familyName"
}
```

### Extended identity propagation

Use extended identity when your app needs to perform actions on behalf of users. Your app can access user data, submit jobs, manage projects, and perform other actions the user is authorized to do in Domino.

When the `com.cerebro.domino.apps.extendedIdentityPropagationToAppsEnabled` configuration control is enabled, apps can request permission to act on behalf of users. Users must explicitly grant consent before the app receives these permissions.

<Note>
  Extended identity propagation can also be set for an [App Preview](/cloud/platform-capabilities/features/apps/develop-and-test-an-app), in the preview **Settings** panel, but only an administrator can set it there. The same `com.cerebro.domino.apps.extendedIdentityPropagationToAppsEnabled` central configuration control governs it.
</Note>

<Warning>
  This feature is enabled by default. Extending identity propagation gives the app access to the viewing user's credentials in addition to the publisher's credentials. It doesn't lock the app into using the viewing user's credentials, the app code has access to both and decides which is appropriate for each action. Individual apps must opt in to extended identity propagation, and only SysAdmins or CloudAdmins can publish apps that have opted in.
</Warning>

**How it works**

1. Users see a consent prompt when they open an app that requests extended permissions.
2. If users grant consent, the app can retrieve a token with full user scope.
3. The app implements code to retrieve the user’s token from the Authorization header.
4. The app calls Domino APIs as the user using this token.
5. Users can choose to remember their consent for an extended period.

Consent has an expiration date and is auditable. The default expiration is 8 hours, but users can specify up to 30 days for an expiration date on the App consent page.

**Retrieve user token**

```python theme={null}
auth_token = request.headers.get('Authorization', '')
token = auth_token[7:] if auth_token.startswith('Bearer ') else ""
```

Without extended identity propagation enabled, the same code can retrieve a token, but it has limited scope instead of full user permissions.

To enable the `com.cerebro.domino.apps.extendedIdentityPropagationToAppsEnabled` setting, the `SecureIdentityPropagationToAppsEnabled` feature flag must also be set to `true`. The app has access to the user’s full credentials only if the publisher enables extended propagation in the app and the user grants consent.

<Warning>
  Apps with extended permissions can access all data the user can access and perform actions as the user. Only enable this for apps you trust. Once you have enabled extended identity propagation on an app, it can’t be disabled.
</Warning>

**To enable extended identity propagation**

1. The configuration record must be enabled.
2. The app must have extended app propagation enabled.
3. The app’s code must actually use the viewer’s token.
4. The viewer must consent.

**Example: Use extended permissions**<br />This Flask app reads the viewer's token from the `Authorization` header and uses it to call the Domino API as the viewing user. The `/projects` route returns the projects the viewer can access, which can differ from what the publisher's credentials can see. Domino injects the API base URL into every app as the `DOMINO_API_HOST` environment variable.

```python theme={null}
import os

import requests
from flask import Flask, jsonify, request

app = Flask(__name__)

API_HOST = os.environ["DOMINO_API_HOST"]


def get_viewer_token():
    """Return the viewing user's token, or None if it isn't present."""
    auth_header = request.headers.get("Authorization", "")
    return auth_header[7:] if auth_header.startswith("Bearer ") else None


def get_publisher_token():
    """Return the publisher's token from the local token service."""
    return requests.get("http://localhost:8899/access-token").text.strip()


@app.route("/projects")
def list_viewer_projects():
    token = get_viewer_token()
    if token is None:
        message = (
            "No viewer token. Confirm that extended identity propagation "
            "is enabled for this app and that you granted consent."
        )
        return jsonify(error=message), 401

    response = requests.get(
        f"{API_HOST}/v4/projects",
        headers={"Authorization": f"Bearer {token}"},
    )
    response.raise_for_status()
    return jsonify(projects=[project["name"] for project in response.json()])
```

The app chooses which credentials to use for each action: call `get_viewer_token()` to act as the viewing user, or `get_publisher_token()` to act as the publisher. The same pattern works for any Domino API call the viewer is authorized to make, such as submitting jobs or reading datasets.

## Next steps

* [Share an App](/cloud/platform-capabilities/features/apps/share-an-app): Set sharing modes, handle access requests, and choose whose identity the App uses.
* [Allow external resources in Apps](/cloud/platform-capabilities/features/apps/authentication-and-identity/whitelist-resources): Apps may be blocked from loading content unless those URLs are explicitly allowlisted.


# Authentication and identity for Apps
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/authentication-and-identity/index

Choose basic, enhanced, or extended identity propagation for Apps, and enforce viewer-specific Dataset and NetApp Volume access through the SDK.

## How Apps use identity

An App acts with the publisher's identity and permissions by default. Basic and enhanced identity propagation let the App identify the current viewer without changing the permissions that the App uses to reach Domino resources.

Extended identity propagation lets a trusted App request a full-scope token and act with the viewer's permissions after the viewer consents. The App must opt in, and only a SysAdmin or CloudAdmin can publish an App that uses it.

[App security and identity](/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security) has complete details about identity propagation, authentication headers, and JWT tokens.

## NetApp Volume and Dataset access in Apps

By default, Apps mount NetApp Volumes and Datasets that the App publisher can access, which means viewers can inherit access to data they cannot reach directly.

To enforce viewer-specific permissions, configure identity propagation and use the `dominodatalab-data` SDK instead of the mounted file path. Pass the raw bearer token from the request's `Authorization` header to the SDK client so Domino's API can apply the viewer's permissions. The `domino-username` header and decoded token claims can identify the viewer, but they do not authenticate an SDK call.

<Tabs>
  <Tab title="NetApp Volumes">
    1. Read the `Authorization` request header and remove its `Bearer ` prefix.
    2. Pass the remaining raw token to the `dominodatalab-data` client.
    3. Use the client to read, write, or download volume files under the viewer's permissions.
  </Tab>

  <Tab title="Datasets">
    1. Read the `Authorization` request header and remove its `Bearer ` prefix.
    2. Pass the remaining raw token to `DatasetClient`.
    3. Use the client to read, write, or download Dataset files under the viewer's permissions. See the [Dataset proxy example](/cloud/reference/api/data_api/datasource_usecases/access-datasets-via-proxy) for Python code.
  </Tab>
</Tabs>

This setup lets you serve a single App to multiple users while tailoring data access according to Domino’s role-based permissions.

## Next steps

* [Apps security and identity](/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security) has complete details about identity propagation, authentication headers, and JWT tokens.
* [Persist data](/cloud/platform-capabilities/features/apps/persist-data-in-apps) using Datasets or external storage to make your Apps more dynamic and interactive.
* [Share an App](/cloud/platform-capabilities/features/apps/share-an-app) explains how to control discovery and viewing access, manage requests, and choose whose identity the App uses.


# Request headers and environment variables for Apps
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/authentication-and-identity/request-headers-and-env-variables-apps

Use domino-username, Authorization, and DOMINO_RUN_HOST_PATH to identify App viewers, validate tokens, and configure runtime routing.

When Domino publishes and runs an App, it injects useful metadata into the runtime environment. These values can help your App adjust its behavior based on user identity, routing, project settings, or access controls.

## Request headers

Domino includes several custom HTTP headers in each request forwarded to your App. These headers are only available if your app framework supports proxied HTTP headers.

| Header | Description |
| - | - |
| `domino-username` | The Domino username of the user making the request. When anonymous App access is enabled, the value is `Anonymous` for unauthenticated users. |
| `Authorization` | (Optional) A JWT token if `SecureIdentityPropagationToAppsEnabled` is enabled. Can be decoded to extract user identity metadata. |

<Note>
  These headers are supported by Dash and Flask by default. For Shiny, you must use [Shiny Server Pro](https://docs.rstudio.com/shiny-server/#proxied-headers) to receive them.
</Note>

## Environment variables

Domino also injects runtime [environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables) when an App is started. These are available to your app logic at startup.

| Variable | Description |
| - | - |
| `DOMINO_RUN_HOST_PATH` | The full URL path prefix for your App. Use this for dynamic route configuration. |

Additional environment variables may be defined at the project or user level:

* **Project-level variables**: Defined in the project settings.

* **User-level variables**: Tied to the user who starts the App.

These variables can be used to:

* Configure your App’s route prefix programmatically.

* Apply logic based on project context or user configuration.

* Pass secrets or credentials securely (via Domino’s environment variable management).

## Next steps

* [Apps in Domino](/cloud/platform-capabilities/features/apps/apps-in-domino) gives an overview of how apps work within the Domino ecosystem.

* Learn more about [how Apps in Domino run](/cloud/platform-capabilities/features/apps/authentication-and-identity) and what identity and permissions are used.

* [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions) explains how publishing a version differs from deploying it, and which settings you can change without creating a version.


# Allow external resources in Apps
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/authentication-and-identity/whitelist-resources

Identify external fonts, APIs, or embedded pages blocked by iFrame or content security policy restrictions, then request allowlist access.

If your Domino instance enforces iFrame restrictions or a content security policy (CSP), some external content may be blocked. In these cases, your administrator must allowlist specific resources before your App can access them.

## When allowlisting is required

Apps may be blocked from loading content - such as fonts, APIs, or embedded pages - unless those URLs are explicitly allowlisted. Contact your Domino admin to get the URL added to the allowlist.

### Identify blocked resources

To find out which external resources your App is being prevented from loading, check your browser’s developer console while the App is running.

1. Open your App in Domino and run it.

2. Once it’s running, open your browser’s web console. For example, in Chrome, use `Cmd+Option+J` or `Ctrl+Shift+J`.

3. Look for errors in red, such as:

   ```shell theme={null}
   Refused to connect to https://example.com because it violates the Content Security Policy.
   ```

4. Copy the URLs of any blocked resources your App depends on.

### Request access

After you identify the blocked URLs, request that your administrator allowlist them so your App can load properly.

1. Send the list of URLs to your Domino administrator and request that they be allowlisted.

2. After the allowlist is updated, stop and restart your App.

3. Once restarted, the App should be able to load the required external resources.

## Next steps

* [Share an App](/cloud/platform-capabilities/features/apps/share-an-app): Control discovery and viewing access, and manage access requests.

* [Use Best Practices for Apps](/cloud/platform-capabilities/features/apps/best-practices-for-apps): Apply best practices to make apps easier to find and maintain.


# Best practices for Domino Apps
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/best-practices-for-apps

Build maintainable Apps with previews, immutable versions, clear configuration, safe concurrent file access, stable environments, and reviewed permissions.

Follow these practices to help your apps stay clear, maintainable, and scalable as adoption grows across your organization.

## Developer practices

### Test in a preview, not by publishing

Publishing to see whether something works is slow and can disturb a deployed version. Instead, iterate in a workspace with [App Preview](/cloud/platform-capabilities/features/apps/develop-and-test-an-app) and live sync, then publish once the App behaves.

* Commit your code. Preview runs the latest committed code, so uncommitted edits are invisible to it.
* Treat side effects as real. A preview isn't sandboxed: it runs with your identity, so Jobs it starts and writes it makes to Datasets or other Domino resources happen for real.
* Validate the Compute Environment in a preview before publishing, so a broken environment never reaches a deployed version.

### Use versions rather than ad-hoc reverts

* Write **Version Notes** as you publish, so the version history explains what changed without a code archaeology session.
* To back out a bad release, [start the previous version](/cloud/platform-capabilities/features/apps/app-versioning-and-reproducibility) instead of reverting code and republishing. Each version restores its pinned code and configuration, although results can differ if Dataset contents or external systems have changed.

### Use clear titles and avoid hardcoded references

Give your App a clear, descriptive title that reflects its purpose, and avoid embedding fixed tokens or URLs that limit reusability.

* Use names like `"Churn Prediction Explorer"` or `"Marketing ROI Dashboard"` rather than `"test-app"` or `"new-dash-demo"`.
* Explain the App’s purpose in the description field: include who it’s for, what data it uses, and any assumptions or limitations.
* Avoid hardcoded tokens, URLs, usernames, or project paths in your App code. Use environment variables or configuration files instead.
* This makes your App easier to share, maintain, and move between environments like dev, staging, and production.

### Organize code cleanly

Structure your App’s code for clarity and maintainability by separating concerns and avoiding bloated single-file implementations.

* Keep your main entry file (for example, `app.py`) focused on layout, routing, or callbacks only.
* Move business logic, data loading, and reusable functions into helper modules.
* Use minimal shell scripts (for example, `app.sh`) for launching the App and handling environment setup if necessary.
* This approach simplifies debugging, encourages reuse, and makes collaboration easier.

### Manage concurrent file access in autoscaled Apps

These best practices apply to Apps that use the autoscaling feature.

Autoscaled Apps share the same disk for temporary storage and project code and files. If different replicas of the autoscaled App try to write to the same file at the same time, it can cause unexpected behavior or data corruption. To avoid issues:

* If your App code writes to disk, ensure that the write location is unique or that other App processes cannot write to the same location at the same time.
* If your App code reads from files or directories that other App code writes to, ensure that your App is reading the correct data.
* Be careful with setup or teardown code. For example, if the App’s compute environment uses pre-setup, post-setup, pre-run, or post-run scripts, ensure that it is OK if multiple processes execute those scripts at the same time. If not, modify those scripts to remove the potential issue, or create a new compute environment that performs the same operations as part of the environment definition.

### Support multiple environments

Use branches and isolated environments to separate development, staging, and production workflows.

* Create dedicated Git branches for each environment to test changes without affecting stable versions.
* Pair each branch with a corresponding Domino compute environment, such as `dev-env`, `prod-env`.
* Preview a branch before you publish it, then publish the branch that passed as a new version.
* This keeps experiments safe, reproducible, and easy to roll forward.

### Version your environment

Avoid unexpected breakages by versioning your compute environments rather than editing them directly.

* Clone your environment when making updates, instead of overwriting an existing one.
* Use semantic versioning or meaningful tags (for example, `v1.2-data-upgrade`) to track changes.
* Document the purpose of each version so you can match it to published Apps later.
* This ensures App behavior stays consistent over time, even as your tooling evolves.

### Review permissions intentionally

Control access to Apps thoughtfully to balance collaboration and security.

* Set clear access levels: use **Anyone in Domino** for internal tools available to every Domino user, and restrict sensitive ones to specific users.
* Review permissions before publishing or republishing, especially when cloning or renaming Apps.
* Sharing an App URL doesn't bypass Domino access controls.
* This helps prevent unintentional exposure and keeps your platform governance strong.

## User experience and support

These tips ensure your Apps are approachable, well-documented, and easy for others to use, support, and extend, whether you’re sharing with a small team or a wider audience.

### Link to supporting documentation

Help users understand how to use your App, and what it’s doing under the hood.

* Add links to internal wikis, technical guides, or dashboards that explain data sources, outputs, and model behavior.
* Mention key assumptions or limitations directly in the App description.
* Consider adding an in-App link - something like a "Learn more" button - to surface docs during usage.
* This reduces confusion and increases the App’s usefulness across teams.

### Provide contact info for feedback

Make it easy for users to reach out when they have questions or run into issues.

* Domino automatically includes an **Email Owner** option in the App toolbar. Encourage users to use it.
* Optionally, include a contact email or Slack handle in the App UI or footer.
* For shared tools, consider a team alias or support channel instead of a personal address.
* This keeps feedback flowing and helps you spot bugs or usability gaps early.

## Next steps

* [Apps in Domino](/cloud/platform-capabilities/features/apps/apps-in-domino) gives an overview of how apps work within the Domino ecosystem.
* [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions) explains how publishing a version differs from deploying it, and which settings you can change without creating a version.
* [App security and identity](/cloud/platform-capabilities/features/apps/authentication-and-identity) explains how to control access to data and resources using identity propagation and viewer-specific permissions.


# Publish a Dash App in Domino
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/common-app-frameworks/dash-app

Build a Dash App with a custom Compute Environment, DOMINO_RUN_HOST_PATH routing, port 8888, a startup script, and publish and sharing settings.

Dash is a popular Python framework for building interactive web applications. This guide shows how to run Dash apps in Domino, without diving into how Dash itself works. If you’re new to Dash or want examples of what’s possible, visit the [Dash Gallery](https://dash.gallery) or the [Dash documentation](https://dash.plotly.com/).

## What you’ll do

* Set up a custom Domino environment that includes Dash
* Create a project and configure it for app publishing
* Add and run Dash sample code that works with Domino routing
* Publish and share your app using the Domino Launchpad interface

### Step 1: Set up a compute environment

First, create a Domino compute environment that includes Dash:

1. In Domino, go to **Govern** > **Environments** > **Create Environment**.
2. Provide a name and description.
3. Select the **Domino Standard Environment** as the base image.
4. Click **Create Environment**.
5. To add Dash, click **Edit Definition** and append the following lines to the Dockerfile:
   ```python theme={null}
   USER root
   RUN pip install dash==3.0.4
   USER ubuntu
   ```
6. Then, click **Build** and wait for the process to complete.

This sets up a custom environment that supports Dash apps.

### Step 2: Create a Project

Now you’ll create a project in Domino to hold your Dash app:

1. From Domino, go to **Develop** > **Projects** > **Create Project**.
2. Name your Project and click **Create**.
3. In the sidebar, go to **Settings** > **Compute environment** and select the environment you just built.
4. Go to **Code** > **Add File**, name the file `app.py`, and click **Save**.

Your project is now configured with the correct environment and is ready for app development.

### Step 3: Add app code

You’ll now add the Python code for your Dash app. Domino requires a few lines of setup to handle routing correctly, but you can use any Dash app layout or logic you like. This basic app confirms that Dash is working in Domino. It renders a static Plotly chart with no data dependencies or interactivity.

To make your Dash app work in Domino, set routing with Domino environment variables and serve the app from `0.0.0.0` on port `8888`.

1. Open the `app.py` file you created earlier.
2. Paste in the following code and save the file:
   ```python theme={null}
   import os
   import plotly.express as px
   from dash import Dash, dcc, html
   # Domino-specific routing
   runurl = os.environ.get("DOMINO_RUN_HOST_PATH")
   # Create app with routing config
   app = Dash(__name__,
               routes_pathname_prefix='/',
               requests_pathname_prefix=runurl)
   # Basic static chart
   fig = px.scatter(x=[1, 2, 3], y=[4, 1, 6], title="Sample Scatter Plot")
   # App layout
   app.layout = html.Div([
       html.H1("Minimal Dash App"),
       dcc.Graph(figure=fig)
   ])
   # Launch on a port and Domino-compatible host
   if __name__ == '__main__':
       app.run(port=8888, host='0.0.0.0', debug=False)
   ```

### Step 4: Add a startup script

Domino runs your app using a startup script. Create it like this:

1. In the project sidebar, go to **Code** > **Add File**.
2. Name the file `app.sh` and add `python app.py` to the file.
3. Click **Save**.

This tells Domino how to launch your app during publishing.

### Step 5: Publish and deploy the App

Before Cloud Release 90, selecting **Publish App** created and deployed the App in one step. Starting with Cloud Release 90, the steps are separate: preview the App in a Workspace, publish a version, then deploy that version.

<Steps>
  <Step title="Create the App">
    On your project's **Deploy > Apps & Agents** page, select **Create**, then **App in Workspace**. Any workspace you already have open works too. See [Create an App](/cloud/platform-capabilities/features/apps/create-an-app).
  </Step>

  <Step title="Preview it">
    In the workspace, open the **App Preview** tab and set the App file, Compute Environment, and Hardware Tier. Commit your code first, because preview runs the latest committed code. See [Develop and test an App](/cloud/platform-capabilities/features/apps/develop-and-test-an-app).
  </Step>

  <Step title="Publish a version">
    Select **Publish App**, then **Publish as a new App**. This records an immutable version and doesn't start it.
  </Step>

  <Step title="Deploy the version">
    On the App details page, select the version and select **Deploy**, then set access under **Access and sharing**. See [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions) and [Share an App](/cloud/platform-capabilities/features/apps/share-an-app).
  </Step>
</Steps>

Set Permissions to **Anyone in Domino** to share the App with every Domino user, and leave **Make globally discoverable** selected so others can find it. You can change both later.

Once the app is Running, click **View App** to open it in a new tab. This launches your Dash app with a Domino toolbar and shareable link.

### Step 6: Share the App

To share your app:

1. Go to **Deployments** > **Apps & Agents**, and click **Copy App Link**.
2. Open the link while signed in as a Domino user who isn't a Project collaborator to test broad access.
3. Share the URL with collaborators.
4. To browse other published apps, go to **Deploy** > **Apps** in the top nav.

Your Dash app is now hosted, versioned, and accessible in Domino. To explore more app designs or features, visit the [Dash Gallery](https://dash.gallery) or the [Dash documentation](https://dash.plotly.com/). These resources offer a wide range of examples you can experiment with and adapt to your own needs.

## Next steps

* [Apps in Domino](/cloud/platform-capabilities/features/apps/apps-in-domino) gives an overview of how apps work within the Domino ecosystem.
* [Create an App](/cloud/platform-capabilities/features/apps/create-an-app) covers the two entry points for a new App, and why creating one no longer deploys it.
* [Learn more about how apps in Domino](/cloud/platform-capabilities/features/apps/authentication-and-identity) run and what identity and permissions are used.


# Publish a Flask App
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/common-app-frameworks/flask-app

Structure a Flask App with package, static, template, view, and startup files, configure port 8888, then preview, publish, deploy, and share it.

This guide structures your Domino project as a Flask-powered website. Get the folders and files in place and the app runs correctly.

## Create the App structure

The final structure is shown below, with each area discussed in turn. Note that `/mnt` is your Files directory on Domino.

Also, the following names matter: `__init__.py`, `/static`, `/templates`.

Besides those, everything can be customized to your liking in this topic.

```shell theme={null}
/mnt
|-- ...
|-- run.py
|-- /flask_app
    |-- __init__.py
    |-- views.py
    |-- /static
    |-- /templates
```

### Create the folders

In the Files section of Domino make a new folder called `flask_app`. This is a Python package that contains the web app.

In the `flask_app` folder put two other folders, one called `static` the other `templates`. In the `static` folder you put your static files and in the `templates` folder you put your template files. The static and templates folders are recognized by Flask to know from where to fetch the appropriate files. It’s possible to change which folder is noticed as the static folder, but common practice is to just leave it as `static`.

### Create the files

Create an `__init__.py` file in the `flask_app` folder. This file makes `flask_app` a Python package. In that file put the following:

```python theme={null}
from flask import Flask

class ReverseProxied(object):
  def __init__(self, app):
      self.app = app
  def __call__(self, environ, start_response):
      script_name = environ.get('HTTP_X_SCRIPT_NAME', '')
      if script_name:
          environ['SCRIPT_NAME'] = script_name
          path_info = environ['PATH_INFO']
          if path_info.startswith(script_name):
              environ['PATH_INFO'] = path_info[len(script_name):]
      # Setting wsgi.url_scheme from Headers set by proxy before app
      scheme = environ.get('HTTP_X_SCHEME', 'https')
      if scheme:
        environ['wsgi.url_scheme'] = scheme
      # Setting HTTP_HOST from Headers set by proxy before app
      remote_host = environ.get('HTTP_X_FORWARDED_HOST', '')
      remote_port = environ.get('HTTP_X_FORWARDED_PORT', '')
      if remote_host and remote_port:
          environ['HTTP_HOST'] = f'{remote_host}:{remote_port}'
      return self.app(environ, start_response)

app = Flask(__name__)
app.wsgi_app = ReverseProxied(app.wsgi_app)
```

Back in your Files section of Domino create a new file called `run.py`.

In `run.py` add the following:

```python theme={null}
from flask_app import app
from flask_app import views

if __name__ == '__main__':
    app.run()
```

Also in your **Files** section of Domino, create a new file with the following content:

```shell theme={null}
#!/usr/bin/env bash
export FLASK_APP=run.py
python -m flask run --host=0.0.0.0 --port=8888
```

The purpose of this file is to provide the configuration including the host and port for your flask app when starting the web hosting process. You select this file as your app file, which Domino then runs within the container that contains your project files.

<Note>
  Domino requires the App service to use host `0.0.0.0` and port `8888`.
</Note>

### Customization

Now we’re going to add a `views.py` file. This file is where you place all your app logic and should be created in the `flask_app` directory.

We’ll make a simple app that just returns “Hello World!”

Add the following to `views.py`:

```python theme={null}
from flask_app import app

@app.route('/')
def index():
    return "Hello World!"
```

If you have a template in the templates folder, you would add:

```python theme={null}
from flask import render_template
from flask_app import app

@app.route('/')
def index():
    return render_template('hello.html')
```

## Publish and deploy the App

Before Cloud Release 90, selecting **Publish App** created and deployed the App in one step. Starting with Cloud Release 90, the steps are separate: preview the App in a Workspace, publish a version, then deploy that version.

<Steps>
  <Step title="Create the App">
    On your project's **Deploy > Apps & Agents** page, select **Create**, then **App in Workspace**. Any workspace you already have open works too. See [Create an App](/cloud/platform-capabilities/features/apps/create-an-app).
  </Step>

  <Step title="Preview it">
    In the workspace, open the **App Preview** tab and set the App file, Compute Environment, and Hardware Tier. Commit your code first, because preview runs the latest committed code. See [Develop and test an App](/cloud/platform-capabilities/features/apps/develop-and-test-an-app).
  </Step>

  <Step title="Publish a version">
    Select **Publish App**, then **Publish as a new App**. This records an immutable version and doesn't start it.
  </Step>

  <Step title="Deploy the version">
    On the App details page, select the version and select **Deploy**, then set access under **Access and sharing**. See [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions) and [Share an App](/cloud/platform-capabilities/features/apps/share-an-app).
  </Step>
</Steps>

Set permissions to **Anyone in Domino** to let any user with a Domino account open the App from its URL, and select **Make globally discoverable** so others can find it. You can change both later. Once the App status says **Running**, select **View App** to load it.

## Share and consume

Use **Anyone in Domino** to share the App with colleagues who have Domino accounts. To test broad access, select **Deployments > App > Copy App Link**, then open the link while signed in as a Domino user who isn't a Project collaborator.

To browse more apps, click **Deploy > Apps** in the top navigation menu.


# Host HTML pages
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/common-app-frameworks/host-html

Serve static HTML and JavaScript from a Domino App with Python's http.server, an app.py entry point, and the required port 8888.

This example integrates a simple Python HTTP server with Domino to serve static HTML. You can extend it with JavaScript or additional pages in your project.

## Files

You’ll need to create a file in your project (in addition to your files required for your page such as `index.html`).

`app.py`

```python theme={null}
import http.server
import socketserver
PORT = 8888
Handler = http.server.SimpleHTTPRequestHandler
httpd = socketserver.TCPServer(("", PORT), Handler)
print ("serving at port", PORT)
httpd.serve_forever()
```

When publishing the App, select `app.py` as the App file. When Domino launches your App, it runs `app.py` in the container that contains your project files.

Apps published in Domino must serve traffic on port `8888`.

## Next steps

* [Apps in Domino](/cloud/platform-capabilities/features/apps/apps-in-domino) gives an overview of how apps work within the Domino ecosystem.
* [Create an App](/cloud/platform-capabilities/features/apps/create-an-app) covers the two entry points for a new App, and why creating one no longer deploys it.
* [Learn more about how Apps in Domino](/cloud/platform-capabilities/features/apps/authentication-and-identity) run and what identity and permissions are used.


# Common App Frameworks
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/common-app-frameworks/index

Start Dash, Streamlit, Flask, and Shiny Apps with minimal code, required host and port settings, startup scripts, and routing configuration.

Domino can run apps built in a variety of popular frameworks. Below are minimal examples that demonstrate how to start each supported type of app. These are intended to help you get a working stub up and running quickly.

<Note>
  Apps in Domino must serve traffic on TCP port `8888`. The examples below use that port.
</Note>

## Dash (Python)

Dash is a lightweight Python framework for building interactive analytic apps with Plotly visualizations.

```python theme={null}
import dash

app = dash.Dash(__name__, routes_pathname_prefix='/')
if __name__ == '__main__':
    app.run(port=8888, host='0.0.0.0', debug=False)
```

* Make sure `dash` is installed in your environment.

* Use `routes_pathname_prefix='/'` for proper routing inside Domino.

## Streamlit (Python)

Streamlit is a rapid app framework for turning Python scripts into shareable web apps with minimal code.

```python theme={null}
import streamlit as st

st.title("Minimal Streamlit App")
st.write("Hello from Domino!")
```

In your `.sh` file, launch with:

```bash theme={null}
streamlit run app.py --server.port=8888 --server.address=0.0.0.0
```

* Streamlit apps require a separate `.sh` such as `app.sh` startup script.

* You can configure the route prefix using `DOMINO_RUN_HOST_PATH`.

## Flask (Python)

Flask is a lightweight Python web framework that is ideal for creating custom endpoints or fully bespoke applications.

```python theme={null}
from flask import Flask
import os

app = Flask(__name__)

@app.route("/")
def index():
    return "Hello from Flask in Domino!"

if __name__ == '__main__':
    app.run(port=8888, host='0.0.0.0')
```

* Works out of the box with Domino’s routing and identity headers.

## Shiny (R)

Shiny is R’s standard framework for building interactive web applications with reactive components.

### Option 1: Inline R script

```r theme={null}
shiny::runApp("./", port = 8888, host = "0.0.0.0")
```

### Option 2: Shell script to launch Shiny

In your `.sh` file:

```sh theme={null}
R -e 'shiny::runApp("./", port=8888, host="0.0.0.0")'
```

Note the following:

* Shiny must run on `0.0.0.0:8888` to work in Domino.

* User-level identity headers are only supported by Shiny Server Pro.

These minimal examples are designed to help you quickly stand up a working app in Domino using your preferred framework. Each framework has specific requirements for launch behavior and routing, but all follow the same basic pattern.

## Next steps

* [Apps in Domino](/cloud/platform-capabilities/features/apps/apps-in-domino) gives an overview of how apps work within the Domino ecosystem.

* [Create an App](/cloud/platform-capabilities/features/apps/create-an-app) covers the two entry points for a new App, and why creating one no longer deploys it.

* [Learn more about how Apps in Domino](/cloud/platform-capabilities/features/apps/authentication-and-identity) run and what identity and permissions are used.


# Publish a Shiny App
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/common-app-frameworks/shiny-app

Create a two-file Shiny App, configure its server and UI scripts to use 0.0.0.0:8888, then preview, publish, deploy, scale, and share it.

In this tutorial, you:

* Create a project and set it up for App publishing
* Publish a simple two file [Shiny](https://shiny.rstudio.com/) App to make it globally discoverable
* Observe how other users in Domino can use the App

You’ll be working with the [Telephones by region](https://shiny.rstudio.com/gallery/telephones-by-region.html) example from the [Shiny gallery](https://shiny.rstudio.com/gallery/). In this example, the application serves an interactive bar chart of consumer telephones by region from 1951 to 1961.

It takes approximately 10 minutes to get this example running in Domino.

## Set up the Project

The first step is creating a project with the settings and content you need to publish your App.

1. From the home page, click **Develop > Projects**.
2. Click **New Project**.
3. Give your project an informative name, choose the **Private** visibility setting, then click **Create Project**.
4. Click **Settings** in the project sidebar, then set the Compute environment to **Domino Standard Environment Py3.8 R4.1**. Read [Domino standard environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) to learn more about the contents of this base image.
5. Click **Code** in the project sidebar, then click **Add File**.
6. Name the file `server.R` in the title field above the editor. It’s important that the file be named exactly this, as launching Shiny application with two files, requires a directory to contain files named `server.R` and `ui.R.`
7. In the body of the file, paste the following example Shiny server code.
   ```r theme={null}
   # Rely on the 'WorldPhones' dataset in the datasets
   # package (which generally comes preloaded).
   library(datasets)
   # Define a server for the Shiny app
   function(input, output) {
     # Fill in the spot we created for a plot
     output$phonePlot <- renderPlot({
     # Render a barplot
     barplot(WorldPhones[,input$region]*1000,
     main=input$region,
     ylab="Number of Telephones",
     xlab="Year")
     })
   }
   ```
8. Click **Save** when finished.
9. Click **Add File** again, and name the new file `ui.R`.
10. Paste in the following content, then click **Save** when finished.
    ```r theme={null}
    # Rely on the 'WorldPhones' dataset in the datasets
    # package (which generally comes preloaded).
    library(datasets)
    # Use a fluid Bootstrap layout
    fluidPage(
      # Give the page a title
      titlePanel("Telephones by region"),
      # Generate a row with a sidebar
      sidebarLayout(
        # Define the sidebar with one input
        sidebarPanel(
          selectInput("region", "Region:",
                      choices=colnames(WorldPhones)),
          hr(),
          helpText("Data from AT&T (1961) The World's Telephones.")
        ),
        # Create a spot for the barplot
        mainPanel(
          plotOutput("phonePlot")
        )
      )
    )
    ```
11. The last thing to do before publishing your App is to create an `app.R` file. This is a script that Domino runs after initializing the host that serves your App. It should contain all commands required to launch your App. In this example, the only command you need is:
    ```r theme={null}
    shiny::runApp("./", port = 8888, host = "0.0.0.0")
    ```
    The `host` and `port` parameters control where Domino directs App traffic. Apps published in Domino must use host `0.0.0.0` and port `8888`. Create the `app.R` file the same way you created `server.R` and `ui.R`, then save it.

## Publish and deploy the App

Creating an App no longer deploys it. Preview the App in a workspace, publish a version, then deploy that version. Select `app.R` as your App file.

<Steps>
  <Step title="Create the App">
    On your project's **Deploy > Apps & Agents** page, select **Create**, then **App in Workspace**. Any workspace you already have open works too. See [Create an App](/cloud/platform-capabilities/features/apps/create-an-app).
  </Step>

  <Step title="Preview it">
    In the workspace, open the **App Preview** tab and set the App file, Compute Environment, and Hardware Tier. Commit your code first, because preview runs the latest committed code. See [Develop and test an App](/cloud/platform-capabilities/features/apps/develop-and-test-an-app).
  </Step>

  <Step title="Publish a version">
    Select **Publish App**, then **Publish as a new App**. This records an immutable version and doesn't start it.
  </Step>

  <Step title="Deploy the version">
    On the App details page, select the version and select **Deploy**, then set access under **Access and sharing**. See [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions) and [Share an App](/cloud/platform-capabilities/features/apps/share-an-app).
  </Step>
</Steps>

Set Permissions to **Anyone in Domino** to let any user with a Domino account open the App from its URL. Once the App status says **Running**, select **View App** to load it. You should see the interactive bar chart with a Domino toolbar above it showing the Project it's published from, plus buttons to email the App owner and open the description panel.

## Scale a Shiny App

Shiny is single-threaded, so autoscale horizontally with session affinity enabled. See [R/Shiny autoscaling guidance](/cloud/platform-capabilities/features/apps/app-logs-and-versioning/autoscaling-apps#rshiny) for recommended settings.

## Share and consume

Use **Anyone in Domino** to share the App with colleagues who have Domino accounts. To test broad access, select **Deployments > App > Copy App Link**, then open the link while signed in as a Domino user who isn't a Project collaborator.

To browse more apps, click **Deploy > Apps** in the top navigation menu.


# Publish a Streamlit LLM App in Domino
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/common-app-frameworks/streamlit-app-llm

Build a Streamlit chatbot that calls Hugging Face and Domino endpoints, configure its environment and port, then preview, publish, and share it.

This guide walks you through building and deploying a chatbot powered by large language models (LLMs) using the [Streamlit framework](https://docs.streamlit.io/) in Domino.

The app combines a couple of models hosted on [Hugging Face](https://huggingface.co/inference-endpoints/dedicated) with a model served via a [Domino endpoint](/cloud/platform-capabilities/features/model-deployment) through a single, interactive UI.

## What you’ll do

* Set up API access and compute environment prerequisites

* Use [Domino’s example code](https://github.com/dominodatalab/docs_llm_app_example) to build a Streamlit chatbot with selectable model backends

* Integrate Hugging Face and Domino endpoints

* Configure app startup and publish it in Domino

* Share the app with others in your organization

## Prerequisites

Before you begin, make sure you have the following in place:

* **Hugging Face API key**\
  Get a Hugging Face API key to access models hosted on Hugging Face:

  1. Create your account on [Hugging Face](https://huggingface.co/join).

  2. Get your Access token [from Hugging Face](https://huggingface.co/settings/tokens).

  3. In Domino, go to **Account Settings** > **Environment Variables** and add a [user-level variable](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables):

     1. **Name**: `HUGGING_FACE_API_TOKEN`

     2. **Value**: `your key`

* **Domino compute environment**\
  If you’re using a **Domino Standard Environment (DSE)**, these are already included. Your environment must include:

  1. Jupyter or JupyterLab

  2. The [jupyter-server-proxy](https://pypi.org/project/jupyter-server-proxy/) package

  3. Streamlit and necessary Python libraries

* **Domino endpoint**\
  To demonstrate multi-model access, this guide uses:

  1. Hugging Face-hosted models, such as Llama 3.2 3B and Mistral 7B.

  2. A model deployed as a Domino-hosted prediction endpoint. It is assumed you already have a Domino endpoint configured and available.

## Step 1: Set up the environment

To run and publish your Streamlit chatbot, first configure a Domino environment with the required packages. Streamlit apps are launched through a streamlit run and are hosted at port `8501` by default. You can configure this to use a specific port later.

1. Go to **Environments** > **Create Environment**.

2. Enter a name and description.

3. Select a base image, such as the **Domino Standard Environment**.

4. In the Dockerfile, add the following if not already present:

   ```python theme={null}
   USER root
   RUN pip install --no-cache-dir \
       streamlit \
       jupyter-server-proxy \
       requests \
       pandas
   USER ubuntu
   RUN pip install transformers torch streamlit
   ```

5. Click **Build** and wait for the environment to finish building.

You now have a compute environment with Streamlit and proxy support, ready to run your chatbot app.

## Step 2: Create the Project and build the App UI

Next, you’ll create a new Domino project and begin building your chatbot app using Streamlit. You’ll start with a basic layout: a title, a sidebar with model selection, and a password field for your Hugging Face API key.

You’ll develop the app interactively inside a Jupyter or JupyterLab workspace and view it live in a browser tab using a URL generated by Domino.

### Create the Domino Project

First, create the Domino Project you’ll use for your app:

1. Go to your Domino home page, or your Projects page, and choose **Develop** > **Projects** > **Create Project**.

2. Name your project something like `streamlit-llm-chatbot` and click **Create**.

3. In the project sidebar, go to **Settings** > **Compute environment** and select the environment you just created.

### Create the app title and sidebar

Next, create the app title and sidebar for the app:

1. Launch a new Jupyter or JupyterLab workspace.

2. In the `/mnt` directory, create a new file named `chatbot.py`.

3. Add the starter code found here in Domino’s [example streamlit LLM app repo](https://github.com/dominodatalab/docs_llm_app_example).

4. Update the following values in your file:

   1. `DOMINO_ENDPOINT_URL`: set to your Domino-hosted model endpoint

   2. `DOMINO_MODEL_ACCESS_TOKEN`: your model’s access token

More information about Domino endpoints and tokens can be found in the [Select Domino endpoint authorization mode](/cloud/platform-capabilities/features/model-deployment/domino-endpoint-authorization) documentation.

### Get your App URL

To preview your app, construct the URL based on your Domino run context.

1. Replace `//your-domino-url/` with your actual Domino domain and run:

   ```python theme={null}
   echo -e "import os\nprint('https://your-domino-url/{}/{}/notebookSession/{}/proxy/8501/'.format(os.environ['DOMINO_PROJECT_OWNER'], os.environ['DOMINO_PROJECT_NAME'], os.environ['DOMINO_RUN_ID']))" | python3
   ```

   <Note>
     Port `8501` is Streamlit's default in a Workspace. Published Apps must serve traffic on port `8888`; [Step 4](#step-4-configure-the-app-file-and-prepare-for-publishing) configures that port.
   </Note>

2. Run the app in your workspace:

   ```python theme={null}
   streamlit run chatbot.py
   ```

3. Open the app in a browser tab using the generated URL.

You now have a live app preview with a sidebar and model selector ready for user interaction and backend logic.

## Step 3: Add model logic and user interaction

Now that the UI is in place, you’ll add logic to handle chat input, store conversation history, and route model calls based on user selection.

The app supports both a Hugging Face-hosted model and a Domino-hosted endpoint. This step uses Streamlit session state to maintain chat history across user inputs.

### Store chat messages in session state

This logic is already included in the `chatbot.py` code. It initializes and displays the conversation history.

### Define model query functions

This app already includes logic to:

* Call Hugging Face-hosted models via InferenceClient.

* Fallback to direct HTTP calls if needed.

* Call your Domino-hosted model endpoint.

All function definitions are in the file, including:

* `chat_with_hf(…​)`

* `chat_with_api_direct(…​)`

* `query_domino_endpoint(…​)`

### Capture user input and generate a response

User prompts are captured using `st.chat_input`, and responses are routed depending on the selected model. The logic makes sure that:

* Messages are appended to the session state

* Responses are streamed with a spinner

* Fallbacks are handled gracefully

The `chatbot.py` file has the full implementation details.

### Sync all changes

Once your file is complete and tested interactively in your workspace, you’ll need to [sync all changes](/cloud/platform-capabilities/core-concepts/workspaces/sync-changes-in-a-workspace#sync-changes-to-files).

1. Save all files, including `chatbot.py` and any updates to tokens or environment values.

2. If you’re ready to move on from development, [stop your workspace](/cloud/platform-capabilities/core-concepts/workspaces/manage-workspaces#stop-a-workspace). This makes sure Domino syncs all file changes back to your project.

3. Confirm that both files - `chatbot.py` and `app.sh` - exist in your project root.

Your app is now fully interactive, capturing input, maintaining history, and calling different LLMs based on user selection.

## Step 4: Configure the app file and prepare for publishing

To publish your Streamlit app in Domino, you need a startup script that tells Domino how to launch the app and configure it to use a specific host and port.

* **Problem**: Streamlit defaults to port `8501`, but Apps published in Domino must serve traffic on port `8888`.

* **Solution**: Override the default settings using a `config.toml` file inside your `app.sh` startup script.

### Create the configuration file

In the root of your project, create a file named `app.sh` with the following content:

```python theme={null}
#!/bin/bash

mkdir -p ~/.streamlit

cat << EOF > ~/.streamlit/config.toml
[browser]
gatherUsageStats = true

[server]
port = 8888
enableCORS = false
enableXsrfProtection = false
address = "0.0.0.0"
EOF

streamlit run chatbot.py
```

**Verify**: Make sure the file is executable by running `chmod +x app.sh` in the terminal from inside your workspace, if needed.

### Why this works

* The `config.toml` file tells Streamlit to use the required port `8888` and host `0.0.0.0`.

* Domino runs `app.sh` automatically when launching your app with no manual intervention needed.

You’re now ready to publish the chatbot as a live Domino app.

## Step 5: Publish and deploy the app

Domino apps run inside containers based on your project's environment and are launched by executing the `app.sh` script you configured in the previous step. Creating an App no longer deploys it: preview the App in a workspace, publish a version, then deploy that version.

<Steps>
  <Step title="Create the App">
    On your project's **Deploy > Apps & Agents** page, select **Create**, then **App in Workspace**. Any workspace you already have open works too. See [Create an App](/cloud/platform-capabilities/features/apps/create-an-app).
  </Step>

  <Step title="Preview it">
    In the workspace, open the **App Preview** tab and set the App file, Compute Environment, and Hardware Tier. Commit your code first, because preview runs the latest committed code. See [Develop and test an App](/cloud/platform-capabilities/features/apps/develop-and-test-an-app).
  </Step>

  <Step title="Publish a version">
    Select **Publish App**, then **Publish as a new App**. This records an immutable version and doesn't start it.
  </Step>

  <Step title="Deploy the version">
    On the App details page, select the version and select **Deploy**, then set access under **Access and sharing**. See [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions) and [Share an App](/cloud/platform-capabilities/features/apps/share-an-app).
  </Step>
</Steps>

Give the App an informative title and description, for example **LLM Chatbot with Hugging Face and Domino Endpoints**. Set the permissions to **Anyone in Domino** so other users on your Domino deployment can view it, and leave **Make globally discoverable** selected so they can find it from the **Deploy** > **Apps** view. You can change both settings later.

Once the app status changes to **Running**, select **View App** to open your fully interactive chatbot in a new browser tab, with model selection, live input, and responses from either the Hugging Face-hosted or Domino-hosted model.

## Step 6: Share the app

Once your chatbot is running, you can share it with colleagues or verify that another Domino user can open it.

Access depends on the permissions you set during publishing. **Anyone in Domino** allows any user with a Domino account to view the App through its URL.

### Copy the App link

* From the **Deployments** > **App screen**, click **Copy App Link**.

* Share the link with others who have access to your Domino instance.

### Test access

Open the App URL while signed in as a Domino user who isn't a Project collaborator to confirm broad access.

If you’re prompted for a Hugging Face token, enter it in the sidebar to test API access as a new user.

### Browse or discover more apps

* From the top nav, go to **Deploy > Apps** to explore other published tools within your organization.

You’ve now successfully published and shared an LLM-powered Streamlit chatbot with interactive inputs and backend model integration.

## Next steps

* [Apps in Domino](/cloud/platform-capabilities/features/apps/apps-in-domino) gives an overview of how apps work within the Domino ecosystem.

* [Create an App](/cloud/platform-capabilities/features/apps/create-an-app) covers the two entry points for a new App, and why creating one no longer deploys it.

* [Learn more about how apps run in Domino](/cloud/platform-capabilities/features/apps/authentication-and-identity) and what identity and permissions are used.


# Create an App
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/create-an-app

The two entry points for creating an App, in a workspace or from an existing file, and why creating an App no longer deploys it.

From your project's **Deploy > Apps & Agents** page, select **Create**, then choose an entry point. **App in Workspace** opens a workspace; Domino creates its Draft App when you start the first preview. **Create from file** creates a published App and its first version. Neither path deploys the App automatically.

## Choose an entry point

<Columns>
  <Card title="App in Workspace">
    Labeled **App in Workspace**. Domino starts a workspace where you build the App with the Draft App, preview, and publish workflow. Use this for new Apps.
  </Card>

  <Card title="Create from file">
    Initializes the App from an existing file in your project and goes straight to the publish workflow, creating the App with an immutable first version.
  </Card>
</Columns>

The workspace workflow isn't limited to workspaces Domino starts for you. Any workspace in the **Workspaces** section works the same way, so you can start previewing from a workspace you already have open. [Develop and test an App](/cloud/platform-capabilities/features/apps/develop-and-test-an-app) covers that workflow.

## Creating an App doesn't make it reachable

Creation, publication, and deployment are separate steps. After you create an App and publish a version, you still have to deploy that version before anyone can open it. Continue with [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions).

<Note>
  In earlier releases, **Publish App** was the only way to create an App, and publishing also deployed it. **Create** replaces that entry point, and publishing and deploying are now separate steps. If your administrator disables the `EnableAppsSDLC` feature flag, which is on by default, the legacy Publish flow is unchanged.
</Note>

## Related

* [Develop and test an App](/cloud/platform-capabilities/features/apps/develop-and-test-an-app): Iterate on App code in a workspace and preview it in a real App runtime.
* [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions): Create an immutable version, then start it.
* [Common App frameworks](/cloud/platform-capabilities/features/apps/common-app-frameworks): Minimal working examples for Streamlit, Dash, Flask, and Shiny.
* [Apps in Domino](/cloud/platform-capabilities/features/apps/apps-in-domino): How Apps, Draft Apps, versions, and instances relate.


# Develop and test an App
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/develop-and-test-an-app

Iterative App development in a workspace, using a Draft App, App Preview, and live sync to test code in a real App runtime without publishing.

Author App code in any workspace and test it with **App Preview**, which runs that code in a real App runtime without publishing it and without touching any deployed App.

## Why preview instead of publishing to test

Workspaces differ from App runtimes in ways that make in-workspace testing unreliable: authentication and token behavior, port and networking configuration, URL patterns, mounts, Compute Environment, and the environment variables Domino injects, such as `DOMINO_RUN_HOST_PATH`. Publishing to test is slow and can disturb a deployed production version.

Preview runs your code in a dedicated App pod on the same infrastructure a deployed App uses, but only you can reach it and it never appears in App discovery.

## How preview works

You develop in an ordinary workspace. The workspace isn't bound to an App up front, and any workspace works. The first time you configure and start a preview, Domino creates a **Draft App** and links it to that workspace. A Draft App is private to its author: it never appears in App lists, discovery, or search for anyone else, and previewing it doesn't change a published App's configuration or deployed instance.

<Steps>
  <Step title="Open the App Preview tab in your workspace">
    You can also use the App Preview shortcut and name the App file directly.
  </Step>

  <Step title="Configure the App under Settings">
    Set the App file, Compute Environment, and Hardware Tier. [Extended identity propagation](/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security) can also be set here, but only by an administrator. Starting the preview creates the Draft App and ties it to this workspace.
  </Step>

  <Step title="Wait for the first load">
    Domino provisions a dedicated App pod and runs your code. The first load is a normal cold start, comparable to starting a deployed App.
  </Step>
</Steps>

While a preview runs, you can sync code changes with live sync, inject query parameters to test parameterized behavior, copy the preview URL (`/preview/<draftAppId>`) to open the preview directly, and view App logs or change preview settings from the workspace sidebar. The preview URL is reachable only by the App author.

The logs panel also offers a **Support bundle** download for the running preview instance, the same troubleshooting archive available from the App details page. The download appears for the App owner when the deployment allows support bundle downloads, and for users who can read Kubernetes resources. [App logs and monitoring](/cloud/platform-capabilities/features/apps/app-logs-and-versioning) describes what the bundle contains.

Only one preview instance runs per Draft App at a time, and starting a new preview stops the existing one. Preview instances don't count against the per-project limit on running Apps, so a running preview never blocks a production App launch.

<Warning>
  **Preview is not sandboxed.** A preview instance is a real App instance running with your full identity and real access to Domino APIs and shared state. If your Draft App starts Jobs, writes to Datasets, or modifies any other Domino resource, it does so for real. Test destructive operations with care.
</Warning>

### Live sync

Preview always runs the latest committed code. When Domino detects new commits in the workspace, live sync applies them to the running preview by restarting only the container. The pod is reused, so iteration is much faster than the first load. A preference controls whether sync runs automatically on every new commit or only when you trigger it.

Live sync applies committed code only, so commit your changes before you expect to see them in the preview. Autoscaling isn't available in preview.

## Draft lifecycle

A Draft App persists across workspace sessions, so you can stop the workspace and resume where you left off. Stopping the workspace stops the preview instance, and deleting the workspace deletes the Draft App. Administrators can also stop preview instances.

## Publish from a Workspace

When the code is ready, select **Publish App** in the workspace. Domino fills the fields from your most recent preview configuration, so you can accept them as-is or change them first. Choose one of two targets:

| Target | What it does |
| :- | :- |
| **Publish as a new App** | Creates an App whose first version captures the configuration of your Draft App. You can change the App file here. |
| **Publish as a new version of an existing App** | Adds a version to an existing App. The target App must be in the same project and have the same App file (entry point) as your Draft App. |

Publishing creates a version but doesn't deploy it. To make the version reachable, continue with [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions).

Publishing is non-destructive. Your Draft App and workspace stay intact and you can keep iterating, so one workspace can publish several Apps and several versions over its life.

## Iterate on a published App

To improve an App that is already serving traffic without disrupting it, open the App's version details page and select **Edit in Workspace**. Domino creates a workspace and Draft App seeded from that version's configuration, then starts a preview. Iterate as above, then select **Publish App** and choose **Update existing** to publish a new version of the same App.

<Note>
  **Edit in Workspace** isn't available for Apps in projects that use the Domino File System (DFS). Use a Git-based Project for this workflow.
</Note>

## Related

* [Create an App](/cloud/platform-capabilities/features/apps/create-an-app): Start an App from a workspace or from an existing file.
* [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions): Turn a Draft App into an immutable version, then start it.
* [App versioning and reproducibility](/cloud/platform-capabilities/features/apps/app-versioning-and-reproducibility): What a version captures and how to start a previous one.
* [Apps in Domino](/cloud/platform-capabilities/features/apps/apps-in-domino): How Apps, Draft Apps, versions, and instances relate.


# Govern an App
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/govern-an-app

Govern App versions with policies, evidence, approvals, bundle snapshots, deployment gates, version comparisons, and required policy enforcement.

Governance for Apps subjects an App to your organization's review and approval workflows. Attach policies to an App to collect evidence, obtain approvals, and gate deployment. You can attach policies at any point in an App's lifecycle, not only at creation.

## How Apps and bundles work together

Each governed App version maps 1:1 to a bundle. Only the latest App version holds a read-write bundle; bundles attached to earlier versions become evidence read-only. Approvals, findings, and attachments on those earlier bundles remain editable.

When you create a new version of a governed App, Domino:

* Snapshots the previous version's bundle by copying its evidence, attachments, and approvals into a new read-only bundle attached to the previous version.

* Attaches the read-write bundle to the new version, so governance carries forward without re-collecting evidence.

Findings and comments are not copied to the snapshot, to avoid duplication.

## Attach a policy to an App

1. Open the App from **Deployments** > **Apps & Agents**.

2. Click **Add to Bundle** to link the App to a new or existing bundle.

3. Confirm the link in the bundle's **Attachments** section.

Once a policy is attached, the App is governed. Its next version reuses the bundle automatically.

## Collect evidence and obtain approvals

Evidence for Apps typically covers security and data privacy:

* **Scripted checks**: Run automated tests to verify dependencies, check for vulnerabilities, or validate data access permissions.

* **Manual evidence**: Provide screenshots, UAT sign-offs, or data source documentation.

Follow the same collection and approval flow used for other governed assets. [Use the Evidence notebook](/cloud/platform-capabilities/features/governance/work-with-bundles/use-evidence-notebook) has the details.

<Tip>
  You can use gates to block the **Deploy** action when an App's bundle is non-compliant. See [Gates](/cloud/platform-capabilities/features/governance/gating-domino-governance).
</Tip>

## Compare evidence across App versions

Open a governed App and go to **Governance** > **Changes** to see how evidence has changed between versions. The **Changes** tab compares evidence for the same policy across each version's bundle and shows what was added, removed, or modified.

* If a policy was upgraded between versions, the tab still pairs the two bundles on the shared policy and marks any modified questions as added, removed, or changed.

* If a policy is present in one version and missing in the other, every question in that policy appears as added or removed, depending on the direction of the comparison.

## Required policies for Apps

Governance admins can define a set of policies that Domino attaches automatically to every App bundle, deployment-wide. Enforced policies always attach at their latest version, and practitioners cannot remove them from a bundle. Governance, System, and Cloud admins can remove one if needed.

Enforcement runs when an App version is created:

* **New App**: Domino creates the bundle with every currently enforced policy attached.

* **New version of an ungoverned App**: If the App has no bundle yet, Domino creates one with every enforced policy attached.

* **New version of a governed App**: Domino attaches any enforced policies that are missing from the bundle and upgrades already-attached enforced policies to their latest version.

If an admin removed an enforced policy from an earlier App version's bundle, the policy attaches again on the next App version as long as it is still in the enforced set.

See [Enforce required policies](/cloud/platform-capabilities/features/governance/define-governance-policies#enforce-required-policies) for how admins configure the enforced set.

## Next steps

* [Work with bundles](/cloud/platform-capabilities/features/governance/work-with-bundles): package assets for review

* [Define policies](/cloud/platform-capabilities/features/governance/define-governance-policies): build the policies that govern Apps

* [Gates](/cloud/platform-capabilities/features/governance/gating-domino-governance): block deployment until approvals are in place


# Build and deploy Apps
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/index

Turn Project code into an interactive web application: create it, iterate on it in a workspace, publish immutable versions, and deploy one to viewers.

**[Apps in Domino](/cloud/platform-capabilities/features/apps/apps-in-domino)** are built for flexibility, letting you run custom web-based tools and expose them to others while Domino manages infrastructure, permissions, and routing.

<Frame>
  ```mermaid placement="top-right" theme={null}
  flowchart TD
    C["Create"] --> D["Develop and preview<br/>Iterate with live sync"]
    D --> P["Publish version"]
    P --> R["Deploy<br/>Choose any published version"]
  ```
</Frame>

## Build and release

<Columns>
  <Card title="Create an App" href="/cloud/platform-capabilities/features/apps/create-an-app">
    Start an App in a workspace or from an existing file in your project.
  </Card>

  <Card title="Develop and test an App" href="/cloud/platform-capabilities/features/apps/develop-and-test-an-app">
    Iterate on App code in a workspace and preview it in a real App runtime, without publishing.
  </Card>

  <Card title="Publish and deploy App versions" href="/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions">
    Create an immutable version, then start it to make the App reachable.
  </Card>

  <Card title="App versioning and reproducibility" href="/cloud/platform-capabilities/features/apps/app-versioning-and-reproducibility">
    What a version pins, and how to restore pinned inputs or roll back.
  </Card>
</Columns>

## Operate and govern

<Columns>
  <Card title="Share an App" href="/cloud/platform-capabilities/features/apps/share-an-app">
    Control who can discover and open a deployed App, and handle access requests.
  </Card>

  <Card title="Govern an App" href="/cloud/platform-capabilities/features/apps/govern-an-app">
    Attach policies, collect evidence and approvals, and gate deployment.
  </Card>

  <Card title="App logs and monitoring" href="/cloud/platform-capabilities/features/apps/app-logs-and-versioning">
    Read per-instance logs and resource consumption, and troubleshoot with support bundles.
  </Card>

  <Card title="Authentication and identity for Apps" href="/cloud/platform-capabilities/features/apps/authentication-and-identity">
    How Apps identify viewers, and what identity and permissions they run with.
  </Card>
</Columns>

## Build reliable Apps

<Columns>
  <Card title="Common App frameworks" href="/cloud/platform-capabilities/features/apps/common-app-frameworks">
    Minimal working examples for Streamlit, Dash, Flask, and Shiny.
  </Card>

  <Card title="Best practices for Apps" href="/cloud/platform-capabilities/features/apps/best-practices-for-apps">
    Techniques that keep Apps clear, maintainable, and scalable as usage grows.
  </Card>

  <Card title="Persist data from Apps" href="/cloud/platform-capabilities/features/apps/persist-data-in-apps">
    Save results to Datasets or external storage, because App containers don't persist file changes.
  </Card>
</Columns>


# Persist data from Apps
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/persist-data-in-apps

Persist App output in Domino Datasets, databases, object storage, or APIs, and protect shared resources from concurrent writes and race conditions.

Apps in Domino do not automatically save file changes back to your project’s repository. This differs from **Workspaces** or **Jobs**, which can commit outputs at the end of execution.

If your App needs to persist results, such as saving user input, logging events, or generating output files, you’ll need to handle persistence directly within your App code.

## Ways to persist data from an App

You can persist results using any of the following approaches:

* **Mounted Datasets**: Datasets provide read/write storage and are versioned independently from your project files. Use them to store:

  * Generated files

  * Log outputs

  * Snapshots or other intermediate results

* **Databases or External Data Sources**: Use standard client libraries to write to:

  * SQL or NoSQL databases

  * Cloud object storage (for example, S3, Azure Blob)

  * Enterprise data warehouses (ensure your runtime environment has credentials and network access)

* **APIs**: Your App can invoke internal or external APIs to:

  * Record state

  * Send metrics or events

  * Trigger workflows

  * Store data in external services

## Concurrency considerations

Apps can serve **multiple users at the same time**, which means your app code must be designed to handle **concurrent access safely**.

<Warning>
  Domino does **not** serialize or isolate access to shared resources across App users. You are responsible for implementing any necessary safeguards in your App code.
</Warning>

Here are important safeguards:

* Use locking mechanisms or atomic operations if multiple users write to the same resource.

* Prefer per-user namespaces, directories, files, or database tables to isolate individual user actions.

* Watch for race conditions or state inconsistencies, especially in long-running or interactive Apps.

## Next steps

* [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions): Create a version and start it for viewers.

* Learn more about [how Apps in Domino run](/cloud/platform-capabilities/features/apps/authentication-and-identity) and what identity and permissions are used.


# Publish and deploy App versions
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions

Publishing creates an immutable App version; deploying starts one. How to do each, and which settings you can change without creating a version.

Publishing and deploying are independent actions. **Publish** creates a new immutable version and doesn't start it. **Deploy** starts a published version, making it reachable by authorized viewers.

You can therefore publish several versions before deploying any of them, and redeploy an older version at any time.

## Publish a version

Publish from a workspace that has an active Draft App by selecting **Publish App** (see [Develop and test an App](/cloud/platform-capabilities/features/apps/develop-and-test-an-app)), or directly from the App list page without a Draft App.

A version captures the App identity and version-defining content listed below. When you publish directly without a Draft, provide an entry point, code commit, Compute Environment and revision, and Hardware Tier. Other fields, such as Data Plane, volume and Dataset configuration, and identity propagation, can use their defaults. The result is a new version in the App's history, with the Git reference resolved and the commit pinned at publish time.

<Note>
  When you publish into an existing App, the entry point and the code configuration type must match the target App's. Domino rejects a mismatch. The entry point is fixed for the life of an App, so create a new App to change it.
</Note>

## Deploy a version

Select a version on the App details page and select **Deploy**. A dialog lets you review and update deployment settings before the version starts. An App has at most one running instance at a time.

In that dialog you can set:

* **Custom URL path** for easier sharing (for example, `/apps/demo`).
* **Iframe rendering** to control whether the App renders inside the Domino iframe wrapper or is served directly at its URL. This setting applies only when deep linking is off. When deep linking is on, Domino serves the App directly at its URL and doesn't render it in an iframe. Turn iframe rendering off when your framework does not work inside an iframe. Changing this setting restarts the running instance but does not create a new version.
* **Enable deep linking** to let viewers link to a specific view inside the App by appending query parameters to its URL. Apps with deep linking on are never rendered in an iframe. Deep linking needs the `SecureIdentityPropagationToAppsEnabled` feature flag, so ask your administrator to check [Configuration settings for Apps](/cloud/platform-capabilities/features/apps/app-logs-and-versioning/app-configuration-settings) if the option is missing.

Any published version can be deployed, not just the newest. [App versioning and reproducibility](/cloud/platform-capabilities/features/apps/app-versioning-and-reproducibility) covers restoring pinned inputs and rolling back to a previous version.

<Note>
  If Governance is enabled and a policy applied to the App gates the **Deploy** action, the Deploy dialog lists any missing approvals and disables **Deploy App version** until they are granted. See [Govern an App](/cloud/platform-capabilities/features/apps/govern-an-app) for the full flow.
</Note>

## What creates a version, and what doesn't

Deployment settings govern how and where an App runs rather than what it runs, so editing them doesn't produce a version.

| Setting group | Fields | Version behavior |
| :- | :- | :- |
| **App identity** | Entry point; code configuration type | Captured by every version, but fixed for the life of the App. Create another App to change either field. |
| **Version-defining content** | Source code (the Git reference and resolved commits for the main repository, Domino File System, and imported Project repositories); Compute Environment and environment revision; Data Plane; external data volume mounts; NetApp Volume mounts; Dataset mount configuration; extended identity propagation | Publishing a Draft or editing one of these fields creates a version. The fields are immutable for that version, and an edit stops the running instance. |
| **Deployment settings** | Hardware Tier; autoscaling; custom (vanity) URL; iframe rendering, deep linking, and query parameters | Updates the existing version in place and restarts the running instance. No new version is created. |
| **Version metadata** | Version description (Version Notes); Tags | Updates the existing version in place without restarting the running instance. No new version is created. |

Starting and stopping an App never create a version.

## Manage Apps through the API

The `/api/apps/v1/` endpoints cover the same operations: create an App, create a Draft App for a workspace, publish, list and create versions, and start, stop, or live-sync a version. Publishing into an App whose entry point or configuration type doesn't match returns `422`.

The older `/api/apps/beta/` endpoints remain active and unchanged, so existing callers need no migration. App instance reads (list instances, logs, real-time logs, views, and instance delete) are available only on the beta endpoints.

### Filter and sort the App list

`GET /api/apps/v1/apps` and `GET /api/apps/beta/apps` accept the same query parameters, so you can script an App inventory instead of paging through the UI. Filter by `projectId`, `name`, `ownerId`, `status`, `publisherId`, or `tagIds`, and combine any of them in one request. Two filters answer type and access questions directly:

* `configurationType` keeps only Apps with one configuration type: `STANDARD` or `AISYSTEM`.
* `accessStatus` filters Apps by your own access: `granted` (you can open the App), `pending` (your access request awaits a decision), or `noAccess` (no access and no pending request).

Sort with `sortField` (`name`, `lastUpdated`, `status`, `views`, `lastViewed`, or `published`, the time the App's current instance was published) and `sortOrder` (`asc` or `desc`).

For example, list the Apps you can already open, most recently published first:

```python theme={null}
import os
import requests

response = requests.get(
    f"{os.environ['DOMINO_API_HOST']}/api/apps/v1/apps",
    headers={"Authorization": f"Bearer {os.environ['DOMINO_API_TOKEN']}"},
    params={
        "accessStatus": "granted",
        "sortField": "published",
        "sortOrder": "desc",
    },
)
response.raise_for_status()

for app in response.json().get("items", []):
    print(app["name"])
```

Both surfaces are documented in the [Domino Platform API reference](/cloud/reference/api/domino-open-api) under the Apps, AppVersions, and AppInstances tags.

## Related

* [Create an App](/cloud/platform-capabilities/features/apps/create-an-app): The two entry points for a new App.
* [Develop and test an App](/cloud/platform-capabilities/features/apps/develop-and-test-an-app): Iterate in a workspace before publishing.
* [Share an App](/cloud/platform-capabilities/features/apps/share-an-app): Control who can discover and open a deployed App.
* [App versioning and reproducibility](/cloud/platform-capabilities/features/apps/app-versioning-and-reproducibility): What a version pins and how to start an older one.


# Share an App
Source: https://docs.domino.ai/cloud/platform-capabilities/features/apps/share-an-app

Control who can discover and open a deployed App, handle access requests, and decide whose identity the App acts with.

Domino enforces App access at the proxy level, before any request reaches your App code. Set it when you publish, or at any time afterwards from the **Share App** dialog.

## Choose a sharing mode

| Mode | Who can view | Access & sharing view |
| :- | :- | :- |
| **Restricted** | Only the users you list, plus project collaborators. | Lets you name the users or organization that can reach the App. |
| **Anyone in Domino** | Anyone with a Domino account. | Shows no user management options. |

Project collaborators can also edit the App, so you add an editor by adding that person as a collaborator on the project that holds the App.

<Frame>
  <img alt="The Share App dialog, showing sharing mode and the list of users with access." />
</Frame>

## Handle access requests

The **Share App** dialog is where you review pending requests, grant or deny access, and track notifications.

* **Pending requests** appear with **Accept** and **Deny** options.
* **Access request notifications** go out by email and in Domino when someone requests access, naming the requester, the App, the project, and linking to the **Share App** dialog.
* **Access granted notifications** go out by email and in Domino when you grant access, naming the App and linking to its details page.

Domino sends a notification whenever you grant access, whether you accepted a request or assigned access directly.

## Decide whose identity the App uses

By default an App acts with the identity and privileges of its publisher, so every viewer shares the publisher's permissions when the App reaches Domino resources. Enable **Allow App to act for viewers in Domino** when the App has to query Datasets under row-level security, start Jobs for a viewer, use Data Sources that need viewer credentials, or take any other action as the viewer. The change takes effect after you restart the App.

Only SysAdmins or CloudAdmins can enable this setting.

<Warning>
  An App with this setting has full access to Domino on behalf of each viewer, including running Jobs, reaching files and data, and managing Projects. Enable it only for Apps you trust to act responsibly with user-level permissions. [App security and identity](/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security) covers the configuration steps and the security implications.
</Warning>

## Related

* [App security and identity](/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security): Identity propagation levels, consent, and what an App can do as a viewer.
* [Publish and deploy App versions](/cloud/platform-capabilities/features/apps/publish-and-deploy-app-versions): Create a version and start it before you share it.
* [Allow external resources in Apps](/cloud/platform-capabilities/features/apps/authentication-and-identity/whitelist-resources): Apps can be blocked from loading content unless the URLs are allowlisted.


# Manage MPI dependencies
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/manage-dependencies

Add Python packages and pinned versions to both MPI cluster and execution Compute Environments through their Docker Instructions.

<Note>
  The Open MPI feature is only available with Domino versions 5.1.1 and later. Previous versions of Domino do not support this feature.
</Note>

The Domino Open MPI base image contains common dependencies like Pandas and NumPy. You can modify your environments to add additional packages or a specific package version.

You can package and manage dependencies for your [MPI-enabled compute environments](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/configure-mpi-prerequisites). This creates flexibility in managing dependencies for individual projects or workloads without shared cluster worker complexities.

* To add a new cluster worker, add the appropriate statements in the **Docker Instructions** section of both the MPI cluster and Open MPI execution environments. For example, to add a particular version of `PyTorch` or the `mpi4py` package include the following:

```dockerfile theme={null}
### Optionally specify version if desired
RUN pip install PyTorch==1.9.0
RUN pip install mpi4py
```


# Work with your MPI cluster
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/work-with-your-cluster

Create MPI clusters for Workspaces and Jobs, configure workers and storage, submit distributed runs, tune jobs, and understand cluster security.

<Note>
  The Open MPI feature is only available with Domino versions 5.1.1 and later. Previous versions of Domino do not support this feature.
</Note>

You can use MPI clusters with Domino workspaces, jobs, and scheduled jobs.

<Note>
  MPI clusters are not available for apps, launchers, or Domino endpoints.
</Note>

## Create an MPI cluster with Workspaces

### Create an on-demand MPI cluster attached to a Domino Workspace

1. Go to **Workspaces** > **New Workspace**.

2. In the **Launch New Workspace** dialog click **Compute Cluster**. For your compute cluster, set the following cluster configuration attributes after selecting **MPI**:

   * **Number of Workers** - The number of MPI node workers making up the cluster when it starts. The combined capacity of the workers is available for your workloads.

   * **Worker Hardware Tier** - The amount of compute resources (CPU, GPU, and memory) made available to each MPI node worker.

   * **Cluster Compute Environment** - Specifies the compute environment for the MPI cluster.

   * **Dedicated local storage per executor** - The amount of dedicated storage in Gigabytes (2 ^ 30 bytes) available to each MPI worker. The storage is automatically mounted to `/tmp`. The storage is automatically provisioned when the cluster is created and de-provisioned when it is shut down.

     <Warning>
       The local storage per worker must not be used for storing any data that must be available after the cluster is shut down.
     </Warning>

3. To launch your workspace, click **Launch**.

After the workspace launches it can access the MPI cluster you configured.

<Note>
  The cluster worker nodes can take longer to spin up than the workspace. The status of the cluster is displayed in the navigation bar.
</Note>

### Connect to your cluster

When provisioning your on-demand MPI cluster, Domino initializes environment variables. These variables contain the information to connect to your cluster. A `hostfile` is created with the worker node name. That file is located at `/domino/mpi/hosts`. Submit your `mpirun` jobs from your preferred IDE. The jobs use these default settings:

* Hosts are set to all workers. This does not include the workspace/job container.

* Slots are allotted based on the CPU cores in the selected worker hardware tier. The minimum is one. If you are using GPU worker nodes this is set to the minimum CPU/GPU requests from the hardware tier.

* Defaults are set for mapping (socket) and binding (core).

<Note>
  You must perform an MPI file sync before you run commands on the worker nodes. See the [File sync MPI clusters](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi/file-sync-mpi-clusters) for more information.
</Note>

1. Run a Python script:

```shell theme={null}
mpirun python my-training.py
```

You can pass arguments to the `mpirun` command. See [Open MPI docs](https://www.open-mpi.org/doc/v4.0/man1/mpirun.1.php). The following are some frequently used parameters:

* `--x <env>` - Distribute an existing environment variable to all workers. You can specify only one environment variable per `-x` option.

* `--hostfile <your hostfile>` - Specify a new hostfile with your custom slots.

* `--map-by <core>` - Map to the specified object. The default is `socket`. Create a `rmaps_base_mapping_policy` environment variable to set a new mapping policy.

  1. Create a `hwloc_base_binding_policy` environment variable to set a new binding policy.

For example to add `-bind-to none`, to specify that a training process not be bound to a single core, and `-map-by slot`, use the following code:

```shell theme={null}
mpirun
--bind-to none
--map-by slot
python my-training.py
```

<Note>
  If you are using Horovod, Domino recommends running it directly using MPI. If not, enter a command specifying the number of worker nodes. For example, `-H server1:1,server2:1,server3:1` specifies three worker nodes. See the [Horovod docs](https://horovod.readthedocs.io/en/stable/mpi_include.html) for instructions on converting Horovod commands to MPI.
</Note>

## Create an MPI cluster with Jobs

### Create an on-demand MPI cluster attached to a Domino Job

1. Go to **Jobs** > **Run**.

2. In the **Start a Job** dialog, click **Compute Cluster**.

   For your compute cluster, set the following cluster configuration attributes:

   * **Number of Workers** - The number of MPI node workers for the cluster when it starts. The combined capacity of the workers is available for your workloads.

   * **Worker Hardware Tier** - The amount of compute resources (CPU, GPU, and memory) available to each MPI node worker.

   * **Cluster Compute Environment** - Specifies the compute environment for the MPI cluster.

   * **Dedicated local storage per executor** - The amount of dedicated storage in Gigabytes (2 ^ 30 bytes) available to each MPI worker. The storage is automatically mounted to `/tmp`. The storage is automatically provisioned when the cluster is created and de-provisioned when it is shut down.

     <Warning>
       Do not store any data in the local storage that must be available after the cluster is shut down.
     </Warning>

3. To launch your job, click **Launch**.

4. Enter the cluster settings and launch your job.

   The job start command is wrapped in a `mpirun` prefix. The defaults were described previously.

   <Note>
     The Domino job is executing the `mpirun` command. Other scripts cannot run `mpirun` while that job is executing.
   </Note>

   When writing scripts that are used in a job MPI cluster, you must save artifacts to an external location or to a Domino dataset, not to the project’s files page (`/mnt`). Each project contains a default [read/write dataset](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets), accessed using the `/domino/datasets/local/${DOMINO_PROJECT_NAME}` path.

   To tune the Open MPI run in jobs, create an `mpi.conf` file at your root directory with the parameters you want. The root of your project is the `/mnt` working directory. This is linked to the `Files` page, or the `/mnt/code` directory if you are using a Git-based project. See [Open MPI run-time tuning](https://www.open-mpi.org/faq/?category=tuning#setting-mca-params) for more information.

   Parameters are set one per line, for example:

   * `mca_base_env_list = EXISTING_ENV_VAR`, where the environment variable is set in the workspace.

   * `orte_hostfile = <new hostfile location`, copy the default hostfile from `domino/mpi/hosts` to `/mnt` and set `orte_hostfile=/mnt/hosts`.

   * `rmaps_base_mapping_policy core`, equivalent to `--map-by <core>`

   * `hwloc_base_binding_policy socket`, equivalent to `--bind-to socket`

## Cluster lifecycle

When the workspace or job starts, a Domino on-demand MPI cluster is automatically provisioned with your MPI cluster settings. The cluster is attached to the workspace or job when the cluster becomes available.

When the workspace or job shut down the on-demand MPI cluster and all associated resources are automatically stopped and de-provisioned. This includes any compute resources and storage allocated for the cluster.

## Cluster network security

The on-demand MPI clusters created by Domino are not meant for sharing between multiple users. Each cluster is associated with a given workspace or job instance. Access to the cluster is restricted to users who can access the workspace or the job attached to the cluster. This restriction is enforced at the networking level and the cluster is only reachable from the execution that provisioned the cluster.


# Run HPC workloads with Slurm
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/slurm/index

Run existing Slurm batch scripts on Domino, from a Workspace or the Jobs page, on worker nodes Domino autoscales.

Domino runs High-Performance Computing (HPC) workloads on [Slurm](https://slurm.schedmd.com/), the standard workload manager in HPC. Submit your existing Slurm job scripts from inside a Domino Workspace with the standard `sbatch`, `squeue`, and `scancel` commands, without rewriting the scripts or changing how you work. Job scripts driven by common R and Python frameworks such as `mirai` and `clustermq` run unchanged.

## How it works

Slurm is a cluster resource manager and job scheduler: it allocates nodes, CPUs, GPUs, and memory to jobs and runs arbitrary executables, with no knowledge of the computation inside.

Reach for a Slurm cluster when you have batch-oriented HPC workloads, such as simulations, statistical ensembles, or parallel parameter sweeps, that are already written as Slurm job scripts or that fit the batch scheduling model.

When you launch a Workspace with an attached Slurm cluster, the Workspace acts as your Slurm login node. Domino injects the configuration and credentials the Slurm CLI needs, and you submit jobs from the Workspace terminal exactly as you would on a traditional HPC cluster.

The **HPC** tab on the Project's Jobs page is the second path: it submits a script that lives in the Project, without the Slurm CLI, once an administrator enables it. See [Submit HPC Jobs from the Jobs page](#submit-hpc-jobs-from-the-jobs-page).

### Key terms

| Term | Definition |
| :- | :- |
| HPC Job | A batch job submitted to and scheduled by Slurm from inside Domino. |
| Job array | A set of related jobs submitted as one unit, indexed by an array task ID, such as a parameter sweep. |
| Partition | A Slurm scheduling queue: a logical grouping of worker nodes. |
| Worker | A compute node in the Slurm cluster. Domino autoscales workers up and down based on queued work. |

## Prerequisites

* Your administrator has deployed Domino with Slurm support enabled on at least one [Data Plane](/cloud/platform-capabilities/core-concepts/data-plane). See [Administer Slurm for HPC workloads](/cloud/admin/infrastructure-and-compute/administer-slurm).

* Two Compute Environments are available to you: one carrying the Slurm CLI, which runs the Workspace, and one that runs on the Slurm worker nodes. Your administrator creates both.

* A worker Hardware Tier sized for Slurm workloads is available to you. Your administrator defines one tier per common workload profile, such as CPU-bound, memory-bound, and GPU-bound work, and you pick the one that fits your jobs when you attach the cluster.

* A [Dataset](/cloud/platform-capabilities/core-concepts/data/datasets) is configured on the Slurm-enabled Data Plane. Slurm copies no files between the Workspace and the worker nodes, so a shared Dataset is the common file system for job scripts, inputs, and outputs.

## Attach a Slurm cluster to a Workspace

<Steps>
  <Step title="Select the Slurm Workspace Environment">
    In the [Workspace launcher](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace), on the **Environment** tab, select the Slurm-capable Workspace Environment your administrator created.
  </Step>

  <Step title="Select a Slurm-enabled Data Plane">
    On the **Data Plane & Hardware** tab, select a Hardware Tier associated with a Data Plane that has Slurm installed. Confirm that a Dataset is listed under **Datasets available**.
  </Step>

  <Step title="Attach the cluster">
    On the **Compute Cluster** tab, select **Slurm**, then set the **Worker Hardware Tier** for the Slurm worker nodes, the **Max Number of Workers** that Domino scales the partition up to based on queued jobs, and the **Cluster Compute Environment** designated for Slurm.
  </Step>

  <Step title="Launch the Workspace">
    The Slurm CLI is available in the Workspace terminal once the Workspace starts.
  </Step>
</Steps>

If the **Slurm** option is disabled, see [Troubleshooting](#troubleshooting).

## Verify the cluster

Open a terminal in your Workspace and run `sinfo` to list the available partitions:

```bash theme={null}
$ sinfo
PARTITION                      AVAIL  TIMELIMIT  NODES  STATE NODELIST
slurm-69f2150f95cca76755bcf00e    up   infinite      0    n/a
all*                              up   infinite      0    n/a
```

Zero nodes is normal for an idle cluster. Domino provisions workers on demand when jobs enter the queue.

## Slurm commands

All standard Slurm commands are available from the Workspace terminal:

| Command | Purpose | Example |
| :- | :- | :- |
| `sinfo` | Show partitions and node availability | `sinfo` |
| `sbatch` | Submit a batch job script | `sbatch job.sh` |
| `squeue` | Show queued and running jobs | `squeue -j <jobid>` |
| `scontrol` | Show full details of a queued or running job | `scontrol show job <jobid>` |
| `sacct` | Show accounting data for completed jobs | `sacct -j <jobid>` |
| `scancel` | Cancel a queued or running job | `scancel <jobid>` |

Give resource requests either as `sbatch` command-line options or as `#SBATCH` directives in the script header. Command-line options override header directives:

```bash theme={null}
sbatch --job-name=hello_job --time=00:01:00 --cpus-per-task=1 --mem=256M job.sh
```

<Tip>
  Prefer specific `sacct` queries, filtered by job ID, user, or time range, over broad ones such as `sacct -a`. Broad queries after high job volume drive memory growth in the Slurm REST API.
</Tip>

## Submit your first HPC Job

Slurm workers share files with your Workspace through the Dataset, so job scripts must write output to the Dataset path rather than to Workspace storage. The example below uses `/domino/datasets/local/quick-start`. Adjust it to match the Dataset configured in your Project and Data Plane.

<Steps>
  <Step title="Create the job script">
    In your Workspace, create `sample-job.sh` in the default working directory with the following contents. No permission changes are needed. The job runs three array tasks, each printing diagnostic information about the worker node it ran on. It redirects its own output to the Dataset, because Domino controls `--output` and `--error` and overwrites whatever you set them to.

    ```bash theme={null}
    #!/bin/bash
    #SBATCH --job-name=hello-cluster
    #SBATCH --array=1-3
    #SBATCH --nodes=1
    #SBATCH --ntasks=1

    OUTDIR="/domino/datasets/local/quick-start"
    OUTFILE="${OUTDIR}/output_task_${SLURM_ARRAY_TASK_ID}.txt"

    {
      echo "===== Task Info ====="
      echo "Job ID:          ${SLURM_JOB_ID}"
      echo "Array Job ID:    ${SLURM_ARRAY_JOB_ID}"
      echo "Array Task ID:   ${SLURM_ARRAY_TASK_ID}"
      echo "Running on node: $(hostname)"
      echo "Started at:      $(date -u '+%Y-%m-%dT%H:%M:%SZ')"

      echo ""
      echo "===== CPU Info ====="
      nproc
      lscpu | grep -E '^(CPU\(s\)|Model name|Architecture)'

      echo ""
      echo "===== Memory Info ====="
      free -h

      echo ""
      echo "===== Shared Storage Check ====="
      df -h "${OUTDIR}"

      echo ""
      echo "===== Simple Compute ====="
      python3 -c "
    import time, math
    start = time.time()
    result = sum(math.sqrt(i) for i in range(1, 10_000_000))
    elapsed = time.time() - start
    print(f'Sum of sqrts: {result:.4f}')
    print(f'Elapsed:      {elapsed:.3f}s')
    "

      echo ""
      echo "Finished at: $(date -u '+%Y-%m-%dT%H:%M:%SZ')"
    } > "${OUTFILE}" 2>&1

    echo "Task ${SLURM_ARRAY_TASK_ID} wrote output to ${OUTFILE}"

    exit 0
    ```
  </Step>

  <Step title="Submit the job">
    Submit the script with `sbatch`. Slurm returns the job ID it assigned:

    ```bash theme={null}
    $ sbatch sample-job.sh
    Submitted batch job 1
    ```
  </Step>

  <Step title="Check the job status">
    Check the queue immediately with `squeue`:

    ```bash theme={null}
    $ squeue
                 JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
               1_[1-3] slurm-69f hello-cl     1000 PD       0:00      1 (PartitionConfig)
    ```

    State `PD` means the job is pending while Domino provisions workers, which is normal on an idle cluster.
  </Step>

  <Step title="Watch the lifecycle">
    While the job is queued and running:

    * `sinfo` shows worker nodes provisioned as the partition scales up.
    * `squeue` shows the job move from pending (`PD`) to running (`R`).
    * `scontrol show job <jobid>` gives full details while the job is queued or running.
    * `sacct -j <jobid>` gives the accounting record once the job completes.

    <Tip>
      This job finishes in seconds, so it can complete before you see it running. To watch the lifecycle unfold, add `sleep 180` at the end of the script. You can also submit the job several times: jobs queue and are dispatched as workers become available.
    </Tip>
  </Step>

  <Step title="Read the output">
    When `squeue` returns empty, the job has finished. Check the output files in your shared Dataset directory:

    ```bash theme={null}
    ls /domino/datasets/local/quick-start/
    cat /domino/datasets/local/quick-start/output_task_1.txt
    ```
  </Step>

  <Step title="Stop the Workspace">
    The cluster's dedicated worker nodeset is tied to the Workspace, so stopping the Workspace releases the compute resources.
  </Step>
</Steps>

## Submit HPC Jobs from the Jobs page

The **HPC** tab on a Project's [Jobs](/cloud/platform-capabilities/core-concepts/jobs) page carries the history of the Project's HPC Jobs, with each job's status and metrics, no matter how the job was submitted.

A **Run** button on that tab submits an HPC Job without the Slurm CLI. The script has to live in the Project, and you give its path relative to the Project root rather than a Dataset path, which is the one place this differs from submitting through `sbatch`. Alongside the script you pick the Data Plane to run on and, optionally, the CPU, GPU, memory, node, task, and wall-time requests.

The [Domino API](/cloud/reference/api/domino-open-api) exposes the same operation as `POST /api/hpc/v1/jobs`, with companion endpoints to list, inspect, cancel, and fetch logs for HPC Jobs.

<Note>
  The **Run** button appears only where an administrator has enabled the `ShortLived.EnableHPCJobs` feature flag. Domino gates UI and API submission behind it while the submission experience is refined. Submitting from a Workspace terminal needs no flag.
</Note>

## Supported HPC Job types

| Job type | Example | Notes |
| :- | :- | :- |
| Serial shell scripts | `sbatch job.sh` | Resource requests through command-line options or the `#SBATCH` header. |
| SMP (single-node, multi-core) | `#SBATCH --cpus-per-task=4`, `#SBATCH --mem=1G` | `--cpus-per-task` sets cores per task, `--mem` sets memory per node. |
| Array jobs | `#SBATCH --array=0-9` | Use `$SLURM_ARRAY_JOB_ID` and `$SLURM_ARRAY_TASK_ID` in the script to give each task its own output path. |
| Dependent jobs | `sbatch --dependency=afterok:<jobid> job.sh` | Run a job only after another completes, with `afterok`, `afterany`, or `afternotok`. |

## Monitor and manage HPC Jobs

Track jobs from the Workspace terminal with `squeue`, `scontrol show job <jobid>`, and `sacct`, and cancel them with `scancel <jobid>`.

The Control Plane captures the runtime statistics of every HPC Job and stores them permanently, so the record outlives the job, the worker nodes that ran it, and the Workspace that submitted it.

Inside your job scripts, express resource requests in Slurm-native terms: CPUs, GPUs, memory, and wall time. Slurm schedules each job onto workers of the Worker Hardware Tier you selected for the cluster, and Domino autoscales those workers up and down based on the queue.

## Limitations

* A Slurm cluster is attached at Workspace launch, and the Workspace is the only place the Slurm CLI runs. A standard Domino Job cannot carry an attached Slurm cluster.

* Data moves between the Workspace and the workers only through [Datasets](/cloud/platform-capabilities/core-concepts/data/datasets), a shared network file system. Workers cannot reach [Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors) or [NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes).

* Interactive and scheduled Slurm jobs are not supported.

<Warning>
  Jobs from different users can run on the same worker node and are not sandboxed from one another. All Domino users share a single Unix group on the workers, so files written with group-readable or group-writable permissions are visible to other users. Set a restrictive `umask` on sensitive output files.
</Warning>

## Troubleshooting

<AccordionGroup>
  <Accordion title="The Slurm option is disabled in the Workspace launcher">
    Two conditions must both hold: at least one Environment labeled with the **Slurm** supported-cluster type is available to you, and the Data Plane selected on the **Data Plane & Hardware** tab has Slurm installed. Check your Environment selection and Data Plane, or ask your administrator.
  </Accordion>

  <Accordion title="Job output files are missing">
    Domino overwrites `--output` and `--error` on every HPC Job so that it can collect job output into the system of record, so both resolve to predefined locations and setting them in your script has no effect. To keep your own copy of a job's output, redirect it inside the script to the shared Dataset directory, as the sample job script does. Worker nodes cannot write to your Workspace file system, so any path outside the Dataset fails.
  </Accordion>

  <Accordion title="Jobs stay pending longer than expected">
    Domino autoscales workers on demand, so a cold cluster needs time to provision nodes. If jobs never dispatch, check that your resource request fits within the Worker Hardware Tier and the configured maximum number of workers, or ask your administrator about scheduling limits.
  </Accordion>
</AccordionGroup>

## Related

* [Administer Slurm for HPC workloads](/cloud/admin/infrastructure-and-compute/administer-slurm): enable Slurm, create the Environments, and size the worker Hardware Tiers.
* [On-demand distributed computing](/cloud/platform-capabilities/features/compute-clusters-hpc): the other cluster types Domino provisions.
* [Distributed GPUs with Open MPI](/cloud/platform-capabilities/features/compute-clusters-hpc/mpi): run MPI programs on a Domino-managed MPI cluster.
* [Datasets](/cloud/platform-capabilities/core-concepts/data/datasets): the shared storage Slurm jobs read and write.


# Use spot instances
Source: https://docs.domino.ai/cloud/platform-capabilities/features/compute-clusters-hpc/use-spot-instances

Choose Spot Hardware Tiers for interruptible workloads and understand how Domino retries or falls back to on-demand capacity when configured.

Spot instances reduce compute costs by using discounted, interruptible cloud capacity. They work well for workloads that can handle interruptions, such as batch jobs, distributed training runs, and parallel analytics tasks.

Domino supports spot instances across Jobs, Workspaces, Apps, Endpoints, and Launchers on AWS. When you select a hardware tier marked with the **Spot** tag, Domino attempts to use the cloud provider’s spot pricing model to provision your workload.

If your deployment is configured with [matching spot and on-demand node pools](/cloud/admin/infrastructure-and-compute/manage-compute-resources/configure-spot-instances), Domino handles spot capacity issues automatically:

* If spot capacity isn’t available when you submit, Domino runs the job on on-demand infrastructure.

* If a running spot workload is interrupted, Domino retries it on available spot capacity (which may use a different instance type) or falls back to on-demand infrastructure.

If automatic fallback isn’t configured, the job fails when spot capacity is unavailable.

## When to use spot instances

When submitting a workload in Domino, you may see one or more hardware tiers marked with the **Spot** tag.

<img alt="Select Spot instance" />

Use the following guidance to decide whether spot instances are a good fit for your workload:

**Ask yourself**:

* Can this job fail or restart without causing issues?

* Is this part of an experiment or test run?

* Does this workload serve live users or downstream systems?

* How long does this job or session run?

If your workload is **short**, **experimental**, and **not critical**, spot instances are a great choice. If your workload is **long**, **stateful**, or **user-facing**, choose an on-demand hardware tier instead.

## Next steps

* [Launch a Workspace](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace): Create a workspace so you can use your preferred tools in a reproducible and customizable environment.

* [Create and run Jobs](/cloud/platform-capabilities/core-concepts/jobs/create-and-run-jobs) to provide a structured approach for running code files. Jobs organize hardware, compute environment, data resources, and security protocols to ensure highly reproducible execution.


# Advanced Flows
Source: https://docs.domino.ai/cloud/platform-capabilities/features/flows/advanced-flows

Configure Domino Flows with task caching, reusable subflows, and human approval steps to reduce recomputation and control execution.

There are advanced capabilities that you can use in Domino Flows.

It is recommended that you familiarize yourself with some of the core functionality before using the following advanced capabilities.

<Note>
  Some Flyte features are not yet available in Flows, such as [dynamic workflows](https://www.union.ai/docs/v1/flyte/user-guide/core-concepts/workflows/dynamic-workflows/) and [map tasks](https://www.union.ai/docs/v1/flyte/user-guide/core-concepts/tasks/map-tasks/).
</Note>

## Caching

Caching allows you to avoid redundant computations by reusing cached results from previous executions. You can enable caching at the task level by configuring parameters in the task definition.

Whether a task execution uses the cache is determined by the state of the cache and the task execution’s task metadata, cache version, and cache key.

For a task execution to use the cache instead of re-computing the result, caching must be turned on for the task (more on that later) and there must be a result in the cache for the exact same:

1. Domino Project.

2. Task name. This is the `name` kwarg to the `DominoJobTask()`.

3. Task cache version. The cache version is formed by combining:

   * The "Flyte task cache version".

     * If the `cache` kwarg to the `DominoJobTask()` is a boolean, then the Flyte task cache version is the `cache_version` kwarg to the `DominoJobTask()`.

     * If instead the `cache` kwarg is a `Cache` object, then the Flyte task cache version is the result of calling the `Cache` object’s `get_version()` method.

   * The "domino job config task cache version", which comes from arguments to the task’s `DominoJobConfig()`. All args affect the version except:

     * `Title`

     * `VolumeSizeGiB`

     * `ComputeClusterProperties.WorkerHardwareTierId`

     * `HardwareTierId`

     * `ComputeClusterProperties.WorkerCount`

     * `ComputeClusterProperties.WorkerStorageGiB`

     * `ComputeClusterProperties.MaxWorkerCount`

     * `ComputeClusterProperties.MasterHardwareTierId`

   * The resolved commits for all `git` inputs to the task. See [`DominoJobconfig()`](/cloud/platform-capabilities/features/flows/define-flows#class-constructors) for details about floating `git` reference resolution on first run and relaunch.

   * The task’s input and output names and types, which come from the `inputs` and `outputs` kwargs to the `DominoJobTask()`.

4. Task cache key. The cache key is formed from the task execution’s runtime input values.

If any of the above parameters change between task executions such that the new combination does not have a result in the cache, then the subsequent execution does not use the cache. Conversely, if caching is enabled for a task and there is a result in the cache with the exact same above parameters, then that result is reused instead of executing the task again. Note that "workflow" is not mentioned above. So, two separate tasks in two different (or even the same) workflows can re-use the same result.

By default, caching is disabled for tasks. There are two ways to enable caching for a task.

The first way is to set the `cache` parameter to the boolean value `True` and specify a `cache_version` to use:

```python theme={null}
@workflow
def cache_bool_workflow(a: int, b: int) -> float:

    # Create first task w/ caching enabled via the boolean True
    add_task = DominoJobTask(
        name='Add numbers',
        domino_job_config=DominoJobConfig(Command="python add.py"),
        inputs={'first_value': int, 'second_value': int},
        outputs={'sum': int},
        cache=True, # This boolean enables caching
        cache_version="1.0", # Bumping the version is akin to invalidating the cache
        use_latest=True
    )
    sum = add_task(first_value=a, second_value=b)

    # Create second task w/ caching disabled
    sqrt_task = DominoJobTask(
        name='Square root',
        domino_job_config=DominoJobConfig(Command="python sqrt.py"),
        inputs={'value': int},
        outputs={'sqrt': float},
        use_latest=True
    )
    sqrt = sqrt_task(value=sum)

    return sqrt
```

When running the example above with arbitrary workflow inputs `a=X` and `b=Y`, both tasks execute initially. Then, subsequent reruns of the flow with `a=X` and `b=Y` use the cache for the first task instead of re-computing the result. This reduces overall compute cost and flow run time.

In the above example, the `cache_version` parameter forms part of the version key for a task’s cached results. Changing the `cache_version` value is a method to force a rerun of the task, even if no other task parameters have changed.

With Flyte 1.15 onward, there is a second way to turn on caching for a task: by setting the `cache` parameter to a `Cache` object:

```python theme={null}
from flytekit.core.cache import Cache, CachePolicy, VersionParameters

# create a class that implements the CachePolicy interface
class SimpleCachePolicy(CachePolicy):
    def get_version(self, salt: str, params: VersionParameters) -> str:
        return "2.0"

@workflow
def cache_obj_workflow(a: int, b: int) -> float:

    # create a Cache object that uses your CachePolicy
    cache_obj = Cache(policies=SimpleCachePolicy())

    # Create first task w/ caching enabled via the Cache object
    add_task = DominoJobTask(
        name='Add numbers',
        domino_job_config=DominoJobConfig(Command="python add.py"),
        inputs={'first_value': int, 'second_value': int},
        outputs={'sum': int},
        cache=cache_obj, # This kwarg enables caching
        # the cache_version kwarg is not used when the cache kwarg is a Cache object
        use_latest=True
    )
    sum = add_task(first_value=a, second_value=b)

    # Create second task w/ caching disabled
    sqrt_task = DominoJobTask(
        name='Square root',
        domino_job_config=DominoJobConfig(Command="python sqrt.py"),
        inputs={'value': int},
        outputs={'sqrt': float},
        use_latest=True
    )
    sqrt = sqrt_task(value=sum)

    return sqrt
```

Under the hood, the `Cache` object’s `get_version()` method is called to get the "Flyte task cache version" discussed above. In the above example, calling `get_version()` on the `Cache` object calls the cache policy’s `get_version()` method, which returns "2.0". So, in this simple example, bumping the string returned by `SimpleCachePolicy.get_version()` is akin to bumping `cache_version` when the `cache` kwarg is a boolean instead of a `Cache` object.

You can choose to have the system ignore some parameters when calculating the cache version and cache key.

* To ignore specific task inputs when the `cache` kwarg is a boolean, use the `cache_ignore_input_vars` kwarg to `DominoJobTask()`.

  * `cache_ignore_input_vars` takes a tuple of strings corresponding to variable names. Any task input whose name appears in the list is excluded from the cache key calculation.

  * In the first code example above, providing `cache_ignore_input_vars=("second_value",)` means that a task execution could use a cached result even if the runtime value of `second_value` is different from the runtime value of the execution that originally wrote the result to the cache.

* To ignore specific task inputs when the `cache` kwarg is a `Cache` object, use the `ignored_inputs` kwarg to `Cache()`.

  * `ignored_inputs` functions the same way as `cache_ignore_input_vars`.

* To ignore specific args to the `DominoJobConfig()`, use the `cache_ignore_vars` kwarg to `DominoJobConfig()`.

  * `cache_ignore_vars` takes a tuple of strings corresponding to `DominoJobConfig()` args (`"CommitId"`, `"EnvironmentId"`, etc.). Args in the tuple are excluded from the cache version calculation.

  * So, providing `cache_ignore_vars=("EnvironmentId",)` means that a task execution could use a cached result even if its `DominoJobConfig.EnvironmentId` is different from the execution that originally wrote the result to the cache.

You can see which flow tasks have or have not been recomputed by the caching symbol that appears in the flow graph view.

<img alt="Cache" />

<Note>
  Caching is different than recovering from failures. While recovery retrieves the actual outputs that were written to the blob storage, caching uses the task definition to compose a unique key that is used to retrieve results from an actual cache. This means that cached results can be used across multiple flows if the task definitions are the same.
</Note>

## Subflows

Subflows allow you to create nested flows by packaging a whole flow into a node. Subflows are useful when you want to abstract away complex flows into modular components that can be easily inserted into a broader flow. This also enables collaboration via assigning responsibilities at a subflow level.

To use subflows:

1. Start by defining your subflow like you would define any other flow. As an example, we use the flow created in the [getting started](/cloud/platform-capabilities/features/flows/get-started-with-flows) guide as our subflow.

   ```python theme={null}
   @workflow
   def math_subflow(a: int, b: int) -> float:

       # Create first task
       add_task = DominoJobTask(
           name='Add numbers',
           domino_job_config=DominoJobConfig(Command="python add.py"),
           inputs={'first_value': int, 'second_value': int},
           outputs={'sum': int},
           use_latest=True
       )
       sum = add_task(first_value=a, second_value=b)

       # Create second task
       sqrt_task = DominoJobTask(
           name='Square root',
           domino_job_config=DominoJobConfig(Command="python sqrt.py"),
           inputs={'value': int},
           outputs={'sqrt': float},
           use_latest=True
       )
       sqrt = sqrt_task(value=sum)

       return sqrt
   ```

2. In the same file, create your main flow that contains the subflow as a node.

   ```python theme={null}
   @workflow
   def math_subflow(a: int, b: int) -> float:

       # Create first task
       add_task = DominoJobTask(
           name='Add numbers',
           domino_job_config=DominoJobConfig(Command="python add.py"),
           inputs={'first_value': int, 'second_value': int},
           outputs={'sum': int},
           use_latest=True
       )
       sum = add_task(first_value=a, second_value=b)

       # Create second task
       sqrt_task = DominoJobTask(
           name='Square root',
           domino_job_config=DominoJobConfig(Command="python sqrt.py"),
           inputs={'value': int},
           outputs={'sqrt': float},
           use_latest=True
       )
       sqrt = sqrt_task(value=sum)

       return sqrt

   @workflow
   def simple_math_flow(a: int, b: int):

       # Call subflow
       sqrt_of_sum = math_subflow(a=a, b=b)

       # Create first task
       random_task = DominoJobTask(
           name='Do something else',
           domino_job_config=DominoJobConfig(Command="sleep 10"),
           inputs={'subflow_output': float},
           use_latest=True
       )
       random_result = random_task(subflow_output=sqrt_of_sum)

       return
   ```

3. Trigger an execution of the main flow in the same way you would trigger any other flow.

   ```bash theme={null}
   pyflyte run --remote workflow.py simple_math_workflow --a 4 --b 5
   ```

4. Monitor the results. Notice how the subflow is shown as its own node in the broader flow.

   <img alt="Domino Subflow Graph" />

## Approval flow

Flows allow you to add tasks in your flow where it pauses and waits for explicit human approval before proceeding. A use case where this can be useful is when model training has finished and you want someone to review the results before proceeding to downstream tasks that prepare it for deployment.

To use approvals in a flow:

1. Use the `approve` class provided by the Flyte SDK to add a required approval on the output of a task. As an example, we use the flow created in the [getting started](/cloud/platform-capabilities/features/flows/get-started-with-flows) guide.

   ```python theme={null}
   from flytekit import approve
   from datetime import timedelta

   @workflow
   def approval_flow(a: int, b: int) -> float:

       # Create first task
       add_task = DominoJobTask(
           name='Add numbers',
           domino_job_config=DominoJobConfig(Command="python add.py"),
           inputs={'first_value': int, 'second_value': int},
           outputs={'sum': int},
           use_latest=True
       )

       # Approval is added here
       sum = approve(add_task(first_value=a, second_value=b), "Approval", timeout=timedelta(hours=2))

       # Create second task
       sqrt_task = DominoJobTask(
           name='Square root',
           domino_job_config=DominoJobConfig(Command="python sqrt.py"),
           inputs={'value': int},
           outputs={'sqrt': float},
           use_latest=True
       )
       sqrt = sqrt_task(value=sum)

       return sqrt
   ```

2. Trigger an execution of the main flow in the same way you would trigger any other flow.

   ```bash theme={null}
   pyflyte run --remote workflow.py simple_math_workflow --a 4 --b 5
   ```

3. Monitor the results. Notice how the flow execution pauses after the first task and waits for approval. Click the **Resume** button.

   <img alt="Approval Flow Paused" />

4. Review the result and approve it to continue execution, or reject it to stop the execution.

   <img alt="Resume Modal" />


# Define Flow Artifacts
Source: https://docs.domino.ai/cloud/platform-capabilities/features/flows/define-flow-artifacts

Define DATA, MODEL, and REPORT Flow Artifacts in Python, then export selected outputs programmatically to Domino Datasets or NetApp Volumes.

A complex flow may execute many steps, producing hundreds of outputs, the majority of which are typically intermediate computations and not final results. It therefore becomes useful to elevate outputs of flow executions to easily discover, reuse, and inspect them, including the lineage of how they were produced. These task or workflow outputs are called Flow Artifacts.

<img alt="Inspect artifacts produced by a run" />

Flow Artifacts are explicitly defined in code by flow definition authors.

The `Artifact` method is used to generate a named artifact. The named artifact can have individual files added to it *anywhere* in the workflow definition. Typically an `Artifact` is declared at the beginning of a workflow like this:

```python theme={null}
PickleArtifact = Artifact(name="Pytorch Model", type=MODEL)
```

In any part of a workflow where an output might be used, the `.File(name="file.ext")` method can be called to declare that the output is an `Artifact`. For example:

```python theme={null}
@workflow
def single_model() -> PickleArtifact.File(name="model.pkl"):
```

There are 3 types of Flow Artifacts:

* `DATA` artifact: Files that may be added later as a new dataset snapshot.

* `MODEL` artifact: Files that may later be registered as a model.

* `REPORT` artifact: Files that may be part of a collection that make up a report.

In the following example, `DataArtifact.File(name="data.csv")` adds a `FlyteFile` output from the `data_prep_job` to the Flow Artifact declared as `Artifact(name="My Data", type=DATA)`. Unlike the use of `FlyteFile[TypeVar("csv")]` from previous examples, Flow Artifact files automatically infer their type from the given file extension.

```python theme={null}
from flytekit import workflow
from flytekit.types.file import FlyteFile
from flytekitplugins.domino.task import DominoJobConfig, DominoJobTask
from flytekitplugins.domino.artifact import Artifact, DATA, MODEL, REPORT

# define the artifact name and type which may be REPORT, DATA or MODEL
DataArtifact = Artifact(name="My Data", type=DATA)
ModelArtifact = Artifact(name="My Model", type=MODEL)

@workflow
def training_workflow(data_path: str) -> ModelArtifact.File(name="model.pt"):

    data_prep_job_config = DominoJobConfig(Command="python prep-data.py")
    data_prep_job = DominoJobTask(
        name='Prep data',
        domino_job_config=data_prep_job_config,
        inputs={'data_path': str},
        outputs={'processed_data': DataArtifact.File(name="data.csv")},
        use_latest=True
    )
    data_prep_results = data_prep_job(data_path=data_path)

    training_job_config = DominoJobConfig(Command="python train-model.py")
    training_job = DominoJobTask(
        name='Train model',
        domino_job_config=training_job_config,
        inputs={'processed_data': FlyteFile[TypeVar("csv")]},
        outputs={'model': FlyteFile[TypeVar("pt")]},
        use_latest=True
    )
    training_results = training_job(processed_data=data_prep_results["processed_data"])

    return training_results["model"] # Final output is returned here
```

There are no limits to the number of Flow Artifacts that can be defined, the number of files that can be added to each Flow Artifact, or where the Flow Artifact files are created. However, an individual Flow Artifact cannot include the same filename more than once. The following is not permitted because `DataArtifact.File(name="data.csv")` is used more than once:

```python theme={null}
DataArtifact = Artifact(name="My Data", type=DATA)

@workflow
def training_workflow(data_path: str) -> DataArtifact.File(name="data.csv"):

    data_prep_job_config = DominoJobConfig(Command="python prep-data.py")
    data_prep_job = DominoJobTask(
        outputs={
            # error - DataArtifact already has another data.csv
            'raw_data': DataArtifact.File(name="data.csv"),
            'processed_data': FlyteFile[TypeVar("csv")],
        }
    )
    data_prep_results = data_prep_job(data_path=data_path)

    return data_prep_results["processed_data"] # Final output is returned here
```

## Programmatically exporting to Domino Datasets and NetApp Volumes

Flow Artifacts can be programmatically exported to Domino Datasets and NetApp Volumes. This is useful for enabling continuous, automated movement of new output data into Datasets and NetApp Volumes.

<Note>
  When you include `run_launch_export_artifacts_task` inside a workflow, Domino generates a **separate export execution** that runs independently of the main workflow run. In the Graph view, the export launch node may appear near the start of the graph (for example, at `n0`) and can execute before downstream tasks have completed. This is expected behavior. The launch task only triggers the export process, and the actual export execution waits for the main workflow to finish and for all referenced Flow Artifacts to be fully generated.

  This design means that the visual placement of the launch node in the Graph view does not reflect the true ordering of when Flow Artifacts are exported. To verify export progress, open the separate export execution linked in the launch task’s logs. That execution contains detailed logs for each exported Flow Artifact.
</Note>

In the following example, `DataArtifact.File(name="data.csv")` is programmatically exported to the Domino Dataset with ID `681d3030ae3a706ef9c7f08b`, and the NetApp Volume with ID `5d1204c2-24c8-47df-a47f-69185efa602b` using the helper method `run_launch_export_artifacts_task`.

```python theme={null}
from flytekit import workflow
from flytekit.types.file import FlyteFile
from flytekitplugins.domino.task import DominoJobConfig, DominoJobTask
from flytekitplugins.domino.artifact import Artifact, DATA, ExportArtifactToDatasetsSpec, ExportArtifactToNetAppVolumesSpec, MODEL, REPORT, run_launch_export_artifacts_task

# define the artifact name and type which may be REPORT, DATA or MODEL
DataArtifact = Artifact(name="My Data", type=DATA)
ModelArtifact = Artifact(name="My Model", type=MODEL)

@workflow
def training_workflow(data_path: str) -> ModelArtifact.File(name="model.pt"):

    data_prep_job_config = DominoJobConfig(Command="python prep-data.py")
    data_prep_job = DominoJobTask(
        name='Prep data',
        domino_job_config=data_prep_job_config,
        inputs={'data_path': str},
        outputs={'processed_data': DataArtifact.File(name="data.csv")},
        use_latest=True
    )
    data_prep_results = data_prep_job(data_path=data_path)

    training_job_config = DominoJobConfig(Command="python train-model.py")
    training_job = DominoJobTask(
        name='Train model',
        domino_job_config=training_job_config,
        inputs={'processed_data': FlyteFile[TypeVar("csv")]},
        outputs={'model': FlyteFile[TypeVar("pt")]},
        use_latest=True
    )
    training_results = training_job(processed_data=data_prep_results["processed_data"])

    # Programmatic export is enabled here
    run_launch_export_artifacts_task(
        spec_list=[
            ExportArtifactToDatasetsSpec(
                artifact=DataArtifact,
                dataset_id="681d3030ae3a706ef9c7f08b",
            ),
            ExportArtifactToNetAppVolumesSpec(
                artifact=DataArtifact,
                netapp_volume_id="5d1204c2-24c8-47df-a47f-69185efa602b",
                target_relative_path="flows-data",
            )
            # ... More exports can be defined in this list, if needed.
        ],
        environment_name="Domino Standard Environment",
        hardware_tier_id="small-k8s",
        use_project_defaults_for_omitted=True,
    )

    return training_results["model"] # Final output is returned here
```

The helper method `run_launch_export_artifacts_task` performs the programmatic export.

It is required to specify this task to use the [Domino Standard Environment (DSE)](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) from 6.0 onwards, or a custom environment that is built on top of the DSE >= 6.0, as these contain the required Flyte Python and jq dependencies.

The full list of available parameters to the method `run_launch_export_artifacts_task` include:

<table>
  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type</th>
      <th>Description and Example</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td><p>spec\_list</p></td>
      <td><p>List\[Union\[ExportArtifactToDatasetsSpec, ExportArtifactToNetAppVolumesSpec]]</p></td>
      <td><p>The list of specifications used to determine the Flows artifacts to be exported to the Domino Datasets and NetApp Volumes. See the <code>spec\_list</code> example after the table.</p></td>
    </tr>

    <tr>
      <td><p>use\_project\_defaults\_for\_omitted</p></td>
      <td><p>bool</p></td>
      <td><p>If set to <code>True</code>, this uses the latest project defaults for parameters that were not explicitly provided. For better reproducibility, it is recommended to set this to <code>False</code> and explicitly define the necessary parameters.<br /> <em>Example:</em><br /> <code>False</code></p></td>
    </tr>

    <tr>
      <td><p>environment\_name</p></td>
      <td><p>Optional\[str]</p></td>
      <td><p>Name of the Environment to use in the Domino Job.<br /> <em>Example:</em><br /> <code>Domino Standard Environment</code></p></td>
    </tr>

    <tr>
      <td><p>environment\_id</p></td>
      <td><p>Optional\[str]</p></td>
      <td><p>ID of the Environment to use in the Domino Job. This is recommended over using <code>environment\_name</code> to prevent breaking reproducibility when environment names get changed.<br /> <em>Example:</em><br /> <code>6646530dcbd</code></p></td>
    </tr>

    <tr>
      <td><p>environment\_revision\_id</p></td>
      <td><p>Optional\[str]</p></td>
      <td><p>A specific revisionId of the environment to use.<br /> <em>Example:</em><br /> <code>6659daf5fc8</code></p></td>
    </tr>

    <tr>
      <td><p>hardware\_tier\_name</p></td>
      <td><p>Optional\[str]</p></td>
      <td><p>Name of the hardware tier to use in the Domino Job. It is recommended to use a minimal-resource hardware tier.<br /> <em>Example:</em><br /> <code>Large</code></p></td>
    </tr>

    <tr>
      <td><p>hardware\_tier\_id</p></td>
      <td><p>Optional\[str]</p></td>
      <td><p>ID of the hardware tier to use in the Domino Job. This is recommended over using <code>hardware\_tier\_name</code> to prevent breaking reproducibility when hardware tier names get changed. It is recommended to use a minimal-resource hardware tier.<br /> <em>Example:</em><br /> <code>small-k8s</code></p></td>
    </tr>

    <tr>
      <td><p>retries</p></td>
      <td><p>int</p></td>
      <td><p>Number of times to retry this task during a workflow execution.<br /> <em>Example:</em><br /> <code>0</code></p></td>
    </tr>

    <tr>
      <td><p>timeout</p></td>
      <td><p>Union\[timedelta, int] = timedelta(hours=3)</p></td>
      <td><p>The maximum amount of time for which one execution of this task should be executed. The execution is terminated if the runtime exceeds the given timeout. It is recommended to set this timeout duration to be greater than the workflow duration.<br /> <em>Example:</em><br /> <code>timedelta(hours=3)</code></p></td>
    </tr>
  </tbody>
</table>

`spec_list` example:

```python theme={null}
[
    ExportArtifactToDatasetsSpec(
        artifact=DataArtifact,
        dataset_id="681d3030ae3a706ef9c7f08b",
    ),
    ExportArtifactToNetAppVolumesSpec(
        artifact=DataArtifact,
        netapp_volume_id="5d1204c2-24c8-47df-a47f-69185efa602b",
        target_relative_path="flows-data",
        file_collision_setting=FileCollisionSetting.RENAME,
    ),
]
```

Each entry in `spec_list` is either an `ExportArtifactToDatasetsSpec` or an `ExportArtifactToNetAppVolumesSpec`.

`ExportArtifactToDatasetsSpec` parameters:

<table>
  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type</th>
      <th>Description and Example</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td><p>artifact</p></td>
      <td><p>Artifact</p></td>
      <td><p>The Flow Artifact to export.<br /> <em>Example:</em><br /> <code>DataArtifact</code></p></td>
    </tr>

    <tr>
      <td><p>dataset\_id</p></td>
      <td><p>str</p></td>
      <td><p>ID of the Domino Dataset to export the artifact to.<br /> <em>Example:</em><br /> <code>681d3030ae3a706ef9c7f08b</code></p></td>
    </tr>
  </tbody>
</table>

`ExportArtifactToNetAppVolumesSpec` parameters:

<table>
  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type</th>
      <th>Description and Example</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td><p>artifact</p></td>
      <td><p>Artifact</p></td>
      <td><p>The Flow Artifact to export.<br /> <em>Example:</em><br /> <code>DataArtifact</code></p></td>
    </tr>

    <tr>
      <td><p>netapp\_volume\_id</p></td>
      <td><p>str</p></td>
      <td><p>ID of the NetApp Volume to export the artifact to.<br /> <em>Example:</em><br /> <code>5d1204c2-24c8-47df-a47f-69185efa602b</code></p></td>
    </tr>

    <tr>
      <td><p>target\_relative\_path</p></td>
      <td><p>str</p></td>
      <td><p>Path on the NetApp Volume to export the artifact to. Defaults to an empty string, which writes to the volume root.<br /> <em>Example:</em><br /> <code>flows-data</code></p></td>
    </tr>

    <tr>
      <td><p>file\_collision\_setting</p></td>
      <td><p>FileCollisionSetting</p></td>
      <td><p>How to resolve a naming conflict when a file with the same name already exists at the target path. Must be one of <code>FileCollisionSetting.RENAME</code>, <code>FileCollisionSetting.IGNORE</code>, or <code>FileCollisionSetting.OVERWRITE</code>. Defaults to <code>FileCollisionSetting.RENAME</code>.<br /> <em>Example:</em><br /> <code>FileCollisionSetting.RENAME</code></p></td>
    </tr>
  </tbody>
</table>

## Additional examples

* **Data artifact with a single file:**

  *Example scenario:* A flow produces a single ADaM dataset with the file name `adae.sas7bdat` and the user wants to track the single file under its own data entity called `adae`.

  <Accordion title="Expand for example code.">
    ```python theme={null}
    from typing import TypeVar
    from flytekitplugins.domino.helpers import Output, run_domino_job_task
    from flytekitplugins.domino.artifact import Artifact, DATA
    from flytekit import workflow
    from flytekit.types.file import FlyteFile

    DataArtifact = Artifact("adae", DATA)

    @workflow
    def single_adam() -> DataArtifact.File(name="adae.sas7bdat"):
        return run_domino_job_task(
            flyte_task_name="Produce single ADaM Dataset",
            command="single_adam.py",
            output_specs=[
                Output(name="adae", type=FlyteFile[TypeVar("sas7bdat")]),
            ],
            use_project_defaults_for_omitted=True,
        )
    ```
  </Accordion>

* **Data artifact with multiple files:**

  *Example scenario:* A flow produces multiple ADaM datasets (`adae.sas7bdat`, `advs.sas7bdat`, and `adsl.sas7bdat`) and the user wants to track the collection of files under a single data entity called `adam`.

  <Accordion title="Expand for example code.">
    ```python theme={null}
    from typing import Tuple, TypeVar
    from flytekitplugins.domino.helpers import Output, run_domino_job_task
    from flytekitplugins.domino.artifact import Artifact, DATA
    from flytekit import workflow
    from flytekit.types.file import FlyteFile

    DataArtifact = Artifact("adam", DATA)

    @workflow
    def multiple_adam() -> Tuple[
        DataArtifact.File(name="adae.sas7bdat"),
        DataArtifact.File(name="advs.sas7bdat"),
        # if name does not include the extension, then provide the type kwarg
        DataArtifact.File(name="adsl dataset", type="sas7bdat"),
    ]:
        # files in an Artifact can be produced by different Flows tasks
        # in this example, one task produces two of the files, and another task produces the third
        adae_dataset, advs_dataset = run_domino_job_task(
            flyte_task_name="Produce adae and advs Datasets",
            command="produce_adae_and_advs.py",
            output_specs=[
                Output(name="adae", type=FlyteFile[TypeVar("sas7bdat")]),
                Output(name="advs", type=FlyteFile[TypeVar("sas7bdat")]),
            ],
            use_project_defaults_for_omitted=True,
        )
        adsl_dataset = run_domino_job_task(
            flyte_task_name="Produce adsl Dataset",
            command="produce_adsl.py",
            output_specs=[
                Output(name="adsl", type=FlyteFile[TypeVar("sas7bdat")]),
            ],
            use_project_defaults_for_omitted=True,
        )
        return adae_dataset, advs_dataset, adsl_dataset
    ```
  </Accordion>

* **Model artifact with a single file:**

  *Example scenario:* A flow produces a single model file with the name `model.pkl` and the user wants to track the single file as its own model entity.

  <Accordion title="Expand for example code.">
    ```python theme={null}
    from typing import TypeVar
    from flytekitplugins.domino.helpers import Output, run_domino_job_task
    from flytekitplugins.domino.artifact import Artifact, MODEL
    from flytekit import workflow
    from flytekit.types.file import FlyteFile

    ModelArtifact = Artifact("My Model", MODEL)

    @workflow
    def single_model() -> ModelArtifact.File(name="model.pkl"):
        return run_domino_job_task(
            flyte_task_name="Produce model",
            command="produce_model.py",
            output_specs=[
                # name of the Output can differ from the name of the ArtifactFile
                Output(name="my_model", type=FlyteFile[TypeVar("pkl")]),
            ],
            use_project_defaults_for_omitted=True,
        )
    ```
  </Accordion>

* **Model artifact with multiple files:**

  *Example scenario:* A flow produces multiple files relating to a model (`model.pkl`, `classes.txt`) and the user wants to track the collection of files as a single model entity.

  <Accordion title="Expand for example code.">
    ```python theme={null}
    from typing import Tuple, TypeVar
    from flytekitplugins.domino.helpers import Output, run_domino_job_task
    from flytekitplugins.domino.artifact import Artifact, MODEL
    from flytekit import workflow
    from flytekit.types.file import FlyteFile

    ModelArtifact = Artifact("My Model", MODEL)

    @workflow
    def multiple_model() -> Tuple[
        ModelArtifact.File(name="model.pkl"),
        # if name does not include the extension, then provide the type kwarg
        ModelArtifact.File(name="classes", type="txt"),
    ]:
        return run_domino_job_task(
            flyte_task_name="Produce model with classes",
            command="produce_model_with_classes.py",
            output_specs=[
                # name of the Output can differ from the name of the ArtifactFile
                Output(name="my_model", type=FlyteFile[TypeVar("pkl")]),
                Output(name="my_classes", type=FlyteFile[TypeVar("txt")]),
            ],
            use_project_defaults_for_omitted=True,
        )
    ```
  </Accordion>

* **Report artifact with a single file:**

  *Example scenario:* A flow produces a single TFL report with the file name `t_vscat.pdf` and the user wants to track the single file as its own report entity.

  <Accordion title="Expand for example code.">
    ```python theme={null}
    from typing import TypeVar
    from flytekitplugins.domino.helpers import Output, run_domino_job_task
    from flytekitplugins.domino.artifact import Artifact, REPORT
    from flytekit import workflow
    from flytekit.types.file import FlyteFile

    ReportArtifact = Artifact("TFL Report", REPORT)

    @workflow
    def single_report() -> ReportArtifact.File(name="t_vscat.pdf"):
        return run_domino_job_task(
            flyte_task_name="Produce PDF",
            command="produce_t_vscat.py",
            output_specs=[
                Output(name="t_vscat", type=FlyteFile[TypeVar("pdf")]),
            ],
            use_project_defaults_for_omitted=True,
        )
    ```
  </Accordion>

* **Report artifact with multiple files:**

  *Example scenario:* A flow produces multiple TFL reports (`t_vscat.pdf`, `t_ae_rel.pdf`) at different steps in the workflow and the user wants to track the collection of files as a single report entity.

  <Accordion title="Expand for example code.">
    ```python theme={null}
    from typing import Tuple, TypeVar
    from flytekitplugins.domino.helpers import Output, run_domino_job_task
    from flytekitplugins.domino.artifact import Artifact, REPORT
    from flytekit import workflow
    from flytekit.types.file import FlyteFile

    ReportArtifact = Artifact("TFL Reports", REPORT)

    @workflow
    def multiple_report():
        # files in an Artifact can be produced by different Flows tasks
        # in this example, one task produces one file, and another task produces the other
        vscat_pdf = run_domino_job_task(
            flyte_task_name="Produce vscat PDF",
            command="produce_t_vscat.py",
            output_specs=[
                Output(name="t_vscat", type=ReportArtifact.File(name="t_vscat.pdf")),
            ],
            use_project_defaults_for_omitted=True,
        )
        ae_rel_pdf = run_domino_job_task(
            flyte_task_name="Produce ae_rel PDF",
            command="produce_t_ae_rel.py",
            output_specs=[
                # if name does not include the extension, then provide the type kwarg
                Output(name="t_ae_rel", type=ReportArtifact.File(name="t_ae_rel tfl report", type="pdf")),
            ],
            use_project_defaults_for_omitted=True,
        )
    ```
  </Accordion>

Find out more about how to inspect, bookmark, and declare Flow Artifacts in [Examine Flow Artifacts](/cloud/platform-capabilities/features/flows/examine-flow-artifacts).

## Next steps

Once you have properly defined the Flow and Flow Artifacts, learn how to:

* [Launch an execution of the Flow.](/cloud/platform-capabilities/features/flows/launch-flows)


# Define Flows
Source: https://docs.domino.ai/cloud/platform-capabilities/features/flows/define-flows

Define Domino Flows and tasks in Python with typed inputs and outputs, Job configuration, data mounts, caching, retries, and timeouts.

Flow definitions are defined using Flyte's Python library.

## Flow files

### Workflow basics

Flows are defined inside a Python file. In this document, we call the file `workflow.py`, but the name can be anything.

```python workflow.py lines icon=python theme={null}
from flytekit import workflow

@workflow
def training_workflow():
    return
```

<Note>
  We're using the `@workflow` decorator to declare a function as a flow. The method name, `training_workflow`, is implicitly used as the name of the flow.
</Note>

### Inputs & outputs

Your flow can be parameterized so that you can re-run it with new inputs without modifying the code of your flow. Inputs can be either structured data or files.

<AccordionGroup>
  <Accordion title="Files">
    Declare file inputs and outputs using the generic `FlyteFile` type.

    To enable file rendering in the UI, this type must either:

    * Be combined with a `TypeVar` that defines the file extension type, for example, `FlyteFile[TypeVar("csv")]` and `FlyteFile[TypeVar("pdf")]`.
    * Be a [Flow Artifact](/cloud/platform-capabilities/features/flows/define-flow-artifacts) like `Artifact(name="My Data", type=DATA).File(name="data.csv")`
  </Accordion>

  <Accordion title="Python base types">
    Use base Python types for passing structured data through flows:

    `str`, `bool`, `int`, `float`, `list`, `dict`, `datetime`
  </Accordion>

  <Accordion title="Data science types">
    A number of common data science types are supported as well:

    `np.ndarray`, `pandas.DataFrame`, `pyspark.DataFrame`, `torch.Tensor / torch.nn.Module`, `sklearn.base.BaseEstimator`, `tf.keras.Model`
  </Accordion>
</AccordionGroup>

<Note>
  Explicit, strongly typed inputs and outputs ensure that tasks pass data to each other correctly. For instance, a data processing task may emit a Pandas dataframe as output, while a training task accepts a Pandas dataframe as input. Flows analyze workflow submissions to prevent user error before execution, providing actionable feedback for incorrectly specified parameters that don't satisfy task contracts.
</Note>

<Tip>
  Input and output types are also used for caching, data lineage tracking, and previewing in Domino.
</Tip>

Here we parameterize our flow by accepting a parameter (`n`) to our `@workflow` function and returning one of our results (the `"results"` output in `sim`):

```python workflow.py highlight={2-3,6,17} icon=python theme={null}
from flytekit import workflow
from flytekit.types.file import FlyteFile
from typing import TypeVar, NamedTuple

@workflow
def simulation(n: int) -> FlyteFile[TypeVar("csv")]:

    # Simulate data using workflow parameter
    sim = run_domino_job_task(
        flyte_task_name="Simulate data",
        command="python sim-data.py",
        inputs=[Input(name="n", type=int, value=n)],
        output_spec=[Output(name="results", type=FlyteFile[TypeVar("csv")])],
        use_project_defaults_for_omitted=True
    )

    return sim["results"]
```

## Flow tasks

### Declare tasks

Tasks are the core building blocks within a flow and are isolated within their own container during an execution. A task maps to a single Domino Job.

<Accordion title="Flow Task Jobs vs standalone Jobs">
  While all tasks trigger a unique Domino Job, there are some differences between jobs launched by a flow and standalone jobs. More specifically, for jobs launched by a flow:

  * Only Domino Dataset snapshots and NetApp Volume snapshots can be mounted, unlike standalone Domino Jobs which support snapshot mounts and read-write mounts. For data to be used in a flow, it must be part of a versioned Dataset snapshot (`version 0` of a dataset, that is, the read-write directory is NOT considered a snapshot), or part of a NetApp Volume snapshot.

  * Only one snapshot of a Dataset or NetApp Volume may be mounted at a time. When leveraging the `use_latest` flag, the latest snapshot is mounted.

  * Snapshots are `read-only` and cannot be modified during a job. Any processed data that needs to be persisted should be defined and written as a task output and therefore written to the Flow blob storage.

  * Snapshots are mounted to a standard location that doesn't include a snapshot ID in the path (the same location used for the `latest` version of a dataset or NetApp Volume in a workspace).

    For DFS projects, the path is:

    * `/domino/datasets/local/{name}` for local dataset snapshots.

    * `/domino/datasets/{name}` for shared dataset snapshots.

    * `/domino/netapp-volumes/{name}` for NetApp Volume snapshots.

    For Git-based projects, the path is:

    * `/mnt/data/{name}` for local dataset snapshots.

    * `/mnt/imported/data/{name}` for shared dataset snapshots.

    * `/mnt/netapp-volumes/{name}` for NetApp Volume snapshots.

  * Snapshots of the project code and artifacts (results) are not taken at the end of the job. Any results that need to be persisted should be defined and written as a task output and therefore written to the Flow blob storage.

  * There are two additional directories: `/workflow/inputs` and `/workflow/outputs`. These are where the task inputs/outputs of the flow are stored. A job status is considered failed if the expected task outputs are not produced by the end of execution. See the [Write task code](#write-task-code) section for more details on how to write your code accordingly.

  * Stopping a job orchestrated by a Flow in the Jobs UI stops the entire flow, including other jobs that are running as part of it.

  * Completed jobs cannot be re-run through the Domino Jobs UI. They must be relaunched by re-running the task from the Flows UI.

  * Additional job metadata is captured and displayed in the Job Details to reference the flow and task that launched it.

  These differences help to guarantee the reproducibility of flow executions by ensuring the triggered jobs adhere to a strict contract and remain side-effect free.
</Accordion>

Tasks for a flow can be defined in one of the following ways:

* Use the base [`DominoJobConfig`](#dominojobconfig) and [`DominoJobTask`](#dominojobtask) classes. These provide more flexibility and direct control of the exact parameters to use.

* Use the [`run_domino_job_task`](#run_domino_job_task) helper method. This offers a more user-friendly abstraction that enables definition and execution of a task in the same line of code.

<CodeGroup>
  ```python Class constructors expandable theme={null}
  @workflow
  def training_workflow(data_path: str) -> FlyteFile[TypeVar("csv")]:

      # Define our data prep task
      data_prep_job_config = DominoJobConfig(Command="python prep-data.py")
      data_prep_job = DominoJobTask(
          name='Prep data',
          domino_job_config=data_prep_job_config,
          inputs={'data_path': str},
          outputs={'processed_data': FlyteFile[TypeVar("csv")]},
          use_latest=True
      )

      # Get our task results
      data_prep_results = data_prep_job(data_path=data_path)

      return data_prep_results["processed_data"]
  ```

  ```python Helper functions expandable theme={null}
  @workflow
  def training_workflow(data_path: str) -> FlyteFile[TypeVar("csv")]:

      # Define our data prep task
      data_prep_results = run_domino_job_task(
          flyte_task_name="Prepare data",
          command="python prep-data.py",
          inputs=[Input(name="data_path", type=str, value=data_path)],
          output_spec=[Output(name="processed_data", type=FlyteFile[TypeVar("csv")])],
          use_project_defaults_for_omitted=True
      )

      return data_prep_results["processed_data"]
  ```
</CodeGroup>

In both cases, tasks trigger a Domino Job with the specified settings and return the results.

### Pass data

It's common for one task to **depend** on another task; that is, one task accepts an input that is produced by another task as an output. This ensures that the dependent task does not start execution until outputs from the other task are produced first.

To create dependent tasks, you can use either the base classes or helper methods to define them. In the example below, note how the second task uses the output from the first task by calling `data_prep_results["processed_data"]`.

<CodeGroup>
  ```python Class constructors expandable highlight={26} theme={null}
  @workflow
  def training_workflow(data_path: str):

      # Define our data prep task
      data_prep_job_config = DominoJobConfig(Command="python prep-data.py")
      data_prep_job = DominoJobTask(
          name='Prep data',
          domino_job_config=data_prep_job_config,
          inputs={'data_path': str},
          outputs={'processed_data': FlyteFile[TypeVar("csv")]},
          use_latest=True
      )

      # Define our training task
      training_job_config = DominoJobConfig(Command="python train-model.py")
      training_job = DominoJobTask(
          name='Train model',
          domino_job_config=training_job_config,
          inputs={'processed_data': FlyteFile[TypeVar("csv")]},
          outputs={'model': FlyteFile[TypeVar("pt")]},
          use_latest=True
      )

      # Get our task results
      data_prep_results = data_prep_job(data_path=data_path)
      training_results = training_job(processed_data=data_prep_results["processed_data"])

      return
  ```

  ```python Helper functions expandable highlight={17} theme={null}
  @workflow
  def training_workflow(data_path: str):

      # Define our data prep task
      data_prep_results = run_domino_job_task(
          flyte_task_name="Prepare data",
          command="python prep-data.py",
          inputs=[Input(name="data_path", type=str, value=data_path)],
          output_spec=[Output(name="processed_data", type=FlyteFile[TypeVar("csv")])],
          use_project_defaults_for_omitted=True
      )

      # Define our training task
      training_results = run_domino_job_task(
          name="Train model",
          command="python train-model.py",
          inputs=[Input(name="data", type=FlyteFile[TypeVar("csv")], value=data_prep_results["processed_data"])],
          outputs=[Output(name="model", type=FlyteFile[TypeVar("pt")])]
      )

      return
  ```
</CodeGroup>

### Return outputs

You can set the output of the flow by returning it in the method. Note that defining an overall Flow output is not required and does not elevate this particular output in the UI. Please see [Define Flow Artifacts](/cloud/platform-capabilities/features/flows/define-flow-artifacts) for elevating important task outputs.

<CodeGroup>
  ```python Class constructors expandable highlight={2,28} theme={null}
  @workflow
  def training_workflow(data_path: str) -> FlyteFile[TypeVar("pt")]:

      # Define our data prep task
      data_prep_job_config = DominoJobConfig(Command="python prep-data.py")
      data_prep_job = DominoJobTask(
          name='Prep data',
          domino_job_config=data_prep_job_config,
          inputs={'data_path': str},
          outputs={'processed_data': FlyteFile[TypeVar("csv")]},
          use_latest=True
      )

      # Define our training task
      training_job_config = DominoJobConfig(Command="python train-model.py")
      training_job = DominoJobTask(
          name='Train model',
          domino_job_config=training_job_config,
          inputs={'processed_data': FlyteFile[TypeVar("csv")]},
          outputs={'model': FlyteFile[TypeVar("pt")]},
          use_latest=True
      )

      # Get our task results
      data_prep_results = data_prep_job(data_path=data_path)
      training_results = training_job(processed_data=data_prep_results["processed_data"])

      return training_results["model"]
  ```

  ```python Helper functions expandable theme={null}
  @workflow
  def training_workflow(data_path: str) -> FlyteFile[TypeVar("pt")]:

      # Define our data prep task
      data_prep_results = run_domino_job_task(
          flyte_task_name="Prepare data",
          command="python prep-data.py",
          inputs=[Input(name="data_path", type=str, value=data_path)],
          output_spec=[Output(name="processed_data", type=FlyteFile[TypeVar("csv")])],
          use_project_defaults_for_omitted=True
      )

      # Define our training task
      training_results = run_domino_job_task(
          name="Train model",
          command="python train-model.py",
          inputs=[Input(name="data", type=FlyteFile[TypeVar("csv")], value=data_prep_results["processed_data"])],
          outputs=[Output(name="model", type=FlyteFile[TypeVar("pt")])]
      )

      return training_results["model"]
  ```
</CodeGroup>

### Write task code

Domino code execution aims for consistency across all execution modalities. To support this, flows can leverage the same data mount paths used in a workspace or standalone job.

As an input, declared inputs that reside under `/mnt` are populated for reading. Data written back to data paths can optionally be used to create a new snapshot of the data at the end of the flow run.

However, for explicit inputs and outputs, flows additionally mounts data for input and output in two special directories under `/workflow`. Files can be read from `/workflow/inputs` and written to `/workflow/outputs`. In addition to files, flows also allows for the passing of structured data such as dataframes and lists. These objects also show up in the `/workflow` directories, but require special handling to read them as structured data in Python.

#### Read inputs

For each input that is defined for a task, a file is created and accessible within a Job within `/workflow/inputs/` in a file whose name exactly matches that of the named input.

For file type inputs, the file can be read directly at the given path.

<CodeGroup>
  ```python task.py icon=python theme={null}
  named_input = "processed_data"
  data_path = "/workflow/inputs/{}".format(named_input)
  df = pd.read_csv(data_path)
  ```

  ```r task.R icon=R theme={null}
  named_input <- "processed_data"
  df <- read.csv(file.path("/workflow", "inputs", named_input))
  ```
</CodeGroup>

For Python non-file types (`str`, `bool`, `int`, `list`, `dict`, etc.), the blob contents contain the input value. Example usage:

```python theme={null}
input_name = "data_path"
input_location = f"/workflow/inputs/{input_name}"
with open(input_location, "r") as file:
    input_value = file.read()
```

<Expandable title="Configuring inputs for local task execution">
  The Domino Python helper library has functions that facilitate setting up input data. These functions allow you to use inputs or outputs from past Flyte executions.\
  Here is an example that shows setting up local input data for a task that expects five inputs:

  1. `my_csv_one`, using the output of a previous Flow execution.
  2. `my_csv_two`, using the input of a previous Flow execution.
  3. `my_csv_three`, using the local file `test_data.csv`.
  4. `my_str`, using the string `"my string input data"`.
  5. `my_int`, using the integer `42`.

  ```python theme={null}
  import shutil
  from flytekitplugins.domino.helpers import BlobDataLocation, PrimitiveDataLocation, setup_workflow_data

  blobs = [
      BlobDataLocation(
          # you can find blob URLs by inspecting node executions in the Domino Flyte UI
          "s3://flyte-data-bucket/past/execution1/output/some_csv",
          # the input name will incorrectly be inferred as "some_csv" from the blob url,
          # so the local_filename kwarg must be provided and match the input name expected by the task code
          local_filename="my_csv_one",
      ),
      BlobDataLocation("s3://flyte-data-bucket/past/execution2/input/my_csv_two"),
  ]

  primitives = [
      PrimitiveDataLocation("my string input data", "my_str"),
      PrimitiveDataLocation(42, "my_int")
  ]

  setup_workflow_data(blobs, primitives)
  shutil.copyfile("test_data.csv", "/workflow/inputs/my_csv_three")
  ```
</Expandable>

#### Write outputs

Outputs defined for a task must be written into `/workflow/outputs/` to a file that exactly matches the name of the output.

<CodeGroup>
  ```python task.py icon=python theme={null}
  named_output = "processed_data"
  df.to_csv("/workflow/outputs/{}".format(named_output))
  ```

  ```python task.R icon=R theme={null}
  named_output <- "processed_data"
  write.csv(df, file.path("/workflow", "outputs", named_output))
  ```
</CodeGroup>

<Warning>
  If the defined outputs do not exist at the end of a Domino Job the task fails.
</Warning>

<Note>
  Flow jobs do not make commits or dataset snapshots. Results should always be written as outputs. Snapshots are taken at the end of flow execution if configured.
</Note>

## Domino Flyte plugin

Domino provides a set of Flyte-compatible classes, as well as utility functions for defining tasks using these classes, in a way that integrates nicely with Domino Jobs. Notably, the `flytekitplugins.domino` library provides groups of utilities in the `helpers`, `task`, and `artifact` submodules.

### Class constructors

The following example defines a task using the base classes.

```python theme={null}
from typing import TypeVar, NamedTuple
from flytekit import workflow
from flytekitplugins.domino.task import DominoJobConfig, DominoJobTask

@workflow
def training_workflow(data_path: str) -> FlyteFile[TypeVar("pt")]:

    # Defining a task using Domino classes
    data_prep_job_config = DominoJobConfig(Command="python prep-data.py")
    data_prep_job = DominoJobTask(
        name='Prepare Data',
        domino_job_config=data_prep_job_config,
        inputs={'data_path': str},
        outputs={'processed_data': FlyteFile[TypeVar("csv")]},
        use_latest=True
    )
    data_prep_results = data_prep_job(data_path=data_path)

    return
```

To pin a specific branch, commit, or tag for one or more of the project's imported Git repos on a per-task basis, pass `ImportedGitRepoOverrides` to `DominoJobConfig` (Git-based projects only):

```python theme={null}
training_job_config = DominoJobConfig(
    Command="python train-model.py",
    MainRepoGitRef=GitRef(Type="branches", Value="main"),
    ImportedGitRepoOverrides=[
        ImportedGitRepoOverride(
            Name="shared-utils",
            Ref=GitRef(Type="commitId", Value="2f1cb9bf696921f0858"),
        ),
    ],
)
```

#### `DominoJobConfig()`

The `DominoJobConfig` defines the configuration for the Domino Job that is triggered by the task.

<Expandable title="Parameters">
  <ParamField type="String">
    The title that is given to the Domino Job. Ignored when calculating the config's cache version.

    *Example:* `"My training job"`
  </ParamField>

  <ParamField type="String">
    The command that is executed in the Domino Job.

    *Example:* `"python train.py"`
  </ParamField>

  <ParamField type="String">
    For projects hosted in the Domino File System, this refers to the commit ID of the code. For Git-based projects, this refers to the commit ID of Artifacts.

    *Example:* `"953f66f3153b71658d"`
  </ParamField>

  <ParamField type="GitRef">
    Reference a specific branch, commit, or tag (for Git-based projects only). See the [API guide](/cloud/reference/api/domino-open-api) for more details.

    *Example:* `GitRef(Type="commitId", Value="2f1cb9bf696921f0858")`
  </ParamField>

  <ParamField type="String">
    The resolved commit hash to use. In most cases, if a specific commit is needed, it is best to use the `Value` argument to `GitRef`. Specifying the resolved commit explicitly can be used to track a floating reference, but resolve to an older commit.

    *Example:* `2f1cb9bf696921f0858`
  </ParamField>

  <ParamField type="Boolean">
    When `True`, floating `git` references specified in `MainRepoGitRef` and `ImportedGitRepoOverrides` are resolved to the latest commit when the Flow is run or relaunched. When `False`, floating `git` references resolve the reference on the first run, which is reused when the Flow is relaunched.

    *Example:* `True`
  </ParamField>

  <ParamField type="String">
    The ID of the Domino Hardware Tier. Note that this is different than the name of the hardware tier. Ignored when calculating the config's cache version.

    *Example:* `small-k8s`
  </ParamField>

  <ParamField type="String">
    The ID of the Domino Environment. Note that this is different than the name or revisionId of the Environment.

    *Example:* `6646530dcbd87f1a3dec0050`
  </ParamField>

  <ParamField type="EnvironmentRevisionSpecification">
    The revision of the specified Domino Environment. See the `EnvironmentRevisionSpec` example after the table.
  </ParamField>

  <ParamField type="Optional[ClusterProperties]">
    Compute cluster properties for the job. See the `ComputeClusterProperties` example after the table.
  </ParamField>

  <ParamField type="Float">
    The amount of disk space (in GiB) to allocate. Ignored when calculating the config's cache version.

    *Example:* `10.0`
  </ParamField>

  <ParamField type="List[DatasetSnapshot]">
    List of the dataset snapshots to include in the job. Note that `version 0` of a dataset cannot be used, since it is mutable. You must take a snapshot first before using a dataset in a flow.

    *Example:* `[DatasetSnapshot(Id="6615af2820a4", Version=1)]`
  </ParamField>

  <ParamField type="List[NetAppVolumeSnapshot]">
    List of the NetApp Volume snapshots to include in the job. Note that `version 0` of a NetAppVolume cannot be used, since it is mutable. You must take a snapshot first before using a NetApp Volume in a flow.

    *Example:* `[NetAppVolumeSnapshot(Id="5d1204c2-24c8", Version=1)]`
  </ParamField>

  <ParamField type="List[String]">
    List of the external data volume mounts (referenced by ID) to include in the Job.

    *Example:* `["9625af24kida4dc035aa881b7"]`
  </ParamField>

  <ParamField type="List[ImportedGitRepoOverride]">
    Pin a specific branch, commit, or tag for one or more of the project's imported Git repos, per task (for Git-based projects only). `Name` must match an imported repository configured on the project, and `Ref` is a `GitRef`.

    *Example:* `[ImportedGitRepoOverride(Name="shared-utils", Ref=GitRef(Type="branches", Value="main"))]`
  </ParamField>

  <ParamField type="Optional[Tuple[str, ...]]">
    Optional tuple of names of arguments to the `DominoJobConfig` to ignore when calculating the config's cache version.

    *Example:* `("EnvironmentId",)`
  </ParamField>
</Expandable>

#### `DominoJobTask()`

Calling the Domino Job task with the relevant inputs (`data_prep_job(data_path=data_path)`) runs the Domino Job and returns the results as a `Promise`, which can be used as an input to downstream tasks.

<Expandable title="Parameters">
  <ParamField type="String">
    The name that is given to the task.

    *Example:* `My training task`
  </ParamField>

  <ParamField type="DominoJobConfig">
    The job configuration, as defined above.

    *Example:* `DominoJobConfig(Command="python prep-data.py")`
  </ParamField>

  <ParamField type="Dict[String, Type]">
    Inputs that are required by the task. See above for different input types that are supported. Inputs may be specified from the workflow OR may be outputs from other tasks.

    *Example:* `{'data_path': str}`
  </ParamField>

  <ParamField type="Dict[String, Type]">
    Outputs that are produced by the task. See above for different output types that are supported.

    *Example:* `{'processed_data': FlyteFile[TypeVar("csv")]}`
  </ParamField>

  <ParamField type="Boolean">
    If set to `True`, this uses the latest project defaults for parameters that were not explicitly provided, like the compute environment version and hardware tier. For better reproducibility, it is recommended to set this to `False` and explicitly define the necessary parameters.

    *Example:* `False`
  </ParamField>

  <ParamField type="Union[bool, Cache]">
    Indicates if caching should be enabled. Can be a `bool` or a Flyte `Cache` object. Setting `cache=true` is a simple way to enable caching for a task, but using a `Cache` object may be more flexible depending on your use case. See the [caching documentation](/cloud/platform-capabilities/features/flows/advanced-flows#caching) for a full explanation of how caching works.

    *Example:* `True`
  </ParamField>

  <ParamField type="String">
    When `cache=true`, the cache version to use. Changes to the task signature automatically trigger a cache miss, but you can always manually update this field as well to force a cache miss. You should also manually bump this version if the function body/business logic has changed, but the signature hasn't.

    *Example:* `"1.0"`
  </ParamField>

  <ParamField type="Tuple[str, ...]">
    When `cache=true`, variable names of `inputs` to the `DominoJobTask` that should not be included when calculating the hash used for caching. If not provided, all input variables are included when calculating the hash.

    *Example:* `(batch_size,)`
  </ParamField>

  <ParamField type="Integer">
    Number of times to retry this task during a workflow execution. This can be used to help automatically mitigate intermittent failures.

    *Example:* `0`
  </ParamField>

  <ParamField type="Union[timedelta, int]">
    The maximum amount of time for which one execution of this task should be executed. The execution is terminated if the runtime exceeds the given timeout.

    *Example:* `timedelta(hours=3)`
  </ParamField>
</Expandable>

#### `EnvironmentRevisionSpec()`

```python theme={null}
EnvironmentRevisionSpecification(
    EnvironmentRevisionType=EnvironmentRevisionType.SomeRevision,
    EnvironmentRevisionId="6659daf5fc8de",
)
```

#### `ComputeClusterProperties()`

```python theme={null}
ClusterProperties(
    ClusterType=ComputeClusterType.Dask,
    ComputeEnvironmentId="computeEnvId",
    WorkerHardwareTierId="workerHardwareTierId",
    WorkerCount=3,
    WorkerStorageGiB=100.0,
    MaxWorkerCount=10,
    ComputeEnvironmentRevisionSpec=EnvironmentRevisionSpecification(
        EnvironmentRevisionType.SomeRevision,
        "revisionId",
    ),
    MasterHardwareTierId="masterHardwareTierId",
    ExtraConfigs={"foo": "bar"},
)
```

### Helpers

#### `run_domino_job_task()`

Helper methods reduce the amount of code necessary to invoke a task. Instead of separately defining a `DominoJobConfig` and passing it to a `DominoJobTask` in the examples above, use `run_domino_job_task` to define the task config, job, inputs and outputs all using a single function call.

<Note>
  To pin imported Git repos for a task, use the base `DominoJobConfig` and `DominoJobTask` classes with `ImportedGitRepoOverrides`. The `run_domino_job_task` helper also accepts `imported_git_repo_overrides`.
</Note>

```python theme={null}
@workflow
def training_workflow(data_path: str) -> FlyteFile[TypeVar("pt")]:

    # First task using helper method
    data_prep_results = run_domino_job_task(
        flyte_task_name="Prepare data",
        command="python prep-data.py",
        inputs=[Input(name="data_path", type=str, value=data_path)],
        output_spec=[Output(name="processed_data", type=FlyteFile[TypeVar("csv")])],
        use_project_defaults_for_omitted=True
    )

    # Output from the task above will be used in the next step

    return # Outputs will be returned here
```

<Expandable title="Parameters">
  <ParamField type="String">
    The title that is given to the task.

    *Example:* `My training task`
  </ParamField>

  <ParamField type="String">
    The title that is given to the Domino Job.

    *Example:* `My training job`
  </ParamField>

  <ParamField type="Boolean">
    If set to `True`, this uses the latest project defaults for parameters that were not explicitly provided. For better reproducibility, it is recommended to set this to `False` and explicitly define the necessary parameters.

    *Example:* `False`
  </ParamField>

  <ParamField type="String">
    For projects hosted in the Domino File System, this refers to the commit ID of the code. For Git-based projects, this refers to the commit ID of the artifacts.

    *Example:* `953f66f3153b71658`
  </ParamField>

  <ParamField type="GitRef">
    Reference a specific branch, commit, or tag (for Git-based projects only). See the [API guide](/cloud/reference/api/domino-open-api) for more details.

    *Example:* `GitRef(Type="commitId", Value="2f1cb9bf696921f0")`
  </ParamField>

  <ParamField type="String">
    Name of the Environment to use in the Domino Job.

    *Example:* `Domino Standard Environment`
  </ParamField>

  <ParamField type="String">
    ID of the Environment to use in the Domino Job. This is recommended over using `environment_name` to prevent breaking reproducibility when environment names get changed.

    *Example:* `6646530dcbd`
  </ParamField>

  <ParamField type="String">
    A specific revisionId of the environment to use.

    *Example:* `6659daf5fc8`
  </ParamField>

  <ParamField type="String">
    Name of the hardware tier to use in the Domino Job.

    *Example:* `Large`
  </ParamField>

  <ParamField type="String">
    ID of the hardware tier to use in the Domino Job. This is recommended over using `hardware_tier_name` to prevent breaking reproducibility when hardware tier names get changed.

    *Example:* `small-k8s`
  </ParamField>

  <ParamField type="List[Input]">
    Inputs that are required by the task. See above for different input types that are supported. See [`Input()`](#input) for details.
  </ParamField>

  <ParamField type="List[Output]">
    Outputs that are produced by the task. See above for different output types that are supported.

    *Example:* `[Output(name="processed_data", type=FlyteFile[TypeVar("csv")])]`
  </ParamField>

  <ParamField type="Integer">
    The amount of disk space (in GiB) to allocate.

    *Example:* `10`
  </ParamField>

  <ParamField type="List[DatasetSnapshot]">
    List of the dataset snapshots to include in the Job. Note that `version 0` of a dataset cannot be used, since it is mutable. You must first take a snapshot before using a dataset in a flow.

    *Example:* `[DatasetSnapshot(Id="6615af2820a4", Version=1)]`
  </ParamField>

  <ParamField type="List[NetAppVolumeSnapshot]">
    List of the NetApp Volume snapshots to include in the Job. Note that `version 0` of a NetApp Volume cannot be used, since it is mutable. You must first take a snapshot before using a NetApp Volume in a flow.

    *Example:* `[NetAppVolumeSnapshot(Id="5d1204c2-24c8", Version=1)]`
  </ParamField>

  <ParamField type="List[str]">
    List of the external data volume mounts (referenced by ID) to include in the Job.

    *Example:* `[9625af24kida4dc035aa881b7]`
  </ParamField>

  <ParamField type="Union[bool, Cache]">
    Indicates if caching should be enabled. Can be a `bool` or a Flyte `Cache` object. Defaults to `false`. Setting `cache=true` is a simple way to enable caching for a task, but using a `Cache` object may be more flexible depending on your use case. See the [caching documentation](/cloud/platform-capabilities/features/flows/advanced-flows#caching) for a full explanation of how caching works.

    *Example:* `False`
  </ParamField>

  <ParamField type="String">
    When `cache=true`, the cache version to use. Changes to the task signature automatically trigger a cache miss, but you can always manually update this field as well to force a cache miss. You should also manually bump this version if the function body/business logic has changed, but the signature hasn't.

    *Example:* `"1.0"`
  </ParamField>

  <ParamField type="Tuple[str, ...]">
    When `cache=true`, variable names of `inputs` to the `DominoJobTask` that should not be included when calculating the hash used for caching. If not provided, all input variables are included when calculating the hash.

    *Example:* `(batch_size,)`
  </ParamField>

  <ParamField type="Integer">
    Number of times to retry this task during a workflow execution.

    *Example:* `0`
  </ParamField>

  <ParamField type="Union[timedelta, int]">
    The maximum amount of time for which one execution of this task should be executed. The execution is terminated if the runtime exceeds the given timeout.

    *Example:* `timedelta(hours=3)`
  </ParamField>
</Expandable>

#### `Input()`

Specify a Flows Task input. Inputs include values and files, and must be declared for each Task.

```python theme={null}
Input(name="number", type=int, value=13)

Input(
    name="data",
    type=FlyteFile[TypeVar("csv")],
    path="/mnt/data/numbers/data.csv"
)
```

<Expandable title="Parameters">
  <ParamField type="String">
    The name of the input. When used in a `DominoJobTask`, this name appears in the Flyte UI as the name of an input to the task.
  </ParamField>

  <ParamField type="Type">
    The programmatic type of the input. When used in a `DominoJobTask`, this type appears in the Flyte UI as the type of an input to the task.
  </ParamField>

  <ParamField type="Any">
    The value of the input. When used in a `DominoJobTask`, this value is passed to the task for the associated input name, and the value appears in the Flyte UI. The value must be of the type defined by the `type` parameter.
  </ParamField>

  <ParamField type="String">
    The path to the input file. When used in a `DominoJobTask`, this path is used to download the file to the local disk. If not provided, the file is downloaded to the default location in `/workflow/inputs`.

    <Tip>
      For consistency with workspace execution, paths can be passed to the Dataset (`/mnt/data`) or NetAppVolume (`/mnt/netapp-volumes/`) data locations. Data in these locations can be both an input and an output, populating the files when the Task starts and propagating the files as outputs for downstream tasks.
    </Tip>

    *Example:* `"/mnt/netapp-volumes/project-data/data.csv"`
  </ParamField>
</Expandable>

#### `Output()`

Specify a Flows Task output.

```python theme={null}
Output(name="number", type=int)

Output(
    name="data",
    type=FlyteFile[TypeVar("csv")],
    path="/mnt/data/numbers/data.csv"
)
```

<Expandable title="Parameters">
  <ParamField type="String">
    The name of the input. When used in a `DominoJobTask`, this name appears in the Flyte UI as the name of an input to the task.
  </ParamField>

  <ParamField type="Type">
    The programmatic type of the input. When used in a `DominoJobTask`, this type appears in the Flyte UI as the type of an input to the task.
  </ParamField>

  <ParamField type="String">
    The path to the input file. When used in a `DominoJobTask`, this path is used to download the file to the local disk. If not provided, the file is downloaded to the default location in `/workflow/inputs`.

    <Tip>
      For consistency with workspace execution, paths can be passed to the Dataset (`/mnt/data`) or NetAppVolume (`/mnt/netapp-volumes/`) data locations. Data in these locations can be both an input and an output, populating the files when the Task starts and propagating the files as outputs for downstream tasks.
    </Tip>

    *Example:* `"/mnt/netapp-volumes/project-data/data.csv"`
  </ParamField>
</Expandable>

## Next steps

Once you have properly defined the flow, learn how to:

* [Define Flow Artifacts.](/cloud/platform-capabilities/features/flows/define-flow-artifacts)

* [Launch an execution of the flow](/cloud/platform-capabilities/features/flows/launch-flows).


# Examine Flow Artifacts
Source: https://docs.domino.ai/cloud/platform-capabilities/features/flows/examine-flow-artifacts

Inspect, preview, download, and bookmark Flow Artifacts from runs, then assign data artifacts to Domino Datasets while retaining lineage.

After you have defined Flow Artifacts, you can inspect artifacts produced by executions and discover bookmarked artifacts within a project.

## Inspect flow artifacts produced by executions

Flow Artifacts are surfaced in a dedicated section in the run view, where they can be easily discovered, inspected, and downloaded.

Navigate to **Flows** > **Flow name** > **Run name** > **Artifacts** to inspect the artifacts produced by that particular run.

<img alt="Inspect artifacts produced by a run" />

### Bookmark artifacts

As many flows are executed, specific flow executions may generate outputs that are especially noteworthy to the project. These Flow Artifacts can be bookmarked by clicking on the **Bookmark artifact** icon on the right panel:

<img alt="Bookmark an artifact with the icon" />

Flow Artifacts can also be bookmarked in bulk by selecting multiple artifacts on the left panel, and clicking on the **Bookmark** button above:

<img alt="Bookmark artifacts in bulk" />

### Access artifacts

Artifacts that have smaller file sizes may be viewed directly within Domino. The right panel provides two additional means of accessing artifacts, either by downloading them or by accessing them via code.

* The **Download** button downloads all the files that are part of the selected artifact.

  In this example, all five files (`model.pkl`, `python_env.yaml`, `requirements.txt`, `conda.yaml`, and `MLmodel`) are downloaded at once:

  <img alt="Download files of a selected artifact" />

* The **Code snippet** button generates Python code to directly download stored artifacts to a local workspace. Copy the code using the **Copy** button and run it inside a workspace:

  <img alt="Access artifacts through code snippets" />

* The default syntax in the code snippet copies all files within the Flow Artifact to the temporary `/workflow/inputs/` directory within your workspace.

  To persist your Flow Artifacts outside of Flows so they are consumable for downstream processes, such as web apps or a batch job, change the `local_dir` parameter in the code snippet to write your files to a Domino Dataset or the DFS Artifacts file system.

## Discover bookmarked flow artifacts

Artifacts that have already been bookmarked are displayed at the top level view for Flows to allow for easier discoverability and reuse.

* Navigate to **Flows** > **Bookmarked Artifacts**, and view all the artifacts that were bookmarked on the current project, across flows and runs. This view includes the artifact name, type, and version, as well as additional metadata.

  <img alt="View bookmarked artifacts" />

* Clicking on the **Source** column navigates to the run that created the artifact, while clicking on the **Artifact** column displays a view of the artifact that includes all of its files.

  <img alt="Display files of an artifact" />

* This paginated view allows filtering files by name, as well as downloading one or more selected files at once. Clicking on a file name displays a larger file preview, alongside an option to download the file.

  <img alt="Display a larger file preview" />

## Assign flow artifacts to Datasets

Once you have created a Flow Artifact, you can assign it to a dataset. You can choose an existing dataset or create a new one. There is also an option to automatically take a dataset snapshot after registration. The data’s lineage is automatically maintained between the Flow Artifact and the Dataset Snapshot.

<img alt="Add artifact to Dataset" />

After the artifact is added successfully to the dataset, you’ll see a confirmation message.

## Next steps

* Maximize the [reproducibility](/cloud/platform-capabilities/features/flows/reproducibility-flows) of Flows.


# Get started with Flows
Source: https://docs.domino.ai/cloud/platform-capabilities/features/flows/get-started-with-flows

Build and run basic math and model training flows with Flyte Python tasks, typed inputs and outputs, caching, and the Flyte CLI.

Make sure you have a good understanding of the [key concepts](/cloud/platform-capabilities/features/flows#important-terms) before you get started with Domino Flows.

## Basic flow

This section demonstrates a basic example that:

* Takes two integers as an input to a flow.

* The first task adds the integers together and passes the result as an input to the second task.

* The second task takes the square root of the input and returns the result as the final output of the flow.

This example flow can be visualized as follows:

<img alt="Simple Math Flow Graph" />

To create a Domino Flow:

1. Create a workspace using the [Domino Standard Environment (DSE)](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) from 6.0 onwards, or a custom environment that is built on top of the DSE >= 6.0, as these contain the required Flyte Python libraries.

2. Create a file named `add.py` in the root directory. Add the following code to the file to add two integer inputs together:

   ```python theme={null}
   from pathlib import Path

   # Read inputs
   a = Path("/workflow/inputs/first_value").read_text()
   b = Path("/workflow/inputs/second_value").read_text()

   # Calculate sum
   sum = int(a) + int(b)
   print(f"The sum of {a} + {b} is {sum}")

   # Write output
   Path("/workflow/outputs/sum").write_text(str(sum))
   ```

3. Create a file named `sqrt.py` in the root directory. Add the following code to the file to calculate the square root of the input:

   ```python theme={null}
   from pathlib import Path

   # Read input
   value = Path("/workflow/inputs/value").read_text()

   # Calculate square root
   sqrt = int(value) ** 0.5
   print(f"The square root of {value} is {sqrt}")

   # Write output
   Path("/workflow/outputs/sqrt").write_text(str(sqrt))
   ```

4. Create a file named `workflow.py` in the root directory (see the [Flytekit Python library](https://github.com/dominodatalab/flytekit) documentation on GitHub). Add the following code to the file to define the flow:

   ```python theme={null}
   from flytekit import workflow
   from flytekitplugins.domino.task import DominoJobConfig, DominoJobTask

   @workflow
   def simple_math_workflow(a: int, b: int) -> float:

       # Create first task
       add_task = DominoJobTask(
           name='Add numbers',
           domino_job_config=DominoJobConfig(Command="python add.py"),
           inputs={'first_value': int, 'second_value': int},
           outputs={'sum': int},
           use_latest=True
       )
       sum = add_task(first_value=a, second_value=b)

       # Create second task
       sqrt_task = DominoJobTask(
           name='Square root',
           domino_job_config=DominoJobConfig(Command="python sqrt.py"),
           inputs={'value': int},
           outputs={'sqrt': float},
           use_latest=True
       )
       sqrt = sqrt_task(value=sum)

       return sqrt
   ```

5. **Commit the code** and run the following command in the Workspace terminal to register and run the flow:

   ```bash theme={null}
   pyflyte run --remote workflow.py simple_math_workflow --a 10 --b 6
   ```

6. Once you run the command above, navigate to **Flows** > **Flow name** > **Run Name** in the Domino UI to monitor the results and view the outputs that were produced by the execution.

   <img alt="Monitor simple flow" />

7. To visualize the full execution flow, click the **Graph** pivot.

   <img alt="Simple Flow Graph" />

## Training flow

This section demonstrates a basic training flow example with the following steps:

1. Data is loaded in from two different sources and a snapshot of the data is taken.

2. The data is merged together as a single dataset.

3. Basic preprocessing is done on the dataset.

4. A model is trained using the cleaned dataset.

The training flow can be visualized as follows:

<img alt="Training Flow Graph" />

To create the training flow:

1. Make a fork of the template [GitHub repository](https://github.com/dominodatalab/domino-ai-flows).

2. Create a Workspace using the [Domino Standard Environment (DSE)](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) from 6.0 onwards, or a custom environment that is built on top of the DSE >= 6.0.

   <img alt="Workspace" />

3. Inspect the `mlops_flow.py` file for the definition of the flow. Note how a helper method, called `run_domino_job_task`, is used here instead of the `DominoJobConfig` and `DominoJobTask` in the basic example above.

   ```python theme={null}
   task1 = run_domino_job_task(
       flyte_task_name='Load Data A',
       command='python /mnt/code/scripts/load-data-A.py',
       inputs=[Input(name='data_path', type=str, value=data_path_a)],
       output_specs=[Output(name='datasetA', type=FlyteFile[TypeVar('csv')])],
       use_project_defaults_for_omitted=True,
       environment_name=environment_name,
       hardware_tier_name="Small",
       cache=cache,
       cache_version="1.0"
   )

   task2 = run_domino_job_task(
       flyte_task_name='Load Data B',
       command='python /mnt/code/scripts/load-data-B.py',
       inputs=[Input(name='data_path', type=str, value=data_path_b)],
       output_specs=[Output(name='datasetB', type=FlyteFile[TypeVar('csv')])],
       use_project_defaults_for_omitted=True,
       environment_name=environment_name,
       hardware_tier_name="Small",
       cache=cache,
       cache_version="1.0"
   )

   # Additional tasks
   ```

4. Run the following command in the Workspace terminal to register and run the flow:

   ```bash theme={null}
   pyflyte run --remote mlops_flow.py model_training --data_path_a /mnt/code/data/datasetA.csv --data_path_b /mnt/code/data/datasetB.csv
   ```

5. Navigate to **Flows** > **Flow name** > **Run name** to monitor the results and view the outputs that were produced by the execution.

   <img alt="Training Flow Run" />

## Next steps

* [Define Flows](/cloud/platform-capabilities/features/flows/define-flows)

* [Define Flow Artifacts](/cloud/platform-capabilities/features/flows/define-flow-artifacts)

* [Launch Flows](/cloud/platform-capabilities/features/flows/launch-flows)

* [Monitor Flows](/cloud/platform-capabilities/features/flows/monitor-flows)

* [Examine Flow Artifacts](/cloud/platform-capabilities/features/flows/examine-flow-artifacts)

* [Flows Reproducibility](/cloud/platform-capabilities/features/flows/reproducibility-flows)

* [Advanced Flows](/cloud/platform-capabilities/features/flows/advanced-flows)


# Orchestrate with Flows
Source: https://docs.domino.ai/cloud/platform-capabilities/features/flows/index

Orchestrate reproducible, versioned Domino Jobs as typed Flow tasks, connect dependencies, run tasks in parallel, and inspect lineage.

<img alt="Domino Flows Icon" />

Domino Flows enables efficient orchestration and monitoring of complex, interconnected multi-step processes while ensuring full lineage and reliable reproducibility. The processes, implemented as Domino jobs, are **tasks** and the complete structure of connections between tasks is a **workflow**. Tasks produce outputs that become inputs to other tasks, forming the basis for the connections. A Flow definition constructs a DAG (directed acyclic graph).

<Note>
  To support reproducibility, each task must be side-effect free by reading versioned inputs, and writing defined outputs.
</Note>

## When to use flows

Flows is flexible enough to declaratively model arbitrarily complex processes. Dependency relationships between tasks determine the order in which they run and whether they can be parallelized. Scenarios spanning machine learning, data engineering, and data analytics benefit from this level of control and reproducibility.

For instance, Flows would be an ideal choice for scenarios like:

* Executing a data processing workflow in Dask prior to a training workflow in XGBoost.

* Running a clinical study pipeline by loading SDTM datasets to produce ADaM datasets and TFL reports.

* Collecting image metadata from S3 with Spark and performing model inference with PyTorch.

* Loading financial data from Snowflake, processing it for use in a Ray training job that registers a model in MLflow.

* Processing a local protein database to search for a nucleotide sequence and generating a scatterplot.

Flows may not be the most appropriate choice to use when modeling a process that accesses a single dataset and performs many small computations in a homogeneous environment. Tasks that write to mutable shared state (like read-write datasets) cannot be used in Flows, but can be made compatible with modifications.

## Flow tasks vs standalone Domino Jobs

Flows extends the Domino Job system with key new functionality including:

* **Programmatic Python based authoring** of versioned, reusable, repeatable, immutable workflows.

* **Strongly typed definitions** of inputs being consumed and outputs being produced for each task.

* **Automatic lineage and versioning** of all task and workflow inputs and outputs.

* **Heterogeneous, isolated environment** support for any task.

* **Stronger reproducibility** requirements and guarantees.

* **Visualization of the workflow execution graph** and the ability to inspect and monitor each task, its inputs and outputs.

* **Parallel execution** of tasks at scale.

* **Configurable caching** and task result reuse anywhere within the workflow.

* **[Flow Artifacts](/cloud/platform-capabilities/features/flows/define-flow-artifacts)** for discovery, inspection and reuse of specially annotated outputs within a project.

* **Automatic recovery** from intermittent failures and manual recovery of partial executions.

Read more about the [differences between Flows-generated tasks that run Domino jobs vs standalone Domino Jobs](/cloud/platform-capabilities/features/flows/define-flows).

Flows is built on the open source framework [Flyte](https://flyte.org/).

## Important terms

Some key terms to understand before getting started with Flows include:

| Term | Definition |
| - | - |
| Task | Tasks are the core building blocks within a flow and are isolated within their own container during an execution. A task maps to a single Domino Job. |
| Flow | A flow is a composition of multiple tasks or other flows (called subflows). Flows can be triggered through a single command and are tracked as a single, fully reproducible entity. |
| Node | A node represents a unit of execution or work within a flow (they show up as individual blocks in the graph views). A node can contain either a single task or a whole flow (called subflows). |
| Task inputs | Task inputs are strongly typed parameters that can be defined on individual tasks. Inputs allow tasks to be rerun with different settings through the UI, without the need to modify the code itself. Inputs can be read and used within executions. |
| Task outputs | Task outputs are strongly typed parameters that define the results that are produced by a task. Outputs are tracked and stored in discrete blob storage, so that they can be used as input to another task. |
| Flow inputs/outputs | Flow inputs/outputs are similar to the task inputs/outputs but are defined at the flow level. Inputs defined for a flow can be passed into relevant tasks, and outputs from tasks can be returned as the overall output for a flow. |

## Next steps

* See [Get started with Flows](/cloud/platform-capabilities/features/flows/get-started-with-flows) to understand the key concepts before you get started with Domino Flows.

* [Define Flows](/cloud/platform-capabilities/features/flows/define-flows) via a code-first approach using Flyte’s Python SDK.

* Explicitly [define Flow Artifacts](/cloud/platform-capabilities/features/flows/define-flow-artifacts) in your code.

* Once Flows are defined, you can [register and launch](/cloud/platform-capabilities/features/flows/launch-flows) them.

* Use the comprehensive Domino Flows user experience to [monitor Flows](/cloud/platform-capabilities/features/flows/monitor-flows).

* After you have defined Flow Artifacts, you can [examine](/cloud/platform-capabilities/features/flows/examine-flow-artifacts) them.

* Find out how every flow, task, and execution are uniquely versioned in Domino Flows to guarantee [reproducibility](/cloud/platform-capabilities/features/flows/reproducibility-flows).

* Learn more about the [advanced capabilities](/cloud/platform-capabilities/features/flows/advanced-flows) that you can use in Domino Flows.


# Launch Flows
Source: https://docs.domino.ai/cloud/platform-capabilities/features/flows/launch-flows

Register and launch Flows with the Flyte CLI, rerun them with new inputs or task overrides, and recover failed runs from the UI.

Once flows are defined, they can be registered and launched using the Flyte CLI through a Workspace or Job. Flows can also be relaunched from the UI once they have been registered into Domino.

## Launch a new flow

A new flow must be launched using the Flyte CLI via the `pyflyte run` command.

In the example below, we assume that:

* A flow is defined inside a `workflow.py` file.

* The flow is defined using a method called `test_workflow`.

* The flow takes two input arguments called `inputA` (int) and `inputB` (string).

Therefore, the following command can be used to run the flow:

```bash theme={null}
pyflyte run --remote workflow.py test_workflow --inputA 5 --inputB "Hello"
```

The above command can be executed inside a Workspace terminal or as the start command for a Job. Once it is executed, the flow is automatically registered and execution should start immediately.

### Set a name

A name is automatically generated for each run, but you can explicitly set the name as well:

```bash theme={null}
pyflyte run --remote --name MY_CUSTOM_NAME workflow.py test_workflow --inputA 5 --inputB "Hello"
```

<Note>
  Flow run names must be unique across all your flows within the same project.
</Note>

### Add tags

You can also add tags to a run execution:

```bash theme={null}
pyflyte run --remote --tag MY_TAG=MY_VALUE workflow.py test_workflow --inputA 5 --inputB "Hello"
```

## Flow versioning

Flows are automatically versioned inside Domino. Runs of each flow version are grouped together and the runs of the latest flow version are shown by default. You can view and select versions in the Runs view of your flow.

<img alt="Flow Versions" />

Any changes made to the `workflow.py` definition file cause a new version to be serialized and registered to Domino when that flow is launched from the CLI. This flexibility allows for rapid iteration: new workflow definitions can be written, serialized, and tested in flows without requiring an immediate commit to Git or DFS. This speeds up development and debugging as workflows can be quickly adjusted and re-registered.

However, since the registered flows are stored in a serialized format, it is important to ensure that the workflow code is eventually committed to Git or DFS. Without this, there is a risk that only the compiled workflow in the Domino UI remains available and that you’ll lose the syntax that built it. A good practice is to leverage this iterative flexibility while maintaining awareness of when a workflow has reached a stable state that should be preserved in version control. Regularly syncing workflow code with Git ensures that development remains traceable and reproducible.

## Relaunch an existing flow

After a flow has been triggered for the first time, it can be relaunched directly through the UI. To relaunch an existing flow:

1. Navigate to **Flows** > **Flow name** > **Flow Version** > **Run name** and click **Re-run** from the top-right corner of the page.

   <Frame>
     <img alt="Rerun Flow" />
   </Frame>

2. Provide a name for the new run and choose whether to overwrite caching. For more details on caching, see the [Advanced Flows](/cloud/platform-capabilities/features/flows/advanced-flows) section.

   <Frame>
     <img alt="Configure run details in the first step of the rerun wizard" />
   </Frame>

3. Choose whether to reuse the latest inputs or enter new input parameters, then click **Launch Run** to trigger the relaunch.

   <Frame>
     <img alt="Customize workflow inputs as part of the new run" />
   </Frame>

4. Override task-specific settings such as the Domino environment, hardware tier or Git reference for the task code. Overrides can be configured for each individual task. When a Git override is used, the files used as task inputs must exist in the new Git reference.

   <Frame>
     <img alt="Customize task specific parameters" />
   </Frame>

<Note>
  When relaunching a Flow, the behavior of floating `git` references specified using `GitRef()` depend on the `use_latest_git_ref` argument to `DominoJobTask()`. See [Advanced Flows](/cloud/platform-capabilities/features/flows/advanced-flows) for further details.
</Note>

### Recover from failure

If a task fails due to unexpected transient issues (job timed out, out-of-capacity issues, spot instance pre-emption, internal server error, etc.), it is possible to continue execution from the point of failure rather than restarting the entire flow execution from the beginning.

When this happens, you can click the **Recover** button of the execution view in the Domino Flows UI. This resumes the execution from the point of failure by re-using the outputs that have already been computed.

<img alt="Recover From Failure" />

<Note>
  This capability is meant for transient issues only. If an execution has failed due to a user error in the code, you must relaunch the flow from the CLI after making your changes so that a new version of the code can be registered.
</Note>

### Rerun a single node

During experimentation, you may be focused on specific parts of the flow and want to execute just a single node with different input parameters. To do this in Flows:

1. Navigate to the relevant node in the execution view of the Domino UI, and click **Rerun Node**.

   <img alt="Node Rerun Button" />

2. Provide a name for the new run and choose whether to overwrite caching. For more details on caching, see the [Advanced Flows](/cloud/platform-capabilities/features/flows/advanced-flows) section.

   <img alt="Node Rerun Modal Step 1" />

3. Choose whether to reuse the latest inputs or enter new input parameters. Then click **Launch Run** to trigger the relaunch of the node.

   <img alt="Node Rerun Modal Step 2" />

<Note>
  Relaunching flows through the UI only works in scenarios where the flow definition has not changed. If changes are being made to the original flow definition, they must be launched through the CLI so that a new version can be registered.
</Note>

## Next steps

Once you have properly launched the flow, learn how to [monitor flow results](/cloud/platform-capabilities/features/flows/monitor-flows).


# Monitor Flows
Source: https://docs.domino.ai/cloud/platform-capabilities/features/flows/monitor-flows

Monitor Flows, tasks, runs, nodes, inputs, outputs, logs, lineage, and execution graphs in the Domino and native Flyte interfaces.

Domino Flows comes with a comprehensive user experience to monitor flows. This includes an integrated Domino UI, along with access to the native Flyte UI.

This document goes through some of the available monitoring experiences.

## Domino UI

The Domino Flows UI can be accessed by navigating to the **Flows** page within your Project view. This section documents some of the key experiences that are available on this page.

### Flow and task list

Upon landing in the Flows UI, you have access to a list of all the flows that have been registered into the Project. In this view, you can:

* Keep track of and monitor all the flows in your Project.

* Click any of the flows to navigate to the details view for the flow.

* View the lineage of Flow artifacts attached to a Dataset.

* Filter between active/archived flows or search for flows by name.

* Change the number of entries and columns to display in the table.

<img alt="Flow List" />

Similarly, clicking on the `Flow Tasks` pivot navigates you to a similar view but at the task level. This view is an aggregate list of tasks from across all flows registered in this project.

<img alt="Task List" />

### Flow and task details

Clicking onto a flow navigates you to its details view. In this view, you can:

* Run the flow.

* Visualize the flow structure by clicking on the **Graph** pivot.

* Keep track of and monitor the status of all the runs for that flow.

* Filter between runs of a specific version or view runs of all versions.

* Click any of the runs to navigate to a run details page.

* Rerun or archive a run by clicking on the overflow button (right side of a row).

* Filter between active/archived runs or search for runs by name.

* Change the number of entries and columns to display in the table.

* Archive the flow (overflow button on the top right corner).

<img alt="Run List" />

Clicking on a task navigates you to a similar view but at the task level.

<img alt="Run List" />

### Run details

To see more details about a run, click the relevant run name to access the **Run details** page. In this view, you can:

* View a list of all the nodes along with their execution status.

* Click into a node to view more details about the results.

* Toggle to a graph view of the run details by clicking on the **Graph** pivot.

* Rerun or archive the run by clicking on the overflow button (top right).

<img alt="Node List" />

### Node details

To see more details about a node, click the relevant node name, either via the **Nodes** or **Graph** pivot, to access the **Node details** panel. In this view, you can:

* Visualize all the versioned inputs that went into the node.

  <img alt="Node Inputs" />

* Visualize all the versioned outputs that came out of the node.

  <img alt="Node Outputs" />

* Access execution details about the node, including a link to the corresponding Domino Job.

  <img alt="Node Details" />

* Access execution logs.

  <img alt="Node Logs" />

<Note>
  To enable file rendering, you must explicitly specify the file type when you define the input or output. See [defining flows](/cloud/platform-capabilities/features/flows/define-flows) for more details.
</Note>

### Lineage details

Once you register a Flow Artifact to a Dataset, you can view its lineage through the **Details** pane. You can trace the lineage path throughout the Dataset by clicking the links at the bottom.

<img alt="View Data Lineage" />

### Graph view

Click into the **Graph** pivot to access a graph view of the execution. This helps validate that the execution flow and dependencies are defined correctly, in addition to narrowing the root cause of errors.

<img alt="Graph View" />

### Launch inputs view

Click into the **Launch Inputs** pivot to view the list of the flow run’s launch parameters. Clicking into any individual launch input displays the value that was set for that run.

<img alt="Launch Inputs View" />

## Flyte UI

The Flyte UI can be accessed directly at `https://{DOMINO-URL}/flows/console/select-project`. Within the Flyte UI, you can search for a specific Project or directly click into any of the other pages on the left. There are also several entry points from the Domino UI that take you directly to specific pages in the Flyte experience.

<Note>
  There is a 1:1 mapping between Projects in Flyte and Domino.
</Note>

<img alt="Flyte Landing Page" />

The Flyte UI offers many of the same capabilities as the Domino UI but also includes some additional, useful features. This section documents some of the key additional views that can help to monitor flows.

<Note>
  Not all native Flyte capabilities are fully supported, so there may be some actions in the Flyte UI that don’t currently work. Additional capabilities are planned for future releases.
</Note>

### Timeline view

Click into any execution in the Flyte UI to access a timeline view of the flow. This helps identify execution bottlenecks and to know what the critical execution path is.

<img alt="Timeline View" />

Hovering over tasks in the timeline view allows you to see an even more granular breakdown.

<img alt="Timeline View - Hover" />

## Flyte output Blob URIs

The Domino UI renders output files for each task and allows you to download them to your local machine. If you need to access the Flyte blob directly, you can find the URI for each blob in the Flyte UI.

<img alt="Flyte Output Blob URI" />

## Next steps

* [Examine Flow Artifacts.](/cloud/platform-capabilities/features/flows/examine-flow-artifacts)

* Maximize the [reproducibility](/cloud/platform-capabilities/features/flows/reproducibility-flows) of Flows.


# Flows reproducibility
Source: https://docs.domino.ai/cloud/platform-capabilities/features/flows/reproducibility-flows

Preserve reproducible Flows through serialized definitions, pinned Job settings, versioned data, explicit task outputs, and immutable inputs.

Every flow, task, and execution are uniquely versioned in Domino Flows to guarantee reproducibility.

Each time a flow is executed through the CLI, the exact definition is serialized and uploaded to be stored in the Flyte blob storage. This includes:

* The structure of the flow and tasks as defined in the function wrapped by the `@workflow` decorator at the time of execution. Implicitly, this automatically captures the exact code definition for the flow at the time of execution, without the need to make a manual commit.

* The Domino Job config that is defined as part of each task and includes a precise code commit, environment version, hardware tier, dataset snapshots, and all other properties that were defined as part of a job.

* The input and output parameters that can be configured or modified for each execution.

Whenever the serialized definition of the flow/tasks do not match a previously registered version, a new version of the entity is created and is tied to every execution that is instantiated from it. This is what ensures that all re-executions of a flow with the same parameters always produce consistent results.

## Best practices

While the automatic versioning of entities in flow helps to maximize reproducibility, there are additional practices that are recommended to guarantee it. More specifically:

* When the `use_latest` parameter is set when defining tasks, Domino uses project defaults or the latest versions of Domino entities for parameters that have not been explicitly defined at the time of registration. While this is useful for quick prototyping, it is recommended to explicitly define all parameters before putting a flow into production. This ensures that if you make changes to your flow definition in the future, you do not accidentally pick up changed project default settings when registering your new flow version.

Note that `use_latest` in this example is useful for development and iteration:

```python theme={null}
DominoJobTask(
    name='Add numbers',
    domino_job_config=DominoJobConfig(Command="python add.py"),
    inputs={'first_value': int, 'second_value': int},
    outputs={'sum': int},
    use_latest=True
)
```

A better definition for production use omits `use_latest` in favor of explicitly defining all parameters:

```python theme={null}
DominoJobTask(
    name='Add numbers',
    domino_job_config=DominoJobConfig(
        Command="python add.py",
        CommitId="4e6f2ee71e3bd64eaa90ce826c7d523b29f179cd",
        MainRepoGitRef=None,
        HardwareTierId="large-k8s",
        EnvironmentId="66b5324c495c6c124cfb0a28",
        EnvironmentRevisionSpec=EnvironmentRevisionSpecification(EnvironmentRevisionType.SomeRevision, "4"),
        ComputeClusterProperties=None,
        VolumeSizeGiB=10,
        DatasetSnapshots=[
            DatasetSnapshot(Name="quick-start", Version=1)
        ],
        ExternalVolumeMountIds=[],
    ),
    inputs={'first_value': int, 'second_value': int},
    outputs={'sum': int}
)
```

* When loading data from external data sources, it is recommended to have an initial task for making a snapshot of the data (that is, a task that loads the data into Domino and copies it directly as an output). This ensures that changes in external data sources won’t affect the ability to reproduce a result.

* Always write results as explicit outputs for the task, rather than to an external data location.

## Next steps

* [Advanced Flows](/cloud/platform-capabilities/features/flows/advanced-flows)


# Governance
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/index

How Domino governs work on the platform: bundles connect the work in your Projects to policies that define the evidence, approvals, and gates your organization requires.

Governance is how you implement your organization's validation, review, and compliance processes inside Domino. Because reproducibility, auditability, and lineage are built into the platform's core, governance works alongside development instead of interrupting it: builders keep working in their Projects while Domino collects evidence, tracks approvals, and records what happened.

Governance is not tied to any one kind of work or process. Anything developed in a [Project](/cloud/platform-capabilities/core-concepts/projects) can be governed: models, Apps, API endpoints, files, reports, and agents. What compliance means is defined entirely by your policies, whether they reflect an internal review checklist, a corporate standard, or a regulatory framework.

Policy-based governance is one facet of Domino's Governance Center. The others are [FinOps](/cloud/platform-capabilities/features/finops), for managing the cost of work on the platform, and the [Audit Trail](/cloud/platform-capabilities/features/governance/audit-trail), which records actions taken across the platform.

## How it works

To govern anything in Domino, it must be in a bundle, and every bundle originates in a Project. Policies applied to the bundle define what compliance looks like, and Domino tracks the bundle against them:

* **Bundles** hold the materials or work items you want to govern as a group, such as models, files, and Apps.
* **Policies** define the governance lifecycle a bundle must go through: all stages, evidence, and approvals required for compliance. Policies are fully customizable, so they can reflect your internal compliance processes (a "Data Privacy Policy" or "Infosec Policy") as well as industry frameworks ("NIST AI RMF" or "SR 11-7"). A bundle can carry multiple policies, and a policy can be reused across bundles.
* **Stages** are the phases you expect work to go through, for example Use Case Definition, Data Selection, Deployment, and Ongoing Monitoring.
* **Evidence** is the information each stage requires, collected as static responses, attachments, or computed results. Domino can [generate much of it automatically](/cloud/platform-capabilities/features/governance/evidence/automated-evidence-collection).
* **Approvals** specify who must sign off on the provided evidence before a stage is complete. Approvals can also control transitions between stages.
* **Gates** enforce that certain actions, such as deploying an App or Agent or using restricted hardware tiers, cannot be performed until the required approvals are complete.
* **Findings** document issues discovered while reviewing evidence, so they can be tracked to resolution.

<Frame>
  <img alt="A governed bundle: Models, Files, and Data feed into a Bundle, which carries attached policies (NIST AI RMF, SR 11-7, Model Risk Management) and moves through a policy workflow of Ideation, Development, Review, and Retirement." />

  <img alt="A governed bundle: Models, Files, and Data feed into a Bundle, which carries attached policies (NIST AI RMF, SR 11-7, Model Risk Management) and moves through a policy workflow of Ideation, Development, Review, and Retirement." />
</Frame>

Governance is not a one-time review. Approvals can be configured to expire and renew on a recurring schedule ([periodic revalidation](/cloud/platform-capabilities/features/governance/approvals#periodic-revalidation)), so every governed entity is resurfaced for review on the cadence your organization requires.

```mermaid theme={null}
flowchart LR
    project["Project"] -->|originates| bundle["Governed bundle:<br/>models, Apps, files, agents"]
    policies["Policies:<br/>stages, evidence, approvals"] -.-> bundle
    bundle -->|approvals complete| gated["Gated actions:<br/>App and Agent deployment,<br/>hardware tiers"]
    gated -->|periodic revalidation| bundle
    gated ~~~ spacer[" "]
    style spacer fill:none,stroke:none
```

<Frame>
  <img alt="The governance policy workflow: a Governance Administrator creates policies, a Practitioner answers evidence, and an Approver verifies it, producing approvals or findings across three policy stages before gating platform actions like App and agent deployment, model deployment, and restricted GPU usage." />

  <img alt="The governance policy workflow: a Governance Administrator creates policies, a Practitioner answers evidence, and an Approver verifies it, producing approvals or findings across three policy stages before gating platform actions like App and agent deployment, model deployment, and restricted GPU usage." />
</Frame>

### Who does what

Governance uses role-based access built on existing Project permissions, plus one new global role:

* **Practitioners** create bundles, add artifacts and evidence, and submit bundles for review, all through their existing Project permissions.
* **Approvers** review evidence, approve stages, and create findings to document issues.
* **Governance Administrators** (the **GovernanceAdmin** global role) create, edit, and publish policies, and have access to compliance views, the Audit Trail, and Governance APIs.

See [Roles and security](/cloud/platform-capabilities/features/governance/roles-and-security) for the full permission matrix and role assignment.

### Platform-wide accountability

Beyond bundle reviews, the [Audit Trail](/cloud/platform-capabilities/features/governance/audit-trail) records user activity across your entire Domino deployment for compliance verification, security investigations, and audit reporting, with purpose-built UI views and APIs. Domino retains audit data for up to 30 years, depending on your deployment's retention policy.

## Work with Governance

* [Work with bundles](/cloud/platform-capabilities/features/governance/work-with-bundles): Connect assets to policies and start the governance workflow.
* [Define policies](/cloud/platform-capabilities/features/governance/define-governance-policies): Create and configure policies (for Governance Administrators).
* [Evidence](/cloud/platform-capabilities/features/governance/evidence): Configure manual and automated evidence collection.
* [Approvals](/cloud/platform-capabilities/features/governance/approvals): Define who must sign off, and when.
* [Gating](/cloud/platform-capabilities/features/governance/gating-domino-governance): Block deployments until approvals are complete.
* [Roles and security](/cloud/platform-capabilities/features/governance/roles-and-security): Understand permissions and role assignment.
* [Audit Trail](/cloud/platform-capabilities/features/governance/audit-trail): Review the log of user and system actions across the platform.
* [FinOps](/cloud/platform-capabilities/features/finops): Manage and attribute the cost of work on the platform.
* [Governance glossary](/cloud/platform-capabilities/features/governance/governance-glossary): The full governance vocabulary.


# Add findings to a bundle
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/work-with-bundles/add-findings-to-bundle

Add findings to governed bundles, assign severity, owners, and due dates, then track comments and resolution through Governance tasks.

As bundles progress through the governance lifecycle and evidence is collected, **Findings** document issues, review notes, and follow-up actions. Each finding assigns a concern to someone for resolution within a defined timeframe. Contributors can add findings at any stage. Approvers can add findings to stages they haven’t approved yet.

Each finding is tied to a specific approval stage and can include supporting evidence. Multiple findings can be added to the same bundle. Findings help teams:

* **Identify risks** such as bias, security gaps, or regulatory non-compliance.

* **Evaluate compliance** with governance standards and ethical guidelines.

* **Suggest improvements** to enhance performance, fairness, or accountability.

Assigned tasks help contributors respond to findings and keep the review process moving forward. Review status options are:

* **Approved**: Grant full approval with no conditions.

* **Conditional approval**: Grant approval with attached findings. Approval is automatically revoked if the findings are not resolved by their due dates.

* **Reject**: Decline approval and document the findings that must be addressed.

## Add findings

Access the governed bundle directly to add or view findings:

1. Go to **Projects** and open the relevant project.

2. From the sidebar, select **Govern** > **Bundles**.

3. Choose the bundle where you want to add or view findings.

<img alt="Find bundle list" />

You can add findings from two tabs:

* **Evidence**: Scroll down and click **Add Finding**.

* **Findings**: Click an existing finding to view, or click **Add Finding** to add a new one.

Fill out the required fields, including the finding name, approval level, severity, assignee, and approver.

<img alt="Add finding" />

You can also add an optional description and due date. Repeat as needed for additional findings. Click **Add Finding** when finished.

## Address findings from tasks

If you’re assigned a task related to a finding:

1. Go to **Projects**, open the relevant project, and select **Govern** > **Tasks**.

2. Click the task to open it, then click the finding link in the task description to view details.

3. On the **Findings** tab, check the **Status** menu to see the current stage.

4. Leave a comment to provide information or context. You can use Markdown and MathJax in your comment, or @mention people.

5. Click **Send** when finished.

## Next steps

* [Review and approve bundles](/cloud/platform-capabilities/features/governance/work-with-bundles/review-bundles): submit bundles for formal approval

* [Evidence](/cloud/platform-capabilities/features/governance/evidence): learn about manual evidence, automated checks, and monitoring checks

* [Gates](/cloud/platform-capabilities/features/governance/gating-domino-governance): control governed actions based on approval status


# Attachments
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/work-with-bundles/attachments

Distinguish linked models, endpoints, Apps, Agents, and files from supporting documents, Dataset assets, and NetApp Volume evidence in bundles.

Attachments are the technical assets and artifacts linked to a bundle. They provide the objective proof required for stakeholders to validate and approve a project.

## Linked assets

These are the primary objects being governed. An asset appears here once you’ve associated it with the bundle from its page in Domino:

* Models and model endpoint versions

* Apps

* Agents

* Files

## Supporting evidence

These artifacts provide the technical context and justification for the responses in the Evidence notebook:

* **Files**: Uploaded PDFs, validation reports, or whitepapers

* **Data**: Domino datasets files or snapshots

* **NetApp Volumes**: Files or snapshots

## Next steps

* [Govern models, apps, agents, and files](/cloud/platform-capabilities/features/governance/work-with-bundles/govern-models-apps-files): link assets to a bundle for review

* [Use the Evidence notebook](/cloud/platform-capabilities/features/governance/work-with-bundles/use-evidence-notebook): document work and collect evidence

* [Add findings to a bundle](/cloud/platform-capabilities/features/governance/work-with-bundles/add-findings-to-bundle): document issues and track resolution


# Govern models, apps, agents, and files
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/work-with-bundles/govern-models-apps-files

Link models, Apps, Agents, and files to governed bundles, collect evidence and approvals, and compare evidence across App versions.

Governing an asset links it to a bundle, subjecting it to your organization’s review and approval workflows. Depending on the asset type, governance may be required before deployment, publishing, or use in validated workflows.

## Governance workflow

The workflow is the same for all asset types:

1. **Link the asset:** Navigate to the asset and click **Add to Bundle** to link it to a new or existing bundle. Confirm the link in the bundle’s **Attachments** section.

2. **Collect evidence:** Provide the documentation and artifacts required by your policies. Evidence types vary by asset.

3. **Obtain approvals:** Request reviewer approvals to reach compliance.

### Models

From the **Model** card or Model endpoint version page, click **Add to Bundle** to link the model.

For models, you can use two specialized evidence types:

* **Model metrics**: Capture performance data such as accuracy, RMSE, and custom KPIs.

* **Metrics checks**: Define pass/fail thresholds that automatically validate performance and auto-create findings.

<Tip>
  You can use gates to block Model endpoint deployment when a bundle is non-compliant.
</Tip>

### Apps and agents

From the app or agent page, click **Add to Bundle** to link the asset. You can attach a policy at any point in an app's lifecycle, not only at creation.

For apps and agents, evidence typically covers security and data privacy:

* **Scripted checks** Run automated tests to verify dependencies, check for vulnerabilities, or validate data access permissions.

* **Manual evidence**: Provide screenshots, UAT sign-offs, or data source documentation.

<Tip>
  You can use gates to block deployment of a published App or an agent when a bundle is non-compliant.
</Tip>

#### How bundles track app versions

Each governed app version maps 1:1 to a bundle. Only the latest app version holds a read-write bundle; bundles attached to earlier versions become evidence read-only. Approvals, findings, and attachments on those earlier bundles remain editable.

When you create a new version of a governed app, Domino:

* Snapshots the previous version's bundle by copying its evidence, attachments, and approvals into a new read-only bundle attached to the previous version.

* Attaches the read-write bundle to the new version, so governance carries forward without re-collecting evidence.

Findings and comments are not copied to the snapshot, to avoid duplication.

#### Compare evidence across app versions

Open a governed app and go to **Governance** > **Changes** to see how evidence has changed between versions. The **Changes** tab compares evidence for the same policy across each version's bundle and shows what was added, removed, or modified.

* If a policy was upgraded between versions, the tab still pairs the two bundles on the shared policy and marks any modified questions as added, removed, or changed.

* If a policy is present in one version and missing in the other, every question in that policy appears as added or removed, depending on the direction of the comparison.

### Files

From the **Code** section of your Project, locate the file. Click the three dot-menu, then select **Add to Bundle**. Confirm the link in the bundle’s **Attachments** section.

For files, evidence typically focuses on validation and traceability. This is particularly relevant in Life Sciences, where scripts, data pipelines, and clinical reports must meet GxP standards before use in validated studies or production workflows.

* **Scripted checks**: Run automated tests to verify data integrity, perform checksums, or validate formatting standards such as CDISC.

* **Manual evidence**: Upload code review logs, validation plans, or QA reports.

## Next steps

* [Use the Evidence notebook](/cloud/platform-capabilities/features/governance/work-with-bundles/use-evidence-notebook): document work and collect evidence

* [Add findings to a bundle](/cloud/platform-capabilities/features/governance/work-with-bundles/add-findings-to-bundle): document issues and track resolution

* [Review and approve bundles](/cloud/platform-capabilities/features/governance/work-with-bundles/review-bundles): submit bundles for formal approval


# Work with bundles
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/work-with-bundles/index

Create governed bundles that connect assets with policies, evidence, attachments, findings, and approvals, then track policy and bundle compliance.

**Bundles** connect assets that need governance with the policies that govern them. Bundles let you contribute, collect, and review evidence to make sure assets achieve and maintain policy compliance.

Each bundle includes the following components:

* One or more policies

* Evidence for each policy

* Attachments

* Findings

* Approvals

Bundles originate in a Domino Project and track its governable materials.

When you add a policy to a bundle, Domino creates an **Evidence** notebook where modelers, reviewers, and validators collaborate to achieve compliance.

The [Evidence notebook](/cloud/platform-capabilities/features/governance/work-with-bundles/use-evidence-notebook) supports different types of evidence depending on the policy definition. When you identify issues with evidence, you can create Findings to document and track them. You can also add **Attachments**, which are supporting evidence in the form of files, data, or other project items. When sufficient evidence is gathered, you can request and obtain approval.

## Bundle compliance

In Domino, compliance is a status derived from the state of a bundle’s underlying policies. Domino calculates it as follows:

* **Policy compliance**: A policy is compliant when you’ve obtained every stage approval up to the policy’s current stage.

* **Bundle compliance**: A bundle is compliant when all its policies are compliant.

Once all requirements are met, you can mark a bundle Completed or Archived. Both are terminal states that freeze the Evidence notebook.

## Create a bundle

You can create a bundle directly from a Domino Project to start the formal review process.

<img alt="Create a bundle and add artifacts" />

1. Open your Project and select **Govern** > **Bundles** from the left navigation.

2. Click **Create Bundle** and provide a name and summary that describes the purpose of your model, app, or agent.

3. Select one or more policies that correspond to your project’s risk profile or regulatory requirements, then click **Create**.

Domino creates your Evidence notebook and populates the stages and approval requirements defined by your organization. You can now begin documenting your work for review.

## Next steps

* [Govern models, apps, agents, and files](/cloud/platform-capabilities/features/governance/work-with-bundles/govern-models-apps-files): link assets to a bundle for review

* [Use the Evidence notebook](/cloud/platform-capabilities/features/governance/work-with-bundles/use-evidence-notebook): document work and collect evidence

* [Attachments](/cloud/platform-capabilities/features/governance/work-with-bundles/attachments): distinguish linked models, endpoints, Apps, Agents, and files from supporting documents, Dataset assets, and NetApp Volume evidence

* [Add findings to a bundle](/cloud/platform-capabilities/features/governance/work-with-bundles/add-findings-to-bundle): document issues and track resolution

* [Review and approve bundles](/cloud/platform-capabilities/features/governance/work-with-bundles/review-bundles): submit bundles for review, approve or reject stages, and grant conditional approval with findings


# Review and approve bundles
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/work-with-bundles/review-bundles

Submit governed bundles for review, approve or reject stages, grant conditional approval with findings, and reset sequential review workflows.

After evidence is collected and findings are documented, bundles are submitted for formal stakeholder review. Practitioners can submit bundles for review at any stage, and approvers are notified when their input is needed.

A single bundle can now include multiple policies, allowing related governance requirements to be reviewed together. Approval flows also support per-bundle customization, so different bundles can have distinct approvers, even under the same policy.

Each approval stage is defined by the Governance administrator and may vary by policy type. Approvers can approve, reject, or grant conditional approval pending resolution of findings or required changes.

## How approvals work

During a bundle review, approvers can document findings to capture and track issues that block compliance. [Add findings to a governed bundle](/cloud/platform-capabilities/features/governance/work-with-bundles/add-findings-to-bundle) has details on findings attributes and workflows.

These attributes help drive progress toward production readiness. If a bundle includes multiple policies, review stages for each policy are grouped in the same interface to streamline approvals across overlapping requirements. An example approval flow might look something like this:

<img alt="Example of a review flow" />

[Findings](/cloud/platform-capabilities/features/governance/work-with-bundles/add-findings-to-bundle) are tied to approval stages and can include supporting evidence.

Once all findings are resolved, the approver marks the stage as **Stages Complete** to finalize the review.

### How conditional approval works

A reviewer can approve a stage conditionally by attaching findings to the approval. These findings form a remediation plan.

If all findings are resolved by their due dates, the approval remains valid. If any findings are not resolved on time, the approval is automatically revoked.

Conditional approval allows practitioners to proceed to the next stage while outstanding findings are tracked and addressed.

## Submit a bundle for review (practitioners)

After [creating a governed bundle](/cloud/platform-capabilities/features/governance/work-with-bundles) or editing an existing one, you can submit it for stakeholder review. This is typically done after completing requirements for each stage.

<img alt="Start an evidence review" />

1. Go to **Project**, open the relevant project, and select **Govern** > **Bundles** from the sidebar.

2. Select the bundle that you want to send for review.

3. On **Stage 1**, fill in the required information and proceed through each stage.

4. When you reach a stage with **Approval required**, click **Start an evidence review**.

5. Complete all required fields and click **Request Review**.

6. To confirm your submission, go to the **Domino Welcome** page and verify that a review task was created.

## Review and respond to a bundle (approvers)

When a practitioner submits a bundle for review, you receive an email notification. Select **Review & Approve** in the email to open the Evidence notebook, which contains the approval questionnaire for the bundle.

1. Complete the approval questionnaire, if one exists.

2. Optionally, select **Add Comment** or **Add Finding** to document issues.

3. Select **Approve**. If there are open findings, a dialog displays them. Selecting a finding changes the button to **Conditionally approve**.

   * To grant conditional approval, select the findings to attach as conditions, then select **Conditionally approve**.

   * To grant full approval, select **Approve** without selecting any findings.

**Reject**: Decline approval and document the findings that must be addressed.

## Bundle reset

Reviewers can reject a bundle and reset it back to the first stage when a fundamental error is identified during review. This is distinct from withholding approval: a reset invalidates the current review cycle and requires work to restart from the beginning.

### Prerequisites

Bundle reset is only available for bundles whose policy has sequential stage ordering enabled. Bundles with non-sequential policies do not support this workflow.

### What happens on reset

* **Evidence:** All answers to evidence questions are cleared and must be re-entered.

* **Approvals:** All pending and approved approval requests are revoked. Stage assignees are also cleared and must be reassigned.

* **Findings:** All findings are preserved and remain visible in the Findings list and alongside their associated evidence.

* **Comments:** All comments are retained and remain visible, even on cleared evidence fields.

* **Attachments:** During the reset, reviewers can choose to remove one or more attachments. Any attachments not selected for removal are carried forward to the reset bundle.

### Audit Trail

Every reset is captured as an immutable event in the bundle’s History tab, recording the rejecting reviewer, timestamp, reason, and the state of attachments at the time of reset. This lineage is available to auditors at any time.

<img alt="Reset bundle" />

## Next steps

* [Gates](/cloud/platform-capabilities/features/governance/gating-domino-governance): control governed actions based on approval status.

* [Add findings to a bundle](/cloud/platform-capabilities/features/governance/work-with-bundles/add-findings-to-bundle): document review decisions and track issues.

* [Work with bundles](/cloud/platform-capabilities/features/governance/work-with-bundles): package models and artifacts for review


# Use the Evidence notebook
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/work-with-bundles/use-evidence-notebook

Use the Evidence notebook to document governed work, collect and review policy evidence, grant approvals, and export an audit-ready PDF.

The **Evidence** notebook is the primary workspace for documenting work and justifying project advancement. It combines narrative context and executable code, letting practitioners generate results programmatically rather than filling out static forms.

## Core features

The **Evidence** notebook provides a unified environment for evidence collection, review, and approval. The following features support the governance workflow.

### Integrated evidence and review

The notebook is the single interface for satisfying policy requirements. Modelers, reviewers, and validators collaborate in one place to achieve compliance. Modelers answer risk questions and provide technical evidence. Reviewers and validators access the same environment to examine and approve. If a reviewer identifies a gap, they log a **finding** within the bundle, creating a feedback loop for remediation.

### Approvals and compliance

Stakeholders grant formal approvals from within the notebook. A bundle reaches a **compliant** state once you’ve secured all required approvals for the current stage.

### Completion and archival

Once all requirements are met, you can mark a bundle **Completed** or **Archived**. Both are terminal states that freeze the Evidence notebook, so proceed carefully.

## Access the evidence notebook

Domino creates the Evidence notebook when you attach a policy to a bundle.

<img alt="Evidence notebook interface" />

1. Navigate to your project’s **Govern** tab.

2. Select the relevant bundle.

3. Click a linked policy.

You can download the Evidence notebook as a PDF report for auditing and tracking purposes.

## Next steps

* [Add findings to a bundle](/cloud/platform-capabilities/features/governance/work-with-bundles/add-findings-to-bundle): document issues and track resolution

* [Review and approve bundles](/cloud/platform-capabilities/features/governance/work-with-bundles/review-bundles): submit bundles for formal approval

* [Attachments](/cloud/platform-capabilities/features/governance/work-with-bundles/attachments): link assets and supporting evidence to a bundle


# Example Python client: Asynchronous Domino endpoint
Source: https://docs.domino.ai/cloud/platform-capabilities/features/model-deployment/async-example-client

Use a Python client to submit an asynchronous prediction request, retry transient failures, poll its status, and retrieve results or errors.

This example shows a Python client application that creates a prediction request from an asynchronous Domino endpoint, polls periodically for completion, and retrieves the result.

```python theme={null}
import json
import logging
import requests
import sys
import time

logging.basicConfig(stream=sys.stdout, level=logging.INFO)  # change logging setup as required

# TO EDIT: update the example request parameters for your model
REQUEST_PARAMETERS = {
    "param1": "value1",
    "param2": "value2",
    "param3": 3
}
# TO EDIT: copy these values from "Calling your endpoint" on the Domino endpoint overview page
DOMINO_URL = "https://domino.mycompany.com:443"
MODEL_ID = "5a4131c5aad8e00eefb676b7"
MODEL_ACCESS_TOKEN = "o2pnVAqFOrQBEZMCuzt797d676E6k4eS3mZMKJVKbeid8V6Bbig6kOdh6y9YSf3R"

# DO NOT EDIT these values
MODEL_BASE_URL = f"{DOMINO_URL}/api/modelApis/async/v1/{MODEL_ID}"
SUCCEEDED_STATUS = "succeeded"
FAILED_STATUS = "failed"
QUEUED_STATUS = "queued"
TERMINAL_STATUSES = [SUCCEEDED_STATUS, FAILED_STATUS]
PENDING_STATUSES = [QUEUED_STATUS]
MAX_RETRY_DELAY_SEC = 60

### CREATE REQUEST ###

create_response = None
retry_delay_sec = 0
while (
        create_response is None
        or (500 <= create_response.status_code < 600)  # retry for transient 5xx errors
):
    # status polling with a time interval that backs off up to MAX_RETRY_DELAY_SEC
    if retry_delay_sec > 0:
        time.sleep(retry_delay_sec)
    retry_delay_sec = min(max(retry_delay_sec * 2, 1), MAX_RETRY_DELAY_SEC)

    create_response = requests.post(
        MODEL_BASE_URL,
        headers={"Authorization": f"Bearer {MODEL_ACCESS_TOKEN}"},
        json={"parameters": REQUEST_PARAMETERS}
    )

if create_response.status_code != 200:
    raise Exception(f"create prediction request failed, response: {create_response}")

prediction_id = create_response.json()["asyncPredictionId"]
logging.info(f"prediction id: {prediction_id}")

### POLL STATUS AND RETRIEVE RESULT ###

status_response = None
retry_delay_sec = 0
while (
        status_response is None
        or (500 <= status_response.status_code < 600)  # retry for transient 5xx errors
        or (status_response.status_code == 200 and status_response.json()["status"] in PENDING_STATUSES)
):
    # status polling with a time interval that backs off up to MAX_RETRY_DELAY_SEC
    if retry_delay_sec > 0:
        time.sleep(retry_delay_sec)
    retry_delay_sec = min(max(retry_delay_sec * 2, 1), MAX_RETRY_DELAY_SEC)

    status_response = requests.get(
        f"{MODEL_BASE_URL}/{prediction_id}",
        headers={"Authorization": f"Bearer {MODEL_ACCESS_TOKEN}"},
    )

if status_response.status_code != 200:
    raise Exception(f"prediction status request failed, response: {create_response}")

prediction_status = status_response.json()["status"]
if prediction_status == SUCCEEDED_STATUS:  # succeeded response includes the prediction result in "result"
    result = status_response.json()["result"]
    logging.info(f"prediction succeeded, result:
{json.dumps(result, indent = 2)}")
elif prediction_status == FAILED_STATUS:  # failed response includes the error messages in "errors"
    errors = status_response.json()["errors"]
    logging.error(f"prediction failed, errors:
{json.dumps(errors, indent = 2)}")
else:
    raise Exception(f"unexpected terminal prediction response status: {prediction_status}")
```


# Select Domino endpoint authorization mode
Source: https://docs.domino.ai/cloud/platform-capabilities/features/model-deployment/domino-endpoint-authorization

Choose unrestricted or restricted Domino endpoint access, generate tokens, and use Basic authentication for sync or Bearer tokens for async requests.

Domino offers authorization options to support various Domino endpoint applications.

## Authorize your Domino endpoint in unrestricted or restricted mode:

1. Go to the Domino endpoint’s **Settings** page and click **Invocation**.

2. Select one of the following:

   * **Unrestricted**: If your Domino endpoint is unrestricted, anyone with network access to Domino can call your Domino endpoint. No access token is required when sending a request.

   * **Restricted**: If your Domino endpoint is restricted, callers must send a valid access token with their requests to use the endpoint. See the code examples in the overview tab to see how to send access tokens with your requests.

     <Note>
       There is a difference in how to invoke synchronous and asynchronous restricted Domino endpoints:

       * Synchronous Domino endpoints require the caller to use HTTP basic authentication with the username and password both set to the value of the access token.

         For example, if the token is `c2b2532ed234f54`, then the Domino endpoint must be invoked with the HTTP username `c2b2532ed234f54` and password `c2b2532ed234f54`.

       * Asynchronous Domino endpoints require the caller to use HTTP bearer (or token) authentication with the value of the access token.

         For example, if the token is `c2b2532ed234f54`, then the Domino endpoint must be invoked with the HTTP header `Authorization: Bearer c2b2532ed234f54`.
     </Note>

3. Click **Add Model Access Token** to generate access tokens for your Domino endpoint. You can generate as many access tokens as you want. Use the name field to help you track which tokens have been issued, to whom, and for what purpose.

   To revoke or regenerate a token, click the gear button.


# Deploy Domino endpoints
Source: https://docs.domino.ai/cloud/platform-capabilities/features/model-deployment/index

Deploy synchronous or asynchronous Domino endpoints from the UI, scheduled Jobs, or API, package model assets, and request predictions.

Domino makes it easy to deploy your model as an HTTP endpoint. Domino streamlines deployment so you can start sending HTTP requests to your endpoint to get predictions from your models quickly. Learn how to:

* Deploy a model for synchronous inference (real-time).

* Deploy a model for asynchronous inference.

* Access Domino assets in a Domino endpoint image.

* Request predictions from a Domino endpoint.

* Retrain a model.

* See an example of an asynchronous inference application.

## How deploy works

When you deploy a model as a Domino endpoint, Domino packages the project’s files as a Flask application. By default, Domino copies all project files into the endpoint image, including the compute environment, project files, a Flask/Plumber harness that exposes the HTTP interface, and an authentication and load balancing layer.

A Domino endpoint is an HTTP API endpoint wrapped around your inference code. You supply the arguments to your inference code as parameters in the HTTP request payload, and the response from the endpoint includes the prediction. When a Domino Endpoint is published, Domino runs the script that contains the function. As the process waits for input, any objects or functions from the script remain in memory. Each request sent to the endpoint runs the function. Since the script is only sourced at publish time, expensive initialization processes occur only once rather than on each request.

## Synchronous vs asynchronous

Decide if your model needs to be made available as a synchronous or asynchronous Domino endpoint.

**Synchronous inference**\
Deploy the model so it can receive an HTTP request, make a prediction, and return results synchronously with low latency. This is useful for interactive applications where predictions must be made synchronously.

**Asynchronous inference**\
Deploy the model so it can process and make predictions asynchronously for computationally intensive workloads. The asynchronous HTTP interface polls for results after a request is queued. After completion, the endpoint returns the result as a response to the polling request.

Domino limits the size of the request and response packets to 10 KB, so users are expected to send and receive the payloads by reference. After Domino picks up the processing request, it keeps the request alive for 30 minutes. Predictions are written to an external data store that you define.

## Package requirements

<Tabs>
  <Tab title="Python">
    To support the Flask harness that exposes a model written in Python, the following packages are required in your endpoint image:

    * uWSGI

    * Flask

    * Six

    * Prometheus-client

    Domino returns an error if these packages are not found in your compute environment.

    Additionally, asynchronous Domino endpoints require the `seldon-core` package.

    When your model is registered with Domino’s Model Registry feature, the `mlflow` Python package attempts to infer package requirements for your model and write them to a `requirements.txt` file. Domino installs these requirements during the build process. At the moment, no other `requirements.txt` files are installed when endpoint images are built.

    All of the required packages are pre-installed in all the flavors of Domino’s Compute Environment images, except the minimal image, which does not have `seldon-core` or `mlflow`.
  </Tab>

  <Tab title="R">
    To support the Plumber harness that exposes a model written in R, the following packages are required in your endpoint image:

    * plumber

    * future

    * openmetrics

    Domino tries to install these packages from CRAN if they are not found in your compute environment.

    Domino does not currently support publishing asynchronous endpoints in R or registering them with the Model Registry feature, so there is no requirement for the `seldon-core` or `mlflow` packages.
  </Tab>
</Tabs>

## Deploy a model

Domino supports a few different methods for deploying a model as a Domino endpoint:

* **Deploy from the UI** - For quick, one time deployments.

* **Deploy with a scheduled job** - Schedule a job and deploy a model from the job. Especially useful for keeping models trained on the latest data on a regular cadence.

* **Deploy with the API** - Use the Domino API to schedule jobs from other applications.

### Deploy from the UI

Use the Domino UI to deploy your model directly from your browser.

1. In your project, go to **Deployments > Endpoints > Create Domino Endpoint**.

2. Provide a name and description, and select the prediction code that executes when the model is called.

3. (Optional) Select a custom compute environment to build the deployed model container.

4. (Optional) Configure scaling. The number of instances and compute resources attached to each instance. See [Scale model deployments](/cloud/admin/platform-configuration/manage-model-deployments#scale-domino-endpoints) for more information.

   <Note>
     You can also include a GPU if your administrator configured GPUs in your node pool.
   </Note>

5. Under **Request Type**, select **Sync** for real-time predictions or **Async** for long-running predictions.

### Update from a scheduled Job

You can use [scheduled jobs](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job) to update an existing endpoint. This is especially useful for keeping models trained on the latest data. Select the Domino endpoint from the **Update Domino Endpoint** field when scheduling a job. This setting uses the state of the project’s files after the run to build and deploy a new version of an existing Domino endpoint.

You can use automated deploy jobs to automatically keep your Domino endpoint up to date by using a training script in the job that trains on fresh data.

### Deploy with the Domino API

Use Domino’s APIs to programmatically build and deploy models. For more information, see [`createModelApi` in the Domino Platform API reference](/api-reference/model-apis/post-apimodelservingv1modelapis).

Set the `isAsync` parameter to true in the API call to create an asynchronous inference endpoint.

## Deploy Domino endpoints on a remote Data Plane

Users may prefer to host their endpoints in a separate cluster or data plane than where the model was built, either for scalability or data proximity reasons. Domino facilitates this through its **Create Endpoint** feature, allowing models to be packaged and deployed close to where they’ll have the most impact.

When deploying Model Endpoints to a remote data plane, select the hardware tier associated with the desired local or remote data plane from the **Deployment Target** list (previously resource quotas) in the **New Model Endpoint** form.

Domino containerizes the model (MLflow or Project) as a Domino endpoint image, deploys it as an endpoint, and exposes a URL local to that remote cluster where a Nexus data plane has been deployed. Users can invoke it via the exposed URL and data movement is restricted to this remote data plane only.

**Limitations:** Currently, Domino endpoints deployed to a remote data plane do not support asynchronous requests (described below) and [integrated model monitoring](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints).

## Access Domino artifacts from endpoint images

You can still access Domino artifacts from endpoint images. However, there are nuances to each artifact type. Learn how to use Domino artifacts from your endpoint image and keep them up to date.

### Domino endpoint environments

When you deploy a Domino endpoint, select the [compute environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments) to include in the endpoint image. The environment bundles packages required by the inference script execution ahead of time.

* Domino endpoint hosts don’t read `requirements.txt` files or execute commands defined in the pre-setup, post-setup, pre-run, or post-run scripts of your environment. If your project uses `requirements.txt` or any setup scripts to install specific packages or repositories, add them to the Dockerfile instructions of your environment.

* Your Domino endpoint doesn’t inherit environment variables set at the project level. However, you can set Domino endpoint-specific environment variables on the Domino endpoint settings page. This separation decouples the management of projects and deployed models. See [Store Project credentials](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables) for more details.

* Domino endpoints run using the `uid` and `gid` of `12574` (the `domino` user). A user with this `uid` and `gid` must exist in the selected environment for the Domino endpoint. See [Create a Domino environment image](/cloud/platform-capabilities/core-concepts/compute-environments/use-external-images/manually-create-an-environment#create-the-image) for more details and instructions on how to use an existing image to create an environment.

* Domino endpoint image builds run using the `USER` directive last specified in the `Dockerfile` for the selected compute environment. Please ensure that the environment image sets the desired user for the Domino endpoint image build (for example `USER root` if you want the Domino endpoint image build to run as `root`).

### Domino endpoint Project files

Your Domino endpoint can access files from the project. Domino loads the project files onto the Domino endpoint host, similar to a Run or Workspace executor host, with a few important differences:

* Domino adds project files to the image when you build a Domino endpoint. Starting or stopping the Domino endpoint won’t change the files on the model host. To update the model host files, you need to create a new version.

* Domino pulls [Git repositories](/cloud/platform-capabilities/core-concepts/projects/dfs-projects#import-git-repositories) attached to projects when you build a Domino endpoint. Starting or stopping the Domino endpoint won’t change the files on the model host. To update the model host files, you need to create a new version.

* The Domino endpoint host mounts project files at `/mnt/<username>/<project_name>`. This location differs from the default location for Runs or Workspaces, `/mnt/`. You can use the [Domino environment variable](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables) `DOMINO_WORKING_DIR` to reference the directory where your project is mounted.

* The Domino endpoint image excludes project files listed in the `.modelignore` file that are located in the project’s root directory. Excluded files are not mounted to the Domino endpoint host.

### Add a Kubernetes volume to a synchronous Domino endpoint container

When you load inference data or write the response to an external volume, you can add Kubernetes volumes:

1. Select a Domino endpoint from the **Endpoints** page.

2. Go to **Settings** > **Advanced** > **Add Volume**.

   <Note>
     Only hostPaths are supported.
   </Note>

3. Enter the values required.

   1. **Name** - Kubernetes volume name.

   2. **Mount Path** - mount point in the Domino endpoint container.

   3. **path** - the path of the Kubernetes host node that must be mounted in the Domino endpoint container, as configured by your administrator.

   4. **Read Only?** - the read/write permission of the mounted volume.

See the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/volumes/) for more details.

## Request predictions

After you deploy the Domino endpoint, and its status changes to **Running**, try test inputs with the **Tester** tab in the Domino endpoint UI.

<Tip>
  Use the **Request** window to make calls to the Domino endpoint from the Domino web application. You find additional tabs with code samples to send requests to the Domino endpoint with other tools and in various programming languages.
</Tip>

### JSON requests

Send your requests as JSON objects. Depending on how you wrote your prediction script, you need to format the JSON request as follows:

* If you use named parameters in your function definition, use a dictionary or parameter array in your JSON request. For example, `my_function(x, y, z)`, use `{"data": {"x": 1, "y": 2, "z": 3}}` or `{"parameters": [1, 2, 3]}`.

* If you use a dictionary in your function definition, use only a parameter array. For example, `my_function(dict)` and your function then uses `dict["x"]`, `dict["y"]`. Send the request: `{"parameters": [{"x": 1, "y": 2, "z": 3}]}`.

* In Python, you can also use kwargs to pass in a variable number of arguments. If you do this: `my_function(x, **kwargs)` and your function then uses `kwargs["y"]` and `kwargs["z"]`, you can use a data dictionary to call your endpoint: `{"data": {"x": 1, "y": 2, "z": 3}}`.

Domino converts JSON data types to the following R and Python data types.

| **JSON Type** | **Python Type** | **R Type** |
| - | - | - |
| dictionary | dictionary | named list |
| array | list | list |
| string | str | character |
| number (int) | int | integer |
| number (real) | float | numeric |
| true | True | TRUE |
| false | False | FALSE |
| null | None | N/A |

The endpoint returns the result object, which is a literal, array, or dictionary.

<Tip>
  The result of your endpoint function gets serialized to JSON before being sent in the response. However, some return types aren't JSON serializable by default. If you return a NumPy object from a Python Domino endpoint, you might see `dataTypeError: don't know how to serialize class` or `TypeError: <result> is not JSON serializable`. Convert NumPy values to Python primitives before returning them (for example, call `numpy.asscalar`). For `Decimal` values, cast to `Float` instead.
</Tip>

### Synchronous requests

* **Request a prediction and retrieve the result**: Pass the URL of the Domino Domino endpoint, authorization token, and input parameters. The response object contains the status, response headers, and result.

  ```python theme={null}
  response = requests.post("{DOMINO_URL}/models/{MODEL_ID}/latest/model",
     auth=("{MODEL_ACCESS_TOKEN}", "{MODEL_ACCESS_TOKEN}"),
     json={"data": {"start": 1, "stop": 100}}
  )

  print(response.status_code)
  print(response.headers)
  print(response.json())
  ```

### Asynchronous requests

<Note>
  * Domino cannot guarantee the processing order of predictions. The order in which predictions are handled can differ from the order of requests.

  * The same prediction request may execute them more than once, so the prediction functions must be idempotent (not affected by repeated executions).

  * Use model environment variables for secrets.
</Note>

* **Request a prediction**: Pass the URL of the Domino Domino endpoint, authorization token, and input parameters. Users typically pass a reference to the payload location for large payloads (>10 KB). A prediction identifier is returned for the user to poll for completion and retrieve the result.

  ```python theme={null}
  prediction_id = requests.post(
     "{DOMINO_URL}/api/modelApis/async/v1/{MODEL_ID}",
     headers={"Authorization": f"Bearer {MODEL_ACCESS_TOKEN}"},
        json={"parameters": {
  "input_file": "s3://example/filename.ext"}})
  ```

* **Poll for completion / retrieve the results**: Use the `prediction_id` from the request to retrieve the result. The response includes one of the following statuses: `SUCCEEDED`, `FAILED`, or `QUEUED`. The results field contains the output from the predict function for a successful completion.

  ```python theme={null}
  status_response = requests.get(
      f"{MODEL_BASE_URL}/{prediction_id}",
      headers={"Authorization": f"Bearer {MODEL_ACCESS_TOKEN}"},
  )

  if prediction_status == SUCCEEDED_STATUS:  # succeeded response includes the prediction result in "result"
     result = status_response.json()["result"]
  ```

## Retrain a model

**Deploy a new version**\
After you retrain your model with new data or switch to a different machine learning algorithm, publish a new version of the Domino endpoint. To follow best practices, stop a previous version of the Domino endpoint and then deploy a new version.

<Tip>
  Models are limited to 500 MB. If your project includes a large trained model dataset, exclude it before publishing a new version; otherwise the build might fail with a storage-related error such as `IOException: Server returned HTTP response code: 500 for URL \<model_url\>`.
</Tip>

## Asynchronous example

This example shows a Python client application that creates a prediction request from an asynchronous Domino endpoint, polls periodically for completion, and retrieves the result.

```python theme={null}
import json
import logging
import requests
import sys
import time

logging.basicConfig(stream=sys.stdout, level=logging.INFO)  # change logging setup as required

# TO EDIT: update the example request parameters for your model
REQUEST_PARAMETERS = {
    "param1": "value1",
    "param2": "value2",
    "param3": 3
}
# TO EDIT: copy these values from "Calling your Model" on the Domino endpoint overview page
DOMINO_URL = "https://domino.mycompany.com:443"
MODEL_ID = "5a4131c5aad8e00eefb676b7"
MODEL_ACCESS_TOKEN = "o2pnVAqFOrQBEZMCuzt797d676E6k4eS3mZMKJVKbeid8V6Bbig6kOdh6y9YSf3R"

# DO NOT EDIT these values
MODEL_BASE_URL = f"{DOMINO_URL}/api/modelApis/async/v1/{MODEL_ID}"
SUCCEEDED_STATUS = "succeeded"
FAILED_STATUS = "failed"
QUEUED_STATUS = "queued"
TERMINAL_STATUSES = [SUCCEEDED_STATUS, FAILED_STATUS]
PENDING_STATUSES = [QUEUED_STATUS]
MAX_RETRY_DELAY_SEC = 60

### CREATE REQUEST ###

create_response = None
retry_delay_sec = 0
while (
        create_response is None
        or (500 <= create_response.status_code < 600)  # retry for transient 5xx errors
):
    # status polling with a time interval that backs off up to MAX_RETRY_DELAY_SEC
    if retry_delay_sec > 0:
        time.sleep(retry_delay_sec)
    retry_delay_sec = min(max(retry_delay_sec * 2, 1), MAX_RETRY_DELAY_SEC)

    create_response = requests.post(
        MODEL_BASE_URL,
        headers={"Authorization": f"Bearer {MODEL_ACCESS_TOKEN}"},
        json={"parameters": REQUEST_PARAMETERS}
    )

if create_response.status_code != 200:
    raise Exception(f"create prediction request failed, response: {create_response}")

prediction_id = create_response.json()["asyncPredictionId"]
logging.info(f"prediction id: {prediction_id}")

### POLL STATUS AND RETRIEVE RESULT ###

status_response = None
retry_delay_sec = 0
while (
        status_response is None
        or (500 <= status_response.status_code < 600)  # retry for transient 5xx errors
        or (status_response.status_code == 200 and status_response.json()["status"] in PENDING_STATUSES)
):
    # status polling with a time interval that backs off up to MAX_RETRY_DELAY_SEC
    if retry_delay_sec > 0:
        time.sleep(retry_delay_sec)
    retry_delay_sec = min(max(retry_delay_sec * 2, 1), MAX_RETRY_DELAY_SEC)

    status_response = requests.get(
        f"{MODEL_BASE_URL}/{prediction_id}",
        headers={"Authorization": f"Bearer {MODEL_ACCESS_TOKEN}"},
    )

if status_response.status_code != 200:
    raise Exception(f"prediction status request failed, response: {create_response}")

prediction_status = status_response.json()["status"]
if prediction_status == SUCCEEDED_STATUS:  # succeeded response includes the prediction result in "result"
    result = status_response.json()["result"]
    logging.info(f"prediction succeeded, result:
{json.dumps(result, indent = 2)}")
elif prediction_status == FAILED_STATUS:  # failed response includes the error messages in "errors"
    errors = status_response.json()["errors"]
    logging.error(f"prediction failed, errors:
{json.dumps(errors, indent = 2)}")
else:
    raise Exception(f"unexpected terminal prediction response status: {prediction_status}")
```

### Use spot instances for Domino endpoints (PREVIEW)

We support serving Domino endpoints using cost-effective Spot instances. Select a hardware tier that uses node pools using Spot instances.

If AWS interrupts a spot instance, endpoints deployed on that instance are affected and either stop responding, if there are no remaining replicas of the endpoint on other AWS instances, or the remaining replicas still running on unaffected instances experience an increased load and may have their performance degraded or also stop working, depending on the runtime characteristics of the model. If this happens, the remediation is to change the hardware tier of the endpoint to use a non-spot node pool until AWS spot instances of the requested type become available again.

## Next steps

* [Monitor model endpoints](/cloud/platform-capabilities/features/model-deployment/monitor-model-endpoints): Track performance metrics, configure health checks, and access logs for troubleshooting.

* [Access and security for Domino endpoints](/cloud/platform-capabilities/features/model-deployment/security): Control access and add contributors to Domino endpoints.

* [Scale and route Domino endpoints](/cloud/platform-capabilities/features/model-deployment/scale): Scale horizontally and vertically for optimal performance.

* [Select Domino endpoint authorization mode](/cloud/platform-capabilities/features/model-deployment/domino-endpoint-authorization): Authorize your Domino endpoint in unrestricted or restricted mode.

* [Example of a Python client application that uses an asynchronous Domino endpoint](/cloud/platform-capabilities/features/model-deployment/async-example-client).


# Monitor model endpoints
Source: https://docs.domino.ai/cloud/platform-capabilities/features/model-deployment/monitor-model-endpoints

Monitor model endpoint CPU, memory, traffic, status codes, health checks, timeouts, alerts, and logs to troubleshoot and optimize serving.

Monitor your deployed model endpoints to track performance, configure health checks, and access logs for troubleshooting. Domino provides comprehensive observability features that help you understand resource usage, detect bottlenecks, and maintain reliable model serving in production.

Use these monitoring capabilities throughout the model lifecycle, from development and testing to production deployment and ongoing optimization. Track CPU usage, memory consumption, error rates, and endpoint health directly from the Domino interface. Configure health check parameters to optimize reliability and access detailed logs to troubleshoot issues during development and production deployment.

Use these capabilities to establish performance baselines during development, validate model behavior in staging environments, and maintain production reliability through continuous monitoring and optimization.

<Note>
  All users can access basic performance metrics through the Domino endpoint interface. Domino administrators have additional access to advanced Grafana dashboards with detailed metrics, alerting, and historical analysis capabilities.
</Note>

## Access model performance metrics

View real-time and historical performance metrics for your deployed model endpoints through the Domino interface:

1. Navigate to **Endpoints** in your Domino Workspace.

2. Select the endpoint you want to monitor.

3. Click **Versions** to see all available model versions.

4. Select the specific version you want to monitor.

From the version details page, you can view:

* **CPU and memory usage**: Real-time resource consumption metrics

* **Traffic volume**: Request patterns and throughput statistics

* **Status code distribution**: Success rates and error patterns by HTTP status codes

* **Grafana dashboard link** (for Domino administrators): Direct access to advanced monitoring dashboards

If you have Domino administrator privileges, you’ll see a **Grafana** link that takes you directly to the model-specific monitoring dashboard with the model ID and version pre-selected.

<img alt="Model endpoint version page showing CPU, memory, and traffic metrics with Grafana link for admins" />

## Understand resource usage metrics

Monitor how your model consumes computational resources to identify optimization opportunities and ensure cost-effective deployments.

The Domino endpoint interface provides basic resource metrics that help you understand your model’s performance:

* **CPU usage graphs**: Visual representation of processor utilization over time

* **Memory consumption charts**: Memory usage patterns and trends

* **Traffic volume metrics**: Request rates and distribution patterns

* **Status code breakdowns**: Success rates and error patterns

For more detailed analysis and alerting capabilities, Domino administrators can access comprehensive Grafana dashboards with advanced metrics and historical data.

### Monitor workflow

Integrate monitoring into your development and deployment pipeline:

**Development phase**

* Establish baseline performance metrics during initial model testing

* Use resource metrics to right-size compute resources and optimize code

* Validate performance under realistic load conditions before deployment

**Staging and testing**

* Monitor performance regressions between model versions

* Test health check configurations and timeout settings

* Validate monitoring and alerting configurations

**Production deployment**

* Monitor deployment health and performance immediately after release

* Set up proactive alerting based on baseline metrics

* Document performance expectations and escalation procedures

**Ongoing optimization**

* Use historical trends to identify optimization opportunities

* Correlate performance metrics with business impact

* Implement continuous improvement based on monitoring insights

### CPU utilization

Track processor usage to understand computational load:

* **Normal range**: 20-60% for most models during steady load

* **Optimization needed**: Sustained usage >80% indicates inefficient code or undersized resources

* **Investigation triggers**: Sudden spikes or erratic patterns

### Memory consumption

Monitor memory usage patterns to detect leaks and optimize allocation:

* **Steady state**: Memory usage should stabilize after model initialization

* **Memory leaks**: Continuously increasing memory over time

* **Optimization opportunities**: High memory usage with low throughput

<img alt="Resource usage panels showing CPU and memory metrics with threshold indicators and trend analysis" />

### What to watch for

| Metric | Warning signs | Optimization actions |
| - | - | - |
| CPU Usage | >80% sustained usage, erratic spikes | Vectorize operations, optimize loops, implement caching |
| Memory Usage | Continuously increasing, >1 GB for simple models | Process data in batches, use generators, clear unused variables |
| Error Rate | >1% failed requests, 5xx status codes | Review model code, check input validation, monitor resource limits |

## Track errors and status codes

Monitor request success rates and identify failure patterns to ensure reliable model serving.

### HTTP status code patterns

* **2xx (Success)**: Successful predictions - target >99%

* **4xx (Client errors)**: Invalid requests, authentication failures - investigate if >1%

* **5xx (Server errors)**: Model failures, resource exhaustion - critical if >0.1%

### Common error scenarios

* **Authentication issues (401)**: Check API key configuration and permissions

* **Bad requests (400)**: Review input validation and data format requirements

* **Timeout errors (504)**: Optimize model performance or increase timeout settings

* **Resource exhaustion (503)**: Scale resources or optimize memory usage

<img alt="HTTP status code distribution chart showing success rates and error patterns" />

## Configure health checks and timeouts

Domino monitors every Domino endpoint’s health and ability to respond to new inference requests. You can adjust health check settings to optimize reliability and response behavior for your specific model requirements.

<Warning>
  When you update health check settings, the Domino endpoint automatically restarts.
</Warning>

To configure health check settings:

1. Navigate to **Endpoints** and select a Domino endpoint.

2. Go to **Settings** > **Advanced**.

3. Adjust the health check fields based on your model’s requirements and performance characteristics.

### Health check parameters

Configure these parameters based on your model’s performance characteristics and reliability requirements:

**Initial delay**\
The time (in seconds) Domino waits before a new endpoint can receive incoming requests. Set this based on your model’s initialization requirements:

* **Simple models**: 10-30 seconds

* **Complex models with large dependencies**: 60-120 seconds

* **Models requiring external resource loading**: 120+ seconds

**Health check period**\
How often (in seconds) Domino checks endpoint health. The formula **Health check period** × **Failure threshold** must be greater than the **Override request timeout**. Recommended settings:

* **High-availability models**: 10-15 seconds for quick failure detection

* **Standard models**: 30 seconds for balanced monitoring

* **Resource-intensive models**: 60+ seconds to avoid unnecessary checks

**Health check timeout**\
The time (in seconds) Domino waits before a health check request is considered as *failed*. Set this based on your model’s typical response time:

* Set to **2-3 times** your model’s P95 response time

* **Minimum recommended**: 5 seconds

* **For slow models**: 30+ seconds

**Failure threshold**\
The number of consecutive failed health checks before Domino considers the endpoint instance *unrecoverable* and restarts it. Balance between quick failure detection and avoiding unnecessary restarts:

* **Critical production models**: 2-3 failures for quick recovery

* **Development/testing**: 5+ failures to allow debugging

* **Resource-constrained environments**: 3-5 failures to balance stability

**Override request timeout**\
The time (in seconds) before Domino times out an inference request and responds with `504 Gateway Timeout`. The default is 60 seconds. Set based on your model’s expected response times:

* **Real-time inference**: 5-30 seconds

* **Batch processing models**: 60-300 seconds

* **Complex analytical models**: 300+ seconds

You must restart the endpoint for timeout setting changes to take effect.

### Timeout behavior

When requests exceed the configured timeout:

* Domino responds with HTTP 504 Gateway Timeout

* Endpoint remains available for new requests

* Consider optimizing model performance or increasing timeout for consistently slow models

## Optimize model performance

Use monitoring insights to improve your model’s efficiency and reliability.

### Performance optimization workflow

1. **Establish a baseline**: Record initial performance metrics after deployment

2. **Generate test traffic**: Use consistent load testing to identify bottlenecks

3. **Analyze patterns**: Look for resource spikes, memory growth, or latency increases

4. **Implement optimizations**: Apply code improvements based on monitoring insights

5. **Validate improvements**: Compare metrics before and after optimization

### Compare performance across model versions

One of the key advantages of Domino’s monitoring interface is the ability to easily switch between model versions and compare their performance:

1. Navigate to your endpoint and select **Versions**

2. Click between different version numbers to view their respective metrics

3. Compare CPU usage, memory consumption, traffic patterns, and error rates across versions

4. Identify performance regressions or improvements between deployments

For example, you might notice that version 2 of your model shows significantly higher CPU usage and error rates compared to version 1, indicating a performance regression that needs investigation.

**Tip:** Document baseline performance metrics for each version to track trends and quickly identify when new deployments impact performance.

### Common optimization strategies

**Reduce CPU usage**

* Use vectorized operations instead of loops

* Implement result caching for repeated computations

* Optimize data structures and algorithms

**Optimize memory usage**

* Process data in smaller batches

* Clear unused variables and intermediate results

* Use memory-efficient data structures

**Improve response times**

* Preload models and dependencies during initialization

* Implement asynchronous processing where possible

* Optimize data preprocessing pipelines

<img alt="Before optimization showing high resource usage" />

<img alt="After optimization showing improved resource efficiency" />

## Set up performance alerts

Configure alerts to proactively identify performance issues.

### Recommended alert thresholds

When working with IT administrators to set up alerts, consider these thresholds based on the basic metrics available in the Domino interface:

* **CPU usage**: Alert if >80% for more than 10 minutes

* **Memory usage**: Alert if >1 GB increase in 1 hour

* **Error rate**: Alert if >2% of requests fail

For advanced alerting, including response time metrics, work with your IT administrators who have access to comprehensive Grafana dashboards.

### Collaborate with IT administrators

**For regular users**

* Review basic performance metrics (CPU, memory, traffic volume, status codes) directly in the Domino endpoint interface

* Report performance issues with specific metrics and timeframes to IT administrators

* Work with IT teams to interpret trends and plan optimizations

**For Domino administrators**

* Access comprehensive Grafana dashboards with advanced metrics and alerting

* Set up shared dashboards for model performance monitoring

* Configure appropriate alert thresholds for your use case

* Establish escalation procedures for critical performance issues

* Review resource allocation and scaling policies

**Navigation for administrators**

* Use the **Grafana** link on endpoint version pages for direct dashboard access with pre-selected model filters

* Or access through **Admin** > **Reports** > **Grafana** > **Dashboards** > **Model Endpoints**

## Example: Debug a resource-heavy model

Consider a model with high resource usage. Use the monitoring dashboard to:

1. **Identify the problem**: CPU usage consistently >90%, memory growing over time

2. **Analyze patterns**: High resource usage correlates with specific request types

3. **Investigate code**: Review model inference logic for inefficiencies

4. **Implement fixes**: Optimize data processing, add result caching

5. **Validate improvement**: Monitor metrics to confirm optimization success

### Performance comparison

**Before optimization**

* CPU usage: 90-95% sustained

* Memory usage: 1.2 GB growing to 2.5 GB over 24 hours

* Error rate: 3% (mostly timeouts and 5xx errors)

* Response times: Noticeably slow, frequent timeouts

**After optimization**

* CPU usage: 30-45% average

* Memory usage: Stable at 150 MB

* Error rate: \<0.1%

* Response times: Significantly improved performance

IT administrators can access detailed response time percentiles (P50, P95, P99) and latency trends through Grafana dashboards, providing precise metrics for response time optimization that aren’t available in the basic Domino interface.

<img alt="Grafana dashboard showing poor latency performance before optimization" />

<img alt="Grafana dashboard showing improved latency performance after optimization" />

## Access endpoint logs

Domino provides comprehensive logging for troubleshooting and auditing your endpoints. Access logs through the **Logs** column for any endpoint version to review deployment events, runtime behavior, and error details.

### Log types

**Build logs**\
Events and processes during image building. Review these to understand build failures, dependency issues, or environment setup problems. Contains build definitions and metadata needed to complete the build process.

**Export logs**\
Details about the endpoint export process. Use these to troubleshoot issues with model packaging and deployment preparation. Shows how Domino processes your model files and creates the deployable endpoint.

**Instance logs**\
Runtime logs from individual containers for a given endpoint instance. Essential for debugging model execution issues, request processing problems, and runtime errors. Filter by endpoint and container to focus on specific components.

**Deployment logs**\
Chronological events related to deployment lifecycle. Includes heartbeats, jobs, deployments, and Kubernetes events with detailed payloads containing pod and status information. Critical for understanding deployment state and infrastructure issues.

### Use logs for troubleshooting

* **Performance issues**: Check instance logs for memory warnings, slow processing messages, or resource constraints.

* **Failed requests**: Review instance logs around the time of errors to identify specific failure causes and stack traces.

* **Deployment problems**: Examine deployment logs for Kubernetes events, pod startup issues, or resource allocation failures.

* **Build failures**: Use build logs to identify missing dependencies, environment configuration errors, or package conflicts.

* **Health check failures**: Look for patterns in instance logs that correlate with health check timeout issues.

<Tip>
  A storage-related error such as `IOException: Server returned HTTP response code: 500` in the build log usually means your model exceeds the 500 MB size limit; exclude large trained model datasets before publishing a new version.
</Tip>

## Best practices

### Development workflow integration

* **Establish monitoring early**: Implement performance monitoring during model development, not just after deployment

* **Set performance baselines**: Record initial performance metrics for each model version to track regressions

* **Use monitoring for architecture decisions**: Let resource usage patterns guide compute sizing and optimization strategies

* **Test with production-like loads**: Validate performance under realistic data volumes and request patterns before deployment

* **Automate performance testing**: Integrate monitoring validation into your CI/CD pipelines

### Production monitoring strategy

* **Implement layered monitoring**: Combine basic Domino metrics with advanced Grafana dashboards for comprehensive coverage

* **Set up proactive alerting**: Configure alerts based on your established baselines and business requirements

* **Monitor deployment health**: Track performance immediately after releases to catch deployment-related issues

* **Document escalation procedures**: Create clear runbooks for performance issues and recovery procedures

* **Regular performance reviews**: Schedule weekly or monthly reviews of performance trends and optimization opportunities

### Performance optimization workflow

* **Data-driven optimization**: Use monitoring data to identify specific bottlenecks rather than guessing

* **Version comparison**: Leverage Domino’s version switching to compare performance across model iterations

* **Resource right-sizing**: Use CPU and memory metrics to optimize compute resource allocation

* **Health check tuning**: Adjust health check parameters based on actual model performance characteristics

* **Continuous improvement**: Implement feedback loops between monitoring insights and model development

## Next steps

* [Secure model deployments](/cloud/platform-capabilities/features/model-deployment/security) by implementing security best practices for production model endpoints.

* [Scale model deployments](/cloud/platform-capabilities/features/model-deployment/scale) by configuring auto-scaling and resource management for high-traffic models.

* [IT Admin: Model endpoint monitoring dashboards](/cloud/admin/operations/grafana-workload-monitoring/model-endpoint-monitoring) allow advanced Grafana dashboards for comprehensive infrastructure monitoring.


# Domino endpoint scaling and routing
Source: https://docs.domino.ai/cloud/platform-capabilities/features/model-deployment/scale

Scale Domino endpoints across instances or Hardware Tiers, set Python worker parallelism, and route traffic to latest or promoted versions.

Scale [Domino endpoints](/cloud/platform-capabilities/features/model-deployment) horizontally and vertically for optimal performance. You can also use Domino endpoint routing to deploy simultaneous APIs for testing and production.

## Scale horizontally

Scale horizontally for throughput-constrained Domino endpoints. Typically these are endpoints that have many concurrent users. Consider horizontal scaling when downstream applications see long queues and running times from your endpoint.

By default, Domino schedules endpoint instances to run on separate nodes when possible for availability and fault tolerance. However, you can set a model instance to run strictly on different nodes by setting the `strictNodeAntiAffinity` parameter to true in the API call when creating a new Domino endpoint.

Endpoint instances are also scheduled to run in separate availability zones when possible.

When you publish a Domino endpoint, select the number of Domino endpoint instances that you want to run at any given time. Domino automatically load-balances requests to the endpoint between these instances. A minimum of two instances (default) provides a high-availability setup. Domino supports up to 32 instances per Domino endpoint.

<Note>
  Domino admins use the `com.cerebro.domino.modelmanager.instances.defaultNumber` configuration records key to change the default number of instances.
</Note>

## Scale vertically

Scale vertically for resource-constrained Domino endpoints. Consider whether your endpoint requires complex tasks with more processing power. Scale Domino endpoints vertically when downstream applications see long-running jobs for complex processes.

When you publish a Domino endpoint, select a [hardware tier](/cloud/admin/platform-configuration/manage-model-deployments#domino-endpoint-hardware-tiers) that determines the amount of RAM and CPU/GPU resources available to each Domino endpoint instance.

<Tip>
  The scaling settings are under **Endpoints** > **\<endpoint name>** > **Settings** > **Deployment**.
</Tip>

<Note>
  If you make changes to scale the Domino endpoint, you must restart it.
</Note>

You can set the degree of parallelism and scale all *Python* Domino endpoints:

1. From the Admin screen, go to **Platform settings** > **Configuration records**.

2. Click the pencil icon for the `com.cerebro.domino.modelmanager.uWsgi.workerCount` config key to update it.

   <Tip>
     If the `com.cerebro.domino.modelmanager.uWsgi.workerCount` config key is not listed on the Configuration Management page, click **Add Record** and enter the config key name at the end of the list in the key column.
   </Tip>

3. In the value column for `com.cerebro.domino.modelmanager.uWsgi.workerCount`, set the value to greater than 1, which is the default. See the [uWSGI project](https://uwsgi-docs.readthedocs.io/en/latest/) for more information.

4. Click **Save** if you edit the key or **Create** if you add a new one.

5. Under the **Configuration Management** page title, the system displays: “Changes here do not take effect until services are restarted. Click here to restart services.” Click the **here** link and follow the directions to restart the services.

## Route your Domino endpoint

Domino supports basic and advanced routing modes to help you manage development and test deployments. To change routing modes, go to **Settings** > **Deployment** for each Domino endpoint.

**Basic mode**\
In basic mode, one exposed endpoint always points to the latest successfully deployed Domino endpoint version. When you deploy a new version, the old version is shut down and replaced with the new one to maintain availability. Basic mode routes have the following signature:

**Latest:** `/models/<modelId>/latest/model`

**Advanced mode**

In advanced mode, a promoted version and the latest version exist simultaneously. Advanced mode lets you point your clients to the promoted, production version, while giving you the ability to test with the latest version. When the latest version is ready for production, switch it to the promoted version without downtime. Advanced mode routes have this signature:

**Latest:** `/models/<modelId>/latest/model`

**Promoted:** `/models/<modelId>/labels/prod/model`

## Next steps

* [Export models to SageMaker](/cloud/platform-capabilities/features/models/deploy-sagemaker)

* [Export models to NVIDIA fleet command](/cloud/platform-capabilities/features/models/export-nvidia)

* [Export models to use with Snowflake queries](/cloud/platform-capabilities/features/models/snowflake-queries)


# Domino endpoint security
Source: https://docs.domino.ai/cloud/platform-capabilities/features/model-deployment/security

Set endpoint visibility and prediction authorization, authenticate restricted requests, and assign viewer, editor, or owner permissions.

Domino offers visibility and authorization controls for [deployed Domino endpoints](/cloud/platform-capabilities/features/model-deployment) to limit who can see and interact with your endpoints. Learn how to control access and add contributors to Domino endpoints.

## Public or private Domino endpoint

Configure your endpoint to be accessible by certain authorized users (private) or by anyone with access to your Domino deployment (public).

On the Domino endpoint page, go to **Settings** > **Access and Sharing**.

**Public**\
Anyone with access to your Domino deployment can search, discover, and view your Domino endpoint. Only collaborators can modify or deploy versions or settings.

**Private**\
Only collaborators can search, discover, and view your Domino endpoint. Only collaborators can modify or deploy versions or settings.

## Unrestricted or restricted authorization

Authorization settings specify which users can access the Domino endpoint’s prediction endpoint.

**Restricted**\
A restricted Domino endpoint only authorizes specific users with valid access tokens to request predictions. Users must send the valid token with their requests. Code examples in the endpoint’s **Overview** tab show sample requests.

Generate an endpoint access token from the Domino endpoint’s **Settings** > **Invocation** tab. Use the name field to track which tokens are issued, to whom, and for what purpose.

**Unrestricted**\
Unrestricted endpoints allow anyone who can access Domino remotely to request predictions. No access token is required.

## Invoke restricted endpoints

Restricted Domino endpoints use different authentication methods depending on the endpoint type.

### Synchronous endpoints

Use HTTP basic authentication with the username and password both set to the access token.

For example, if the token is `c2b2532ed234f54`, then the Domino endpoint must be invoked with the HTTP username `c2b2532ed234f54` and password `c2b2532ed234f54`.

### Asynchronous endpoints

Use HTTP bearer (or token) authentication with the access token.

For example, if the token is `c2b2532ed234f54`, then the Domino endpoint must be invoked with the HTTP header `Authorization: Bearer c2b2532ed234f54`.

<table>
  <thead>
    <tr>
      <th />

      <th>Public</th>
      <th>Private</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td><p><strong>Unrestricted</strong></p></td>
      <td><ul> <li><p>Anyone with access to your Domino deployment can search, discover, and view your Domino endpoint.</p></li> <li><p>Only collaborators can modify or deploy versions or settings.</p></li> <li><p>No access token is required to request a prediction.</p></li> </ul></td>
      <td><ul> <li><p>Only collaborators can search, discover, and view your Domino endpoint.</p></li> <li><p>Only collaborators can modify or deploy versions or settings.</p></li> <li><p>No access token is required to request a prediction.</p></li> </ul></td>
    </tr>

    <tr>
      <td><p><strong>Restricted</strong></p></td>
      <td><ul> <li><p>Anyone with access to your Domino deployment can search, discover, and view your Domino endpoint.</p></li> <li><p>Only collaborators can modify or deploy versions or settings.</p></li> <li><p>An access token is required to request a prediction.</p></li> </ul></td>
      <td><ul> <li><p>Only collaborators can search, discover, and view your Domino endpoint.</p></li> <li><p>Only collaborators can modify or deploy versions or settings.</p></li> <li><p>An access token is required to request a prediction.</p></li> </ul></td>
    </tr>
  </tbody>
</table>

## Add collaborators

You can add **Domino endpoint collaborators** to view and manage specific Domino endpoints (but not the whole project). Domino endpoint collaborators can also invoke private prediction endpoints.

To add **Domino endpoint collaborators**:

1. Go to the Domino endpoint page > **Settings** > **Access**.

2. Add collaborators by their username or email address. You can also add organizations as collaborators and grant permissions to all members.

If you are the project owner, you can set the following access levels for collaborators:

**Viewers**\
Viewers can view the Domino endpoint's overview, its list of versions, and its tester. They cannot open version details, view logs, monitoring results, or settings, edit settings, or publish new versions. A viewer cannot see access tokens. To let a collaborator view version details, logs, or monitoring results, assign the Editor role.

**Editors**\
Editors with collaborator access to the underlying project can deploy new versions. They can view logs, view audit history, and change most settings. They cannot invite new collaborators or change Domino endpoint visibility. An editor can see all access tokens and generate new ones.

**Owners**\
Owners have all of the above permissions, and they can invite new collaborators, change the visibility, and transfer ownership. An owner can see and revoke all access tokens and generate new ones.

You can also add **Project collaborators** to grant them access to Domino endpoints within a project. Project collaborators assigned the `Contributor` role can create Domino endpoints, becoming the endpoint owner with the permissions listed above. All project collaborators can list Domino endpoints in the project, but they do not have access to the endpoints unless they are the endpoint owner or explicitly granted access.

## Next steps

* [Scale model deployments](/cloud/platform-capabilities/features/model-deployment/scale) by configuring auto-scaling and resource management for high-traffic models.

* [Monitor model endpoints](/cloud/platform-capabilities/features/model-deployment/monitor-model-endpoints) to track health, performance, and logs for your deployed endpoints.


# Configure batch scoring Job
Source: https://docs.domino.ai/cloud/platform-capabilities/features/models/batch-scoring

Configure a batch scoring Job with inference code, compute resources, schedules, logs, notifications, and result files for bulk predictions.

Domino makes it easy to deploy models for batch scoring to make inferences (predictions) in bulk over a large batch of records. Learn how to use Domino jobs to perform batch prediction with the web UI and API.

Batch scoring jobs are a good choice for "fire-and-forget" scoring jobs that don’t require a strict SLA (Service Level Agreements) for results. If you need a scoring endpoint with better response times, especially for customer-facing applications, use the [Domino endpoint REST endpoints](/cloud/platform-capabilities/features/model-deployment).

## Configure a batch scoring Job

1. Set up the executable inference file.

   1. Create a prediction script that is the entry point for the batch scoring program.

   2. Use the prediction script as the executable when you configure a new job run. See how to [start a Domino job](/cloud/platform-capabilities/core-concepts/jobs).

2. Configure execution and compute resources.

   1. Select a custom Domino environment if you require additional packages not supported by the default environment.

   2. Select the size of the storage volume in the job.

   3. Select the appropriate compute resources needed. For simplification, Domino offers a list of available Hardware Tiers that represents a pool of nodes your Domino Administrator configures for such executions.

   4. If your scoring code works with large-scale compute clusters like Spark, Ray, or Dask, attach one of those compute clusters to the job to enable their use. Domino provides on-demand Spark, Ray, Dask, and OpenMPI clusters made available to the executable in the job.

## Deploy the scoring Job

Run the job created in the previous section. Optionally, [run the job on a schedule](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job) to process periodically generated data that requires batch scoring.

## Monitor the scoring Job

Batch job durations depend on the volume of processed data and the available compute resources. [Monitor the job’s progress](/cloud/platform-capabilities/core-concepts/jobs/monitor-jobs) on the jobs dashboard.

## Retrieve results

Configure the notifications settings to receive the results of an execution. By default, you are notified of all failed runs via email. On successful execution, you receive an email with the last few lines of stdout and up to ten results files from your execution. Domino detects which files were added or changed during your execution and captures those as the execution’s results. You can also [set up custom emails](/cloud/platform-capabilities/core-concepts/jobs/customize-job-results) as part of the notification.

The results of an execution are the set of files that your scoring code generates or modifies when it runs. View the results of a given execution through the **Runs** tab of your Project or on the **Results** tab of your Job. See [Execution results](/cloud/platform-capabilities/core-concepts/jobs/monitor-jobs) to learn how to retrieve results, customize the results dashboard, or automate the delivery of results via an Airflow pipeline.

## Troubleshooting

To troubleshoot your scoring job, Domino provides access to log files related to the execution. If you need help, contact [Domino Support](/help/help#contact-technical-support) to review and resolve execution issues.

* For errors that occur before execution, that is, during setup, download the Setup Output files.

* Download the User Output log file for errors in the scoring job to examine and address any changes to your scoring code.

* For runtime errors related to the deployment itself, download the Deployment Logs file to analyze the failure and share it with Domino Support if needed.

## Next steps

For model deployments that require interactivity, see [Deploy model endpoints](/cloud/platform-capabilities/features/model-deployment).


# Deploy models to SageMaker
Source: https://docs.domino.ai/cloud/platform-capabilities/features/models/deploy-sagemaker

Deploy a registered model to an Amazon SageMaker endpoint, configure resources, scaling, access, and streaming, then test and manage it.

Domino provides model deployment to SageMaker. Leverage Domino’s flexible development experience to build your models before deploying them for production on SageMaker, while still maintaining centralized governance and tracking of all of your models. Administrators apply guardrails around cost, performance, and security, and the platform picks the resources needed to meet them.

New versions of your models are automatically published to the target environment, validated, and made available for production after they clear the necessary measures.

<Note>
  Your admin must set up an external deployment target before you can deploy models to SageMaker. See the [admin documentation](/cloud/admin/platform-configuration/manage-model-deployments#external-deployment-targets) for more details. Deploying models to SageMaker is only supported when the Domino control plane is running on an [AWS EKS cluster](https://aws.amazon.com/eks).
</Note>

Domino also provides an intelligent system that can profile your models and expected scale of usage and recommend a hosting strategy. This helps enable:

* A smoother transition from development to production through automated packaging, transfer, and deployment of models to SageMaker.

* Unified governance and tracking for all models through a single interface, regardless of deployment environment.

* The advanced capabilities of SageMaker facilitate the scalable and cost-effective deployment of the latest models, including LLMs.

## Prerequisites

Before you deploy a model to SageMaker, make sure that you have an existing registered model. For details on how to register a model, see the [model registry](/cloud/platform-capabilities/features/models/register-and-govern) documentation.

Verify that your admin set up an external deployment target so you can deploy endpoints to SageMaker. See the [admin documentation](/cloud/admin/platform-configuration/manage-model-deployments#external-deployment-targets) for more details.

## Create the endpoint

1. Navigate to the **Endpoints** page and click the **External** pivot. Then click **Create external endpoint**.

   <img alt="Endpoint Landing Page" />

2. Provide a **Name** for the endpoint, select the **Type** and optionally enter a **Description**.

   <img alt="Create Endpoint - Details" />

3. Select the **Model** that you want to deploy, followed by the **Environment** that is required to run the model.

   <img alt="Create Endpoint - Model" />

4. Choose the **Deployment Target** and **Resources** that you want to use for the deployment.

   <img alt="Create Endpoint - Deployment" />

5. Select whether to enable **Streaming** and the **Minimum** / **Maximum** number of instances for your endpoint.

   <img alt="Create Endpoint - Configuration" />

   <Note>
     Setting a higher maximum than minimum value enables auto-scaling. If auto-scaling is enabled, it triggers a scale-up when the **CPU utilization** exceeds the target threshold of **80% utilization**. To use a different scaling policy or add additional scaling policies, modify the **Scaling Policies** section.
   </Note>

6. Set the **Visibility** of the endpoint and add additional **Collaborators** if you like.

   <img alt="Create Endpoint - Access & Sharing" />

## View, test, and edit the endpoint

Once the endpoint has been created, you can click into it to view details for the endpoint. You can also **Edit** or **Delete** it.

From this view, you can get a code snippet for testing the endpoint. Paste this code snippet into a workspace and replace the request body with a valid inference request to test the deployment.

In the same view, you can also stop the endpoint. If you click **Stop endpoint**, the SageMaker endpoint is deleted and a new one is created when it is started up again.

<img alt="Endpoint Details" />


# Export models to NVIDIA Fleet Command
Source: https://docs.domino.ai/cloud/platform-capabilities/features/models/export-nvidia

Export a Domino endpoint image and Helm chart to NVIDIA Fleet Command, track the export by ID, and configure ports for edge deployment.

Domino’s integration with NVIDIA Fleet Command simplifies model deployment to the edge.

Scaling AI at the edge is a critical way of deploying data science work, and is useful for scenarios like anomaly detection at cellphone towers and quality control in manufacturing.

## Export to NVIDIA Fleet Command

1. [Deploy a Domino endpoint](/cloud/platform-capabilities/features/model-deployment) from any project and run tests to ensure your Domino endpoint is functioning properly.

2. After the deployment status changes to **Ready to Run**, invoke the Domino API call to register the endpoint image with NVIDIA.

   `/:modelId/:modelVersionId/exportImageForNvidia`

3. Wait for the API to register the Domino endpoint container image and an associated Helm chart to the Fleet Command registry.

   See the Domino endpoint details to learn how to invoke the export API to push the image to Fleet Command and check the status endpoint to track the progress of the export.

4. Use Fleet Command to configure the distributed edge system.

With Domino’s model export catalog, you can see a unified view of all exported models and their performance. With this view, you can track and manage all production assets from a single pane.

## Invoke the export API

```shell theme={null}
curl =
--location --request POST 'https://<deployment>/v4/models/<model-id>/<model-version-id>/exportImageForNvidia'
--header 'Content-Type: application/json'
--header 'X-Domino-Api-Key: <domino-api-key>'
--data-raw '{
 "containerRepository": "<path-to-container-repo>",
 "tag": "<image-tag>",
 "helmRepository": "<path-to-helm-repo>",
 "helmVersion": "<version>",
 "ngcApiKey": "<ngc-api-key>"
}'
```

Here is a description of the parameters:

* `model-id`: unique identifier for the Domino endpoint

* `Model-version-id`: version ID for the Domino endpoint

* `containerRepository`: Repository name for a private container repository in NVIDIA NGC. For example, **\<organization-name>/\<team-name>/\<container-repository-name>**. The user can provide a container repository that already exists in NVIDIA NGC. Here are the acceptable elements for a Container repository name:

  * Alphanumeric characters

  * Lowercase characters

  * Dashes and slashes (for organization and team names)

  * No spaces

* `tag`: Tag for the container. The tag must be alphanumeric. It can also contain underscores, periods, and dashes.

* `helmRepository`: Repository name for a private helm repository in NVIDIA NGC. For example, \<organization-name>/\<team-name>/\<container-repository-name>. The user can provide a helm repository that already exists in NVIDIA NGC. Here are the acceptable elements in a Helm repository name:

  * Alphanumeric characters

  * Lowercase characters

  * Dashes and slashes (for organization and team names)

  * No spaces

* `helmVersion`: Version of the Helm chart. Helm versions must be in [Semver](https://semver.org/) format.

* `ngcApiKey`: NVIDIA NGC API key. See [generate an NVIDIA NGC API Key](https://docs.nvidia.com/ngc/gpu-cloud/ngc-overview/index.html#generating-api-key).

  <Note>
    * The port for the endpoint on the edge device is hardcoded to 30008. If you deploy two Domino models on the same edge device, the second application deployed fails. Override the port in `values.yaml` in the NVIDIA Fleet Command UI to prevent this failure.

    * The model cannot use anything Domino-specific like predictive sets or domino datasets.

    * The model must be built in Domino first, and there must be a version of the Model.

    * If you export a model with an image tag or Helm chart version number that already exists, the export fails.

    * You cannot deploy different versions of the same NVIDIA Application to the same edge device. If you want to deploy multiple Domino models on a single-edge device, override the port when you deploy.
  </Note>

## Track the status of the export

Use the `export-id` you get as the return value from the export endpoint to monitor the export status.

Here is a sample command:

```shell theme={null}
curl --include
    --header 'Content-Type: application/json'
    --header 'Accept: application/json'
    --header 'X-Domino-Api-Key: <domino-api-key>'
 'https://<deployment>/v4/models/<export-id>/getExportImageStatus'
```

## Next steps

[Integrate endpoint deployment with CI/CD workflows](/cloud/platform-capabilities/features/models/integrate-ci-cd-workflows)


# Deploy models
Source: https://docs.domino.ai/cloud/platform-capabilities/features/models/index

Choose real-time endpoints, batch scoring Jobs, or exports to Amazon SageMaker, NVIDIA Fleet Command, Snowflake, and external CI/CD pipelines.

[Deploy models as REST API endpoints](/cloud/platform-capabilities/features/model-deployment) with a single click for asynchronous (batch) or synchronous (real-time) predictions on new data.

Domino offers built-in security, scaling, and management features:

* [Monitor endpoint performance](/cloud/platform-capabilities/features/model-deployment/monitor-model-endpoints).

* [Endpoint security](/cloud/platform-capabilities/features/model-deployment/security).

* [Endpoint scaling and routing](/cloud/platform-capabilities/features/model-deployment/scale).

* [Endpoint authorization mode](/cloud/platform-capabilities/features/model-deployment/domino-endpoint-authorization).

* [Asynchronous endpoints](/cloud/platform-capabilities/features/model-deployment/async-example-client).

In addition, you can score large batches of data with [batch scoring Jobs](/cloud/platform-capabilities/features/models/batch-scoring).

To serve a Large Language Model (LLM) instead, see [Host an LLM](/cloud/platform-capabilities/features/llms/host-an-llm), which deploys it as an endpoint with an OpenAI-compatible API.

## Deploy models to third-party platforms

Domino gives you the flexibility to use your models and endpoints on other supported platforms.

* [Deploy models to SageMaker](/cloud/platform-capabilities/features/models/deploy-sagemaker) to leverage auto-scaling and streaming capabilities.

* [Export to NVIDIA Fleetcommand](/cloud/platform-capabilities/features/models/export-nvidia) for edge deployments.

* [Export models to Snowflake](/cloud/platform-capabilities/features/models/snowflake-queries) to bring the model to where your data lives.

## Integrate Domino models with existing CI/CD workflows

Domino can deploy natively, export to third party platforms, or [integrate model deployments with existing CI/CD workflows](/cloud/platform-capabilities/features/models/integrate-ci-cd-workflows) with the Model Export API. The choice is yours.

## Register and govern models

Use [Register and govern models](/cloud/platform-capabilities/features/models/register-and-govern) to create a system of record for your models and apply organizational policies and regulations before deployment.


# Integrated deployment to CI/CD workflows
Source: https://docs.domino.ai/cloud/platform-capabilities/features/models/integrate-ci-cd-workflows

Use the Model Export API to build a model image, export it to a registry, test it on Kubernetes, and promote it through a CI/CD pipeline.

Domino can integrate with existing CI/CD workflows and registries outside of Domino. Use Domino’s Model Export API to integrate model development in Domino with other production environments.

With a single API call, Domino turns your trained model into a container image with a REST interface that you can deploy into a separate production environment. You may need to host models outside of Domino for specific scenarios such as:

* Support for sudden bursts in traffic

* Strict latency and uptime SLAs

* Security, legal, or locality restrictions

## Model export API

Domino’s Model Export API exports Docker images of Domino-built models, model artifacts, project files, dependent packages, base environments, and more. The API registers the images with Domino’s internal image registry. You then fetch it, customize it, and deploy it inside your runtime environment with existing CI/CD pipelines to facilitate tests and production-readiness steps.

By default, all files present in a project are copied into the generated model image. The default image includes the specified compute environment, project files, a Flask/Plumber harness that exposes the REST interface, and an authentication and load balancing layer.

<img alt="CI/CD pipeline workflow" />

To control the sizes of such images, include names of specified files in the `.modelignore` file in the project’s root folder to omit them. Filenames, folder names, or UNIX shell regular expressions are allowed in this file.

<img alt="CI/CD pipeline workflow continued" />

## 1. Train the model in Domino

A data scientist creates a project in Domino, associates an environment with it, and conducts multiple experiments to identify the best production model. The data scientist or MLOps engineer then configures the CI/CD pipeline to run through the next steps to deploy the model from the project.

## 2. Build the model image

Your CI/CD pipeline calls the **Build Model Image API** to trigger a build process. This call builds and saves the generated model image in Domino’s internal container registry. The API response includes the model ID and the model version ID, which are used to poll for the build status or to fetch build logs. The model build process comprises Preparing, Building, Complete, or Failed.

Invoke the build process:

```script theme={null}
curl --location --request POST 'https://<YOUR_DOMINO_APP_PATH>/v4/models/buildModelImage'
--header 'Content-Type: application/json'
--header 'X-Domino-Api-Key: <YOUR_DOMINO_API_TOKEN>'
--data-raw '{
  "projectId": "5e5efb400cf22210b88f36df",
  "inferenceFunctionFile": "model.py",
  "inferenceFunctionToCall": "predict",
  "environmentId": "5e4c6cc6cec7a117386a5c0a",
  "modelName": "ModelExportDemo",
  "logHttpRequestResponse": true,
  "description": ""
}'
```

Poll for completion:

```script theme={null}
curl --location --request GET 'https://<YOUR_DOMINO_APP_PATH>/v4/models/<modelID>/<modelVersionId>/getBuildStatus'
--header 'Content-Type: application/json'
--header 'X-Domino-Api-Key: <YOUR_DOMINO_API_TOKEN>'
```

## 3. Export the model image to your registry

After the image is built and registered in Domino’s internal registry, export it to your registry with Domino’s **Export Model Image API**. The API responds with an export process ID, which is used to poll for the status or fetch export operation logs. As part of a production workflow, ensure the image is first deployed for staging.

Invoke the build process:

```script theme={null}
curl --location --request POST 'https://<YOUR_DOMINO_APP_PATH>/v4/models/<modelID>/<modelVersionId>/exportImageToRegistry'
--header 'Content-Type: application/json'
--header 'X-Domino-Api-Key: <YOUR_DOMINO_API_TOKEN>'
--data-raw '{
  "registryUrl": "quay.io/sample_url",
  "repository": "sample_prod_repo",
  "tag": "sample_tag",
  "username": "username",
  "password": "secret_token"
}'
```

Poll for completion:

```script theme={null}
curl --location --request GET 'https://<YOUR_DOMINO_APP_PATH>/v4/models/<exportId>/getExportImageStatus'
--header 'Content-Type: application/json'
--header 'X-Domino-Api-Key: <YOUR_DOMINO_API_TOKEN>'
```

## 4. Run the exported image in a staging environment

Stand up the image in your test environment, which is often a separate Kubernetes cluster dedicated to stage and test workflows. You can also configure your CI/CD pipeline to customize the image to prepare it for staging.

Deploy an exported model Image on an existing Kubernetes cluster with these steps. This export assumes your K8s cluster is running and kubectl is installed. Then create the K8s manifest files, which describe your application & service and use kubectl to deploy them.

Use these Sample Kubernetes manifest files and procedures to deploy the Image.

my-model-deployment.yaml

```YAML theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mymodel
  labels:
    app: mymodel
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mymodel
  template:
    metadata:
      labels:
        app: mymodel
    spec:
      containers:
        - name: mymodel
          image: docker.io/myrepo:mytag
          ports:
            - name: mymodel
              containerPort: 8888
```

my-model-service.yaml

```YAML theme={null}
apiVersion: v1
kind: Service
metadata:
  name: mymodel-service
spec:
  type: NodePort
  selector:
    app: mymodel
  ports:
  - nodePort: 31440
    name: mymodel-port
    port: 90
    targetPort: 8888
```

<Note>
  If you pull from a private registry, [add a secret](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) to your Kubernetes cluster so the pod can authenticate with the docker registry.
</Note>

## 5. Start the deployment and service

```shell theme={null}
$ kubectl apply -f my-model-deployment.yaml
$ kubectl apply -f my-model-service.yaml
```

Get the URL:

```shell theme={null}
# If the kubernetes is in minikube then get the minikube IP
$ MYMODEL_IP="$(minikube ip)"
```

```shell theme={null}
# Else get IP of the mymodel-service
$ MYMODEL_IP="$(kubectl get service/mymodel-service -o jsonpath='{.spec.clusterIP}')"
```

Get the port:

```shell theme={null}
MYMODEL_PORT="$(kubectl get service/mymodel-service -o jsonpath='{.spec.ports[?(@.name=="mymodel-port")].nodePort}')"
```

Get the Model URL:

```shell theme={null}
echo http://$MYMODEL_IP:$MYMODEL_PORT/model
>> http://192.168.99.100:31440/model
```

### Test the exported model

After the image is up and running, test the deployed model.

Sample invocation to test your python model:

```shell theme={null}
curl --location --request POST 'http://192.168.99.100:31440/model' --header 'Content-Type: application/json' --data-raw '{"data": {"start": 1, "stop": 100}}'
```

The content of the “data” JSON object is the input to your model.

## 6. Deploy to a production environment

After you test the model in the staging environment and it clears the requisite production criteria, your CI/CD pipeline can deploy the Domino endpoint image to a production environment. If you want to use a separate Kubernetes cluster for this environment, substitute your desired cluster in the [Run the exported image in a staging environment](#4-run-the-exported-image-in-a-staging-environment) and [Test the exported model](#test-the-exported-model) steps.


# Register models
Source: https://docs.domino.ai/cloud/platform-capabilities/features/models/register-and-govern/index

Use the Model Registry and model review process to create a system of record for models and apply organizational policies and regulations.

Use Domino’s model registry and integrated model review process to ensure that models meet organizational policies and regulations.

## Track and manage models with the model registry

[Register your models to the model registry](/cloud/platform-capabilities/features/models/register-and-govern/model-registry) to create a system of record for your machine learning models.


# Manage models with model registry
Source: https://docs.domino.ai/cloud/platform-capabilities/features/models/register-and-govern/model-registry

Register MLflow models through the API or web UI, manage versions and model cards, control access, and make models discoverable.

Track and manage your machine learning models with Domino’s MLflow-based model registry. The model registry lets you do the following:

* Discover models in project-scoped and deployment-scoped registries.

* Record model metadata and lineage for auditability and reproducibility.

* Create custom model cards to capture notes on fairness, bias, and other important information.

* Manage model versions and deploy models to Domino-hosted or externally hosted, endpoints.

* Use RBAC (role-based access control) to limit who can view, edit, and collaborate on registered models.

<Note>
  Model registry doesn’t support R-based models.
</Note>

## Register your model

Domino uses MLflow to track and manage Domino experiments. Each Domino experiment can be associated with one or more models that you can publish to the model registry. You can register models using the MLflow API or through the Domino web UI.

### Register models using the MLflow API

Register your model to the model registry programmatically with the MLflow API. This method is especially useful if you are already using [Domino’s experiment management workflow](/cloud/platform-capabilities/features/development/track-monitor) to log training runs using MLflow. After you successfully train a model that you want to register, add the following code to name, tag, and register the model.

Use MLflow tags to add custom attributes or versions to a model to track additional metadata.

<Note>
  Model registry tags are distinct and cannot be used with [Project tags](/cloud/platform-capabilities/core-concepts/projects/project-settings#tag-a-project) or [Model Monitoring tags](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/9-model-monitoring-tags).
</Note>

```python theme={null}
import random
import string
random_string = ''.join(random.choice(string.ascii_letters) for _ in range(3))
import mlflow.sklearn
from mlflow.store.artifact.runs_artifact_repo import RunsArtifactRepository
from mlflow import MlflowClient
from sklearn.ensemble import RandomForestRegressor

client = MlflowClient()

# Register model name in the model registry
name = "RandomForestRegression_" + random_string
registered_model = client.create_registered_model(name)

# create an experiment run in MLflow
params = {"n_estimators": 3, "random_state": 42}
rfr = RandomForestRegressor(**params).fit([[0, 1]], [1])
# Log MLflow entities
with mlflow.start_run() as run:
    mlflow.log_params(params)
    model_info = mlflow.sklearn.log_model(rfr, artifact_path="sklearn-model")
    runs_uri = model_info.model_uri

    # Create a new model version of the RandomForestRegression model from this run
    desc = "A testing version of the model"
    model_src = RunsArtifactRepository.get_underlying_uri(runs_uri)
    mv = client.create_model_version(name, model_src, run.info.run_id, description=desc)
    print("Name: {}".format(mv.name))
    print("Version: {}".format(mv.version))
    print("Description: {}".format(mv.description))
    print("Status: {}".format(mv.status))
    print("Stage: {}".format(mv.current_stage))
print("You created your first registered model!")
print(f"Go to the Models UI and click on the {name} model to see the Model Card.")
```

Starting with Domino 5.11.1, you can now upload large artifact files directly to blob storage without needing to go through the MLflow proxy server. This is an experimental feature that must be enabled inside user notebook code by setting the environment variable `MLFLOW_ENABLE_PROXY_MULTIPART_UPLOAD` to `true`.

```python theme={null}
import os

# Enable MLflow multipart uploads
os.environ['MLFLOW_ENABLE_PROXY_MULTIPART_UPLOAD'] = "true"

# (default: `10_485_760` (10 MB)) - set to 200MB instead here.  Select size based on maximum of 1000 chunks.
os.environ['MLFLOW_MULTIPART_UPLOAD_CHUNK_SIZE'] = '200_000_000'
```

This feature is useful for both `log_artifact` calls and registering large language models. It is currently supported only in AWS and GCP environments. There are 2 additional settings available for configuring this feature:

* `MLFLOW_MULTIPART_UPLOAD_MINIMUM_FILE_SIZE` - the minimum file size required to initiate multipart uploads.

* `MLFLOW_MULTIPART_UPLOAD_CHUNK_SIZE` - the size of each chunk of the multipart upload. Note that a file may be divided into a maximum of 1000 chunks.

See the [MLflow multipart upload for proxied artifact access](https://mlflow.org/docs/latest/self-hosting/architecture/artifact-store/#multipart-upload-for-proxied-artifact-access) documentation for further details.

### Register models using the web UI

You can also register a model using the Domino web UI after an experiment run has been completed.

1. Go to **Experiment** and select your experiment.

2. Select your run.

3. Select **Register model from run**.

## Model cards

Domino helps you manage your models through model cards. Model cards are automatically created whenever you publish a model to the registry. Model cards help you do the following:

* Ensure reproducibility and auditability by automatically tracking metadata, lineage, and downstream usage for each version of your models.

* Document the model with information like data science techniques, limitations, performance, fairness considerations, and explainability.

### Lineage and metadata tracking

Domino automatically records the code, environment, workspace settings, and any datasets or data sources used for each experiment and associates them with the registered model. This system of record lets you track and reproduce the training context for your models for auditing and validation. Domino automatically adds all of this information to every model card for easy reference.

### Custom model cards

When publishing your model, you can either import model cards from other sources or add your own markdown content to the model’s description. Custom model cards can be used to bring together other resources to provide a single point of truth for your model. For example, you could link to relevant research, dashboards, or external applications that help you evaluate models for fairness and bias.

You can update model card markdown programmatically, as in this example:

```python theme={null}
with open('Model Card v2.md', 'r') as file:
    markdown_description = file.read()
    client.update_registered_model(name, markdown_description)
```

You can also directly edit the model card in the web UI:

1. Go to **Models** > **Overview**.

2. Select the **Edit** icon and edit the model card.

## Collaborate securely

Domino uses project-based access controls to determine who can view, manage, and deploy models. Users must authenticate into Domino to use MLflow and Domino APIs for model tasks. Project user roles govern the model registry actions that a user can perform. For more information on model registry user roles, see [model registry roles](/cloud/admin/identity-and-access/manage-users/roles#model-registry-actions).

## Model discoverability

You can toggle Models in the Model Registry to be "publicly" discoverable to other Domino users within the organization. By allowing models to be publicly discoverable, organizations can improve visibility, reduce redundancy, and increase collaboration between otherwise siloed projects.

For example, a data scientist may want to explore models from other teams to avoid duplicating work. Meanwhile, a data science leader may want a consolidated view of all models across the organization to measure the success of the data science team.

In the Model Registry, users see two lists:

**Global Models**: A list of all the models that are marked publicly discoverable, regardless of whether the user has adequate project permissions to use the models. This serves as a dashboard for data scientists who want to explore and use models from other teams.

**Collaborating Models**: A list of all the models that a user is allowed to see. This includes models marked as publicly discoverable (annotated by the globe icon). Data science leaders can use this as a consolidated dashboard to track model development, usage, and metrics across projects.

### Create a publicly discoverable model

To make a new model publicly discoverable:

1. [Register a model from a run](#register-models-using-the-web-ui).

2. Check the checkbox for **Model Discoverability**.

<Note>
  The checkbox’s default setting is determined by the `com.cerebro.domino.registeredmodels.newModelDiscoverability.default` configuration records setting, which is initially set to `true`.
</Note>

### Toggle an existing model as discoverable

Once the model is created, a System Admin or Project Owner can toggle the model in and out of public discoverability using one of two ways.

1. Through the model card UI:

2. Through the public API PATCH endpoint (API not published on [API guide](/cloud/reference/api)).

<Note>
  If you turn off a Model’s public discoverable setting, Domino users without project permissions can no longer see the model in their global model list or view the model card.
</Note>

## Read-only users

Domino shows a limited, read-only version of the model card for publicly discoverable models to users without project permissions. The read-only model card hides the discoverability toggle, doesn’t allow edits to the model description, and does not render Domino endpoint components. Additionally, links on various components of the model card may lead to 403 error pages if the user doesn’t have the proper permissions to view assets such as Data Sources, experiment runs, code, etc.

You can request access to a read-only model from the project owners:

1. Go to the Model Card.

2. In the Model Card header, click **How can I use this Model?**

## Example MLflow-supported models

See examples of how to use various MLflow-supported models such as XGBoost, Sklearn, PyTorch, TensorFlow, and custom Pyfunc models for tasks like registering various flavors of MLflow models, enabling integrated monitoring, and [creating Domino endpoints](/cloud/platform-capabilities/features/model-deployment).

1. [Registering XGBoost model with integrated model monitoring](https://github.com/dominodatalab/reference-project-domino-mlflow-supported-models/blob/main/domino-mlflow-model-xgboost-imm.ipynb).

2. [Registering Sklearn model with integrated model monitoring](https://github.com/dominodatalab/reference-project-domino-mlflow-supported-models/blob/main/domino-mlflow-model-sklearn-imm.ipynb).

3. [Registering MLflow models (with column and tensor-based inputs) as Python functions in Domino](https://github.com/dominodatalab/reference-project-domino-mlflow-supported-models/blob/main/domino-mlflow-models.ipynb).


# Snowflake queries
Source: https://docs.domino.ai/cloud/platform-capabilities/features/models/snowflake-queries

Deploy Python models as Snowpark user-defined functions in Snowflake, push new versions, monitor drift and quality, and troubleshoot exports.

Snowflake is one of the most prominent data storage services. When you move large volumes of data to the model, it increases costs and involves security risks. As a result, many shift to an approach where the model comes to the data, also known as in-database scoring. Domino pushes models down to Snowflake via Snowpark integration and connects the data back to the Domino platform to monitor those models for drift and accuracy.

## How it works

After you train your model in Domino and write the necessary inference logic that you want to execute from within Snowflake, Domino lets you push model artifacts and associated Project Files necessary to execute the inference code in Snowflake as a Snowpark UDF.

<Note>
  [Snowpark](https://www.snowflake.com/en/data-cloud/snowpark/) is a library that lets users build applications (with a high level language like Scala) that run in Snowflake compute environments where the data resides. The core abstraction in Snowpark is the DataFrame. In addition to built-in capabilities in the Snowpark library, users create user-defined functions (UDFs) for custom operations. Both the DataFrame and UDFs are optimized to execute lazily on the Snowflake server.
</Note>

## Meet the requirements

The model that you create in Domino must meet these requirements to be hosted in Snowflake:

* Create your model in Python 3.8.

* Create a file named `requirements.txt` in the root folders (for example, the `root` folder of the main Git-based project, an imported repository, or imported projects)

  This file defines the list of required packages for the model. See [https://github.com/binder-examples/requirements/blob/master/requirements.txt](https://github.com/binder-examples/requirements/blob/master/requirements.txt) for an example and [Add Packages to Environments](/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments) for more details.

  * These packages must be in the Snowflake Anaconda list: [https://repo.anaconda.com/pkgs/snowflake/](https://repo.anaconda.com/pkgs/snowflake/).

  * If you use external packages, your Snowflake instance must enable the usage of [Anaconda](https://repo.anaconda.com/pkgs/snowflake/) packages.

* Create Python-type hints for the model function input and return variables.

For the model to be monitored, you must meet these requirements:

* A `timestamp` column in your prediction dataset

* The name of the table that contains the prediction data

Domino looks for entries in the table with new timestamps to continuously ingest new data in batches.

## Push models to Snowflake

1. Train a model with Domino Workbench to produce a model artifact and associated Project Files needed to invoke an inference with the model.

2. Instrument the inference function in the form of a Snowpark UDF with Snowflake packages. (See example below)

   1. The prediction UDF typically accepts a DataFrame as an input to a prediction.

   2. The result of the prediction (also typically a DataFrame) is returned as a response to the caller of the UDF

3. With a Domino publish flow (via the Domino web application), deploy the model as a Snowpark UDF to a Snowflake endpoint. The presented dialog prompts you to enter the necessary information to create a session with Snowflake and upload the custom function and necessary artifacts to create a Snowpark UDF.

   <Tip>
     In **Snowflake URL**, enter the URL that uniquely identifies the Snowflake account in your organization in the form `https://<account-id>.snowflakecomputing.com`.
   </Tip>

   1. Information to create a Snowflake session includes key identifiers for the account (URL, role, credentials via username, and password) and JDBC settings (stage, warehouse, DB, and schema).

   2. Domino adds the model artifacts (model file, UDF contained python file, and dependent python modules) to the Snowflake session.

   3. Domino registers the UDF with Snowflake.

4. (Optional) If you make changes, deploy a new version. This action packages the current state of the project (also known as the HEAD commit on the main branch) as a new version of the exported model. Then, Snowflake overwrites the published UDF.

   1. From the **Exports** page, click the name of the model for which you want to create a new version.

   2. Go to **Actions > Push new version**.

   3. Enter the name of the file that contains the predict function and the name of the predict function to call from the external provider.

   4. If necessary, select a different **Environment** in which to run the model.

   5. On the **Next** page, enter your **Snowflake Username**, enter you **Snowflake Password**, and select a **User Role** for the external provider. Then, click **Authenticate & Export**.

5. (Optional) After your exported model has been used for inference and it has saved model inputs and predictions to a supported data store, configure the model to monitor for data drift and model quality in Domino. See [Model Requirements](#meet-the-requirements) and [Set up Model Monitor](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor).

   1. Go to **Grafana Monitoring** > **Configure Monitoring > Data**.

   2. From the Configure Data window, select the **Training Data** and the **Version** for the Domino training set on which the model was trained. See [Domino Training Sets](/cloud/platform-capabilities/core-concepts/data/data-source-connectors).

   3. From **Model type**, select **Classification** or **Regression** depending on your model type. If your Training Set code includes prediction data defined in target\_columns, select the model type that matches your Training Set:

      * If target\_columns is a categorical column, select **Classification**.

      * If target\_columns is a numerical column, select **Regression**.

   4. After the model is registered, the system shows options to add **Prediction Data** and **Ground Truth Data**.

      <Note>
        You must wait until Training Data is ingested to add this information.
      </Note>

      After the data is ingested, click the **open this model** link for the data to add.

      * See [Set Up Drift Detection](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/4-set-up-drift-detection-mm).

      * See [Set Up Model Quality Monitoring](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/5-set-up-model-qual-monitor-mm)

   5. Click **Save**.

   6. See [Test defaults](/cloud/platform-capabilities/features/monitoring/7-monitor-settings/4-tests) to set the targets for your data drift and model quality metrics and click **Next** when you’re done.

   7. See [Set Scheduled Checks](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/8-set-scheduled-checks-mm) to define the schedule for when monitoring results are calculated and updated and click **Next** when you’re done.

   8. To send email notifications if thresholds are breached based on the Scheduled checks, in **Send alerts to these email addresses**, type or paste a comma- or semicolon-separated list of email addresses.

   9. Click **Save & Test**.

6. (Optional) Use the Exports page to monitor your externally hosted data science models. The model monitoring feature detects and tracks data drift in the model’s input features and output predictions. See [Use Monitoring](/cloud/platform-capabilities/features/monitoring/5-use-monitoring) for more information about model monitoring.

   1. In the navigation pane, click **Exports** to review the Drift and Model Quality columns.

   2. Inspect the results.

      * A green circle with a checkmark indicates that the drift and model quality metrics are healthy.

      * A red circle with a number indicates the number of alerts that exceed the threshold for the drift or metrics during the most recent check.

      * A gray circle without any icon or number indicates a null value; no metrics are available.

After this setup, call the UDF from inside your Snowpark Python application to perform inference against a Domino trained model.

## Troubleshooting

* To gain the most insight, examine logs for the export:

  1. Go to the Jobs dashboard of the Project and examine the last Domino Job for the export.

  2. Use the Export UI (view export logs) to navigate to the Domino Job that was used for this export.

  3. Examine the User Output under the Job.

     <Note>
       For Snowflake models, when you hover over the command on the Jobs page, the password `SNOWFLAKE` represents an environment variable, not your actual password.
     </Note>

* If your Domino model uses an external file, such as a pickle file in the predict function, you must specify a different directory path for Snowflake than for Domino. See [Reading Files from a UDF](https://docs.snowflake.com/en/developer-guide/snowpark/python/creating-udfs.html#reading-files-from-a-udf).

* Authentication or networking issues might cause exports to fail.

  If an export to Snowflake fails because of code issues, you might have to update the model. Then, you can [push a new version](#push-models-to-snowflake).

<Note>
  If you push a new version, the version number increments the next time you export the model.
</Note>

## Next steps

[Integrate model deployment with CI/CD workflows](/cloud/platform-capabilities/features/models/integrate-ci-cd-workflows)


# Monitor workflows
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/1-monitor-workflows

Choose Domino endpoint monitoring or Model Monitor based on deployment location and how prediction, training, and ground truth data are managed.

In Domino, you can monitor models deployed on Domino and elsewhere. For Domino models deployed as Domino endpoints, you can use our Domino endpoint monitoring workflow, so you don’t have to maintain access to external data sources. For models deployed in other formats (such as an [App](/cloud/platform-capabilities/features/apps), [Launcher](/cloud/platform-capabilities/features/launchers), or [Job](/cloud/platform-capabilities/core-concepts/jobs)) or external to Domino, you can use our Model Monitor workflow.

Use Domino’s Model Monitor to see a single list of monitored models no matter how they are deployed.

## Domino endpoint monitoring

When a model is deployed on Domino as a Domino endpoint, Domino:

* Analyzes the training data to extract the model schema (if you register a Domino [TrainingSet](/cloud/reference/api/data_api/trainingsets_usecases/create_trainingset)).

* Captures predictions as Domino datasets for monitoring.

* Generates drift detection and model quality analysis on a schedule (if you share the ground truth dataset with Domino), and alerts you if any thresholds are exceeded.

* Allows you to easily [reproduce the environment](/cloud/platform-capabilities/features/monitoring/6-remediation/2-remediate-a-domino-endpoint#reproduce-the-environment) with access to the captured predictions to diagnose and fix issues with your model.

See [Set up monitoring for Domino endpoints](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints).

If you do not want Domino to manage the prediction data collection, use the Model Monitor to configure monitoring, even for Domino endpoints.

## Model Monitor

For models deployed as other assets on Domino ([App](/cloud/platform-capabilities/features/apps), [Launcher](/cloud/platform-capabilities/features/launchers), or [Job](/cloud/platform-capabilities/core-concepts/jobs)) or external to Domino, you can use Domino to:

* Connect to the data source where the training, prediction, and ground truth data reside.

* Register a model’s entry along with its schema.

* Set up drift detection and model quality monitoring by registering the location of every new batch of prediction or ground truth data.

* Set up a schedule for Domino to run drift and model quality checks periodically and alert you if thresholds are exceeded.

See [Set up Model Monitor](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor).

Contact your Domino support person if you need more help in determining the best workflow.


# Troubleshoot the Model Monitor
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/10-troubleshoot

Resolve Model Monitor delays, missing data, disabled monitoring, unusable Training Set versions, package errors, and notification failures.

This topic provides solutions for common error messages you might see in the model monitoring interface.

## Data ingestion taking longer than expected

Data is ingested sequentially. When multiple users and/or models ingest data at the same time, the ingest jobs are queued and executed sequentially. This can cause ingest jobs to take longer than expected, occasionally delaying ingesting a small dataset if a larger dataset is ahead of it in the queue.

## Model not running

If your model is still building or starting, wait for its status to change to **Running**.

If your model is stopped, use the following steps to start the model:

1. In the Domino endpoint, click **Versions**.

2. In the **Actions** column for the version you want to start, click the three dots to open the menu.

3. Click **Start Version**.

4. Wait for the model status to change to **Running**.

## Monitoring is not enabled

This happens for the following reasons:

* Your model’s prediction data might not be configured.

* Your model’s `predict()` function might not include the `domino.log()` function needed to enable monitoring. See [Set up Prediction Capture](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/1-set-up-prediction-capture) for instructions.

* Your model’s [training data](/cloud/reference/api/data_api/trainingsets_usecases/create_trainingset) might not be configured. Training data is required for monitoring data drift. See [Set up Drift Detection](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/2-set-up-drift-detection).

* Your model’s ground truth data might not be configured. Ground truth data is required for monitoring model quality. See [Set up Model Quality Monitoring](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/3-set-up-model-qual-monitor).

* The first scheduled check might not have occurred yet. Check the [monitoring schedule](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/6-set-scheduled-checks) to see when the next monitoring data check occurs.

  <Note>
    [Prediction Data](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/1-set-up-prediction-capture) is analyzed through 23:59 of the previous day. Data from the current day is not included. Domino reads the timestamps in the dataset, if they are present; if not, then it uses the ingestion timestamp.
  </Note>

## Waiting for data

The Model Monitor is waiting for sufficient data to start rendering monitoring results.

1. From the Domino endpoint, click **Grafana Monitoring.**

2. Click **Configure monitoring > Target Ranges**. Go to **Date Filter > Today** to view the data that has been analyzed so far.

## Model quality monitoring is not enabled

Model quality is based on ground truth data. If you see this message on your model’s monitoring page, then your model’s ground truth data might not be configured. See [Set up Model Quality Monitoring](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/3-set-up-model-qual-monitor).

## Drift monitoring is not enabled

If you see this message on your model’s monitoring page, then your model’s training data might not be set up. See [Set up Drift Detection](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/4-set-up-drift-detection-mm).

## Selected training set version cannot be used

* The selected Training Set Version cannot currently be used for monitoring because it doesn’t contain a schema definition.

* The Training Set version you selected might be empty, you might have selected the wrong model type, or you might need to also add Target to the Categorical feature while creating the Training set. See [Set up Drift Detection](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/4-set-up-drift-detection-mm).

## Error in library(“DominoDataCapture”)

`Error in library(“DominoDataCapture”): there is no package called ‘DominoDataCapture`

Your project’s environment does not include the prediction data capture library. To add it, do one of the following:

* [Change your project’s compute environment to the 5.0 DSE](/cloud/platform-capabilities/core-concepts/projects/project-settings#set-the-default-environment).

* [Install Packages for Model Monitoring](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/1-install-packages-for-model-monitoring).

## Notifications not being sent

See [Configure Notifications for Model Monitoring](/cloud/admin/platform-configuration/manage-model-deployments#model-monitoring-notifications).


# Data drift and quality monitoring
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/2-data-drift-and-model-quality

Compare production and training distributions for data drift, measure predictions against ground truth, and use Cohort Analysis to target remediation.

Domino monitors data drift and model quality.

<img alt="Data drift metrics compare training data and prediction data, while model quality metrics compare prediction data and ground truth data" />

## Data drift

Data drift occurs when production data diverges from the model’s original training data. Data drift can happen for many reasons, including a changing business environment, evolving user behavior and interest, modifications to data from third-party data sources, data quality issues, and even issues in upstream data processing pipelines. Data drift monitoring compares live predictions with the model’s training data, and then sends an alert when live predictions diverge too much from the training data. See [Analyze Data Drift](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/3-analyze-data-drift).

## Model quality

Model quality monitoring compares the model’s predicted values against the actual results (or labels for the predictions) using ground truth data to generate quality metrics. For classification models, Domino reports the following metrics:

1. Accuracy

2. Precision

3. Recall

4. F1

5. AUC ROC

6. Log Loss

7. Gini (Normalized)

Domino also provides a confusion matrix and classification report for further quality evaluation.

For regression models, Domino reports the following metrics.

1. Mean Square Error (MSE)

2. Mean Absolute Error (MAE)

3. Mean Absolute Percentage Error (MAPE)

4. R-Squared (R2)

5. Gini (Normalized)

See [Metrics for classification and regression models](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/4-analyze-model-quality#metrics-for-classification-and-regression-models) for details.

See [Analyze Model Quality](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/4-analyze-model-quality).

## Remediation

When data drift or model quality issues are detected, the Cohort Analysis feature identifies underperforming cohorts of data so you can take remedial action. See [Cohort Analysis](/cloud/platform-capabilities/features/monitoring/6-remediation/1-review-cohort-analysis).

Use Domino to [recreate the development environment](/cloud/platform-capabilities/features/monitoring/6-remediation/2-remediate-a-domino-endpoint#reproduce-the-environment) originally used to train the model. Then, you can use this environment to access the prediction data to diagnose issues, update code, and retrain your model with the latest production data.


# Set up prediction capture
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/1-set-up-prediction-capture

Instrument Python or R endpoint code with DataCaptureClient, test capture in developer mode, and validate stored prediction data.

Prediction data is a combination of the inputs to the model and the predictions that are output from the model. Inputs are the values of the features that were input as [API requests](/cloud/platform-capabilities/features/monitoring/9-model-monitoring-apis) into the Domino endpoint. When you incorporate a Domino-provided data capture library in your Domino endpoint code, Domino automatically captures the prediction data.

The data ingestion client is part of the Domino Standard Environment (DSE) with the latest version of the client library. The client library records prediction data for deployed models.

## Capture prediction data

To add the DominoDataCapture library details to your model, you must add the following lines so that this logic is executed when the model is deployed. See [call a Domino endpoint](/cloud/platform-capabilities/features/model-deployment#synchronous-requests) for details.

1. In the top navigation pane, click **Develop > Projects**.

2. In the navigation pane of your project, click **Workspaces**.

3. Start the appropriate workspace.

4. Edit your prediction code:

   Edit your prediction code to add the DataCaptureClient. See the following examples for Python and R:

```python theme={null}
import datetime
import uuid

from domino_data_capture.data_capture_client import DataCaptureClient

feature_names = ["dropperc", "mins", "consecmonths", "income", "age"]
feature_values = ["dropperc", "mins", "consecmonths", "income", "age"]

features_dict = dict(zip(feature_names, feature_values))

predict_names = ["y"]
predict_values = [100]

predict_dict = dict(zip(predict_names, predict_values))

# Record eventID and current time
event_id = uuid.uuid4()
event_time = datetime.datetime.now(datetime.timezone.utc).isoformat()

# Custom metadata I want to track for this event
metadata_names = ["cohort"]
metadata_values = ["cohort_id"]
prediction_probability = [0.1, 0.9]
sample_weight = 0.3

data_capture_client = DataCaptureClient(feature_names, predict_names, metadata_names)

data_capture_client.capturePrediction(
    feature_values,
    predict_values,
    metadata_values=metadata_values,
    event_id=event_id,
    timestamp=event_time,
    prediction_probability=prediction_probability,
    sample_weight=sample_weight,
)
```

```r theme={null}
library(“DominoDataCapture”)
data_capture_client <- DataCaptureClient(feature_names, predict_names, metadata_names)
data_capture_client.capturePrediction(feature_values, predict_values, metadata_values, event_id, timestamp, prediction_probabily, sample_weight)
```

The following table explains the parameters from the `DataCaptureClient` statement:

| Parameter | Type | Description |
| - | - | - |
| `feature_names` | `Array[String]` | The feature names against which the user calculates the prediction. |
| `predict_names` | `Array[String]` | The prediction names collection. This value must be an array. |
| `metadata_names` (Optional) | `Array[String]` | Collection of any metadata keys to pass. |

The following table explains the parameters from the `capturePrediction` statement:

| Parameter | Type | Description |
| - | - | - |
| `feature_values` | `Array[float/String]` | The feature values against which the user calculates the prediction. |
| `predict_values` | `Array[float/String]` | The prediction values collection. This value must be an array. |
| `metadata_values` | `Array[float/String]` | Collection of any metadata values to pass. |
| `event_id` (Optional) | `String` | A unique record ID for each prediction. If not provided the client library generates one. |
| `timestamp` (Optional) | `Int` | The event timestamp. If not provided the client library generates it. |
| `prediction_probability` (Optional) | `Array[float]` | The collection of prediction probabilities. This value must be an array. |
| `sample_weight` (Optional) | `Array[float]` | The collection of associated sample weights. This value must be an array. |

### Run DataCaptureClient

Use the `DominoDataCapture` library to capture prediction data in the Domino endpoint or in developer mode. Use developer mode to test the library calls to verify that the data capture works, without actually capturing data. After verifying that the data capture works, you must invoke the Domino endpoint code in a workspace (for example, an iPython notebook), where you can review the output of the library calls, validate, and debug the code.

<Note>
  Domino endpoints support capturing up to 8 GB of data per 24-hour period. Bursting has been tested up to five times this limit, beyond which there might be errors or warnings in the Domino endpoint log.
</Note>

**Step 1: Run DataCaptureClient in developer mode**

<Tabs>
  <Tab title="Run an example in Python">
    1. Open a Python Prediction Client workspace.

    2. Go to **New > Python3**.

    3. Add the following lines and update them for your model:

       * Import the predict function: `from python_model_with_logging import *`

       * Invoke the predict method with parameters: `predict_iris_variety(5.3,3,1.1,0.1, 1)`
  </Tab>

  <Tab title="Run an example in R">
    1. Open R Studio.

    2. Go to **File > New File > R Notebook**.

    3. Add the following lines and update them for your model:

       * The source for the model instrumented with data capture: `source("model_to_predict_iris_variety.R")`

       * Invoke the predict method with parameters: `predict_iris_variety(5.3,3,1.1,0.1, 1)`

    4. Click **Run** and view the console.
  </Tab>
</Tabs>

**Step 2: Run DataCaptureClient in Domino endpoint**

The following are examples of models that use Domino data capture:

```python theme={null}
import datetime
import pickle
import pandas as pd
import uuid

from sklearn import metrics
from domino_data_capture.data_capture_client import DataCaptureClient

feature_names = ['sepal.length', 'sepal.width', 'petal.length', 'petal.width']
predict_names = ['variety']
pred_client = DataCaptureClient(feature_names, predict_names)

loaded_model = pickle.load(open("model.pkl", 'rb'))

def predict_iris_variety(sepal_length, sepal_width, petal_length, petal_width, event_id):
    feature_values = [sepal_length, sepal_width, petal_length, petal_width]
    predict_values = loaded_model.predict([feature_values])

    event_time = datetime.datetime.now(datetime.timezone.utc).isoformat()

    pred_client.capturePrediction(feature_values, predict_values, event_id=event_id, timestamp=event_time)

    return dict(predict_value=predict_values[0])
```

```R theme={null}
library("randomForest")
library("DominoDataCapture")

training_data <- read.csv(
 "iris.csv",
 header = TRUE
)

training_data$variety <- factor(training_data$variety)
iris_rf <- randomForest(
 variety ~ .,
 data = training_data,
 importance = TRUE,
 proximity = TRUE
)

data_capture_client <- DataCaptureClient(
 feature_names = c("sepal.length", "sepal.width", "petal.length", "petal.width"),
 predict_names = c("variety")
)

predict_iris_variety <- function(sepal_length, sepal_width, petal_length, petal_width, event_id) {
 mytestdata <- data.frame(
   "sepal.length" = sepal_length,
   "sepal.width" = sepal_width,
   "petal.length" = petal_length,
   "petal.width" = petal_width
 )
 predicted_value_factor <- predict(iris_rf, mytestdata)
 predicted_value <- as.character(predicted_value_factor)
 data_capture_client$capturePrediction(
   feature_values = c(sepal_length, sepal_width, petal_length, petal_width),
   predict_values = c(predicted_value),
   event_id = event_id
 )
 return(list(variety = predicted_value))
}
```

### Data capture examples

See more examples of MLflow-supported models that use Domino data capture:

1. [Registering XGBoost model with integrated model monitoring](https://github.com/dominodatalab/reference-project-domino-mlflow-supported-models/blob/main/domino-mlflow-model-xgboost-imm.ipynb).

2. [Registering Sklearn model with integrated model monitoring](https://github.com/dominodatalab/reference-project-domino-mlflow-supported-models/blob/main/domino-mlflow-model-sklearn-imm.ipynb).

## (Optional) customize Domino environments

If you want to use a specific version of the client library, or enable client libraries in another environment:

1. In your Environment, click **Edit Definition**.

2. In the **Dockerfile Instructions**, add the following lines to enable the library:

   ```python theme={null}
   USER root
   RUN pip install domino-data-capture
   USER ubuntu
   ```

   ```r theme={null}
   RUN R --no-save -e "install.packages(c(DominoDataCapture))"
   ```

3. Select **Full rebuild without cache** and click **Build**.

4. From the navigation bar, click **Endpoints**.

5. Click **New endpoint** and create the endpoint from the newly built image.

## Test the Domino endpoint

See [Validate your Setup](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/2-set-up-drift-detection#validate-your-setup) to confirm your prediction data is being captured.

After you [publish your Domino endpoint](/cloud/platform-capabilities/features/model-deployment#deploy-a-model) and it is running, call the Domino endpoint to capture prediction data.

1. Go to the Domino endpoint to test.

2. From the **Tester** tab, enter the values from your model’s schema.

3. Click **Send**. The **Response** field shows a prediction in the form of key-value pair.

After the logged predictions are captured and processed by Domino, you can see a preview of the drift results. See [Validate Your Setup](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/2-set-up-drift-detection#validate-your-setup) for more information.

## Advanced configuration

To tune the prediction data capture feature, please reach out to Domino Support.


# Domino endpoint drift detection
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/2-set-up-drift-detection

Register a Training Set for a Domino endpoint, select classification or regression, and validate that prediction data is ready for drift analysis.

To monitor a model’s data drift you must ingest training data. Training data is the data used while constructing a model that leverages machine learning techniques. Training data records must include both model input values and a previously known corresponding output for each input value combination from which the model can learn.

Your model’s training data is used to [analyze data drift](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/3-analyze-data-drift). This topic describes how to register a training set for monitoring purposes.

## Prerequisites

Before you proceed, you must create a [training set](/cloud/reference/api/data_api/trainingsets_usecases/create_trainingset) that was used to build a model (`.pkl` file), and then that model was deployed into a Domino endpoint.

## Select the training set

1. In the top navigation pane, click **Deploy > Endpoint**.

2. Select the endpoint you would like to configure and navigate to the **Grafana Monitoring** tab.

3. Click **Configure Monitoring > Data**.

4. From the Configure Data window, select the **Training Data** and **Version**.

5. Select **Classification** or **Regression**, depending on your model type. If your Training Set code includes prediction data defined in `target_columns`, select the model type that matches your Training Set:

   * If `target_columns` is a `categorical` column, select **Classification**.

   * If `target_columns` is a `numerical` column, select **Regression**.

6. Click **Save**.

<Note>
  When using the **Configure Data** window to register a model for monitoring, the default binning strategy is used to create bins for both numerical and categorical variables. To change the binning strategy, including the number of bins to use for numerical or categorical variables, reach out to your Domino representative. See [Supported binning methods](/cloud/platform-capabilities/features/monitoring/8-json/1-supported-binning) for more details on binning.
</Note>

## Validate your setup

Domino captures prediction data in a Domino Dataset in hourly batches.

Validate your setup without waiting for the Scheduled Check to run:

1. On the Endpoint’s Monitoring tab, go to **Configure Monitoring > Target Ranges**.

2. If the Configure Tests and Thresholds page populates with data, then your setup is complete. See [Analyze Data Drift](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/3-analyze-data-drift) to learn how to configure tests and thresholds.


# Set up model quality monitoring
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/3-set-up-model-qual-monitor

Register ground truth data for an endpoint with a JSON config and row identifiers, then verify ingestion and calculate model quality metrics.

Ground truth data consists of known correct outputs for each input. Model quality monitoring compares the model’s predicted values against the actual results (or labels for the predictions) using ground truth data to generate quality metrics.

After you set up your endpoint to [capture prediction data](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/1-set-up-prediction-capture), you can ingest the model’s ground truth data to monitor the quality of predictions made by the model. For this analysis, you must have declared a `row_identifier` column for the prediction data schema. This `row_identifier` column must also be present in the ground truth data to map a prediction uniquely to the ground truth.

When you apply a date filter, Domino uses the timestamp values in the prediction data to filter the results. Then, it matches the filtered predictions with the ground truth labels ingested in the last 90 days and calculates the model quality metrics for the matched predictions.

## Ingest ground truth data

You can use the Domino application or a REST API to ingest ground truth data. For details about using the REST API, see [GET /v2/api/model/\{model\_id}/traffic/ground-truth](/cloud/platform-capabilities/features/monitoring/9-model-monitoring-apis).

1. In the navigation pane, click **Model Monitor**.

2. Click the model for which you want to ingest ground truth data.

3. Click **Model Quality**.

4. On the Model Quality page of the registered model, click **Register Ground Truth \< Upload Ground Truth Config**.

   In the Register Ground Truth window, you can upload a Ground Truth Config file or paste a JSON config. This JSON must capture all the information needed to register ground truth data.

   The following is a sample file. Your file must define the `variableType` as `ground_truth`. Do not include any other variableTypes. The configuration file must always include the `datasetDetails` and `modelMetadata` information. See [Monitoring config JSON](/cloud/platform-capabilities/features/monitoring/8-json) for details about each field in the JSON.

   ```console theme={null}
   {
   "variables": [
     {
       "name": "y_gt",
       "variableType": "ground_truth",
       "valueType": "categorical",
       "forPredictionOutput": "y"
     }
   ],
   "datasetDetails": {
     "name": "GT-labels.csv",
     "datasetType": "file",
     "datasetConfig": {
       "path": "GT-labels.csv",
       "fileFormat": "csv"
     },
     "datasourceName": "abc-shared-bucket",
     "datasourceType": "s3"
   }
   }
   ```

5. Click **Register Ground Truth**. On the Model Quality Analyze page, the Model Monitor starts to calculate model quality metrics for this dataset.

6. Click the bell icon to disable the alerts. Doing this reduces noise by controlling the metrics included in the Scheduled Checks.

### Check the status of ground truth data

After you register the ground truth data for ingestion, you might want to check its status.

1. From the navigation pane, click **Model Monitor**.

2. Click the name of the model for which you set up ground truth data ingestion.

3. Click **Ingest History** to check to see if the status is Done.

After the data is ingested, you can perform an ad-hoc [model quality analysis](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/4-analyze-model-quality) to validate your setup. Then, [set up a schedule](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/6-set-scheduled-checks) to perform model quality analysis automatically.


# Set up Cohort Analysis for Domino endpoints
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/4-set-up-cohort-analysis

Enable Cohort Analysis for a regression endpoint, verify data ingestion, and tune cohort counts, samples, bins, and report output in config.yaml.

Cohort analysis provides information about your model, including feedback about what might be going wrong with it. Cohort analysis is a job that runs on the model’s prediction and ground truth data for a regression model. The result is a PDF report and raw data in JSON format.

To use Cohort Analysis, enable data analysis for each regression model. Doing this for a model starts a series of ingest and analysis jobs.

<Note>
  This feature is only available for regression models with numerical features.
</Note>

## Prerequisites

* [Set up prediction data capture](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/1-set-up-prediction-capture)

* The model is [published](/cloud/platform-capabilities/features/model-deployment#deploy-a-model).

* The Domino endpoint is running.

* [Ground truth data](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/3-set-up-model-qual-monitor) is available.

* The model is a regression type model

## Set up Cohort Analysis for a Domino endpoint

1. From the navigation pane, click **Domino endpoint**.

2. Click the name of the model for which you want to set up Cohort Analysis.

3. Click **Grafana Monitoring**.

4. Click **Configure Monitoring > Data**.

5. On the Configure Data page, click **open this model** to open the configuration in the Model Monitor.

6. Click **Register Ground Truth Data > Upload Ground Truth Config**.

7. In the Register Ground Truth window, click **Register with Cohort Analysis**.

## Check the status of the data ingestion

After you set up the data analysis for a regression model, you might want to check its status.

1. From the navigation pane, click **Model Monitor**.

2. Click the name of the model for which you set up Cohort Analysis.

3. Click **Ingest History** to check to see if the status is Done.

After the data is ingested, Domino finds underperforming cohorts and the features that make those cohorts distinct from the rest of the data.

## Configure the Cohort Analysis

You can configure the cohort analysis for a model to customize the report.

### Prerequisites

* Check the status of the [data ingestion](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/4-set-up-cohort-analysis#check-the-status-of-the-data-ingestion) to confirm the data analysis is complete.

### Configure the Cohort Analysis

1. In the navigation pane, click **Projects**.

2. Click the **DominoCohortAnalysis** project. The DominoCohortAnalysis Project is created automatically as a private project under the model’s project owner’s account.

3. In the navigation pane, click **Code**.

4. Click `config.yaml` and then click **Edit**.

   You can configure the following parameters:

   **min\_k**\
   The minimum number of cohorts.

   **max\_k**\
   The maximum number of cohorts.

   **max\_samples\_for\_clustering**\
   The maximum number of samples to use to find cohorts

   **num\_bins**\
   The number of bins to use to compute the feature histograms and the contrast score.

   **max\_num\_top\_cohorts\_for\_report**\
   The maximum number of cohorts to show in the Cohort Summary and Detailed Cohort Analysis sections of the Cohort Analysis report.

   **max\_num\_top\_cohorts\_for\_report**\
   The maximum number of features to show per cohort in the Detailed Cohort Analysis section of the Cohort Analysis report.

5. Click **Save**.

6. In the navigation pane, click **Jobs**.

7. Select the **cohort\_analysis** job and click **Run** to generate new results.


# Domino endpoint notifications
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/5-set-up-notifications

Configure email recipients for an endpoint so scheduled drift and model quality checks notify them when monitored thresholds are breached.

A scheduled check is an automated instance of a running drift or model quality computation over a range of data. You can view the results on the monitoring dashboard, and automated email notifications are sent if thresholds are breached.

This topic describes how to configure notifications.

## Set up notifications for a specific Domino endpoint:

1. In the navigation pane, click **Endpoints**.

2. Click the name of the model for which you want to set up notifications.

3. Click **Grafana Monitoring**.

4. Click **Configure Monitoring > Alert Recipients**.

5. Type or paste a list of comma-separated or semicolon-separated email addresses.

6. Click **Save**.


# Domino endpoint scheduled checks
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/6-set-scheduled-checks

Schedule endpoint drift and model quality checks by frequency and data window, then preserve captured predictions for diagnosis and retraining.

Use Scheduled Checks to ensure that you are notified if data drift or model quality metrics degrade beyond the threshold for any period. For data drift, Domino uses the timestamps of the predictions to select data for the scheduled checks. For model quality, Domino uses the timestamp of the prediction data to select the prediction and ground truth data on which to run periodic model quality checks. You can specify how often to repeat the checks and the time range of the data to be used for calculations for those checks. When a check fails, emails are sent. See [Set up Notifications](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/5-set-up-notifications).

## Prerequisites

You must have ground truth data available for the date ranges selected for the Schedule Checks. To do this, set up the [data source](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/2-connect-a-data-source) and notify Domino of new [ground truth data](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/3-set-up-model-qual-monitor#ingest-ground-truth-data).

## Create a scheduled check

1. From the navigation pane, click **Endpoints**.

2. Click the name of the model for which you want to set up scheduled checks.

3. Click **Grafana Monitoring**.

4. Go to **Configure Monitoring > Schedule**.

5. Type a name for the check.

6. Set up the frequency at which the check must run.

7. In the **Select Data to Check** area, select one of the following:

   <Warning>
     * When you make a selection, remember that you must update the data manually and ensure that there is enough time for the data to be ingested. For example, if you set the check to repeat every day at 11 PM but you load new ground truth data at 10:45 PM, this might not be enough time for the data to be ingested.

     * If you select **Use new data since last check**, the system assumes new data was ingested. If none was ingested, you will see an empty dashboard for the specified date and time on the Analyze tab.
   </Warning>

   * **Use new data since last check time**

     * For data drift, this checks predictions with timestamps later than the last scheduled check.

     * For model quality, this checks only ground truth labels ingested into the Model Monitor after the last scheduled check ran and matches them with historical predictions made by the model.

   * **Data since last x \<time period>**

     * For data drift, this checks predictions whose timestamps are within the last specified interval (for example, the last three days). For model quality, this checks only the ground truth labels ingested within the last specified interval (for example, within the last three days), and matches them with historical predictions made by the model.

8. Click **Save**.

Domino automatically creates data in a Domino Dataset named `prediction_data` for every project. When you [reproduce a workspace](/cloud/platform-capabilities/features/monitoring/6-remediation/2-remediate-a-domino-endpoint#reproduce-the-environment), this data is included. Predictions are in Parquet format and are updated hourly as the Domino endpoint processes inputs. All the data captured is stored in the file and populated in a workspace so you can diagnose flaws in the model and retrain the model as needed.


# Set up monitoring for Domino endpoints
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/index

Set up endpoint monitoring with prediction capture, a Training Set, drift detection, ground truth data, notifications, checks, and Cohort Analysis.

You must set up model monitoring for each Domino endpoint that you want to monitor.

## Prerequisites

* When you [trained the model](/cloud/reference/api/data_api/trainingsets_usecases/create_trainingset), you registered or used a Domino Training Set in the same project from which you’re looking to publish a Domino endpoint.

* If you perform Model Quality analysis, set up your data sources for ground truth data. See [Connect a Data Source](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/2-connect-a-data-source).

### Set up monitoring for Domino endpoints

The following topics explain the steps:

1. [Set up Prediction Data Capture](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/1-set-up-prediction-capture). Prediction data is required for monitoring both data drift and model quality.

2. [Publish the Domino endpoint](/cloud/platform-capabilities/features/model-deployment#deploy-a-model). Domino starts to log prediction data (and convert it to Parquet files), but it does not yet produce monitoring data.

3. [Set up Drift Detection](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/2-set-up-drift-detection). Register a training set to monitor data drift.

   * Optional: Configure [Set up Notifications](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/5-set-up-notifications) or change the [scheduled checks](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/6-set-scheduled-checks).

4. [Validate your Setup](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/2-set-up-drift-detection#validate-your-setup). Confirm that your predication data is being captured.

5. [Set up Model Quality Monitoring](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/3-set-up-model-qual-monitor). Ingest ground truth data to monitor the quality of the model’s predictions.

   * Optional: Configure [Set up Notifications](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/5-set-up-notifications) or change the [scheduled checks](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/6-set-scheduled-checks).

6. [Set up Cohort Analysis](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/4-set-up-cohort-analysis). Get insights into model quality so you can find underperforming data hotspots for model remediation.


# Install packages for Model Monitoring
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/1-install-packages-for-model-monitoring

Install the Domino Data Capture client for Python or R and the Python-only Domino Data API in an Environment used for model monitoring.

Model Monitoring requires specific packages be included in your Environment.

You can either use a [Domino Standard Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) from your current version, or add the below packages to your existing environments.

## Install the Domino data capture client

You must use the Domino Data Capture client for integrated model monitoring.

Install it as follows:

```python theme={null}
RUN pip install domino-data-capture
```

```r theme={null}
RUN R -e "install.packages(c('DominoDataCapture'))"
```

## Install the Domino Data API

The [Domino Data API](/cloud/reference/api) currently supports Python only.

Install it as follows:

```shell theme={null}
RUN python -m pip install "dominodatalab[data]"
```


# Configure Model permissions
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/10-model-permissions

Make monitored models private, add Viewer or Editor collaborators, change their roles, and remove access through Model Monitor permissions.

By default, models registered for monitoring are viewable and editable by all licensed Domino users. The model owner can choose to make a model private, and limit access to users that have been explicitly added as collaborators.

Model permissions can only be configured by the owner of the model or a Domino administrator.

Start by navigating to the model permissions page:

1. In the navigation pane, click **Model Monitor**.

2. Click the name of the model for which you want to configure permissions.

3. Click **Permissions**.

<Note>
  Monitoring permissions for [models deployed as REST APIs](/cloud/platform-capabilities/features/model-deployment) can be configured from the [Domino endpoint collaborator settings](/cloud/platform-capabilities/features/model-deployment/security#add-collaborators).
</Note>

**Make a model private:**

* Turn off the **Enable access for all licensed Domino users** option.

  **Add collaborators:**

  1. Find and select the desired users by typing usernames into the search box.

  2. Select between **Viewer** and **Editor** access for the users being added.

  3. Click the **Add** button.

**Change collaborator role:**

1. Find the desired collaborator in the **Current Collaborators** list.

2. In the **Role** column, select the desired role from the dropdown.

**Remove a collaborator:**

1. Find the desired collaborator in the **Current Collaborators** list.

2. Click the **Delete** icon.


# Unregister a Model
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/11-unregister-a-model

Unregister a model to remove it and its checks from monitoring, correct its Monitoring Config JSON, and register the model again.

If you must modify the [Monitoring Config JSON](/cloud/platform-capabilities/features/monitoring/8-json), you can unregister the model. Make the necessary corrections to Config JSON and then [register the model](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/3-register-a-model) again.

1. From the navigation pane, click **Models**.

2. On the Models page, go to the model that you want to unregister.

3. Click the three vertical dots at the end of the model’s row, and click **Unregister**.

4. Click **Yes** to confirm your action. The model and its checks are removed from monitoring.


# Connect a Data Source
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/2-connect-a-data-source

Connect Model Monitor to S3, Azure Blob, Azure Data Lake, HDFS, or Snowflake and configure each source’s required authentication.

Model monitoring ingests and processes a model’s training, prediction, and ground truth data to monitor the model. You must set up the data source from which the data is read. After you set up the data source, it is available to all users and can be used in any model.

You can have multiple data sources linked to the model monitor, and multiple instances for each data source type.

<Note>
  Enable **read** and **list** access for each data source.
</Note>

The Model Monitor supports data from the following sources:

* Amazon S3

* Generic S3

* Azure Blob

* Azure Data Lake Gen 1

* Azure Data Lake Gen 2

* Google Cloud Storage

* HDFS

* Snowflake

## Connect a Data Source

1. If you haven’t set up a data source, go to the **Data** page (from the navigation pane, go to **Model Monitor > Monitoring Data Sources**).

2. Click **Add Data Source**.

3. Complete the details as needed. The following describes source-specific configurations:

   <table>
     <thead>
       <tr>
         <th>Data source</th>
         <th>Required fields</th>
         <th>Authentication</th>
       </tr>
     </thead>

     <tbody>
       <tr>
         <td><p>Amazon S3</p></td>
         <td><p>Data Source Name</p> <p>S3 Bucket Name</p> <p>S3 Region</p></td>
         <td><p>If the S3 buckets can be authenticated to using IAM roles, select the <strong>Load Credentials from IAM Role</strong> attached to the instance checkbox.</p> <p>Enter an <strong>AWS Access Key</strong> and an <strong>AWS Secret Key</strong>.</p></td>
       </tr>

       <tr>
         <td><p>Generic S3</p></td>
         <td><p>Data Source Name</p> <p>S3 Bucket Name</p> <p>S3 Endpoint</p></td>
         <td><p>Enter an <strong>Access Key</strong> and a <strong>Secret Key</strong>.</p></td>
       </tr>

       <tr>
         <td><p>Azure Blob Store</p></td>
         <td><p>Data Source Name</p> <p>Account name</p> <p>Container</p></td>
         <td><p>Access Key</p> <p>SaaS Token</p></td>
       </tr>

       <tr>
         <td><p>Azure Data Lake Gen 1</p></td>
         <td><p>Data Source name</p> <p>Container</p></td>
         <td><p>Select the <strong>Authentication Type</strong>:</p> <ol> <li><p>If you select <strong>Client Credentials</strong>, enter:</p> <ul> <li><p>Token Endpoint</p></li> <li><p>Application ID</p></li> <li><p>Secret Key</p></li> </ul></li> <li><p>If you select <strong>Managed Identity</strong> , you can enter an optional <strong>Port Number</strong>.</p></li> </ol> <p>This method applies when the Model Monitor is deployed on Azure VMs configured with service identities that can access Azure Data Lake.</p></td>
       </tr>

       <tr>
         <td><p>Azure Data Lake Gen 2</p></td>
         <td><p>Data Source name</p> <p>Account Name</p> <p>Container</p></td>
         <td><p>Select the <strong>Authentication Type</strong>:</p> <ol> <li><p><strong>Shared Key</strong>.</p></li> <li><p><strong>Client Credentials</strong>. Then, enter:</p> <ul> <li><p><strong>Endpoint</strong></p></li> <li><p><strong>Client ID</strong></p></li> <li><p><strong>Client Secret</strong></p></li> </ul></li> <li><p><strong>Username Password</strong>. Then, enter:</p> <ul> <li><p><strong>Endpoint</strong></p></li> <li><p><strong>Username</strong></p></li> <li><p><strong>Password</strong></p></li> </ul></li> <li><p><strong>Refresh Token</strong>. Then, enter:</p> <ul> <li><p><strong>Refresh Token</strong></p></li> <li><p><strong>Client ID</strong></p></li> </ul></li> <li><p><strong>Managed Identity</strong>. This method applies when the Model Monitor has been deployed on Azure VMs configured with service identities that can access Azure Data Lake Gen2. Then, you can enter the following optional information:</p> <ul> <li><p><strong>Tenant ID</strong></p></li> <li><p><strong>Client ID</strong></p></li> </ul></li> </ol></td>
       </tr>

       <tr>
         <td><p>Google Cloud Storage</p></td>
         <td><p>Data Source Name</p> <p>Bucket</p></td>
         <td><p>JSON Key File</p></td>
       </tr>

       <tr>
         <td><p>HDFS</p></td>
         <td><p>Data Source Name</p> <p>Host</p> <p>Port (optional)</p></td>

         <td />
       </tr>

       <tr>
         <td><p>Snowflake</p></td>
         <td><p>All fields</p></td>
         <td><p>Enter the following:</p> <ol> <li><p><strong>Account URL</strong> which uniquely identifies the Snowflake account in your organization.</p></li> <li><p>Enter your <strong>Username</strong> and <strong>Password</strong>.</p></li> <li><p>In <strong>Database</strong>, enter the name of the Snowflake database that contains the data.</p></li> <li><p>In <strong>Schema</strong>, enter the name of the active schema for the session.</p></li> <li><p>In <strong>Warehouse</strong>, enter the name of a compute resource cluster that provides the resources in Snowflake.</p></li> <li><p>Select a <strong>Role</strong>.</p>&#x20;
         &#x20;<p>All fields must match Snowflake’s requirements for <a href="https://docs.snowflake.com/en/sql-reference/identifiers-syntax.html">object identifiers</a>.</p> </li> </ol></td>
       </tr>
     </tbody>
   </table>

4. Click **Add**.


# Register a Model
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/3-register-a-model

Register a model with Monitoring Config JSON that defines monitored features, training data location, schema, and model metadata.

Before monitoring a model, it must be registered so that Domino captures information about the model through a [Monitoring Config JSON](/cloud/platform-capabilities/features/monitoring/8-json). Use the JSON config to describe the model’s schema, as well as the source of the training data, and metadata about the model. For models with many features, you can include only those features that you want to monitor.

1. From the navigation pane, click **Model Monitor**.

2. From the Models page, click **Register Model > Upload Model Config File**.

3. In the Register Model window, upload or paste a JSON configuration that describes the features to monitor, the location of the training dataset, and the model metadata.

   The following is a sample file. Your file must define all features that you want to monitor for drift in this configuration. These features must be specified as `“variableType”: “feature”`. Other variableTypes are optional when registering a model, although `sample_weight` and `prediction_probability` don’t apply. The configuration file must always include the datasetDetails and modelMetadata information.

   See [Monitoring Config JSON](/cloud/platform-capabilities/features/monitoring/8-json) for details about each field in the JSON.

   ```console theme={null}
   {
    "variables": [
      {
        "name": "age",
        "valueType": "numerical",
        "variableType": "feature"
      },
      {
        "name": "y",
        "valueType": "categorical",
        "variableType": "prediction"
      },
      {
        "name": "date",
        "valueType": "datetime",
        "variableType": "timestamp"
      },
      {
        "name": "RowId",
        "valueType": "string",
        "variableType": "row_identifier"
      }
    ],
    "datasetDetails": {
      "name": "TrainingData.csv",
      "datasetType": "file",
      "datasetConfig": {
        "path": "TrainingData.csv",
        "fileFormat": "csv"
      },
      "datasourceName": "abc-shared-bucket",
      "datasourceType": "s3"
    },
    "modelMetadata": {
      "name": "test_psg",
      "modelType": "classification",
      "version": "2",
      "description": "",
      "author": "testadmin"
    }
   }
   ```

4. Click **Register Model** and the model is listed on the Models page.


# Drift detection for monitored models
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/4-set-up-drift-detection-mm

Register prediction data with Monitoring Config JSON, monitor ingestion status, and configure drift tests, thresholds, date ranges, and alerts.

[Prediction data](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/1-set-up-prediction-capture#capture-prediction-data) is the output produced by a model. You must add data that was trained on historical data to your model’s code to start ingesting and storing prediction data for monitoring.

<Note>
  You can register prediction data for a model that uses a Snowflake data source once per table.
</Note>

If you registered a `timestamp` variable or if you register it with the ground truth dataset, Domino automatically retrieves new data every 24 hours at 12:00 AM UTC. See [Monitoring Config JSON](/cloud/platform-capabilities/features/monitoring/8-json). When you deregister the model, Domino stops retrieving the data.

If you don’t register the `timestamp` variable, the prediction data is ingested once, and for every new batch, you must register a new dataset.

1. Click the registered model and go to the Data Drift page.

   <Note>
     If you want to do this from the Overview page, go to **Add Data** > **Prediction Data**.
   </Note>

2. Click **Register Prediction \< Upload Prediction Config**.

3. In the Register Prediction window, upload or paste a JSON configuration that describes the prediction variables and the location of the prediction dataset.

   The following is a sample file. This file doesn’t require any variable information. You can include prediction variableTypes, although, you typically include these when you define your training data to get a complete set of metrics. If you define the variables here, you only get drift predictions. You can also define the following variables: `sample_weight`, `prediction_probability`, and `timestamp`. The configuration must always include the `datasetDetails` and `modelMetadata` information.

   See [Monitoring Config JSON](/cloud/platform-capabilities/features/monitoring/8-json) for details about each field in the JSON.

   ```console theme={null}
   {
    "variables": [
      {
        "name": "campaign",
        "variableType": "sample_weight",
        "valueType": "numerical"
      }
    ],
    "datasetDetails": {
      "name": "Predictions.csv",
      "datasetType": "file",
      "datasetConfig": {
        "path": "Predictions.csv",
        "fileFormat": "csv"
      },
      "datasourceName": "abc-shared-bucket",
      "datasourceType": "s3"
    }
   }
   ```

4. Click **Register Prediction** and the Model Monitor starts calculating drift on the registered dataset.

## Check the status of prediction data

After you add the prediction data, you might want to check its status.

1. From the navigation pane, click **Model Monitor**.

2. Click the name of the model for which you set up ground truth data ingestion.

3. On the Overview page’s **Data Traffic** tab, you can see the history of the ingested data and review the model’s metadata and schema.

4. Click **Ingest History** to check to see if the status is `Done`. The following are other statuses you might see:

   **`Completed with Warnings`**\
   Click the three vertical dots at the end of the row and click **View Details**.

   **`Failed`**\
   Click the three vertical dots at the end of the row and click **View Details**.

   If the ingestion failed, Domino stops the continuous ingestion. After you fix the issue, re-register the prediction data and Domino restarts continuous ingestion from this point.

   <Note>
     If you re-register the same dataset, Domino processes the entire table again. This causes duplicate records which affects the drift and model quality calculations.
   </Note>

   **`Skipped`**\
   The dataset had no rows to fetch.

   You can click **Status** to select checkboxes to filter by status.

5. Click **Refresh** to see a graph of the **Data Ingest Timeline**.

   After the data is ingested, you can perform an ad-hoc [model quality analysis](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/3-analyze-data-drift) to validate your setup. Then, [set up a schedule](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/8-set-scheduled-checks-mm) to perform model quality analysis automatically.

## Analyze data drift

After the drift calculations are complete, you can see the divergence value for each feature in the Drift column. The computation used the default **Test Type** and **Threshold** values.

1. To access this page, in the navigation pane, click **Model Monitor**.

2. Go to the model whose data drift you want to see and then click **Data Drift**.

   <Note>
     To change the default settings to experiment with other Test Types, Conditions, and Default Threshold Values, in the navigation pane click **Settings**. Then, click [Test Defaults](/cloud/platform-capabilities/features/monitoring/7-monitor-settings/4-tests), make your changes and click **Save**.
   </Note>

   <img alt="The Data Drift page labeled with A, B, and C" />

   A - Use the Date Filter to refine the data and review the Drift and its trends for specific periods.

   B - If your model had a timestamp column declared, it’s used to get the timestamp of different predictions in the dataset. If the timestamp wasn’t declared, then the data’s ingestion time in the Model Monitor is used as its timestamp.

   C - To reduce alert noise, click the bell icon next to features to disable alerts to exclude them from the Scheduled Checks.

   D - If you made changes to the test types, thresholds, and so on and want to save them as the default configuration for the model to be used for running scheduled checks, go to **Other Actions > Save Checks Config** to save the changes. If you experimented with configurations and wanted to load the model’s default configuration to reset the Analyze table, go to **Other Actions > Restore Checks Config**.


# Set up model quality monitoring
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/5-set-up-model-qual-monitor-mm

Register ground truth data with matching row identifiers, monitor ingestion status, and calculate quality metrics for matched predictions.

Ground truth data consists of known correct outputs for each input. Model quality monitoring compares the model’s predicted values against the actual results (or labels for the predictions) using ground truth data to generate quality metrics.

After you set up your model to [capture prediction data](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/1-set-up-prediction-capture#capture-prediction-data), you can ingest the model’s ground truth data to monitor the quality of predictions made by the model. For this analysis, you must have declared a `row_identifier` column for the prediction data schema. This `row_identifier` column must also be present in the ground truth data to map a prediction uniquely to the ground truth.

When you apply a date filter, Domino uses the timestamp values in the prediction data to filter the results. Then, it matches the filtered predictions with the ground truth labels ingested in the last 90 days and calculates the model quality metrics for the matched predictions.

<Note>
  You can register ground truth data for a model that uses a Snowflake data source once per table.
</Note>

If you registered a `timestamp` variable or if you register it with the ground truth dataset, Domino automatically retrieves new data every 24 hours at 12:00 AM UTC. See [Monitoring Config JSON](/cloud/platform-capabilities/features/monitoring/8-json). When you deregister the model, Domino stops retrieving the data.

If you don’t register the timestamp variable, the ground truth data is ingested once, and for every new batch, you must register a new dataset.

## Ingest ground truth data

You can use the Domino application or a REST API to ingest ground truth data. See [Model Monitoring APIs](/cloud/platform-capabilities/features/monitoring/9-model-monitoring-apis) for details.

1. In the navigation pane, click **Model Monitor**.

2. Click the model for which you want to ingest ground truth data.

3. Click **Model Quality**.

   <Note>
     If you want to do this from the Overview page, go to **Add Data > Ground Truth Data**.
   </Note>

4. On the Model Quality page of the registered model, click **Register Ground Truth > Upload Ground Truth Config**.

   In the Register Ground Truth window, you can upload a Ground Truth Config file or paste a JSON config. This JSON must capture all the information needed to register ground truth data.

   The following is a sample file. You must specify the target column, shown below as `y_gt` and the row identifier that maps to a corresponding prediction record, shown below as `gt_uuid`. Note that your prediction dataset needs to have a column that contains unique row identifiers that match. When you registered your prediction dataset with Domino, you also supplied a variable that denotes the `row_identifier` similarly. The configuration must always include the `datasetDetails` and `modelMetadata` information. See [Monitoring Config JSON](/cloud/platform-capabilities/features/monitoring/8-json) for details about each field in the JSON.

   ```console theme={null}
   {
   "variables": [
     {
       "name": "y_gt",
       "variableType": "ground_truth",
       "valueType": "categorical",
       "forPredictionOutput": "y"
     },
     {
        "name": "gt_uuid",
        "variableType": "row_identifier",
        "valueType": "string"
     }
   ],
   "datasetDetails": {
     "name": "GT-labels.csv",
     "datasetType": "file",
     "datasetConfig": {
       "path": "GT-labels.csv",
       "fileFormat": "csv"
     },
     "datasourceName": "abc-shared-bucket",
     "datasourceType": "s3"
   }
   }
   ```

5. Click **Register Ground Truth**. On the Model Quality Analyze page, the Model Monitor starts to calculate model quality metrics for this dataset.

6. Click the bell icon to disable the alerts. Doing this reduces noise by controlling the metrics included in the Scheduled Checks.

### Check the status of ground truth data

After you register the ground truth data for ingestion, you might want to check its status.

1. From the navigation pane, click **Model Monitor**.

2. Click the name of the model for which you set up ground truth data ingestion.

3. On the Overview page’s **Data Traffic** tab, you can see the history of the ingested data and review the model’s metadata and schema.

4. Click **Ingest History** to check to see if the status is `Done`. The following are other statuses you might see:

   **`Completed with Warnings`**\
   Click the three vertical dots at the end of the row and click **View Details**.

   **`Failed`**\
   Click the three vertical dots at the end of the row and click **View Details**.

   If the ingestion failed, Domino stops the continuous ingestion. After you fix the issue, re-register the prediction data and Domino restarts continuous ingestion from this point.

   <Note>
     If you re-register the same dataset, Domino processes the entire table again. This causes duplicate records which affects the drift and model quality calculations.
   </Note>

   **`Skipped`**\
   The dataset had no rows to fetch.

   You can click **Status** to select checkboxes to filter by status.

5. Click **Refresh** to see a graph of the **Data Ingest Timeline**.

   After the data is ingested, you can perform an ad-hoc [model quality analysis](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/3-analyze-data-drift) to validate your setup. Then, [set up a schedule](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/8-set-scheduled-checks-mm) to perform model quality analysis automatically.


# Set up Cohort Analysis for Model Monitor
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/6-set-up-cohort-analysis-mm

Enable Cohort Analysis for a numerical regression model, verify ingestion, and configure cohort counts, samples, bins, and report contents.

Cohort analysis provides information about your model, including feedback about what might be going wrong with it. Cohort analysis is a job that runs on the model’s prediction and ground truth data for a regression model. The result is a PDF report and raw data in JSON format.

To use Cohort Analysis, you must register ground truth data to enable data analysis for each regression model. Doing this for a model starts a series of ingest and analysis jobs.

<Note>
  This feature is only available for regression models with numerical features.
</Note>

## Set up Cohort Analysis for this model monitor

### Prerequisites

* The model is a regression type model

* [Ground truth data](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/5-set-up-model-qual-monitor-mm) is available.

### Set up Cohort Analysis

1. From the navigation pane, click **Model Monitor**.

2. Click the name of the model for which you want to set up Cohort Analysis.

3. Click **Model Quality**.

4. Click **Register Ground Truth Data > Upload Ground Truth Config**.

5. In the Register Ground Truth window, click **Register with Cohort Analysis**.

## Check the status of the data ingestion

After you set up the data analysis for a regression model, you might want to check its status.

1. From the navigation pane, click **Deploy > Model Monitor**.

2. Click the name of the model for which you set up Cohort Analysis.

3. Click **Ingest History** to check to see if the status is Done.

After the data is ingested, Domino finds underperforming cohorts and the features that make those cohorts distinct from the rest of the data.

## Configure the Cohort Analysis

You can configure the cohort analysis for a model to customize the report.

### Prerequisites

* [Set up Cohort Analysis](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/6-set-up-cohort-analysis-mm).

* Check the status of the [data ingestion](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/4-set-up-cohort-analysis#check-the-status-of-the-data-ingestion) to confirm the data analysis is complete.

### Configure the Cohort Analysis

1. In the navigation pane, click **Projects**.

2. Click the **DominoCohortAnalysis** project. The DominoCohortAnalysis Project is created automatically as a private project under the model’s project owner’s account.

3. In the navigation pane, click **Code**.

4. Click `config.yaml` and then click **Edit**.

   You can configure the following parameters:

   **min\_k**\
   The minimum number of cohorts.

   **max\_k**\
   The maximum number of cohorts.

   **max\_samples\_for\_clustering**\
   The maximum number of samples to use to find cohorts

   **num\_bins**\
   The number of bins to use to compute the feature histograms and the contrast score.

   **max\_num\_top\_cohorts\_for\_report**\
   The maximum number of cohorts to show in the Cohort Summary and Detailed Cohort Analysis sections of the Cohort Analysis report.

   **max\_num\_top\_cohorts\_for\_report**\
   The maximum number of features to show per cohort in the Detailed Cohort Analysis section of the Cohort Analysis report.

5. Click **Save**.

6. In the navigation pane, click **Jobs**.

7. Select the **cohort\_analysis** job and click **Run** to generate new results.


# Set up notifications
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/7-set-up-notifications-mm

Configure SMTP and recipient overrides for monitored models, or call /api/metricAlerts/v1 to send threshold-based custom metric alerts.

A scheduled check is an automated instance of a running drift or model quality computation over a range of data. You can view the results on the monitoring dashboard, and automated email notifications are sent if thresholds are breached.

This topic describes how to configure notifications.

## Prerequisites

Set up the SMTP server:

1. In the navigation pane, click **Model Monitor**.

2. Click **Settings** and then click **Notification Channels**.

3. Click **Edit**.

4. Type the SMTP details in the fields provided.

5. To use SMTPS instead of SMTP+STARTTLS, switch the **Enable TLS** toggle to the off position.

6. Click **Save Mail Config**.

## Configure global notification settings for monitored models

These notification settings are used for all monitored models that do not override them.

1. In the navigation pane, click **Model Monitor**.

2. Click the name of the model for which you want to set up notifications.

3. Click **Notifications**.

4. If you do not want to **Use email address from global settings**, switch the toggle to the off position. See [Notification Channels](/cloud/platform-capabilities/features/monitoring/7-monitor-settings/3-notification).

5. Click **Edit**.

6. In the **To Addresses** field, type or paste a comma- or semicolon-separated list of email addresses.

7. Click **Save Mail Config**.

## Override the global notification settings for a monitored model

To use different notification settings for specific models, you can override the global notification settings:

1. In the navigation pane, click **Model Monitor**.

2. Click **Settings** and then click **Notification Channels**.

3. Click **Edit**.

4. Type the SMTP details in the fields provided.

5. Type or paste a comma- or semicolon-separated list of email addresses.

6. To use SMTPS instead of SMTP+STARTTLS, switch the **Enable TLS** toggle to the off position.

7. Click **Save Mail Config**.

<Warning>
  The global list was set up by your administrator during installation. This procedure overrides the global default notifications for all Model Monitors.
</Warning>

## Send notifications with the API

In your project code, you can use the Domino API to send alerts or notifications. The [`/api/metricAlerts/v1`](/api-reference/custom-metrics/send-a-metric-alert) REST API endpoint sends email when any of your monitoring metrics falls outside the thresholds you have defined in your code. Deploy the code in a [job](/cloud/platform-capabilities/core-concepts/jobs) (for one-time execution) or as a [scheduled job](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job) (for continuous evaluation) to compute your model’s metrics and send alerts.

Alerts are sent to the recipients configured in the global notification settings for monitored models, except for models that override those notification settings as described above.

<Note>
  When notifications are sent using the API, Domino uses the SMTP server for global notifications, *not* the server configured for model monitoring notifications.
</Note>

The code examples below show how to construct the [payload](/cloud/reference/api/domino-open-api) and send it to the `/api/metricAlerts/v1` endpoint.

<Note>
  You need your [API key](/cloud/reference/api/domino-api-authentication#authenticate-with-an-api-key-legacy) to access this endpoint.
</Note>

```python theme={null}
import requests
import json

url = "https://<DOMINO_HOST>/api/metricAlerts/v1"
api_key = "<DOMINO_API_KEY>"

payload = json.dumps({
  "modelMonitoringId": "62471956481e88a77ae91210",
  "metric": "testMetric",
  "value": 1.0,
  "targetRange": {
    "lowerLimit": 2.0,
    "upperLimit": 3.0,
    "condition": "between"
  }
})

headers = {
  'X-Domino-Api-Key': api_key,
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)
response.ok
```

```curl theme={null}
curl --location --request POST 'https://<DOMINO_HOST>/api/metricAlerts/v1'
--header 'X-Domino-Api-Key: <DOMINO_API_KEY>'
--header 'Content-Type: application/json'
--data-raw '{
    "modelMonitoringId": "62471956481e88a77ae91210",
    "metric": "testMetric",
    "value": 1.0,
    "targetRange": {
        "lowerLimit": 2.0,
        "upperLimit": 3.0,
        "condition": "between"
    }
}'
```

The values for `condition` are:

* `lessThan`

* `lessThanEqual`

* `greaterThan`

* `greaterThanEqual`

* `between`

You can also add an optional `description` string to include in the notifications.

<Tip>
  Make sure that `modelMonitoringId` refers to the *monitoring* model ID and not the ID of the Domino endpoint. To find the monitoring model ID, select your model in the Model Monitor and scroll down on the model’s Overview page.
</Tip>


# Set scheduled checks
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/8-set-scheduled-checks-mm

Schedule Model Monitor drift and quality checks by frequency and data window, receive breach notifications, and review Checks History.

In a production setting, APIs typically push prediction and ground truth data into the Model Monitor.

You can use Scheduled Checks to ensure that you are notified if data drift or model quality metrics degrade beyond the threshold for any period. For data drift, Domino uses the timestamps of the predictions to select data for the scheduled checks. For model quality, Domino uses the ingestion time of the ground truth labels to select data. You can specify how often to repeat the checks and the time range of the data to be used for calculations for those checks. When a check fails, email notifications are sent. See [Set up Notifications](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/7-set-up-notifications-mm).

## Prerequisites

For scheduled checks on data drift and model quality, [prediction](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/4-set-up-drift-detection-mm) and [ground truth data](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/5-set-up-model-qual-monitor-mm) must be available periodically.

## Set scheduled checks

1. From the navigation pane, click **Model Monitor**.

2. Click the name of the model for which you want to set up scheduled checks.

3. Click **Data Drift** or **Model Quality** and then click **Scheduled Checks**.

4. Type a name for the check.

5. Set up the frequency at which the check must run.

6. In the **Select Data to Check** area, select one of the following:

   <Warning>
     * When you make a selection, remember that you must update the data manually and ensure that there is enough time for the data to be ingested. For example, if you set the check to repeat every day at 11 PM but you load new ground truth data at 10:45 PM, this might not be enough time for the data to be ingested.

     * If you select **Use new data since last check**, the system assumes new data was ingested. If none was ingested, you will see an empty dashboard for the specified date and time on the Analyze tab.
   </Warning>

   * **Use new data since last check time**

     * For data drift, this checks predictions with timestamps later than the last scheduled check.

     * For model quality, this checks only ground truth labels ingested into the Model Monitor after the last scheduled check ran and matches them with historical predictions made by the model.

   * **Data since last x \<time period>**

     * For data drift, this checks predictions whose timestamps are within the last specified interval (for example, the last three days). For model quality, this checks only the ground truth labels ingested within the last specified interval (for example, within the last three days), and matches them with historical predictions made by the model.

7. Click **Save**.

In the Model Monitor, you can click **Checks History** to see historical data about scheduled checks.


# Model Monitoring tags
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/9-model-monitoring-tags

Add or remove Model Monitoring tags to organize monitored models, and use the API to list or archive tags independently of other tag types.

Model Monitoring tags help collaborators organize and discover monitored models in Domino. You could use tags to generate summary views of monitored models for a specific audience - like an "executive summary" for executives or "production" models for developers.

## Tag a model

You can add and remove model tags by going to **Model Monitor** > Select the model you want to edit > Click the **Edit** icon.

## Manage tags

Domino admins can manage the tags for the Domino Model Monitor through the [Model Monitoring API](/cloud/reference/api/model_monitoring/dmm-api).

Admins can archive a tag to remove it from all models with that particular tag.

```python theme={null}
import requests

headers = {"X-Domino-Api-Key": f"{API_TOKEN}"}

# Fetch available tags that can be added to a given model
response = requests.get(f"{DOMINO_URL}/model-monitor/v2/api/model/{DMM_MODEL_ID}/tags", headers = headers)
print(response.json())

# Archive the first tag from the list
tag_id = response.json()[0]['id']
requests.patch(f"{DOMINO_URL}/model-monitor/v2/api/model-tag/{tag_id}/archive", headers = headers)
```

## Limitations

Model Monitoring tags are distinct and cannot be used with [model registry tags](/cloud/platform-capabilities/features/models/register-and-govern/model-registry) or [Project tags](/cloud/platform-capabilities/core-concepts/projects/project-settings#tag-a-project).


# Set up Model Monitor
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/index

Set up Model Monitor for external models by connecting data, registering schemas, configuring checks and alerts, and managing tags and permissions.

Use the Model Monitor to configure monitoring for models that aren’t deployed on Domino endpoints or for those models deployed outside Domino.

The following topics explain the steps:

1. [Install packages for Model Monitoring](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/1-install-packages-for-model-monitoring). Model Monitoring requires certain packages to be included. If you aren’t using the Domino Standard Environment, follow these instructions to install the packages manually in your own environment.

2. [Connect a Data Source](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/2-connect-a-data-source). Connect to external Data Sources to access your training, prediction, and ground truth data.

3. [Register a Model](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/3-register-a-model). Register external models so that Domino can capture information about them through their Monitoring Config JSON files.

4. [Set up Drift Detection](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/4-set-up-drift-detection-mm). Add data that was trained on historical data to your model’s code to start ingesting and storing prediction data for monitoring.

   * Optional: [Configure notifications](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/7-set-up-notifications-mm) or change the [scheduled checks](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/8-set-scheduled-checks-mm).

5. [Analyze data drift](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/4-set-up-drift-detection-mm#analyze-data-drift). See the divergence value for features and experiment with test types, thresholds, and other conditions.

   * Optional: [Configure notifications](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/7-set-up-notifications-mm) or change the [scheduled checks](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/8-set-scheduled-checks-mm).

6. [Set up Model Quality Monitoring](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/5-set-up-model-qual-monitor-mm). Ingest ground truth data to monitor the quality of the model’s predictions.

   * Optional: [Configure notifications](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/7-set-up-notifications-mm) or change the [scheduled checks](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/8-set-scheduled-checks-mm).

7. [Set up Cohort Analysis](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/6-set-up-cohort-analysis-mm). Get insights into model quality so you can find underperforming data hotspots for model remediation.

8. [Configure model tags](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/9-model-monitoring-tags) to add metadata to your Model for better organization and searchability.

9. [Configure model permissions](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/10-model-permissions) to define who can view or edit your model.

10. If you must modify the monitoring config JSON, you can [unregister the model](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/11-unregister-a-model) and then [register it](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/3-register-a-model) again.


# Access the Monitor dashboard
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/5-use-monitoring/1-access-monitor-dashboard

View monitored models, health status, prediction traffic, drift, and quality in the Monitor dashboard, then inspect threshold breaches.

This topic describes how to access Domino’s dashboard which shows all endpoints being monitored.

To see all models, in the navigation pane, click **Deploy > Model Monitor**.

You can see a list of all models, their health status, and prediction traffic. If one or more features for the drift value or model quality for a model doesn’t meet the test criteria, the model status is marked in red to indicate the features have drifted beyond the threshold. You can hold your mouse over the red circle in the appropriate column to see more details about the issues:

You can set up notifications to alert on breached thresholds for [drift](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/5-set-up-notifications) or [model quality](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/7-set-up-notifications-mm).

<Tabs>
  <Tab title="Domino endpoint monitoring">
    1. In the navigation pane, click **Endpoints**.

    2. Click the endpoint whose dashboard you want to see.

    3. Click **Grafana Monitoring**. The dashboard opens and shows the Data Drift and Model Quality charts, if both are enabled.
  </Tab>

  <Tab title="Model Monitor">
    1. In the navigation pane, click **Model Monitor**.

    2. Click the model whose dashboard you want to see.

    3. Click **Data Drift** or **Model Quality** to see the respective dashboard.
  </Tab>
</Tabs>


# Custom Model Monitoring metrics
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/5-use-monitoring/2-custom-metrics

Use the custom metrics Python SDK to log and retrieve metric values, attach tags, define target ranges, and trigger alerts for monitored models.

Use Domino’s Model Monitoring custom metrics Python SDK to define custom metrics and use them alongside out-of-the-box drift and model quality metrics that are monitored in Domino Model Monitor. With this SDK, you can register new metrics and define the logic to compute them. You can author this logic and evaluate it from within a Domino project.

For every model that you register for monitoring, you can select a registered metric, associate the data sources from which the metric is computed, and set up the execution environment to compute this metric on a periodic basis. You are notified by email when a metric behaves abnormally based on threshold definitions.

For end-to-end working code with a description of the workflow, see the `/custom metrics example` folder in the quick-start project.

## Use custom metrics

Use the following library functions from the custom metrics Python SDK to build and deploy your own monitoring metrics.

1. Instantiate the client. Enter your `DMM model ID`:

   ```python theme={null}
   import domino
   dmm_model_id = "XXXXXXXXXXXXX"
   d = domino.Domino("integration-test/quick-start")
   metrics_client = d.custom_metrics_client()
   ```

2. Log the custom metrics:

   ```python theme={null}
   metrics_client.log_metric(dmm_model_id, "accuracy", 7.1234, "2022-10-08T00:00:00Z", { "example_tag1" : "value1", "example_tag2" : "value2" })

   metrics_client.log_metrics([
   { "modelMonitoringId" : dmm_model_id, "metric" : "accuracy", "value" : 7.1234,
   "timestamp" : "2022-10-08T00:00:00Z",
   "tags" : { "example_tag1" : "value1", "example_tag2" : "value2" }
   ]
   },
   { "modelMonitoringId" : dmm_model_id, "metric" : "accuracy", "value" : 8.4567,
   "timestamp" : "2022-10-09T00:00:00Z" }
   ])
   ```

   * `modelMonitoringId`: ID of the monitored model to send metric alerts for

   * `metric`: Name of the metric to send alert for

   * `value`: Value of the metric

   * `timestamp`: Timezone is in UTC in ISO 8601 format.

   * `tags`: Custom metadata for metric represented as key-value string pairs

3. Send a custom metrics alert:

   ```python theme={null}
   metrics_client.trigger_alert(dmm_model_id, "metric_name", 3.14,
   condition = metrics_client.BETWEEN,
   lower_limit=2.0, upper_limit=3.0,
   description = "Breached 2.0-3.0 range." )
   ```

   * `modelMonitoringId`: ID of the monitored model for which to send metric alerts.

   * `metric`: Name of the metric for which to send the alert.

   * `value`: Value of the metric.

   * `condition`: Target range for the metric defined by lower and upper limit bounds.\
     The following are potential values for the `condition` argument:

     * `metrics_client.LESS_THAN = "lessThan"`

     * `metrics_client.LESS_THAN_EQUAL = "lessThanEqual"`

     * `metrics_client.GREATER_THAN = "greaterThan"`

     * `metrics_client.GREATER_THAN_EQUAL = "greaterThanEqual"`

     * `metrics_client.BETWEEN = "between"`

   * `lower_limit`: The lower limit for the `condition`.

   * `upper_limit`: The upper limit for the `condition`.

   * `description`: Optional message included in the alert.

4. Retrieve the custom metrics:

   ```python theme={null}
   res = metrics_client.read_metrics(dmm_model_id, "accuracy",
   "2022-10-08T00:00:00Z", "2022-10-10T00:00:00Z" )
   ```

   * `modelMonitoringId`: ID of the monitored model for which to retrieve the metric values.

   * `metric`: Name of the metric for which to retrieve the metric values.

   * `start_timestamp`: The start timestamp of the range when the metrics were logged. The timezone is in UTC in ISO 8601 format.

   * `end_timestamp`: The end timestamp of the range when the metrics were logged. The timezone is in UTC in ISO 8601 format.

## Sample output

```
{
  "metricValues": [
    {"timestamp": "2022-10-08T00:00:00Z", "value": 3.123, "tags": []},
    {"timestamp": "2022-10-08T00:00:00Z", "value": 3.123, "tags": []},
    {"timestamp": "2022-10-08T00:00:00Z", "value": 3.123, "tags": []},
    {"timestamp": "2022-10-08T00:00:00Z", "value": 3.123, "tags": []},
    {"timestamp": "2022-10-08T00:00:00Z", "value": 7.1234, "tags": {"example_tag1": "value1", "example_tag2": "value2"}},
    {"timestamp": "2022-10-09T00:00:00Z", "value": 8.4567, "tags": []}
  ],
  "metadata": {"requestId": "97e61f63-3be5-4766-a07a-06c931d91268", "notices": []}
}
```

Results:

* `timestamp`: The time this metric was logged.

* `value`: The value of the metric.

* `tags`: Custom metadata for metric represented as key-value string pairs.

See also the [custom metrics REST API endpoints](/cloud/reference/api/domino-open-api).


# Analyze data drift
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/5-use-monitoring/3-analyze-data-drift

Compare training and prediction distributions, select drift tests and thresholds per feature, recalculate results, and save check defaults.

Domino’s monitoring feature uses the training data to calculate the probability distributions of all features and prediction columns to detect data drift for input features and output predictions of your model. It approximates these columns by creating bins, and then counting the frequency for each bin. This acts as the reference pattern.

The monitor feature uses prediction data to calculate the probability distribution, using the same bins, and then applies the specified statistical divergence or distance test to quantify the dissimilarity between the training and prediction distributions for each column.

By default, drift checks are scheduled to occur daily at noon UTC.

<Note>
  Prediction data is analyzed through 23:59 of the previous day. Data from the current day is not included. Domino reads the timestamps in the dataset if they are present; if not, it uses the ingestion timestamp.
</Note>

## Prerequisites

* Model is [registered](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/3-register-a-model) for monitoring

* [Prediction data](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/1-set-up-prediction-capture) is available for Domino to monitor

## Modify the data drift analysis

Out-of-the-box, the Model Monitor supports several statistical tests. Each feature can have a different test. When a new model is registered, it inherits the [global default test settings](/cloud/platform-capabilities/features/monitoring/7-monitor-settings/4-tests). However, you can change the test types and thresholds to values suitable for each model.

If you save these changes, they are used for subsequent automated checks, such as when new predication data is uploaded or when Scheduled Checks run for the model.

### Modify the test features

* If you are using the monitoring feature for a Domino endpoint, go to **Configure monitoring > Target Ranges** to access the Configure Tests and Thresholds page.

* If you are using the Model Monitor, in the navigation pane, click **Model Monitor**. Select the model that you want to monitor and then click **Data Drift**.

<img alt="The Data Drift page is labeled with the letters A through E" />

<Note>
  If your model had a timestamp column declared, it’s used to get the timestamp of different predictions in the dataset. If the timestamp wasn’t declared, then the data’s ingestion time in the Model Monitor is used as its timestamp.
</Note>

A - Use the **Date Filter** to refine the date range and review the Drift and its trends for specific periods.

B - To reduce alert noise, click the bell icon next to features to disable alerts to exclude them from the Scheduled Checks.

C - Select the **Test Type**, the **Acceptable Range**, and the **Threshold**. See [Data drift statistical tests](#data-drift-statistical-tests) for more information about test types.

Click **Recalculate** to see the results.

D - If you made changes to the tests and want to set them as the defaults for the model in subsequent automated checks, go to **Other Actions > Save Checks Config** to save the changes.

E - If you experimented with changes to the tests and wanted to load the model’s default configuration to reset the table, go to **Other Actions > Restore Checks Config**.

If you are using the monitoring feature for a Domino endpoint, click the X in the top right corner to close the page when you are finished.

## Data drift statistical tests

Out-of-the-box, the Model Monitor supports several statistical tests.

<Note>
  The lower the Population Stability Index (PSI) value, the better the prediction data matches the training data. If there is no difference between the data sets, the value is 0. Typically:

  * PSI \< 0.1 = no significant change

  * PSI \< 0.2 = moderate change

  * PSI >= 0.2 = significant change
</Note>

**Kullback–Leibler Divergence (Recommended)**\
Kullback–Leibler divergence (also called relative entropy) is a measure of how one probability distribution is different from a second, reference probability distribution.

The divergence can range from zero to infinity. A value of zero means there is no difference between the data sets.

This test works for different distributions and therefore is most commonly used to detect drift.

**Chi-square Statistic**\
Chi-square test in another popular divergence test well-suited for categorical data.

The chi square statistic is a statistical hypothesis testing technique to test how two distributions of categorical variables are related to each other. Specifically, the chi-square statistic is a single number that quantifies the difference between the observed counts versus the counts that are expected if there was no relationship between the variables at all.

The divergence can range from zero to infinity. A value of zero means there is no difference between the data sets.

**Population Stability Index**\
Population Stability Index (PSI) is a popular metric in the finance industry to measure changes in distribution for two datasets. It produces less noise and has the advantage of a generally accepted threshold of 0.2-0.25.


# Analyze model quality
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/5-use-monitoring/4-analyze-model-quality

Compare predictions with ground truth, filter by date, set acceptable ranges and thresholds, and interpret classification or regression metrics.

The monitoring feature uses the row identifiers in the prediction data and ground truth data to match the predicted and expected value. Based on those matches, it calculates the model quality metrics.

By default, model quality checks are scheduled to occur daily at noon UTC.

<Note>
  Prediction data is analyzed through 23:59 of the previous day. Data from the current day is not included. Domino reads the timestamps in the dataset if they are present; if not, it uses the ingestion timestamp.
</Note>

## Prerequisites

* [Set up Prediction Data](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/1-set-up-prediction-capture)

* [Publish the Model](/cloud/platform-capabilities/features/model-deployment)

* [Set up Drift Detection](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/4-set-up-drift-detection-mm)

* [Set up Model Quality Monitoring](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/5-set-up-model-qual-monitor-mm)

* [Set up Cohort Analysis](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/6-set-up-cohort-analysis-mm)

## Modify the model quality analysis

Classification models and regression models show different prediction quality metrics. See [Metrics for classification and regression models](#metrics-for-classification-and-regression-models) for details.

If you are looking at a classification model, a confusion matrix and classification report for the data in the selected time range is shown in a Charts section under the metrics table. See [classification models](#classification-models) for details.

For classification models:

* Model monitoring uses [weighted method](#weighted-method) to calculate the metrics.

* AUC, ROC, Log Loss, and Gini Norm are only calculated if the Predication Probability type is declared as part of the schema. See the [Note](/cloud/platform-capabilities/features/monitoring/8-json#forPredictionOutput) in [Monitoring Config JSON](/cloud/platform-capabilities/features/monitoring/8-json).

* Sample Weights is used in calculations for the Gini Norm metric only.

### Modify the test features

* If you are using the monitoring feature for a Domino endpoint:

  * Go to **Configure monitoring > Data**.

  * In the Configure Data dialog, in the Ground Truth Data section, click **open this model**.

* If you are using the Model Monitor, in the navigation pane, click **Model Monitor**. Select the model that you want to monitor and then click **Model Quality**.

  <img alt="Model Quality page labeled from A-E" />

  A - Use the **Date Filter** to refine the date range and review the Model Quality and its trends for specific time periods.

  B - To reduce alert noise, click the bell icon next to a feature to disable alerts and exclude it from the Scheduled Checks.

  C - Select the **Acceptable Range** and the **Threshold**. Click **Recalculate** to see the results.

  D - If you made changes to the tests and want to set them as the defaults for the model in subsequent automated checks, go to **Other Actions > Save Checks Config** to save the changes.

  E - If you experimented with changes to the tests and wanted to load the model’s default configuration to reset the table, go to \*Other

### Metrics for classification and regression models

#### Classification models

The following are the supported prediction quality metrics for Classification models. All metrics are calculated using the weighted average method with the weight relating to the proportion of occurrences of a prediction class in the dataset.

<span />

**Accuracy**\
Accuracy is the most common metric used to evaluate the performance of a classification model. It is calculated as the proportion of examples in the evaluation dataset that were predicted correctly, divided by all predictions that were made on the evaluation dataset. + A confusion matrix represents the model’s true positives, true negatives, false positives, and false negatives for each prediction class. The classification accuracy can be calculated from this confusion matrix as the sum of true positives and true negatives divided by the sum of the true positives, true negatives, false positives and false negatives.

**Precision**\
Precision quantifies the number of positive identifications that were actually correct. It is defined as the ratio of true positives to the sum of true positives and false positives

**Recall**\
Recall quantifies the proportion of positive identifications was actually correct. It is defined as the ratio of true positives to the sum of true positives and false negatives

**F1**\
The F(n) score allows us to give precision or recall a higher weight based on what is important to the use case being evaluated. For F1, both precision and recall are given equal weight and is calculated as the harmonic mean of the precision and recall. + The formula for F1 is (2 \* precision \* recall) / (precision + recall)

**ROC**\
An ROC curve (receiver operating characteristic curve) is a graph showing the performance of a classification model at all classification thresholds. This curve plots two parameters: true positive rate on the y-axis and false positive rate on the x-axis. Typically, lowering the classification threshold results in increasing both false positives and true positives and the curve is used to decide a classification threshold that is a good tradeoff between the two.

**AUC**\
AUC stands for Area under the ROC curve and measures the 2D area under the ROC curve. The AUC provides an aggregate measure of performance across all possible classification thresholds. AUC ranges in value from 0 to 1. A model whose predictions are 100% wrong has an AUC of 0 and one whose predictions are 100% correct has an AUC of 1. The AUC is desirable as it measures the quality of the model’s predictions irrespective of what classification threshold is chosen.

**Log Loss**\
Log-loss is indicative of how close the prediction probability is to the corresponding actual/true value (0 or 1 in case of binary classification). The more the predicted probability diverges from the actual value, the higher is the log-loss value. It’s hard to interpret raw log-loss values, but log-loss is a good metric for comparing models. For any given model, a lower log-loss value means better predictions

**Gini coefficient (Normalized)**\
The (normalized) Gini coefficient gets its inspiration from the Lorentz curve in economics and is typically used when the datasets are imbalanced. In the context of Machine Learning the equivalent of the Lorenz curve is built using the values of the (predicted) class probabilities as opposed to “income” that is used in economics models. The Gini coefficient is then defined as the area under the Lorentz curve. Dividing the Gini coefficient computed from the Lorenz curve of the predicted class probabilities by the Gini coefficient computed from the actual class probabilities results in a normalized Gini coefficient for classification models.

#### Regression models

The following are the supported prediction quality metrics for Regression models.

**Mean Square Error (MSE)**\
Mean squared error (MSE) measures the amount of error (also called residual) in regression models. It assesses the average squared difference between the observed and predicted values. When a model has no error, the MSE equals zero. As model error increases, its value increases. The mean squared error is also known as the mean squared deviation (MSD) and is calculated by dividing the sum of squared errors by the sample size.

**Mean Absolute Error (MAE)**\
The MAE is computed by taking the average of the absolute difference between the actual or true values and the values that are predicted by the regression model for every datapoint in the evaluation dataset. It is therefore measuring the average magnitude of errors of the predictions irrespective of whether the prediction was less than or greater than the actual value.

**Mean Absolute Percentage (MAPE)**\
Mean absolute percentage error (MAPE) is commonly used as a loss function for regression problems and in model evaluation. It is the percentage equivalent of MAE and as it is a percentage is an intuitive way to quantify on average how far the model’s predictions are off from the actual values. Both MAPE and MAE are resistant to outliers but have the disadvantage of being biased towards predictions that are less than the actual values themselves.

**R-Squared (R2)**\
R-Squared (R2) is a statistical measure of how close the data is to the fitted regression hyperplane. It is defined as the percentage of variation in the target variable that is explained by the regression model. R-Squared (R2) lies between 0 and 1 ( or 0% and 100%) and 0 (0%) indicates that the model explains none of the variability of the target data around its mean and 1 (100%) indicates that the model explains all the variability of the target data around its mean

**Gini (Normalized)**\
The (normalized) Gini coefficient gets its inspiration from the Lorentz curve in economics and is typically used when the datasets are imbalanced i.e there are big (and) small values in the dataset. In the context of regression models, the equivalent of the Lorenz curve is built using the predicted values of the target variable as opposed to “income” that is used in the case of economics models. The Gini coefficient is then defined as the area under the Lorenz curve. Dividing the Gini coefficient computed from the Lorenz curve of the predictions by the Gini coefficient computed from the actual values results in a normalized Gini coefficient for regression models.


# Model quality methodology
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/5-use-monitoring/6-model-quality-methodology

Trace how Model Monitor matches prediction and ground truth rows, builds hourly aggregates, and calculates regression and classification metrics.

Domino Model Monitoring provides a set of metrics to assess your model’s performance. Learn about the statistical techniques that Domino uses so you can interpret the metrics as new prediction and ground truth data sets are produced.

## Matching ground truth and prediction datasets

To monitor your models effectively, you need to set up Domino Model Monitor and provide corresponding records for both prediction and ground truth data sets. You need to separately set up Domino Model Monitor to ingest prediction datasets (for [Domino hosted models](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/1-set-up-prediction-capture), for [externally hosted models](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/4-set-up-drift-detection-mm)) and [ground truth datasets](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/5-set-up-model-qual-monitor-mm#ingest-ground-truth-data) for each model.

Domino identifies matching records using the 'row\_identifier' [variable type](/cloud/platform-capabilities/features/monitoring/8-json#variabletype) registered in both the prediction and ground truth datasets.

For any duplicate entries in the ground truth data, Domino processes the latest record.

If you only ingest partial ground truth data, it only reflects the accuracy for the predictions that have actuals associated with them. For example, if only 100 out of 1000 entries have ground truth, then the accuracy is calculated based on the 100 values.

If at some point, you re-ingest ground truth data, Domino updates the metrics assuming two different values for the same record ID.

## How Domino computes quality metrics

For the matched rows, Domino creates "aggregates," which are summaries over hourly periods. Aggregates serve as the basis for the model quality metrics. These aggregates capture different information depending on whether the model is for regression or classification.

When the user selects a date start/end period, or when a scheduled check happens, the model quality statistics are calculated on the fly, combining multiple hourly aggregates into a single per-day aggregate, one for each day in the selected period.

The sums are added up trivially. The two distributions are added by an algorithm that combines two distributions. To do so, the lower-sized bin width is chosen as the new bin width, an overall min/max is decided and then values from the distributions being added are redistributed into the closest new bin.

### Re-ingesting ground truth data

If ground truth data is re-ingested and there exist aggregates for the period that has just been re-ingested, for a given hour, the existing record is updated. This implies that new information is added to the existing record. That is, the system is designed to always process new data but not overwrite existing data.

If you wish to override this behavior and perform a clean re-ingestion, it is recommended that you start with re-registering the model, starting afresh. If you need to clear a portion of data saved for an existing model, contact your Domino support representative.

## Regression models

Each hourly aggregate described above is saved with the following information:

* Number of records

* Sum of actual values

* Sum of squares

* Sum of MSE

* Sum of MAE

* Sum of MAPE

* Prediction distribution (binned discrete counts)

* Ground truth distribution (binned discrete counts)

Since the sums of actual values, squares, MSE, MAE, and MAPE are additive, Domino then uses them to calculate final MSE, MAE, MAPE, and R<sup>2</sup> model quality statistics in the user interface.

In turn, prediction distributions and ground truth distributions are used to calculate the Gini Norm model quality statistic.

## Classification models

The model quality methodology for classification models is similar to regression models, except for the types of hourly aggregates saved.

For classification models, each aggregate is saved with the following information:

* Number of records

* Confusion matrix

* Log loss numerator

* Log loss denominator

* Counts of true positives & false positives

* Counts of true negatives & false negatives

When combined, these allow the system to calculate AUC/ROC, accuracy, precision, recall, F1, log loss and Gini Norm model quality statistics.


# Use monitoring
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/5-use-monitoring/index

Track drift and prediction quality across models, inspect historical checks, configure alerts, ingest data through APIs, and automate retraining.

Use Domino’s dashboard to monitor your data science models. The model monitoring feature detects and tracks data drift in the model’s input features and output predictions. If you have ground truth data in the model, Domino can calculate and track the model’s prediction quality using standard measures such as accuracy, precision, and more.

Domino’s monitoring feature checks if the characteristics of the predictions versus the target data that were used to train the model are significantly different. You can set up scheduled checks, use APIs to ingest data, configure notification recipients to continuously monitor your models across your organization, and review historical data about the checks. APIs give you a way to integrate into existing business processes, as well as a programmatic option to automatically retrain models.

Find out more about how to use monitoring in these topics:

* [Access the Monitor dashboard](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/1-access-monitor-dashboard).

* [Custom Model Monitoring metrics](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/2-custom-metrics).

* [Analyze data drift](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/3-analyze-data-drift).

* [Analyze model quality](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/4-analyze-model-quality).

* [Model quality methodology](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/6-model-quality-methodology).


# Use Cohort Analysis
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/6-remediation/1-review-cohort-analysis

Review Cohort Analysis jobs, JSON results, and PDF reports to identify underperforming regression cohorts and the features that distinguish them.

When the Model Monitor detects data drift or a reduction in quality for a regression model, the Cohort Analysis gives you details so you can take remedial action. It identifies underperforming cohorts of data, determines what differentiates this data, and shows you specific features of the data that require examination. See [Set up Cohort Analysis](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/6-set-up-cohort-analysis-mm) for information about how to enable this feature.

<Note>
  This feature is only available for regression models with numerical features.
</Note>

## Review the Cohort Analysis

You can access the job or report for the Cohort Analysis from the Model Monitor.

* The job gives you access to the JSON files that contain the code used to perform the analysis, as well as the cohort statistics. You can copy the code and statistics to create custom reports or to use when remediating your model.

* The report shows a detailed cohort analysis. Use this to drill down into each cohort to understand the primary features that impact the model’s performance.

Review the Cohort Analysis:

1. From the navigation pane, click **Model Monitor**.

2. Select the model for which you generated the Cohort Analysis.

3. Click **Model Quality**.

4. In the Cohort Analysis Report area, click one of the following:

   * **Go to Job** to see the job. Click the **Results** tab to see the JSON data that was used to generate the report. In the navigation pane, you can click **Files** to see the related files.

   * **View Report** to open a Files page that has the .pdf. Click `<filename>.pdf` to view the report. The report has the following sections:

     1. Section 1 shows a visual representation and summary statistics about the model quality for the worst-performing cohorts. You can investigate these cohorts.

     2. Section 2 shows information about the most distinctive features (quantified by a contrast score) for the top worst-performing cohorts. You can investigate a feature or a combination of features for remediation.

     3. Section 3 shows summary statistics for all cohorts listed in the order of model performance. It also lists the features for a cohort that contribute to the cohort’s distinctiveness.

        | <img alt="The cohort summary shows the top 50 worst performing cohorts" /> | <img alt="The detailed cohort analysis shows the top 50 worst performing cohorts" /> | <img alt="The cohort summary statistics show a table of the model quality statistics" /> |
        | - | - | - |


# Remediate a Domino endpoint
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/6-remediation/2-remediate-a-domino-endpoint

Inspect captured Parquet predictions, reproduce an endpoint’s original Workspace, update or retrain the model, and publish a new version.

After reviewing the [data drift](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/3-analyze-data-drift) and [model quality monitor](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/4-analyze-model-quality), use the monitoring results and [Cohort Analysis](/cloud/platform-capabilities/features/monitoring/6-remediation/1-review-cohort-analysis) to determine whether there are concerns with your model. If you do have concerns, [review your data](#review-your-data).

If you want to review the model and its associated code to investigate further, you can [reproduce the original code commits and artifacts](#reproduce-the-environment) in the environment in which you deployed the model. If necessary, you can update the model code or retrain it with the latest production data. Then you can [deploy a new, improved version of the Domino endpoint](#publish-a-new-domino-endpoint).

## Review the Domino endpoint predictions

Domino automatically creates a prediction dataset named `prediction_data` for every project that can be accessed from any workspace. The predictions are in Parquet format and are updated hourly as the Domino endpoint processes inputs. If there is no data in an hour, no file is created. If you configured [data drift monitoring](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/3-analyze-data-drift) or [Set up Model Quality Monitoring](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/4-analyze-model-quality) then the stored prediction data is automatically consumed by the Model Monitor.

By default, a daily job deletes data older than 30 days. Your administrator defines the retention policy for predictions.

### Review your data:

1. In your workspace, open the IDE.

2. Use the following paths to read the data:

   * To load individual Parquet files:

<Tabs>
  <Tab title="To load individual Parquet files for DFS-based projects">
    ```console theme={null}
    /domino/datasets/local/prediction_data/<model_version_id>/$$date$$=<date_in_utc>/$$hour$$=<hour_in_utc>/predictions_<uuid>.parquet
    ```
  </Tab>

  <Tab title="To load individual Parquet files for GIT-based projects">
    ```console theme={null}
    /mnt/data/prediction_data/<model_version_id>/$$date$$=<date_in_utc>/$$hour$$=<hour_in_utc>/predictions_<uuid>.parquet
    ```
  </Tab>

  <Tab title="To load the entire Parquet dataset for DFS-based projects">
    ```console theme={null}
    /domino/datasets/local/prediction_data/<model_version_id>
    ```
  </Tab>

  <Tab title="To load the entire Parquet dataset for Git-based projects">
    ```console theme={null}
    /mnt/data/prediction_data/<model_version_id>/$$date$$=<date_in_utc>/$$hour$$=<hour_in_utc>/predictions_<uuid>.parquet
    ```
  </Tab>
</Tabs>

<Warning>
  Do not rename the generated Parquet files and directories. Doing so can cause inconsistent behavior.
</Warning>

## Reproduce the environment

### Prerequisites

<Tabs>
  <Tab title="For Git-based projects">
    * This feature is only enabled for models published in Domino 5.0 and higher.
  </Tab>

  <Tab title="For Domino File System-based projects">
    * This feature is available for pre-5.0 published models as well as newly published models, as long as no additional Git repositories are involved.
  </Tab>
</Tabs>

### Remediate the model

1. From the navigation pane, click **Endpoints**.

2. Click the model that you want to remediate and then click **Open in Workspace**.

3. From the Open in New Workspace and Branch window, type a name for the workspace.

4. Select a **Hardware Tier**.

5. In New Branch Name, type a name for the code branch.

6. Click **Open**. A Domino workspace opens and is ready for you to take remedial action.

   <Note>
     If you manage file changes and code commits outside of Domino (such as in an external Git client) and only use Domino to publish the Domino endpoint, the window shows a list of tools. Select from these tools to create a new workspace.
   </Note>

## Publish a new Domino endpoint

When you reproduce a workspace, as you did in the [Reproduce the environment](#reproduce-the-environment) topic, Domino creates a branch in every repository involved in the project.

To publish a new model based on this reproduced branch, you must apply the [commit to the master branch](/cloud/platform-capabilities/core-concepts/workspaces/use-git-in-your-workspace) because Domino supports Git-based projects. The way that you do this depends on whether you are working with a Domino File System-based project or a Git-based project.

<Tabs>
  <Tab title="Commit projects based on the Domino File System (DFS)">
    1. Go to the Project.
    2. Click **Code** in the navigation bar.
    3. From the **Branch** list, select the reproduced branch.
    4. Click **Revert Project** to ensure that the commits made in this branch are added in the Master branch.

       <Note>
         **Revert Project** only works for the DFS files. For imported Git repositories or projects, you must revert the artifacts independently.
       </Note>
  </Tab>

  <Tab title="Commit Git-based projects">
    * In a Domino workspace or the Git tool of your choice, merge your latest code update into the master branch.

    <Note>
      Perform the same merge operation for any artifacts (such as `.pkl` files) that you maintain in a separate repository. You can use Domino for the artifacts.

      1. In your Git-based project, in the navigation pane, click **Artifacts**.
      2. Click **Revert Project** to restore the artifacts from your reproduced branch to the master branch.
    </Note>
  </Tab>
</Tabs>

Go to the **Domino endpoint** section of your project to publish a new Domino endpoint or a new version of an existing Domino endpoint. See [Publish the Domino endpoint](/cloud/platform-capabilities/features/model-deployment).


# Remediate model issues
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/6-remediation/index

Use Cohort Analysis and captured prediction data to diagnose performance issues, update model code or training data, and deploy a new endpoint version.

Domino automatically analyzes prediction data to identify cohorts that are influencing a model’s performance.

For models deployed as Domino endpoints, you can use this cohort analysis to do an in-depth investigation into the production data that is in Domino Datasets to get to the root of the model’s performance issues. Then, you can make changes to your model code or retrain the model with new data and deploy a new version.

Find out more about how to remediate model issues in these topics:

* [Use Cohort Analysis](/cloud/platform-capabilities/features/monitoring/6-remediation/1-review-cohort-analysis).

* [Remediate a Domino endpoint](/cloud/platform-capabilities/features/monitoring/6-remediation/2-remediate-a-domino-endpoint).


# API token
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/7-monitor-settings/1-api

Authenticate Model Monitoring API requests with Domino API keys, continue using existing DMM API keys, or revoke deprecated DMM tokens.

DMM API keys have been deprecated in Domino 5.6, and you can no longer generate new DMM API keys. Instead, use Domino API keys to authenticate requests to the Model Monitoring API using the `X-Domino-Api-Key` header. Learn how to [generate Domino API Keys](/cloud/reference/api/domino-api-authentication).

Existing DMM API keys continue to authenticate requests to the Model Monitoring API with the `X-DMM-API-KEY` header. You can revoke DMM API keys based on your security policy by clicking **Revoke Token** from the Model Monitor Settings page.


# Health dashboard
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/7-monitor-settings/2-health

Set how many checks and days of prediction traffic the Model Monitor Health Status displays, then view model health from the Models page.

Use this section to configure the number of checks and number of days of prediction traffic shown in the Health Status area on the Model page.

1. From the navigation pane, click **Model Monitor**.

2. Click **Settings**.

3. Click **Health Dashboard**.

4. Select the **No. of checks** and **Prediction traffic (Days)** as needed.

## View the health status on the Model page

From the navigation pane, click **Models**.


# Notification channels
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/7-monitor-settings/3-notification

Configure global SMTP notification channels for Model Monitor, including sender and recipient addresses, TLS behavior, and model-level overrides.

Use this section to set the global communication channel to send notifications to your stakeholders. You can override these global settings at the model level of any registered model. See [Set up Notifications](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/5-set-up-notifications) (for Domino endpoints) or [Set up Notifications](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/7-set-up-notifications-mm) (for the Model Monitor).

1. From the navigation pane, click **Deploy > Model Monitor**.

2. Click **Settings**.

3. Click **Notification Channels**.

4. Click **Edit**.

5. Type the SMTP details in the fields provided.

6. In the **From Address** field, type email address from which the notifications are sent.

7. In the **To Addresses** field, type or paste a comma- or semicolon-separated list of email addresses.

8. To use SMTPS instead of SMTP+STARTTLS, switch the **Enable TLS** toggle to the off position.

9. Click **Save Mail Config**.


# Test defaults
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/7-monitor-settings/4-tests

Set global Data Drift and Model Quality test types, conditions, and thresholds that Model Monitor applies when registering new models.

Use this section to set global default threshold conditions. You can override these global settings at the model level of any registered model.

* [To modify the test features for data drift](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/3-analyze-data-drift#modify-the-data-drift-analysis), change the data drift default settings at the model level.

* [To modify the test features for model quality analysis](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/4-analyze-model-quality#modify-the-model-quality-analysis), change the model quality analysis at the model level.

## Test the defaults

1. From the navigation pane, click **Deploy > Model Monitor**.

2. Click **Settings**.

3. Click **Test Defaults**.

4. Click the arrow to expand or collapse the following sections:

   * **Data Drift Test Defaults**

   * **Model Quality Test Defaults**

5. For Data Drift, you can do the following:

   * Select the default test type to use when a new model is registered.

   * Set the default test condition and threshold for each test type to use when a new model is registered.

6. For Model Quality, you can do the following:

   * Set the default test condition and threshold for each metric to use when a new model is registered.

7. Click **Save**.


# Apply monitor settings
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/7-monitor-settings/index

Manage the API token, Health Dashboard, notification channels, and test defaults shared by Model Monitor and Domino endpoint monitoring.

The following settings impact both the Model Monitor and monitoring in the Domino endpoints.

## Access the Model Monitor settings

1. From the navigation pane, click **Model Monitor**.

2. Click **Settings**.

3. From the Settings page, click the settings to manage.

Learn more about the different sections in the following topics:

* [API token](/cloud/platform-capabilities/features/monitoring/7-monitor-settings/1-api).

* [Health dashboard](/cloud/platform-capabilities/features/monitoring/7-monitor-settings/2-health).

* [Notification channels](/cloud/platform-capabilities/features/monitoring/7-monitor-settings/3-notification).

* [Test defaults](/cloud/platform-capabilities/features/monitoring/7-monitor-settings/4-tests).


# Supported binning methods
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/8-json/1-supported-binning

Configure numerical and categorical bins for Model Monitor with binsNum, binsEdges, and binsCategories, including guard bins and input limits.

Bins are used to represent probability distributions and divergence values for data drift. The number of bins impacts the quality of drift values and in general, Model Monitor’s performance itself. If you have more than 20 bins, this can cause false alarms which can impact performance.

By default, Model Monitor uses the Freedman Diaconis Estimator method to calculate the number of bins for numerical variables. If this method returns a count higher than 20, then the count is capped at 20.

For numerical variables, the Model Monitor automatically adds one guard bin for values that fall outside the minimum and maximum range of the values present in the training data. For training data this guard bin has a zero count (unless the user uses the ‘binEdges' override strategy). However, for Prediction data, values might fall in this bin, indicating that prediction data has values outside the min-max seen on the training data.

For categorical variables, all unique class values are used as bins. The Model Monitor automatically adds one guard bin ‘Untrained Classes'. For training data, this guard bin has zero counts (unless the `binCategories` override strategy is used). However, for Prediction data, counts of all classes that were not present in the training data fall in this bin. You can use this bin to detect new classes previously unseen during training.

Use the following attributes in the [Monitoring config JSON](/cloud/platform-capabilities/features/monitoring/8-json) to override these defaults and fine tune the bin creation.

<Note>
  After a model is registered, you can’t change bins.
</Note>

For numerical data columns, you can use one of the following approaches:

* **binsNum**

  * This takes a positive integer >= 2 and > 20 as input.

  * The Model Monitor creates that number of equal sized bins for the numerical variable.

  * The Model Monitor uses the max and min value in the training dataset to determine the bin widths.

  * The Model Monitor adds two guard bands in addition to the user-defined bins.

  * For example:

    * `` “binsNum”: 10` ``

* **binsEdges**

  * This takes an array of real numbers as input.

  * Edges can be both positive and negative decimal numbers (except Infinity).

  * These correspond to actual bin edges.

  * To create N user-defined bins, users must provide N+1 bin edges.

  * You can provide a minimum of 3 and maximum of 20 numbers or edges in the array.

  * They must monotonically increase (lowest to highest) from the start of the array to end of the array.

  * This is similar to histogram\_bin\_edges method used in Numpy.

  * The Model Monitor adds two guard bands in addition to the user-defined bins.

  * All provided values must be unique.

  * For example:

    * `“binsEdges”: [-10, -4.5, -0.25, 0, 3.2, 5.11111]`

  * Examples of invalid `“binsEdges”`:

    * `“binsEdges”: [-10, 4, -0.25, 0, 3.2, 5.11111]` –> not monotonically increasing

    * `“binsEdges”: [-10, XYZ, -0.25, 0, 3.2, 5.11111]` –> string value present

    * `“binsEdges”: [1,2]` –> less than 3 edges provided

    * `“binsEdges”: [1,2,2,4,6]` –> duplicates present

For categorical data columns, you can use the following approach:

* **binsCategories**

  * This takes an array of strings as input (length must be less than 100) and creates a bin for each of them.

  * The values must ideally correspond to class values present in the data column in the training data or class values that you expect to find in the prediction data.

  * Counts of all other class values of the training and prediction data columns fall in the 'Untrained Classes' guard bin.

  * If the user has specified an `Untrained Classes` bin as part of the `binsCategories`, then it corresponds to the internal `Untrained Classes` bin.

  * Example:

    * `“binsCategories”: [“red”, “blue”, “green”, “white”, “yellow”]`


# Use the Monitoring Config JSON
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/8-json/index

Define variables, dataset details, model metadata, feature importance, and binning options in the Monitoring Config JSON used to register data.

The Monitoring Config JSON captures all information required to register a model, a prediction dataset, or a ground truth dataset. This section describes the structure of the Monitoring Config JSON file. The following is a sample config:

```JSON theme={null}
{
        "variables": [
                {
                        "name": "age",
                        "valueType": "numerical",
                        "variableType": "feature",
                        "featureImportance": 0.9
                },
                {
                        "name": "y",
                        "valueType": "categorical",
                        "variableType": "prediction"
                },
                {
                        "name": "date",
                        "valueType": "datetime",
                        "variableType": "timestamp"
                },
                {
                        "name": "RowId",
                        "valueType": "string",
                        "variableType": "row_identifier"
                }
        ],
        "datasetDetails": {
                "name": "TrainingData.csv",
                "datasetType": "file",
                "datasetConfig": {
                        "path": "TrainingData.csv",
                        "fileFormat": "csv"
                },
                "datasourceName": "abc-shared-bucket",
                "datasourceType": "s3"
        },
        "modelMetadata": {
                "name": "test_psg",
                "modelType": "classification",
                "version": "2",
                "description": "",
                "author": "testadmin"
        }
}
```

## variables

An array of variables that declare all features and prediction columns that you want to analyze. For each member in the array, specify the `name`, `variableType`, and `valueType`.

### name

Name of the column.

### variableType

`variableType` provides the attribute that identifies the column. Supported types are:

* **feature**

  * Can only be of `valueType` numerical or categorical.

  * Input feature of the model.

  * Data drift is calculated for this data column.

  * Must be declared while registering the model along with its training data.

  * The column must be present in all training and prediction datasets registered with the model.

* **prediction** (optional)

  * When declared, there can only be one Prediction column.

  * Can only be of `valueType` numerical or categorical.

  * Output prediction of the model.

  * Data drift and model quality metrics are calculated for this data column. Include this column when registering your model to ensure both data drift and model quality analysis can be run. If it isn’t included, model quality metrics won’t be computed.

* **timestamp** (optional)

  * When present, there can be only one `timestamp` column.

  * Can only be of datetime `valueType`.

  * Although you can declare this column when adding prediction data for the first time, Domino recommends that it be declared during model registration.

  * Identifies the column that contains the timestamp for the prediction made. If not declared, the ingestion time of the data in the Model Monitor is used as the timestamp of the prediction.

  * Must contain the date/time when the prediction was made. Column values must follow the ISO 8601 time format.

  * When it is not declared, the ingestion time of the prediction dataset into the Model Monitor is substituted as the timestamp of prediction.

  * To use automatic ingestion for Snowflake, you must include this column. [Snowflake’s documentation](https://docs.snowflake.com/en/user-guide/spark-connector-use.html#working-with-timestamps-and-time-zones) recommends setting the timezone to UTC for both the Spark cluster and the Snowflake deployment.

* **row\_identifier** (optional)

  * Can only be of string `valueType`.

  * Uniquely identifies each prediction row. Typically referred to as prediction ID, transaction ID, and so on.

  * When present, there can be only one `row_identifier` column.

  * Although you can declare this column when adding prediction data for the first time, Domino recommends that it be declared during model registration.

  * Values are used to match ground truth data to the predictions to calculate model quality metrics. Model quality metrics are not calculated if this column is not present. If used, must be present in both prediction and ground truth datasets.

* **ground\_truth**

  * Identifies the column that contains the ground truth labels in the ground truth datasets.

* **sample\_weight**

  * Column that contains the weight to be associated with each prediction to calculate the Gini Norm metric.

* **prediction\_probability**

  * Column that contains the probability value for the model’s prediction. Can be a single value (maps to the probability value of the positive class) or a list of values (the length of the list must match the number of unique prediction labels or classes present in the training dataset).

<Note>
  <span />

  For a field of this type, include `forPredictionOutput` to indicate the prediction column for which you are specifying the probability. This column is required if you want to measure AUC, ROC, Log Loss, and Gini Norm as part of the model quality analysis.
</Note>

### valueType

Identifies the value of the column. Supported types are `categorical`, `numerical`, `datetime`, or `string`. Can only be set to `datetime` for one column.

### forPredictionOutput

Specifies which prediction column the ground truth variable represents in Ground Truth Config.

## datasetDetails

### name

The name to associate with this dataset instance. You can use the same name as the file you are selecting.

### datasetType

The supported type is `file`.

### datasetConfig

Defines the actual location of the file.

* `path`: The name of the file. If you are using Snowflake, the path is the name of the Snowflake table.

* `fileFormat`: .csv and parquet are supported.

If you are using Snowflake, the path must be `snowflake`.

### datasourceName

Name you provided when you created the [data source](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/2-connect-a-data-source).

### datasourceType

The supported data source types are: `amazon_s3`, `generic_s3`, `gcs`, `azure_blob`, `azure_data_lake_gen1`, `azure_data_lake_gen2`, `hdfs`, or `snowflake`.

## modelMetadata

Captures metadata related to the model.

Specify the `name`, `modelVersion`, `modelType`, `dataset`, `dateCreated`, `description`. and `author` attributes.

`dateCreated` must be in a valid UTC format (ISO 8601). Valid values for `modelType` are `classification` and `regression`.

## featureImportance

Highlights the overall impact of the feature on predictions made to the model, relative to other features.

## Supported binning methods

Bins are used to represent probability distributions and divergence values for data drift. The number of bins impacts the quality of drift values and in general, Model Monitor’s performance itself.

For more information, see [Supported binning methods](/cloud/platform-capabilities/features/monitoring/8-json/1-supported-binning).


# Use the Model Monitoring APIs
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/9-model-monitoring-apis

Use the Model Monitoring API with an API token to access Model Monitor and ingest prediction data captured from Domino endpoints.

The Model Monitoring API gives you programmatic access to the [Model Monitor](/cloud/platform-capabilities/features/monitoring).

To invoke these APIs, you must pass the API token in the header. See [API Token](/cloud/platform-capabilities/features/monitoring/7-monitor-settings/1-api) for more information.

Domino Model Monitoring can only ingest prediction data for Domino endpoints in the form of [Domino Datasets](/cloud/admin/data-administration/datasets) captured using [Domino’s prediction capture library](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/1-set-up-prediction-capture). To ingest different data types, create a new model in Domino Model Monitor and use a Model Monitor Data Source.


# Monitor models
Source: https://docs.domino.ai/cloud/platform-capabilities/features/monitoring/index

Monitor production models for data drift and model quality, schedule checks and alerts, and use APIs to ingest data or trigger retraining.

Domino’s Model Monitoring uses data from several supported [data sources](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/2-connect-a-data-source) to analyze models in production and alerts you when a model’s performance falls outside the specified range.

Model Monitoring uses training and prediction data to track drift for the model’s input features and prediction variables, and alerts you about every feature that exceeds a configurable threshold. If you have ground truth data for the model’s predicted values, Domino can ingest it to produce model quality metrics using standard measures such as accuracy, precision, and recall. Domino can also alert you about every metric that exceeds a threshold.

You can also:

* Use the monitoring dashboard to observe your data science models.

* Set up your model to be continuously monitored, using APIs to ingest prediction and ground truth data when available, and define scheduled checks to alert about drift and model quality metrics periodically.

* Use APIs to integrate into existing business processes, and programmatically retrain models.

To find out more, see:

* [Monitor workflows](/cloud/platform-capabilities/features/monitoring/1-monitor-workflows).

* [Data drift and quality monitoring](/cloud/platform-capabilities/features/monitoring/2-data-drift-and-model-quality).

* [Set up monitoring for Domino endpoints](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints).

* [Set up Model Monitor](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor).

* [Use monitoring](/cloud/platform-capabilities/features/monitoring/5-use-monitoring).

* [Remediate model issues](/cloud/platform-capabilities/features/monitoring/6-remediation).

* [Apply monitor settings](/cloud/platform-capabilities/features/monitoring/7-monitor-settings).

* [Use the Monitoring Config JSON](/cloud/platform-capabilities/features/monitoring/8-json).

* [Use the Model Monitoring APIs](/cloud/platform-capabilities/features/monitoring/9-model-monitoring-apis).

* [Troubleshoot the Model Monitor](/cloud/platform-capabilities/features/monitoring/10-troubleshoot).


# Set up Domino on EKS, AKS and GKE
Source: https://docs.domino.ai/cloud/admin/architecture-and-deployment/architecture/domino-on-eks-aks-gke

Provision Domino clusters on EKS, AKS, or GKE with the required node pools, networking, storage, domains, and provider-specific resources.

Domino can run on a Kubernetes cluster provided by [AWS Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/), [Azure Kubernetes Service (AKS)](https://azure.microsoft.com/en-us/services/kubernetes-service/), and [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine/).

To find out how to set up Domino on Kubernetes clusters provided by EKS, AKS, or GKE, select the relevant tab below:

<Tabs>
  <Tab title="EKS">
    When running on EKS, the Domino architecture uses AWS resources to fulfill the Domino cluster requirements as follows:

    <Frame>
      <img alt="A map of the Amazon Web Services architecture behind a Domino deployment" />
    </Frame>

    * EKS features a fully managed Kubernetes control plane.

    * Domino uses a dedicated Auto Scaling Group (ASG) of EKS workers to host the Domino platform.

    * ASGs of EKS workers host elastic compute for Domino executions.

    * AWS S3 stores user data, internal Docker registry, backups, and logs.

    * Elastic Container Registry can be configured as an external Docker registry.

    * AWS EFS stores Domino Datasets.

    * The `ebs.csi.aws.com` provisioner creates persistent volumes for Domino executions.

    * [Calico](https://docs.aws.amazon.com/eks/latest/userguide/calico.html) is a network plugin that supports [Kubernetes network policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/).

    * Domino cannot be installed on EKS Fargate, since Fargate does not support stateful workloads with persistent volumes.

    * Domino recommends provisioning via our [Terraform modules](https://github.com/dominodatalab/terraform-aws-eks).

    * Domino recommends nodes have private IPs fronted by a load balancer with proper security controls. Nodes in the cluster can egress to the Internet through a NAT gateway.

    Note that the use of GPU compute instances is optional.

    Your annual Domino license fee will not include any charges incurred from using AWS services. You can find detailed pricing information for the Amazon services listed above at [https://aws.amazon.com/pricing](https://aws.amazon.com/pricing).

    ## Set up an EKS cluster for Domino

    <Warning>
      Domino maintains [Terraform modules](https://github.com/dominodatalab/terraform-aws-eks) that provision AWS infrastructure in your VPC in a manner consistent with how we validate Domino internally. To promote standardization, stability, and easier troubleshooting, Domino highly recommends that you deploy our Terraform, working with a Domino Solutions Architect to customize as needed.
    </Warning>

    This section describes how to configure an Amazon EKS cluster for use with Domino. You should be familiar with the following AWS services:

    * Elastic Kubernetes Service (EKS)

    * Identity and Access Management (IAM)

    * Virtual Private Cloud (VPC) Networking

    * Elastic Block Store (EBS)

    * Elastic File System (EFS)

    * S3 Object Storage

    Additionally, a basic understanding of Kubernetes concepts like node pools, network CNI, storage classes, autoscaling, and Docker are useful when deploying the cluster.

    ### Security considerations

    You must create IAM policies in the AWS console to provision an EKS cluster. Domino recommends that you grant the least privilege when you create IAM policies. Grant elevated privileges when necessary. See information about the [grant least privilege concept](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege).

    ### Service quotas

    Amazon maintains default service quotas for each of the services listed previously. Log in to the [AWS Service Quotas console](https://console.aws.amazon.com/servicequotas/home) to check the [default service quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html) and manage your quotas.

    ### VPC networking

    If you plan to do [VPC peering](https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html) or set up a [site-to-site VPN connection](https://docs.aws.amazon.com/vpn/latest/s2svpn/VPC_VPN.html) to connect your cluster to other resources like data sources or authentication services, [configure your cluster VPC accordingly](https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html) to avoid address space collisions.

    ### Namespaces

    You do not have to configure namespaces prior to installation. Domino will create the following namespaces in the cluster during installation, according to the following specifications:

    | Namespace | Contains |
    | - | - |
    | `domino-platform` | Persistent Domino platform services and metadata required for platform operation (control plane). |
    | `domino-compute` | Ephemeral Domino execution pods launched by user actions in the application (workspaces, Domino endpoints, apps, etc.). |
    | `domino-system` | Domino installation metadata and secrets. |

    ### Node pools

    The EKS cluster must have at least two ASGs that produce worker nodes with the following specifications and distinct node labels, and it might include an optional GPU pool:

    | Pool | Min-Max | Instance | Disk | Labels |
    | - | - | - | - | - |
    | `platform` | 4-6 | m7i-flex.2xlarge | 128G | `dominodatalab.com/node-pool: platform` |
    | `compute` | 1-20 | m6i.2xlarge | 400G | `dominodatalab.com/node-pool: default` `domino/build-node: true` |
    | Optional: `gpu` | 0-5 | g5.2xlarge | 400G | `dominodatalab.com/node-pool: default-gpu` `nvidia.com/gpu: true` |

    The `platform` ASG can run in one availability zone or across three availability zones. If you want Domino to run with some components deployed as highly available [ReplicaSets](https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/) you must use three availability zones. Using two zones is not supported, as it results in an even number of nodes in a single failure domain. All compute node pools you use must have corresponding ASGs in any AZ used by other node pools. If you set up an isolated node pool in one zone, you might encounter volume affinity issues.

    To run the `default` and `default-gpu` pools across multiple availability zones, you must duplicate ASGs in each zone with the same configuration, including the same labels, to ensure pods are delivered to the zone where the required ephemeral volumes are available.

    To get suitable drivers onto GPU nodes, use the [EKS-optimized accelerated AMI distributed by Amazon](https://docs.aws.amazon.com/eks/latest/userguide/gpu-ami.html) as the machine image for the GPU node pool.

    You can add ASGs with distinct `dominodatalab.com/node-pool` labels to make other instance types available for Domino executions. See [Manage Compute Resources](/cloud/admin/infrastructure-and-compute/manage-compute-resources) to learn how these different node types are referenced by labels from the Domino application.

    ### Network plugin

    Domino relies on [Kubernetes network policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) to manage secure communication between pods in the cluster. The network plugin implements network policies, so your cluster must use a networking solution that supports `NetworkPolicy`, such as [Calico](https://www.tigera.io/project-calico/).

    See the [AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/calico.html) about installing Calico for your EKS cluster.

    If you use the [Amazon VPC CNI](https://github.com/aws/amazon-vpc-cni-k8s) for networking, with only NetworkPolicy enforcement components of Calico, ensure the subnets you use for your cluster have CIDR ranges of sufficient size, as every deployed pod in the cluster will be assigned an elastic network interface and consume a subnet address. Domino recommends at least a /23 CIDR for the cluster.

    ### Dynamic block storage

    The EKS cluster must be equipped with an EBS-backed storage class that Domino will use to provision ephemeral volumes for user execution. GP2 and GP3 volume types are supported. See the following for an example storage class specification:

    ```yaml theme={null}
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: dominodisk-gp3
    parameters:
      type: gp3
    provisioner: ebs.csi.aws.com
    reclaimPolicy: Delete
    volumeBindingMode: WaitForFirstConsumer
    ```

    When using GP3, your IAM policy must allow additional permissions to operate on these ephemeral volumes. Use this [example IAM policy](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/helm-chart-aws-ebs-csi-driver-2.1.1/docs/example-iam-policy.json#L85/) as a reference.

    ### Datasets storage

    To store Datasets in Domino, you must configure an EFS (Elastic File System). You must provision the EFS file system and configure an access point to allow access from the EKS cluster.

    Configure the access point with the following key parameters, also shown in the following image.

    * Root directory path: `/domino`

    * User ID: `0`

    * Group ID: `0`

    * Owner user ID: `0`

    * Owner group ID: `0`

    * Root permissions: `777`

    <Frame>
      <img alt="EFS access point" />
    </Frame>

    Record the file system and access point IDs for use when you install Domino.

    ### Blob storage

    When running in EKS, Domino can use Amazon S3 for durable object storage.

    Create the following S3 buckets:

    * One bucket for user data

    * One bucket for the internal Docker registry

    * One bucket for logs

    * One bucket for backups

    Configure each bucket to permit read and write access from the EKS cluster. This means that you must apply an IAM policy to the nodes in the cluster like the following:

    ```json theme={null}
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "s3:ListBucket",
            "s3:GetBucketLocation",
            "s3:ListBucketMultipartUploads"
         ],
          "Resource": [
            "arn:aws:s3:::$your-logs-bucket-name",
            "arn:aws:s3:::$your-backups-bucket-name",
            "arn:aws:s3:::$your-user-data-bucket-name",
            "arn:aws:s3:::$your-registry-bucket-name"
         ]
        },
        {
          "Effect": "Allow",
          "Action": [
            "s3:PutObject",
            "s3:GetObject",
            "s3:DeleteObject",
            "s3:ListMultipartUploadParts",
            "s3:AbortMultipartUpload"
         ],
          "Resource": [
            "arn:aws:s3:::$your-logs-bucket-name/*",
            "arn:aws:s3:::$your-backups-bucket-name/*",
            "arn:aws:s3:::$your-user-data-bucket-name/*",
            "arn:aws:s3:::$your-registry-bucket-name/*"
         ]
        }
     ]
    }
    ```

    Record the names of these buckets for use when you install Domino.

    ### Autoscaler access

    If you intend to deploy the Kubernetes Cluster Autoscaler in your cluster, the instance profile used by your platform nodes must have the necessary [AWS Auto Scaling permissions](https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html).

    See the following example policy:

    ```json theme={null}
    {
     "Version": "2012-10-17",
     "Statement": [
         {
             "Action": [
                 "autoscaling:DescribeAutoScalingGroups",
                 "autoscaling:DescribeAutoScalingInstances",
                 "autoscaling:DescribeLaunchConfigurations",
                 "autoscaling:DescribeTags",
                 "autoscaling:SetDesiredCapacity",
                 "autoscaling:TerminateInstanceInAutoScalingGroup",
                 "ec2:DescribeLaunchTemplateVersions",
                 "ec2:DescribeInstanceTypes"
            ],
             "Resource": "*",
             "Effect": "Allow"
         }
    ]
    }
    ```

    ### Domain

    You must configure Domino to serve from a specific FQDN. To serve Domino securely over HTTPS, you also need an SSL certificate that covers the chosen name. Record the FQDN for use when you install Domino.

    <Warning>
      A Domino install can’t be hosted on a subdomain of another Domino install. For example, if you have Domino deployed at `data-science.example.com`, you can’t deploy another instance of Domino at `acme.data-science.example.com`.
    </Warning>
  </Tab>

  <Tab title="AKS">
    <Frame>
      <img alt="A map of the Azure architecture to set up a Domino deployment" />
    </Frame>

    **Runtime platform ([Azure Kubernetes Service](https://azure.microsoft.com/en-us/products/kubernetes-service)):**\
    **A -** AKS cluster deployed in three [Availability Zones](https://learn.microsoft.com/en-us/azure/reliability/availability-zones-overview). The cluster must match Domino’s compatible Kubernetes versions.

    **Nodes/instances ([AKS node pools](https://learn.microsoft.com/en-us/azure/aks/create-node-pools)):**\
    **B -** System Pool: Scales 1-3 per zone, Standard\_DS4\_v2.\
    **C -** Platform Pool: Scales 1-3 per zone, Standard\_D8S\_v4.\
    **D -** Compute Pool: Scales 0-10 per zone, Standard\_D8S\_v4.\
    **E -** GPU Pool: Scales 0-3 per zone, Standard\_NC6s\_v3.

    **Networking:**\
    **F -** [Ingress load balancer](https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-overview).

    **Storage:**\
    **G -** Environment and model images; [Azure Container Registry](https://azure.microsoft.com/en-us/products/container-registry).\
    **H -** [Storage Account](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview); Azure Files access for shared file system; [Datasets](https://azure.microsoft.com/en-us/products/storage/files); [Azure Blob](https://azure.microsoft.com/en-us/products/storage/blobs) API access for Project files, Logs, and Backups.

    When running on AKS, the Domino architecture uses Azure resources to fulfill the Domino cluster requirements as follows:

    * For a complete Terraform module for Domino-compatible AKS provisioning, see [terraform-azure-aks on GitHub](https://github.com/dominodatalab/terraform-azure-aks).

    * The AKS control plane, with managed Kubernetes masters, handles Kubernetes control.

    * The AKS cluster’s default [node pool](https://docs.microsoft.com/en-us/cli/azure/ext/aks-preview/aks/nodepool?view=azure-cli-latest) is configured to host the Domino platform.

    * Additional AKS node pools provide compute nodes for user workloads.

    * When Domino is deployed in AKS, it is compatible with the `containerd` runtime, which is the AKS default runtime for Kubernetes 1.19 and above.

    * When using the `containerd` runtime, the Azure Container Registry stores Domino images.

    * An [Azure storage account](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-overview) stores Domino blob data and datasets

    * The `kubernetes.io/azure-disk` provisioner creates persistent volumes for Domino executions.

    * The Advanced Azure CNI is used for cluster networking, and Calico enforces network policy.

    * An SSL-terminating [Application Gateway](https://docs.microsoft.com/en-us/azure/application-gateway/overview) that points to a Kubernetes load balancer handles Ingress to the Domino application.

    * Domino recommends provisioning with [Terraform](https://www.terraform.io/) for extended control and customizability of all resources. When you set up your Azure Terraform provider, add a `partner_id` with a value of `31912fbf-f6dd-5176-bffb-0a01e8ac71f2` to enable [usage attribution](https://docs.microsoft.com/en-us/azure/marketplace/azure-partner-customer-usage-attribution#use-terraform). When using AzureRM Provider v4 or later, you must also set `subscription_id` in the provider block or export the `ARM_SUBSCRIPTION_ID` environment variable:

      ```hcl theme={null}
      provider "azurerm" {
        partner_id      = "31912fbf-f6dd-5176-bffb-0a01e8ac71f2"
        subscription_id = var.subscription_id
        features {}
      }
      ```

    ## Set up an AKS cluster for Domino

    This section describes how to configure an AKS cluster for use with Domino.

    ### Resource groups

    You can provision the cluster, storage, and application gateway in an existing [resource group](https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/manage-resource-groups-portal#what-is-a-resource-group). When Azure creates the cluster, it will create a separate resource group that will contain the cluster components themselves.

    ### Namespaces

    You do not have to configure namespaces prior to install. Domino will create three namespaces in the cluster during installation, according to the following specifications:

    | Namespace | Contains |
    | - | - |
    | `platform` | Durable Domino application, metadata, platform services required for platform operation. |
    | `compute` | Ephemeral Domino execution pods launched by user actions in the application. |
    | `domino-system` | Domino installation metadata and secrets. |

    ### Node pools

    The AKS cluster must have at least two node pools: `platform` (for platform nodes) and `default` (for compute nodes). The cluster can also include a `default-GPU` node pool. All node pools must contain worker nodes with the following specifications and distinct node labels:

    | Pool | Min-Max | VM | Disk | Labels |
    | - | - | - | - | - |
    | `platform` | 4-6 | Standard\_DS5\_v2 | 128G | `dominodatalab.com/node-pool: platform` |
    | `default` | 1-20 | Standard\_DS4\_v2 | 128G | `dominodatalab.com/node-pool: default` `domino/build-node: true` |
    | Optional: `default-gpu` | 0-5 | Standard\_NC6\_v3 | 128G | `dominodatalab.com/node-pool: default-gpu` `nvidia.com/gpu: true` |

    The recommended architecture creates the `platform` node pool by configuring the cluster’s initial default node pool. See the following cluster Terraform resource for a complete example.

    ```shell theme={null}
    resource "azurerm_kubernetes_cluster" "aks" {

      name                       = example_cluster
      location                   = "East US"
      resource_group_name        = "example_resource_group"
      dns_prefix                 = "example_cluster"
      private_cluster_enabled    = false

      default_node_pool {
        node_public_ip_enabled = false
        name                  = "platform"
        node_count            = 4
        node_labels           = { "dominodatalab.com/node-pool" : "platform" }
        vm_size               = "Standard_DS5_v2"
        availability_zones    = ["1", "2", "3"]
        max_pods              = 250
        os_disk_size_gb       = 128
        node_taints           = []
        auto_scaling_enabled   = true
        min_count             = 1
        max_count             = 4
      }

      network_profile {
        load_balancer_sku  = "Standard"
        network_plugin     = "azure"
        network_policy     = "calico"
        dns_service_ip     = "100.97.0.10"
        service_cidr       = "100.97.0.0/16"
      }

    }
    ```

    You must add the `default` compute node pool after the cluster is created. This is *not* the initial cluster default node pool, but a separate node pool named `default`. It contains the default Domino compute nodes. See the following node pool Terraform resource for a complete example.

    ```shell theme={null}
    resource "azurerm_kubernetes_cluster_node_pool" "aks" {

      node_public_ip_enabled = false
      kubernetes_cluster_id = "example_cluster_id"
      name                  = "default"
      node_count            = 1
      vm_size               = "Standard_DS4_v2"
      availability_zones    = ["1", "2", "3"]
      max_pods              = 250
      os_disk_size_gb       = 128
      os_type               = "Linux"
      node_labels = {
        "domino/build-node"            = "true"
        "dominodatalab.com/build-node" = "true"
        "dominodatalab.com/node-pool"  = "default"
      }
      node_taints           = []
      auto_scaling_enabled  = true
      min_count             = 1
      max_count             = 20

    }
    ```

    You can add node pools with distinct `dominodatalab.com/node-pool` labels to make other instance types available for Domino executions. See [Manage Compute Resources](/cloud/admin/infrastructure-and-compute/manage-compute-resources) to learn how these different node types are referenced by label from the Domino application. When you add GPU node pools, consider the Azure best practices on [using GPU nodes in AKS](https://docs.microsoft.com/en-us/azure/aks/gpu-cluster).

    ### Network plugin

    The Domino-hosting cluster must use the Advanced Azure CNI with network policy enforcement by Calico. See the following `network_profile` configuration example.

    ```shell theme={null}
    network_profile {
      load_balancer_sku  = "Standard"
      network_plugin     = "azure"
      network_policy     = "calico"
      dns_service_ip     = "100.97.0.10"
      service_cidr       = "100.97.0.0/16"
    }
    ```

    ### Dynamic block storage

    AKS clusters come equipped with several `kubernetes.io/azure-disk` backed storage classes by default. Domino requires use of premium disks for adequate input and output performance. You can use the `managed-premium` class that is created by default. Consult the following storage class specification as an example.

    ```yaml theme={null}
    allowVolumeExpansion: true
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      labels:
        kubernetes.io/cluster-service: "true"
      name: managed-premium
      selfLink: /apis/storage.k8s.io/v1/storageclasses/managed-premium
    parameters:
      cachingmode: ReadOnly
      kind: Managed
      storageaccounttype: Premium_LRS
    reclaimPolicy: Delete
    volumeBindingMode: Immediate
    ```

    ### Persistent blob and data storage

    Domino uses one Azure storage account for both blob data and files. See the following configuration for the two resources required, the storage account itself and a blob container inside the account.

    ```shell theme={null}
    resource "azurerm_storage_account" "domino" {
      name                     = "example_storage_account"
      resource_group_name      = "example_resource_group"
      location                 = "East US"
      account_kind             = "StorageV2"
      account_tier             = "Standard"
      account_replication_type = "LRS"
      access_tier              = "Hot"
    }

    resource "azurerm_storage_container" "domino_registry" {
      name                  = "docker"
      storage_account_name  = "example_storage_account"
      container_access_type = "private"
    }
    ```

    Record the names of these resources for use when you install Domino.

    ### Domain

    Domino must be configured to serve from a specific FQDN. To serve Domino securely over HTTPS, you need an SSL certificate that covers the chosen name. Record the FQDN for use when installing Domino.

    <Warning>
      A Domino install can’t be hosted on a subdomain of another Domino install. For example, if you have Domino deployed at `data-science.example.com`, you can’t deploy another instance of Domino at `acme.data-science.example.com`.
    </Warning>
  </Tab>

  <Tab title="GKE">
    <Frame>
      <img alt="A map of the Google Cloud architecture behind a Domino deployment" />
    </Frame>

    When running on GKE, the Domino architecture uses Google Cloud Provider (GCP) resources to fulfill the Domino cluster requirements as follows:

    * The [GKE cluster](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture) manages Kubernetes control.

    * Domino uses one [node pool](https://cloud.google.com/kubernetes-engine/docs/concepts/node-pools) of three [n1-standard-8](https://cloud.google.com/compute/docs/machine-types) worker nodes to host the Domino platform.

    * Additional node pools host elastic compute for Domino executions with optional GPU accelerators.

    * [Cloud Filestore](https://cloud.google.com/filestore/docs/) stores user data, backups, logs, and Domino Datasets.

    * A [Cloud Storage Bucket](https://cloud.google.com/storage/docs/creating-buckets) stores the Domino Docker Registry.

    * The `kubernetes.io/gce-pd` provisioner creates persistent volumes for Domino executions.

    ## Set up a GKE cluster for Domino

    This section describes how to configure an GKE cluster for use with Domino.

    ### Namespaces

    You don’t have to configure namespaces prior to install. Domino creates the following namespaces in the cluster during installation, according to the following specifications:

    | Namespace | Contains |
    | - | - |
    | `platform` | Durable Domino application, metadata, platform services required for platform operation. |
    | `compute` | Ephemeral Domino execution pods launched by user actions in the application. |
    | `domino-system` | Domino installation metadata and secrets. |

    ### Node pools

    The GKE cluster must have at least two node pools that produce worker nodes with the following specifications and distinct node labels, and it might include an optional GPU pool:

    | Pool | Min-Max | Instance | Disk | Labels |
    | - | - | - | - | - |
    | `platform` | 4-6 | n1-standard-8 | 128G | `dominodatalab.com/node-pool: platform` |
    | `default` | 1-20 | n1-standard-8 | 400G | `dominodatalab.com/node-pool: default` `domino/build-node: true` |
    | Optional: `default-gpu` | 0-5 | n1-standard-8 | 400G | `dominodatalab.com/node-pool: default-gpu` |

    If you want to configure the `default-gpu` pool, you must add a GPU accelerator the node pool. See the GKE documentation about [available accelerators](https://cloud.google.com/kubernetes-engine/docs/how-to/gpus) and deploying a DaemonSet that automatically installs the [necessary drivers](https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#installing_drivers).

    You can add node pools with distinct `dominodatalab.com/node-pool` labels to make other instance types available for Domino executions. See [Manage Compute Resources](/cloud/admin/infrastructure-and-compute/manage-compute-resources) to learn how these different node types are referenced by labels from the Domino application.

    Consult the following [Terraform](https://www.terraform.io/) snippets for code representations of the required node pools.

    **Platform pool**

    ```shell theme={null}
    resource "google_container_node_pool" "platform" {
      name     = "platform"
      location = $YOUR_CLUSTER_ZONE_OR_REGION
      cluster  = $YOUR_CLUSTER_NAME

      initial_node_count = 3
      autoscaling {
        max_node_count = 3
        min_node_count = 3
      }

      node_config {
        preemptible  = false
        machine_type = "n1-standard-8"

        labels = {
          "dominodatalab.com/node-pool" = "platform"
        }

        disk_size_gb    = 128
        local_ssd_count = 1
      }

      management {
        auto_repair  = true
        auto_upgrade = true
      }

      timeouts {
        delete = "20m"
      }
    }
    ```

    **Default compute pool**

    ```shell theme={null}
    resource "google_container_node_pool" "compute" {
      name     = "compute"
      location = $YOUR_CLUSTER_ZONE_OR_REGION
      cluster  = $YOUR_CLUSTER_NAME

      initial_node_count = 1
      autoscaling {
        max_node_count = 20
        min_node_count = 1
      }

      node_config {
        preemptible  = false
        machine_type = "n1-standard-8"

        labels = {
          "domino/build-node"            = "true"
          "dominodatalab.com/build-node" = "true"
          "dominodatalab.com/node-pool"  = "default"
        }

        disk_size_gb    = 400
        local_ssd_count = 1
      }

      management {
        auto_repair  = true
        auto_upgrade = true
      }

      timeouts {
        delete = "20m"
      }
    }
    ```

    **Optional GPU pool**

    ```shell theme={null}
    resource "google_container_node_pool" "gpu" {
      provider = google-beta
      name     = "gpu"
      location = $YOUR_CLUSTER_ZONE_OR_REGION
      cluster  = $YOUR_CLUSTER_NAME

      initial_node_count = 0

      autoscaling {
        max_node_count = 5
        min_node_count = 0
      }

      node_config {
        preemptible  = false
        machine_type = "n1-standard-8"

        guest_accelerator {
          type  = "nvidia-tesla-p100"
          count = 1
        }

        labels = {
          "dominodatalab.com/node-pool" = "default-gpu"
        }

        disk_size_gb    = 400
        local_ssd_count = 1

        workload_metadata_config {
          node_metadata = "GKE_METADATA_SERVER"
        }
      }

      management {
        auto_repair  = true
        auto_upgrade = true
      }

      timeouts {
        delete = "20m"
      }
    }
    ```

    ### Network policy enforcement

    Domino relies on [Kubernetes network policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) to manage secure communication between pods in the cluster. By default, the network plugin in GKE will not enforce these policies. To run Domino securely on GKE, you must enable enforcement of network policies.

    See the [GKE documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/network-policy) for instructions about how to enable network policy enforcement for your cluster.

    ### Dynamic block storage

    The Domino installer will automatically create a storage class like the following example for use provisioning GCE persistent disks as [execution volumes](/cloud/admin/infrastructure-and-compute/manage-compute-resources/manage-persistent-volumes). No manual setup is necessary for this storage class.

    ```yaml theme={null}
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: dominodisk
    parameters:
      replication-type: none
      type: pd-standard
    provisioner: kubernetes.io/gce-pd
    reclaimPolicy: Delete
    volumeBindingMode: WaitForFirstConsumer
    ```

    ### Shared storage

    A Cloud File store instance must be provisioned with at least 10T of capacity and it must be configured to allow access from the cluster. You must provide the IP address and mount path of this instance to the Domino installer, and it will create a Network File System (NFS) storage class like the following.

    ```yaml theme={null}
    allowVolumeExpansion: true
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      labels:
        app.kubernetes.io/instance: nfs-client-provisioner
        app.kubernetes.io/managed-by: Tiller
        app.kubernetes.io/name: nfs-client-provisioner
        helm.sh/chart: nfs-client-provisioner-1.2.6-0.1.4
      name: domino-shared
    parameters:
      archiveOnDelete: "false"
    provisioner: cluster.local/nfs-client-provisioner
    reclaimPolicy: Delete
    volumeBindingMode: Immediate
    ```

    ### Docker registry storage

    You need one [Cloud Storage Bucket](https://cloud.google.com/storage/docs/creating-buckets) accessible from your cluster to be used to store the internal Domino Docker Registry.

    ### Domain

    You must configure Domino to serve from a specific FQDN. To serve Domino securely over HTTPS, you need an SSL certificate that covers the chosen name. Record the FQDN for use when you install Domino.

    <Warning>
      A Domino install can’t be hosted on a subdomain of another Domino install. For example, if you have Domino deployed at `data-science.example.com`, you can’t deploy another instance of Domino at `acme.data-science.example.com`.
    </Warning>

    After Domino is deployed into your cluster, you must set up DNS for this name to point to an [HTTPS Cloud Load Balancer](https://cloud.google.com/load-balancing/docs/choosing-load-balancer) that has an SSL certificate for the chosen name, and forwards traffic to port 80 on your platform nodes.
  </Tab>
</Tabs>

## Next steps

* Deploy workloads across multiple Kubernetes clusters with Domino’s [Nexus Hybrid Architecture](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture).

* Learn about how Domino uses Keycloak to manage [user accounts](/cloud/admin/identity-and-access).


# Architecture overview
Source: https://docs.domino.ai/cloud/admin/architecture-and-deployment/architecture/index

Understand how Domino separates clients, Control Plane services, Data Plane workloads, and external storage, data, Git, and package systems.

Domino is distributed as a set of containerized, [Kubernetes](https://kubernetes.io) native applications. A set of accompanying Helm charts and an installer is used to manage Domino installation and upgrades. Domino is validated on major Kubernetes services in the public cloud (EKS, AKS, and GKE). It is also possible to run Domino in self-managed Kubernetes environments in your private data center.

The following diagram shows Domino’s conceptual architecture.

<Frame>
  <img alt="Clients reach the Control Plane and the Data Plane over HTTPS. The Control Plane is one Kubernetes cluster running the Domino UI, Domino API, message broker, secrets vault, identity, application data stores, experiment manager, model registry, environment image builder, and other supporting services. The Data Plane runs the Data Plane agent, the cluster autoscaler, and Domino workloads. Both planes reach external services." />

  <img alt="Clients reach the Control Plane and the Data Plane over HTTPS. The Control Plane is one Kubernetes cluster running the Domino UI, Domino API, message broker, secrets vault, identity, application data stores, experiment manager, model registry, environment image builder, and other supporting services. The Data Plane runs the Data Plane agent, the cluster autoscaler, and Domino workloads. Both planes reach external services." />
</Frame>

The Domino application is separated into three primary areas of concern:

**Clients**\
External clients interact with Domino via HTTPS. Clients can take the form of a command-line interface, web browsers, and programmatic API clients (including [Domino endpoint](/cloud/platform-capabilities/features/model-deployment) clients).

**Control Plane (Platform)**\
The Domino control plane houses platform components including the Domino UI server, Domino API server, container orchestration, storage of metadata, and various supporting services.

**Data Plane (Compute)**\
The Domino data plane is where user workloads are run including machine learning and AI Jobs, models, Apps, and data science Workspaces.

The remainder of this page describes various services and the role they play.

## Control Plane services

The following services run in the Control Plane:

**Domino UI**\
A Node.js web server that serves a single-page React application to clients using web browsers. Primary users include data science practitioners and Domino admins.

**Secrets Vault**\
Storage for sensitive data including credentials, tokens, certificates, and API keys.

**Domino API**\
Exposes API endpoints that handle REST API requests from various clients (CLI, web browser, programmatic clients).

**Message Broker**\
A message broker (RabbitMQ) facilitating asynchronous communication between Domino services.

**Identity**\
Domino’s authentication service used for storing user identities and properties. Identity federation to SSO systems and third party identity providers is possible.

**Application Data Stores**\
Various application databases (for example, MongoDB, PostgreSQL) store Domino metadata including Projects, users, and organizations. Application data stores can store references to blob storage stored in object storage systems (for example, Amazon S3).

**Experiment Manager and Model Registry**\
Built on MLflow. Tracks and monitors data science experiments with logging, and lets users view and compare across experiments and collaborate securely. Also discovers and tracks models, records model metadata for auditability and reproducibility, and handles model versioning.

**Environment Image Builder**\
The Image Builder is capable of building Domino Environments, based on Docker containers. These Environments can include user-specified libraries needed to support data science work. It also builds Domino endpoint images.

## Data Plane services

The following services run in the Data Plane clusters:

**Data plane agent**\
Applies resource changes to remote data planes. Responsible for running workloads on remote clusters.

**Cluster Autoscaler**\
Adds additional compute capacity to the cluster dynamically based on resource requirements (for example, CPU, memory, GPU).

**Domino Workspaces**\
Used for housing interactive data science Environments (for example, Jupyter Notebooks). Implemented as ephemeral pods that must be synced to long-term storage.

**Domino Apps**\
User-defined web applications able to serve user requests, hosted within Domino. Supports frameworks including Flask, React, FastAPI, Streamlit, Plumber, and Dash.

**Domino endpoints**\
Model inference endpoints hosted within Domino. Implemented as ephemeral pods.

**Domino Jobs**\
Short-lived batch Jobs run by Domino. Implemented as ephemeral pods.

**Distributed Compute Clusters**\
Use clustered compute environments to scale out compute-intensive workloads in Domino, including Spark, Dask, Ray, MPI, Slurm, and Slinky.

<img alt="Tools and frameworks compatible with Domino workloads" />

## External & additional services

The following services execute outside of Domino clusters:

**Object Storage**\
Object (blob) storage, used for durably storing logs, container images, Domino backups, and various user data.

**File shares**\
File-based storage, used to store Domino entities like Datasets.

**Remote data sources**\
Pull data residing in remote data sources into data science environments.

**Git providers**\
Domino uses Git to revision projects and files. Git clients residing in the data plane can also be used to interact with external repositories to access code or data.

**Environment registry**\
Select from a catalog of predefined and user-defined Environments containing libraries needed to support data science.

**Package repository**\
A Compute Environment can pull packages from an internal or enterprise package repository, such as Artifactory, instead of the public internet.

## Next steps

* Set up Domino on a Kubernetes cluster provided by [AWS Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), and Google Kubernetes Engine (GKE)](/cloud/admin/architecture-and-deployment/architecture/domino-on-eks-aks-gke).

* Deploy workloads across multiple Kubernetes clusters with Domino’s [Nexus Hybrid Architecture](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture).

* Work out the resources each type of Domino workload needs with [Size the infrastructure](/cloud/admin/architecture-and-deployment/sizing-the-infrastructure).

* Learn about how Domino uses Keycloak to manage [user accounts](/cloud/admin/identity-and-access).


# Nexus hybrid architecture
Source: https://docs.domino.ai/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture

Deploy workloads across Kubernetes clusters with Domino Nexus, and plan control planes, data planes, connectivity, storage, and load balancers.

Domino Nexus provides a hybrid architecture that enables you to deploy workloads across multiple Kubernetes clusters, including in multiple cloud regions, cloud providers, or on-premises.

<img alt="The Nexus hybrid architecture" />

A Domino Nexus deployment consists of a “control plane”, which is a Kubernetes cluster hosting Domino platform services (above, light blue), and many “data planes” (above, dark blue) which are distinct Kubernetes clusters that run a small set of Domino services and are used for executing user workloads.

The Domino control plane is also capable of executing user workloads in what is called the “local” data plane. This allows for the continued use of Domino features which are not yet supported for remote data planes.

<Note>
  This Domino feature is not available in *remote* data planes: Starburst-powered data sources

  The `local` data plane (hosted in the control plane) supports the full range of Domino features.
</Note>

Your Domino field representative can help you enable Domino Nexus in your deployment. Once it is enabled, you can [manage your data planes](/cloud/admin/platform-configuration/manage-data-planes) and your Domino users can [use data planes](/cloud/platform-capabilities/core-concepts/data-plane).

## Connectivity

User connectivity to Domino Nexus consists primarily of a browser connection to the Domino control plane. For Domino Workspaces, the user connects directly to the data plane where the workspace is running. This avoids the possibility of proxying sensitive or region-locked data through the control plane.

<Frame>
  <img alt="The Domino SaaS control plane AWS account and data plane AWS account, joined by four AWS PrivateLink connections carrying HTTPS 443 for S3, HTTPS 8200 for Vault, AMQPS 5672 and 5552 for RabbitMQ, and HTTPS 443 for Elastic Container Registry. The data plane VPC also peers with the VPC where the data lives." />

  <img alt="The Domino SaaS control plane AWS account and data plane AWS account, joined by four AWS PrivateLink connections carrying HTTPS 443 for S3, HTTPS 8200 for Vault, AMQPS 5672 and 5552 for RabbitMQ, and HTTPS 443 for Elastic Container Registry. The data plane VPC also peers with the VPC where the data lives." />
</Frame>

Each virtual private cloud (VPC) connection below is initiated by the data plane VPC and reaches its counterpart in the single-tenant control plane VPC over its own AWS PrivateLink. This is the PrivateLink topology of the diagram, not the full list of services a control plane exposes to data planes, which [Control Plane load balancers](#control-plane-load-balancers) covers:

| Traffic | Data plane endpoint | Protocol and port |
| :- | :- | :- |
| Log and blob storage in S3 | S3 gateway endpoint | HTTPS 443 |
| Secrets from Vault | Vault VPC endpoint | HTTPS 8200 |
| Messages from RabbitMQ | RabbitMQ VPC endpoint | AMQPS 5672 and 5552 |
| Container images from Elastic Container Registry | ECR gateway endpoint | HTTPS 443 |

Inside the data plane VPC, the data plane cluster stores Datasets on Amazon Elastic File System (EFS). A VPC peering connection reaches the VPC where the data lives: Amazon Redshift, S3, Amazon Bedrock, Elastic Container Registry, Snowflake, and other data.

## Supported Kubernetes versions

Domino validates its software on various Kubernetes versions. Note the following:

* Nexus **control planes** have the same Kubernetes cluster requirements as any other Domino deployment.

* Nexus **data planes** have the same requirements, except:

  * There is no requirement for shared storage (RWX storage class).

  * Only one (compute) namespace is required.

  * Ingress configuration is different (see [Enable a Data Plane for Workspaces](/cloud/admin/platform-configuration/manage-data-planes#enable-a-data-plane-for-workspaces-and-datasets)).

## Control Plane load balancers

Domino control planes expose these services to data planes:

* RabbitMQ

* Vault

* Docker Registry (unless external registry is used)

* Domino API

With the exception of the Domino API, a load balancer must be configured to allow ingress to these services from data planes.

There are a number of important security considerations; see [Control plane security guidance](/cloud/admin/security-and-compliance/control-plane-security).


# Configuration records
Source: https://docs.domino.ai/cloud/admin/architecture-and-deployment/deployment-configuration/configuration-records

Review configuration settings and defaults for Domino Cloud deployments, and contact Domino to request changes to platform limits and behavior.

Configuration records control deployment-wide limits and behavior. On Domino Cloud deployments, only Domino can modify these records. Contact your Domino representative to request a change, specifying the setting and the value you need.

The settings below use their names in the Cloud configuration editor. Unset settings use the listed defaults. Domino applies changes and handles any required service restart.

<Note>
  For self-managed deployments, use the [latest self-managed configuration reference](/self-managed/admin/configuration/central-configuration). It includes the full record names, defaults, and administrator instructions for the latest published self-managed version. Select your deployment's version in the version menu when it differs.
</Note>

## Apps

Controls limits on App publishing within Projects.

<ParamField>
  Maximum number of simultaneously running Apps allowed in a single Project.
</ParamField>

<ParamField>
  Maximum number of Apps that can be created in a single Project.
</ParamField>

## Compute grid

Controls execution queuing behavior across the platform.

<ParamField>
  Maximum number of executions that can be queued across all users at one time.
</ParamField>

<ParamField>
  Maximum number of executions a single user can run concurrently before their additional executions are queued.
</ParamField>

## Environments

Controls environment customization options available to users.

<ParamField>
  When enabled, users can define custom file handlers that override the default behavior for supported file types.
</ParamField>

## Long-running Workspaces

Controls detection, notification, and automatic shutdown policies for long-running Workspaces.

[Configure long-running workspaces](/cloud/platform-capabilities/core-concepts/workspaces/set-up-a-workspace#configure-long-running-workspaces) has more information about the auto shutdown behavior of long-running workspaces.

<ParamField>
  Time in seconds a Workspace must be running before it is considered long-running. Long-running Workspaces are eligible for automatic shutdown and notifications.
</ParamField>

<ParamField>
  Maximum Workspace lifetime in seconds that a user can configure before the platform forces automatic shutdown.
</ParamField>

<ParamField>
  When enabled, users may opt in to receive email notifications for their long-running Workspaces.
</ParamField>

<ParamField>
  When enabled, all users always receive email notifications for their long-running Workspaces, regardless of their personal preference.
</ParamField>

<ParamField>
  When enabled, users can choose to have their long-running Workspaces automatically shut down after the configured threshold.
</ParamField>

<ParamField>
  When enabled, long-running Workspaces are automatically shut down for all users. Users cannot override this behavior.
</ParamField>

<Note>
  * **Auto-Shutdown Required** takes precedence over **Auto-Shutdown Enabled**.

  * **Notifications Required** takes precedence over **Notifications Enabled**.
</Note>

## Restricted environments and Projects

Controls access isolation between environments and Projects.

<ParamField>
  When enabled, administrators can mark individual environments and Projects as restricted. Executions in restricted Projects can only run in environments that are also marked as restricted.
</ParamField>

## Workloads

Controls Workspace provisioning limits per user.

<ParamField>
  Maximum number of Workspaces a single user can have provisioned at one time, across all Projects.
</ParamField>

<ParamField>
  Maximum number of Workspaces a single user can have provisioned within a single Project.
</ParamField>


# Deployment configuration
Source: https://docs.domino.ai/cloud/admin/architecture-and-deployment/deployment-configuration/index

Configure deployment-wide settings through configuration records, and restrict cluster access to trusted networks with IP whitelisting.

Cloud administrators can find the **Deployment Configuration** section on the Domino admin page.

This section explains how to configure different aspects of the Domino Deployment.

[Configuration records](/cloud/admin/architecture-and-deployment/deployment-configuration/configuration-records)\
Set or update global settings or your Domino installation.

[Configure IP whitelisting](/cloud/admin/architecture-and-deployment/deployment-configuration/ip-whitelisting)\
Restrict access to your cluster with IP whitelisting.


# Configure IP whitelisting
Source: https://docs.domino.ai/cloud/admin/architecture-and-deployment/deployment-configuration/ip-whitelisting

Restrict access to trusted IP ranges, automatically retain your current IP, and enable, update, or disable the whitelist from the Admin UI.

IP whitelisting is a security feature that lets you choose which IP addresses can access your Domino cluster. This improves security by allowing only trusted IPs and blocking unauthorized access. You can set up IP whitelisting in the Domino Admin UI.

When you add or remove IP ranges, the changes usually take effect about 10 minutes after you save them.

## Configure IP whitelisting

1. In the admin portal, go to **Platform Settings** > **Deployment Configuration**.

   <img alt="IP Whitelisting configuration main page" />

2. You can enable IP whitelisting by checking the **Enable IP Whitelisting** checkbox.

   <img alt="IP Whitelisting configuration enabled and My IP is showing in the list" />

   It automatically adds your current IP, so there is no way to lock yourself out of the deployment.

3. Click the **Add IP CIDR** button to create a new row where you can add the IP range you want to whitelist.

   1. Optional: add with a description.

4. Click the green check to add the IP to the list.

5. When you are done adding the IP ranges to whitelist, click the **Save** button to save your changes.

   The changes take effect approximately 10 minutes after saving.

## Disable IP whitelisting

To disable IP whitelisting, uncheck the **Enable IP Whitelisting** checkbox, then click the **Save** button.

The changes take effect approximately 10 minutes after saving.


# Domino Cloud
Source: https://docs.domino.ai/cloud/admin/architecture-and-deployment/domino-cloud

Domino Cloud is Domino's fully managed SaaS offering. Because Domino operates the infrastructure, some administrative capabilities available in self-managed deployments are not available.

Domino Cloud is Domino's fully managed software-as-a-service (SaaS) offering, hosted and operated by Domino Data Lab. For an overview and how it compares to the other deployment models, see [Deployment options](/cloud/platform-overview/deployment-options#domino-cloud).

Because Domino operates the underlying infrastructure, some administrative capabilities available in a self-managed deployment are not available in Domino Cloud.

## Limitations

Domino Cloud includes the core functionality of a self-managed Domino installation and most features. The following are not available:

* Attaching external data volumes.
* Some administrator capabilities:
  * Searching over the configuration.
  * Triggering restarts of Domino's backend services.
  * Access to Domino's own MongoDB database.
  * Customizing hardware tiers, resource quotas, configuration records, feature flags, and email settings.
* Access to view the underlying infrastructure, including the Kubernetes cluster and the virtual machine instances.
* Keycloak access for user management.

<Note>
  Domino updates the product over time, so this list evolves.
</Note>


# Kubernetes compatibility
Source: https://docs.domino.ai/cloud/admin/architecture-and-deployment/kubernetes-compatibility

Match Domino Control Planes and Data Planes to validated Kubernetes versions across EKS, AKS, GKE, Red Hat OpenShift, Rancher, and Nexus deployments.

The table below shows the versions of Kubernetes on which Domino validates its software. Validation is performed for the following providers: AWS, Azure, Google Cloud, Red Hat OpenShift, and Rancher.

## Domino Nexus hybrid deployments

Nexus control planes require the same Kubernetes cluster requirements as any other Domino deployment.

However, Nexus data planes have the following differences:

* No requirement for shared storage (RWX storage class).

* Requires only one (compute) namespace.

* Ingress configuration is different, see [Enable a Data Plane for Workspaces](/cloud/admin/platform-configuration/manage-data-planes#enable-a-data-plane-for-workspaces-and-datasets) for more information.

* Kubernetes version compatibility may be different from the control plane (see table below).

## Compatibility matrix

The following matrix shows the supported combinations of Domino and Kubernetes platform versions. For inquiry about unlisted Kubernetes distributions, contact your Domino professional services representative.

End of life (EOL) for Kubernetes versions is dictated by the Kubernetes platform provider. For more information on what EOL means for each platform, see the links in the Kubernetes Platform column.

| | Control plane Domino version |
| - | - |
| **Kubernetes platform** | **6.3.0** |
| [Amazon Elastic Kubernetes Service (EKS)](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar) | 1.34, 1.35 |
| [Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#aks-kubernetes-release-calendar) | 1.34, 1.35 |
| [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine/docs/release-schedule) | 1.34, 1.35 |
| [Red Hat OpenShift](https://access.redhat.com/support/policy/updates/openshift) | 1.34 (4.21) |
| Nexus data plane (non-vCluster \| AWS (EKS)) | 1.34, 1.35 |

## Docker compatibility

Domino doesn’t publish a validated container runtime compatibility matrix and recommends using the default container runtime version provided by your vendor. However, if your Kubernetes nodes use Docker, it should be 20.10.24 or higher.

## Next steps

* [Configure Projects](/cloud/admin/platform-configuration/configure-projects) for users.

* [Manage Compute Environments](/cloud/admin/platform-configuration/manage-compute-environments) used by your organization.


# Size the infrastructure
Source: https://docs.domino.ai/cloud/admin/architecture-and-deployment/sizing-the-infrastructure/index

Plan CPU, memory, node, and storage capacity for the Domino Platform, Compute Grid, data plane agents, and asynchronous endpoints.

Domino runs in [Kubernetes](https://kubernetes.io/), which is an orchestration framework for delivering applications to a distributed compute cluster. The Domino application runs the following types of workloads in Kubernetes; each has different principles to sizing infrastructure:

**Data plane agent**\
Domino recommends increasing resource requests and limits on certain platform pods for larger [Domino Nexus](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture) deployments. The data plane agent is included in this recommendation. See [Size the data plane agent](/cloud/admin/architecture-and-deployment/sizing-the-infrastructure/size-data-plane-agent) for more information.

**Domino Platform**\
These always-on components provide user interfaces, the Domino API server, orchestration, metadata and supporting services. The standard [architecture](/cloud/admin/architecture-and-deployment/architecture) runs the platform on a stable set of nodes for high availability, and the capabilities of the platform are principally managed through vertical scaling. This means changing the CPU and memory resources available on those platform nodes and changing the resources requested by the platform components.

**Asynchronous Domino endpoints**\
Asynchronous Domino endpoints may impose additional storage requirements for MongoDB and RabbitMQ. See [Asynchronous Domino endpoints Capacity Planning](/cloud/admin/architecture-and-deployment/sizing-the-infrastructure/size-async-domino-endpoint) for sizing information.

**Domino Compute Grid**\
These on-demand components run users' data science, engineering, and machine learning workflows. Compute workloads run on customizable collections of nodes organized into node pools. The number of these nodes can be variable and elastic, and the capabilities are principally managed through horizontal scaling. This means changing the number of nodes.

However, when there are more resources present on compute nodes, they can handle additional workloads, and therefore there are benefits to vertical scaling.

Domino uses [Kubernetes requests and limits](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) to manage the CPU and memory resources that Domino pods use. These requests and limits can be scaled to adjust resource consumption and performance. Container workloads such as databases and search systems, whose data integrity is affected by the enforcement of limits, do not have limits added to their configuration and you must not add limits to them. See [Manage Compute Resources](/cloud/admin/infrastructure-and-compute/manage-compute-resources) for sizing information.


# Asynchronous Domino endpoints capacity planning
Source: https://docs.domino.ai/cloud/admin/architecture-and-deployment/sizing-the-infrastructure/size-async-domino-endpoint

Calculate MongoDB and RabbitMQ storage for asynchronous Domino endpoints, and configure minimum available disk thresholds.

Asynchronous Domino endpoints may impose additional storage requirements on MongoDB and RabbitMQ depending on the number of API invocations and other factors.

Here are the default volume sizes for MongoDB and RabbitMQ in a new Domino deployment:

* MongoDB storage volume size: `100 GiB`

* RabbitMQ storage volume size: `25 GiB`

The default configuration can support asynchronous Domino endpoints based on the following parameters:

| Parameter | Value |
| - | - |
| Number of asynchronous Domino endpoints | 100 |
| Number of API requests per endpoint per day | 5,000 |
| MongoDB document size | 10 KiB |
| MongoDB document retention | 2 days |
| RabbitMQ request message size | 10 KiB |
| RabbitMQ response message size | 10 KiB |
| RabbitMQ request input queue retention | 2 days |
| RabbitMQ response output queue retention | 2 days |

The sections below show how to calculate MongoDB and RabbitMQ storage requirements for asynchronous Domino endpoints using the above parameters.

## MongoDB storage requirements

* Number of documents per endpoint per day

  `Number of API requests per endpoint per day`

* Additional MongoDB storage

  `Number of asynchronous Domino endpoints` x `Number of documents per endpoint per day` x `MongoDB document size` x `MongoDB document retention`

Example:

```
100 endpoints x 5,000 documents per endpoint per day x 10 KiB x 2 days =~ `10 GiB`
```

## RabbitMQ storage requirements

* Number of API responses per endpoint per day

  `Number of API requests per endpoint per day`

* RabbitMQ request input queue storage

  `Number of asynchronous Domino endpoints` x `Number of API requests per endpoint per day` x `RabbitMQ request message size` x `RabbitMQ request input queue retention`

* RabbitMQ response output queue storage

  `Number of asynchronous Domino endpoints` x `Number of API responses per endpoint per day` x `RabbitMQ response message size` x `RabbitMQ response output queue retention`

* Additional RabbitMQ storage

  `RabbitMQ request input queue storage` + `RabbitMQ response output queue storage`

Example:

```
(100 endpoints x 5,000 requests per endpoint per day x 10 KiB x 2 days) + (100 endpoints x 5,000 responses per endpoint per day x 10 KiB x 2 days) =~ `20 GiB
```

## Minimum available storage

To prevent asynchronous Domino endpoints from filling the disks, Domino monitors MongoDB and RabbitMQ disk storage metrics and stops accepting new asynchronous Domino endpoint requests when the available disk storage falls below the following configurable thresholds:

| Configuration Records Key | Default Value |
| - | - |
| `com.cerebro.domino.modelApis.async.mongoDb.minimumAvailableBytes` | 500000000 |
| `com.cerebro.domino.modelApis.async.rabbitmq.availableAsyncLimitBytes` | 1000000000 |


# Size the data plane agent
Source: https://docs.domino.ai/cloud/admin/architecture-and-deployment/sizing-the-infrastructure/size-data-plane-agent

Set Helm resource requests and limits for the data plane agent, and scale nucleus-frontend for heavy remote Job file synchronization.

For larger [Domino Nexus](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture) deployments, Domino recommends increasing resource requests and limits on certain platform pods. The data plane agent is included in this recommendation. If you are deploying a remote [data plane](/cloud/admin/platform-configuration/manage-data-planes), you should also consider the resource sizing for that agent.

## Size the Data Plane agent

You can use Helm values like these to size the data plane agent:

```yaml theme={null}
agent:
  deployment:
    resources:
      requests:
        cpu: 2000m
        memory: 2Gi
      limits:
        cpu: 4000m
        memory: 4Gi
```

## Sizing tips

If the Domino control plane is not hosted in AWS, remote data planes upload DFS artifacts via a proxy hosted by the `nucleus-frontend` deployment in the control plane.

If there are a large number of remote jobs syncing a large number of files, performance may be improved by increasing the number of replicas for this deployment and/or the resource requests and limits.


# Datasets administration
Source: https://docs.domino.ai/cloud/admin/data-administration/datasets

Administer Domino Datasets and Dataset Storage across data planes, monitor usage, set limits, delete snapshots, and add storage accounts.

When users have large quantities of data, including collections of many files and large individual files, Domino recommends that users import the data using a [Domino Dataset](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets).

Datasets are collections of Snapshots, where each Snapshot is an immutable image of a filesystem directory from the time when the Snapshot was created. These directories are stored in a network filesystem managed by Kubernetes as a shared Persistent Volume.

To view all the Datasets in your Domino deployment, go to **Admin** > **Manage resources** > **Datasets**. From here you can manage permissions, and rename and delete Datasets.

Domino natively supports Domino Datasets with the following cloud storage services:

* Amazon EFS

* Azure File Storage

* Google Cloud Filestore

The Domino File Store can also be backed with a shared Kubernetes Persistent Volume from a compatible storage class. You can provide an NFS storage service, and Domino installation utilities can deploy the [nfs-client-provisioner](https://github.com/kubernetes-retired/external-storage/tree/master/nfs-client) and configure a compatible storage class backed by the provided NFS system.

Each Snapshot of a Domino Dataset is an independent state, and its membership in a Dataset is an organizational convenience for working on, sharing, and permissioning related data. Domino supports running [scheduled Jobs](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job) that create Snapshots, so users can write or import data into a Dataset as part of an [ongoing pipeline](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets#schedule-jobs-to-update-a-dataset).

You can permanently delete Dataset Snapshots. This is a two-step process to avoid data loss. Users must mark Snapshots to be deleted, then [you must confirm the deletion, if appropriate](#delete-datasets-and-snapshots). This capability makes Datasets the right choice for storing data in Domino that has regulatory requirements for expiration.

## Create and register Dataset storage

Here are some things to keep in mind when creating Dataset Storages:

* Admins cannot create a Dataset Storage on the local data plane because there is usually only one, which is associated with the Domino shared store. Multiple Dataset Storages can only exist in the local data plane if a [multi-storage account](#set-up-multiple-storage-accounts) is set up in that data plane.

* Dataset Storages can only be edited if no Datasets are currently using them, to avoid disrupting ongoing dataset work.

* Admins can edit the name, underlying volume, and the is default field.

### Create Dataset storage

**Prerequisite**: If your Kubernetes cluster is not configured to automatically create a Persistent Volume (VC) when a Persistent Volume Claim (PVC) is used, you must set up a [Kubernetes PV and PVC](/cloud/admin/data-administration/external-data/external-data-volumes#set-up-the-kubernetes-pv-and-pvc) in the cluster first.

You’ll need to add a few things to the PVS specification:

1. Add the label **`dominodatalab.com/dataset-storage: <driver>`**:

   ```yaml theme={null}
   labels: “dominodatalab.com/dataset-storage:EFS"
   ```

2. Add the namespace used in your data plane (usually **compute**):

   ```yaml theme={null}
   namespace: YOUR-DOMINO-COMPUTE-NAMESPACE
   ```

3. Next, click the Add Dataset Storage button and enter this information:

   1. **Name**: specify a name for the Dataset Storage.

   2. **Data Plane**: select the data plane you want to use.

   3. **Volume**: choose the PVC name from the dropdown.

The volume may not be discoverable immediately. If you don’t see the volume you want to use, close the modal and reopen it after a few minutes.

### Register Dataset storage

You’ll need to register your newly created Dataset Storages before users can access it. To register a Dataset Storage:

1. Go to the Admin UI and select **Datasets**.

2. Click **Dataset Storage**.

<img alt="Register or Add Dataset Storage" />

You can only unregister a Dataset Storage if no Datasets are using it. Local Dataset Storage cannot be deleted, but remote Dataset Storages can be. Admins must delete the Persistent Volume Claim (PVC) or Persistent Volume (PV) from the cluster if needed.

Admins can view all Dataset Storages, including names, driver types (like EFS, NFS, SMB), and their associated data planes. The “Is Default” section indicates if a Dataset Storage is the default. This means that when creating a Dataset in that data plane, the default is used unless changed.

### Configurations

The configurations outlined below are set in the config map of a service on **remote data planes** and impact the datasets located there. These configurations serve as a redundancy measure to ensure that temporary files or files marked for deletion are eventually cleaned up, even if the original process that created them fails to do so.

**Note**: Making the grace period settings too short could result in files being deleted while they are still in use.

* **cleanDownloadDirsPeriod**: Temporary files generated during the download process are regularly cleared.

* **cleanDeleteDirsGracePeriod**: Frequency files that are scheduled for deletion have been cleared.

* **cleanDownloadDirsGracePeriod**: Grace period before deleting any temporary files related to the download process.

* **cleanDeleteDirsGracePeriod**: Grace period before deleting any temporary files related to the delete process.

To enable Datasets on a remote data plane, see [Configure Datasets](/cloud/admin/platform-configuration/manage-data-planes#configure-datasets).

## Associate Data Planes with Datasets

Nexus now includes the Domino Datasets feature, allowing users to create and manage Datasets from remote data planes and using the local data plane. Remote Datasets work just like local ones and support file operations such as snap-shotting, uploading, and downloading. They also have the same permissions that users expect. Users must first obtain data plane permission and the relevant dataset permissions to access remote datasets.

We are introducing the concept of Dataset Storage, which refers to the underlying storage that supports a collection of Datasets. Administrators can register, configure, and unregister these Dataset Storages, enabling users to select the Dataset Storage that supports their Datasets.

When upgrading to a new release that includes Dataset Storage, the Domino shared storage, and any other storage set up for [multi-storage accounts](#set-up-multiple-storage-accounts), are linked to the corresponding Dataset Storage.

```mermaid placement="top-right" theme={null}
%%{init: {"flowchart": {"wrappingWidth": 190}}}%%
flowchart TB
  accTitle: Datasets across the Control Plane and Data Planes
  accDescr: A customer machine sends Dataset file requests to the Control Plane ingress and to each remote Data Plane ingress. In the Control Plane, the Dataset service and Jobs read and write Dataset metadata and reach the local Data Plane's Dataset Storage. Each remote Data Plane runs its own Dataset service, which sends metadata requests to the Control Plane and serves its own Dataset Storage. In both Data Planes the Dataset Storage is mounted into executions: Jobs, Workspaces, Launchers, and Apps.
  customer["Customer machine<br/>web browser"]

  subgraph cp["Control Plane"]
    cpingress["Ingress"]
    service["Dataset service<br/>and Jobs<br/>handles local Dataset<br/>operations"]
    metadata[("Dataset<br/>metadata")]
  end

  subgraph localdp["Local Data Plane"]
    localstore["Dataset Storage<br/>PV/PVC"]
    localexec["Executions<br/>Jobs · Workspaces<br/>Launchers · Apps"]
  end

  subgraph remotedp["Remote Data Plane, one or many"]
    remoteingress["Ingress"]
    remoteservice["Remote Dataset<br/>service and Jobs<br/>at parity with the<br/>local Data Plane"]
    remotestore["Dataset Storage<br/>PV/PVC"]
    remoteexec["Executions<br/>Jobs · Workspaces<br/>Launchers · Apps"]
  end

  customer -->|Dataset file requests| cpingress
  customer -->|Dataset file requests| remoteingress
  cpingress --> service
  service <--> metadata
  service --> localstore
  localstore -->|mounted in| localexec
  remoteingress --> remoteservice
  remoteservice --> remotestore
  remotestore -->|mounted in| remoteexec
  remoteservice -->|metadata requests| service
```

### Data Planes

Local data planes, found within the compute namespace in the same cluster as the control plane, are responsible for handling compute operations, such as running a Workspace in the cluster. Remote data planes are the compute planes for other clusters and can connect to the central control plane.

Users receive visual feedback on which data plane a Dataset is located in. They can also filter by data plane when creating a dataset or mounting one for an app or launcher.

Shared Datasets are logical links that make data accessible within a project, they don’t physically move or copy data. The underlying data stays in its original data plane. Because of this, users can only access Datasets from one data plane at a time, whether through the UI or executions.

### Dataset storage

Dataset Storage refers to a method of managing storage in Kubernetes. It works with [PersistentVolumeClaim and PersistentVolume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) (PVC/PV) to provide the storage needed for Datasets. Here are some essential things to know about how dataset storage functions in Domino:

* Each dataset storage is mapped 1:1 to a corresponding PVC/PV pair.

* A particular dataset storage can be associated with anywhere from zero to many Datasets.

* A data plane can have anywhere from zero to many dataset storages.

* Dataset storage can be in the local data plane or in any remote data plane that supports datasets in Nexus.

## Monitor Dataset usage

From the Admin application, go to **Manage resources** > **Datasets**.

The Datasets page shows information for **Datasets** and **Snapshot** such as the following:

* Total storage size used by all stored Datasets/Snapshots.

* The size of all storage used by Datasets/Snapshots marked for deletion.

* A table of all Datasets/Snapshots from the history of the deployment. You can sort the table by status, size, and the name of the containing Project/Dataset.

## Set limits on Dataset usage

If you want to limit the storage consumed in Datasets, please reach out to Domino Support.

If a Dataset reaches one of these limits and a user starts an execution with a Dataset configuration that can output a new Snapshot, they receive an error message. Before additional Snapshots can be written, you must delete old snapshots or increase the limit.

You can authorize individual projects to ignore these limits.

1. Go to the project whose limits you want to ignore.

2. Click **Settings**. In the **Hardware & Environment** tab, select the **Ignore Dataset Limits** checkbox.

## Delete Datasets and snapshots

You (an administrator) can delete entire Datasets or individual Snapshots. Domino recommends the following process for Dataset and Snapshot deletion.

* The user who owns the Dataset marks it for deletion, excluding it from any new executions that start. Non-administrator users can never permanently delete a Dataset or Snapshots.

* You (the administrator) then delete the Dataset or Snapshot, if appropriate.

Delete a Dataset or Snapshot

1. Go to **Admin** > **Manage resources** > **Datasets**.

2. Click the **Datasets** tab or the **Snapshots** tab.

3. At the end of the row for the item to delete, click the three vertical dots and then click **Delete**.

   If you confirm the action, the system permanently deletes the item. If the action was initiated by mistake, you can still recover a dataset or snapshot before the delete grace period (`com.cerebro.domino.dataset.graceTimeForDeletion`) expires.

<Note>
  The delete confirmation identifies the projects using a given Dataset or Snapshot. If you delete Datasets that are used by several projects, it can be disruptive because users no longer have access to this data. Consider notifying users when you think the impact of the deletion might be significant.
</Note>

When a Dataset or a Snapshot is deleted, it is no longer available for future executions. Executions that are in progress are also affected if they attempt to read or write to the dataset that is deleted.

<Tip>
  You can **Delete all marked datasets** or **Delete all marked snapshots** and perform bulk delete confirmations. You can also sort the tables by status to easily find all Datasets of Snapshots marked for deletion.
</Tip>

## Set up multiple storage accounts

When your storage account is filling up, you can configure an additional storage location where Domino can store all future datasets and snapshots. Updates to existing datasets and snapshots are made in their original storage location, while new datasets and snapshots are stored in the secondary location. You can do this again if the secondary storage location also fills up.

<Note>
  We recommend reaching out to your Domino field team to understand whether this solution is appropriate for your scenario.
</Note>

This topic explains how to set up additional storage accounts for datasets. For the steps below, we recommend working with the Domino field team to ensure that your configuration is accurate.

### Set up new storage account & Kubernetes resources

1. Create a new storage account.

   This can be in the cloud or anywhere, as long as it can be registered as a Kubernetes Persistent Volume resource, and made accessible to Domino deployments (namely `nucleus-*` and `run-*`).

2. Create the appropriate PV/PVC pairs for the new storage account in the Domino cluster.

   Two pairs of PV/PVC should be created in this step: one in the platform and one in the compute namespace. The content of the PV/PVC varies according to your choice of storage.

3. Make the PV/PVC accessible to other resources within the cluster.

   The newly created PV/PVC must be accessible by pods in the platform and compute namespace. Depending on your choice of storage, this can mean creating appropriate Kubernetes secrets for the new storage, editing the storages network policies to authorize write-access, and so on.

4. Edit the Kubernetes `nucleus-*` deployments to mount the newly registered PV at a specific `mountPoint`.

5. Once the pods have restarted, test that the PV can be accessed correctly:

   1. Execute inside one of the `nucleus-frontend` pods.

   2. Navigate to the specified `mountPoint`.

   3. Try to write a test file.

### Enable multi storage in the Domino UI

Start this procedure only after successfully completing step 5 above. At this point, you should have a new storage account that is accessible to Domino via its own PV/PVC pairs. To instruct Domino to write every future dataset and snapshot in the new storage account, follow the below steps:

1. Go to **Admin** > **Platform settings** > **Configuration records**, and add the following key/value pair in the `common` namespace:

   * `com.cerebro.domino.datacache.pvc.names` - This is a comma-separated list of all compute-PVC names used for dataset/snapshot storage at Domino. If you only added one additional storage account, this list should contain two PVCs: the original one and the newly created one.

   * `com.cerebro.domino.datacache.pvc.originalName` - The compute-PVC name corresponding to the original Domino storage (this defaults to the first PVC in the comma-separated list of step 1).

   * `com.cerebro.domino.datacache.pvc.primaryName` - The compute-PVC name of the volume in which you would like to store every *next* dataset and snapshot.

   * `com.cerebro.domino.datacache.pvc.<COMPUTE-PVC-NAME>.mountPoint` - Configure this for each PVC name specified in step 1; it is the mount point at which the PVC is mounted in the `nucleus-*` deployments.

   <Note>
     All the PersistentVolumeClaims (PVC) specified in the configuration records settings above must correspond to the claim names in the `*-compute` namespace.
   </Note>

2. Restart the necessary services as specified at the top of the **Configuration records** page.

3. Test that all dataset-related features work properly in the new storage account:

   * Create a dataset and uploading data to it.

   * Take a snapshot.

   * Download data from a dataset/snapshot.

   * Delete a dataset/snapshot.

   * Mount datasets/snapshots to executions.

   Verify that all these actions took place in the newly configured storage account as expected.

   <Note>
     Contact [Domino Support](mailto:support@dominodatalab.com) if the dataset-related features are not working properly, as you may need to adjust your settings.
   </Note>

### Preserve changes to Kubernetes deployments throughout upgrades

To preserve the changes made to the `nucleus-*` deployments and ensure that multi-storage support continues to work properly after a Domino upgrade, you must add the following specs to the custom `agent.yaml`, inside the `nucleus` > `chart_values` section:

```yaml theme={null}
config:
  multiStorageSupportEnabled: true
  additionalStorages:
    - name: "<name of the additional volume(s) as per the modified nucleus-* deployment>"
      mountPath: "<mount path at which volume was mounted>"
      platformPvc: "<platform-PVC name corresponding to the new volume>"
      computePvc: "<compute-PVC name corresponding to the new volume>"
```

## Datasets workflow updates

Here are the general updates for the user interface that apply to all Datasets in version 6.0.0 and the cloud, following the August release.

<img alt="Datasets Interface" />

1. **New Icons**: The Datasets page now shows new icons to display information from left to right:

   * **Dataset size**: total of both read-only and read-write snapshots

   * **Data plane**: origin of that Dataset

   * **Storage amount**: used by the Dataset

   * **Directory**: location of the Dataset

2. **Description** and **Files**: there are a few updates to the functionality of these fields:

   * **Description**: Field is now in a separate white panel to make it more transparent.

   * **Files**: The dropdown to switch between snapshots has been renamed to refer to files from the latest snapshot data.

   * **Dropdown bar**: the width has been reduced to take up less space.

### Other UI updates

In addition to new icons and improvements, there is a new dropdown to select a data plane along with the corresponding Dataset Storage for the following:

* Jobs

* Scheduled Jobs

* Workspaces

You can now also use the new Data Plane dropdown to filter for hardware tiers for:

* Apps

* Launchers


# Manage External Data Volumes
Source: https://docs.domino.ai/cloud/admin/data-administration/external-data/external-data-volumes/index

An External Data Volume mounts a remote file system into Domino executions, backed by a labeled Kubernetes Persistent Volume Claim in the compute namespace.

An External Data Volume (EDV) mounts a remote file system into Domino executions, so practitioners read and write files on storage that Domino does not own. Registered EDVs appear in a table listing the name, type, description, and volume access of each one, plus a Projects column showing which Projects have added it.

<Note>
  [Remote Data Planes](/cloud/platform-capabilities/core-concepts/data-plane) are needed to use EDVs in Domino Cloud.
</Note>

## How EDVs work

Domino runs on Kubernetes, so every EDV is backed by a Kubernetes [Persistent Volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) (PV) bound to a Persistent Volume Claim (PVC). Three things make a PVC eligible:

* It carries the label key `dominodatalab.com/external-data-volume`.

* The value of that label is the storage type: `NFS`, `SMB`, `EFS`, or `Generic`.

* It exists in the Domino compute namespace.

Every properly labeled PVC becomes a candidate you can register on the Domino External Data Volumes page. Creating the PVC exposes it to Domino; registering it makes it usable in Projects.

## Set up the Kubernetes PV and PVC

<Note>
  A Kubernetes administrator must set up the Kubernetes [persistent volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) (PVs) and Persistent Volume Claims (PVCs). Adjust the Domino compute namespace and the PVC names in the examples before you use them.
</Note>

<Tabs>
  <Tab title="NFS">
    A static provisioning example for a Network File System (NFS) share. Provision the PV that refers to the NFS share by any mechanism appropriate for your environment.

    ```yaml theme={null}
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
       name: pvc-nfs
       namespace: YOUR-DOMINO-COMPUTE-NAMESPACE    #change for your deployment
       labels:
          "dominodatalab.com/external-data-volume": "NFS"
    spec:
       storageClassName: ""
       accessModes:
       - ReadWriteMany
       resources:
          requests:
             storage: 30Gi
       volumeName: pv-nfs
    ---
    apiVersion: v1
    kind: PersistentVolume
    metadata:
       name: pv-nfs
    spec:
       storageClassName: ""
       accessModes:
       - ReadWriteMany
       capacity:
          storage: 30Gi
       # Optionally explicitly specify a reference to the PVC that will be using this PV
       # Name and namespace must match the PVC created
       claimRef:
          apiVersion: v1
          kind: PersistentVolumeClaim
          name: pvc-nfs
          namespace: YOUR-DOMINO-COMPUTE-NAMESPACE   #change for your deployment
       persistentVolumeReclaimPolicy: Retain
       nfs:
          path: /mnt/export
          server: 10.0.0.26
    ```
  </Tab>

  <Tab title="EFS">
    A static provisioning example for Amazon Elastic File System (EFS). Only the configuration options that the EFS Container Storage Interface (CSI) driver supports are available. For details, see the [EFS CSI documentation](https://github.com/kubernetes-sigs/aws-efs-csi-driver#example-links).

    ```yaml theme={null}
    apiVersion: v1
    kind: PersistentVolume
    metadata:
       name: pv-efs
    spec:
       storageClassName: ""
       accessModes:
       - ReadWriteMany
       capacity:
          storage: 30Gi
       # Optionally explicitly specify a reference to the PVC that will be using this PV
       # Name and namespace must match the PVC created
       claimRef:
          apiVersion: v1
          kind: PersistentVolumeClaim
          name: pvc-efs
          namespace: YOUR-DOMINO-COMPUTE-NAMESPACE   #change for your deployment
       persistentVolumeReclaimPolicy: Retain
       csi:
          driver: efs.csi.aws.com
          volumeHandle: <EFS file system id>:/<path>    #the path within the file system is optional
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
       name: pvc-efs
       namespace: YOUR-DOMINO-COMPUTE-NAMESPACE
       labels:
          "dominodatalab.com/external-data-volume": "EFS"
    spec:
       storageClassName: ""
       accessModes:
       - ReadWriteMany
       resources:
          requests:
             storage: 30Gi
       volumeName: pv-efs
    ```
  </Tab>

  <Tab title="SMB">
    <Warning>
      The Server Message Block (SMB) driver is disabled by default. To use an EDV backed by an SMB persistent volume, add the following to your installer configuration and then run the installer.

      ```yaml theme={null}
      release_overrides:
        csi-driver-smb:
          installed: true
      ```
    </Warning>

    A static provisioning example for SMB. Only the configuration options that the SMB CSI driver supports are available. For details, see the [SMB CSI documentation](https://github.com/kubernetes-csi/csi-driver-smb/blob/master/docs/driver-parameters.md).

    ```yaml theme={null}
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
       name: pvc-smb
       namespace: YOUR-DOMINO-COMPUTE-NAMESPACE
       labels:
          "dominodatalab.com/external-data-volume": "SMB"
    spec:
       storageClassName: ""
       accessModes:
       - ReadWriteMany
       resources:
          requests:
             storage: 30Gi
    ---
    apiVersion: v1
    kind: PersistentVolume
    metadata:
       name: pv-smb
    spec:
       storageClassName: ""
       accessModes:
       - ReadWriteMany
       capacity:
          storage: 100Gi
       claimRef:
          apiVersion: v1
          kind: PersistentVolumeClaim
          name: pvc-smb
          namespace: YOUR-DOMINO-COMPUTE-NAMESPACE    #change for your deployment
       persistentVolumeReclaimPolicy: Retain
       csi:
          driver: smb.csi.k8s.io
          nodeStageSecretRef:
             name: SMB-SECRET-NAME
             namespace: NAMESPACE-OF-SMB-SECRET
          volumeAttributes:
             source: //10.0.0.11/Share        #SMB server address and path
          volumeHandle: volume-handle-unique  # must be unique id for the cluster
       mountOptions:
       - dir_mode=0777
       - file_mode=0777
       - vers=3.0
    ```
  </Tab>

  <Tab title="Generic">
    The `Generic` type attaches storage that you deploy and maintain in your own environment. Configure the PV according to the documentation for the system you are using; the example below shows the labeling and namespace that Domino requires.

    ```yaml theme={null}
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
       name: generic-EDV
       namespace: YOUR-DOMINO-COMPUTE-NAMESPACE    #change for your deployment
       labels:
          "dominodatalab.com/external-data-volume": "Generic"
    spec:
       storageClassName: ""
       accessModes:
       - ReadWriteMany
       resources:
          requests:
             storage: 30Gi
       volumeName: generic-EDV
    ---
    apiVersion: v1
    kind: PersistentVolume
    metadata:
       name: generic-EDV
    spec:
       storageClassName: ""
       accessModes:
       - ReadWriteMany
       capacity:
          storage: 30Gi
       # Optionally explicitly specify a reference to the generic EDV that will be using this generic EDV
       # Name and namespace must match the generic EDV created
       claimRef:
          apiVersion: v1
          kind: PersistentVolumeClaim
          name: generic-EDV
          namespace: YOUR-DOMINO-COMPUTE-NAMESPACE   #change for your deployment
       persistentVolumeReclaimPolicy: Retain
       nfs:
          path: /mnt/export
          server: 10.0.0.26
    ```
  </Tab>
</Tabs>

## Register External Data Volumes

In a hybrid deployment, one EDV can serve a shared file system reachable from several [Data Planes](/cloud/admin/platform-configuration/manage-data-planes). Domino matches EDVs to PVCs by the name of both the PVC and the underlying PV, and detects the additional Data Planes automatically during registration. If a PVC appears in a Data Plane you do not intend to use with the EDV, rename that PVC or PV so it no longer matches.

<Steps>
  <Step title="Open the registration wizard">
    From the Admin application, go to **Manage resources** > **External Data Volumes**, then click **Register External Volume**.
  </Step>

  <Step title="Select the storage">
    On the **Volume** step, select the storage type: NFS, Windows Share (SMB), EFS, or Generic. From **Available Volumes**, select the backing Kubernetes PVC. In a hybrid deployment, Domino then lists the [Data Planes](/cloud/admin/platform-configuration/manage-data-planes) where the EDV can be used.

    <Frame>
      <img alt="Register an External Data Volume wizard showing the storage type and the Available Volumes list." />
    </Frame>

    Click **Next**.
  </Step>

  <Step title="Configure the EDV">
    On the **Configuration** step:

    | Field | What to enter |
    | :- | :- |
    | **Name** | Defaults to the PVC name. Name the EDV so users recognize it from its use case or your organization's naming convention. |
    | **Relative Mount Path** | Defaults to the PVC name. Must be unique across all registered EDVs. A few words are reserved. |
    | **Mount as read-only** | Select to make the EDV read-only, or clear it to make the EDV read-write. |
    | **Description** | Free text shown to users browsing EDVs. |

    <Note>
      Read-write access is enforced at the Domino layer only. More restrictive access controls at the Kubernetes or NFS layer overrule this setting: if the PVC access mode is read-only, the underlying read-only permission wins regardless of what you select here.
    </Note>

    <Frame>
      <img alt="Configure an External Data Volume step showing name, mount path, and description fields." />
    </Frame>

    Click **Next**.
  </Step>

  <Step title="Set access and register">
    On the **Access** step, select the **Volume Access**: **Everyone** to allow all logged-in users, or **Specific users or organizations** to limit access. Click **Register**.

    <Frame>
      <img alt="Access step of the External Data Volume registration wizard." />
    </Frame>

    <Note>
      Regardless of this setting, you (CloudAdmin) can always access any External Data Volume.
    </Note>
  </Step>
</Steps>

## Associate Data Planes with an EDV

In a hybrid deployment, each EDV is associated with specific [Data Planes](/cloud/admin/platform-configuration/manage-data-planes). For security reasons, that set never changes on its own. Change it by modifying the PVCs in the relevant Data Plane, then editing the EDV.

### Add a Data Plane to an EDV

1. Create the PV and PVC in the new Data Plane, with names matching the PVC already used for the EDV.

2. Click the three-dot menu at the right of the EDV row, then click **Edit**.

   The new Data Plane appears at the bottom of the modal, highlighted in green to show that it is being added. If you do not see it, confirm the name of the PV and the name and label of the PVC.

3. Click **Update**.

### Remove a Data Plane from an EDV

1. Remove the PVC and PV from that Data Plane.

2. Click the three-dot menu at the right of the EDV row, then click **Edit**.

   The Data Plane appears highlighted in red to show that it is marked for removal.

3. Click **Update**.

## View, edit, and unregister EDVs

All three start from **Manage resources** > **External Data Volumes** in the Admin application.

| To | Do this |
| :- | :- |
| See the details of an EDV | Click the **Name** of the EDV. In a hybrid deployment, the list also shows the Data Planes associated with each EDV. |
| Change an EDV | Click the three-dot menu at the end of the EDV row, then click **Edit**. |
| Unregister an EDV | Click the three-dot menu at the end of the EDV row, click **Unregister**, then click **Yes** to confirm. |

## Control EDV visibility

Collaborators on one Project do not necessarily share access to the same EDVs. A user without volume access can never mount that EDV in an execution, but you can choose how much they see of it, trading discoverability against secrecy.

| Censorship level | What a user without access sees | Choose it for |
| :- | :- | :- |
| Full censorship | Only that some inaccessible EDV exists. Neither the count nor any metadata is revealed. | The highest level of security. |
| Inactive censorship (default) | The inaccessible EDVs themselves, including metadata such as name and description. | Discoverability, so users can escalate to you and request access. |

To set the level of censorship, please reach out to Domino Support.

See [Mount an external volume](/cloud/platform-capabilities/core-concepts/data/external-volumes/use-a-mounted-volume) for what your users see.

## Related

* [External Data Volumes](/cloud/platform-capabilities/core-concepts/data/external-volumes) covers what practitioners do with an EDV once you register it.

* [Data Sources](/cloud/admin/data-administration/external-data) connect to external databases and object stores by query rather than by mount.

* [Domino Volumes for NetApp ONTAP](/cloud/admin/data-administration/netapp-volumes) are Domino-managed NetApp-backed storage, distinct from EDVs.


# Manage Data Sources
Source: https://docs.domino.ai/cloud/admin/data-administration/external-data/index

A Data Source is a reusable connection to an external database, data service, or object store that an admin configures once and practitioners query from any execution.

A Data Source is a reusable connection to an external database, data service, or object store. You configure the connection once, and practitioners query it from a Workspace, Job, or app without handling hostnames, ports, or drivers, and write results back to a Domino Dataset.

Admins can expose a Data Source to everyone in the deployment. When a practitioner [creates a Data Source](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources), they must name the users or organizations that can access it.

## Create a Data Source

<Steps>
  <Step title="Open the Data Source wizard">
    Go to **Manage Resources** > **Data Sources**, then click **Connect to External Data**.
  </Step>

  <Step title="Configure the connection">
    Select your Data Source, then enter the configuration details for accessing it. The details vary depending on the Data Source type.
  </Step>

  <Step title="Select the credential type">
    | Credential type | Who supplies the credentials | Use it when |
    | :- | :- | :- |
    | **Individual** | Each user provides their own credentials before using the Data Source. | Access must be attributable to a person, or the underlying system enforces per-user permissions. |
    | **Service Account** | You provide one set of static credentials that Domino applies on behalf of every user with permission to the Data Source. | The team shares one identity against the external system. |

    End users cannot access or extract Service Account credentials, nor can they manage user-level permissions on such Data Sources. Domino stores Data Source credentials in the Domino secret store, which is backed by [HashiCorp Vault](https://www.vaultproject.io/).

    <Note>
      OAuth authentication is not supported for Service Accounts. The Data Source itself does not distinguish end users, because the same credentials authenticate every query.
    </Note>
  </Step>

  <Step title="Enter and test the credentials">
    Enter your credentials for Data Source authentication, then click **Test Credentials** to verify that authentication works, or click **Skip for Now** to continue.
  </Step>

  <Step title="Set access and finish">
    Select whether **Everyone** can use this Data Source or just **Specific users or organizations**, then click **Finish Setup**.
  </Step>
</Steps>

## Authentication methods

Data Sources support multiple authentication methods to meet different security requirements. Domino manages authentication configuration for cloud deployments, so you can focus on connecting to your data.

Authentication methods vary by Data Source, including support for credentials, OAuth, key pair, and certificate authentication.

Contact Domino Support to enable additional authentication methods for your Data Sources or remove existing ones.

## IAM authentication for AWS Data Sources

You can enable users to create Data Sources that authenticate with AWS Identity and Access Management (IAM) for these Data Source types:

* [MySQL](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-mysql)

* [PostgreSQL](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-postgres)

* [Redshift](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-redshift)

* [S3](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-generic-s3)

<Note>
  IAM-authenticated connections can be used only to execute Jobs and Workspaces. Other execution types, such as scheduled Jobs and Domino endpoints, require basic authentication.
</Note>

To set up IAM authentication, please reach out to Domino Support.

## OAuth for Snowflake

OAuth-backed [Snowflake Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-snowflake) authenticate and query Snowflake with a JSON Web Token (JWT). Once it is set up, users can select **OAuth** when they create a Snowflake Data Source.

When an OAuth-configured Snowflake Data Source is queried in a Domino execution, the `dominodatalab-data` library uses the Domino JWT token to authenticate against the underlying Snowflake database. No extra user action is needed.

Setup has two halves. First, an External OAuth Security Integration in your Snowflake account dictates what values Snowflake expects the token to carry. Second, Keycloak mappers propagate those values into the Domino JWT access token.

<Note>
  OAuth is not supported for Service Accounts. OAuth-authenticated connections can be used for any execution type *except* Domino endpoints.
</Note>

<Note>
  Because authentication depends on the Domino JWT token propagating the correct values in the context of an execution, you cannot verify successful authentication when you create a Data Source. The green checkmark that appears when you create a Data Source does not guarantee that authentication succeeds. To confirm successful value propagation in a Domino JWT token, use the `dominodatalab-data` library to query data from an OAuth Data Source in a Domino execution, such as a Workspace.
</Note>

### Configure the Snowflake security integration

To set up an External OAuth Security Integration in Snowflake, see the [Snowflake external OAuth documentation](https://docs.snowflake.com/en/user-guide/oauth-ext-custom.html#configure-custom-clients-for-external-oauth). Get familiar with the Snowflake terminology and the values that Snowflake needs in the token; correct spelling of the values is critical.

The following is a sample command to create a base security integration that allows all account roles to authenticate. Use it as a starting point for a configuration that fits your security needs.

```sql theme={null}
create security integration <external_oauth_name>
    type = external_oauth
    enabled = true
    external_oauth_type = custom
    external_oauth_issuer = 'https://<domino-domain>/auth/realms/DominoRealm'
    external_oauth_rsa_public_key = <keycloak-realm-settings-public-key>
    external_oauth_audience_list=('account')
    external_oauth_scope_mapping_attribute = 'scp'
    external_oauth_token_user_mapping_claim='preferred_username_snowflake'
    external_oauth_snowflake_user_mapping_attribute='login_name';

alter security integration <external_oauth_name> set EXTERNAL_OAUTH_ANY_ROLE_MODE = ENABLE;
```

Find the value for these keys:

* `external_oauth_issuer` in Keycloak's **Realm Settings** > **OpenID Endpoint Configuration** > **Issuer**

* `external_oauth_rsa_public_key` in Keycloak's **Realm Settings** > **Keys** > **Public Key from RS256**

The configured security integration (`desc security integration <external_oauth_name>`) should look similar to the following:

<Frame>
  <img alt="Snowflake security integration description output showing the configured external OAuth keys." />
</Frame>

### Configure Keycloak

Keycloak passes the values that the Snowflake security integration expects by way of mappers in the `domino-play` client. Each mapper is set to **add to access token**, which creates a key-value entry in the Domino JWT token keyed on **Token Claim Name**.

The value depends on the mapper type. A hardcoded-claim mapper takes a value that propagates equally to all users. A user-attribute mapper takes a **User Attribute** name, reads that attribute from each Keycloak user, and propagates its value, so you can vary the token per user.

<Steps>
  <Step title="Open the Keycloak client scopes">
    Go to the Keycloak admin console at `https://<domino-domain>/auth/`, then click **Domino Realm** > **Clients**.

    <Frame>
      <img alt="Keycloak client dashboard listing the realm clients." />
    </Frame>

    From the lookup list, click **domino-play**, then click **Client scopes** > **domino-play-dedicated**.

    <Frame>
      <img alt="Keycloak client scopes tab for the domino-play client." />
    </Frame>

    <Frame>
      <img alt="Keycloak Mappers tab for the domino-play-dedicated client scope." />
    </Frame>
  </Step>

  <Step title="Add the mappers">
    Click **Mappers** > **Add mapper** > **By configuration**, choose a mapper from the listed mappings, and configure it.

    <Frame>
      <img alt="Keycloak Add mapper by configuration dialog." />
    </Frame>

    <Frame>
      <img alt="Keycloak mapper configuration form." />
    </Frame>

    The definitions in [Keycloak mapper reference](#keycloak-mapper-reference) match the base Snowflake security integration above. For the `scp` claim, use the prepopulated Scope Claim Aggregator mapper. You can write your mappers differently, as long as they conform to the requirements of the Snowflake security integration.

    Keycloak propagates the values the mappers produce into the JWT token that authenticates against the External OAuth Security Integration you created in Snowflake.
  </Step>

  <Step title="Add the user attributes">
    If you added a mapper that requires a user attribute, add that attribute at **Users** > **View All Users** > `<user ID>` > **Attributes**. You can automate this to create further roles and mappers.

    <Frame>
      <img alt="Keycloak user attributes tab showing a custom attribute." />
    </Frame>
  </Step>

  <Step title="Enable OAuth for Snowflake in Domino">
    Go to **Platform settings** > **Configuration records**, click **Advanced settings**, then click **Add Record**. Enter the `com.cerebro.domino.datasource.SnowflakeConfig.enabledAuthTypes` key with the comma-separated values `OAuth` and `KeyPair`. Order matters: the first value becomes the default authentication method for all users. Click **Create**.
  </Step>
</Steps>

#### Keycloak mapper reference

<AccordionGroup>
  <Accordion title="SCP">
    | Field | Value |
    | :- | :- |
    | Token Claim Name | `scp` |
    | User Attribute | The name of the user attribute containing the Snowflake scope name. |
    | Static Values (comma-separated) | `all` |
    | Add to access token | `ON` |

    <Frame>
      <img alt="Keycloak mapper configuration for the scp claim." />
    </Frame>
  </Accordion>

  <Accordion title="Issuer">
    | Field | Value |
    | :- | :- |
    | Token Claim Name | `external_oauth_issuer` |
    | Claim JSON Type | `String` |
    | Add to access token | `ON` |
    | Claim value | The name of your OpenID Endpoint Configuration issuer, from **Realm Settings** > **OpenID Endpoint Configuration** > **Issuer**. |

    <Frame>
      <img alt="Keycloak mapper configuration for the issuer claim." />
    </Frame>
  </Accordion>

  <Accordion title="Audience">
    | Field | Value |
    | :- | :- |
    | Token Claim Name | `external_oauth_audience_list` |
    | Claim JSON Type | `String` |
    | Add to access token | `ON` |
    | Multivalued | `ON` |
    | Claim value | The list of audiences the security integration supports. |

    <Frame>
      <img alt="Keycloak mapper configuration for the audience claim." />
    </Frame>
  </Accordion>

  <Accordion title="User mapping">
    | Field | Value |
    | :- | :- |
    | Token Claim Name | `external_oauth_user_mapping_claim` |
    | Claim JSON Type | `String` |
    | Add to access token | `ON` |
    | Claim value | The mapping between the Keycloak user and the Snowflake user. |

    <Frame>
      <img alt="Keycloak mapper configuration for the user mapping claim." />
    </Frame>
  </Accordion>

  <Accordion title="Preferred username">
    | Field | Value |
    | :- | :- |
    | Token Claim Name | `preferred_username_snowflake` |
    | Claim JSON Type | `String` |
    | Add to access token | `ON` |
    | Claim value | The Snowflake username of the authenticating user. |

    This mapper pairs with the security integration above, which sets `external_oauth_token_user_mapping_claim='preferred_username_snowflake'` and so instructs Snowflake to read the value of the `preferred_username_snowflake` key.

    <Frame>
      <img alt="Keycloak mapper configuration for the preferred username claim." />
    </Frame>
  </Accordion>

  <Accordion title="RSA public key value">
    | Field | Value |
    | :- | :- |
    | Token Claim Name | `external_oauth_rsa_public_key_value` |
    | Claim JSON Type | `String` |
    | Add to access token | `ON` |
    | Claim value | The name of your OpenID Endpoint Configuration issuer, from **Realm Settings** > **Keys** > **Public Key from RS256**. |

    <Frame>
      <img alt="Keycloak mapper configuration for the RSA public key claim." />
    </Frame>
  </Accordion>

  <Accordion title="Snowflake user mapping attribute">
    | Field | Value |
    | :- | :- |
    | Token Claim Name | `external_oauth_snowflake_user_mapping_attribute` |
    | Claim JSON Type | `String` |
    | Add to access token | `ON` |
    | Claim value | The attribute that maps the access token to a Snowflake user record. |

    <Frame>
      <img alt="Keycloak mapper configuration for the Snowflake user mapping attribute claim." />
    </Frame>
  </Accordion>
</AccordionGroup>

### Test the OAuth configuration

The user attributes `SCP`, `external_oauth_audience_list`, and `preferred_username_snowflake` must be present in the Domino user JWT token. There are two ways to inject them:

* **Locally in Keycloak**, for quick testing. Add the user attributes manually under the Keycloak user.

* **Through SAML attributes**, for enterprise-level implementations. Passing the user attributes from the identity provider as SAML attributes is the recommended approach. When SAML attributes include the user attributes, additional mappers extract them and make them available to the `domino-play` client.

Configure three identity-provider user attribute mappers:

1. Go to **Domino realm** > **Identity providers** > **Identity provider** > **Mappers**.

2. For **Mapper type**, select `Attribute importer`.

3. In **Attribute Name** and **User Attribute Name**, add the user attributes to pass to the `domino-play` client, for example `SCP`, `external_oauth_audience_list`, and `preferred_username_snowflake`.

<Frame>
  <img alt="Keycloak identity provider mapper configured as an attribute importer." />
</Frame>

To confirm that OAuth works with your Snowflake Data Source, register a Snowflake Data Source, select `OAuth` as the authentication type, open an execution, and query data from the registered Data Source.

<Tip>
  To get the token itself, make a GET request to the [Domino API proxy](/cloud/reference/api/domino-api-authentication#use-the-api-proxy-for-domino-api-authentication) by running `curl $DOMINO_API_PROXY/access-token` in an execution, such as a Workspace terminal or a Job script. Paste the token into [jwt.io](https://jwt.io/) to see which values Keycloak propagated into the Domino JWT token.
</Tip>

## Starburst-powered Data Sources

Starburst-powered Data Sources cover MongoDB, Palantir Foundry, tabular S3 with AWS Glue, Starburst JDBC, and Teradata. Only an admin can create them, because each one needs a restart of the Starburst cluster in Domino.

For additional support in working with Starburst, contact your Domino representative.

### Enable Starburst

To enable Starburst-powered Data Sources in your Domino instance, add the following block to `domino.yaml` and `agent.yaml`.

```yaml theme={null}
release_overrides:
  starburst-enterprise:
    installed: true
```

### Create a Starburst-powered Data Source

Every type starts at **Manage Resources** > **Data Sources** > **Connect to External Data** and ends with a cluster restart. Expand the type you are configuring for the fields it asks for.

<AccordionGroup>
  <Accordion title="MongoDB">
    Create a Data Source for [MongoDB](https://www.mongodb.com/) that your Domino users can access.

    Prerequisites:

    * Starburst is enabled in the deployment. See [Enable Starburst](#enable-starburst).

    * Network connectivity between MongoDB and your Domino deployment.

    * Domino has enabled this feature. If you do not see this Data Source type in your deployment, contact your Customer Success Manager.

    To create the Data Source:

    1. Go to **Manage Resources** > **Data Sources**.

    2. Click **Connect to External Data**.

    3. In the New Data Source window, select **MongoDB**.

    4. Enter your MongoDB hostname or IP address.

    5. Optional: Enter the **Port**.

    6. Enter a **Name** and **Description** for this Data Source.

    7. Click **Next** twice.

    8. Enter your MongoDB credentials.

    9. Click **Next**.

    10. Select whether **Everyone** can access this Data Source or just **Specific users or organizations**.

    11. Click **Finish Setup**. Domino alerts you to restart the cluster.

    12. Click **Click here to trigger a restart**, then click **Restart Cluster** to confirm. The restart takes approximately 5 to 10 minutes. When it completes, Domino users can use the MongoDB Data Source.
  </Accordion>

  <Accordion title="Palantir Foundry">
    Create a Data Source for [Palantir Foundry](https://www.palantir.com/platforms/foundry/) that your Domino users can access.

    Prerequisites:

    * Starburst is enabled in the deployment. See [Enable Starburst](#enable-starburst).

    * Network connectivity between Palantir and your Domino deployment.

    * Domino has enabled this feature. If you do not see this Data Source type in your deployment, contact your Customer Success Manager.

    To create the Data Source:

    1. Go to **Manage Resources** > **Data Sources**.

    2. Click **Connect to External Data**.

    3. In the New Data Source window, select **Palantir**.

    4. Enter a **Name**, **Description**, and the Palantir hostname or IP address. Optionally, enter the **Port**.

    5. Select **Individual** or **Service Account** as the credential type.

    6. Select **Client Credentials** or **OAuth Token** for authenticating to Palantir.

       <Note>
         Individual credentials can only use **OAuth Tokens** to authenticate.
       </Note>

    7. Enter your Palantir credentials.

    8. Select whether **Everyone** can access this Data Source or just **Specific users or organizations**.

    9. Domino alerts you to restart the cluster. Click **Click here to trigger a restart** > **Restart Cluster**. The restart takes approximately 5 to 10 minutes. When it completes, Domino users can [use the Palantir Data Source](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-palantir).

    **Get Palantir client credentials.** To connect with a Service Account, retrieve your Palantir client ID and client secret:

    1. Go to your Palantir instance.

    2. Go to **Control Panel** > **Third-party applications**. An existing Domino integration appears here.

    3. Next to the Domino integration, select **Actions** > **Manage application** to find your client ID.

    If no Domino integration exists, [register Domino as a third-party application](https://www.palantir.com/docs/foundry/platform-security-third-party/register-3pa/index.html). Choose **Confidential client** as the client type and save the client secret, because Palantir does not show it again. If the integration already exists and you cannot see the client secret, contact your Palantir administrator.

    **Generate a Palantir OAuth token.** To connect with an OAuth token, generate a new [Palantir token](https://www.palantir.com/docs/foundry/platform-security-third-party/user-generated-tokens/#generation).
  </Accordion>

  <Accordion title="Tabular S3 with AWS Glue">
    Only Service Accounts are supported for [tabular S3 with AWS Glue](https://docs.aws.amazon.com/athena/latest/ug/data-sources-glue.html).

    Prerequisites:

    * Starburst is enabled in the deployment. See [Enable Starburst](#enable-starburst).

    * Network connectivity between S3 and your Domino deployment.

    * Domino has enabled this feature. If you do not see this Data Source type in your deployment, contact your Customer Success Manager.

    To create the Data Source:

    1. Go to **Manage Resources** > **Data Sources**.

    2. Click **Connect to External Data**.

    3. In the New Data Source window, select **Tabular S3 with AWS Glue**.

    4. Enter your S3 account ID, database, and region.

    5. Enter a name for this Data Source.

    6. Click **Next**. In the Credential Type screen, **Service Account** is preselected. Individual accounts are not supported for this Data Source type.

    7. Click **Next**.

    8. Enter your S3 service account access key ID and secret access key.

    9. Click **Next**.

    10. Select whether **Everyone** can access this Data Source or just **Specific users or organizations**.

    11. Click **Finish Setup**. Domino alerts you to restart the cluster.

    12. Click **Click here to trigger a restart**, then click **Restart Cluster** to confirm. The restart takes approximately 5 to 10 minutes.
  </Accordion>

  <Accordion title="Starburst JDBC">
    Starburst JDBC extends [Domino Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors) to any Starburst-JDBC data entity. IBM DB2, IBM Netezza, and SAP HANA are enabled by default.

    Prerequisites:

    * Starburst is enabled in the deployment. See [Enable Starburst](#enable-starburst).

    * Network connectivity between the data platform and your Domino deployment.

    To create the Data Source:

    1. Go to **Manage Resources** > **Data Sources**.

    2. Click **Connect to External Data**.

    3. Select the data entity you want to connect to under **Powered by Starburst JDBC**.

    4. Enter the connection parameters.

    5. Enter the **Data Source Name** and, optionally, a **Description** that explains the purpose of the Data Source to others.

    6. Select the credential type.

    7. Optionally, enter your credentials. You can enter these later.

    8. Click **Next**.

    9. Select whether **Everyone** can access this Data Source or just **Specific users or organizations**.

    10. Click **Finish Setup**. Domino alerts you to restart the cluster and provides a link to trigger it.

    To connect to a data entity that is not enabled by default, see [Add Starburst JDBC data entities](#add-starburst-jdbc-data-entities).
  </Accordion>

  <Accordion title="Teradata">
    Create a Data Source for [Teradata](https://www.teradata.com/) that your Domino users can access.

    Prerequisites:

    * Starburst is enabled in the deployment. See [Enable Starburst](#enable-starburst).

    * Network connectivity between Teradata and your Domino deployment.

    * Domino has enabled this feature. If you do not see this Data Source type in your deployment, contact your Customer Success Manager.

    To create the Data Source:

    1. Go to **Manage Resources** > **Data Sources**.

    2. Click **Connect to External Data**.

    3. In the New Data Source window, select **Teradata**.

    4. Enter your Teradata hostname or IP address.

    5. Enter a name for this Data Source.

    6. Click **Next**.

    7. Select the credential type.

    8. Optionally, enter your Teradata credentials. You can enter these later.

    9. Click **Next**.

    10. Select whether **Everyone** can access this Data Source or just **Specific users or organizations**.

    11. Click **Finish Setup**. Domino alerts you to restart the cluster.

    12. Click **Click here to trigger a restart**, then click **Restart Cluster** to confirm. The restart takes approximately 5 to 10 minutes.

    <Note>
      Users must specify the database name in their queries, as in this example where `td_demo` is the database name and `yellow_cab` is the table name:

      ```
      example res = ds.query("select * from td_demo.yellow_cab")
      ```
    </Note>
  </Accordion>
</AccordionGroup>

### Add Starburst JDBC data entities

Not every JDBC-powered data entity is available by default. You can enable ClickHouse, Druid, Greenplum, MariaDB, Ignite, SingleStore (MemSQL), Synapse, Vertica, and the generic JDBC capability that Starburst provides.

<Note>
  You may need to contact your Customer Success engineer for assistance to add some additional JDBC data entities.
</Note>

1. Go to **Admin** > **Platform settings** > **Configuration records**.

2. Add or edit the `com.cerebro.domino.datasource.starburstJdbc.enabledConnectors` config option.

3. Add a comma-separated list of data-entity names as the value:

   ```console theme={null}
   clickhouse,druid,db2,synapse,greenplum,ignite,mariadb,singlestore,vertica,generic_jdbc,sap_hana,netezza
   ```

4. Restart the Domino services to apply the new setting, following the prompt on the admin page.

The newly added data entity then appears in the drop-down list.

### Restart the Starburst cluster

Restart the Starburst cluster in Domino after actions including, but not limited to:

* Creating a Data Source

* Updating a Data Source configuration

* Updating credentials

Domino notifies you to restart your Starburst cluster and provides a link to trigger the restart. The restart takes approximately 5 to 10 minutes and disrupts executions that interact with Starburst-powered Data Sources, so plan it for a window when none are running.

<Frame>
  <img alt="Domino prompt to restart the Starburst cluster after a Data Source change." />
</Frame>

### Starburst limitations

* Your Domino representative must enable this feature. If you do not see these Data Source types in your deployment, contact your Customer Success engineer.

* In a hybrid deployment, Starburst-powered Data Sources are only available from the local Data Plane.

## Data Plane accessibility

[Data Planes](/cloud/platform-capabilities/core-concepts/data-plane) in a Domino Nexus hybrid deployment have caveats around which of them can reach a Data Source.

| Data Source type | Remote Data Planes | Local Data Plane |
| :- | :- | :- |
| Domino native Data Sources using individual credentials | Automatically accessible and unconfigurable. | Automatically accessible and unconfigurable. |
| Domino native Data Sources using Service Account credentials | Accessible, with the ability to disable access from any or all remote Data Planes. | Accessible, with the ability to disable access. |
| Starburst-powered Data Sources | Unsupported. | Supported. |

You can disable adding a Data Source in remote Workspaces. For more information, please reach out to Domino Support.

## Related

* [External Data Volumes](/cloud/admin/data-administration/external-data/external-data-volumes) mount remote file systems into executions instead of querying them.

* [Data Source connectors](/cloud/platform-capabilities/core-concepts/data/data-source-connectors) covers what practitioners do with a Data Source once you create it.

* [Monitor Data Source logs](/cloud/admin/operations/audit-logs/monitor-data-source-logs) audits Data Source activity.

* [Domino Datasets](/cloud/platform-capabilities/core-concepts/data/datasets) are where practitioners land data pulled from a Data Source.


# Manage data
Source: https://docs.domino.ai/cloud/admin/data-administration/index

Choose and administer NetApp Volumes, Data Sources, External Data Volumes, Domino Datasets, and Project files for execution data.

Data moves in and out of Domino executions in the following ways:

[Domino Volumes for NetApp ONTAP](/cloud/admin/data-administration/netapp-volumes)\
Domino Volumes for NetApp ONTAP (NetApp Volumes) let Domino practitioners share data more easily across projects by storing files on external NetApp-backed storage. They enhance performance, eliminate duplication, and provide snapshot-based recovery.

[Data Sources](/cloud/admin/data-administration/external-data)\
User code that runs in Domino can use Domino Data Sources or third-party drivers to interact with external databases, APIs, and file systems to which the Domino-hosting cluster can connect. Users can read and write from these external systems, and they can import data into Domino from such systems by saving files to their project [writing files to an output Dataset](/cloud/platform-capabilities/core-concepts/data/datasets/local-data).

[External Data Volumes](/cloud/admin/data-administration/external-data/external-data-volumes)\
External Data Volumes (EDVs) mount remote file systems into executions, so users read and write files on storage that Domino does not own.

[Domino Datasets](/cloud/admin/data-administration/datasets)\
You can configure Domino executions to mount [Domino datasets](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets) for input and output. Datasets are network volumes mounted in the execution environment. Mounting an input Dataset lets executions start quickly. It also gives them access to large quantities of data because the data is not transferred to the local execution volume until user code performs read operations from the mounted volume. Any data written to an output Dataset is saved by Domino as a new snapshot.

[Domino Project file storage](/cloud/admin/data-administration/project-file-storage)\
Every execution occurs in a [Project](/cloud/platform-capabilities/core-concepts/projects). Based on the [Domino File System](/cloud/platform-capabilities/core-concepts/projects/dfs-projects#the-working-directory), the files for the active revision of the project are automatically loaded into the local execution volume for a Job or Workspace. These files are retrieved from the [Domino File Store](/cloud/admin/data-administration/project-file-storage), and any changes to these files are written back to the Domino File Store as a new revision of the project’s files.

The following diagram shows the series of operations that happens when a user starts an execution (Job or Workspace) in Domino, and illustrates when and how various data systems can be used.

```mermaid placement="top-right" theme={null}
%%{init: {"flowchart": {"wrappingWidth": 300}}}%%
flowchart TB
  accTitle: The Domino execution data flow
  accDescr: A browser or API launches a Job or Workspace in Domino. Domino builds an execution pod specification holding the user environment, the executor utility, and the HTTP(S) interface, pulled from the Docker registry. Kubernetes schedules the pod onto the target node pool. The running pod syncs project files with the Domino File Store, mounts Domino Datasets, and reads and writes external data.
  browser["Browser"]
  apiclient["API"]
  domino["Domino"]
  registry[("Docker registry")]
  pod["Execution pod specification<br/>user environment<br/>executor utility<br/>HTTP(S) interface"]
  k8s["Kubernetes"]

  subgraph pool["Target node pool, labeled dominodatalab.com/node-pool: default"]
    podx["Execution pod X"]
    pody["Other execution pods"]
  end

  filestore[("Domino File Store")]
  datasets[("Domino Datasets")]
  extdata[("External data")]

  browser -->|1 launch| domino
  apiclient -->|1 launch| domino
  domino -->|2 pod specification| pod
  registry -->|2 container images| pod
  pod -->|3 schedule| k8s
  k8s --> podx
  k8s --> pody
  podx <-->|4 sync| filestore
  podx <-->|4 mount| datasets
  podx <-->|5 read and write| extdata
```

The numbers on the diagram's arrows match these steps:

<Steps>
  <Step title="A client launches the execution">
    A client launches a Job or Workspace from the Domino application. The operation is scoped to a specific project, environment, and hardware tier.
  </Step>

  <Step title="Domino builds the pod specification">
    Domino constructs a specification for an execution pod from the user's input. The pod holds containers for the user environment, the executor utility, and the HTTP(S) interface, all pulled from the Docker registry.
  </Step>

  <Step title="Kubernetes schedules the pod">
    Domino brokers assignment of the execution pod to Kubernetes with the node label and resource requests the user chose. An ephemeral execution volume is attached to the target node and mounted in the pod.
  </Step>

  <Step title="The executor utility syncs project files and Datasets">
    The executor utility pulls project files from the Domino File Store into the execution volume mounted in the user environment. Datasets are attached as a network file system. When the Job or Workspace syncs, Domino writes changes back to these stores.
  </Step>

  <Step title="User code reaches external data">
    User code that runs inside the Workspace or Job can read and write external databases, APIs, and file systems.
  </Step>
</Steps>

For quick tips to troubleshoot issues with data in Domino, see [Troubleshoot data issues](/cloud/admin/data-administration/troubleshoot-data).


# Manage NetApp Volumes
Source: https://docs.domino.ai/cloud/admin/data-administration/netapp-volumes/index

Domino Volumes for NetApp ONTAP store Project data on external NetApp-backed storage, provisioned through Trident and grouped into ONTAP filesystems.

Domino Volumes for NetApp ONTAP (NetApp Volumes) let practitioners share data across Projects by storing files on external NetApp-backed storage. They improve I/O performance, remove duplication, and support snapshot-based recovery, replacing scattered, Project-specific data solutions with one centralized system.

<Frame>
  <img alt="Domino Volumes for NetApp ONTAP architecture linking Domino Projects to an ONTAP filesystem." />
</Frame>

## How ONTAP filesystems work in Domino

NetApp Volumes live inside ONTAP filesystems. A [NetApp ONTAP filesystem](https://docs.netapp.com/us-en/ontap/concepts/snapshot-copies-concept.html) defines where NetApp Volumes live by linking a Data Plane and a storage class to a named storage container. That binding gives you:

* Clear visibility into where each NetApp Volume is stored.

* Support for multiple accounts and Data Planes, which matters because NetApp Volume counts are capped per account.

* A deletion flow that moves a NetApp Volume to cold storage instead of destroying it, so retention policies are enforceable and accidental data loss is less likely.

Setting up NetApp Volumes is a manual, infrastructure-level task. It is intended for administrators who need complete control over the infrastructure and who deploy into AWS or Google Cloud environments.

## Prerequisites

| Component | Requirement |
| - | - |
| **Domino** | A deployed and configured Domino. |
| **ONTAP filesystem** | A supported ONTAP filesystem, backed by Amazon FSx for NetApp ONTAP (FSxN), Google Cloud NetApp Volumes (GCNV), or NetApp Cloud ONTAP. This is where the NetApp Volumes are made and how they interact with your underlying infrastructure. |
| **Trident** | Version **26.02** or above, installed and working with your filesystem. |
| **Communication** | Verified connectivity between Domino and ONTAP. |
| **Storage configuration** | The root system file name, the base of the ONTAP filesystem on which all other NetApp Volumes are mounted. |
| **Feature flag** | `EnableDominoNetAppVolumes` set to `true`. |

## Set up NetApp Volumes

<Steps>
  <Step title="Configure Kubernetes for Domino">
    A Kubernetes `Role` and `RoleBinding` let the microservice authenticate and communicate with Trident, and a `NetworkPolicy` allows the right Kubernetes resources to reach Trident.

    ```yaml theme={null}
    # role.yaml
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: data-plane-agent
      namespace: trident
    rules:
      - apiGroups: [""]
        resources: ["secrets"]
        verbs: ["get"]
    ```

    ```yaml theme={null}
    # rolebinding.yaml
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: data-plane-agent
      namespace: trident
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: data-plane-agent
    subjects:
      - kind: ServiceAccount
        name: data-plane-agent
        namespace: domino-compute
    ```

    ```yaml theme={null}
    # networkpolicy.yaml
    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: trident-operator
      namespace: trident
    spec:
      podSelector:
        matchLabels:
          app: controller.csi.trident.netapp.io
      policyTypes:
        - Ingress
        - Egress
      ingress:
        - from: []
          ports:
            - protocol: TCP
              port: 8443
            - protocol: TCP
              port: 8001
      egress:
        - to: []
    ```

    Apply all three:

    ```bash theme={null}
    kubectl apply -f role.yaml
    kubectl apply -f rolebinding.yaml
    kubectl apply -f networkpolicy.yaml
    ```
  </Step>

  <Step title="Configure the Trident backend">
    The `TridentBackendConfig` (TBC) custom resource tells Trident how to connect to your ONTAP filesystem. Create `trident-backend.yaml` for your cloud provider, replacing the placeholders with your filesystem's details.

    <Tabs>
      <Tab title="Amazon FSxN">
        Take the values from your Terraform output:

        * `svm`: the name of the Storage Virtual Machine (SVM).
        * `fsxFilesystemID`: the ID of the filesystem.
        * `apiRegion`: the AWS region where your FSx filesystem is deployed.
        * `credentials.name`: the `creds_secret_arn` for the SVM.
        * `credentials.type`: always `awsarn` for FSxN.

        ```yaml theme={null}
        apiVersion: trident.netapp.io/v1
        kind: TridentBackendConfig
        metadata:
          name: backend-tbc-ontap-nas
          namespace: trident
        spec:
          version: 1
          storageDriverName: ontap-nas
          backendName: tbc-ontap-nas
          svm: "<your-svm-name>"
          aws:
            fsxFilesystemID: "<your-fsx-filesystem-id>"
            apiRegion: "<your-aws-region>"
          credentials:
            name: "<your-creds-secret-arn>"
            type: awsarn
        ```
      </Tab>

      <Tab title="Google Cloud NetApp Volumes">
        * `projectNumber`: the Google Cloud project number (not the project ID) that contains the GCNV storage pool.
        * `location`: the Google Cloud region or zone of the GCNV storage pool.
        * `poolID`: the GCNV storage pool that Trident provisions volumes from.
        * `proxyURL`: the GCNV API endpoint, always `https://netapp.googleapis.com`.
        * `credentials.name`: the name of the Kubernetes secret that holds the Google Cloud service account key Trident uses to authenticate to GCNV.
        * `nfsMountOptions`: the NFS mount options applied to all volumes. For example, `nconnect=8,nfsvers=4.1` parallelizes connections and pins the NFS version.

        <Note>
          The GCNV storage pool that `poolID` references must be configured in ONTAP mode.
        </Note>

        ```yaml theme={null}
        apiVersion: trident.netapp.io/v1
        kind: TridentBackendConfig
        metadata:
          name: gcnv-backend
          namespace: trident
        spec:
          version: 1
          storageDriverName: ontap-nas
          gcnv:
            projectNumber: "<gcp-project-number>"
            location: <gcp-region-or-zone>
            poolID: <gcnv-storage-pool-id>
            proxyURL: https://netapp.googleapis.com
          credentials:
            name: <k8s-secret-name>
            type: secret
          nfsMountOptions: <nfs-mount-options>
        ```
      </Tab>
    </Tabs>

    Apply it, then check that the backend is bound and online:

    ```bash theme={null}
    kubectl apply -f trident-backend.yaml
    kubectl -n trident get tbc,tbe
    ```

    ```console theme={null}
    NAME                    BACKEND NAME    BACKEND UUID                           PHASE   STATUS
    backend-tbc-ontap-nas   tbc-ontap-nas   xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx   Bound   Success

    NAME        BACKEND         BACKEND UUID
    tbe-xxxxx   tbc-ontap-nas   xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    ```

    A storage class then lets users dynamically provision NetApp Volumes on that backend:

    ```yaml theme={null}
    # storage-class.yaml
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: ontap-gold
    provisioner: csi.trident.netapp.io
    parameters:
      backendType: "ontap-nas"
    reclaimPolicy: Delete
    allowVolumeExpansion: true
    ```

    ```bash theme={null}
    kubectl apply -f storage-class.yaml
    ```
  </Step>

  <Step title="Create the root PVC">
    The filesystem you created in Amazon FSx, Google Cloud NetApp Volumes, or NetApp Cloud ONTAP includes a root ONTAP volume at the path `/`, the base on which all other NetApp Volumes are mounted. Domino reaches it through a Kubernetes `PersistentVolumeClaim` (PVC) that imports it.

    ```yaml theme={null}
    # root-pvc.yaml
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: fsx-imported-volume                                                     # Arbitrary name
      namespace: domino-compute                                                     # Should always be the compute namespace
      labels:
        dominodatalab.com/netapp-storage: "true"                                    # IMPORTANT. This is needed so the Root PVC can be found during Filesystem creation (in the next step)
      annotations:
        trident.netapp.io/importBackendUUID: "c9c51153-a02d-4f67-b3ca-6e11c8e80ee2" # This can be found by inspecting the tridentbackendconfig custom resource in the cluster, under the status.backendInfo.backendUUID field
        trident.netapp.io/importOriginalName: demo_root                             # Name of the root volume in your Filesystem. For FSx, find it in the AWS FSx UI (the name is usually <deployment_id>_svm_root); for GCNV, find it in the GCP UI
        trident.netapp.io/notManaged: "false"
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 953Mi
      storageClassName: democlass                                                   # Use the storage class of the trident storage class in the cluster
      volumeMode: Filesystem
    ```

    Apply it and confirm that the returned message indicates `STATUS - Success`:

    ```bash theme={null}
    kubectl apply -f root-pvc.yaml
    kubectl get pvc fsx-imported-volume -n domino-compute
    ```

    Your Domino deployment is now configured to use NetApp Volumes.
  </Step>
</Steps>

## Register an ONTAP filesystem

Registering a filesystem is what makes the storage usable from Domino. After that, the same area of the Admin application lets you view, audit, and manage individual NetApp Volumes.

1. In the Admin panel, go to **Manage Resources** > **NetApp Volumes**, then select the **ONTAP** tab.

2. Click **Add Filesystem**.

3. Provide the following:

   * **Name**: 1 to 250 characters, which can include letters, digits, underscores, hyphens, and spaces.

   * **Data Plane**: select from existing Data Planes. This defines where the data resides.

   * **Kubernetes Root PVC**: the name of the root PVC, which must exist in the selected Data Plane.

   * **Storage Class**: the Kubernetes storage class name that uses Trident as the provisioner.

4. Click **Save**.

<Frame>
  <img alt="Register NetApp filesystem form in the Domino Admin panel." />
</Frame>

## Review NetApp Volumes

Go to **Admin panel** > **Manage Resources** > **NetApp Volumes**, then select the **Volumes** tab to see every configured NetApp Volume and its key details.

| Column | Description |
| - | - |
| **Name** | The name of the NetApp Volume as registered in Domino. |
| **Owner Name** | The user or Project that owns the NetApp Volume. |
| **ONTAP Filesystem** | The ONTAP filesystem in which this NetApp Volume resides. |
| **Status** | The current lifecycle state, for example Active, Deleted, or Error. |
| **Permissions** | Access level settings, for example read/write access by owner or Project. |
| **Size** | The allocated capacity. |
| **Number of Snapshots** | Total count of saved snapshots. |

## Related

* [Create NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/create-netapp-volumes) covers what practitioners do once a filesystem is registered.

* [Domino Volumes API](/cloud/reference/api/domino-volumes-api) manages NetApp Volumes programmatically.

* [External Data Volumes](/cloud/admin/data-administration/external-data/external-data-volumes) mount storage that Domino does not manage, distinct from NetApp Volumes.


# Manage Project file storage
Source: https://docs.domino.ai/cloud/admin/data-administration/project-file-storage/index

Project files live at rest in the Domino Blob Store, versioned as a new revision on every sync, and removable only by a system administrator performing a full delete.

At rest, [Project files](/cloud/platform-capabilities/core-concepts/projects) live in a durable object storage system, the Domino Blob Store.

Domino natively supports the Domino File Store with the following cloud storage services:

* Amazon S3

* Azure File Storage

* Google Cloud Storage

The Domino File Store can also be backed with a shared Kubernetes Persistent Volume from a compatible storage class. You can provide a Network File System (NFS) storage service, and the Domino installation utilities can deploy the [nfs-client-provisioner](https://github.com/kubernetes-retired/external-storage/tree/master/nfs-client) and configure a compatible storage class backed by the NFS system.

## Encrypted Project files

Domino supports server-side encryption with customer-provided keys ([SSE-C](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html)) for Amazon S3, and [EBS file system encryption](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html) using the industry-standard AES-256 algorithm on Elastic Block Store.

Domino also supports default encryption keys for Amazon S3, Azure File Storage, and Google Cloud Filestore.

Domino does not provide pre-write encryption for `nfs-client-provisioner` volumes.

## How data is stored in Project files

When a user starts an [execution](/cloud/platform-capabilities/core-concepts/jobs), the files from their Project are fetched from the Domino File Store and loaded into the working directory of the [Domino File System](/cloud/platform-capabilities/core-concepts/projects/dfs-projects#the-working-directory). When the execution finishes, or the user starts a manual [sync](/cloud/platform-capabilities/core-concepts/workspaces/sync-changes-in-a-workspace) in a [Workspace session](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace), any changes to the working directory are written back as a new revision of the Project files. Domino's versioning system tracks file-level changes and provides [rich file difference information](/cloud/platform-capabilities/core-concepts/projects/dfs-projects#compare-file-revisions) between revisions.

Any of the following creates a new revision:

* A user uploads files from the Domino application upload interface.

* A user authors or edits a file in the Domino web application file editor.

* A user syncs local files to Domino from the [Domino Command Line Interface](/cloud/reference/cli/install-the-cli).

* A user uploads files to Domino through the [Domino API](/cloud/reference/api).

* A user runs code in a [Job](/cloud/platform-capabilities/core-concepts/jobs) that writes files to the working directory.

* A user writes files to the working directory during a [Workspace session](/cloud/platform-capabilities/core-concepts/workspaces), then starts a manual sync or commits those files when the session finishes.

Project files are a component of the Domino Reproducibility Engine, so by default Domino keeps every revision indefinitely. Users can always return to and work with past revisions, except for files subjected to a [full delete](#delete-every-instance-of-a-file) by a system administrator.

### Access to data in Project files

Users can read and write files in the Projects they create, on which they automatically hold an Owner role. Owners can add collaborators with roles and associated file permissions. See [Collaborator permissions](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#collaborator-permissions) for details.

Users can also inherit roles from membership in [organizations](/cloud/platform-capabilities/features/collaboration/manage-organizations). Some administrative system roles grant additional access to Project files across the deployment they administer. See [Roles](/cloud/admin/identity-and-access/manage-users/roles) for more information.

## Delete every instance of a file

A system administrator can use a full delete to remove all instances of a file's contents from the Domino File System. It finds the contents across every revision of every Project, erases them, and replaces them with a message you supply. Use it when a file holds data that must be destroyed, for example for compliance reasons.

<Warning>
  A full delete irrevocably removes every file matching the exact *contents* of the file you delete, even where the filenames differ. If several Projects contain a `data.txt` file with identical contents, full-deleting it in one Project deletes it in all of them. Files that share the filename but not the contents are unaffected.
</Warning>

1. In the Project that the file belongs to, click **Code**.

2. Click the three dots at the end of the file's row and click **Full Delete**, or click the file and then click **Full Delete**.

   <Note>
     You can perform a full delete on one file at a time.
   </Note>

3. Enter a commit message and confirm. Anyone who later opens the deleted file finds that message in place of the contents.

The operation leaves the revision history of the affected Projects intact: every commit that contained the file contents still exists, and only the contents are replaced.

<Warning>
  Domino endpoints published from Project revisions that contained a fully deleted file still carry that file inside their images, and those images cannot be purged. Contact [Domino Support](mailto:support@dominodatalab.com) if you have questions about such files.
</Warning>

## Related

* [Permanent deletion](/cloud/platform-capabilities/core-concepts/reproducibility/permanent-deletion) is the practitioner-facing view of a full delete.

* [Application logs](/cloud/admin/operations/audit-logs/application-logs) record Project file activity.


# Troubleshoot data issues
Source: https://docs.domino.ai/cloud/admin/data-administration/troubleshoot-data

Diagnose data access issues by checking network routes, firewall rules, data plane locality, feature support, and External Data Volumes.

These are some quick tips to troubleshoot issues with data in Domino.

## Check the network connection

Ensure that there is a clear route between Domino and the external data store. Confirm that firewall access rules allow traffic between the Domino deployment and the data store.

In the case of a [Domino Nexus hybrid deployment](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture), check that there is a clear route between the Domino control plane and the [data plane](/cloud/admin/platform-configuration/manage-data-planes).

## Check which Data Planes have access to the data

In a [Domino Nexus hybrid deployment](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture), different data stores are available on different [data planes](/cloud/admin/platform-configuration/manage-data-planes).

Some features are only support data available in the `Local` data plane (hosted in the control plane):

* Domino endpoints

* Apps

* Starburst-powered data sources

Other types of data sources can be accessed on both the `Local` and remote data planes, but functionality on remote data planes is in Preview.

EDVs are the primary method for accessing large data sets in Domino Nexus; they have a first-class notion of data locality.


# User accounts
Source: https://docs.domino.ai/cloud/admin/identity-and-access/index

Configure Domino user authentication with Keycloak local accounts, identity federation, or SAML, OAuth, and OIDC identity brokering.

Domino uses Keycloak to manage user accounts. Keycloak supports the following modes of authentication to Domino.

## Local accounts

When using local accounts, anyone with network access to the Domino application can create a Domino account. Users supply a username, password, and email address on the signup page to create a Domino-managed account. You can track, manage, and deactivate these accounts through the application. You can configure Domino with multi-factor authentication and password requirements through Keycloak.

## Identity federation

When identity federation is enabled, local account creation is disabled and Keycloak authenticates users against identities in the external identity provider (IdP) and retrieves configurable properties about those users, such as Domino usernames and email addresses.

See [Keycloak identity federation](https://www.keycloak.org/docs/latest/server_admin/index.html#_user-storage-federation) for more information.

## Identity brokering

You can configure Keycloak to broker authentication between Domino and an external authentication or SSO system. When identity brokering is enabled, Domino redirects users in the authentication flow to a SAML, OAuth, or OIDC service for authentication. Following authentication in the external service, the user is routed back to Domino with a token containing user properties.

See [Keycloak identity brokering](https://www.keycloak.org/docs/latest/server_admin/index.html#_identity_broker) for more information.


# Create and manage users
Source: https://docs.domino.ai/cloud/admin/identity-and-access/manage-users/activate-users

Create, activate, and deactivate Domino users, then clean up their Workspaces, executions, and Projects to prevent orphaned resources.

Create and activate users so they can use Domino. You can also deactivate users temporarily or permanently from the Admin panel.

Email services must be configured in your deployment for user creation to work.

## Create a user

Use the **Admin panel** to manually add users by email and assign their roles. They’ll receive an invitation to set up their account after creation.

<img alt="Add user" />

1. Go to **Admin panel** > **Users** > **View all**.

2. Click **Add User**.

3. Enter the user’s email address and assign roles.

4. Click **Save**.

The user receives two emails:

* A notification

* An invitation to set up their account

They must follow the setup link in the email to complete their profile.

## Activate or deactivate a user

1. Go to **Admin panel** > **Users** > **View all**.

2. Find and select the user in the list.

3. Click **Activate** or **Deactivate**, depending on their current status.

### After deactivation

After deactivating a user, complete the following cleanup tasks to prevent orphaned resources and broken executions.

1. Stop and delete workspaces:

   1. Go to **Reports** > **Workspaces** and search for any workspaces owned by the user.

   2. Delete these workspaces.

2. Check for active executions:

   1. Go to **Reports** > **Executions** to view any running endpoints or apps owned by the user.

   2. Make note of any that must remain active after project transfer. They need to be restarted.

3. Transfer or archive projects:

   1. Go to **Reports** > **Projects**.

   2. Either archive the user’s projects, or transfer ownership to another user.

<Warning>
  If you transfer a project that contains running endpoints, apps, or other executions, those executions stop working until they are manually restarted under the new owner.
</Warning>

## Next steps

* Learn how to [manage and assign user roles](/cloud/admin/identity-and-access/manage-users/roles).

* See how to [reset a user’s password](/cloud/admin/identity-and-access/manage-users/password-reset).


# User management
Source: https://docs.domino.ai/cloud/admin/identity-and-access/manage-users/index

Create and activate Domino user accounts or deactivate them, reset passwords, and review user activity with downloadable CSV reports you can schedule.

Learn how to manage users in Domino to control permissions and track activity.

[Create and activate users](/cloud/admin/identity-and-access/manage-users/activate-users)\
Create and activate users so they can use Domino. You can also deactivate users temporarily or permanently.

[Review user activity](/cloud/admin/identity-and-access/manage-users/run-user-activity-report)\
View information about a user such as their name, whether they are active, and the projects they own. You can also download the user information to a CSV file and schedule regularly generated user activity reports.

[Reset password](/cloud/admin/identity-and-access/manage-users/password-reset)\
Reset user passwords.


# Reset password
Source: https://docs.domino.ai/cloud/admin/identity-and-access/manage-users/password-reset

Reset a Domino user's local password through the Admin panel, and use an external identity provider when it manages authentication.

**CloudAdmins** can reset a user’s password. To initiate the process, the deployment must have email settings configured and utilize local user management and not an external identity provider.

To reset a password as an admin:

1. Go to **Manage resources** > **Users**.

2. Select **Reset Password** for the desired user.

Domino invalidates the user’s current password, and the user receives an email prompting them to create a new password.

When using an external identity provider to manage users, this button can be hidden and the feature disabled. For more information, please reach out to Domino Support.

## Next steps

Learn how to [run a user activity report](/cloud/admin/identity-and-access/manage-users/run-user-activity-report).


# Personal access tokens
Source: https://docs.domino.ai/cloud/admin/identity-and-access/manage-users/personal-access-tokens

Audit, filter, revoke, and delete users' Personal Access Tokens through the admin panel or API, including single and bulk operations.

Personal Access Tokens (PATs) are user-managed credentials that allow Domino users to authenticate against the Domino API. Users create their own tokens, each with a name, an optional description, and a custom expiration time.

PATs differ from [Domino Service Accounts](/cloud/admin/identity-and-access/manage-users/service-accounts) in that they are personal and tied to an individual user’s identity and roles. They are intended for interactive use, personal scripts, and jobs that a user runs on their own behalf, rather than for shared automation pipelines.

As an administrator, you can view and revoke any user’s tokens, perform bulk operations across users, and review token activity across the platform. Administrators cannot generate PATs on behalf of other users. Each user must create their own tokens from their account settings or via the API.

<Warning>
  Administrators can only access PAT metadata, such as the token name, owner, status, and expiration date. The JWT token value itself is never accessible after creation, even to administrators. This is by design: a PAT can only be retrieved by the user who created it, at the moment of creation. Administrators can revoke or delete a token, but cannot read or use it.
</Warning>

## Access the Personal Access Tokens admin panel

You can access the PAT administration panel in two ways:

* Navigate directly to `/admin/personalaccesstokens`.

* From the top navigation bar, go to **Manage resources** > **Personal Access Tokens**.

<img alt="Personal Access Tokens admin panel" />

The panel displays all PATs across every user in the platform. Each row shows the token name, description, owner username, creation date, last used date, and current status.

The **STATUS** column shows either a status badge (**Revoked**, **Expired**) or the token’s expiration date for active tokens.

You can narrow down the list using:

* The **Search by name or username** field filters by partial token name or username.

* The **All Status** dropdown filters by `Active`, `Expiring Soon`, `Expired`, or `Revoked`.

Each row has a checkbox on the left for bulk selection. When one or more tokens are selected, **Revoke selected** and **Delete selected** buttons appear in the top right of the table.

## View all Personal Access Tokens via the API

The admin panel provides a UI view of all tokens, but you can also retrieve them programmatically.

**Example endpoint information**:

```json theme={null}
GET /api/pat/v1/users/tokens
```

**Example `curl` request (calling inside Domino)**:

```shell theme={null}
curl -H "Authorization: Bearer $DOMINO_ACCESS_TOKEN" \
  "$DOMINO_API_PROXY/api/pat/v1/users/tokens"
```

**Example response**:

```json theme={null}
{
  "tokens": [
    {
      "patId": "64b1c2d3e4f5a6b7c8d9e0f1",
      "userId": "5f3a7b2c1d4e6f8a9b0c1d2e",
      "username": "alice",
      "name": "my-domino-token",
      "description": "My personal Domino API token",
      "isValid": true,
      "createdAt": "2024-04-01T10:00:00Z",
      "expiresAt": "2024-04-08T10:00:00Z"
    }
  ],
  "pagination": {
    "offset": 0,
    "limit": 10,
    "total": 1
  }
}
```

You can filter and sort results using the following query parameters:

| Parameter | Default | Description |
| - | - | - |
| `name` | None | Filter by partial token name or username match. |
| `status` | None | Filter by token status: `active`, `expiringSoon`, `expired`, or `revoked`. |
| `sortBy` | `createdAt` | Sort by: `name`, `username`, `createdAt`, `expiresAt`, or `status`. |
| `sortOrder` | `asc` | Sort direction: `asc` or `desc`. |
| `offset` | `0` | Pagination offset. |
| `limit` | `10` | Number of results per page. |

### View tokens for a specific user

**Example endpoint information**:

```json theme={null}
GET /api/pat/v1/users/{userId}/tokens
```

**Example `curl` request (calling inside Domino)**:

```shell theme={null}
curl -H "Authorization: Bearer $DOMINO_ACCESS_TOKEN" \
  "$DOMINO_API_PROXY/api/pat/v1/users/$USER_ID/tokens"
```

Supports the same query parameters as the system-wide listing above.

## Automatic token behavior

Some administrative actions on a user account automatically affect their PATs:

* When a user is **deactivated**, all their PATs are automatically deleted.

* When a user’s **roles are modified**, all their PATs are automatically revoked. The user needs to create tokens to authenticate with their updated permissions.

In both cases, no manual action is required from the administrator. The sections below cover situations where you need to revoke or delete tokens independently of these lifecycle events.

## Revoke a user’s token

Revoking a token immediately invalidates it while keeping its metadata visible. Both the user and administrators still see the token listed with a revoked status, so they are aware it was invalidated and can choose to delete it later.

### Revoke a single token

In the PAT admin panel, click the three-dot menu at the end of the token row and select **Revoke**.

Alternatively, use the API:

**Example endpoint information**:

```json theme={null}
POST /api/pat/v1/users/{userId}/tokens/{patId}/invalidate
```

**Example `curl` request (calling inside Domino)**:

```shell theme={null}
curl -X POST \
  -H "Authorization: Bearer $DOMINO_ACCESS_TOKEN" \
  $DOMINO_API_PROXY/api/pat/v1/users/$USER_ID/tokens/$PAT_ID/invalidate
```

### Revoke all tokens for a user

Use this when you need to immediately invalidate all of a user’s tokens, for example when responding to a suspected credential compromise.

**Example endpoint information**:

```json theme={null}
POST /api/pat/v1/users/{userId}/tokens/invalidate
```

**Example `curl` request (calling inside Domino)**:

```shell theme={null}
curl -X POST \
  -H "Authorization: Bearer $DOMINO_ACCESS_TOKEN" \
  $DOMINO_API_PROXY/api/pat/v1/users/$USER_ID/tokens/invalidate
```

### Revoke multiple tokens in bulk

In the PAT admin panel, select the checkboxes next to the tokens you want to revoke and click **Revoke selected**.

<img alt="Personal Access Tokens admin panel with bulk selection" />

Alternatively, use the API:

**Example endpoint information**:

```json theme={null}
POST /api/pat/v1/users/tokens/invalidate/bulk
{
  "patIds": ["64b1c2d3e4f5a6b7c8d9e0f1", "74c2d3e4f5a6b7c8d9e0f1a2"]
}
```

**Example `curl` request (calling inside Domino)**:

```shell theme={null}
curl -X POST \
  -H "Authorization: Bearer $DOMINO_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"patIds": ["64b1c2d3e4f5a6b7c8d9e0f1", "74c2d3e4f5a6b7c8d9e0f1a2"]}' \
  $DOMINO_API_PROXY/api/pat/v1/users/tokens/invalidate/bulk
```

### Key points

* Revoking a token immediately blocks its use.

* Revoked token metadata remains visible to both the user and administrators until explicitly deleted.

## Delete a user’s token

Deleting a token permanently removes it and all its metadata. Once deleted, neither the user nor administrators can see it. Rather revoke the token if you want the token to remain visible after it has been invalidated.

### Delete a single token

In the PAT admin panel, click the three-dot menu at the end of the token row and select **Delete**.

Alternatively, use the API:

**Example endpoint information**:

```json theme={null}
DELETE /api/pat/v1/users/{userId}/tokens/{patId}
```

**Example `curl` request (calling inside Domino)**:

```shell theme={null}
curl -X DELETE \
  -H "Authorization: Bearer $DOMINO_ACCESS_TOKEN" \
  $DOMINO_API_PROXY/api/pat/v1/users/$USER_ID/tokens/$PAT_ID
```

### Delete all tokens for a user

**Example endpoint information**:

```json theme={null}
DELETE /api/pat/v1/users/{userId}/tokens
```

**Example `curl` request (calling inside Domino)**:

```shell theme={null}
curl -X DELETE \
  -H "Authorization: Bearer $DOMINO_ACCESS_TOKEN" \
  $DOMINO_API_PROXY/api/pat/v1/users/$USER_ID/tokens
```

### Delete multiple tokens in bulk

In the PAT admin panel, select the checkboxes next to the tokens you want to delete and click **Delete selected**.

Alternatively, use the API:

**Example endpoint information**:

```json theme={null}
POST /api/pat/v1/users/tokens/delete/bulk
{
  "patIds": ["64b1c2d3e4f5a6b7c8d9e0f1", "74c2d3e4f5a6b7c8d9e0f1a2"]
}
```

**Example `curl` request (calling inside Domino)**:

```shell theme={null}
curl -X POST \
  -H "Authorization: Bearer $DOMINO_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"patIds": ["64b1c2d3e4f5a6b7c8d9e0f1", "74c2d3e4f5a6b7c8d9e0f1a2"]}' \
  $DOMINO_API_PROXY/api/pat/v1/users/tokens/delete/bulk
```

## Next steps

* See [Domino API authentication](/cloud/reference/api/domino-api-authentication) for how users create and manage their own PATs.

* Explore the [Domino REST API reference](/cloud/reference/api/domino-open-api) for additional API details.


# Enable and assign Data Analyst role
Source: https://docs.domino.ai/cloud/admin/identity-and-access/manage-users/roles/data-analyst

Enable the Data Analyst role, select its Hardware Tier, assign users through Keycloak, and monitor license types in activity reports.

The Data Analyst role is designed for users who have some technical background and coding experience in Python and R. Users in the Data Analyst role don’t have access to Machine Learning Operations (MLOps) capabilities like Domino endpoints, model monitoring, exports, or expensive compute capabilities such as GPU processing.

A Data Analyst user has the privileges of the Practitioner role, except that they cannot:

* Use model-related features

* Edit the hardware tier

* Edit or manage Environment definitions

Learn how to enable the Data Analyst role, assign users to the role, and monitor their activities.

## Enable the data analyst role

1. Log in as an administrator.

2. Go to **Admin** > **Platform settings** > **Configuration records** > Click **Add Record**.

3. Create a new config option `com.cerebro.domino.dataAnalyst.enabled` and set it to `true`.

Select the hardware tier that Domino assigns to all Data Analyst executions:

1. Restart the Domino services to apply the new setting.

2. Go to **Admin** > **Manage resources** > **Hardware tiers**.

3. Next to the hardware tier you want to use for Data Analysts, click **Edit** and select the **Is Tier for Data Analysts** checkbox.

4. Click **Update**.

## Assign the data analyst role

After you enable the Data Analyst role, Keycloak has a `data-analyst-user` role in the `domino-play` client. Use either of the following methods to set up a Keycloak client role.

### Manually assign the user role

Manually assign a single user the Data Analyst role:

1. Log in to Keycloak.

2. Go to **DominoRealm** > **Users** > Click the user ID > **Role Mappings**.

### Use Keycloak groups to assign the role

Assign all members of a Keycloak group to the Data Analyst role.

Regardless of your authentication framework, you must propagate the `data-analyst-user` role in the `domino-play` client and the Practitioner Domino global role. The following steps use the SAML/LDAP authentication method:

1. Log in to Keycloak and go to **DominoRealm**.

2. Click **Groups** and create a new group, such as `data-analyst-group`.

3. In the new group, click **Role Mappings**.

4. Click **Assign role** > **Filter by clients** > Select **data-analyst-user** > **Assign**.

   <img alt="Use Keycloak groups to assign the Data Analyst role" />

Add users to the Keycloak group:

1. Go to **DominoRealm** > **Users** and find the user you want to add.

2. Click the user ID > **Groups**.

3. Click **Join group** > **Data analyst**.

   <img alt="Use Keycloak groups to assign the Data Analyst role" />

## Monitor data analysts

View [user activity reports](/cloud/admin/identity-and-access/manage-users/run-user-activity-report) to view user information and activity.

After you enable the Data Analyst role, the Users page displays a **License** column with the following values:

* **DataScienceProfessional** (standard Domino license)

* **DataAnalyst** (the new Data Analyst license)

* **LiteUser** (a legacy role)


# Work with roles
Source: https://docs.domino.ai/cloud/admin/identity-and-access/manage-users/roles/index

Assign Domino roles and compare their permissions for Projects, files, Workspaces, Jobs, models, Apps, Datasets, and administration.

Administrators assign roles to users based on assignments and responsibilities. Set these roles in the application or map them from your identity provider if you have [SSO integration](/cloud/admin/identity-and-access/sso-for-domino-cloud) enabled. If you start with a completely new Domino installation, the first user to log in is assigned the **SysAdmin** and **Practitioner** roles.

The available roles are:

* **SysAdmin** - Administers instance with full administrative access (not available on Domino Cloud).

* **CloudAdmin** - Administers Domino Cloud instance with limited administrative access ([Domino Cloud](/cloud/platform-overview/deployment-options#domino-cloud) only).

* **ProjectManager** - Manages organizations and project tags.

* **SupportStaff** - Manages compute-related functionality.

* **Practitioner** - Uses compute and file storage.

* **ReadOnlySupportStaff** - View compute-related configuration.

* **Librarian** - Manages project library.

* **LimitedAdmin** - SysAdmin without access to projects and data.

* **LicenseReviewer** - Views license-related content.

* **Lite User** - A user with no role. See [Lite User](#lite-user-role).

* **GovernanceAdmin** - Manages policies with Domino Governance. See [GovernanceAdmin role](#governanceadmin-role).

<Tip>
  **LimitedAdmin** and **LicenseReviewer** roles do not grant any permissions to Projects or Data.
</Tip>

By default, all new users are assigned the **Practitioner** role.

## Edit roles

When multiple roles are assigned to a user, permissions are additive. To grant users roles, you must be a **SysAdmin** or a **CloudAdmin**. SysAdmins can grant any role to any user. CloudAdmins can grant the CloudAdmin and Practitioner roles to CloudAdmins and Practitioners.

1. In the Admin application, click **Users**.

2. Search for the username to grant permissions.

3. Click **Edit** and select the roles.

4. Click **Save**.

## Actions for specific roles

### Project overview actions

| Permission | Practitioner | SysAdmin | CloudAdmin | SupportStaff | ReadOnlySupportStaff | Librarian | Limited Admin | License Reviewer |
| - | - | - | - | - | - | - | - | - |
| Create Project | ✓ | - | - | - | - | - | - | - |
| View Project List | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | - | - |
| Archive Project | ✓ | ✓ | ✓ | ✓ | - | - | - | - |

### File actions

| Permission | Practitioner | SysAdmin | CloudAdmin | SupportStaff | ReadOnlySupportStaff | Librarian | Limited Admin | License Reviewer |
| - | - | - | - | - | - | - | - | - |
| List and View Files | ✓ | ✓ | ✓ | ✓ | ✓ | - | - | - |
| Edit Files | ✓ | - | - | - | - | - | - | - |
| Upload Files | ✓ | - | - | - | - | - | - | - |

### Workspace actions

| Permission | Practitioner | SysAdmin | CloudAdmin | SupportStaff | ReadOnlySupportStaff | Librarian | Limited Admin | License Reviewer |
| - | - | - | - | - | - | - | - | - |
| Start Workspace | ✓ | - | - | - | - | - | - | - |
| Stop Workspace | ✓ | ✓ | ✓ | ✓ | - | - | - | - |
| Open Workspace | ✓ | - | - | - | - | - | - | - |
| View Workspace History | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | - | - |
| Delete Workspace | ✓ | ✓ | ✓ | - | - | - | - | - |

### Job actions

| Permission | Practitioner | SysAdmin | CloudAdmin | SupportStaff | ReadOnlySupportStaff | Librarian | Limited Admin | License Reviewer |
| - | - | - | - | - | - | - | - | - |
| Start Job | ✓ | - | - | ✓ | - | - | - | - |
| Stop Job | ✓ | ✓ | ✓ | ✓ | - | - | ✓ (Public projects only) | - |
| View Job History | ✓ | ✓ | ✓ | ✓ | - | - | - | - |
| Create Scheduled Job | ✓ | - | - | - | - | - | - | - |
| Edit Scheduled Job | ✓ | ✓ | ✓ | - | - | - | - | - |
| Delete Scheduled Job | ✓ | ✓ | ✓ | - | - | - | - | - |

### Project settings actions

| Permission | Practitioner | SysAdmin | CloudAdmin | SupportStaff | ReadOnlySupportStaff | Librarian | Limited Admin | License Reviewer |
| - | - | - | - | - | - | - | - | - |
| View Project Settings | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | - | - |
| Edit Project Settings | ✓ | ✓ | ✓ | ✓ | - | ✓ | - | - |

### Experiment management actions

| Permission | Practitioner | SysAdmin | SupportStaff | ReadOnlySupportStaff | Librarian | Limited Admin | License Reviewer |
| - | - | - | - | - | - | - | - |
| Register a new experiment or a new run of an experiment | ✓ | ✓ | - | - | - | - | - |
| View/list/search experiments and runs (including metadata and artifacts) | ✓ | ✓ | ✓ | - | - | - | - |
| Delete (archive) an experiment or experiment run | ✓ | ✓ | - | - | - | - | - |
| Update an experiment or experiment run (includes logging artifacts, adding tags, etc.) | ✓ | ✓ | - | - | - | - | - |

### Model Registry actions

| Permission | Launcher user | Results consumer | Contributor | Project owner | SysAdmin | CloudAdmin |
| - | - | - | - | - | - | - |
| Register a new model or a new version of a model | - | - | ✓ | ✓ | ✓ | ✓ |
| Archive a registered model version | - | - | Owning user only | ✓ | ✓ | ✓ |
| Update a registered model version | - | - | ✓ | ✓ | ✓ | ✓ |
| View / list / search registered models and their versions | - | ✓ | ✓ | ✓ | ✓ | ✓ |
| Deploy a model as a Domino endpoint | - | - | ✓ | ✓ | ✓ | ✓ |
| Export a model as a Domino endpoint Image | - | - | ✓ | ✓ | ✓ | ✓ |
| Download model artifacts | - | - | ✓ | ✓ | ✓ | ✓ |

### Domino endpoint actions

| Permission | Practitioner | SysAdmin | CloudAdmin | SupportStaff | ReadOnlySupportStaff | Librarian | Limited Admin | License Reviewer |
| - | - | - | - | - | - | - | - | - |
| Create Domino endpoint | ✓ | - | - | - | - | - | - | - |
| Be a Domino endpoint "Owner" | ✓ | - | - | - | - | - | - | - |
| Be a Domino endpoint "Editor" | ✓ | ✓ | ✓ | ✓ | - | - | - | - |
| Be a Domino endpoint "Viewer" | ✓ | - | - | - | - | - | - | - |
| Stop a model version | ✓ | ✓ | ✓ | ✓ | - | - | - | - |
| View model settings | ✓ | ✓ | ✓ | ✓ | ✓ | - | - | - |
| Edit model settings | ✓ | ✓ | ✓ | ✓ | - | - | - | - |
| Promote a model version to Prod | ✓ | - | - | - | - | - | - | - |

### App actions

| Permission | Practitioner | SysAdmin | CloudAdmin | SupportStaff | ReadOnlySupportStaff | Librarian | Limited Admin | License Reviewer |
| - | - | - | - | - | - | - | - | - |
| Publish or Start App | ✓ | - | - | - | - | - | - | - |
| Stop App | ✓ | ✓ | ✓ | ✓ | - | - | - | - |
| View App | ✓ | ✓ | ✓ | ✓ | - | - | - | - |

### Launcher actions

| Permission | Practitioner | SysAdmin | CloudAdmin | SupportStaff | ReadOnlySupportStaff | Librarian | Limited Admin | License Reviewer |
| - | - | - | - | - | - | - | - | - |
| View Launchers | ✓ | ✓ | ✓ | ✓ | - | - | - | - |
| Create or Edit Launcher | ✓ | - | - | - | - | - | - | - |
| Delete Launcher | ✓ | - | - | - | - | - | - | - |
| Run Launcher | ✓ | - | - | - | - | - | - | - |

### Dataset actions

See [Dataset permissions](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#dataset-permissions) and [Dataset roles](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-datasets#dataset-roles) for more information.

| Permission | Practitioner | SysAdmin | CloudAdmin | SupportStaff | ReadOnlySupportStaff | Librarian | Limited Admin | License Reviewer |
| - | - | - | - | - | - | - | - | - |
| Create Dataset | ✓ | - | - | - | - | - | - | - |
| Mount/Unmount Dataset | ✓ | - | - | - | - | - | - | - |
| Delete Dataset Snapshot | - | ✓ | ✓ | ✓ | - | - | - | - |
| List All Datasets on Global Data Page | - | ✓ | ✓ | ✓ | - | - | - | - |
| List All Datasets and Snapshots in Admin Application | - | ✓ | ✓ | ✓ | - | - | - | - |
| Permanently Delete Datasets and Snapshots from the Admin Application | - | ✓ | ✓ | ✓ | - | - | - | - |
| Cancel Delete Requests within the time set by `com.cerebro.domino.dataset.graceTimeForDeletion`. | - | ✓ | ✓ | ✓ | - | - | - | - |
| Edit Any Dataset Permissions | - | ✓ | ✓ | ✓ | - | - | - | - |

### Environment actions

<Note>
  As a reminder, your organization incurs costs when anyone creates or stores environments.
</Note>

| Permission | Practitioner | SysAdmin | CloudAdmin | SupportStaff | ReadOnlySupportStaff | Librarian | Limited Admin | License Reviewer |
| - | - | - | - | - | - | - | - | - |
| List and View Environment | ✓ | ✓ | ✓ | ✓ | ✓ | - | - | - |
| Create Environment | ✓ | ✓ | ✓ | - | - | - | - | - |
| Edit Environment\* | ✓ | ✓ | ✓ | ✓ | - | - | - | - |

\*Only admins can edit global environments. Practitioner users can only edit a global environment if they are an owner.

### Administrator actions

| Permission | Lite User | Practitioner | SysAdmin | CloudAdmin | SupportStaff | ReadOnlySupportStaff | Librarian | Limited Admin | License Reviewer |
| - | - | - | - | - | - | - | - | - | - |
| View Admin UI | - | - | ✓ | ✓ | ✓ | ✓ | - | ✓ | ✓ |
| Edit Settings in Admin UI | - | - | ✓ | ✓ | - | - | - | ✓ | - |
| Edit Configuration records | - | - | ✓ | - | - | - | - | - | - |
| Edit Users | - | - | ✓ | ✓ | - | - | - | - | - |
| Edit Feature Flags | - | - | ✓ | - | - | - | - | ✓ | - |
| Create Global Environments | - | - | ✓ | ✓ | - | - | - | - | - |
| Edit Global Environments | - | - | ✓ | ✓ | - | - | - | - | - |
| View Usage Reports | - | - | ✓ | ✓ | - | - | - | ✓ | ✓ |
| Create Notifications | - | - | ✓ | ✓ | - | - | - | ✓ | - |
| Edit Hardware Tiers | - | - | ✓ | \* | - | - | - | ✓ | - |
| Run MongoDB Queries | - | - | \*\* | - | - | - | - | - | - |
| Manage Executions | - | - | - | - | - | - | ✓ | ✓ | - |
| View Datasets in Admin UI | - | - | ✓ | ✓ | ✓ | - | - | - | - |
| Manage Datasets in Admin UI | - | - | ✓ | ✓ | ✓ | - | - | - | - |
| Use Cost Monitoring | - | - | ✓ | ✓ | - | - | - | - | - |
| Configure Cost Budgets and Alerts | - | - | ✓ | ✓ | - | - | - | - | - |

\*CloudAdmins can only manage hardware tiers if the configuration records key `com.cerebro.domino.dominoCloud.cloudAdmin.canManageHwTiers` is set to `true`.

\*\*MongoDB access is disabled in Domino Cloud.

### Organization actions

| Permission | Lite User | Practitioner | SysAdmin | CloudAdmin | SupportStaff | ReadOnlySupportStaff | Librarian | Limited Admin | License Reviewer |
| - | - | - | - | - | - | - | - | - | - |
| Create Organizations | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | - | ✓ | - |
| Organization Owner Can Add/Remove Members To/From the Organization | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | - |
| Organization Owner Can Make Another User an Owner of the Organization | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | - |
| Add/Remove Members To/From Any Organization | - | - | ✓ | ✓ | - | - | - | - | - |
| Can Make Another User an Owner of Any Organization | - | - | ✓ | ✓ | - | - | - | - | - |
| Select Hardware Tiers Available to Members of the Organization | - | - | ✓ | ✓ | - | - | - | ✓ | - |

<Note>
  You cannot delete organizations after you create them.
</Note>

## More information about specific roles

### Project manager role

When Project Managers are members of organizations, their role grants them owner-level access to all projects that are owned by other members of the organizations. This allows the Project Manager to see these projects and their assets in the Projects Portfolio and Assets Portfolio.

The Project Manager might also have the ability to add users to these organizations, thereby gaining contributor access to those users' projects. For this reason, the Project Manager must be treated as a highly privileged role, similar to System Administrator.

### CloudAdmin role

CloudAdmins are given most of the access SysAdmins have, but not all. CloudAdmins are only available on [Domino Cloud](/cloud/platform-overview/deployment-options#domino-cloud).

CloudAdmins cannot do the following:

* Manage configuration records

* Manage feature flags

* Manage email configuration

* Manage search index

* Manage API keys

* Run MongoDB commands

* View Kubernetes dashboard

* Restart Nucleus

CloudAdmins have full user management capabilities with one restriction: they cannot manage users with the SysAdmin role or assign the SysAdmin role to any user.

### Lite user role

A user with no roles is called a **Lite User** or, in some contexts, a **Results Consumer**. They have restricted feature access and may have a different licensing status.

**Lite Users** have permission to do the following:

* View the project list.

* View files in a project.

* View Workspace history.

* View Job history.

* Be added as collaborators of Domino endpoints.

* View Apps.

* View and run Launchers (if permitted in project settings).

* List and view Environments.

* View experiments.

### Data analyst role

The Data Analyst role is for users who have some technical background and coding experience in Python and R, but who do not need access to all the MLOps features of Domino. For more information, see [Data Analyst role](/cloud/admin/identity-and-access/manage-users/roles/data-analyst).

### GovernanceAdmin role

The GovernanceAdmin role has permission to do the following:

* View bundles.

* View approvals.

* Query audit events.

* View policy overviews.

* Manage policies evidence templates.

For more information, see [Domino Governance policies](/cloud/platform-capabilities/features/governance/define-governance-policies).


# Review user activity
Source: https://docs.domino.ai/cloud/admin/identity-and-access/manage-users/run-user-activity-report

Review user status, roles, workloads, and Projects, then download or schedule CSV activity reports with configurable dates and recipients.

You can review user behavior across Domino.

1. In the navigation pane, click **Manage resources > Users**.

2. The report shows you when they signed up, the number of workloads run, whether they are active, and their roles.

## Generate a user activity report

You can download a CSV of the user activity information.

1. In the navigation pane, click **Admin**.

2. Go to **Reports** > **User Activity Report**.

3. Enter the following:

   * The **Start Date** and **End Date** for the report.

   * The number of days to see in the recent activity section.

   * The organization for which you want to see data.

   * Email addresses to send copies of the report.

4. Click **Download Report**.

## Schedule a user activity report

Configure certain configuration records keys to schedule User Activity reports to be automatically generated and emailed.

1. In the navigation pane, click **Admin**.

2. Go to **Platform Settings** > **Configuration Records**.

3. Set the options below; `com.cerebro.domino.com.cerebro.domino.Usage.ReportRecipients` is required.

   <table>
     <thead>
       <tr>
         <th>Key</th>
         <th>Default</th>
         <th>Description</th>
       </tr>
     </thead>

     <tbody>
       <tr>
         <td><p><code>com.cerebro.domino.email.usageReportRecipient</code></p></td>
       </tr>

       <tr>
         <td />

         <td><p><code>[usage@dominodatalab.com](mailto:usage@dominodatalab.com)</code></p></td>
         <td><p>Sets the default recipient for the user activity report. To access this report, go to <strong>Reports</strong> > <strong>User Activity Report</strong>.</p></td>
       </tr>

       <tr>
         <td><p><code>com.cerebro.domino.email.EmailToDomino</code></p></td>
       </tr>

       <tr>
         <td />

         <td><p><code>true</code></p></td>
         <td><p>When <code>true</code>, the system sends a scheduled user activity report to <a href="mailto:usage@dominodatalab.com">[usage@dominodatalab.com](mailto:usage@dominodatalab.com)</a> to help improve Domino.</p></td>
       </tr>

       <tr>
         <td><p><code>com.cerebro.domino.Usage.RecentUsageDays</code></p></td>
       </tr>

       <tr>
         <td />

         <td><p><code>30</code></p></td>
         <td><p>Specifies the number of days to report for recent activity in the user activity reports. For example, the default value includes activity within the past 30 days in the Recent Activity section.</p>&#x20;
         &#x20;<p>The cron entry is set in UTC.</p></td>
       </tr>

       <tr>
         <td><p><code>com.cerebro.domino.Usage.ReportFrequency</code></p></td>
       </tr>

       <tr>
         <td />

         <td><p><code>0 0 2 \* \* ?</code></p></td>
         <td><p>Defines the frequency for automatically scheduled user activity reports. The default cron string value is set to daily at 02:00.</p>&#x20;
         &#x20;<p>This value must be a QUARTZ cron string expression. See the <a href="http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html">Cron Quartz tutorial</a>.</p></td>
       </tr>

       <tr>
         <td><p><code>com.cerebro.domino.Usage.ReportRecipients</code></p></td>
       </tr>

       <tr>
         <td />

         <td><p>Empty</p></td>
         <td><p>Identifies a comma-separated list of email addresses that receive automatic scheduled user activity reports. This is not shown in the configuration records unless it is set explicitly. Example values are: <a href="mailto:email1@domain.com">[email1@domain.com](mailto:email1@domain.com)</a>, <a href="mailto:email2@domain.com">[email2@domain.com](mailto:email2@domain.com)</a>.</p></td>
       </tr>

       <tr>
         <td><p><code>com.cerebro.domino.Usage.IncludeUsernameAndEmail</code></p></td>
       </tr>

       <tr>
         <td />

         <td><p><code>false</code></p></td>
         <td><p>When <code>true</code>, automatically generated and emailed user activity reports include username and email address columns.</p></td>
       </tr>
     </tbody>
   </table>


# Service accounts
Source: https://docs.domino.ai/cloud/admin/identity-and-access/manage-users/service-accounts

Create Domino Service Accounts for API automation, assign roles, issue and rotate tokens, and manage username, SSH, or token Git credentials.

Domino Service Accounts (DSAs) provide a secure way to access the Domino API without relying on personal user credentials. They are automation principals: technical users managed by administrators, not human users behind a browser. DSAs can’t log in through the Domino web interface or CLI, and they are the only accounts permitted to use long-lived API tokens.

Each DSA can have one or more tokens, each with its own independent lifespan. By default, tokens expire after four months. Tokens inherit the DSA’s roles at creation and can be revoked individually or by deactivating the DSA. This allows administrators to control access precisely while maintaining security. Typical use cases include: CI/CD pipelines, external monitoring or automation tools, and system-to-system integrations.

DSAs simplify automation by providing a stable identity with scoped permissions, independent of personal user accounts.

<img alt="Domino Service Account Lifecycle" />

The DSA lifecycle includes:

* Creating the account.

* Assigning or adjusting roles.

* Generating a token (which inherits roles at creation).

* Using a token to authenticate.

* Reviewing tokens regularly to rotate or revoke them as needed.

## Create a Domino service account

A Domino Service Account (DSA) is created through the [Domino API](/cloud/reference/api/domino-api-authentication). Only [Administrators](/cloud/admin/identity-and-access/manage-users/roles) can create service accounts.

When you create a DSA, it inherits the roles of the user who created it. You can later assign or remove roles directly on the DSA. Tokens generated for the DSA reflect the roles present at the time of token creation.

<Note>
  The easiest way for administrators to call Domino APIs is from a [Workspace](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace) terminal using the [API Proxy](/cloud/reference/api/domino-api-authentication), which handles authentication automatically.

  You can use these examples inside or outside a Domino Workspace. Follow the instructions for your use case.
</Note>

To create a Service Account, the Administrator makes a call to the `/api/serviceAccounts/v1/serviceAccounts` endpoint, supplying the desired username for the Service Account and a valid email address. Domino Service Accounts, just like regular Domino users, need a valid and unique email address. Domino uses this email address to send notifications, inform about failed jobs, and for other purposes.

Domino Service Accounts, just like regular Domino users, need a valid and unique email address. Domino uses this email address to send notifications, inform about failed jobs, and for other purposes.

**Example endpoint information**:

```json theme={null}
POST /api/serviceAccounts/v1/serviceAccounts
{
  "userName": "demo-sa",
  "email": "demo-sa@customer.com"
}
```

**Example `curl` request (calling inside Domino)**:

```shell theme={null}
curl -X POST -H 'Content-Type: application/json' \
 -d '{"username": "demo-sa", "email": "demo-sa@customer.com"}' \
$DOMINO_API_PROXY/api/serviceAccounts/v1/serviceAccounts
```

**Example response**:

```json theme={null}
{
  "id": "abcd1234",
  "userName": "demo-sa",
  "email": "demo-sa@customer.com",
  "idpld": "ca72dc97-504b-4b3c-98d6-3a306b30bd24"
}
```

### Key points

* The `userName` and `email` values identify the DSA.

* DSAs do not have passwords and cannot log in interactively.

* Roles inherited at creation may or may not be sufficient. Adjust them before generating tokens if necessary.

## Assign roles to a DSA

Roles define what a Domino Service Account (DSA) can do. When you create a DSA, it inherits the roles of the user who created it. You can add or remove roles later to match the needs of the automation or integration. You can [manage roles using the Admin panel](/cloud/admin/identity-and-access/manage-users/roles) in Domino.

### Key points

* Roles control the scope of access for any tokens generated from the DSA.

* Tokens capture the DSA’s roles at the time of creation. If you later change roles, generate a new token to reflect the updated permissions.

* Common roles include:

  * `Practitioner` - Uses compute and file storage

  * `SupportStaff` - Manages compute-related functionality

## Generate a token for a Domino service account

After you create a Domino Service Account (DSA), generate a token so it can authenticate in API requests or automation workflows.

### Role behavior

* A token inherits the DSA’s roles at the time of token creation.

* If you later add or remove roles on the DSA, existing tokens are not updated.

* To capture role changes, generate a new token.

### Token `name` field

The `name` field in the token creation request is an arbitrary label. It does not need to match the DSA’s `userName` or `email`. Use a descriptive value that indicates the purpose of the token.

The `serviceAccountIdpId` uniquely identifies the DSA and is required in API calls.

**Example endpoint information**:

```json theme={null}
POST /api/serviceAccounts/v1/serviceAccounts/$IDPID/tokens
{
  "name": "token-for-circleci"
}
```

**Example `curl` request (calling inside Domino)**:

```shell theme={null}
curl -X POST -H 'Content-Type: application/json' \
  -d '{"name": "token-for-circleci"}' \
$DOMINO_API_PROXY/api/serviceAccounts/v1/serviceAccounts/$IDPID/tokens
```

**Example response**:

```json theme={null}
{
  "createdAt" : "2024-02-15T00:39:35.521Z",
  "expiresAt" : "2024-06-14T00:39:35.513201660Z",
  "isValid" : true,
  "name" : "token-for-circleci",
  "serviceAccountIdpId" : <IDPID>,
  "token" : "eyJhbGciOiJSUzI1N..."
}
```

### Key points

* The `token` is active immediately after creation.

* The `name` is for identification only. It does not affect permissions.

* The `serviceAccountIdpId` uniquely identifies the DSA and is required in API calls.

* Save the value of the IdpId property in the IDPID [environment variable](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables) for future use.

* Store tokens securely. You need to pass it as a bearer token in API requests or configure it in automation systems that act on behalf of the DSA.

## Use a token

Tokens authenticate automation systems or external tools on behalf of a Domino Service Account (DSA). Use them in place of personal credentials.

### Pass the token in API requests

Include the token as a bearer token in the request header:

```shell theme={null}
curl -X GET "https://your-domino-domain.com/v4/projects" \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIs..."
```

### Configure tokens in automation

* Examples of where to store and use tokens:

  * Environment variables in CI/CD pipelines

  * Secret managers such as Vault, AWS Secrets Manager, or Kubernetes Secrets

  * Configuration files for monitoring or automation scripts

### Security guidelines

* Treat tokens as sensitive credentials.

* Store them securely and never check them into version control.

* Rotate tokens periodically or immediately if you suspect compromise.

### Key points

* Tokens act with the full permissions of the DSA.

* Use tokens only in trusted automation systems.

* Rotate and revoke tokens regularly to maintain security.

## Rotate or revoke tokens

Tokens act as credentials for a Domino Service Account (DSA). DSA tokens expire by default four months after creation. If a token expires, a new one must be created.

You can also rotate or revoke tokens regularly to maintain security and to align with changes in role assignments.

### When to rotate

* You update the DSA’s roles and need a token with the new permissions.

* Security policies require periodic token rotation.

* You suspect a token has been exposed or compromised.

### How to rotate

1. Generate a new token for the DSA.

2. Update automation systems or scripts to use the new token.

3. Revoke the old token once the new one is in use.

### List tokens for a DSA

Use the Domino API to return all tokens associated with a DSA. This is useful for auditing or identifying tokens to revoke.

The `serviceAccountIdpId` uniquely identifies the DSA and is required in API calls.

**Example endpoint information**:

```json theme={null}
GET /api/serviceAccounts/v1/serviceAccounts/$IDPID/tokens
```

**Example `curl` request (calling inside Domino)**:

```shell theme={null}
curl -s
$DOMINO_API_PROXY/api/serviceAccounts/v1/serviceAccounts/$IDPID/tokens
```

**Example response**:

```json theme={null}
[
  {
    "id": "efgh5678",
    "name": "token-for-circleci",
    "createdAt": "2024-02-15T00:39:35.521Z",
    "expiresAt": "2024-06-14T00:39:35.513201660Z",
    "isValid": true
  },
  {
    "id": "ijkl9012",
    "name": "token-for-monitoring",
    "createdAt": "2024-03-01T10:22:11.100Z",
    "expiresAt": "2024-07-01T10:22:11.100Z",
    "isValid": false
  }
]
```

### Revoke a token

Use the Domino API to invalidate a token.

The `serviceAccountIdpId` uniquely identifies the DSA and is required in API calls.

**Example endpoint information**\*:

```json theme={null}
POST /api/serviceAccounts/v1/serviceAccounts/$IDPID/tokens/$TOKEN_NAME/invalidate
```

**Example `curl` request (calling inside Domino)**:

```shell theme={null}
curl -H 'Accept: application/json' -X POST \
$DOMINO_API_PROXY/api/serviceAccounts/v1/serviceAccounts/$IDPID/tokens/token-for-circleci/invalidate
```

### Delete a token

Use the Domino API to delete a token.

The `serviceAccountIdpId` uniquely identifies the DSA and is required in API calls.

**Example endpoint information**:

```json theme={null}
DELETE /api/serviceAccounts/v1/serviceAccounts/$IDPID/tokens/$TOKEN_NAME
```

### Add Git credentials to a service account

Use the Domino API to associate Git credentials with a service account.

The `serviceAccountIdpId` uniquely identifies the DSA and is required in API calls.

Domino supports three different types of Git credentials, each using a different endpoint.

<Tabs>
  <Tab title="Username/Password">
    **Example endpoint information:**

    ```json theme={null}
    POST /api/serviceAccounts/v1/serviceAccounts/{serviceAccountIdpId}/passwordgitcredentials
    {
      "name": "password-git-creds",
      "gitServiceProvider": "github",
      "username": $USERNAME,
      "password": $PASSWORD,
    }
    ```

    **Example `curl` request (calling inside Domino):**

    ```shell theme={null}
    curl -X POST \
    $DOMINO_API_PROXY/api/serviceAccounts/v1/serviceAccounts/$IDPID/tokens/passwordgitcredentials \
    -d '{"name": "password-git-creds", "gitServiceProvider": "github", "username": $USERNAME, "password": $PASSWORD}'
    ```
  </Tab>

  <Tab title="SSH">
    **Example endpoint information:**

    ```json theme={null}
    POST /api/serviceAccounts/v1/serviceAccounts/{serviceAccountIdpId}/sshgitcredentials
    {
      "name": "ssh-git-creds",
      "gitServiceProvider": "github",
      "key": $KEY,
      "passphrase": $PASSPHRASE,
    }
    ```

    **Example `curl` request (calling inside Domino):**

    ```shell theme={null}
    curl -X POST \
    $DOMINO_API_PROXY/api/serviceAccounts/v1/serviceAccounts/$IDPID/tokens/sshgitcredentials \
    -d '{"name": "ssh-git-creds", "gitServiceProvider": "github", "key": $KEY, "passphrase": $PASSPHRASE}'
    ```
  </Tab>

  <Tab title="Token">
    **Example endpoint information:**

    ```json theme={null}
    POST /api/serviceAccounts/v1/serviceAccounts/{serviceAccountIdpId}/tokengitcredentials
    {
      "name": "token-git-creds",
      "gitServiceProvider": "github",
      "token": $TOKEN
    }
    ```

    **Example `curl` request (calling inside Domino):**

    ```shell theme={null}
    curl -X POST \
    $DOMINO_API_PROXY/api/serviceAccounts/v1/serviceAccounts/$IDPID/tokens/tokengitcredentials \
    -d '{"name": "token-git-creds", "gitServiceProvider": "github", "token": $TOKEN}'
    ```
  </Tab>
</Tabs>

<Note>
  * **`gitServiceProvider`**: Only certain values for this field are defined. The value should match `bitbucket`, `bitbucketServer`, `github`, `githubEnterprise`, `gitLab`, `gitLabEnterprise`, or `unknown`.

  * **`domain`**: If the `gitServiceProvider` field has the value `bitbucketServer`, `githubEnterprise`, `gitLabEnterprise`, or `unknown`, then the `domain` field must be passed in the body of a `POST` to this endpoint.
</Note>

All three endpoints have the same response structure.

**Example response**:

```json theme={null}
{
  "credential": {
    "id": $ID,
    "name": $NAME,
    "gitServiceProvider": "Github",
    "domain": "github.com",
    "fingerprint": $FINGERPRINT,
    "protocol": "https"
  },
  "metadata": {
    "requestId": $REQUEST_ID,
    "notices": []
  }
}
```

### Remove Git credentials from a service account

Use the Domino API to dissociate Git credentials from a service account.

The `serviceAccountIdpId` uniquely identifies the DSA and is required in API calls.

The `credentialId` is returned in the response to the API call to associate a Git credential to a service account.

**Example endpoint information:**

```json theme={null}
DELETE /api/serviceAccounts/v1/serviceAccounts/{serviceAccountIdpId}/gitcredentials/{credentialId}
```

**Example `curl` request (calling inside Domino):**

```shell theme={null}
curl -X DELETE \
$DOMINO_API_PROXY/api/serviceAccounts/v1/serviceAccounts/$IDPID/tokens/gitcredentials/$CREDENTIALID \
```

### Key points

* Use `GET /tokens` to review existing tokens for auditing.

* Revoking a token immediately invalidates it.

* Always revoke tokens you no longer use.

* Keep only active, necessary tokens in circulation.

## Advanced: Multiple service accounts with plus addressing

When creating multiple DSAs, you can use **plus addressing** to distinguish them while reusing the same base email address. For example:

```
automation+ci@customer.com

automation+monitoring@customer.com

automation+etl@customer.com
```

Domino treats each address as a unique identity, even though they all route to the same base mailbox - `automation@customer.com`. This approach simplifies administration while keeping service accounts separate in Domino.

## Next steps

* Explore the [Domino REST API reference](/cloud/reference/api/domino-open-api).

* See other methods users can use to [authenticate against the Domino API](/cloud/reference/api/domino-api-authentication).


# Authentication & SSO
Source: https://docs.domino.ai/cloud/admin/identity-and-access/sso-for-domino-cloud

Integrate Domino Cloud with SAML SSO, map user attributes and roles from Okta or Azure, and provide the required metadata to Domino Support.

Domino Cloud can be integrated into SAML2 SSO IDPs with minimal required configuration. Note that for security reasons, this integration comes disabled by default.

<Note>
  This document provides guidance for configuring Domino Cloud instances, which come with a pre-built SSO provider. If you want to manually configure a non-cloud instance, see [Domino Cloud SSO integration](/cloud/admin/identity-and-access/sso-for-domino-cloud) for more details.
</Note>

## Cluster endpoint

The redirect URI for your cluster has the following pattern:

`https://{your-cluster-name}.domino.tech/auth/realms/DominoRealm/broker/saml/endpoint`

This URL is both the recipient and destination of the Single Sign-On process, as well as the Entity ID (audience URI).

### SAML descriptor

You can download the SAML descriptor metadata from the following URL:

`https://{your-cluster-name}.domino.tech/auth/realms/DominoRealm/broker/saml/endpoint/descriptor`

This includes expected attributes, as well as the configured nameID format.

## Required attributes

### NameID, username and email

Users are uniquely identified by their email, and their username is automatically derived from it. The resulting username is obtained by keeping the email username (without the domain) and normalizing them.

If the provided NameID is in the email format, this is done automatically. No extra attribute is required for the email.

```xml theme={null}
<saml2:NameID Format="urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress">john.smith@acme.org</saml2:NameID>
```

Typically, however, the SAML endpoint returns a NameID which is a GUID that is not suitable for a username. In this case, an explicit email attribute is used instead. This attribute must be called `email`.

### Data attributes

If you don’t want your users to be required to provide their personal details on the first login, add these attributes to prepopulate them.

* `firstName`

* `lastName`

## Integrating roles with Domino Cloud

If you also want to use the third-party provider for authorization, then set the `domino-system-roles` attribute to include the desired roles. You then need to request Domino Support to enable role synchronization for your cluster.

Once this is enabled, all roles need to be explicitly provided. This includes previously defaulting roles such as `Practitioner`.

### Available roles

Available roles for Domino Cloud users are:

* `Practitioner`

* `CloudAdmin`

* `Librarian`

* `ProjectManager`

<Note>
  If roles are integrated with the IDP, changes done through Domino’s admin panel are not reflected and are overwritten the next time the user logs in. You need to make changes to the provided `domino-system-roles` attribute sent instead.
</Note>

### Multiple roles support

The `domino-system-roles` attribute supports the multi-valued attribute format. Provide each role in a separate `AttributeValue` in the authentication claim. CSV is currently not supported.

#### Okta-specific configurations

If you are using Okta as your SSO provider, you can either:

* Use Okta groups to control the attributes being sent.

* Use attributes for every user logging in through SSO using Okta expressions.

##### Using Okta groups

1. Create a group in your Okta environment for every Domino role you want users linked to. The group names must exactly match Domino’s role names.

2. Create a Group Attribute Statement called `domino-system-roles`, and set the format as `unspecified` or `basic`.

3. Add a filter so that only Domino-related groups are sent. You can use a regex filter like `Practitioner|CloudAdmin|Librarian|ProjectManager`.

At login time, Okta provides a list of all groups the user is currently assigned to (that match the provided filter) under the `domino-system-roles` attribute.

```xml theme={null}
<saml2:Attribute Name="domino-system-roles"
    NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified">
    <saml2:AttributeValue
        xmlns:xs="http://www.w3.org/2001/XMLSchema"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">Practitioner
    </saml2:AttributeValue>
    <saml2:AttributeValue
        xmlns:xs="http://www.w3.org/2001/XMLSchema"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">CloudAdmin
    </saml2:AttributeValue>
</saml2:Attribute>
```

##### Using Okta expressions

1. Add an attribute statement to your SAML configuration with the name `domino-system-roles`, and choose `basic` as the format.

2. Provide the Okta expression that evaluates to the appropriate roles array at login time.

For fixed roles (if you want all users logging in through SSO to have the same roles), you can use a simple expression such as the inline array `{"CloudAdmin", "Practitioner"}`. For more complex use cases, see the Okta expression reference: [Okta Expression Language#Array Functions](https://developer.okta.com/docs/reference/okta-expression-language/#array-functions). Note that the result of the expression needs to be an array, not a CSV string.

#### Azure-specific configurations

If you are using Azure as your SSO provider:

* Make sure the created application is an Enterprise application.

* Make sure that SAML is selected as the SSO authentication mechanism.

* Update the SSO attribute claims to provide the required data attributes. Microsoft ships some claims out of the box, but the names they are generated with do not match the attribute names expected by Domino. Also make sure to remove the namespaces for the provided attribute claims, as they won’t match otherwise.

* Update the entity ID so it matches with the SSO URL. You can do this by [downloading the SAML descriptor metadata](/cloud/admin/identity-and-access/sso-for-domino-cloud#saml-descriptor), and re-uploading it to your application configuration.

The expected entity ID is similar to:

`https://{your-cluster-name}.domino.tech/auth/realms/DominoRealm`

The expected Reply URL is similar to:

`https://{your-cluster-name}.domino.tech/auth/realms/DominoRealm/broker/saml/endpoint`

Logout URL is similar to:

`https://{your-cluster-name}.domino.tech/auth/realms/DominoRealm/broker/saml/endpoint`

##### Using Azure groups

<Note>
  You need an Azure subscription capable of assigning user groups to Enterprise Applications to do this. This is only required for using Azure as the authorization provider, and is not required if you only use Azure for authentication purposes.
</Note>

1. Open Enterprise Applications, select the application in the list, select **Single Sign-On configuration**, and then select **User Attributes & Claims**.

2. Add a new Group Claim.

3. For the group type emitted in the token, select **Groups assigned to the application**.

4. For the Source Attribute, select **Cloud-only group display names**.

5. In advanced settings of the group claim, check "Customize the name of the Group Claim". Use `domino-system-roles` as the claim name, and no Namespace.

6. In the **Groups** tab, assign one group to the application per Domino role you wish to map to. Group names need to match [available roles for Domino Cloud](/cloud/admin/identity-and-access/sso-for-domino-cloud#available-roles).

For more details, see [Microsoft Entra - Emit cloud-only group display name in token](https://learn.microsoft.com/en-us/entra/identity/hybrid/connect/how-to-connect-fed-group-claims#emit-cloud-only-group-display-name-in-token)

## Reaching back to Domino Support

Once you have configured the SSO provider accordingly, the following needs to be provided to Domino Support to complete enabling the integration:

* The Single Sign-On Service URL to be used for authentication. For Okta, this is usually in the format of `https://{customer-okta-domain}.okta.com/app/{okta-app-name}/{s0m3rand0m1d}/sso/saml`

* The certificate for signature validation.

* Whether roles should be integrated with the SSO provider or not.

* A custom name to be used on the cluster login page for the button, if desired. Defaults to `saml` for a "Continue with saml" button.

### IDP-initiated flows

Domino Cloud currently does not support initiating the authentication flow from the SSO provider. Attempting to do so results in errors similar to this:

```
We are sorry...

Invalid request
```

SSO can only be initiated from the Domino Cloud cluster’s login page, through the `"Continue with {my sso}"` button.


# Admin overview
Source: https://docs.domino.ai/cloud/admin/index

Administer Domino infrastructure, data, users, operations, deployment settings, and integrations for governed data science and AI workloads.

Domino unifies infrastructure and data to drive innovation, enhance control, and lower costs for all data science and AI initiatives.

It lets you empower data science teams with the tools they love and with the governance guardrails to ensure safe, compliant, and cost-effective AI. Domino helps IT deliver the promise of AI value with the most flexible, scalable, and safe architecture for your AI operations.

* **Improve productivity**: Deliver self-serve tools, secure access to data anywhere, and standardized computing in IT-governed sandboxes.

* **Run AI anywhere**: On hybrid, multicloud, or on-prem to optimize costs, ensure compliance, and avoid vendor lock-in.

* **Slash operational costs**: Simplify AI management by unifying commercial and open source tools, disparate stacks, and data silos on one future-ready platform.

<Frame>
  <img alt="Any data, any language and dev tool, and any compute framework feed the Domino platform, which provides a system of record, a workbench, MLOps, and infrastructure orchestration on top of data platforms and cloud or on-premises infrastructure, and delivers APIs, apps, and edge deployments." />

  <img alt="Any data, any language and dev tool, and any compute framework feed the Domino platform, which provides a system of record, a workbench, MLOps, and infrastructure orchestration on top of data platforms and cloud or on-premises infrastructure, and delivers APIs, apps, and edge deployments." />
</Frame>

## How do we do it?

Domino is an open system that provides self-service access to data and tools, enables the reuse of materials, and facilitates collaboration with other teams in your organization while enforcing best practices, enhancing knowledge, and improving efficiency.

* [Data](/cloud/platform-capabilities/core-concepts/data/data-source-connectors): Domino connects to external data sources such as databases, data warehouses, and data lakes. The complete list of supported data sources is available under [Data Source Connectors](/cloud/platform-capabilities/core-concepts/data/data-source-connectors).

* [LLMs](/cloud/platform-capabilities/features/llms): Access external Large Language Model (LLM) providers such as OpenAI, Anthropic, and AWS Bedrock, or host your own model as a Domino endpoint.

* [Software](/cloud/platform-capabilities/core-concepts/projects): Domino can be connected to [GitHub](/cloud/platform-capabilities/features/coding-assistants), [MLflow](cloud/platform-capabilities/features/agents/develop), and [Sagemaker](/cloud/admin/platform-configuration/manage-model-deployments#external-deployment-targets). This enables integration of your data science workflows and allows users to track progress on data science projects.

* [Languages](/cloud/platform-overview/getting-started-with-domino): Domino allows data scientists to use their preferred languages and tools, such as Python, SAS, MATLAB, and R.

* [IDEs and Tools](/cloud/platform-capabilities/core-concepts/workspaces): You can use IDEs and tools like Jupyter Notebook, JupyterLab, RStudio, VS Code, MATLAB, and SAS with Domino.

* [Packages and Libraries](/cloud/admin/platform-configuration): We support a range of packages and libraries, including open source options such as Python, R, TensorFlow, PyTorch, and others.

* [Compute Environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/8-compute-environment-catalog): You can add different environments to any Domino installation by visiting [quay.io](https://quay.io/). Our [documentation](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/3-create-an-environment) has step-by-step instructions for setting up these environments.

## Administering Domino

These sections provide some information for Domino CloudAdmins.

* [Platform management](/cloud/admin/platform-configuration): Manage project files and basic project settings for users to make sure the needs of your organization are fulfilled.

* [User management](/cloud/admin/identity-and-access/manage-users): Learn how to manage users in Domino to control permissions, set up accounts, and track activity.

* [Operations](/cloud/admin/operations): Monitor and optimize the performance of Domino systems.

* [Deployment configuration](/cloud/admin/architecture-and-deployment/deployment-configuration): Configure different aspects of the Domino Deployment, including configuration records and IP whitelisting.


# Administer Slurm for HPC workloads
Source: https://docs.domino.ai/cloud/admin/infrastructure-and-compute/administer-slurm

Architecture, setup, and operation of Slurm on a Domino Data Plane, so practitioners can run batch HPC Jobs from a Workspace.

Slurm support adds a persistent Slurm cluster to a Data Plane, with autoscaled worker capacity attached to it, so practitioners can submit batch [HPC Jobs](/cloud/platform-capabilities/features/compute-clusters-hpc/slurm) from a Workspace using the standard Slurm CLI. Domino enables Slurm when a Data Plane is deployed. After that, your work is creating the two Compute Environments Slurm needs, sizing the worker Hardware Tiers, and making sure a shared Dataset is available on the Data Plane.

## How it works

Domino runs High-Performance Computing (HPC) workloads on [Slurm](https://slurm.schedmd.com/), deployed on Kubernetes with [Slinky](https://slinky.schedmd.com/), SchedMD's Slurm operator. Slurm is supported through the Distributed Compute Operator (DCO), like the other compute cluster types, but its deployment model differs in one important way: rather than one ephemeral cluster per execution, each Data Plane runs a single persistent Slurm cluster shared across users, with ephemeral, autoscaled compute attached to it. Slurm is a stateful scheduler, where priority, fair-share, backfill, and limits all depend on recent global state, so its central scheduling and accounting services persist on the Data Plane while worker capacity scales up and down with demand.

* **Slinky** manages all Slurm cluster resources on Kubernetes, including Slurm's central services: the controller, which schedules jobs, and the accounting service, which records the job runtime data used for quotas, access control, and Domino's job monitoring.

* **Slurm is optional**, and adds services to the Data Plane that raise its baseline resource usage.

* **Slurm is part of the Data Plane.** The Control Plane requires no additional components, so a configuration with Slurm absent from the Control Plane and the local Data Plane but installed on a remote Data Plane is valid. Domino enables the Slurm controls in the UI from the component information each Data Plane reports, and the Control Plane collects Slurm job statuses and metrics from the Data Plane no matter how a job was submitted.

* **Runtime statistics are stored permanently.** The Control Plane captures the runtime statistics of every HPC Job and keeps them indefinitely, so the record outlives the job, the worker nodes that ran it, and the Workspace that submitted it.

Workspaces with an attached Slurm cluster act as Slurm login nodes. Domino injects the configuration and credentials the Slurm CLI needs.

### Resources created at runtime

* **Nodesets**: groups of worker nodes with identical characteristics, managed by the DCO. Each Workspace gets its own dedicated nodeset.

* **Partitions**: logical scheduling queues, currently one per nodeset, managed by the Slurm controller through the nodeset custom resource.

* **Workers**: compute nodes, scaled up and down automatically based on queued work. Each worker belongs to exactly one nodeset and one partition.

### Execution flow

When a user launches a Workspace with an attached Slurm cluster:

<Steps>
  <Step title="The Control Plane builds the cluster manifest">
    The dispatcher creates a `slurmcluster` custom resource from the launch parameters and the Project configuration, and sends the manifest to the Data Plane.
  </Step>

  <Step title="The Data Plane Agent applies it">
    The agent applies the custom resource on the Data Plane cluster.
  </Step>

  <Step title="The Distributed Compute Operator provisions the cluster">
    The DCO creates a nodeset resource, from which the Slurm controller creates up to the user's maximum number of worker pods, initially scaled to zero. It creates a token resource, from which the Slurm controller creates a JWT secret for the execution. It then adds the launching user to the `slurm-users` ConfigMap.
  </Step>
</Steps>

## Prerequisites

* Slurm enabled on at least one Data Plane in your deployment. Slurm is enabled when the Data Plane is deployed and cannot be added afterward, so contact Domino Support to enable it.

* A [Dataset](/cloud/platform-capabilities/core-concepts/data/datasets) configured on the Slurm-enabled Data Plane. See [Configure a shared Dataset](#configure-a-shared-dataset).

## Create the Compute Environments

Slurm requires two [Compute Environments](/cloud/platform-capabilities/core-concepts/compute-environments): one with the Slurm CLI installed, which users select for the Workspace itself, and one that runs on the Slurm worker nodes. Create both by [creating an Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/3-create-an-environment) with the attributes below. Get the current image tags from the [Compute Environment catalog](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/8-compute-environment-catalog).

### Workspace Compute Environment

* **Environment Base**: **Start from a custom base image**, then enter the Slurm Workspace image.

  ```
  quay.io/domino/domino-slurm-environment:<tag>
  ```

* **Supported Clusters**: **None**. This Environment runs the Workspace, not the workers.

* **Pluggable Workspace Tools**: add the workspace tool definitions.

  ```yaml theme={null}
  jupyter:
    title: "Jupyter (Python, R, Julia)"
    iconUrl: "/assets/images/workspace-logos/Jupyter.svg"
    start: [ "/opt/domino/workspaces/jupyter/start" ]
    supportedFileExtensions: [ ".ipynb" ]
    httpProxy:
      port: 8888
      rewrite: false
      internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
      requireSubdomain: false
  jupyterlab:
    title: "JupyterLab"
    iconUrl: "/assets/images/workspace-logos/jupyterlab.svg"
    start: [ "/opt/domino/workspaces/jupyterlab/start" ]
    httpProxy:
      internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
      port: 8888
      rewrite: false
      requireSubdomain: false
  vscode:
    title: "vscode"
    iconUrl: "/assets/images/workspace-logos/vscode.svg"
    start: [ "/opt/domino/workspaces/vscode/start" ]
    httpProxy:
      port: 8888
      requireSubdomain: false
  rstudio:
    title: "RStudio"
    iconUrl: "/assets/images/workspace-logos/Rstudio.svg"
    start: [ "/opt/domino/workspaces/rstudio/start" ]
    httpProxy:
      port: 8888
      requireSubdomain: false
  ```

### Cluster Compute Environment

* **Environment Base**: **Start from a custom base image**, then enter the Slurm cluster image.

  ```
  quay.io/domino/slurm-cluster-environment:<tag>
  ```

* **Supported Clusters**: **Slurm**.

<Note>
  The **Slurm** option in the Workspace launcher is enabled only when at least one Environment carrying the Slurm supported-cluster label is available to the user. Make both Environments visible to your intended users or their Organizations.
</Note>

## Create worker Hardware Tiers

Users select the worker tier that best fits their jobs when they attach a Slurm cluster to a Workspace. A typical deployment defines a small set of [Hardware Tiers](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers) matched to common workload profiles:

| Hardware Tier | Workload profile | Backing Kubernetes nodes |
| :- | :- | :- |
| Slurm Compute | CPU-bound: simulations and parameter sweeps | Standard nodes with a high CPU-to-memory ratio |
| Slurm Big Memory | Memory-bound: large in-memory models and datasets | High-memory nodes |
| Slurm GPU | GPU-bound: training, GPU-accelerated simulation | GPU-enabled nodes |

For predictable scheduling, size each Hardware Tier to match its underlying Kubernetes node. That avoids one Kubernetes node hosting several Slurm workers, and a Slurm worker sharing a node with unrelated workloads.

When sizing, leave headroom for the small per-node overhead that infrastructure `DaemonSets` add. For a Kubernetes node with 8 CPUs and 32 GiB of memory, for example, a worker Hardware Tier of 6 CPUs and 28 GiB works well.

## Configure a shared Dataset

Slurm has no internal mechanism for copying files between the Workspace and the worker nodes, so data is exchanged through a shared network file system. In Domino that is a [Dataset](/cloud/platform-capabilities/core-concepts/data/datasets), which makes configuring one on the Slurm-enabled Data Plane essential rather than optional.

* **Local Data Plane**: configured by default at `/domino/datasets/local/<project-name>`.

* **Remote Data Plane**: enable Data Plane storage and Workspaces on the Data Plane. See [Manage Data Planes](/cloud/admin/platform-configuration/manage-data-planes).

## Set scheduling policies and limits

Configure Slurm's supported scheduling policies and limits, including priority, fair-share, and per-user and per-Project limits on queued or running jobs, directly in Slurm with the standard Slurm administration tools. The entry point for CLI-based Slurm configuration is a Workspace on the Slurm-enabled Data Plane. The [Slurm documentation](https://slurm.schedmd.com/) covers the available policy and limit options.

## Enable HPC Job submission from the Jobs page

The **HPC** tab on a Project's Jobs page carries the history of the Project's HPC Jobs, and lists them no matter how they were submitted. Submission from that tab is gated behind the `ShortLived.EnableHPCJobs` feature flag. Set it to `true` and a **Run** button appears on the tab, letting users submit HPC Jobs from the Domino UI and the API rather than from a Workspace terminal. Domino ships the gate because the submission experience is still being refined. Submission from a Workspace terminal is unaffected and needs no flag.

## Map Domino users to Slurm users

Each Slurm user name derives from the name of the Domino user who launched the Workspace, so Slurm accounting, fair-share, and per-user limits apply per person rather than to one shared account.

A Domino user name has to be a valid Linux user name for this mapping to work: a lowercase letter or underscore followed by up to 31 lowercase letters, digits, underscores, or hyphens, matching `^[a-z_][a-z0-9_-]{0,31}`. It must also not be a reserved system name such as `root`, `daemon`, or `nobody`.

<Warning>
  Launching a Workspace with an attached Slurm cluster fails when the Domino user name breaks either rule. Check user names against both before you make Slurm available to a group of users.
</Warning>

A Domino user holding the SysAdmin role maps instead to Slurm's built-in `slurm` user, which is a Slurm system administrator. The [SysAdmin role](/cloud/admin/identity-and-access/manage-users/roles) is not available on Domino Cloud, so this mapping applies only to self-managed deployments.

## Limitations

* A Slurm cluster is attached at Workspace launch, and the Workspace is the only place the Slurm CLI runs. A standard Domino Job cannot carry an attached Slurm cluster.

* Nodesets and partitions are created per Workspace, so each Workspace gets its own dedicated nodeset.

* Data is exchanged between the Workspace and the workers only through Datasets, over NFS. Workers cannot reach [Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors) or [NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes).

* Integration with external, customer-managed Slurm clusters is not supported.

* Slurm cannot be added to a Data Plane that is already deployed.

<Warning>
  Jobs on worker nodes are not sandboxed from one another. Jobs from different users can run on the same worker node, and all Domino users share a single Unix group there, so files written with group-readable or group-writable permissions are visible to other users. Tell users to set a restrictive `umask` on sensitive output files.
</Warning>

## Known issues

### Slurm REST API memory growth

Broad `sacct` queries, such as `sacct -a`, run after high job volume cause sustained memory growth in the Slurm REST API. Memory is not promptly released, which can lead to out-of-memory restarts of the service.

To mitigate it, tell users to filter `sacct` queries by job ID, user, or time range rather than querying all jobs. If the service restarts repeatedly, contact Domino Support.

## Troubleshooting

<AccordionGroup>
  <Accordion title="The Slurm option is disabled in the Workspace launcher">
    Two conditions must both hold: at least one Environment labeled with the **Slurm** supported-cluster type is available to the user, and the Data Plane they selected has Slurm installed.
  </Accordion>

  <Accordion title="The Slurm option is disabled when launching a Job">
    Standard Jobs do not support attached Slurm clusters. Users submit HPC Jobs from a Workspace terminal, or from the **HPC** tab of the Project's Jobs page once you turn on the flag described in [Enable HPC Job submission from the Jobs page](#enable-hpc-job-submission-from-the-jobs-page).
  </Accordion>

  <Accordion title="Jobs never dispatch">
    Confirm that the requested resources fit within the worker Hardware Tier and the maximum number of workers set on the cluster. If workers are never created at all, contact Domino Support.
  </Accordion>
</AccordionGroup>

## Related

* [Run HPC workloads with Slurm](/cloud/platform-capabilities/features/compute-clusters-hpc/slurm): the practitioner-facing guide to submitting HPC Jobs.
* [Manage Hardware Tiers](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers): create and size the worker tiers.
* [Manage Data Planes](/cloud/admin/platform-configuration/manage-data-planes): register a Data Plane and enable it for storage and Workspaces.
* [Datasets](/cloud/platform-capabilities/core-concepts/data/datasets): the shared storage Slurm jobs read and write.


# Domino infrastructure
Source: https://docs.domino.ai/cloud/admin/infrastructure-and-compute/index

Manage Hardware Tiers, node pools, scaling, queue limits, persistent volumes, accelerators, and Slurm resources for Domino workloads.

[Manage Compute Resources](/cloud/admin/infrastructure-and-compute/manage-compute-resources)\
[Scale compute capacity](/cloud/admin/infrastructure-and-compute/manage-compute-resources/scale-compute-capacity), [set execution queue limits](/cloud/admin/infrastructure-and-compute/manage-compute-resources/execution-queue-limits), [view platform and compute nodes details](/cloud/admin/infrastructure-and-compute/manage-compute-resources/view-details-of-platform-and-compute-nodes), and [use Intel Habana accelerators](/cloud/admin/infrastructure-and-compute/manage-compute-resources/use-intel-habana-accelerators) to make sure your users have enough compute power to run Domino clusters.

[Manage node pools](/cloud/admin/infrastructure-and-compute/manage-compute-resources/node-pools)\
View, add, and delete node pools, and see how many hardware tiers use each one.

[Manage Hardware Tiers](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers)\
Create, edit, archive, and tune Hardware Tiers that define the compute resources available to Domino executions and endpoints.

[Administer Slurm](/cloud/admin/infrastructure-and-compute/administer-slurm)\
Set up and operate Slurm on a Data Plane so your users can run batch HPC Jobs from a Workspace.

[Manage Persistent Volumes](/cloud/admin/infrastructure-and-compute/manage-compute-resources/manage-persistent-volumes)\
Manage the Kubernetes persistent volumes that back Domino Jobs and Workspaces, including storage cost, performance, and durability.


# Configure spot instances
Source: https://docs.domino.ai/cloud/admin/infrastructure-and-compute/manage-compute-resources/configure-spot-instances

Run fault-tolerant workloads on spot capacity by mapping spot node pools to Hardware Tiers and configuring on-demand fallback.

Spot instances reduce compute costs by using discounted cloud capacity. They’re ideal for fault-tolerant, stateless, or short-lived workloads like batch jobs, distributed training runs, and parallel analytics tasks.

Your Domino administrator can configure spot instances by creating node pools with spot instance types and mapping them to hardware tiers. Users can then select hardware tiers marked with the **Spot** tag to run their workloads on spot capacity.

Domino uses infrastructure services like Karpenter or EKS to manage spot capacity. When matching spot and on-demand node pools is configured, Domino handles capacity issues automatically:

* If spot capacity isn’t available when a user submits a workload, the infrastructure provisions it on on-demand capacity.

* If a running spot workload is interrupted, Domino retries it on available spot capacity (which may use a different instance type) or falls back to on-demand infrastructure.

Without a matching on-demand node pool, workloads fail when spot capacity is unavailable.

## Prerequisites

You’ll need to understand how to create [hardware tiers](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/manage-hardware-tier) in Domino.

## Step 1: Create a node pool with spot instances

1. From the **Admin Panel**, click **Platform settings** > **Deployment Configuration** > **Create node pool**.

2. Fill out the node pool details.

3. Select **Support Spot**.

## Step 2: Create the hardware tier

Map your node pool to a hardware tier so users can select it.

1. Create a new hardware tier for the target data plane.

2. In the Node Pool field, enter the value you set for `dominodatalab.com/node-pool` in your node pool configuration.

   * For example, use the value `flex` if you used `dominodatalab.com/node-pool:flex` when you created the node pool.

3. Select **Configured with spot instance support (capacity type: spot)**.

Once configured, the infrastructure determines whether spot or on-demand instances are more economical at the time of provisioning. Spot is generally preferred when available.

If spot capacity is interrupted or unavailable, the infrastructure automatically falls back to the on-demand node pool if one exists with the same name and the infrastructure supports automatic fallback.

## Best practices

* **Use multiple instances per node pool** - Include several similar instance types to increase the chances of successful spot allocation and improve availability. Don’t rely on a single instance type. You can also configure your node pool to use any available instance type by not specifying instance sizes or families. To avoid expensive instance types, use exclusion rules to restrict specific types while allowing all others.

* **Check the AWS Spot Instance Advisor** - Review typical interruption frequencies by instance type and region. This helps you select more stable spot instances.

* **Distribute across Availability Zones** - Spot capacity varies by zone. Be flexible about where workloads run to reduce allocation failures.

* **Use the capacity-optimized allocation strategy** - This strategy helps auto-scaling groups select Spot pools with the most available capacity, reducing interruptions.

* **Enable proactive capacity rebalancing** - Proactive rebalancing adds replacement Spot instances before termination notices are issued, improving workload stability.

* **Keep spot settings consistent** - If spot is enabled on a node pool, make sure the corresponding HWT also has Spot enabled, and vice versa.

## Known issues and limitations

* **EBS volumes are tied to Availability Zones** - If a node pool uses spot instances across multiple Availability Zones and a spot instance is interrupted, workloads with attached EBS volumes can’t restart in a different zone. The EBS volume remains in the original zone.

If you encounter this issue, switch to a hardware tier that uses a non-spot node pool.

## Next steps

* [Create a hardware tier](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/manage-hardware-tier) in Domino.

* [Scale compute capacity](/cloud/admin/infrastructure-and-compute/manage-compute-resources/scale-compute-capacity) explains how Domino relies on Kubernetes to autoscale environments for executions to run on compute resources.


# Execution queue limits
Source: https://docs.domino.ai/cloud/admin/infrastructure-and-compute/manage-compute-resources/execution-queue-limits

Limit queued execution slots globally or per user, and understand how Workspaces, Jobs, Apps, and distributed clusters consume them.

To prevent too many queued executions from overwhelming a Domino deployment, limit the number of execution slots that can be queued globally or per user. To do this, please reach out to Domino Support.

Once a queue limit is reached, additional executions are rejected with a failure status. The count includes executions for Domino workspaces, jobs, web applications, and any executions that make up an on-demand distributed compute cluster. For example, an on-demand Spark cluster consumes an execution slot for the master and each Spark executor.


# Advanced hardware tier settings
Source: https://docs.domino.ai/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/advanced-hardware-tier-settings

Advanced pod-level customization options for hardware tiers, for administrators with specialized resource or compute cluster requirements.

The following advanced hardware tier settings provide you more control:

## Restrict to a compute cluster

You can dynamically orchestrate on-demand compute clusters on top of the Domino managed infrastructure. These workloads often have unique resource requirements, and you might want to create dedicated hardware tiers that are available only for compute cluster workloads. In this case, use the **Restrict to compute cluster** setting to specify the compute cluster types, such as Spark or Ray, that must exclusively use a given hardware tier.

When configuring resources for a cluster, the options are limited to hardware tiers where the **Restrict to compute cluster** setting is selected for the cluster type. Hardware tiers configured this way are not available when selecting resources outside of cluster configuration.

<Note>
  For a given cluster type, hardware tier choices are filtered only after at least one restricted hardware tier is configured.
</Note>

## Maximum simultaneous executions

If you want to limit the deployment-wide capacity for a hardware tier, but you do not want to create a dedicated node pool for the hardware tier, you can specify the **Maximum Simultaneous Executions**.

This ensures that no more than the specified number of executions can use the selected hardware tier at the same time. Additional executions beyond the limit are queued.

## Overprovision pods

On cloud deployments enabled for autoscaling, when a capacity request is made, new nodes are provisioned. This option minimizes cost, but provisioning a new node can take several minutes causing users to wait. The situation can be particularly time-consuming in the mornings when many users first log onto a system that has scaled down overnight.

You can address this problem by overprovisioning several warm slots for popular hardware tiers. Domino automatically pre-provisions nodes that might be necessary to accommodate the specified number of overprovisioned executions using this hardware tier. This minimizes the chance that a user must wait for a new node to spin up.

To keep costs under control, you can apply overprovisioning on a scheduled basis for periods when many new users are expected.

To do this:

1. Set the number of **Overprovisioning Pods**.

2. Select the **Enable Overprovisioning Pods On a Schedule** checkbox.

3. Specify the schedule as needed.

## Use a custom GPU resource name

By default, Domino requests GPU resources of type `nvidia.com/gpu`. This works well for most NVIDIA GPU-enabled devices, but when your deployment is backed by different GPU devices (for example, NVIDIA MIG GPUs, AMD GPUs, AWS vGPUs, or Xilinx FPGA), you must use a different name for the GPU resources.

Select **Use custom GPU resource name** and enter the GPU resource name that corresponds to the name of the GPU devices being discovered and reported by Kubernetes.

For example, with an NVIDIA A100 GPU configured in MIG Mixed Mode, you can use resources like `nvidia.com/mig-1g.5gb`, `nvidia.com/mig-2g.10gb`, or `nvidia.com/mig-3g.20gb`.

## Allow executions to exceed the default shared memory limit

You can allow hardware tiers to exceed the default limit of 64 MB for shared memory. This is especially beneficial for applications that can use shared memory.

To enable this, select the **Allow executions to exceed the default shared memory limit** checkbox. This overrides the `/dev/shm` (shared memory) limit, and any shared memory consumption counts toward the overall memory limit of the hardware tier.

Incorporate the size of `/dev/shm` in any memory usage calculations for a hardware tier with this option enabled. If this option is enabled, you must input the shared memory limit. For example, suppose the hardware tier has an overall memory limit of 4 GB and you want to limit shared memory to 2 GB, then you should input "2.0" for the shared memory limit.

Setting a shared memory limit greater than the overall memory limit allows usage of shared memory only up to the overall limit. For example, if you set a shared memory limit of 6 GB when the hardware tier has an overall memory limit of 4 GB, an execution on the hardware tier can still only use up to 4 GB of shared memory.

<Warning>
  `/dev/shm` is considered part of the overall memory footprint of an execution container. If you allow the hardware tier to exceed the default limit of 64 MB shared memory, be sure that the container’s shared memory usage plus regular memory usage is below the overall memory limit, or Kubernetes terminates the container.
</Warning>

## Advanced pod customization

When creating or editing a hardware tier in the Admin UI, there is a small **Advanced** link near the bottom of the form. Clicking this link reveals text inputs for the following attributes:

* Pod resource requests

* Pod resource limits

* Pod annotations

* Pod labels

* Hugepages

* Capabilities

These attributes apply to all Execution pods, including Workspaces, Jobs, and Apps, as well as Compute Cluster pods. However, they do not apply to Domino Endpoint pods.

<Note>
  These attributes cannot override any existing Domino-managed fields and are ignored.
</Note>

<AccordionGroup>
  <Accordion title="Pod resource requests / Pod resource limits">
    Resource requests and limits are applied to the `run` container of Execution pods and the master/worker containers of Compute Cluster pods. The inputs should consist of newline-delimited key-value pairs in the form `key: value`.

    For example:

    ```yaml theme={null}
    ephemeral-storage: 10Gi
    ```

    <Note>
      Administrators should not specify any CPU or memory requests/limits here, as these values are overridden by the existing form inputs: `Cores Requested`, `Cores Limit`, `Memory Requested (GiB)`, `Memory Limit (GiB)`. The same applies to GPU resources and hugepages.
    </Note>
  </Accordion>

  <Accordion title="Pod annotations">
    Annotations are applied to all Execution pods and Compute Cluster pods. The input should consist of newline-delimited key-value pairs in the form `key: value`.

    For example:

    ```yaml theme={null}
    prometheus.io/scrape: "true"
    prometheus.io/path: "/metrics"
    prometheus.io/port: "9090"
    ```

    This translates to the following pod specification:

    ```yaml theme={null}
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
        prometheus.io/port: "9090"
    ```
  </Accordion>

  <Accordion title="Pod labels">
    Labels are applied to all Execution pods and Compute Cluster pods. The input should consist of newline-delimited key-value pairs in the form `key: value`.

    For example:

    ```yaml theme={null}
    example.com/application: my-app
    example.com/environment: production
    ```

    This translates to the following pod specification:

    ```yaml theme={null}
    metadata:
      labels:
        example.com/application: my-app
        example.com/environment: production
    ```
  </Accordion>

  <Accordion title="Hugepages">
    Hugepages are applied to the `run` container of Execution pods and the master/worker containers of Compute Cluster pods. The input should consist of a key-value pair in the form `page_size: total_size`.

    For example:

    ```yaml theme={null}
    2Mi: 100Mi
    ```

    This translates to the following pod specification:

    ```yaml theme={null}
        containers:
          resources:
            limits:
              hugepages-2Mi: 100Mi
            requests:
              hugepages-2Mi: 100Mi
          volumeMounts:
          - mountPath: /hugepages-2Mi
            name: hugepages-2mi
        volumes:
        - name: hugepages-2mi
          emptyDir:
            medium: HugePages-2Mi
    ```
  </Accordion>

  <Accordion title="Capabilities">
    Capabilities are applied to the `run` container of Execution pods and the master/worker containers of Compute Cluster pods. The input should consist of a newline-delimited list of capabilities.

    For example:

    ```text theme={null}
    IPC_LOCK
    NET_BIND_SERVICE
    ```

    This translates to the following pod specification:

    ```yaml theme={null}
        securityContext:
          capabilities:
            add:
            - IPC_LOCK
            - NET_BIND_SERVICE
    ```

    <Note>
      In order for the capabilities to be added, the Configuration record `com.cerebro.domino.computegrid.kubernetes.nonRootExecutions.enabled` must not be set to `true`.
    </Note>
  </Accordion>
</AccordionGroup>


# Hardware tier best practices
Source: https://docs.domino.ai/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/hardware-tier-best-practices

Recommendations for administrators configuring hardware tier requests and limits to avoid resource contention and pod eviction.

Domino Hardware Tiers define Kubernetes requests and limits and link them to specific node pools. Domino recommends the following best practices:

* Account for overhead

* Isolate workloads and users using node pools

* Isolate compute cluster workloads

* Set resource requests and limits to the same values

## Account for overhead

When designing hardware tiers, consider what resources are available on a given node when Domino submits your workload for execution. Not all physical memory and CPU cores of your node are available due to system overhead.

Consider the following overhead components:

1. Kubernetes management overhead

2. Domino daemon-set overhead

3. Domino execution sidecar overhead

Overhead is relevant if you want to define a hardware tier dedicated to one execution at a time per node, such as for a node with a single physical GPU. It is also relevant if you absolutely must maximize node density.

## Kubernetes management overhead

Kubernetes typically reserves a portion of each node’s capacity for daemons and pods that are required for Kubernetes itself. The amount of reserved resources usually scales with the size of the node, and also depends on the Kubernetes provider or distribution.

See the following for information about reserved resources by cloud-provider managed Kubernetes providers:

* [AWS EKS](https://awslabs.github.io/amazon-eks-ami/usage/al2/)

* [Azure AKS](https://docs.microsoft.com/en-us/azure/aks/concepts-clusters-workloads#resource-reservations)

* [Google GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture#node_allocatable)

### Get available resources for your instance

1. Run the following command to check one of your compute nodes:

   ```shell theme={null}
   kubectl describe nodes
   ```

2. Find the `Allocatable` section of the output. This shows the memory and CPU available for Domino.

## Domino daemon-set overhead

Domino runs a set of management pods that reside on each of the compute nodes. Use these for log aggregation, monitoring, and environment image caching.

The overhead of these daemon-sets is roughly 0.5 CPU core and 0.5 Gi RAM. This overhead is taken from the allocatable resources on the node.

### Domino execution overhead

For each Domino execution, there is a set of supporting containers in the execution pod that manages authentication, handling request routing, loading files, and installing dependencies. These supporting containers make CPU and memory requests that Kubernetes considers when scheduling workspace, job, and app pods.

The supporting container overhead currently is roughly 1 CPU core and 1.5 GiB RAM. You can configure this so it might vary for your deployment.

**Example:**

Consider an `m5.2xlarge` EC2 node with a raw capacity of 8 CPU cores and 32 GiB of RAM.

When used as part of an EKS cluster, the node reports the following allocatable capacity of \~27GiB of RAM and 7910m CPU cores.

```yaml theme={null}
Capacity:
  attachable-volumes-aws-ebs:  25
  cpu:                         8
  ephemeral-storage:           104845292Ki
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      32120476Ki
  pods:                        58
Allocatable:
  attachable-volumes-aws-ebs:  25
  cpu:                         7910m
  ephemeral-storage:           95551679124
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      28372636Ki
  pods:                        58
```

Also, account for 500m CPU and 0.5GiB RAM for the Domino and EKS daemons.

Lastly, for a single execution add 1000m CPU and 1.5GiB RAM for sidecars, and you are left with roughly 6410m CPU and 25GiB RAM that you can use for a single large hardware tier.

If you want to partition the node into smaller hardware tiers, you must account for the sidecar overhead for every execution that you want to co-locate.

As a general rule, larger nodes allow for more flexibility as Kubernetes takes care of efficiently packing your executions onto the available capacity.

**See which pods are running on a specific node:**

1. In the Admin application, click **Infrastructure**.

2. Click the name of a node. In the following image, there is a box around the execution pods. The other pods handle logging, caching, and other services.

<img alt="execution pod resource utilization" />

## Isolate workloads and users using node pools

Define node pools:

1. Go to **Advanced > Hardware Tiers**.

2. Create or edit a hardware tier.

3. In the **Node Pool** field, enter `your-node-pool` which must match the node pool label such as: `dominodatalab.com/node-pool=<your-node-pool>`. You can name a node pool anything you like, but Domino recommends naming them something meaningful given the intended use.

Domino typically comes pre-configured with `default` and `default-gpu` node pools, with the assumption that most user executions run on nodes in one of those pools. As your compute requirements become more sophisticated, you might want to keep certain users separate from one another or provide specialized hardware to certain groups of users.

For example, if there’s a data science team in New York City that needs a specific GPU machine that other teams don’t need it, you can use the following label for the appropriate nodes: `dominodatalab.com/node-pool=nyc-ds-gpu`. In the hardware tier form, you would specify `nyc-ds-gpu`. To ensure only that team has access to those machines, create a `NYC` organization, add the correct users to the organization, and give that organization access to the new hardware tier that uses the `nyc-ds-gpu` node pool label.

## Isolate compute cluster workloads

Domino on-demand compute clusters often require pooling a large amount of compute resources on specialized hardware (for example, using larger nodes compared to your other workloads).

Consider a use case where you want a set of extra large nodes to be available for on-demand Ray workloads but not available for regular workloads.

1. In the admin panel, go to **Manage Resources** > **Hardware Tiers**.

2. Create or edit a hardware tier.

3. From **Restrict to compute cluster**, select the **Ray** checkbox.

For clusters that are comprised of multiple workload types (for example, Ray head and worker), Domino recommends that you create a separate dedicated Hardware Tier that match the requirements for each type. In the Ray example, you can create `ray-head` and `ray-worker` Hardware Tiers.

## Set resource requests and limits to the same values

With Kubernetes, resource limits must be greater than or equal to resource requests. So, if your memory request is 16 GiB, your limit must be greater than or equal to 16 GiB. But, although setting a request greater than limits can be useful, and there are cases where allowing bursts of CPU or memory can be useful, this is also dangerous. Kubernetes might evict a pod using more resources than initially requested. For Domino workspaces or jobs, this would cause the execution to be terminated.

For this reason, Domino recommends setting memory and CPU requests equal to limits. In this case, Python and R cannot allocate more memory than the limit, and execution pods are not evicted.

On the other hand, if the limit is higher than the request, a user can use resources that another user’s execution pod must be able to access. This is the noisy neighbor problem that you might have experienced in other multi-user environments. But, instead of allowing the noisy neighbor to degrade performance for other pods on the node, Kubernetes evicts offending pods when necessary to free up resources.

User data on disk is not lost, because Domino stores user data on a persistent volume that can be reused. But, anything in memory is lost and the execution must be restarted.


# Manage hardware tiers
Source: https://docs.domino.ai/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/index

Create, edit, archive, and tune Hardware Tiers that define the compute resources available to Domino executions and endpoints.

Domino allows you to create and manage hardware tiers that your users use to host executions.

[Manage a hardware tier](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/manage-hardware-tier)\
Information about how to create, edit, and archive hardware tiers, and specify hardware tiers as Domino endpoints.

[Advanced settings](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/advanced-hardware-tier-settings)\
Information about advanced hardware tier settings that give you more control.

[Best practices](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/hardware-tier-best-practices)\
Domino’s recommendations on best practices for when you work with hardware tiers.


# Manage a hardware tier
Source: https://docs.domino.ai/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/manage-hardware-tier

Create, edit, and archive hardware tiers.

## Overview

Hardware tiers let you define the CPU, memory, and GPU resources available to a Domino execution, so users can pick the right amount of compute without managing the underlying infrastructure themselves. Use this page to create a hardware tier when you need a new resource combination, edit one as your cluster or team's needs change, and archive one you no longer want available for selection.

## Create a Hardware Tier

Users select a hardware tier when launching an execution, which determines the available resources and the node type for the execution.

1. Go to **Manage Resources** > **Hardware Tiers**.

2. Click **New**.

3. Set the following fields to configure the resource requests and limits for execution pods:

   * **Name**\
     The name of the hardware tier that shows to users.

     <Note>
       Name each hardware tier as descriptively as possible so users can identify the underlying compute resources, for example `GPU (g5.4xlarge)`.
     </Note>

   * **Architecture**\
     The CPU architecture used by executions on this hardware tier. Choose `amd64` or `arm64`. New hardware tiers default to `amd64`, and tiers created before this field existed keep the `amd64` default. Set the value to match the architecture of the nodes in the tier's node pool, so Domino schedules executions onto compatible nodes.

     The field is optional on the public hardware tier API at `/api/hardwaretiers/v1/hardwaretiers`. Requests that omit `architecture` leave the stored value unchanged. Domino rejects values other than `amd64` or `arm64`.

   * **Cores Requested**\
     The number of requested CPUs.

   * **Memory Requested (GiB)**\
     The amount of requested memory.

   * **Memory Limit (GiB)**\
     The maximum amount of memory. Domino recommends that this is the same as the request.

     If **Allow executions memory limit to exceed request** is not selected, Domino automatically applies a memory limit equal to the request.

     <Warning>
       Use the option to allow the memory limit to exceed the request with caution since it can make executions more likely to be evicted under memory pressure. See [the Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/) to learn how memory requests and limits influence Kubernetes eviction decisions.
     </Warning>

   * **Cores Limit**\
     The maximum number of CPUs. Domino recommends that this is the same as the request.

     If **Allow executions to exceed request when unused CPU is available** is not selected, Domino automatically applies a CPU limit equal to the request.

   * **Number of GPUs**\
     The number of requested GPUs. The request values, CPU Cores, Memory, and GPUs, are thresholds used to determine whether a node has the capacity to host an execution pod. These requested resources are effectively reserved for the pod. The limit values control the amount of resources a pod can use above and beyond the amount requested. If there’s additional headroom on the node, the pod can use resources up to this limit.

     However, if resources are in contention, and a pod is using resources beyond those it requested and thereby causing excess demand on a node, the offending pod might be evicted from the node by Kubernetes, and the associated Domino Run is terminated. For this reason, Domino recommends that you set the requests and limits to the same values.

     Ensure that the CPU, memory, and GPU requests in your hardware tier do not exceed the available resources in the target node pool, considering overhead. Otherwise, an execution using such a hardware tier never starts. If you need more resources than are available on existing nodes, you might have to add a new node pool with different specifications. This might mean adding individual nodes to a static cluster or configuring new auto-scaling components that provision new nodes with the required specifications and labels.

   * **Cents Per Minute Per Run**\
     Enter how much it costs (in cents) per minute to run each machine in the hardware tier. You might notice that when users select a hardware tier, they see the cost in dollars. The cost is converted from cents to dollars (rounded to the fourth floating digit).

   * **Node Pool**\
     Specify the underlying machine type on which a Domino execution runs.

     Nodes that have the same value for the `dominodatalab.com/node-pool` [Kubernetes node label](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/) form a **Node Pool**. Executions with a matching value in the **Node Pool** field then run on these nodes. [Configure spot instances](/cloud/admin/infrastructure-and-compute/manage-compute-resources/configure-spot-instances) has information on creating node pools with spot instances.

     For example, you can set a hardware tier named `large-k8s` to use the `default` node pool.

     The following diagram shows a cluster configured with two node pools for Domino, one named `default` and one named `default-gpu`. You can label node pools with the same scheme `dominodatalab.com/node-pool=<node-pool-name>` to make additional node pools available to Domino. The arrows in the diagram represent Domino requesting that a node with a given label be assigned to an execution. Kubernetes then assigns the execution to a node in the specified pool that has sufficient resources.

     <img alt="Node pools" />

     By default, Domino creates a node pool with the label `dominodatalab.com/node-pool=default` and all compute nodes Domino creates in cloud environments are assumed to be in this pool. In cloud environments with automatic node scaling, you configure scaling components like AWS Auto Scaling Groups or Azure Scale Sets with these labels to create elastic node pools.

   * Data Plane\
     In a [Domino Nexus](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture) deployment, the form also carries a dropdown for the Data Plane this hardware tier runs workloads on. Each hardware tier is assigned to one Data Plane, so this dropdown decides where executions using the tier run.

     Confirm that the resources the tier requests are available on the nodes in the chosen Data Plane, that those nodes carry the `dominodatalab.com/node-pool` label, and that the label value matches the **Node Pool** you set above.

4. Click **Insert**.

<Tip>
  To add a tier for a Data Plane you just registered, go to **Manage resources** > **Hardware tiers**, click **Clone** next to an existing tier, and change its Data Plane, rather than building one from scratch. Cloning a "small" tier is an easy way to test a new Data Plane.
</Tip>

<Note>
  See [Advanced hardware tier settings](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/advanced-hardware-tier-settings)
</Note>

## Restrict access to a Hardware Tier

Clearing **Is Globally Available** limits who can run on the resources a hardware tier defines, and in a Domino Nexus deployment that also limits who can run workloads on its Data Plane.

1. In the admin portal, go to **Manage resources** > **Hardware tiers**.

2. Next to the hardware tier, click **Edit**.

3. Clear **Is Globally Available** at the bottom of the form.

4. Go to the **Hardware tiers** tab on the organization's page and select the restricted hardware tier.

## Define a Hardware Tier as a Domino Endpoint

You can define specific hardware tiers for machine learning models deployed as [Domino endpoints](/cloud/admin/platform-configuration/manage-model-deployments#domino-endpoint-hardware-tiers). This allows you to cater your hardware tiers to the unique demands of machine learning model deployment.

Domino endpoint tiers and regular hardware tiers cannot be interchanged, meaning Domino endpoints can only use Domino endpoint tiers, and Domino endpoint tiers cannot be used outside of Domino endpoints.

To learn more, see [Scale models with Domino endpoint hardware tiers](/cloud/admin/platform-configuration/manage-model-deployments#domino-endpoint-hardware-tiers).

## Edit a Hardware Tier

1. In the admin panel, go to **Manage Resources** > **Hardware Tiers**.

2. Click **Edit**.

3. Make the changes needed and click **Update**.

## Archive a Hardware Tier

<Warning>
  An archived hardware tier cannot be un-archived.
</Warning>

1. In the admin panel, go to **Manage Resources** > **Hardware Tiers**.

2. Click **Archive** next to the hardware tier you want to archive.

3. Click **Archive Hardware Tier**.


# Manage compute resources
Source: https://docs.domino.ai/cloud/admin/infrastructure-and-compute/manage-compute-resources/index

Manage Domino compute resources: spot instances, autoscaling, execution queue limits, platform and compute node details, and Habana accelerators.

[Configure spot instances](/cloud/admin/infrastructure-and-compute/manage-compute-resources/configure-spot-instances)\
Learn how to create node pools with spot instances and mapping them to hardware tiers.

[Scale Compute Capacity](/cloud/admin/infrastructure-and-compute/manage-compute-resources/scale-compute-capacity)\
Learn how Domino relies on Kubernetes to autoscale environments for executions to run on compute resources.

[Execution Queue Limits](/cloud/admin/infrastructure-and-compute/manage-compute-resources/execution-queue-limits)\
You can limit the number of execution slots that can be queued to prevent queued executions from overwhelming a Domino deployment.

[View Details of Platform and Compute Nodes](/cloud/admin/infrastructure-and-compute/manage-compute-resources/view-details-of-platform-and-compute-nodes)\
View a list and the details of the current platform and compute nodes in your compute grid.

[Use Intel Habana Accelerators](/cloud/admin/infrastructure-and-compute/manage-compute-resources/use-intel-habana-accelerators)\
Add Habana Gaudi accelerators to a hardware tier.


# Manage persistent volumes
Source: https://docs.domino.ai/cloud/admin/infrastructure-and-compute/manage-compute-resources/manage-persistent-volumes

Manage the Kubernetes persistent volumes that back Domino Jobs and Workspaces, for admins tuning storage cost, performance, and durability.

## Overview

Persistent volumes (PVs) let Domino give each Job and Workspace fast local storage for project files, instead of reading and writing directly to the Domino blob store on every operation. As an administrator, you manage how these volumes are provisioned, reused, backed up, and cleaned up, so that your cluster balances storage cost against the performance benefit of reusing volumes.

## How it works

When not in use, Domino project files are stored and versioned in the Domino blob store. When a Domino execution is started from a project, the project files are copied to a Kubernetes persistent volume (PV) that is attached to the compute node and mounted in the Run.

You can run the following command to see all [current PVs](https://kubernetes.io/docs/concepts/storage/persistent-volumes/), sorted by last-used:

```shell theme={null}
kubectl get pv --sort-by='.metadata.annotations.dominodatalab\.com/last-used'
```

## Storage workflow for Jobs

When a user starts a new Job, Domino assigns a new execution pod to the cluster. This pod has an associated [Persistent Volume Claim (PVC)](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims) which defines what type of storage it requires from Kubernetes.

If an idle PV exists that matches the PVC, Kubernetes mounts that PV on the node it assigns to host the pod, and the Job starts. If an appropriate idle PV does not exist, Kubernetes creates a new PV according to the [storage class](https://kubernetes.io/docs/concepts/storage/storage-classes/).

When the user completes their Job, the PV data is written to the Domino File System, and the PV unmounts and sits idle until it is either reused for the user's next Job or until [garbage collection](#garbage-collection). By reusing PVs, users who are actively working in a project do not have to repeatedly copy data from the blob store to a PV.

A Job only matches with a fresh PV or one previously used by that project. PVs are not reused between projects.

### Garbage collection

<Note>
  This feature applies to Idle PVs used for jobs and scheduled jobs, not durable workspaces.
</Note>

Domino has configurable values to help you tune your cluster to balance performance with cost controls. The more idle volumes you allow, the more likely it is that users can reuse a volume and avoid having to copy project files from the blob store. However, this comes at the cost of keeping additional idle PVs.

By default, Domino does the following:

* Limit the total number of idle PVs to `32`.

* Terminate any idle PV that has not been used in `7` days.

For more information, please reach out to Domino Support.

## Storage workflow for Workspaces

Workspace volumes are handled differently than volumes for Jobs. Workspaces are potentially long-lived development environments that users stop and resume repeatedly without writing data back to the Domino File System each time. As a result, the PV for the Workspace is a similarly long-lived resource that stores the user's working data.

These workspace PVs are durably associated with the durable Workspace they are initially created for. Each time that Workspace is stopped, the PV is detached and preserved so that it's available the next time the user starts the Workspace. When the Workspace starts again, it reattaches its PV and the user sees all their working data saved during the last session.

The contents of the project files in the workspace PV are written back to the Domino File System only when a user chooses to initiate a sync. A durable workspace PV is deleted only if the user deletes the associated Workspace.

### Durable Workspace volume backups on AWS

Since the data in durable workspace volumes is not automatically written back to the Domino File System, users can lose work if the volume is lost or deleted. When Domino is running on AWS, it safeguards against this by backing up the EBS volume that backs up the workspace PV with EBS snapshotting to S3.

EBS snapshots can be expensive, especially if you have a lot of users. To manage snapshots for your organization, please reach out to Domino Support.

If you accidentally deleted or lost a durable workspace volume that contains data you want to recover, contact Domino Support for assistance in restoring from the snapshot.

### Enable automatic Workspace deletion

Domino can automatically delete unused workspaces to make more storage available.

When a Workspace has been stopped for 30 days, Domino marks it for deletion and notifies the workspace user.

The user is notified several times during the 14-day grace period. Domino notifies the user once more when the Workspace is deleted.

To enable this feature or extend the periods, please reach out to Domino Support.

## Enable volume provisioning recommendations

When a user starts a [Workspace](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace#launch) or a [Job](/cloud/platform-capabilities/core-concepts/jobs), Domino can provide volume provisioning recommendations based on project size and previous usage to reduce storage costs and maximize performance. This feature is enabled by default.

## Configure volume size

The volume size can be configured on a per-project basis on the **Project Settings** page. This includes setting the default volume size for new projects, as well as the minimum and maximum volume size allowed.

For more information, please reach out to Domino Support.


# Manage node pools
Source: https://docs.domino.ai/cloud/admin/infrastructure-and-compute/manage-compute-resources/node-pools

View, create, edit, and delete node pools, choose their data plane, accelerator and instance types, and set maximum CPU and memory.

You can manage node pools in your Domino cluster. You can manage node pools in the local Domino data plane and in any Domino-managed data plane.

You can view existing node pools, add node pools, edit node pool limits, and delete existing node pools.

## Manage node pools

In the admin portal, go to **Platform Settings** > **Deployment Configuration**.

<img alt="Node pools configuration main page" />

You can view the existing node pools in the table. The table shows the following information for each node pool:

* **Node pool**: The name of the node pool.

* **Data plane**: The data plane where the node pool is located.

* **Accelerator**: The accelerator used by the node pool. It can be CPU only, GPU, or Neuron.

* **Instance types**: The instance types configured in the node pool.

* **Hardware tiers**: The number of hardware tiers that have configured this node pool.

* **Rate/hr**: An estimate of the cost per hour for the node pool, based on the instance types configured in the node pool.

* **Max CPU**: The maximum number of virtual CPUs the node pool can use, if a limit is set.

* **Max memory**: The maximum amount of memory the node pool can use, if a limit is set.

* **Description**: A description of the node pool.

## Add a new node pool

1. To add a new node pool, click the **Add Node Pool** button.

   <img alt="Add Node Pool dialog" />

2. In the **Add Node Pool** dialog, fill in the following fields:

   * **Name**: The name of the node pool.

   * **Description**: A description of the node pool.

   * **Data plane**: The data plane where the node pool is created.

   * **Accelerator type**: The accelerator type for the node pool. It can be CPU only, GPU, or Neuron.

3. Select the instance types you want to add to the node pool by checking the checkboxes next to the instance types.

   <img alt="Select instance types for the node pool" />

   For each instance type, you see the following information:

   * **Instance type**: The name of the instance type.

   * **vCPUs**: The number of virtual CPUs in the instance type.

   * **Memory (GiB)**: The amount of memory in the instance type.

   * **GPU count**: The number of GPUs in the instance type, if applicable.

   * **Est. hourly rate**: An estimate of the cost per hour for the instance type.

4. You can set the limits for the node pool in the **Set Limits** section. The limits are the maximum number of nodes that can be created in the node pool. This helps you to avoid creating a large number of nodes and helps keep costs under control.

   Limits are defined by the following fields:

   * **Maximum CPU**: The maximum number of virtual CPUs that can be used in the node pool (integer).

   * **Maximum memory**: The maximum amount of memory that can be used in the node pool (binary SI units, for example, `16Gi`; see [Binary SI](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-memory)).

     <Note>
       To limit the node pool to a number of instances, you can multiply the vCPU value of the instance type by the number of instances you want to limit the node pool to. For example, if you want to limit the node pool to 10 instances of `c5.2xlarge`, you can set the maximum CPU to `80` (10 \* 8 vCPUs).
     </Note>

     <img alt="Set limits for the node pool" />

5. Click the **Create Node Pool** button to create the node pool.

## Edit a node pool

You can update the description and limits on a node pool you created, without recreating it or its hardware tiers. Instance types, data plane, and accelerator type are fixed at creation.

<Note>
  You cannot edit the built-in node pools that ship with your deployment (`compute`, `platform`, `gpu`, and `trainium`). Their limits are managed through your deployment configuration, so the edit action is hidden for them.
</Note>

1. In the node pools table, click the three dots menu in the node pool row and select **Edit**.

2. In the **Edit Node Pool** dialog, update any of the following fields:

   * **Description**: The description of the node pool.

   * **Maximum CPU**: The maximum number of virtual CPUs the node pool can use (integer). Leave empty to remove the limit.

   * **Maximum memory**: The maximum amount of memory the node pool can use, in binary SI units, for example `16Gi`. Leave empty to remove the limit. See [Binary SI](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-memory).

3. Click **Save** to apply the changes.

   Any field you leave unchanged keeps its current value. To clear an existing description or limit, empty the field before saving.

## Delete a node pool

1. If the node pool is not used by any hardware tier, you can delete it by clicking the three dots menu in node pool row and selecting **Delete**.

2. A confirmation dialog appears. You see one of the following messages:

   1. If the node pool is not being used by any hardware tier:

      <img alt="Delete node pool dialog" />

      You can delete it by clicking the **Remove node pool** button.

   2. If the node pool is used by some hardware tier:

      <img alt="Delete node pool used by hardware tier" />

      You can click each hardware tier name to go to the hardware tier configuration page and remove the node pool from the hardware tier.

      Once the node pool is not used by any hardware tier, you can delete it by clicking the **I’ve updated the hardware tiers** button.


# Scale compute capacity
Source: https://docs.domino.ai/cloud/admin/infrastructure-and-compute/manage-compute-resources/scale-compute-capacity

Plan Kubernetes autoscaling limits so Domino adds compute nodes for demand, removes idle capacity, and queues executions when resources run out.

The amount of compute power required for your Domino cluster fluctuates as users start and stop executions. Domino relies on Kubernetes to find space for each execution on existing compute resources. In cloud autoscaling environments, if there’s not enough CPU or memory to satisfy a given execution request, the Kubernetes cluster autoscaler starts new compute nodes to fulfill that increased demand. In environments with static nodes, or in cloud environments where you have reached the autoscaling limit, the execution request is queued until resources are available.

Autoscaling Kubernetes clusters shut nodes down when they are idle for more than a configurable duration. This reduces your costs by ensuring that nodes are used efficiently, and terminated when not needed.

Cloud autoscaling resources have properties like the minimum and maximum number of nodes they can create. You must set the node maximum to whatever you are comfortable with given the size of your team and expected volume of workloads. If everything else is equal, it is better to have a higher limit than a lower one, as compute node cost is cheap to start up and shut down, while your users' time is valuable. If the cluster cannot scale up further, your users' executions wait in a queue until the cluster can service their request.


# Use Intel Habana accelerators
Source: https://docs.domino.ai/cloud/admin/infrastructure-and-compute/manage-compute-resources/use-intel-habana-accelerators

Configure Intel Habana Gaudi accelerators with a node pool, Hardware Tier, Kubernetes device plugin, and Gaudi-enabled container image.

Gaudi accelerators from Habana Labs (an Intel company) deliver low-cost-to-train deep learning models for natural language processing, object detection, and image recognition use cases.

AWS currently offers a dl1.24xlarge EC2 image with eight Gaudi accelerators available. See [Amazon EC2 DL1 Instances](https://aws.amazon.com/ec2/instance-types/dl1/).

## 1. Add node pool and hardware tiers

1. Create or add a Gaudi-enabled node to an existing node-pool in your Domino cluster.

2. Create a [hardware tier](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/manage-hardware-tier) so users can use this resource in Domino.

   1. Select the **Use custom GPU resource name** checkbox.

   2. In **GPU Resource Name** enter `habana.ai/gaudi`.

## 2. Install the Habana device plugin for Kubernetes

**Prerequisite:** Admin `kubectl` permissions to your cluster

1. Use the kubectl command to add the [Habana device plugin for Kubernetes](https://docs.habana.ai/en/latest/Installation_Guide/Additional_Installation/Kubernetes_Installation/Intel_Gaudi_Kubernetes_Device_Plugin.html).

   ```shell theme={null}
   kubectl create -f
   https://vault.habana.ai/artifactory/docker-k8s-device-plugin/habana-k8s-device-plugin.yaml
   ```

2. Run the following command to verify the plugin is running:

   ```shell theme={null}
   kubectl get pods -n habana-system
   ```

## 3. Use Gaudi-enabled containers

Many Intel Habana environment containers work natively in Domino. To use a custom image to create a new environment, paste the Docker registry path into the **FROM** field.\
For instance:

```
vault.habana.ai/gaudi-docker/1.5.0/ubuntu20.04/habanalabs/tensorflow-installer-tf-cpu-2.9.1
```

See the [Habana Developer Catalog](https://www.intel.com/content/www/us/en/developer/platform/gaudi/models/catalog.html) for more pre-built container images that can be used in Domino.


# View platform and compute nodes details
Source: https://docs.domino.ai/cloud/admin/infrastructure-and-compute/manage-compute-resources/view-details-of-platform-and-compute-nodes

Inspect platform and compute nodes in the Infrastructure report, including labels, available resources, hosted pods, and node pool assignments.

You can view a list and the details of the current platform and compute nodes in your compute grid.

1. In the Admin panel, click **Reports > Infrastructure**. A list of Platform and Compute nodes opens.

   Non-Platform nodes with a value in the **Node Pool** column are compute nodes that can be used for Domino executions by configuring a Hardware Tier to use the pool.

2. Click the name of a node. A complete `kubectl describe` description for the node opens, including all applied labels, available resources, and currently hosted pods.


# Monitor application execution logs
Source: https://docs.domino.ai/cloud/admin/operations/audit-logs/application-logs

Collect execution and application logs for Domino services, inspect front-end pods with kubectl, and aggregate key Kubernetes namespaces.

## Execution logs

User code running as [Jobs](/cloud/platform-capabilities/core-concepts/jobs), [Workspaces](/cloud/platform-capabilities/core-concepts/workspaces), [Apps](/cloud/platform-capabilities/features/apps), and [Domino endpoints](/cloud/platform-capabilities/features/model-deployment) output execution logs.

Your users can access these from the [Jobs dashboard](/cloud/platform-capabilities/core-concepts/jobs#monitor-jobs-and-view-results), [Workspaces Dashboard](/cloud/platform-capabilities/core-concepts/workspaces/manage-workspaces), [App Dashboard](/cloud/platform-capabilities/features/apps/app-logs-and-versioning#app-logs), and Domino endpoint instance logs. This data is a key part of the Domino reproducibility model and is kept indefinitely in the [Domino File Store](/cloud/admin/data-administration/project-file-storage).

The installation configuration file at `blob_storage.logs` defines which system these logs are written to.

## Application logs

All Domino services use the [standard Kubernetes logging architecture](https://kubernetes.io/docs/concepts/cluster-administration/logging/) to output their logs. Relevant logs are printed to `stdout` or `stderr` as indicated, and Kubernetes captures them.

## Support bundle

A zip file that contains logs and reports from Domino components with information about a Domino execution. See [Support Bundles](/help/help#support-bundles).

## View your front-end logs

1. List your all namespaces to find the name of your platform namespace:

   ```shell theme={null}
   kubectl get namespace
   ```

2. List all the pods in your platform namespace to find the name of a front-end. You likely have more than one frontend pod.

   ```shell theme={null}
   kubectl get pods  -n <namespace for your platform nodes>
   ```

3. Print the front-end logs for one of your frontends.

   ```shell theme={null}
   kubectl logs <pod name of your frontend pod> -n <namespace for your platform nodes> -c nucleus-frontend
   ```

Domino recommends that you attach a Kubernetes log aggregation utility to monitor the following [Kubernetes namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) used by Domino to aggregate the logs.

* Platform namespace

  This namespace hosts the core application components of the Domino application, including API servers, databases, and web interfaces. The installer configuration file defines the name of this namespace at `namespaces.platform.name`. The following components running in this namespace produce the most important logs:

  | Component | Logs |
  | - | - |
  | `nucleus-frontend` | The nucleus-frontend pods host the frontend API server that routes all requests to the Domino application. Its logs contain details about HTTP requests to Domino from the application or another API client. If you see errors in Domino with HTTP error codes like 500, 504, or 401, you can find corresponding logs here. |
  | `nucleus-dispatcher` | The nucleus-dispatcher pod hosts the Domino scheduling and brokering service that sends user execution pods to Kubernetes for deployment. Errors in communication between Domino and Kubernetes result in corresponding logs from this service. |
  | `keycloak` | The keycloak pod hosts the Domino authentication service. The logs for this service contain a record of authentication events, including additional details on any errors. |
  | `cluster-autoscaler` | This pod hosts the open source Kubernetes cluster autoscaler, which controls and manages autoscaling resources. The logs for this service contain records of scaling events, both scaling up new nodes in response to demand and scaling down idle resources, including additional details on any errors. |

* Compute grid namespace

  This namespace hosts user executions plus Domino environment builds. The installer configuration file defines the name of this namespace at `namespaces.compute.name`.

  Logs in this namespace correspond to ephemeral pods hosting user work. Each pod contains a user-defined environment container, whose logs are [execution logs](#execution-logs). There are additional supporting containers in the pods, and their logs might contain additional information about errors or behavior seen in specific Domino executions.

  Domino advises that you aggregate and keep at least 30 days of logs to facilitate debugging. You can harvest these with a variety of Kubernetes log aggregation utilities, including:

  * [Loggly](https://www.loggly.com/solution/kubernetes-logging/)

  * [Splunk](https://docs.splunk.com/Documentation/InfraApp/2.0.2/Admin/AddDataKubernetes)

  * [New Relic](https://docs.newrelic.com/docs/logs/enable-logs/enable-logs/kubernetes-plugin-logs)

## Next steps

* [Project audit logs](/cloud/admin/operations/audit-logs/monitor-project-logs) for project-level auditing including user permissions and Dataset-related actions.

* The [Audit trail data glossary](/cloud/platform-capabilities/features/governance/audit-trail/audit-trail-glossary) contains Domino audit event names along with their descriptions.

* [Data Source audit logs](/cloud/admin/operations/audit-logs/monitor-data-source-logs) create an audit trail for Data Source activity and records the WHO, WHERE, WHEN & WHAT for user activity.


# Require e-signature for critical actions
Source: https://docs.domino.ai/cloud/admin/operations/audit-logs/esignature

Require e-signatures for critical Dataset and admin actions, configure reason-for-change options, and inspect events in the Unified Audit Trail.

Admins can require an e-signature by enabling the configuration record `com.cerebro.domino.eSignatureWorkflowEnabled`. E-signatures can be viewed in the Unified Audit Trail.

## How reason for change (RFC) works

When enabled, critical actions trigger an **RFC modal**:

* User provides a reason for change (select from dropdown or enter additional context).

* User re-authenticates (password or SSO).

* Action is submitted, and details are logged in the audit trail with:

  * Action performed

  * Reason for change

  * Electronic signature and timestamp

<img alt="Reason for change UI" />

## Critical actions that require an e-signature

If `Global` or `PerProject` is enabled, the following actions require an e-signature:

* Uploading, renaming, or deleting dataset files.

* Creating dataset snapshots.

* Editing dataset permissions.

* Downloading files from datasets or artifacts.

* Admin actions such as dataset deletion.

## Configure e-signature requirements

Administrators can control e-signature requirements using configuration record keys:

* `com.cerebro.domino.requireESignatureWorkflow`

  * `Disabled/None`: No RFC enforcement (default).

  * `Global`: Required for all projects and admin actions.

  * `PerProject`: Project owners/admins may enable RFC enforcement on a per-project basis.

* `com.cerebro.domino.eSignatureActionOptions`

  * Allows admins to define custom dropdown values for "Reason for Change".

  * Example: `"Regulatory Submission, Data Correction, Test Activity"`

<Note>
  Only **system admins and project owners** can enable this functionality at the project level. This configuration change is also captured in the audit trail.
</Note>

## View e-signatures

E-signature events can be viewed in the [Unified Audit Trail](/cloud/platform-capabilities/features/governance/audit-trail). In the Audit Trail logs, admins can add columns to see the reason for the changes, the user who e-signed, and any additional information entered by the user.

## Next steps

* The [Audit Trail Data Glossary](/cloud/platform-capabilities/features/governance/audit-trail/audit-trail-glossary) has a complete list of events being tracked. You can use these in the event filter to search for events.

* [Data Source audit logs](/cloud/admin/operations/audit-logs/monitor-data-source-logs) create an audit trail for Data Source activity and records the WHO, WHERE, WHEN, and WHAT for user activity.


# Use execution logs
Source: https://docs.domino.ai/cloud/admin/operations/audit-logs/index

Use application, Data Source, and Project logs to audit executions, permissions, Dataset activity, and actions that require e-signatures.

Application logs\
[Application logs](/cloud/admin/operations/audit-logs/application-logs) capture application-level logging including user execution from Jobs, Workspaces, and front-end logs. These can also be aggregated in enterprise logging platforms such as Splunk & New Relic.

Require e-signature for critical actions\
Admins can [require an e-signature](/cloud/admin/operations/audit-logs/esignature) for critical actions.

Data Source logs\
[Data Source logs](/cloud/admin/operations/audit-logs/monitor-data-source-logs) capture events like Data Source creation, access, deletion, and permission setting.

Project logs\
[Project logs](/cloud/admin/operations/audit-logs/monitor-project-logs) capture project-scoped events including user permission setting and Dataset-related events.


# Monitor Data Source logs
Source: https://docs.domino.ai/cloud/admin/operations/audit-logs/monitor-data-source-logs

Audit Data Source creation, access, ownership, permissions, and Project associations through the web UI or filtered REST API requests.

Domino automatically creates an audit trail for [Domino Data Source](/cloud/admin/data-administration/external-data) activity that records the WHO, WHERE, WHEN & WHAT for user activity. Audit trails can provide a means to help accomplish several security-related objectives, including the following:

* Individual accountability

* Reconstruction of events or security breaches

* Intrusion Detection

* Problem Analysis

* Ensuring compliance with industry regulations

<Tip>
  We recommend you use at least `dominodatalab-data==5.7.2` [PyPI version](https://pypi.org/project/dominodatalab-data/5.7.2/) for the most thorough metadata event tracking. For R users, we recommend at least `0.3.0` of [DominoDataR](https://github.com/dominodatalab/DominoDataR/releases).
</Tip>

Domino logs the following Data Source events:

* Create Data Sources

* Delete Data Sources

* Edit Data Source permissions

* Edit Data Source ownership

* Access Data Sources from Domino executions

* Add Data Sources to projects

* Remove Data Sources from projects

<Note>
  Data Source access events are logged for all executions (Workspaces, Jobs, Scheduled Jobs, Apps, and Launchers), except Domino endpoints.
</Note>

Access the audit logs via the web UI or [Public REST API endpoint](/cloud/reference/api/domino-open-api), `/api/datasource/v1/audit`.

## Data Source API

Use the Data Source audit [Public REST API](/cloud/reference/api/domino-open-api) endpoint, `/api/datasource/v1/audit`, which takes several optional filter parameters, to access Data Source logs programmatically.

All requests must contain the header `X-Domino-Api-Key` with a value corresponding to a **CloudAdmin** API key.

### Unfiltered Data Source audit API sample

The following is a request with no filter parameters, it returns all events within the last 24 hours.

```json theme={null}
// request
https://<domino-url>/api/datasource/v1/audit

// response: with no filter parameters, all events in the last 24 hours are returned
[
    {
        "dataSourceId": "64e67c6005e8b2388f992545",
        "dataSourceName": "snowflake-ds",
        "dataSourceType": "SnowflakeConfig",
        "eventKind": "DeleteDataSource",
        "metadata": {
            "performedByUsername": "sample-admin"
        },
        "performedBy": "64e6363605e8b2388f9924c8",
        "timestamp": "2023-08-23T21:40:21.669Z"
    },
    {
        "dataSourceId": "64e67c6005e8b2388f992546",
        "dataSourceName": "s3-ds",
        "dataSourceType": "S3Config",
        "eventKind": "AccessDataSource",
        "metadata": {
            "runType": "Workspace",
            "projectId": "64e6371205e8b2388f9924d4",
            "dataPlaneId": "000000000000000000000000",
            "performedByUsername": "sample-user",
            "runId": "64e6632005e8b2388f9924eb"
        },
        "performedBy": "64e6371105e8b2388f9924d2",
        "timestamp": "2023-08-21T21:30:05.439Z"
    }
]
```

### Filtered Data Source audit API sample

The following is a request with several filter parameters, including `startTime`, `endTime`, multiple `dataSourceNames`, and `eventKinds`, for a more granular response.

```json theme={null}
// request
https://<domino-url>/api/datasource/v1/audit/?startTime=2022-08-17T23:09:24.921Z&endTime=2023-08-25T23:09:24.921Z&dataSourceNames=snowflake-ds&dataSourceNames=s3-ds&eventKinds=AccessDataSource

// response
[
    {
        "dataSourceId": "64e67c6005e8b2388f992546",
        "dataSourceName": "s3-ds",
        "dataSourceType": "S3Config",
        "eventKind": "AccessDataSource",
        "metadata": {
            "runType": "Workspace",
            "projectId": "64e6371205e8b2388f9924d4",
            "dataPlaneId": "000000000000000000000000",
            "performedByUsername": "sample-user",
            "runId": "64e6632005e8b2388f9924eb"
        },
        "performedBy": "64e6371105e8b2388f9924d2",
        "timestamp": "2023-08-21T21:30:05.439Z"
    },
    {
        "dataSourceId": "64e67c6005e8b2388f992545",
        "dataSourceName": "snowflake-ds",
        "dataSourceType": "SnowflakeConfig",
        "eventKind": "AccessDataSource",
        "metadata": {
            "runType": "Workspace",
            "projectId": "64e6371205e8b2388f9924d4",
            "dataPlaneId": "000000000000000000000000",
            "performedByUsername": "sample-user",
            "runId": "64e6632005e8b2388f9924eb"
        },
        "performedBy": "64e6371105e8b2388f9924d2",
        "timestamp": "2023-02-05T21:40:01.449Z"
    },
]
```

## Next steps

* [Project audit logs](/cloud/admin/operations/audit-logs/monitor-project-logs) for information on project-level auditing including user permissions and Dataset-related actions.

* The [Audit trail data glossary](/cloud/platform-capabilities/features/governance/audit-trail/audit-trail-glossary) contains Domino audit event names along with their descriptions.

* [Data Source audit logs](/cloud/admin/operations/audit-logs/monitor-data-source-logs) create an audit trail for Data Source activity and records the WHO, WHERE, WHEN & WHAT for user activity.


# Monitor Project logs
Source: https://docs.domino.ai/cloud/admin/operations/audit-logs/monitor-project-logs

Download a Project audit log as a PDF to review user permission changes and Dataset creation, deletion, restoration, and sharing events.

Domino automatically logs actions performed in Domino Projects. You can use this data to meet certain audit requirements requested by your regulators. Project actions include, but are not limited to, the following:

* Create datasets/snapshots

* Delete datasets/snapshots

* Restore datasets/snapshots

* Edit dataset permissions

* Add/remove dataset to/from a project

* User permission changes

Domino endpoint data is not included in the Domino Project audit log.

This report cannot be modified.

For advanced audit procedures that track data uploads and downloads to and from Domino Datasets, contact [support@dominodatalab.com](mailto:support@dominodatalab.com).

## Download log

1. Go to the project for which you want to download the audit log.

2. Go to **Settings** > **Access & Sharing**.

3. In the **Collaborators and permissions** section, click **Download Audit Log** to download the PDF.

   The PDF includes all [permission](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-datasets) changes made to users that belong to the project. It records who made the change to what user, when the change was made, and the specific attributes that were changed. It also records changes to datasets:

   * When a user marks a dataset for deletion.

   * When a user restores or deletes a dataset.

   * When a user creates a dataset.

   * When a user mounts or unmounts a shared dataset.

     The audit log lists a timestamp and the name of the user who made the changes. For shared datasets, the logs also list the project from which the changes originate.

<Note>
  If you started your project while using Domino 5.3, the start of the date range is the project start date. If you started your project prior to using Domino 5.3 and then upgraded to Domino 5.3, the start date of the date range is the date on which your organization deployed Domino 5.3.
</Note>

## Next steps

* [Project audit logs](/cloud/admin/operations/audit-logs/monitor-project-logs) for project-level auditing including user permissions and Dataset-related actions.

* The [Audit trail data glossary](/cloud/platform-capabilities/features/governance/audit-trail/audit-trail-glossary) contains Domino audit event names along with their descriptions.

* [Data Source audit logs](/cloud/admin/operations/audit-logs/monitor-data-source-logs) create an audit trail for Data Source activity and records the WHO, WHERE, WHEN & WHAT for user activity.


# Export Compute and Spend data with the API
Source: https://docs.domino.ai/cloud/admin/operations/control-center/export-control-center-data

Export Compute and Spend run data as JSON or CSV with the Domino API, paginate by batch ID, and retrieve Workspace utilization by date.

The Compute and Spend report provides several views about deployment usage, broken down by hardware tier, project, or user. However, if you want to do a more detailed, custom analysis, you can use the API to export Control Center data for examination with Domino’s data science features or external business intelligence applications.

The endpoint that serves this data is `/v4/gateway/runs/getByBatchId`.

## Prerequisites

* The API key for your account.

* An administrator account to access the full deployment’s Control Center data.

## Get the API key

1. Log in as an administrator, then click **Account** > **Account Settings**.

2. Under **Account Settings**, click **API Key**. Click **Regenerate** if you have not generated a key before.

3. Copy the key and store it carefully as you need it to make requests to the API.

   <Warning>
     Anyone with this key can authenticate to the Domino API as you so treat it like a sensitive password.
   </Warning>

Domino recommends repeating this process every 60-days to maintain security best practices.

## Use the data gateway endpoint

The following is a basic call to the data export endpoint, executed with [curl](https://curl.haxx.se/):

```shell theme={null}
curl --include -H "X-Domino-Api-Key: <your-api-key>"
'https://<your-domino-url>/v4/gateway/runs/getByBatchId'
```

By default, the endpoint starts with the oldest available run data, starting from January 1, 2018. Older data isn’t available. The command also has a default limit of 1000 runs worth of data. As written, the preceding call returns data on the oldest 1000 runs available.

To try this example, enter `your-api-key` and `your-domino-url` in the previous command.

The standard JSON response object you receive has the following scheme:

```json theme={null}
{
  "runs": [
    {
      "batchId": string,
      "runId": string,
      "title": string,
      "command": string,
      "status": string,
      "runType": string,
      "userName": string,
      "userId": string,
      "projectOwnerName": string,
      "projectOwnerId": string,
      "projectName": string,
      "projectId": string,
      "runDurationSec": integer,
      "hardwareTier": string,
      "hardwareTierCostCurrency": string,
      "hardwareTierCostAmount": number,
      "queuedTime": date-time ,
      "startTime": date-time,
      "endTime": date-time,
      "totalCostCurrency": string,
      "totalCostAmount": number,
      "computeClusterDetails : {
        "computeClusterType": string,
        "masterHardwareTierId": string,
        "masterHardwareTierCostPerMinute": number,
        "workerCount": integer,
        "workerHardwareTierId": string,
        "workerHardwareTierCostPerMinute": number
      }
    }
 ],
  "nextBatchId": string
}
```

The Control Center gives each run recorded a `batchId`, which is an incrementing field that can be used as a cursor to fetch data in multiple batches. You can see in the previous response, after the array of `runs` objects, a `nextBatchId` parameter points to the next run to include. Use that ID as a query parameter in a subsequent request to get the next batch:

```shell theme={null}
curl --include
-H "X-Domino-Api-Key: <your-api-key>"
'https://<your-domino-url>/v4/gateway/runs/getByBatchId?batchId=<your-batchId-here>'
```

You can also include a header with `Accept: text/csv` to request the data as CSV. On the Unix shell, you can write the response to a file with the `>` operator. This is a quick way to get data suitable for import into analysis tools:

```shell theme={null}
curl --include
-H "X-Domino-Api-Key: <your-api-key>"
-H 'Accept: text/csv'
'https://<your-domino-url>/v4/gateway/runs/getByBatchId' > your_file.csv
```

## Get all Workspaces

Use the following API call to retrieve all workspaces:

```bash theme={null}
GET /controlCenter/utilization/runsPerDay
```

<Note>
  The `com.cerebro.domino.controlCenter.cacheTimeToLiveInMinutes` configuration key specifies the cache refresh as 30 minutes, by default. This might cause a delay in retrieving some workspaces.
</Note>

```shell theme={null}
curl -X GET "<your-domino-url>/v4/controlCenter/utilization/runs?startDate=20220503&endDate=20220503" -H  "accept: application/json"
```

| Parameter | Required | Description |
| - | - | - |
| `projectId` | | |
| `startingUserId` | | |
| `organizationId` | | |
| `hardwareTierId` | | |
| `startDate` | ✓ | Range must be in `YYYYMMDD` format |
| `endDate` | ✓ | Range must be in `YYYYMMDD` format |

### Example: Get all data

The following code shows a Python script that fetches all Control Center data from the earliest available to a configurable end date, and writes it to a CSV file. Enter the date of the last known completed run to fetch all available historical data.

```python theme={null}
import requests
import json
import pandas as pd
import os
from datetime import datetime
from datetime import timedelta

URL = "https://<your-domino-url>/v4/gateway/runs/getByBatchId"
headers = {'X-Domino-Api-Key': '<your-api-key>'}
last_date = 'YYYY-MM-DD'

last_date = datetime.strftime(
    datetime.strptime(last_date, '%Y-%m-%d') + timedelta(days = 1),
    '%Y-%m-%d',
)

try:
  os.remove('output.csv')
except:
  pass

batch_ID_param = ""
while True:
    batch = requests.get(url = URL + batch_ID_param, headers = headers)
    parsed = json.loads(batch.text)
    batch_ID_param = "?batchId=" + parsed['nextBatchId']
    df = pd.DataFrame(parsed['runs'])
    df[df.endTime <= last_date].to_csv(
        'output.csv',
        mode="a+",
        index=False,
        header=True)
    if len(df.index) < 1000 or len(df.index) > len(df[df.endTime <= last_date].index):
        break
```

You can run a script like this periodically to import fresh data into your tools for custom analysis. Work with the data in a Domino project, or make it available to third-party tools like Tableau.


# Use the Control Center
Source: https://docs.domino.ai/cloud/admin/operations/control-center/index

Track deployment-wide compute usage and spend in Control Center, drill into Hardware Tiers, Projects, and users, and export data for analysis.

Use the Control Center to see important data about your Domino deployment.

<Warning>
  Only Domino Admins can access the Control Center. If you need access to the Control Center, contact your local Domino Administrator or email [support@dominodatalab.com](mailto:support@dominodatalab.com).
</Warning>

For more information on how to use the Control Center, have a look at these topics:

* [View Control Center metrics](/cloud/admin/operations/control-center/view-metrics-in-cc), including deployment-wide usage of compute resources by hours of runtime or spend in USD.

* Drill down into statistics about [hardware tiers](/cloud/admin/operations/control-center/view-hardware-tier-information), [Projects](/cloud/admin/operations/control-center/view-project-spend), and [user activity](/cloud/admin/operations/control-center/view-user-activity).

* [Export Compute and Spend data](/cloud/admin/operations/control-center/export-control-center-data) to create your own reports or analyses.

* Use the Projects Portfolio to [view the status of all your Projects](/cloud/admin/operations/control-center/view-the-status-of-all-projects).


# View hardware tier execution information
Source: https://docs.domino.ai/cloud/admin/operations/control-center/view-hardware-tier-information

Inspect Hardware Tier performance averages, completed runs, and Run Logs, then open a selected run from the Project Jobs page.

<Warning>
  Domino recommends using [FinOps](/cloud/admin/operations/finops) features to view and control cloud compute costs. With FinOps, you can set budgets, alerts, store usage data for analysis, and directly integrate with cloud billing APIs to provide the most accurate representation of your cloud bills.
</Warning>

1. In the navigation pane, click **Govern** > **Compute & Spend**.

2. In the second chart, switch the data to **Hardware Tiers**.

3. Click a bar in the chart.

   This page shows performance averages for runs that use the specified hardware tier, and tracks completed runs. The **Run Logs** table lists details about all runs performed on the specified hardware tier.

4. Click an entry in the table to view the specified run in the project **Jobs** page.


# View Control Center metrics
Source: https://docs.domino.ai/cloud/admin/operations/control-center/view-metrics-in-cc

Compare compute spend, runtime, queue time, and usage across Projects, users, organizations, and Hardware Tiers in Control Center.

<Warning>
  Domino recommends using [FinOps](/cloud/admin/operations/finops) features to view and control cloud compute costs. With FinOps, you can set budgets, alerts, store usage data for analysis, and directly integrate with cloud billing APIs to provide the most accurate representation of your cloud bills.
</Warning>

1. In the navigation pane, click **Govern** > **Cost center**.

   The first chart is a bar chart of deployment compute spend in USD, along with memory costs of executions, for each day in the current month.

   <img alt="Cost center dashboard" />

   Compute spend is based on settings applied by admins when creating and managing [hardware tiers](https://tickets.dominodatalab.com/hc/en-us/articles/360058497671-Hardware-tiers-and-cluster-controls). Compute spend data is available in this report only if the **Cents per minute** field is set for the [hardware tier](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers) in use. These numbers also only represent active usage and do not reflect other potential spend such as idle cloud resources or storage.

   1. Use the date menu to change the date range.

   2. Use the **Compute Spend (\$)** menu to change the chart to show compute usage by hours of runtime.

The next chart shows more granular data about projects, users, organizations, and hardware tiers across the selected date range. You can chart these by metrics such as the following:

* Projects can be charted by compute spend (USD) or compute hours.

* Users can be charted by compute spend (USD) or compute hours.

* Hardware tiers can be charted by average run queue time in minutes.

This chart shows the top five results for the selected metric. If you set this chart to show data about users, you can click the **View all** link to load a paginated table with detailed usage statistics for all users.

By default, this table shows data for the date range set on the previous page. Use the menu to change the date range.

## Drill down for more details

You can drill down into most of the tables and charts support for more details. Click a bar in a bar chart to see a more detailed page about the related project, user, or hardware tier.

Some of these pages also show a table of related runs. You can click an entry in a **Run Logs** table to view the specified run in the project **Runs** page.


# View Project spend
Source: https://docs.domino.ai/cloud/admin/operations/control-center/view-project-spend

Break down Project spend across Apps, Batch Runs, Endpoints, Launchers, Scheduled Runs, and Workspaces, then inspect Run Logs.

<Warning>
  Domino recommends using [FinOps](/cloud/admin/operations/finops) features to view and control cloud compute costs. With FinOps, you can set budgets, alerts, store usage data for analysis, and directly integrate with cloud billing APIs to provide the most accurate representation of your cloud bills.
</Warning>

1. In the navigation pane, click **Govern** > **Compute & Spend**.

2. In the second chart, switch the data to **Projects**.

3. Click a bar in the chart.

   This page breaks down project spend across Apps, Batch Runs, Endpoints, Launchers, Scheduled Runs, and Workspaces. The **Run Logs** table lists details about all runs performed on the specified hardware tier.

4. Click an entry in the table to view the specified run in the project **Jobs** page.


# View the status of all Projects
Source: https://docs.domino.ai/cloud/admin/operations/control-center/view-the-status-of-all-projects

Use the Projects Portfolio to list every Project your data science team works on, see who works on each one, and filter by Project name, owner, or username.

Use the Projects Portfolio to see all the Projects that your data science team is working on.

In the navigation pane, click **Govern** > **Projects Portfolio**.

You can see a list of every Project, including Jira-linked Projects, and who is working on them.

* Use the filter to limit the project list with text entered in the **Project Name**, **Owner**, or **Username**.


# View user activity
Source: https://docs.domino.ai/cloud/admin/operations/control-center/view-user-activity

Inspect a user's runs, Projects, Hardware Tiers, and compute hours in Control Center, then open individual runs from Run Logs.

1. In the navigation pane, click **Govern** > **Compute & Spend**.

2. In the second chart, from the menu, click **Users**.

3. Click a bar in the chart.

   This page shows detailed data on a user’s activity in Domino. The top of the page shows the types of runs this user starts, which projects the user works on, and which hardware tiers the user uses. You can click bars in the compute hours, project, and hardware tier charts to view the object represented. The **Run Logs** table lists details about all runs performed on the specified hardware tier.

4. Click an entry in the table to view the specified run in the project **Jobs** page.


# Configure Billing Tags, Budgets, and Alerts
Source: https://docs.domino.ai/cloud/admin/operations/finops/1-budgets-alerts

Configure Cost Center billing tags, budget limits and overrides, spending alerts, plus Dataset snapshot and storage quotas.

You can use Domino Cost Center to establish budgets and send alerts to team members when they approach their spending limits on compute and storage resources. The addition of Billing Tags gives you added flexibility to track and manage project costs horizontally across your organization.

These features make it easier to track costs across different projects. Administrators can also set limits on the amount of storage for Datasets.

## Configure billing tags

You can aggregate Cost Center or organization usage by using billing tags within Domino. Billing tags are a way of tagging projects so that they can be grouped by tag for accounting purposes. Projects may have multiple billing tags, and Administrators have the option to require the use of billing tags or not. To create billing tags:

1. Go to **Admin** > **Manage resources** > **Cost tracking** > **Billing tags**.

2. In the **Configure Billing Tags** section, you can set the **Billing Tag** feature to `Disabled`, `Optional`, or `Required` for new projects. Existing projects are unaffected, but you can still apply billing tags to them manually.

3. For Billing Tags, enter one or more names and click **Add Billing Tags**.

<img alt="cost center dashboard" />

After the billing tags are created, you can filter available projects and assign billing tags to them using the **Manually apply Billing Tags** field. This is useful for adding a newly defined billing tag to an existing project.

## Configure budgets to prevent overspending

Budgets help control costs for organizations and projects by preventing overspending. By default, all projects and organizations have no budget limit. To set and configure your budgets:

1. Go to **Manage Cost**.

2. Go to **Admin** > **Manage resources** > **Cost tracking** > **Budget & alerts**.

3. In the **Default Limits** section, set a default monthly (by calendar month) budget that applies to all organizations, projects, or billing tags.

4. Use **Budget Overrides** to change the default budget limits for specific organizations, projects, or billing tags.

<img alt="Configure budgets to prevent overspending" />

## Configure alerts to notify stakeholders

Alerts notify stakeholders when expenses for their organizations are nearing the budget threshold. These alerts are delivered through both Domino notifications and email.

Alerts are not configured by default. Billing tag notifications are only sent by email and require email to be configured to allow alerts. To set and configure alerts:

1. Go to **Admin** > **Manage resources** > **Cost tracking** > **Budget & alerts**.

2. Under **Alert Settings**, select **Alert on 75% and 100%** of budget thresholds to enable alerts.

3. Add additional emails to receive notifications.

4. Go to **Platform settings** > **Email** to configure your email and allow email alerts.

Domino sends out daily notifications when 75% of the budget is reached.

## Configure limits per Dataset

Admins can also use configuration keys related to read-write Datasets to set limits and quotas on the amount of storage used by Datasets, saving money on storage expenses and monitoring usage.

For more information, please get in touch with Domino Support.

### Set Dataset snapshot limits

When a Dataset reaches the snapshot limit, users receive an error message when they attempt to create an additional snapshot.

To limit the number of snapshots per Dataset, please reach out to Domino Support.

You must delete old snapshots or increase the limit before writing additional snapshots.

### Override snapshot limits

You can authorize individual projects to ignore the snapshot limits that you set:

1. Go to the **Project** that you want to set snapshot limits on.

2. Go to **Settings** > **Hardware & Environment**.

3. Select the **Ignore Dataset Limits** checkbox.

Selecting **Ignore Dataset Limits** does not impact any quotas for which hard limits have been set.

### Dataset sizing

When a user interacts with a dataset by uploading a file, deleting a file, creating a snapshot, or ending a run, the dataset is queued to be sized in the background. This allows large datasets to be sized quickly and accurately without timing out.

To avoid multiple expensive calls to resize the dataset, Domino waits seven days between each sizing request.

### Dataset quota limits

You can also set dataset quotas to notify users if their datasets exceed the allocated storage. Quotas are "soft" quotas, meaning they never block users from creating more datasets or snapshots.

There are two types of dataset quotas. To set quotas:

1. Navigate to **Admin** > **Datasets**.

2. Choose the type of quota to set:

   * `Global quota`: Creates a limit that applies to every user in the deployment.

   * `Quota override`: Sets individual limits on specific users and overrides the `global quota`.

Quota overrides only work when a global quota is also configured.

### Dataset quota calculations

Domino calculates quotas based on a user’s total dataset storage footprint, which includes all the Datasets they own, except for datasets in specific states:

* `MarkedForDeletion`

* `DeletionInProcess`

* `Deleted`

* `Failed`

The size of a dataset is determined by summing the sizes of each snapshot within the dataset, excluding any deleted snapshots. If a dataset has multiple owners, the size of that dataset is counted towards each owner’s quota.

[Dataset Roles](/cloud/platform-capabilities/core-concepts/data/sharing-and-security/share-datasets#dataset-roles) has more information about how the roles work together.

### Quota notifications

The thresholds for the Read-write Datasets options can be configured. Contact your Domino Administrator if you need help with this.

When a user reaches certain thresholds of their quota, they receive notifications, emails, and UI warnings on Dataset pages. These actions are triggered by the following events:

* Uploading files to a Dataset (both UI and CLI).

* Creating a snapshot (both UI and CLI).

* Creating a Dataset from a snapshot (both UI and CLI).

* Deleting files from a Dataset in the UI.

When a user receives an email about their Dataset storage quota, administrators automatically receive a similar email. You can also add additional recipients under **Quota Notifications**.

## Next steps

Here are some other ways Domino Cost Center can help you optimize the performance of your AI projects while managing cloud expenditures:

* [Automatically allocate usage-based costs](/cloud/admin/operations/finops/3-analyze-costs-API) to Projects, organizations, and users.

* Optimize [Dataset storage usage](/cloud/admin/data-administration/datasets#set-limits-on-dataset-usage) to reduce cloud storage costs.

* Generate [chargeback/showback reports](/cloud/admin/operations/finops/3-analyze-costs-API) and use APIs to facilitate cost recovery across organizations and engage with high-expenditure users to optimize infrastructure usage.


# Configuration Options for Cost Center
Source: https://docs.domino.ai/cloud/admin/operations/finops/2-configuration-options-cost-center

Configure Cost Center data export, long-term blob storage, federated storage, and cloud billing integration for cost analysis.

This guide describes several configuration options that you can customize for the [Domino Cost Center](/cloud/admin/operations/finops).

You can download Domino Cost Center data in JSON format. You can then connect this JSON file to your [Tableau](https://help.tableau.com/current/pro/desktop/en-us/examples_json.htm) or [Power BI dashboard](https://learn.microsoft.com/en-us/power-bi/connect-data/desktop-quickstart-connect-to-data). This allows you to visualize the data and share it with your team members as needed.

Domino also offers the ability to integrate with cloud provider billing APIs, allowing for precise tracking and reporting of Domino’s cloud expenses.

## Configure long-term storage for Domino Cost Center

Domino Cost Center lets you retain your cost data in blob storage for long-term analysis. Non-AWS users can manually provision their blob store and set up federated storage for their Cost Center. To provision your blob store and set up Cost Center federated storage:

1. In your Domino cluster, create a new file called `federated-store.yaml`.

2. Format the contents of the file according to each cloud provider’s specifications:

   * [GCP storage configuration](https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=configuration-gcp-multi-cluster-storage)

   * [Azure storage configuration](https://www.ibm.com/docs/en/kubecost/self-hosted/2.x?topic=configuration-azure-multi-cluster-storage)

3. Add a secret using `federated-store.yaml: kubectl create secret generic <secret_name> -n <namespace> --from-file=federated-store.yaml`.

4. Set `.Values.kubecostModel.federatedStorageConfigSecret` to use your Kubernetes secret name.

## Cloud integration for Domino Cost Center

Cost Center is preconfigured for AWS. Non-AWS users can manually provision their cost usage report and set up cloud integration for Cost Center. To retrieve billing information from your cloud provider, contact your Domino representative to set up Cost Center Cloud Billing Integration.

## Next steps

* Prevent overspending by [setting budgets and sending alerts](/cloud/admin/operations/finops/1-budgets-alerts) to stakeholders, administrators, and team members approaching their limits.

* Learn more about [automatic allocation of usage-based costs](/cloud/admin/operations/finops/3-analyze-costs-API).


# Analyze costs with Domino Cost Center
Source: https://docs.domino.ai/cloud/admin/operations/finops/index

Use Cost Center to filter, compare, and allocate CPU, GPU, storage, and shared service costs by Project, user, organization, and workload.

Domino’s Cost Center dashboard gives administrators clear visibility into cloud costs for your Domino deployment. They can quickly see how resource usage and cost trends over time map to your organization from within Domino.

Administrators can view total expenses and break them down by resource type, such as CPU, GPU, storage, and cloud services. They can also analyze spending by time period and top contributors, including users, projects, organizations, billing tags, workloads, and workload types. This feature helps with both overall cost tracking and detailed analysis of usage patterns. Domino Cost Center helps reduce project costs by:

* [Automatic allocation of usage-based costs](/cloud/admin/operations/finops/3-analyze-costs-API) to organizational entities (users, projects, etc) and workload.

* Optimization of [dataset storage usage](/cloud/admin/data-administration/datasets#set-limits-on-dataset-usage) to reduce cloud storage costs.

* Proactive spend control by [setting budgets and sending alerts](/cloud/admin/operations/finops/1-budgets-alerts) to stakeholders, administrators, and team members approaching their limits.

* Generation of [chargeback/showback reports](/cloud/admin/operations/finops/3-analyze-costs-API) using APIs to facilitate cost recovery across organizations and engage with high-expenditure users to optimize infrastructure usage.

Domino Cost Center automatically tags all infrastructure usage with essential metadata, associating costs with specific Domino projects, organizations, users, and billing tags, allowing for detailed, usage-based cost allocations.

## Access the Cost Center dashboard

The Cost Center dashboard provides a pre-configured canvas to get you started. Access it from the main Domino page as an Administrator:

1. From Domino, go to the top navigation bar.

2. Open **Govern** > **Cost center**.

3. The Cost Center dashboard loads with the default view.

<img alt="cost center dashboard" />

* **Date picker**: Click the date picker in the top left to select a date range, either 7 days, 30 days, 6 months, or a custom range (limited to 1 year).
* **Filter by…​** Use the filter controls at the top of the dashboard to narrow your data by category: **Project**, **user**, **billing tag**, or **workload type**.
* **total cost** bar: Shows numerical cost data over the selected date range. Cost is broken down by: **CPU**, **GPU**, **Storage**, **Cloud** services. Note that the cloud costs represent shared infrastructure expenses. This cannot be attributed to specific users, projects, or workloads. Even though they are included in the overall total, cloud costs do not appear in charts and summaries after filters are applied.
* **Total costs by period**: Displays a time-based breakdown of expenditures. Hover over a bar to view detailed costs by resource type.
* **Top** contributors: Scroll down to explore top contributors, organized by category: **Top users**, **Top projects**, **Top organizations**, **Top billing tags**, **Top workloads**, or **Top workload types**. Hovering over a bar reveals the exact spend associated with that contributor.

## Next steps

* Prevent overspending by [setting budgets and sending alerts](/cloud/admin/operations/finops/1-budgets-alerts) to stakeholders, administrators, and team members approaching their limits.

* Customize [configuration options for Cost Center](/cloud/admin/operations/finops/2-configuration-options-cost-center), including data export, long-term storage, and cloud billing integration.

* Learn more about [automatic allocation of usage-based costs](/cloud/admin/operations/finops/3-analyze-costs-API).

* Reduce cloud storage costs by optimizing [dataset storage usage](/cloud/admin/data-administration/datasets#set-limits-on-dataset-usage).

* Learn about [how costs are calculated](/cloud/admin/operations/finops/4-cost-calculation-methodology).


# Use data plane agent monitoring dashboards
Source: https://docs.domino.ai/cloud/admin/operations/grafana-workload-monitoring/data-plane-agent-monitoring

Use Grafana dashboards to track Data Plane Agent health, message processing, Kubernetes API latency, resource usage, and scaling.

Use these dashboards to monitor the health and performance of the Domino Data Plane Agents that manage executions and Kubernetes resources.

## Data Plane health and connectivity

Track agent status and communication health. Identify outages, disconnections, and service disruptions.

**Panels:** Agent Health, Data Plane State

* **Indicators**

  * State: Healthy, Unhealthy, Not Running

  * Connectivity to Nucleus, RabbitMQ

* **What to watch**

  * Disruptions, extended outages, unstable state changes

* **Targets**

  * Availability >99.9%

  * Health check \<5 sec

## Message processing performance

Measure how efficiently agents process execution-related messages. Detect delays and capacity issues.

**Panels:** Message Duration, Throughput, p95 Response Times

* **Metrics**

  * p95 roundtrip \<2 sec

  * Average message duration \<1 sec

* **Bottlenecks**

  * Long durations: API or processing delays

  * Low throughput: Capacity issues

* **Message types**

  * CREATE, UPDATE, DELETE, GET

## Kubernetes API performance

Monitor how the agent interacts with the Kubernetes API. Spot slow operations and API-related failures.

**Panels:** Kube API Request Duration

* **Targets**

  * p95 \<500 ms (general)

  * CREATE \<2 sec, UPDATE/DELETE \<1 sec, GET \<200 ms

* **Operations**

  * Pod/Service/Config/Namespace management

* **Optimization**

  * Monitor API performance and network latency

  * Adjust RBAC and resource specs

## Resource utilization and scaling

Check agent CPU and memory usage. Identify overuse, leaks, or scaling limits.

**Panels:** Memory and CPU Use vs Requests and Limits

* **Targets**

  * Memory \<80% limit

  * CPU \<70% average, \<90% peak

* **Optimization**

  * Right-size resources

  * Watch for memory leaks

  * Set alerts on usage thresholds

## Next steps

* [Use execution monitoring dashboards](/cloud/admin/operations/grafana-workload-monitoring/execution-monitoring) - Track workload performance and identify issues early, and optimize execution across your deployment.

* [Work with model endpoint monitoring dashboards](/cloud/admin/operations/grafana-workload-monitoring/model-endpoint-monitoring) - Monitor the health, reliability, and performance of your model APIs to detect issues quickly and improve model serving.


# Use execution monitoring dashboards
Source: https://docs.domino.ai/cloud/admin/operations/grafana-workload-monitoring/execution-monitoring

Use Grafana dashboards to assess execution status, startup phases, node pool capacity, resource use, and Compute Environment image pulls.

Use these dashboards to monitor workload execution, spot performance issues early, and optimize performance across your deployment.

## Execution health and status

Use these dashboards to monitor workload execution, spot issues early, and optimize performance across your deployment.

**Panels:** Active Executions, Success Rate, Execution Failures

* **Key metrics**

  * Execution status breakdown (Running, Pending, Preparing, Queued, Finishing)

  * Success rate over 30-minute intervals

  * Failure types (System vs User)

* **What to watch**

  * High Pending: Resource bottlenecks

  * Long Preparing: Image pull or volume mount delays

  * Success rate \<85%: Immediate investigation needed

  * Growing Queued: Scheduler issues

* **Targets**

  * Success rate: 95–100% (healthy), 85–95% (watch), \<85% (investigate)

  * Pending time: \<5 minutes

## Startup performance and timing

Check workload status and failure trends in real time. Use this dashboard to detect execution problems and track system health.

**Panels:** Time to Available, Startup Duration, Time in Checkpoint

* **Phases**

  * NodeAssigned: Resource allocation

  * ImagesPulled: Container setup

  * VolumesMounted: Storage readiness

  * FilesPrepared: Git/file staging

  * ExecutionAvailable: Ready state

* **Optimization tips**

  * Delay in NodeAssigned: Scale node pools

  * Bottlenecks in ImagesPulled: Use smaller images

  * Slow VolumesMounted: Investigate storage

  * Lag in FilesPrepared: Check repo size or network

## Resource capacity and scaling

Track resource availability and scaling behavior. Use this dashboard to adjust pool sizes and avoid resource constraints.

**Panels:** Node Pool Size, Resource Utilization

* **Indicators**

  * Pool size trends

  * Resource usage per pool

  * Auto-scaling effectiveness

* **Strategies**

  * Overprovision spare nodes for fast start

  * Tune pool size for cost/performance balance

  * Adjust scaling thresholds

## Image pull performance

Monitor how long container images take to download. Use this data to optimize images and reduce start up delays.

**Panels:** Compute Environment Pull Statistics

* **Metrics**

  * Pull count per image

  * Average pull duration

* **Performance thresholds**

  * \<30 sec: Optimal

  * 30–120 sec: Investigate

  * 120 sec: Action required

* **Improvements**

  * Use optimized, smaller container images

## Next steps

* [Work with model endpoint monitoring dashboards](/cloud/admin/operations/grafana-workload-monitoring/model-endpoint-monitoring) - Monitor the health, reliability, and performance of your model APIs to detect issues quickly and improve model serving.

* [Run data plane agent monitoring dashboards](/cloud/admin/operations/grafana-workload-monitoring/data-plane-agent-monitoring) - Track the health and performance of Domino Data Plane Agents to monitor execution management and Kubernetes resource activity.


# Use Grafana workload monitoring in Domino
Source: https://docs.domino.ai/cloud/admin/operations/grafana-workload-monitoring/index

Access Grafana workload monitoring to inspect execution activity, Model API performance, Data Plane resources, scaling, and alerts.

Domino provides a dedicated Grafana monitoring interface for monitoring workload execution, model performance, and infrastructure coordination. This monitoring is focused on the applications and jobs running in your Domino environment, but not the underlying infrastructure Domino manages.

Cloud administrators can use these dashboards to:

* Monitor active workloads and execution status.

* Troubleshoot performance issues and startup delays.

* Track model API health, response times, and traffic.

* Observe resource usage and scaling behavior across data planes.

This guide shows you how to access the monitoring interface, interpret key dashboards, and identify issues affecting system performance and reliability.

## Prerequisites

Before continuing, make sure the following are in place:

* **Cloud Administrator**: You are assigned this role in your organization. This is required for access to all workload monitoring dashboards and features.

* **Network Access**: You can reach `https://<your-domino-domain>/grafana-workload` from your local environment or VPN.

* **Grafana Familiarity (optional)**: You have basic knowledge of how to read and interpret Grafana dashboards.

Only users with the Cloud Administrator role can access Grafana workload monitoring. This role grants full access to dashboards, alerting, and monitoring configurations.

## Access the Grafana Workload dashboard

Grafana integrates with Domino authentication. If you are logged into Domino with a Cloud Administrator role, you are automatically authenticated. Otherwise, you see the login screen.

You can open the workload monitoring dashboards in either of two ways:

* From the Domino Admin UI, go to **Admin Panel** > **Reports** > **Customer Grafana Monitoring**.
* Open `https://<your-domino-domain>/grafana-workload` in your browser.

Either entry point opens the workload monitoring dashboards, where you can track execution activity, resource usage, and performance metrics.

## Next steps

* [Use execution monitoring dashboards](/cloud/admin/operations/grafana-workload-monitoring/execution-monitoring) - Track workload performance and identify issues early, and optimize execution across your deployment.

* [Work with model endpoint monitoring dashboards](/cloud/admin/operations/grafana-workload-monitoring/model-endpoint-monitoring) - Monitor the health, reliability, and performance of your model APIs to detect issues quickly and improve model serving.

* [Run data plane agent monitoring dashboards](/cloud/admin/operations/grafana-workload-monitoring/data-plane-agent-monitoring) - Track the health and performance of Domino Data Plane Agents to monitor execution management and Kubernetes resource activity.


# Use model endpoint monitoring dashboards
Source: https://docs.domino.ai/cloud/admin/operations/grafana-workload-monitoring/model-endpoint-monitoring

Use Grafana dashboards and alerts to monitor Model API errors, latency, traffic, authentication, pod health, and resource utilization.

Use these dashboards to monitor the health, reliability, and performance of your model APIs. Identify issues quickly and optimize model serving to ensure production-ready model endpoints that meet your SLA requirements.

As an IT administrator, you can treat model endpoints as first-class production services with comprehensive monitoring, alerting, and troubleshooting capabilities.

<Warning>
  The Cloud Administrator role is required to access Grafana workload monitoring dashboards. Regular users can view basic metrics through the Domino endpoint interface, but advanced monitoring and alerting features require administrator privileges.
</Warning>

## Access the model endpoint monitoring dashboards

As a Cloud Administrator, you can access model endpoint monitoring through two methods:

### Method 1: Direct Grafana access

1. Navigate to the **Admin** section in Domino.

2. Go to **Reports** > **Grafana**.

3. Select **Dashboards** from the Grafana interface.

4. Choose the **Model Endpoints** dashboard.

5. Use the filters to select your specific model ID and version.

### Method 2: From model endpoint details

1. Navigate to **Endpoints** in your Domino Workspace.

2. Select the endpoint you want to monitor.

3. Click **Versions** and select the specific version.

4. On the version details page, click the **Grafana** link (available for Domino administrators).

5. This opens the Grafana dashboard with the model ID and version automatically pre-selected.

Both methods provide access to the same comprehensive monitoring dashboards with detailed metrics and alerting capabilities.

## Model health and reliability

Track status codes and success rates to identify errors, uptime issues, and model failures. Monitor authentication patterns and detect potential security issues or misconfigurations.

**Panels:** Model Performance Summary, HTTP Status Codes, Success Rate, Authentication Status

* **Key metrics**

  * Status code distribution (2xx, 4xx, 5xx)

  * Success rate (excluding 401 responses)

  * Request volumes and patterns

  * Authentication success/failure rates

  * API key usage patterns

* **What to watch**

  * **Success rate \<99%**: Reliability concerns requiring immediate investigation

  * **High 4xx rates**: Client-side misconfiguration or authentication issues

  * **High 5xx rates**: Model or infrastructure failures

  * **Authentication failures**: Potential security threats or credential issues

* **Targets and thresholds**

  * **Success rate**: >99% (excellent), 95–99% (acceptable), \<95% (investigate immediately)

  * **4xx errors**: \<1% of requests

  * **5xx errors**: \<0.1% of requests

  * **Authentication success rate**: >98%

### Setting up alerts for health metrics

Configure alerts to proactively detect issues:

```promql theme={null}
# High error rate alert
sum(rate(http_requests_total{status=~"5.*|4.*"}[5m])) / sum(rate(http_requests_total[5m])) > 0.05

# Authentication failure alert
sum(rate(http_requests_total{status="401"}[5m])) / sum(rate(http_requests_total[5m])) > 0.02
```

## Response time and latency

Measure model responsiveness and detect performance bottlenecks. Use latency percentiles to understand the user experience and identify models that need optimization.

**Panels:** Latency Percentiles, Request Times, Response Time Distribution

* **Key metrics**

  * P50, P90, P95, P99 latency percentiles

  * Upstream vs total response time differentiation

  * Response time trends over time

* **Understanding the metrics**

  * **P50 (median)**: Represents typical user experience

  * **P95/P99**: Captures worst-case scenarios and outliers

  * **Upstream time**: Time spent in model inference

  * **Total time**: Includes network overhead and request processing

* **Focus areas**

  * **High P50**: Core model slowness affecting all users

  * **High P95/P99**: Sporadic performance issues affecting some users

  * **Increasing trends**: Performance degradation requiring intervention

  * **Large upstream vs total gaps**: Network or infrastructure bottlenecks

### Response time alerting

Configure alerts for response time degradation:

```promql theme={null}
# Slow responses alert (95th percentile)
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, model_id, version)) > 2

# Median response time alert
histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, model_id, version)) > 1
```

## Traffic patterns and load

Monitor request volume and distribution to identify spikes, uneven traffic patterns, and scaling needs. Understand usage patterns to proactively prepare for peak loads and optimize resource allocation.

**Panels:** Request Volume, Model Paths, Request Rate, Traffic Distribution

* **Key indicators**

  * Requests per second (RPS) and total request volumes

  * Path-level distribution and endpoint usage patterns

  * Load balancing efficiency across model instances

  * Peak usage times and traffic trends

* **Actionable insights**

  * **Prepare for peak loads**: Identify usage patterns to pre-scale resources

  * **Address skewed distribution**: Balance traffic across model instances

  * **Track growth trends**: Plan capacity based on historical usage

  * **Detect anomalies**: Identify unusual traffic spikes or drops

### Traffic-based alerts

Monitor for unusual traffic patterns:

```promql theme={null}
# High request rate alert
sum(rate(http_requests_total[5m])) by (model_id) > 100

# Traffic spike detection
increase(http_requests_total[10m]) > 1000
```

## Resource utilization and performance

Check pod health, CPU, memory, and network usage to detect resource constraints and failures. Monitor resource efficiency to optimize costs and ensure reliable model serving.

**Panels:** Pods, Container Restarts, CPU Utilization, Memory Usage, Network I/O

* **Critical metrics**

  * CPU usage (average and peak utilization)

  * Memory consumption and allocation patterns

  * Pod restart frequency and reasons

  * Network throughput and bandwidth usage

* **Target thresholds**

  * **CPU usage**: \<70% average, \<90% peak for sustained periods

  * **Memory usage**: \<80% of allocated limits

  * **Container restarts**: \<1 restart per day per pod

  * **Network I/O**: Monitor for saturation during peak traffic

* **Optimization opportunities**

  * Scale resources based on actual usage patterns

  * Address frequent restarts indicating stability issues

  * Monitor network throughput for bottlenecks

  * Right-size resource allocations to reduce costs

### Resource utilization alerts

Monitor resource constraints:

```promql theme={null}
# High CPU usage alert
avg(rate(process_cpu_seconds_total{job="model-endpoints"}[5m])) by (model_id, version) > 0.8

# High memory usage alert
avg(process_resident_memory_bytes{job="model-endpoints"}) by (model_id, version) / avg(container_spec_memory_limit_bytes) by (model_id, version) > 0.8

# Container restart alert
increase(kube_pod_container_status_restarts_total[1h]) > 0
```

## Model-specific performance

Compare performance across models and versions to identify underperformers and optimize resource allocation. Use these insights to make data-driven decisions about model deployment and resource management.

**Panels:** Response Times Table, Response Size, Request Breakdown, Model Comparison

* **Comparative metrics**

  * Latency per model and version

  * Payload sizes and data transfer patterns

  * Request frequency and error rates by model

  * Resource efficiency ratios (requests per CPU/memory unit)

* **Insights for optimization**

  * **Identify slow models**: Spot models consistently performing below thresholds

  * **Track performance regressions**: Compare versions to detect degradation

  * **Allocate resources per model**: Right-size resources based on actual usage

  * **Cost optimization**: Identify high-resource, low-value models

## Troubleshooting common issues

Use these dashboards to diagnose and resolve common model endpoint problems:

### High error rates

* Check the HTTP Status Codes panel for error distribution

* Correlate with recent deployments or traffic changes

* Review authentication metrics for 401 errors

* Examine resource usage for 5xx errors caused by resource exhaustion

### Performance degradation

* Compare current latency percentiles with historical baselines

* Check CPU and memory usage for resource constraints

* Review request patterns for traffic spikes

* Analyze individual model performance for regressions

### Resource issues

* Monitor container restart patterns for stability problems

* Check CPU and memory trending for capacity planning

* Review network I/O for bandwidth limitations

* Examine pod health for infrastructure issues

## Best practices for IT administrators

### Daily monitoring checklist

* Review HTTP status code distributions across all models

* Check resource usage patterns and identify anomalies

* Verify authentication success rates and security metrics

* Monitor alert status and resolve any active incidents

### Weekly performance review

* Analyze performance trends and identify degradation patterns

* Review models with resource inefficiencies

* Check alert history and tune thresholds if needed

* Collaborate with data science teams on optimization opportunities

### Monthly maintenance tasks

* Update alert thresholds based on historical performance data

* Fine-tune dashboards for emerging usage patterns

* Review access logs and security metrics

* Plan capacity based on growth trends

### Alert strategy recommendations

* Set up tiered alerting (warning, critical, emergency)

* Configure different notification channels for different severity levels

* Implement alert correlation to reduce noise

* Document runbooks for common alert scenarios

## Next steps

* [Use execution monitoring dashboards](/cloud/admin/operations/grafana-workload-monitoring/execution-monitoring) to track workload performance and identify issues early, and optimize execution across your deployment.

* [Run data plane agent monitoring dashboards](/cloud/admin/operations/grafana-workload-monitoring/data-plane-agent-monitoring) to track the health and performance of Domino Data Plane Agents to monitor execution management and Kubernetes resource activity.

* [Monitor model endpoints (User Guide)](/cloud/platform-capabilities/features/model-deployment/monitor-model-endpoints) helps data scientists understand how to use monitoring during model development and optimization.


# Operations
Source: https://docs.domino.ai/cloud/admin/operations/index

Find Domino administration resources for monitoring, costs, audit logs, Control Center, disaster recovery, and upgrades and maintenance.

[Monitor Domino and infrastructure](/6.3/admin/operations/monitoring)\
Monitoring Domino involves tracking several key application metrics to reveal the health of the application and provide advance warning of any issues or failures of Domino components.

[Analyze costs with Domino Cost Center (FinOps)](/cloud/admin/operations/finops)\
Domino Cost Center (FinOps) empowers organizations to optimize the performance of their AI projects while managing their cloud expenditures.

[Analyze costs with Domino Cost Center](/cloud/admin/operations/finops)\
Domino Cost Center empowers organizations to optimize the performance of their AI projects while managing their cloud expenditures.

[Use execution logs](/cloud/admin/operations/audit-logs)\
Domino audit logs let you meet the security and audit compliance needs of your business while also enabling access to the logs that help troubleshoot issues.

[Use the Control Center](/cloud/admin/operations/control-center)\
Use the Control Center to see important data about your Domino deployment.

[Use Grafana workload monitoring in Domino](/cloud/admin/operations/grafana-workload-monitoring)\
Use a dedicated Grafana monitoring dashboard for monitoring workload execution, model performance, and infrastructure coordination.

[Disaster recovery](/cloud/admin/operations/disaster-recovery)\
Domino Data are stored and backed up in AWS, as well as on-premises.

[Upgrades & maintenance](/cloud/admin/operations/upgrades-maintenance)\
How Domino Cloud delivers product updates, and what a platform administrator needs to track or act on.


# Projects configuration
Source: https://docs.domino.ai/cloud/admin/platform-configuration/configure-projects

Platform-wide Project defaults, including the quick-start Projects Domino creates for every new user at signup.

Domino creates a quick-start Project for every new user when they sign up. It holds example files that show how to use Domino features, along with a detailed `README`. You can replace that quick-start with Projects of your own, so new users land in content your organization wrote.

## Default Projects for new users

Point Domino at one or more of your own Projects, and every new user owns a copy of each of them at signup, under their own username.

<Steps>
  <Step title="Create the Projects new users should start from">
    Give each one a name, a description, and a `README` that describes what is in it. Any user can own these Projects, but each one must be **Private**.
  </Step>

  <Step title="Record the owner and Project name paths">
    For example:

    ```text theme={null}
    admin-user/getting-started-project
    admin-user/sample-app-project
    ```

    Domino reproduces the Project name for each new user, so with the paths above, every new user owns copies at `new-username/getting-started-project` and `new-username/sample-app-project`.
  </Step>

  <Step title="Set the override configuration record">
    When the Projects are ready, set `com.cerebro.domino.frontend.overrideDefaultProject` to the comma-separated list of paths you recorded:

    ```text theme={null}
    admin-user/getting-started-project, admin-user/sample-app-project
    ```

    On Domino Cloud, contact Domino Support to set this configuration record.
  </Step>
</Steps>


# Configure Domino
Source: https://docs.domino.ai/cloud/admin/platform-configuration/index

Configure deployment-wide defaults for Projects, Compute Environments, data, models, Data Planes, and notifications in Domino Cloud.

Platform configuration covers the settings that apply to the whole deployment rather than to one Project: the defaults new users inherit, the Environments and Data Planes their workloads run on, and the platform-level controls behind deployed models.

<Columns>
  <Card title="Cloud configuration records" href="/cloud/admin/architecture-and-deployment/deployment-configuration/configuration-records">
    Review platform settings and defaults. Only Domino can modify these records on Cloud deployments; contact Domino to request a change.
  </Card>

  <Card title="Projects configuration" href="/cloud/admin/platform-configuration/configure-projects">
    Set the default Projects Domino creates for every new user.
  </Card>

  <Card title="Compute Environments administration" href="/cloud/admin/platform-configuration/manage-compute-environments">
    Curate the Environments your organization builds on, and control how they are created and cached.
  </Card>

  <Card title="Data administration" href="/cloud/admin/data-administration">
    Manage Project files, Datasets, and external data.
  </Card>

  <Card title="Data Planes administration" href="/cloud/admin/platform-configuration/manage-data-planes">
    Register, configure, upgrade, monitor, and archive the Data Planes in a Domino Nexus deployment.
  </Card>

  <Card title="Model deployment administration" href="/cloud/admin/platform-configuration/manage-model-deployments">
    Configure Model Monitoring notifications and access, Domino endpoint resources, and external deployment targets.
  </Card>

  <Card title="Notifications" href="/cloud/admin/platform-configuration/notifications">
    Send in-app messages to users about Domino.
  </Card>
</Columns>


# AMI operations
Source: https://docs.domino.ai/cloud/admin/platform-configuration/manage-compute-environments/cache-images-in-eks/ami-operations

Prepare an Amazon Machine Image for EKS by installing Docker and GPU dependencies, pulling Environment images, and validating GPU access.

The following sections describe how to perform important operations on an EC2 instance to set it up as the template for a new AMI suitable for Domino.

## Install Docker

See the official instructions about how to [install Docker](https://docs.docker.com/engine/install/).

## Pull environment images

To pre-cache environment images, you must run `docker pull` for the base images that those environments are built on, or the built environments from the internal registry itself.

To pull the [Domino Standard Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments) base images, your command would look like this (substituting the version string for the image you want to cache):

```shell theme={null}
docker pull quay.io/domino/base:<desired version>
```

To pull a built image from the Domino internal registry, you must find its URI on the **Revisions** tab in the Environment details page.

For example, to cache revision #9 of the environment shown in the previous screenshot, you would run:

```shell theme={null}
docker pull 100.97.56.113:5000/domino-5d7abf2715f3690007f23081:9
```

## Install NVIDIA Docker 2.0 (GPU AMIs only)

See the official instructions for [ installing the nvidia-docker 2.0 runtime](https://github.com/NVIDIA/nvidia-docker/wiki/Installation-\(version-2.0\)).

## Install GPU drivers (GPU AMIs only)

To use the GPU on a GPU node, install the appropriate driver on the machine image. Domino does not have a requirement for any specific driver version, however, if you want to use a Domino Standard Environment, it must be a version that is compatible with the current version of Cuda shown in standard environments.

View a [compatibility matrix](https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver).

If you’d like to install the GPU drivers manually, you can follow these [instructions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/install-nvidia-driver.html#obtain-nvidia-driver-linux).

To validate that your GPU machine is configured properly, reboot the machine and run the following:

```shell theme={null}
docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi
```

This shows the driver number and GPU devices if installed successfully.

## Change the default Docker runtime (GPU AMIs only)

See the [official instructions from NVIDIA](https://docs.nvidia.com/dgx/nvidia-container-runtime-upgrade/index.html#using-nv-container-runtime) about using the container runtime.

You must restart Docker before this works.


# AMI caching
Source: https://docs.domino.ai/cloud/admin/platform-configuration/manage-compute-environments/cache-images-in-eks/complete-ami-caching-procedure

Cache Environment images in an Amazon Machine Image, update EKS Auto Scaling Group launch configurations, and reduce execution startup time.

1. Determine which AMI you want to use as the base for the new AMI. If you’re performing this operation on an operational Domino node pool, you must use the AMI that’s currently used in the active launch configuration.

   <img alt="The launch configuration name" />

   After you’ve identified the name of the active launch configuration, view its details to see the AMI ID it uses.

   <img alt="The AMI ID" />

2. Launch a new EC2 instance from the base AMI.

3. Connect to the instance through SSH and perform any of the operations listed previously that you want to apply to your new AMI, including pulling any environment images you want to cache.

4. Snap a new AMI from the EC2 instance.

5. Create a copy of the launch configuration currently used by any ASGs you want to switch to using the new AMI.

6. Edit the AMI for the copied launch configuration to be the ID of the new AMI you snapped.

7. For any ASGs that you want to start using the new AMI, switch them over to the new launch configuration.

After you complete the final step, any ASGs you switched to using the new launch configuration start using the new AMI whenever they create nodes. These new nodes have any environment images you pulled onto the AMI template cached, and are fast to start new Domino Runs.


# Cache Environment images in EKS
Source: https://docs.domino.ai/cloud/admin/platform-configuration/manage-compute-environments/cache-images-in-eks/index

Pre-cache Compute Environment images and GPU dependencies in Amazon Machine Images for EKS nodes to reduce execution startup time.

When a user launches a Domino execution, part of the start-up process is loading the user’s environment onto the node that hosts the execution. For large images, the process of transferring the image to a new node can take several minutes. After an image has been loaded onto a node once, it gets cached, and future executions that use the same environment start up faster.

When running Domino on EKS, you can pre-cache popular environments and base images on the Amazon Machine Image (AMI) used for new nodes. This can significantly speed up the start time of executions on new nodes. This topic describes how to create a new AMI with cached environments and configure EKS to use it for new nodes.

In addition to any dependencies required by Kubernetes, your AMI must contain the following:

* Docker

* Cache of Domino’s compute environments

* NVIDIA-Docker 2 (GPU nodes only)

* NVIDIA GPU driver 410+ (GPU nodes only)

* Change the default Docker runtime (GPU nodes only)

For simplicity, Domino recommends that you use the official EKS default AMIs, which come pre-configured with Docker and the GPU tools.

* See the [official EKS AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html) Domino recommends for default compute nodes.

* See the [official EKS AMI](https://docs.aws.amazon.com/eks/latest/userguide/gpu-ami.html) Domino recommends for GPU nodes.

You can also use [Amazon’s build scripts](https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-eks-ami-build-scripts-available-on-github/) to create your own AMI for use with EKS.

## Next steps

* [AMI operations](/cloud/admin/platform-configuration/manage-compute-environments/cache-images-in-eks/ami-operations)

* [AMI caching](/cloud/admin/platform-configuration/manage-compute-environments/cache-images-in-eks/complete-ami-caching-procedure)


# Clean your Environment catalog
Source: https://docs.domino.ai/cloud/admin/platform-configuration/manage-compute-environments/clean-environment-catalog

Review Environment usage, consolidate global Environments, rebuild derived images, and archive unused entries without breaking reproducibility.

It’s inevitable that the number of environments grows. Therefore, you must occasionally review your environment catalog to remove unused environments, update active ones, and consolidate wherever possible. Depending on the size of your organization and your use of Domino, this might be a yearly or quarterly task.

## Review current environment usage

Review all Environments in your deployment

1. In the Admin application, go to **Manage resources > Environments**.

2. Sort the table by the **# of Projects** column to get a quick understanding of which environments are in common use.

3. Click the name of an environment to see its Dockerfile details.

   1. Click the **Projects** and **Models** tabs to see the list of projects and models using the environment. The list also includes the date of the last run for each project and model.

4. Watch for environments that make duplicate changes to global base environments, as well as unused or poorly maintained environments.

## Plan changes to global Environments

After you [review existing environments](/cloud/admin/platform-configuration/manage-compute-environments/clean-environment-catalog), you must plan an updated set of global environments that include the tools and features frequently added by users. It might be as easy as adding a few packages to an existing global environment. You can also create a new global environment when necessary, but Domino recommends erring on the side of larger, more consolidated environments. Doing so makes it easier for your users to choose an environment. It is also easier for you to manage and maintain the collection of environments in your deployment.

Before you execute your plan and change the available global environments, inform your users of the impending changes and solicit their feedback. Explain the changes to existing environments, announce when you create additional ones, and provide recommendations for which environment to use for various projects.

### Rebuild environments after updates

When you change a base environment’s settings, you can decide whether to rebuild all derived environments when the base updates. You can choose which derived environments to rebuild or include new ones with the base update.

Derived environments that auto-update display an indication in the user interface, showing that they are subscribed to that base. Users can keep a specific environment version as **Active** while still receiving updates on the base environment. This lets them test the new version before connecting their Jobs, Apps, and Model APIs to it.

Please note that archived images are not rebuilt automatically.

[Create a Domino Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/3-create-an-environment) has more details about creating and updating base and derived environments.

## Sunset environments

When necessary, you must sunset Environments that are deprecated or no longer used. You can archive these Environments as long as they are not currently set as the [Project Default Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/6-set-a-default-environment) for any Projects.

<Tip>
  An alternative to archiving Environments is to rename them to indicate that they are End-of-Life.
</Tip>

Archiving an Environment does the following:

* Removes it from the Environments menu so users can no longer select it.

* Encourages adoption of new, up-to-date, and consolidated Environments.

It is important to note the following about archived Environments:

* Historical runs still reference archived Environments, so archiving does not break reproducibility.

* Environments can be unarchived if necessary.


# Create Environments from external custom images
Source: https://docs.domino.ai/cloud/admin/platform-configuration/manage-compute-environments/create-envs-from-external-custom-images

Configure the Domino image builder to authenticate with an external registry, update its Kubernetes secret, and verify custom Environment builds.

This topic describes how to configure Domino so that you can create Environments based on custom images hosted in external, authenticated image registries.

This topic is useful if you want to:

* Enable your users to access an internal registry of images to use in Domino.

* Enable your users to access an authenticated third-party registry containing images related to your data science workflows. For example, the NVIDIA Container Registry (NGC).

## Pre-requisites

* Access to the Kubernetes cluster where Domino is running and permissions to read and edit secrets in the Domino `compute` namespace.

* Access to and familiarity with using `kubectl`. You must be comfortable editing Kubernetes objects.

* Access to Basic Authentication (username and password) credentials for the authenticated image registry you want to configure.

* The registry must support HTTPS. If it doesn’t, contact support or your field engineer.

<Note>
  The supplied registry credentials are configured Domino-wide. Therefore, they must be safe for use by all Domino platform users, not just specific users. Domino recommends creating a service account in the image registry specifically designated for use with the Domino platform.
</Note>

## Add authentication details

Use this procedure to add authentication details to your registry secret. Domino’s image builder uses this secret to authenticate with registries during environment builds.

1. **Determine the registry authentication secret name and namespace**

   1. Edit the secret. The default location for the secret is in the Domino `compute` namespace, under the secret name, `domino-quay-repos`. However, another user might have changed the location or secret name.

   2. Because the secret name and namespace values can be changed, go to **Platform Settings > Configuration records** to confirm that you do not have `com.cerebro.domino.builder.remoteRegistryCredentials.secretName` or `com.cerebro.domino.builder.remoteRegistryCredentials.secretNamespace` set. If either of these are set use those values instead of the default values listed.

2. **Fetch the current secret value**

   1. Using the secret name and namespace from the previous steps, run the following command:

      ```shell theme={null}
      kubectl get -n <secret namespace> secret <secret name> -o jsonpath='{.data}'
      ```

      Your output should look similar to the following: `{".dockerconfigjson":"…​"}`

      The value of `.dockerconfigjson` is the value in the double quotes ("") after `.dockerconfigjson`.

      Retain this value as it is used in subsequent steps.

   2. Using the value you retained in the previous step, run the following command:

      ```shell theme={null}
      echo '<the copied value>' | base64 --decode
      ```

   3. Copy the output of this command into a text editor of your choice. Ensure you do not copy the final `%` symbol. The value you copy should look similar to this: `{"auths": {"[quay.io]": {"username": "…​", "password": "…​", "email": "."}}}`

3. **Add the new registry authentication**

   1. Use your text editor to add your new registry to the previous content. You’ll want to add it at the same level as the existing repositories within the `auths` dictionary.

   2. The `key` is the repository host. For example `nvcr.io` for NGC. The value of this key is a dictionary with a username and password entry with your username and password.

   3. You can use this template, replacing the values with the pre-existing values and your new values:

      ```json theme={null}
      {"auths": {"quay.io": {"username": "...", "password": "...", "email": "."}, "<new registry>": {"username": "...", "password": "..."}}}
      ```

      <Note>
        If you are using an NGC container, you can use this template with the required username:

        ```json theme={null}
        &#123;"auths": &#123;"quay.io": &#123;"username": "...", "password": "...", "email": "."&#125;, "nvcr.io": &#123;"username": "$oauthtoken", "password": "..."&#125;&#125;&#125;
        ```
      </Note>

      You might want to keep your the text editor open until all the steps are complete. This saves time if you must debug an error later.

4. **Encode the new secret content**

   1. Copy the new text from your editor. Ensure it is all on one line and does NOT have a newline at the end.

   2. Using the text copied from your text editor, run this command:

      ```shell theme={null}
      echo -n '<your copied content>' | base64 -w 0`.
      ```

      You MUST use single quotes to properly escape the double quotes in the copied content.

   3. Copy the output of that command. Ensure you do not to copy the trailing `%` character.

5. **Edit the secret**

6. If you want to edit the secret interactively, run:

   ```shell theme={null}
   kubectl edit -n <secret namespace> secret <secret name>
   ```

   1. In the system editor, remove the content after `.dockerconfigjson` replacing it with the previous encoded text.

   2. If you want to edit the secret non-interactively, run:

      ```shell theme={null}
      kubectl get -n <secret namespace> secret <secret name> -o json | jq '.data[".dockerconfigjson"]="<your copied content>"' | kubectl apply -f -
      ```

7. **Validate your edits**

   1. Repeat the step where you’ve fetched the current secret value before and ensure the decoded content has your updates. Carefully validate the dictionary structure.

8. **Verify your changes**

   1. Go to Domino and create a new Domino Environment in the Environments tab.

   2. Use an image URI from the new authenticated registry as the custom base image for the Environment.

   3. Watch the build progress in the Environment revisions tab. Ensure the build succeeds.

   4. If the build succeeds, delete all credentials you saved on your system in the text editor.

      If the build fails, use the revision build logs to determine if the error is related to authentication. Double check that your credentials are correct and validate that you edited the secret properly, repeating the previous steps as necessary.

      If your edits do not work and you can not find any errors, see the previous steps to edit the secret but use the original encoded content that you retained in the step where you fetched the current secret value.


# Manage Compute Environments
Source: https://docs.domino.ai/cloud/admin/platform-configuration/manage-compute-environments/index

Curate Compute Environments by limiting global images, maintaining clear Dockerfiles, sharing ownership, caching images, and removing old entries.

This topic covers best practices for managing compute environments. As a Domino administrator, you must curate the environments used by your organization. A proactive approach to environment management can prevent sprawl, avoid duplicate environments, and equip users with the tools they have to succeed in Domino. You must find the balance between giving users the freedom to be agile in development, while also maintaining enough control that you don’t end up with duplicate or unnecessary environments. Because there is no limit to the number of environments that administrators and users can create, you must manage this to ensure that your users can find what they need.

Domino recommends these best practices:

**Limit global environments**\
The benefits of focusing on a small number of global environments with broad applications outweighs the benefits of creating many niche global environments. When a user requests an environment with new capabilities, consider whether you can add their requested features to an existing global environment instead of creating a new one.

**Keep global images up-to-date and comprehensive**\
Strive to have global images that cover the majority of users' needs. Users should only have to make minor additions to global environments when they create their own environments, such as installing a specific version of a package. You don’t want users to have to re-install Python or make other major changes, as this results in a bloated and poorly performing environment.

**Use clear, descriptive names for your environments**\
A common reason why environments get duplicated is that a user cannot tell whether an existing environment meets their needs. Clear, descriptive names on all environments makes the entire catalog comprehensible to new users and administrators, and makes Domino easier to work with and maintain.

**Add comments to the Dockerfile for each environment**\
You can add comments to a Dockerfile like this:

```shell theme={null}
# This is a comment that can provide a helpful description of the code to follow
RUN echo 'This is an executed Dockerfile instruction'

# Here's a block that installs Ruby
RUN
  apt-get update &&
  apt-get install -y ruby
```

Each section must have a clear heading and comments to explain its purpose and implementation.

**Share responsibility for environment management**\
If you have multiple teams or departments doing separate work in Domino, they must maintain their team-specific environments. Find an advanced user in each team and make them a deputy for environment management. This person must be responsible for planning and understanding the environments their team needs, and must work with you on implementation. This reduces the administrators' workload and ensures that environments are designed by someone with context about what users need.

**Cache global environments on the executor machine image**\
Cache your global environments in your executor template machine image. This ensures that each new executor starts up with the base Docker image for any environment already cached. If users are setting up environments that have base images that differ significantly from what is cached on the machine image, it can lead to long pull times when they launch executors. Contact [Domino Support](mailto:support@dominodatalab.com) for help modifying your machine image.

**Clean up old or poorly maintained environments**\
Create a culture of tidiness around environment creation and management. Enforce a standard of quality in naming and Dockerfile commenting, and be assertive about pruning unnecessary environments. See [Clean Your Environment Catalog](/cloud/admin/platform-configuration/manage-compute-environments/clean-environment-catalog) for a walk-through.

For more information, have a look at these topics:

* [Use Restricted Environments](/cloud/admin/platform-configuration/manage-compute-environments/restricted-environments): Enable `Restricted Environments` in your Domino instance, mark your Project as `Restricted`, and prevent package installation at runtime.

* [Clean your Environment catalog](/cloud/admin/platform-configuration/manage-compute-environments/clean-environment-catalog): Review current Environment usage, plan changes to global Environments, and sunset Environments.

* [Cache Environment images in EKS](/cloud/admin/platform-configuration/manage-compute-environments/cache-images-in-eks): Pre-cache popular Environments and base images on the Amazon Machine Image (AMI) used for new nodes.

* [Create custom Environments](/cloud/admin/platform-configuration/manage-compute-environments/create-envs-from-external-custom-images): Configure Domino so that you can create Environments based on custom images hosted in external, authenticated image registries.


# Use Restricted Environments
Source: https://docs.domino.ai/cloud/admin/platform-configuration/manage-compute-environments/restricted-environments

Restrict Compute Environment revisions and Projects to approved packages, and block runtime installs with R, host, and pip configuration.

You may want to ensure that a Compute Environment remains frozen and unmodifiable to guarantee that only certain packages are used for modeling or analysis. This can be important to ensure compliance with internal policies or external regulations.

Domino lets you designate Compute Environments as “Restricted”. Environment Restriction ensures that users can’t use an Environment that has been modified after it has been certified as restricted.

Restricted Environments provide the following benefits:

* **Improve security and reduce compliance risk** by enforcing sensitive workloads to use only approved environments.

* **Reduce costs and management overhead** by providing flexibility to run sensitive and non-sensitive workloads on the same Domino deployment, without compromising on compliance. By marking certain environments as "Restricted", admins can ensure that they can enforce compliance for sensitive workloads while allowing data scientists the flexibility to use the latest innovations (new tools and libraries) on the same Domino deployment.

## Use restricted environments

To enable Restricted Environments in your Domino instance, please reach out to Domino Support.

Only an admin can mark a revision of an Environment as Restricted.

To designate a Compute Environment as Restricted, go to **Environments**, select the environment and mark a specific Revision of the Environment as Restricted.

You can also quickly set the latest environment revision to Restricted from the environment’s **Overview** tab by clicking the **Restricted** checkbox.

Only a single Environment revision can be Restricted. If you mark another revision as Restricted, the previous revision becomes unrestricted.

Once you mark a revision as Restricted, that Environment can’t be unrestricted. You can, however, choose which revision is the Restricted (that is, frozen) revision. This limitation ensures that anyone using the environment is always using a revision that was explicitly designated as Restricted.

Even when an Environment is Restricted and changes to its Dockerfile aren’t permitted, data scientists can still install packages at runtime. To prevent these types of modifications, see [prevent dynamic installation](#prevent-package-installation-at-runtime).

## Restricted Projects

To enforce that a Project only uses Restricted Environments, mark the Project as `Restricted` when you create the Project.

<Warning>
  A Restricted project can only be classified as **Restricted** at the time of creation. Existing Projects cannot be classified as **Restricted**. Once a Restricted project is classified as **Restricted**, it cannot be reverted to unrestricted. This ensures that every action in a Restricted project uses a Restricted Environment at all times.
</Warning>

When running workloads from a Restricted Project, Domino limits your choice of Compute Environment to Restricted Environments for all actions including but not limited to:

* Start a manual job run.

* Start a scheduled job run.

* Start a Workspace.

* Rerun a Job.

* Start an App.

* Start a Launcher.

### Copy a restricted Project

When a Restricted Project is copied, the copied Restricted is also classified as Restricted. This is especially useful when the source Restricted is a [Restricted template](/cloud/platform-capabilities/core-concepts/projects/manage-projects#copy-a-git-based-project), since it ensures that Projects created from a template are Restricted if the template is Restricted.

## Prevent package installation at runtime

Even when an Environment is frozen and changes to its Dockerfile aren’t permitted, data scientists may still install packages at runtime through various ways (for example, running a `pip` command or installing an R package from CRAN). To prevent these types of modifications, we recommend adding the below commands in your Environment’s Dockerfile. These commands prevent common ways of installing packages at runtime.

<Warning>
  These examples are not meant to be exhaustive. We highly recommend that you include additional instructions depending on your organization’s policies and technologies.
</Warning>

### Set read-only libpath and block well known R package repositories

Add the following lines to the compute environment’s Dockerfile to set the `R_LIBS_USER directory` to a folder the Domino user can’t write to.

```shell theme={null}
RUN chmod 755 /usr/local/lib/R/site-library &&
    sed -i 's/^R_LIBS_USER/#&/' /usr/local/lib/R/etc/Renviron
```

### Overwrite package install functions

Add the following lines to the compute environment’s Dockerfile to modify the package install functions to empty functions that produce a message that the environment is frozen.

```shell theme={null}
RUN echo 'assign("utils::install.packages", function ()  cat("
 Frozen R environment, no package installation allowed

"), envir = globalenv())' >> /home/domino/.Rprofile &&
    echo 'assign("install.packages", function (pkg)  cat("
 Frozen R environment, no package installation allowed

"), envir = globalenv())' >> /home/domino/.Rprofile &&
    echo 'assign("devtools::install_github",  function () cat("
 Frozen R environment, no package installation allowed

"), envir = globalenv())' >> /home/domino/.Rprofile &&
    echo 'assign("install_github",  function (pkg) cat("
 Frozen R environment, no package installation allowed

"), envir = globalenv())' >> /home/domino/.Rprofile &&
    echo 'assign("devtools::install_gitlab",  function () cat("
 Frozen R environment, no package installation allowed

"), envir = globalenv())' >> /home/domino/.Rprofile &&
    echo 'assign("install_gitlab",  function (pkg) cat("
 Frozen R environment, no package installation allowed

"), envir = globalenv())' >> /home/domino/.Rprofile &&
    chown domino: /home/domino/.Rprofile
```

### Block package repos by routing package repo requests to loopback address

The following pre-setup script modifies the `/etc/hosts` file to re-route requests to a popular package repo back to localhost.

```shell theme={null}
echo "127.0.0.1 packagemanager.rstudio.com" >> /etc/hosts
echo "127.0.0.1 cran.r-project.org" >> /etc/hosts
echo "127.0.0.1 cran.microsoft.com" >> /etc/hosts
echo "127.0.0.1 bioconductor.org" >> /etc/hosts
```

### Block pip installs by routing pip to a non-existent URL

One way to disable dynamic installation of packages from within a Workspace or a Job is to set the global index URL in the pip configuration to a non-existing URL:

```shell theme={null}
pip config set global.index-url [https://path/to/a/non-existing/URL]
```

## Limitations & caveats

* If a Job is rerun and the previous Environment revision is no longer Restricted, the rerun uses the current Restricted revision of the Environment. If there is no Restricted revision available, the rerun fails.

* Scheduled runs use the Restricted Environment revision at the time of scheduling. If the Environment revision changes before the scheduled run executes, it continues to use the revision from the time of scheduling.


# Data Planes administration
Source: https://docs.domino.ai/cloud/admin/platform-configuration/manage-data-planes

Register, configure, upgrade, monitor, and archive the Data Planes in a Domino Nexus deployment.

A Data Plane is a compute location where Domino runs workloads. A [Domino Nexus](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture) deployment can run several, in different regions or clouds. Manage all of them from the admin portal at **Manage resources** > **Data Planes**.

## Register a Data Plane

Registration connects a Data Plane cluster that is already installed to your control plane, exchanging configuration data and cryptographic keys between the two. Contact your Domino Admin if you need help with the installation.

Before you start, confirm you have admin access to the control plane and `kubectl` access to the Data Plane cluster.

<Steps>
  <Step title="Register the Data Plane in the Domino UI">
    In the admin portal, go to **Manage resources** > **Data Planes** > **Register Data Plane**, then set these fields:

    | Field | What it sets |
    | :- | :- |
    | **Name** | An arbitrary, human-readable name for the Data Plane. |
    | **Namespace** | The Kubernetes namespace to install the Data Plane into on the Data Plane cluster. This namespace is created if it does not already exist. |
    | **Storage Class** | The Kubernetes StorageClass name to use for execution volumes in this Data Plane. If blank, it defaults to the control plane's storage class specified by the `com.cerebro.domino.computegrid.kubernetes.volume.storageClass` configuration record (default: `dominodisk`). |
    | **Data Plane Hostname** | The subdomain component of the DNS name used for the Data Plane. If the control plane is `example.com` and the Data Plane is `dp.example.com`, enter `dp`. Append `:` and a port number if your Data Plane load balancer runs on a port other than 443. See [Enable a Data Plane for Workspaces and Datasets](#enable-a-data-plane-for-workspaces-and-datasets). |

    Three more fields sit behind **Advanced**:

    | Advanced field | What it sets |
    | :- | :- |
    | **Disable File Sync** | Turns off Domino File System (DFS) file sync in this Data Plane. See [Disable file sync](#disable-file-sync). |
    | **Enable Istio** | Indicates that Istio runs in this Data Plane's cluster. Data Planes with Istio *must* have this set; setting the Helm value alone is not sufficient. |
    | **Override S3 Endpoint URL** | Overrides the default S3 endpoint URL for this Data Plane. |

    Click **Done**.
  </Step>

  <Step title="Generate a Data Plane token">
    When registration is complete, create a bootstrap token:

    1. In the admin portal, go to **Manage Resources** > **Data Planes**.
    2. In the **Actions** column, click **Setup** for the Data Plane you are registering.
    3. Click **Load bootstrap token**.
    4. When the token loads, click **Copy token**.

    <Note>
      The token is valid for *10 minutes*. If you exceed the 10 minute window, load a new token through the same steps.
    </Note>
  </Step>

  <Step title="Run the bootstrap script">
    Make sure your default Kubernetes context is the cluster you want to deploy the Data Plane into.

    1. Obtain the Kubernetes context of the Data Plane:

       ```shell theme={null}
       tsh login && tsh kube login <dp-deploy-id>
       ```

    2. Download the `bootstrap-data-plane.sh` script from [the Nexus repository on GitHub](https://github.com/dominodatalab/nexus/blob/main/bootstrap-data-plane.sh) and save it as an executable file:

       ```shell theme={null}
       curl -Sso bootstrap-data-plane.sh \
         https://raw.githubusercontent.com/dominodatalab/nexus/refs/heads/main/bootstrap-data-plane.sh && \
         chmod +x bootstrap-data-plane.sh
       ```

    3. Run the script, replacing `<token>` with the token you copied:

       ```shell theme={null}
       ./bootstrap-data-plane.sh <token>
       ```

    The script reports that it created a secret and exits. The Data Plane becomes healthy in the admin portal within a couple of minutes.
  </Step>
</Steps>

### Verify the registration

In the admin portal, go to **Manage resources** > **Data Planes** and confirm the Data Plane reports **Healthy**. From the Data Plane cluster, confirm every node is `Ready` and every pod is `Running` or `Completed`:

```shell theme={null}
kubectl get nodes
kubectl get pods -n domino-compute
```

### Troubleshoot the registration

<AccordionGroup>
  <Accordion title="The token expired">
    The bootstrap token is valid for 10 minutes. If more time passed between copying it and running the script, load a new token and run the script again.
  </Accordion>

  <Accordion title="The Data Plane stays unhealthy">
    Read the agent logs, which name the reason the agent has not completed its handshake:

    ```shell theme={null}
    kubectl logs -n domino-compute -l app.kubernetes.io/name=data-plane-agent
    ```
  </Accordion>

  <Accordion title="The keys expired after a long outage">
    A Data Plane that stayed offline for more than 4 days can have expired security keys. Force a re-bootstrap:

    ```shell theme={null}
    ./bootstrap-data-plane.sh --force <token>
    ```

    Only use `--force` when the agent logs report key expiration. If they report `waiting...`, run the script without it.
  </Accordion>
</AccordionGroup>

<Warning>
  Never delete or modify the `data-plane-config` or `data-plane-config-extra` ConfigMaps. Recovering a Data Plane after either one is lost is extremely difficult.
</Warning>

## Enable a Data Plane for Workspaces and Datasets

Workspaces need a direct connection from the user's browser to the Data Plane, so a Data Plane that serves interactive workloads needs its own hostname, load balancing, and TLS.

### Configure the hostname

Serve the Data Plane from a *subdomain* of the domain used for the control plane. If users connect to Domino at `example.com`, then Data Planes must be served from `data-plane.example.com`.

### Configure load balancing

Resolve that hostname to a load balancer that routes traffic to port 8080 on pods with this label selector:

```yaml theme={null}
app.kubernetes.io/component: auth-proxy
app.kubernetes.io/instance: auth-proxy
app.kubernetes.io/name: auth-proxy
```

Use a `NodePort` service plus a load balancer on-premises, or a `LoadBalancer` service type in the major cloud providers.

### Configure TLS

Users must connect to Data Planes over TLS (HTTPS). If a load balancer routes traffic to the Data Plane, the simplest option is to configure that load balancer to serve valid TLS certificates for the domain.

To serve certificates that you provide from the Data Plane itself:

1. Create a Kubernetes secret containing the certificates:

   ```shell theme={null}
   kubectl create secret tls custom-certs -n <data plane namespace> --cert=<cert file> --key=<key file>
   ```

2. Set this value when you deploy the Data Plane Helm chart:

   ```shell theme={null}
   --set auth-proxy.config.nginx.tlsSecretName=custom-certs
   ```

### Configure Datasets

Datasets on a remote Data Plane need the volume operator and the filetask operator enabled under `release_overrides` in the Data Plane's configuration file. Ask your Domino Admin to apply these if you do not manage that file:

```yaml theme={null}
release_overrides:
  data-plane:
    chart_values:
      enable_volume_operator: true
      enable_filetask_operator: true
```

## Edit a registered Data Plane

Go to **Manage resources** > **Data Planes**, click the three-dot menu next to the Data Plane, and select **Edit**. What you can change varies by field:

| Field | What changing it means |
| :- | :- |
| **Name** | Change it freely. Tell your users, since they select Data Planes by name. |
| **Namespace** | Migrating a Data Plane to a new namespace is technically possible but not generally supported. Contact Domino for support. |
| **Storage Class** | Does not affect running executions, but applies to new ones. This can disrupt existing Workspaces. |
| **Hostname** | Affects running executions. Re-run the [registration setup](#register-a-data-plane) for the Data Plane, then restart the `auth-proxy` pod in the Data Plane namespace. |
| **Disable File Sync** | Change it at any time. It applies to newly launched workloads. |
| **Enable Istio** | Enabling or disabling Istio support on an existing Data Plane is possible but not trivial. Contact Domino for support. |

## Disable file sync

When data must not leave a specific Data Plane, disable file sync on it to prevent anyone pushing data from that Data Plane to the control plane or to another Data Plane.

On a Data Plane with file sync disabled:

* Users can pull Project files.
* Users can pull and push any mounted Git repositories.
* Users cannot push changes to the Domino File System (DFS). Domino warns before starting an execution that cannot push to DFS, and the job history records that no artifacts or output files were captured because the execution ran in a restricted Data Plane.

To disable file sync:

1. In the admin portal, go to **Manage resources** > **Data Planes**.
2. Click the three-dot menu next to the Data Plane and click **Edit**.
3. Click **Advanced**.
4. Select **Disable File Sync**, then click **Save**.

## Hardware Tiers for Data Planes

Hardware tiers are how users reach a Data Plane. Every hardware tier is assigned to one Data Plane, so granting a user access to a hardware tier grants them access to run workloads on that Data Plane with the resources the tier defines.

Create and restrict those tiers from [Manage a hardware tier](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/manage-hardware-tier), which covers the Data Plane field on the tier form, the node labels a Data Plane's nodes need, and how to limit a tier to specific organizations.

## Upgrade a Data Plane

Keep your remote Data Planes in sync with the control plane to maintain compatibility and access new features. A Data Plane upgrade replaces the `fleetcommand-agent` and Domino services running in the Data Plane cluster.

On Domino Cloud, most Data Plane upgrades run from the admin portal. If the UI Upgrade action is unavailable for your Data Plane, Domino Support performs the upgrade for you.

### Before you start

* The control plane must already be running the target version. Running a Data Plane newer than its control plane is not supported.
* Check [version compatibility](#version-compatibility) below.
* Notify users in the affected regions about a maintenance window. Ongoing executions can typically continue, but stopping them avoids lost work if reconciliation surfaces an issue.

### Version compatibility

Starting with Domino 6.1, control planes and Data Planes are [compatible](/cloud/admin/architecture-and-deployment/kubernetes-compatibility#compatibility-matrix) across two regular releases.

A control plane is compatible with the current and one past regular release of the Data Plane. For instance, control plane `6.2.x` is compatible with Data Planes `6.2.x` and `6.1.x`.

<Warning>
  Upgrade the control plane before any Data Plane. Running a Data Plane version newer than the control plane is not supported.
</Warning>

When a Data Plane's version falls outside that range, the admin portal reports it:

| Status | What it means |
| :- | :- |
| Degraded | The Data Plane runs an older version. New executions are not permitted until you upgrade it. |
| Error | Communication between the Data Plane and the control plane is failing. |
| Disconnected | The Data Plane is not reachable. |

### Upgrade from the UI

Use the Domino UI when the Upgrade action is available. The action requires both planes to be running Domino 6.1.2 or later, and the Data Plane to have online access to the deployment artifacts.

1. In the admin portal, go to **Manage resources** > **Data Planes**.
2. Identify the Data Plane that needs an upgrade by looking for a status message indicating "Data Plane is running an older version" or "Data Plane is running a different version".
3. Click the three vertical dots next to the Data Plane.
4. Select **Upgrade** from the dropdown menu.

The control plane instructs the Data Plane's operator to reconcile against the target version. Any `release_overrides` set on the Data Plane's Domino custom resource are preserved.

### When the UI upgrade is unavailable

The Upgrade option is greyed out when the Data Plane already matches the control plane's version, when it runs 6.1.0 or 6.1.1, or when the upgrade path is not supported from the UI. Contact the [Domino Support](mailto:support@dominodatalab.com) team. They coordinate the manual upgrade against your Data Plane cluster.

Have the following ready when you file the request:

* The Data Plane's current version and the target version.
* Whether the Data Plane cluster is air-gapped.
* Any `release_overrides` currently applied to the Data Plane. These live under `.spec.config.release_overrides` on the Data Plane's Domino custom resource and are preserved across upgrades, but Support may need to review them for schema changes.

### About `release_overrides`

`release_overrides` live under `.spec.config.release_overrides` on the Data Plane's Domino custom resource. They override Helm chart values for the services that the `fleetcommand-agent` deploys into the Data Plane. A typical shape, adding CPU requests and limits to a service:

```yaml theme={null}
spec:
  config:
    release_overrides:
      domino-connect:
        chart_values:
          resources:
            requests:
              cpu: "2"
            limits:
              cpu: "4"
```

Both the UI Upgrade button and Support-driven upgrades read the existing `.spec.config`, so overrides remain in place across upgrades. There is no need to re-apply them from scratch.

<Warning>
  Starting with the Helm 4 upgrade, chart values are applied to Kubernetes with server-side apply, so any mistyped or invalid field in `release_overrides` now fails the upgrade instead of being silently ignored. If you want to add or change an override, work with Domino Support to validate the shape before the upgrade runs.
</Warning>

### Verify the upgrade

1. In the admin portal, go to **Manage resources** > **Data Planes** and confirm the Data Plane is marked **Healthy** and its reported version matches the target.
2. Start a Workspace or run a job pinned to this Data Plane to confirm Workspace connectivity, ingress, and any hardware tiers still behave as expected.
3. If Domino Support drove the upgrade, confirm with them that no `HelmRelease` resources landed in a stalled state during reconciliation.

If the Data Plane is unhealthy after the upgrade, read the `data-plane-agent` logs and, if they report expired keys, re-bootstrap it. See [Troubleshoot the registration](#troubleshoot-the-registration) for both commands.

## Monitor a Data Plane

Check a Data Plane's health, and see which Kubernetes nodes, user executions, and Workspaces are running in it, from the admin portal:

| To see | Go to | Where the Data Plane appears |
| :- | :- | :- |
| Data Plane health | **Manage resources** > **Data Planes** | The status marks a Data Plane **Healthy** when it is available for use. |
| Kubernetes nodes across all your clusters | **Reports** > **Infrastructure** | The column on the right indicates the Data Plane that each node is in. |
| All running user executions | **Reports** > **Executions** | The rightmost column indicates the Data Plane for the execution. |
| All durable Workspaces, running and stopped | **Reports** > **Workspaces** | The rightmost column indicates the Data Plane for the Workspace. |

Data Plane health is a simple check that the Data Plane is contactable. If a Data Plane indicates `Error`, `Disconnected`, or `Degraded`, communication between the Data Plane services and the control plane may have a problem. Contact Domino for help troubleshooting.

<Note>
  Some of the node detail views are not yet implemented for nodes in remote Data Planes.
</Note>

## Archive a Data Plane

<Warning>
  Archiving a Data Plane effectively deletes it. An archived Data Plane cannot be un-archived.
</Warning>

1. [Stop all executions](/cloud/platform-capabilities/core-concepts/workspaces/manage-workspaces) in the Data Plane.
2. [Delete all Workspaces](/cloud/platform-capabilities/core-concepts/workspaces/manage-workspaces) in the Data Plane.
3. [Archive all hardware tiers](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/manage-hardware-tier#archive-a-hardware-tier) in the Data Plane.
4. In the admin portal, go to **Manage resources** > **Data Planes**.
5. Click the three-dot menu next to the Data Plane and select **Archive**.
6. Click **Archive Data Plane**.
7. Use Helm to delete the `data-plane` release in the Data Plane namespace on the Data Plane Kubernetes cluster.

## Related

* [Domino Nexus hybrid architecture](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture)
* [Manage a hardware tier](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/manage-hardware-tier)
* [Kubernetes compatibility](/cloud/admin/architecture-and-deployment/kubernetes-compatibility)


# Model deployment administration
Source: https://docs.domino.ai/cloud/admin/platform-configuration/manage-model-deployments

Platform settings behind deployed models: Model Monitoring notifications and access, Domino endpoint scaling, and external deployment targets.

Once practitioners deploy a model, platform administration decides who receives Model Monitoring alerts, who can open Model Monitoring at all, how much compute Domino endpoints get, and which external accounts models can be deployed to.

## Model Monitoring notifications

Users can send notifications to recipients about thresholds that are breached during scheduled checks. Domino sends those notifications by email, so the email server must be configured before any of them go out.

<Steps>
  <Step title="Set the email server configuration records">
    In the admin portal, go to **Platform settings** > **Configuration records**, then set the necessary keys.
  </Step>

  <Step title="Restart services">
    Click **here** in the message at the top of the page to restart services.
  </Step>
</Steps>

Once the server is configured, users choose where notifications go:

* [Notification channels](/cloud/platform-capabilities/features/monitoring/7-monitor-settings/3-notification) covers the channels available.
* [Set up notifications](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/5-set-up-notifications) covers a specific Domino endpoint.
* [Set up notifications for a Model Monitor](/cloud/platform-capabilities/features/monitoring/4-set-up-model-monitor/7-set-up-notifications-mm) covers a Model Monitor.

## Model Monitoring access control

Access to Domino Model Monitor is granted by membership of the `dmm-users` group in Keycloak. Use the Keycloak UI for a handful of users, and the access control script for a list of them.

<Tabs>
  <Tab title="Keycloak UI">
    1. Open the admin UI at `https://<deployment-url>/auth/admin` and log in to Keycloak.

    2. Select the **DominoRealm** realm from the dropdown:

       <Frame>
         <img alt="Navigate to the realm" />
       </Frame>

    3. Go to the **Users** tab, find the user (you may need to click **View all users**), and click **edit**.

       <Frame>
         <img alt="Find and edit the user" />
       </Frame>

    4. Go to the **Groups** tab.

       To grant access, click **Available Groups** > **dmm-users**, then click **Join**.

       To revoke access, click **Group Membership** > **dmm-users**, then click **Leave**.

       <Frame>
         <img alt="Manage access for groups" />
       </Frame>

    5. Repeat for each user.
  </Tab>

  <Tab title="Access control script">
    <Warning>
      If there is a high volume of users, adjust the Keycloak **AccessToken** settings in the UI so the script runs uninterrupted:

      1. Log in to Keycloak.

      2. Go to **Realm Settings** in the **Master** realm.

      3. Click the **Tokens** tab.

      4. Increase the **Access Token Lifespan** to something sufficiently large.

      5. Save the changes, then run the script.

      Undo the changes after the script finishes.

      <Frame>
        <img alt="Manage access token lifespan" />
      </Frame>
    </Warning>

    1. To use the [Domino Model Monitor Access Control Script](/attachments/5.5/keycloak_dmm_access_control.sh), set environment variables based on your deployment:

       * Domino host URL (`DOMINO_HOST`)

       * Keycloak admin username (`KEYCLOAK_ADMIN_USERNAME`)

       * Keycloak admin password (`KEYCLOAK_ADMIN_PASSWORD`)

       * The Keycloak realm (`KEYCLOAK_REALM`) defaults to `DominoRealm` and can be overridden if needed.

       Replace the placeholders and run these commands in the terminal:

       ```shell theme={null}
       export DOMINO_HOST=https://placeholder.domino.tech
       export KEYCLOAK_ADMIN_USERNAME=placeholder_username
       export KEYCLOAK_ADMIN_PASSWORD=placeholder_password
       ```

       Make sure that the trailing "/" is not included in the `DOMINO_HOST` variable.

    2. Run the script with a list of usernames and the command to execute. The command is either `GRANT` or `REMOVE`.

       ```shell theme={null}
       ./keycloak_dmm_access_control.sh '("foo", "bar")' GRANT
       ```

       Check the spelling of every username. If a username is misspelled or does not exist, the script silently skips that user and continues with the rest.
  </Tab>
</Tabs>

## Domino endpoint Hardware Tiers

Domino endpoints often have different requirements from Workspaces and Jobs, so Domino keeps a separate class of hardware tier for them. Domino endpoint tiers and regular hardware tiers are not interchangeable: Domino endpoints can only use Domino endpoint tiers, and Domino endpoint tiers cannot be used outside Domino endpoints.

To create one:

1. From the admin home page, go to **Advanced** > **Hardware Tiers**.

2. Click **New** to create a hardware tier, or click **Edit** to modify an existing one or set a default.

3. Select the hardware tier values. See [Manage a hardware tier](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/manage-hardware-tier) for what each field controls.

4. Select **Is Domino endpoint Tier**.

5. Optionally, make this tier the default for all Domino endpoints.

<Frame>
  <img alt="Create Domino endpoint hardware tier" />
</Frame>

## Scale Domino endpoints

Scaling the hardware attached to a [Domino endpoint](/cloud/platform-capabilities/features/model-deployment) is one lever. The other is the degree of parallelism, which Domino sets deployment-wide for all Python Domino endpoints.

<Note>
  Only synchronous models support the degree of parallelism setting.
</Note>

1. Go to **Admin** > **Platform settings** > **Configuration records**.

2. Set `com.cerebro.domino.modelmanager.uWsgi.workerCount` to a value greater than its default of `1` to increase the uWSGI worker count. See the [uWSGI documentation](https://uwsgi-docs.readthedocs.io/en/latest/) for what the worker count changes.

3. Domino shows the message `Changes here do not take effect until services are restarted. Click here to restart services.` Click **here** to restart the services.

## External deployment targets

An external deployment target is a connection to an external account that models can be deployed to from Domino. Typically, a target is one account and region.

<Note>
  Amazon SageMaker is the only type of external deployment target currently supported. Deploying models to SageMaker is only supported when the Domino control plane runs on an [AWS EKS cluster](https://aws.amazon.com/eks).
</Note>

To create a target, go to **External deployment targets** under **Resources** in the admin panel, click **Create deployment target**, and complete these fields:

<Frame>
  <img alt="The External deployment targets page in the admin panel" />
</Frame>

| Field | What it sets |
| :- | :- |
| **Name** | The name of the target. Use something a data scientist can choose confidently at deployment time, such as `Production West`. |
| **Account ID** | The AWS account ID to use with this target. |
| **Use IRSA role credentials** | Whether to assume the execution role in the target AWS account using the Domino external deployments IAM role. |
| **Credentials file contents** | The contents of an AWS credentials file holding credentials for a user that can assume the execution role in the target account. See the [AWS credentials file format](https://docs.aws.amazon.com/sdkref/latest/guide/file-format.html). Ignored when IRSA role credentials are enabled. |
| **Execution role name** | The name of the execution role to assume when accessing AWS. This role must already exist in the target account. |
| **ECR repository name** | The ECR repository to store deployment images in. It is created if it does not exist in the target account. |
| **Artifacts S3 bucket name** | The S3 bucket to store deployment artifacts in. It is created if it does not exist in the target account. |
| **Region** | The AWS region code that deployments using this target create resources in, for example `us-east-1` or `eu-west-2`. |
| **Resource configurations** | The compute resources available in this target, which practitioners choose from when they deploy. Confirm the selected instance types are available in the selected region and that your AWS account has quota for them. |
| **Permissions** | Whether the target is available to everyone or only to specific users and organizations. |

When Domino runs on an AWS EKS cluster, a default deployment target for the Domino AWS account is created automatically, so users can deploy external endpoints inside the AWS account that runs the Domino platform. This can be disabled when Domino is deployed, and admins can change or remove the default target afterwards.

<Frame>
  <img alt="Default deployment target" />
</Frame>

To configure an AWS account for use as a Domino deployment target, use the Terraform module in the [dominodatalab/terraform-external-sagemaker](https://github.com/dominodatalab/terraform-external-sagemaker) repository.

## Related

* [Deploy models to SageMaker](/cloud/platform-capabilities/features/models/deploy-sagemaker)
* [Manage a hardware tier](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers/manage-hardware-tier)
* [Model Monitoring](/cloud/platform-capabilities/features/monitoring)


# Notifications
Source: https://docs.domino.ai/cloud/admin/platform-configuration/notifications

Create, target, schedule, expire, and delete Domino notifications, and control their priority, recipients, timezone, and duration.

Use notifications to send in-app messages to users about Domino. For example, you might want to inform users about planned maintenance, remind them to save their work, or send onboarding messages to a small group of new hires.

The navigation bar alerts users if they have notifications. See [Event notifications](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#view-event-notifications) for information about how your users experience these notifications.

Domino polls for notifications every 30 seconds.

## Send custom notifications

Administrators can create up to 100 notifications.

1. From the Admin application, click **Manage resources > Notifications**.

2. Click **Create Notification**.

3. In the new notification window:

   1. Set the **Priority** to indicate the criticality of the message. Messages set to Critical are prioritized to the top of the user’s list.

   2. Select one or more recipients. The default is to send to all users and organizations. Remove this option to limit the recipients to those that you select.

   3. Type the title and text for the notification.

   4. If necessary, change the **Timezone** to set the start and (optional) end time for the notification. The timezone defaults to your system timezone which is used to timestamp when you created the notification.

   5. Specify when you want the notification to be sent. If you set an end time, the message expires at this time. When the message expires, it is moved to the bottom of the user’s list. If you do not set an end time, the notification expires after the default expiration period that you configure.

   6. Click **Create notification**.The new notification is listed in the Notifications table.

## View notifications

View the Domino home page to see a list of recent notifications. Click **View all** to see all notifications.

* Use the **Type** list to sort by Active or Expired notifications.

* Use the **Priority** list to see **All notifications** or limit the table to **Critical** or **Default** notifications only.

## Expire notifications

If an issue is resolved, and you want to terminate a notification before its expiration date:

1. From the Admin application, click **Manage resources > Notifications**.

2. At the end of the row for the notification to terminate, click the three vertical dots.

3. Click **Expire**.

## Delete notifications

You can manage your list of notifications, delete outdated or incorrect notifications. When you delete a notification, no one can access it.

1. From the Admin application, click **Manage resources > Notifications**.

2. At the end of the row for the notification to delete, click the three vertical dots.

3. Click **Delete**.


# AutoML Studio
Source: https://docs.domino.ai/cloud/platform-capabilities/features/extensions/auto-ml

Train tabular, time series, or multimodal models with AutoGluon, review diagnostics, export notebooks, and register or deploy the selected model.

Train, evaluate, and deploy machine learning models through a unified no-code interface.

## Overview

AutoML is a Domino extension that enables data scientists and domain experts to build, evaluate, and deploy machine learning models through a streamlined, no-code interface.

Powered by AutoGluon, AutoML automates the end-to-end model training pipeline - from data profiling and feature engineering through model selection, hyperparameter tuning, and ensembling - so you can go from raw data to a production-ready model in minutes.

AutoML primarily provides a Model Training workflow. This allows you to configure and run an AutoGluon training job that automatically trains multiple model types, ranks them on a leaderboard, and produces deployment-ready artifacts.

<Warning>
  Exported models and notebooks should only be shared with authorized users.
</Warning>

You can access AutoML from the left navigation sidebar of any Domino project under the **Extensions** section.

## Get started

### Prerequisites

* A Domino project with the AutoML extension enabled.

* A dataset in CSV or Parquet format stored in a Domino Dataset.

* Permission to launch Domino Jobs if you plan to train models or run Data Exploration as a Domino Job.

### Installation

An administrator installs AutoML Studio from the Admin panel. See [Install Domino Official Extensions](/cloud/platform-capabilities/features/extensions/install-domino-official-extensions).

The [AutoML\_Extension repository](https://github.com/dominodatalab/AutoML_Extension) holds the source and [manual install instructions](https://github.com/dominodatalab/AutoML_Extension/blob/main/INSTALL.md) for reference.

### Access AutoML

1. Navigate to your Domino project.

2. In the left sidebar, scroll down to the **Extensions** section.

3. Click **AutoML**.

The AutoML landing page displays all existing training jobs. From here you can start a new training job.

<img alt="AutoML landing page" />

## AutoML model training

AutoML model training uses AutoGluon to automatically train, tune, and rank multiple machine learning models. The training job wizard guides you through four steps: selecting your data source, choosing a model type, configuring training parameters, and reviewing your settings before launch.

To begin, click **New training job** from the AutoML landing page.

### Step 1: Select a data source

Select a file in the dataset you want to use for training. Once your file is loaded, click **Continue** to proceed.

You can also click the shortcut in the dataset picker to open the project’s Domino Dataset area in a new browser tab so you can upload files before starting the run.

<img alt="Select a data source for a new training job" />

### Step 2: Select a model type

Choose the AutoGluon predictor that best fits your use case. Two model types are currently available:

| Model type | Description | Example use cases |
| - | - | - |
| **Tabular** | For structured data with rows and columns. Supports classification and regression tasks. | Classification and regression, equipment failure prediction, well log analysis. |
| **Time Series** | For forecasting sequential data where observations are ordered over time. | Production forecasting, demand prediction, anomaly detection. |

**Problem Type (Optional)**

For tabular jobs, AutoGluon can auto-detect the problem type from your target column, but you can also specify it explicitly:

| Problem type | Description |
| - | - |
| **Binary Classification** | Predict one of two classes (for example, `Failed: 0 or 1`). |
| **Multiclass Classification** | Predict one of multiple classes (for example, `CriticalityLevel: 1, 2, or 3`). |
| **Regression** | Predict a continuous value (for example, `FlowRatePSI`). |

Click **Continue** to proceed to configuration.

<img alt="Select a model type for a new training job" />

### Step 3: Configure training

The configuration step lets you define your training job’s name, target column, and AutoGluon-specific settings.

#### Basic configuration

| Setting | Description |
| - | - |
| **Job Name** | A descriptive name for this training run (for example, `My training job`). |
| **Description (optional)** | A free-text description to help you identify this job later. |
| **Target Column** | The column in your dataset that the model should predict. Select it from the dropdown list of available columns. |

For time series jobs, this step also includes:

* **Time Column:** The timestamp field used to order observations.

* **ID Column (optional):** A grouping key for multi-series forecasting.

* **Prediction Length:** The forecast horizon.

#### AutoGluon settings

| Setting | Description |
| - | - |
| **Preset** | Controls the trade-off between model quality and training speed. |
| **Time Limit (seconds)** | Maximum wall-clock time for the entire training run. AutoGluon trains as many models as possible within this limit. |
| **Evaluation Metric (optional)** | The metric used to rank models on the leaderboard, for example, `accuracy` for classification and `RMSE` for regression. |
| **Experiment Name (optional)** | An optional Domino Experiment name for tracking this run. If left blank, a name is auto-generated. |

<img alt="Configure the training of a new training job" />

#### Advanced configuration

Click **Advanced Configuration** to access fine-grained controls for AutoGluon training.

<img alt="Advanced configuration of the model training" />

### Step 4: Review and launch

Review all your selected settings on the summary screen. If everything looks correct, click **Start Training** to launch the job.

You are taken to the training run’s overview page where you can monitor progress in real time.

<img alt="Review and launch the training job" />

## Training results

Once a training job is launched, its results page provides comprehensive information about the run’s progress and outputs.

<Tabs>
  <Tab title="Overview">
    The **Overview** tab displays the training job’s metadata and real-time progress. You can access the Data Exploration view, the Domino Job details page, and Domino Experiment used to track generated models.

    <img alt="Overview tab of a completed training job" />
  </Tab>

  <Tab title="Model Leaderboard">
    The **Leaderboard** tab ranks all trained models by their validation score. The top-ranked model is the one AutoGluon recommends for deployment.

    Models are displayed in a table sorted by rank, with metadata related to AutoGluon scoring metrics.

    Use the **All Models** dropdown to filter the leaderboard by specific model families.

    <Tip>
      The `WeightedEnsemble` models combine predictions from multiple base models and typically achieve the highest scores. They may have longer prediction times, so consider this trade-off for latency-sensitive applications.
    </Tip>

    <img alt="Results of the trained job" />
  </Tab>

  <Tab title="Diagnostics">
    The **Diagnostics** tab provides visual insights into model behavior. The available sub-tabs depend on the problem type.

    **Feature Importance:** A horizontal bar chart ranking features by their contribution to model predictions. Features with the highest importance have the greatest influence on predictions. Use this chart to understand which variables drive your model’s decisions and to identify potential features for engineering or removal.

    **Model Comparison:** Side-by-side comparison of all trained models showing model names with color-coded performance bars, validation scores for each model, and a visual comparison helping you understand how different algorithms performed on your data. The best-performing model is highlighted at the top of the comparison.

    **Classification Metrics** (binary and multiclass classification only):

    This view provides comprehensive classification-specific metrics.

    | Component | Description |
    | - | - |
    | **Confusion Matrix** | A grid showing predicted vs. actual class labels. Cells are color-coded by frequency, helping identify where the model makes correct predictions and where it confuses classes. |
    | **Accuracy** | Overall proportion of correct predictions. |
    | **Precision** | Weighted average precision across all classes (proportion of positive predictions that were correct). |
    | **Recall** | Weighted average recall across all classes (proportion of actual positives that were correctly identified). |
    | **F1-Score** | Weighted average F1-score (harmonic mean of precision and recall). |
    | **ROC Curve** | For binary classification only. Plots the true positive rate against the false positive rate at various threshold settings. Includes the AUC (Area Under Curve) score. |

    **Regression Diagnostics** (regression only):

    This view provides regression-specific metrics and visualizations.

    | Component | Description |
    | - | - |
    | **R² (R-Squared)** | Coefficient of determination indicating the proportion of variance explained by the model. Values closer to 1.0 indicate better fit. |
    | **RMSE** | Root Mean Square Error: average magnitude of prediction errors, in the same units as the target variable. |
    | **MAE** | Mean Absolute Error: average absolute difference between predicted and actual values. |
    | **MSE** | Mean Square Error: average squared difference between predicted and actual values. |
    | **Predicted vs Actual Plot** | Scatter plot comparing predicted values (Y-axis) against actual values (X-axis). Points close to the diagonal line indicate accurate predictions. |
    | **Residuals Plot** | Scatter plot showing residuals (prediction errors) against predicted values. Ideally, residuals should be randomly distributed around zero with no visible patterns. |

    <img alt="Diagnostics of the trained job" />
  </Tab>

  <Tab title="Metrics">
    The **Metrics** tab provides two comparison charts and a summary statistics panel:

    **Validation Scores:** A horizontal bar chart comparing every model’s validation score. The best-performing model appears at the top.

    **Training Times:** A horizontal bar chart showing how long each model took to train. This helps you evaluate the cost-performance trade-off for each model.

    **Training Statistics:** A summary panel displaying key metrics for the overall run.

    **Fit Summary:** When available, a raw training summary is shown below the charts for deeper inspection.

    | Metric | Description |
    | - | - |
    | **Models Trained** | Total number of models trained during the run (for example, `12`). |
    | **Best Score** | The highest validation score achieved across all models (for example, `0.860`). |
    | **Total Train Time** | Total wall-clock time for all model training (for example, `25s`). |
    | **Avg Train Time** | Average training time per model (for example, `2.1s`). |

    <img alt="Metrics of the trained job" />
  </Tab>

  <Tab title="Outputs">
    The **Outputs** tab provides a **Download Notebook** to get a Jupyter notebook with the full training configuration, evaluation code, and deployment instructions. This notebook serves as a reproducible record of the experiment and a starting point for further customization.

    <img alt="Outputs of the trained job" />
  </Tab>

  <Tab title="Deployments">
    The **Deployments** tab groups operational follow-up tasks into three sections:

    * **Model Registry:** Shows whether the model has been registered in Domino and lets you register a model.

    * **Model API:** Shows Model API deployment status and lets you deploy a registered model as a Domino Model API.

    * **Experiment Tracking:** Surfaces Domino Experiment and MLflow Run links for the job.

      <img alt="Deployments tab of a completed training job" />
  </Tab>
</Tabs>

## Data exploration

Accessible via the dataset link in the Training Results view, the Data Exploration tool lets you analyze data quality, distributions, and correlations, and prepare transformations for subsequent model training.

<img alt="Data Exploration overview with data preview" />

Data Preview

The **Data Preview** tab shows a sample of your raw data in table format. You can browse columns and rows to get a quick sense of the dataset’s structure and values.

If your dataset is large or takes a long time to profile, you can re-profile the data with a Domino Job using the Execution dropdown.

Column Analysis

The **Column Analysis** tab provides detailed profiling information for each column in your dataset.

<img alt="Data Exploration overview with column analysis" />

Correlations

The **Correlations** tab displays a correlation matrix heatmap for all numeric columns in the dataset. Correlation values range from −1 (strong negative correlation) to +1 (strong positive correlation), with color coding to highlight the strength and direction of each relationship.

<img alt="Data Exploration overview with correlations" />

Use the **Min correlation** slider to filter out weak correlations and focus on the strongest relationships. This view is helpful for identifying multicollinearity between features and understanding which variables are most associated with your target column.

The UI also lists highly correlated column pairs below the matrix to help you quickly identify candidates for removal or further review.

Data Quality

The **Data Quality** tab gives you an at-a-glance assessment of your dataset’s readiness for model training. It includes the following sections:

* **Missing Values by Column:** A horizontal bar chart showing the number and percentage of missing values for each column that has any. Columns are color-coded by severity: green for less than 5% missing, orange for 5–20%, yellow for 20–50%, and red for more than 50%.

* **Missing Value Pattern:** A compact visual representation of where missing values occur across columns, helping you identify whether missingness is concentrated or scattered.

* **Warnings:** Issues that could affect training quality. For example, `Small dataset – consider using best_quality preset for better results`.

* **Recommendations:** Actionable suggestions to improve model performance. These are tagged by priority (high or medium) and category (Target, Preprocessing). Examples include:

Transformations

The **Transformations** tab lets you define preprocessing steps that are applied to the dataset before model training. AutoML analyzes your data and suggests recommended transformations based on detected issues.

**Recommended transformations**

AutoML may recommend transformations for columns with the following issues:

* **Identifier columns:** Columns where all values are unique and are likely row identifiers (for example, `PumpId`). *Recommended action:* drop the column.

* **High cardinality:** Columns that may be identifiers due to a large number of unique values (for example, `ManufacturerModel`). *Recommended action:* review and potentially drop.

* **Missing values:** Columns with a significant percentage of null values (for example, `OperatingYears at 19.9% missing`). *Recommended action:* fill missing values.

* **Extreme outliers:** Numeric columns with extreme values detected (for example, `FlowRatePSI with 6.9% outliers`). *Recommended action:* clip or remove outliers.

* **High missing rate:** Columns with a high percentage of missing data (for example, `WellSector at 77% missing`). *Recommended action:* drop the column.

Transformations are included in the exported notebook and can be reviewed and modified in code.

<img alt="Data Exploration page" />

## Export a notebook

At any point during data exploration, you can click the **Export Notebook** button in the top-right corner to download a Jupyter notebook that contains all of your data profiling results and any transformations you have selected.

This notebook can be opened in a Domino Workspace for further analysis or custom preprocessing.

## Deploy a model

After training is complete, you can operationalize the best model from the **Deployments** tab in the Training Results view.

<img alt="Register model dialog from the Deployments tab" />

### Register a model

To register your model in Domino’s Model Registry for versioning, governance, and deployment:

1. Open the completed training run.

2. Navigate to the **Deployments** tab.

3. In the **Model Registry** section, click **Register Model** and fill out the details.

4. Click **Register** to save the model to the registry.

Once registered, the model appears in Domino’s Model Registry and can be promoted into downstream deployment workflows.

### Deploy a model API

After a model is registered:

1. In the **Model API** section of the **Deployments** tab, click the deploy link.

2. Complete the deployment dialog.

Once deployment finishes, the section shows the Model API status and, when available, a direct link back to Domino.

## Best practices

* **Review data quality for subsequent training.** Use the Data Exploration tool to inspect missing values, outliers, and column types. Address high-priority recommendations (especially dropping or imputing columns with more than 30% missing data) before starting a training job.

* **Use the dashboard to manage iteration cycles.** Search, filter, and switch views on the AutoML landing page as your project accumulates runs. Bulk deletion is useful for clearing experimental jobs that are no longer relevant.

* **Choose the right preset.** The Medium Quality (Faster) preset is a good starting point for rapid iteration. Once you have identified a promising dataset and target, switch to Best Quality for production models, especially on smaller datasets where the additional training time yields meaningful improvements.

* **Set an appropriate time limit.** A longer time limit allows AutoGluon to train more models and explore more hyperparameter configurations. For initial exploration, 600–1800 seconds is often sufficient. For production runs, consider 3600 seconds or more.

* **Exclude identifier columns.** Columns with all unique values (flagged as identifiers) do not contribute meaningful signal and should be dropped before training. AutoML flags these in both Column Analysis and Recommended Transformations.

* **Examine feature importance.** After training, review the Feature Importance chart on the Diagnostics tab. If unimportant features dominate, consider removing them and retraining to reduce noise and improve generalization.

* **Consider the prediction time trade-off.** Ensemble models (for example, WeightedEnsemble\_L2) typically achieve the best validation scores but may have higher inference latency. For real-time applications, compare leaderboard scores with prediction times and consider selecting a simpler model that meets your latency requirements.

* **Use the exported notebook for reproducibility.** Download the Training Notebook from the Outputs tab to preserve a complete record of your training configuration. This notebook can be re-run in a Domino Workspace and serves as the starting point for any custom modifications.

* **Use the Deployments tab for operational follow-through.** Register the model after a successful run, then create a Domino Model API from the same page when you are ready to serve predictions.


# Auto Model Documentation
Source: https://docs.domino.ai/cloud/platform-capabilities/features/extensions/auto-model-documentation

Draft structured documentation for machine learning models with citations back to project code, model artifacts, and governance evidence.

Auto Model Documentation is a Domino extension that drafts structured documentation for your machine learning models. It reads your project code, model metrics and model info, registered models, and optional Domino Governance bundles, then produces a Word document and an editable Jupyter notebook with citations back to the source material.

Use it for model reviews, handoffs, and compliance workflows when you want a structured first draft to review rather than writing every section from scratch.

## Contents

* [What it does](#what-it-does)
* [Requirements](#requirements)
* [Data handling and privacy](#data-handling-and-privacy)
* [Getting started](#getting-started)
* [Generate a document](#generate-a-document)
* [Templates and specs](#templates-and-specs)
* [Citations](#citations)
* [Governance bundles](#governance-bundles)
* [Outputs and history](#outputs-and-history)
* [What to expect](#what-to-expect)
* [For administrators](#for-administrators)
* [Troubleshooting](#troubleshooting)

## What it does

* Documentation templates you can preview, edit, upload, or browse from project datasets.
* Automatic analysis of model code, training artifacts, and registry metadata.
* Optional governance grounding from Domino Governance bundles (policy, evidence, findings).
* Citations to source code, model runs, artifacts, and governance records.
* A Word document and an editable Jupyter notebook stored in your project's autodoc dataset.
* Generation runs as a Domino job in the target project, so heavy work does not run in the Studio app.

## Requirements

* Domino 6.3.0 or later.
* Extension installed via **Admin > Extensions > Manage Domino-official Extensions**.
* An LLM provider API key (`ANTHROPIC_API_KEY` or `OPENAI_API_KEY`) available to documentation jobs in the target project. You can set it on the compute environment, the project, or your user account. User-level values take precedence over project and compute environment values.
* Read access to the target project's code, datasets, and (if used) governance bundles.

## Data handling and privacy

Documentation jobs run inside your Domino project and call the LLM provider you select in Advanced options. Before any code is sent to the provider, the scanner redacts likely secrets and credentials.

**What gets redacted before LLM calls:**

* API keys, secret tokens, passwords, and bearer tokens matched by pattern.
* AWS access key IDs and secret access keys.
* GitHub, Slack, and Anthropic API tokens.
* Entire contents of files named `.env`, `.pem`, `.key`, `credentials`, or `secrets`.

Each redacted value is replaced with a `[REDACTED]` placeholder. Files larger than the configured size limit are truncated before being sent.

**What is sent to the provider:**

* The code, model metrics and parameters, registered model metadata, and governance evidence needed to write the document. The LLM provider receives this content to generate prose.

**You control where the call goes.** In Advanced options you choose the provider (Anthropic or OpenAI and OpenAI-compatible endpoints) and can override the API base URL to route through a proxy or gateway your organization operates. A call is only made when you click **Generate Documentation**.

<Note>
  If your organization requires a specific provider endpoint or proxy, set `ANTHROPIC_BASE_URL` or `OPENAI_BASE_URL` on the job compute environment and leave the in-app base URL at its default.
</Note>

## Getting started

### Where to open it

After your administrator installs the extension, open **Auto Model Documentation** from:

* The **project sidebar** (Extensions section) while viewing any project.
* The **model** page when viewing a registered model. This pre-populates model filters and can auto-select a governance bundle.

The Studio always runs in the context of a Domino project. It receives a `projectId` from Domino when launched from a mount point.

On first open, Domino may ask you to authorize the extension app to act on your behalf. Click **Authorize** to continue.

<Frame>
  <img alt="Permissions consent dialog" />
</Frame>

### Prerequisites

1. **Extension installed.** Your Domino administrator has installed Auto Model Documentation from the official extensions installer.
2. **Project access.** You can open the target project and run jobs in it.
3. **LLM credentials.** The project (or its default compute environment) has `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` set. Jobs inherit these variables when they run. You can also set the key on your user account (**Account Settings > User environment variables**). User-level values take precedence over project and compute environment values.
4. **Model artifacts (recommended).** Your project has model metrics, model info, or registered models you want documented. The tool can run on code alone, but results are richer when training artifacts exist.

<Tip>
  Ask your administrator to set LLM API keys on the extension project or on compute environments used for documentation jobs if generation fails with authentication errors.
</Tip>

### Confirm the Studio loaded correctly

You should see:

* **Choose a template** on the left with templates in the gallery.
* **Document outline** preview on the right (after you select a template).
* A **Generate Documentation** button at the bottom (enabled after you select a template).

If templates stay on the "Loading templates" state, see [Troubleshooting](#troubleshooting).

<Frame>
  <img alt="Studio main view with template gallery" />
</Frame>

### First-run checklist

1. Select a template
2. Review the document outline preview
3. Expand **Filters** if you want to limit models or experiments
4. Select a **Governance bundle** if your project has governance enabled
5. Open **Advanced options** and confirm provider, model, hardware tier, and environment
6. Click **Generate Documentation**

## Generate a document

The Studio is a two-step wizard:

1. **Choose a template** and configure options.
2. **Results** with live job status, preview, and download links.

### Step 1: Choose a template

Click a template card in the gallery to select it. The **Document outline** panel shows every section the template generates.

<Frame>
  <img alt="Standard ML template selected with document outline" />
</Frame>

Sections marked **once per model** expand into one subsection per registered model that matches your filters.

You can also **Browse** spec files in project datasets, **Upload** a `.yaml` or `.yml` file, or **Edit** the selected template's YAML inline and **Save** or **Revert**.

<Frame>
  <img alt="Browse spec files modal" />
</Frame>

<Frame>
  <img alt="Inline template YAML editor" />
</Frame>

#### Filters

Expand **Filters** to narrow which models and experiments are included:

| Field | Description |
| :- | :- |
| **Model names** | Comma-separated names. Use `*` wildcards (for example, `churn-*`). |
| **Experiment names** | Comma-separated experiment names. Wildcards supported. |
| **Latest version only** | When checked, include only the latest version of each registered model. |

Leave filters empty to include all models and experiments the scanner can access.

<Frame>
  <img alt="Filters expanded" />
</Frame>

#### Advanced options

Click **Advanced options** to configure the Domino job and LLM settings:

| Option | Purpose |
| :- | :- |
| **Branch** | Git branch for the job (git-backed projects only). Leave blank for the project default. |
| **Code path** | Path to source code to analyze. Defaults to the project code mount. |
| **Hardware tier** | Compute tier for the documentation job. |
| **Environment** / **Revision** | Compute environment the job runs in. |
| **Provider** | `anthropic` or `openai` (and OpenAI-compatible endpoints). |
| **Provider API base URL** | Override the API endpoint (for proxies or compatible gateways). |
| **Model** | Model name passed to the provider. |

Default models:

| Provider | Default model |
| :- | :- |
| Anthropic | `claude-haiku-4-5` |
| OpenAI | `gpt-5.4-mini` |

To use a different model, enter its name in the **Model** field. See the [Anthropic Models overview](https://docs.anthropic.com/en/docs/about-claude/models) and the [OpenAI Models reference](https://platform.openai.com/docs/models) for current names and availability. For OpenAI-compatible gateways, use the model names that gateway exposes.

The footer notes that `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` must be available to the job environment.

<Frame>
  <img alt="Advanced options with Anthropic provider selected" />
</Frame>

Click **Generate Documentation**. The wizard moves to the results view and shows a **Submitting job** message while the run is queued.

### Step 2: Results

While the job runs, status updates appear in the results panel. Open **History** (top right of the outline panel) to see past runs.

<Frame>
  <img alt="Results view while documentation is generating" />
</Frame>

When the job succeeds:

* A success banner appears.
* **Document Preview** loads an HTML preview of the generated Word document.
* Download links open the `.docx` file and job artifacts in Domino.

<Frame>
  <img alt="Document preview of generated output" />
</Frame>

If the job fails, check the Domino job logs from the **View job** link in History.

### Tips for better output

<Tip>
  * Scope models with filters when a project has many registered models. Generation time and output length scale with the number of files and models scanned.
  * Keep code and model artifacts in sync. Missing artifacts produce shorter sections that explicitly state what could not be evidenced, by design.
</Tip>

## Templates and specs

Documentation structure is defined by **spec files**: YAML documents that list sections, titles, and per-section hints for the LLM.

Templates in the gallery are synced to your project's autodoc dataset on first load. Edits in the Studio save to the dataset copy.

### Spec file structure

```yaml theme={null}
slug: my_template
card_title: "My Template"
card_description: "Short description shown on the gallery card."

title: "Model Documentation Title"
authors: "Your Team"

sections:
  - Executive Summary
  - Data Overview
  - "Model Performance: per_model"
  - Appendix

hints:
  "Executive Summary": >
    What to emphasize in this section. Only document evidenced facts.
```

Append `: per_model` to a section name to generate one subsection per matching registered model:

```yaml theme={null}
sections:
  - "Development History: per_model"
  - "Model Performance: per_model"
```

The `hints` map keys must match section names (without the `: per_model` suffix). Hints can include evidence gates so the generator only states metrics, compliance claims, or governance status that are supported by source material.

Specs live in the project's **autodoc** dataset. Use **Browse** to pick specs from other datasets.

### Supported code languages

The scanner reads source files for the following languages:

| Language | Extensions |
| :- | :- |
| Python | `.py` |
| R | `.R`, `.r`, `.Rmd`, `.rmd` |
| SAS | `.sas`, `.SAS` |
| MATLAB | `.m` |

Files matching test directories, virtual environments, and other exclude patterns are skipped. Binary files are skipped. README content (truncated) is read when present.

## Citations

Generated prose cites its sources inline so a reviewer can trace any claim back to the artifact it came from. Citations appear as bracketed labels in the document body and resolve to entries in a References section.

**Citation types:**

| Source | Format |
| :- | :- |
| Code file | `Code:path/to/file.py#symbol:L42-L58` |
| Model run | `ExperimentName-RunName` |
| Model artifact | `ExperimentName-RunName:artifact.ext` |
| Governance field | `governance.<field>` |
| Governance evidence | `evidence.<slug>` |
| Governance finding | `finding.<id>` |

In the rendered Word document, an inline citation shows as a short label (for example, `[train_churn.py · fit model]` or `[Risk tier: High]`) and links to a full entry in the References section. Model run citations include the experiment and run name. Artifact citations include the artifact filename. Governance citations include the bundle name and short id.

The Governance & Risk section quotes risk tier, validation status, approval state, and open findings verbatim from the selected bundle's evidence, each with its citation. Other sections leave governance status to the **Governance & Risk** section rather than restating it.

## Governance bundles

When your Domino project uses **Governance**, the Studio can load bundle evidence and findings so governance sections reflect your governed model of record.

### Selecting a bundle

The **Governance bundle** field appears when the project has at least one bundle you can access. Review the pre-selected bundle, or choose another.

<Frame>
  <img alt="Governance bundle selector" />
</Frame>

When you open the extension from a **model** mount point, the Studio may pre-populate **Model names** and auto-select a bundle linked to that model.

### What governance data is used

For the selected bundle, the pipeline reads policy and evidence attachments and open findings. Governance-oriented sections (for example, **Governance & Risk**) use this material. Other sections still draw primarily from code and model metrics.

### Evidence boundaries

Templates guide the LLM to state risk tier, approval state, compliance certifications, and fairness metrics only when evidence supports them. When evidence is thin, expect shorter sections that state what could not be assessed.

**Development History** describes model candidates explored during development. **Governance & Risk** describes the bundle-linked governed model. The templates keep that distinction explicit so a development candidate is never read as a governed model of record.

## Outputs and history

### Output files

Primary output: `model_docs.docx` at:

```
autodoc dataset / docs / <run-id> / model_docs.docx
```

`<run-id>` is the first eight characters of the Domino job run ID. Jobs also generate a Jupyter notebook in the same folder, named `model_docs.ipynb`.

### The notebook output

The `.ipynb` file is the editable counterpart of the Word document. Each section is a cell, so you can revise wording, add commentary, or drop in extra analysis before re-exporting. Run the export cell the to rebuild the `.docx` file, so the Word file stays in sync with your edits instead of being regenerated from scratch.

Use the notebook when you want to hand-edit content while keeping the citation structure and References section intact. Use the `.docx` directly when the generated draft is good enough.

### Access outputs

From the Studio after a successful run:

1. **Document Preview** in the results view.
2. Download or **Open** links for the `.docx`.
3. **History** for past runs (**View** logs, **Open** in dataset, **Preview** completed documents).

<Frame>
  <img alt="History drawer with a completed run" />
</Frame>

You can also browse the **autodoc** dataset in the project **Data** tab under `docs/`.

### Job history scope

History lists documentation jobs for the **current project** only. Other users with project access see the same list.

<Frame>
  <img alt="History drawer before any runs" />
</Frame>

The extension does not automatically delete old runs. Plan dataset retention with your team if autodoc grows large.

## What to expect

Auto Model Documentation produces a structured first draft, not a finished compliance artifact. Knowing the shape of the output up front saves review cycles.

**Duration.** Generation runs as a Domino job. Time scales with the number of code files scanned and the number of registered models that match your filters. A small project with a handful of models typically finishes in a few minutes. A large project with many files and per-model sections takes longer. Use filters to narrow scope when you only need a subset documented.

**Output length.** Sections are only as long as the evidence supports. When code, model metrics, or governance data is missing for a topic, the section is short and says so explicitly. This is by design.

**Human review.** A human reviewer should verify citations, add business context the code cannot convey, and confirm any compliance claims against the governing evidence. The tool assembles and cites; it does not sign off.

**Cost.** Generation calls the LLM provider you select. Usage scales with the amount of code and metadata sent. Route through your organization's proxy or gateway in Advanced options if you need billing or egress controls.

## For administrators

This section is for Domino administrators installing and configuring the extension. End users can skip this section.

### Install

1. Open **Admin > Extensions > Manage Domino-official Extensions**.
2. Select **Auto Model Documentation**.
3. Choose a version (for example 1.0.0).
4. Click **Install** and wait for the pipeline to finish.

The installer creates the extension project, compute environment, app (`modeldocs` vanity URL, entry point `app.sh`), and extension record with `projectSidebar` and `model` mount points.

For the full installer reference, including upgrades, retries, and uninstalls, see [Install Domino Official Extensions](/cloud/platform-capabilities/features/extensions/install-domino-official-extensions).

### Post-install configuration

No environment variables are required for a baseline 1.0.0 install.

<Note>
  **Recommended:** set `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` on the extension project or job compute environments. Optional: `ANTHROPIC_BASE_URL`, `OPENAI_BASE_URL`.
</Note>

Upgrades install from the same tab.

## Troubleshooting

### Studio does not load

| Message | Fix |
| :- | :- |
| **Project ID required** | Open from a Domino mount point (sidebar or model page), not a bare app URL. |
| **Access to this project's Datasets is required** | Request dataset access. The Studio needs access to datasets. |
| **Session or consent required** | Reload and complete Domino consent prompts. |

<Frame>
  <img alt="Session or consent required error" />
</Frame>

### Templates

The **Loading templates** state never finishes: check dataset access and that the extension app is **Running**.

Upload or save fails: confirm write access to the autodoc dataset.

### Other issues and actions

* Generate button disabled: No template selected, click a gallery card to enable it.
* Compute environment errors: contact your administrator.

### Job failures

Open **History > View** for logs.

| Symptom | Fix |
| :- | :- |
| LLM authentication errors | Set `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` on the job environment. |
| Code directory not found | Check **Code path** in Advanced options. |
| Hardware tier / environment rejected | Choose a hardware tier and environment you have access to in this project. |


# Create and manage Extensions
Source: https://docs.domino.ai/cloud/platform-capabilities/features/extensions/create-manage-extensions

Build Extensions from Apps, configure Project, Dataset, file, Model, or Admin mount points, handle context parameters, and manage App versions.

An Extension adds functionality from a Domino App into the Domino UI without changing the core Domino codebase. It shows up where your users already work: a link in the Project sidebar, an entry on a Dataset details page, an entry on a Model details page, an entry in the three-dot menu for files in a Dataset, or a new page in the Admin panel. When a user clicks it, Domino opens the underlying App and passes along the context of the page it came from.

An Extension is not an App, and an App is not an Extension. An Extension points to an App. You build the App once, then create one or more Extensions that surface it at specific places in the UI. The same App can power several Extensions, and you can change which App version an Extension points to without rebuilding anything.

This page covers building your own Extension. To install a pre-built Extension that Domino develops and maintains, such as [AutoML Studio](/cloud/platform-capabilities/features/extensions/auto-ml), [Clinical Data Explorer](/cloud/platform-capabilities/features/extensions/data-visualization), or [Auto Model Documentation](/cloud/platform-capabilities/features/extensions/auto-model-documentation), see [Install Domino Official Extensions](/cloud/platform-capabilities/features/extensions/install-domino-official-extensions). The installer creates the App and Extension for you.

The App behind an Extension is yours to build. Three requirements shape it. It must have extended identity propagation enabled, which is what lets it call Domino APIs as the person viewing it, and which needs a framework that can read proxied HTTP headers: [Flask and Dash support them by default, and Shiny needs Server Pro](/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security#extended-identity-propagation). A SysAdmin or CloudAdmin publishes it. Viewers consent before it can act as them.

<Note>
  Only SysAdmins and CloudAdmins can publish an App with extended identity propagation enabled, and only Admins can create an Extension from it. Both are release steps: what an Extension does comes from the App.
</Note>

## How it works in Domino

An Extension connects three things: an App that does the work, a mount point that decides where the Extension appears, and the context that the mount point passes to the App as query parameters. The workflow is the same regardless of where you mount it:

1. **Build an App** with extended identity propagation enabled. This is the only hard requirement on the App itself.

2. **Choose a mount point**. The mount point determines where the Extension link appears in the UI and which context it passes.

3. **Handle the context** in your App if you want it to respond to the page it was opened from. This step is optional.

4. **Create and configure the Extension**, selecting the App version and one or more mount points.

5. **Use the Extension** by navigating to a page where it is mounted and clicking the link.

6. **Manage the Extension** from the Admin panel: enable, disable, edit, or delete it.

The rest of this page walks through each step, then covers the mount point reference and the guidelines your App should follow.

## Step 1. Build an App

Create an App to use as the Extension’s backend, or reuse an existing one.

* Use a standard Compute Environment such as the DSE if it has what your App needs. If your App depends on packages or other dependencies that a standard environment doesn’t provide, create your own Compute Environment first and use it with the App.

* The App must be published with extended identity propagation enabled, which only a SysAdmin or CloudAdmin can do. In the App publish flow, this is the **Allow App to act for viewers in Domino** option on the **Access & sharing** step. With this turned on, the App acts using each viewer’s Domino identity, so it can read files and data and run Jobs with that viewer’s \* permissions. Domino enforces this requirement because the identity propagation mechanism is what supplies the per-request token your App uses to call Domino APIs. An App without it cannot back an Extension.

* Build the App in a framework that can read proxied HTTP headers, since that is how the per-request token reaches it. [Flask and Dash support them by default, and Shiny needs Server Pro](/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security#extended-identity-propagation). Viewers grant consent before the App can act as them, and that consent expires.

* Deep linking with query parameters is optional but recommended to allow context-aware behavior in the Extension’s App. This allows the Extension’s App to, for example, assume the Project context it was entered from, or know what file to operate on, etc.

* Users must have permission to view the underlying App to use the Extension.

## Step 2. Choose a mount point

A mount point is the location in the Domino UI where the Extension link appears. Five mount point types are available:

| Mount point | Where it appears |
| - | - |
| Projects | The navigation sidebar for every Project |
| Files | The three-dot menu for files in Datasets |
| Datasets | The details page for every Dataset |
| Models | The details page for Models |
| Admin | A new page in the Admin panel |

A single Extension can use more than one mount point. Pick the places where the Extension’s function is relevant to what the user is looking at. Typically, one App corresponds to one Extension, but if your Extension’s App behaves differently for different mount points or, depending on the context passed in, you want to expose different versions of an App, you can create multiple Extensions from the same underlying App.

## Step 3. Handle the context

When a user opens an Extension, the mount point passes the context of the originating page to your App as query parameters on the root route (`/`). Handling them is optional, but it lets the App respond to the specific Project, Dataset, file, or Model the user came from.

Each mount point passes a fixed set of parameters:

| Mount point | Query parameters passed |
| - | - |
| Projects | `projectId` |
| Files | `datasetId`, `datasetSnapshotId`, `datasetFilePath` |
| Datasets | `datasetId` |
| Models | `modelId`, `modelVersionId` |
| Admin | none |

* To use the context, read these parameters on your App’s root route and act on them.

* Validate that required parameters are present and return a clear error when they are missing, since a user can reach the App’s URL without them.

* Authenticate to Domino with the per-request bearer token from the `Authorization` header rather than an API key.

The example below is a Flask App mounted on Datasets. It reads `datasetId`, then calls the Domino Datasets API with the forwarded token to fetch that Dataset’s details. The same pattern applies to any mount point: read the parameters from the table above and pass the token through.

```python theme={null}
import flask
import requests

@app.route("/")
def index():
    dataset_id = flask.request.args.get("datasetId")
    if not dataset_id:
        return flask.Response("Missing required query parameter: datasetId", status=400)

    # Forward the per-request token that identity propagation supplies.
    auth_header = flask.request.headers.get("Authorization", "")
    url = f"{DOMINO_HOST}/api/datasetrw/v1/datasets/{dataset_id}"

    try:
        response = requests.get(url, headers={"Authorization": auth_header})
        response.raise_for_status()
    except requests.HTTPError as e:
        return flask.Response(f"Failed to fetch dataset: {e}", status=e.response.status_code)

    dataset = response.json().get("dataset", {})
    # Render dataset details in your App.
```

A Project’s mount point reads `projectId`, a Model’s mount point reads `modelId` and `modelVersionId`, and a File’s mount point reads `datasetId`, `datasetSnapshotId`, and `datasetFilePath`. An Admin mount point passes no context, so its root route needs no parameter handling.

## Step 4. Create and configure the extension

1. There are two ways to open the Create Extension modal:

   * From your App’s three-dot menu, select **Create Extension**.

   * In the **Extensions** tab of the Admin panel, select **Create Extension**.

2. In the modal, choose the App version to associate with the Extension, then add one or more mount points under **Add Extension to**, specifying any relevant scope (like file extension) if applicable.

3. Use **Add Extension point** to mount the same Extension in more than one place.

4. Select **Create Extension** to save.

The Extension is now available to all users who have permission to view the App. If a user does not have that permission, they still see the Extension listed but are prompted to request access to the App.

## Step 5. Use the extension

1. Navigate to a page that has a mount point you configured.

2. Click the Extension link that appears in the Project sidebar of the **Extensions** panel on a Dataset or Model details page, or in the file three-dot menu.

3. Domino opens a new tab with the App’s contents and passes the page’s context as query parameters.

## Step 6. Manage the extension

Manage existing Extensions from the **Extensions** tab in the Admin panel, reached through **Platform settings**.

* Each Extension has a toggle.

  * Switch it off to disable the Extension, which removes it from its mount points without deleting it.

  * Switch it on to restore it.

* Select **Manage Extension** for the rest of the actions.

  * **Edit** changes the mount point configuration and the App version the Extension points to.

  * **Delete** soft-deletes the Extension and removes it from everywhere in the UI.

<Note>
  Publishing a new version of the source App does not update the Extension automatically. An Extension keeps pointing to the App version it was created with until you change it. To move it to a newer version, select **Manage Extension**, then **Edit**, choose the new version under **Version**, and select **Save & close**.
</Note>

<Note>
  Extensions installed through the Domino Official Extensions installer appear in this tab like any other Extension, marked as Domino Official. Enable or disable them here, but perform upgrades and uninstalls from the **Manage Domino-official Extensions** tab, and avoid editing their underlying Project, Compute Environment, or App. See [Install Domino Official Extensions](/cloud/platform-capabilities/features/extensions/install-domino-official-extensions).
</Note>

## App guidelines and advanced use cases

These practices keep an Extension’s App secure and predictable. The list reflects current guidance and grows over time.

* Call Domino APIs only from your App’s backend code, never from frontend code such as React.

* Authenticate to Domino APIs with the JWT bearer token that is available on a per-request basis in the App backend. Do not use API keys or any other authentication method. The extended identity propagation mechanism places this token in each request, which is why Extensions are limited to Apps that have the feature enabled.

* An Extension inherits its access controls from the underlying App. There is no separate permission layer to configure.

* Write stateless Apps. If your App must keep state, give any stored data access controls equivalent to the source it came from. For instance, if the App pulls information from a Domino API and caches it outside request-scoped data structures, that cached copy needs controls functionally equivalent to the API it was read from.

* To start Domino Jobs that run your App’s code rather than the target Project’s code, include that code in your App’s Compute Environment. When you start the Job through an API call, set the run command to the path of your file inside the Compute Environment, and pass your Compute Environment’s ID and revision ID, both of which are available as environment variables in the App runtime.

* Where you can, declare all dependencies in the Dockerfile of your App’s Compute Environment and install nothing at runtime.

## Next steps

If you don't have an App yet, start there. [Publish Apps](/cloud/platform-capabilities/features/apps) covers building and publishing one, and [Common App frameworks](/cloud/platform-capabilities/features/apps/common-app-frameworks) has worked examples.

The Extensions Domino publishes are built on this framework, and the [Extensions overview](/cloud/platform-capabilities/features/extensions) links the public source for each one.

* [Clinical Data Explorer](/cloud/platform-capabilities/features/extensions/data-visualization): Explore, filter, and visualize data through an intuitive web interface.

* [AutoML](/cloud/platform-capabilities/features/extensions/auto-ml): Train, evaluate, and deploy machine learning models through a unified no-code interface.

* [Auto Model Documentation](/cloud/platform-capabilities/features/extensions/auto-model-documentation): Automatically generate professional ML model documentation from code and Experiment Manager artifacts.


# Clinical Data Explorer
Source: https://docs.domino.ai/cloud/platform-capabilities/features/extensions/data-visualization

Explore CSV, Parquet, SAS, and CDISC Dataset-JSON files, filter and visualize tabular data, share views, and ask questions through chat.

Clinical Data Explorer is a Domino App for exploring, filtering, and visualizing clinical and general datasets.

## Overview

Clinical Data Explorer is a dataset analysis application that runs as a Domino App. It lets users browse, filter, and visualize tabular data through an intuitive web interface, and ask questions about their data in plain English. It retrieves data via Domino APIs from datasets and Netapp Volumes accessible to the user, and supports CSV, Parquet, SAS (`.sas7bdat`), SAS Transport (`.xpt`), and CDISC Dataset-JSON formats.

## Who it's for

Clinical Data Explorer is aimed at clinical data scientists, statistical programmers, data managers, and reviewers who work with clinical trial data (with particular focus on CDISC dataset standards), as well as anyone who needs a quick, no-code way to explore tabular data on Domino. It is especially useful for life-sciences teams, but it works equally well for general datasets.

## Key features

* **Table View**: Browse and filter datasets in an interactive table, with built-in missing-values analysis, summary statistics, and distinct-value counts.
* **Explore**: Build visualizations, including histograms, X-Y plots, and category bar charts.
* **Chat**: Ask questions about your data in natural language (requires an LLM provider to be configured).
* **Friendly Names**: Toggle human-readable labels in place of raw column names (for example, for ADaM variables).
* **Shareable views**: Copy a link that preserves your current filters so colleagues see the exact same view.

## How to use it

1. **Load a dataset.** Choose a file from the dataset picker and click **Load**.
2. **Filter the data.** Use **+ Add Filter** to build column/operator/value filters, or use **Expression** filters written in SAS WHERE, R dplyr, or Python pandas syntax. Click **Copy Link** to share a filtered view.
3. **Explore visually.** Switch to the **Explore** tab and pick a column for a histogram, or choose X and Y axes for an X-Y plot. Optionally group by a category.
4. **Ask questions.** If chat is configured, use the **Chat** tab to ask things like "What columns are in this dataset?" or "Show me the distribution of AGE."

<Frame>
    <img alt="Table View of a loaded dataset with missing-values analysis" />
</Frame>

<Frame>
    <img alt="Histogram of a selected column with a category-level bar chart on the side" />
</Frame>

## Installation and setup

An administrator installs Clinical Data Explorer from the Admin panel. See [Install Domino Official Extensions](/cloud/platform-capabilities/features/extensions/install-domino-official-extensions).

The [clinical-data-explorer repository](https://github.com/dominodatalab/clinical-data-explorer) holds the source and [manual install instructions](https://github.com/dominodatalab/clinical-data-explorer/blob/main/INSTALL.md) for reference.

## Troubleshooting

* **"No datasets found"** - Configure the user permissions to access any datasets or Netapp Volumes in the given project context where the extension was opened from.
* **Chat shows "Not Configured"** - Set the LLM environment variables; for cloud providers verify `LLM_API_KEY`.
* **Filters not persisting in links** - Ensure "Deep linking and query parameters" is enabled in the Domino App settings.

## Advanced settings

Most installs work with the defaults. Use these settings when you need to support larger datasets, higher concurrency, or a specific LLM provider.

### LLM and chat configuration

The **Chat** tab uses an OpenAI-compatible API. By default, the app assumes OpenAI, but it can also point at other providers by changing the base URL and model.

* **OpenAI:** Set `LLM_API_KEY` and optionally set `LLM_MODEL` (for example, `gpt-4o`). If `LLM_BASE_URL` is not set, the app uses OpenAI's default API URL.
* **Domino LLM Gateway:** For customer accounts with Domino's installable LLM Gateway module, point `LLM_BASE_URL` at the gateway's OpenAI-compatible endpoint, set `LLM_MODEL` to the gateway model/deployment name, and provide `LLM_API_KEY` if the gateway requires one. This lets the app use centrally governed models without changing app code.
* **Azure OpenAI or other hosted gateways:** Set `LLM_BASE_URL` to the provider's OpenAI-compatible endpoint, then set `LLM_API_KEY` and `LLM_MODEL` as required by that provider.
* **Local Ollama:** Set `LLM_BASE_URL` to an OpenAI-compatible local endpoint such as `http://localhost:11434/v1` and set `LLM_MODEL`. An API key is not required for local Ollama.
* **Chat history:** Use `CHAT_AGENT_MESSAGE_HISTORY_CACHE_SIZE_B` to control the in-memory chat history cache size, and `CHAT_AGENT_MESSAGE_HISTORY_CAP` to cap how many messages are retained per chat session.

### Dataset size, caching, and concurrency

To configure your Domino project settings with regards to dataset size, concurrency, and how downloaded files, MCP server DataFrames, and session metadata are cached, you can follow the guidelines in the [Clinical Data Explorer README.md](https://github.com/dominodatalab/clinical-data-explorer/blob/main/README.md).

<Note>
  The README points to the `main` branch on GitHub. If you're on a different release / version of the app, the instructions on `main` may not match your version. Switch to the corresponding version tag on GitHub that matches your release from the branch / tag dropdown.
</Note>

### Production server settings

See the [Clinical Data Explorer README.md](https://github.com/dominodatalab/clinical-data-explorer/blob/main/README.md) for information on configuring your production server settings.


# Extensions
Source: https://docs.domino.ai/cloud/platform-capabilities/features/extensions/index

Surface Apps in the Domino UI through five Extension mount points, pass page context to Apps, and install or build Extensions for specific tasks.

An Extension surfaces a Domino App at a mount point in the Domino UI: a link in the Project sidebar, an entry on a Dataset details page, an entry on a Model details page, an entry in the three-dot menu for files in a Dataset, or a new page in the Admin panel. Domino opens the App in a new tab and passes the context of the page the user came from as query parameters, so the App can act on the Project, Dataset, file, or Model that user was looking at.

The App behind an Extension is yours to build. One App can back several Extensions, and pointing an Extension at a newer App version is a configuration change an admin makes, not a rebuild.

Three requirements shape the App. It must have extended identity propagation enabled, which needs a framework that can read proxied HTTP headers: [Flask and Dash support them by default, and Shiny needs Server Pro](/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security#extended-identity-propagation). A SysAdmin or CloudAdmin publishes it, and only Admins create the Extension itself. Viewers consent before the App can act as them.

[Create and manage Extensions](/cloud/platform-capabilities/features/extensions/create-manage-extensions) covers the App requirements, the five mount points and the context each one passes, and the guidelines an Extension's App should follow.

## Extensions Domino publishes

These three Extensions are built on the same framework, and the source for each is public in the `dominodatalab` GitHub organization. An administrator installs, upgrades, or uninstalls any of them from the Admin panel, without building the App. See [Install Domino Official Extensions](/cloud/platform-capabilities/features/extensions/install-domino-official-extensions).

| Extension | What it does | Source repository |
| :- | :- | :- |
| [Clinical Data Explorer](/cloud/platform-capabilities/features/extensions/data-visualization) | Explore, filter, and visualize data through an intuitive web interface. | [clinical-data-explorer](https://github.com/dominodatalab/clinical-data-explorer) |
| [AutoML Studio](/cloud/platform-capabilities/features/extensions/auto-ml) | Train, evaluate, and deploy machine learning models through a unified no-code interface. | [AutoML\_Extension](https://github.com/dominodatalab/AutoML_Extension) |
| [Auto Model Documentation](/cloud/platform-capabilities/features/extensions/auto-model-documentation) | Automatically generate professional ML model documentation from code and Experiment Manager artifacts. | [AutoDocumentation\_Extension](https://github.com/dominodatalab/AutoDocumentation_Extension) |


# Install Domino Official Extensions
Source: https://docs.domino.ai/cloud/platform-capabilities/features/extensions/install-domino-official-extensions

Install, upgrade, and uninstall Domino-built Extensions from the Admin panel.

Domino Official Extensions are pre-built Extensions that Domino develops, tests, and distributes, such as [AutoML Studio](/cloud/platform-capabilities/features/extensions/auto-ml), [Clinical Data Explorer](/cloud/platform-capabilities/features/extensions/data-visualization), and [Auto Model Documentation](/cloud/platform-capabilities/features/extensions/auto-model-documentation). Instead of building an App and creating an Extension yourself, an administrator installs one from a catalog in the Admin panel. The installer creates and configures every entity the Extension needs, and handles upgrades and uninstalls.

Installing, upgrading, and uninstalling a Domino Official Extension requires a SysAdmin or CloudAdmin role.

To build your own Extension, see [Create and manage Extensions](/cloud/platform-capabilities/features/extensions/create-manage-extensions).

## How installation works

Each Domino Official Extension is published in a public GitHub repository maintained by Domino. Every released version of an Extension includes a manifest that tells Domino how to set that version up. When you install a version, Domino creates four managed entities in your deployment, in order:

1. A **Project** that hosts the Extension, with the Extension's GitHub repository attached.
2. A **Compute Environment** for the Extension's App, either a Domino standard environment or a custom environment built from the Extension's Dockerfile.
3. An **App**, published from the released code and configured per the manifest.
4. The **Extension**, enabled at the mount points the manifest defines and marked as Domino Official.

Installation runs in the background. The installer shows per-step progress and logs in the Admin panel, and you can retry or cancel if a step fails. Because the manifest defines the App configuration and mount points, there is nothing to configure during installation.

## Prerequisites

* A SysAdmin or CloudAdmin role.
* Extensions enabled on the deployment. Extensions and the identity propagation controls Apps rely on (`SecureIdentityPropagationToAppsEnabled` and `com.cerebro.domino.apps.extendedIdentityPropagationToAppsEnabled`) are on by default in Domino Cloud.
* Network access from the Domino platform to GitHub. The installer downloads the catalog, manifests, and code from Domino's public GitHub repositories.

## Browse available extensions

1. In the Admin panel, open **Extensions**.
2. Select the **Manage Domino-official Extensions** tab.

Each Domino Official Extension is listed with a version selector. The list shows only versions compatible with your Domino version.

## Install an extension

1. In the **Manage Domino-official Extensions** tab, find the Extension.
2. Select the version to install. Choose the latest available version unless you have a specific reason not to.
3. Select **Install**. The installation is queued and continues in the background.
4. Monitor per-step progress and logs from the same tab. You can leave the page, and installation continues.

When installation completes, the Extension appears in the **Extensions** tab marked as Domino Official and is enabled at the mount points its manifest defines. Users who have permission to view the underlying App can use the Extension immediately.

## Upgrade an extension

When a newer compatible version is released, it appears in the Extension's version selector.

1. Select the new version.
2. Select **Install**.

The installer updates the managed entities in place. The existing Project and App are updated to the new release, and the Compute Environment is rebuilt only if it changed between versions. The installer does not create a duplicate set of entities.

Downgrades are not supported. To move to an older version, uninstall the Extension first, then install the version you want.

## Retry a failed install or upgrade

If a step fails, the install pauses and the step's logs explain what went wrong. Two retry options are available:

* **Retry from failed step** undoes only the failed step, then runs it again. Try this first.
* **Clean retry** undoes all completed steps and starts the installation from the beginning.

## Cancel an in-progress install

Cancelling stops the installation and reverts every step that already completed, returning the deployment to its prior state.

## Uninstall an extension

Uninstalling removes the Extension and archives or deletes all the managed entities the installer created: the Project, the Compute Environment, and the App. If Domino cannot remove an entity immediately, the installer reports it and keeps retrying the cleanup in the background. You can start a fresh install without waiting.

## Do not modify managed entities

Nothing prevents an administrator from editing an installed Extension's Project, Compute Environment, App, or Extension configuration, but avoid it. The Extension was tested only with its installed configuration, and manual changes can break the Extension or complicate a future upgrade. Domino records the configuration each version was installed with, and Domino support can compare that record against the live configuration to diagnose issues.

Routine management is fine. Use the toggle in the **Extensions** tab to enable or disable an installed Domino Official Extension, the same as any other Extension. Use the **Manage Domino-official Extensions** tab, not **Manage Extension > Edit**, to change versions.

## Limitations

* An Extension created manually cannot be converted to a Domino Official Extension. The Domino Official designation is only applied by the installer.
* Downgrades are not supported. Uninstall the current version, then install the older version.
* The installer creates entities with fixed names, such as the Project name and the App's vanity URL. If your deployment already has a resource with a conflicting name, the install fails at that step. Resolve the conflict, then retry. Contact [Domino support](https://support.domino.ai/) if you cannot resolve it.

## Next steps

* [Create and manage Extensions](/cloud/platform-capabilities/features/extensions/create-manage-extensions) covers building your own Extension from a Domino App.
* The [Extensions overview](/cloud/platform-capabilities/features/extensions) lists the Extensions Domino publishes and links their public source.


# FinOps
Source: https://docs.domino.ai/cloud/platform-capabilities/features/finops

Cost management for AI work in Domino: attribute cloud spend to the users, Projects, organizations, and billing tags that produced it, then budget against it and report on it.

FinOps attributes your cloud bill to the work that produced it. Domino tags infrastructure usage with the Project, organization, user, and billing tags behind it, so a single deployment-wide invoice resolves into per-Project and per-organization numbers you can budget against, charge back, and act on.

FinOps is one facet of Domino's Governance Center, alongside [policy-based Governance](/cloud/platform-capabilities/features/governance) and the [Audit Trail](/cloud/platform-capabilities/features/governance/audit-trail). Platform administrators configure and read it through the Cost Center dashboard and the Domino Cost API.

## How costs are attributed

Cost Center integrates with Kubecost, which integrates in turn with the major cloud providers, AWS, Azure, and GCP among them. Every hour it ingests your provider's billing data, and reconciles the CPU, GPU, and RAM discounts you negotiated against what you were actually billed. Its allocation engine then maps each hour of spend onto Domino entities: users, Projects, organizations, billing tags, workloads, and workload types.

```mermaid placement="top-right" theme={null}
flowchart TB
    bill["Cloud provider billing<br/>AWS, Azure, or GCP"] --> kubecost["Kubecost<br/>hourly ingest and discount reconciliation"]
    kubecost --> entities["Attribution to Domino entities"]
    entities --> dash["Cost Center<br/>dashboard"]
    entities --> api["Domino<br/>Cost API"]
    entities --> budget["Budgets<br/>and alerts"]
```

Compute and storage spend resolves to an entity this way. Spend that maps to neither, such as shared infrastructure, appears as **Cloud Services**: it counts toward the total, but it drops out of the charts and summaries as soon as you apply a filter, because there is no entity to attribute it to. [Cost calculation methodology](/cloud/admin/operations/finops/4-cost-calculation-methodology) gives the exact formulas, including how provider discounts and credits are applied.

<Note>
  Cost data is retained for up to 1 year. The most recent 36 to 48 hours may be incomplete, because cloud providers report with a delay. Domino reconciles it once the full billing detail arrives.
</Note>

## Where cost data appears

Start with the Cost Center dashboard, at **Govern** > **Cost center**. Move to the API when you need the numbers in another system.

| Surface | Best for | Reference |
| :- | :- | :- |
| Cost Center dashboard | Reviewing spend by resource type, period, and top contributor, filtered by Project, user, billing tag, or workload type | [Analyze costs with Domino Cost Center](/cloud/admin/operations/finops) |
| Domino Cost API | Chargeback and showback reports, and feeding cost data to another system | [Analyze costs using API](/cloud/admin/operations/finops/3-analyze-costs-API) |
| JSON download | Building a Tableau or Power BI dashboard on Cost Center data | [Configuration options for Cost Center](/cloud/admin/operations/finops/2-configuration-options-cost-center) |

The Cost API aggregates by the labels of Domino domain objects, not by arbitrary Kubernetes objects such as clusters. Its `/allocation` and `/asset` paths are restricted to [CloudAdmins](/cloud/admin/identity-and-access/manage-users/roles), and Domino recommends calling it with a [Service Account](/cloud/admin/identity-and-access/manage-users/service-accounts).

## Control spend

Attribution tells you where the money went. Billing tags, budgets, and alerts are what stop it going there again, once cloud billing is connected.

<Steps>
  <Step title="Connect cloud billing">
    Cost Center is preconfigured for AWS. On other providers, provision the cost usage report yourself and ask your Domino representative to set up Cost Center Cloud Billing Integration. [Configuration options for Cost Center](/cloud/admin/operations/finops/2-configuration-options-cost-center) also covers retaining cost data in blob storage for long-term analysis.
  </Step>

  <Step title="Define billing tags">
    Billing tags group Projects for accounting purposes, and a Project can carry several, so you can track cost along a dimension the Project and organization hierarchy doesn't capture. Under **Admin** > **Manage resources** > **Cost tracking** > **Billing tags**, set the billing tag requirement for new Projects to `Disabled`, `Optional`, or `Required`, then create the tag names. See [Configure Billing Tags, Budgets, and Alerts](/cloud/admin/operations/finops/1-budgets-alerts#configure-billing-tags).
  </Step>

  <Step title="Set budgets">
    Projects and organizations start with no budget limit. Under **Admin** > **Manage resources** > **Cost tracking** > **Budget & alerts**, set a default monthly limit, measured by calendar month, that applies to organizations, Projects, and billing tags, then use budget overrides for the ones that need a different figure.
  </Step>

  <Step title="Turn on alerts">
    Alerts are off by default, so a budget on its own notifies nobody. Under **Alert settings**, select **Alert on 75% and 100%** of budget thresholds, and configure email under **Platform settings** > **Email**. Domino then sends a notification daily once a budget reaches 75%, and sends billing tag alerts by email only.
  </Step>
</Steps>

<Note>
  Changing the billing tag requirement doesn't affect existing Projects. They stay untagged until someone applies tags by hand, which is why the requirement is worth settling before a deployment fills up with Projects.
</Note>

## Related

<Columns>
  <Card title="Governance" href="/cloud/platform-capabilities/features/governance">
    Policies, evidence, approvals, and gates: the Governance Center facet that defines what compliance means and blocks actions until it is met.
  </Card>

  <Card title="Audit Trail" href="/cloud/platform-capabilities/features/governance/audit-trail">
    The record of user activity across the deployment, for compliance verification, security investigations, and audit reporting.
  </Card>

  <Card title="Set limits on Dataset usage" href="/cloud/admin/data-administration/datasets#set-limits-on-dataset-usage">
    Cap the storage Datasets consume, and let individual Projects opt out of the limits.
  </Card>
</Columns>


# Approvals
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/approvals

Define governance approvals with approvers, evidence requirements, conditional findings, and recurring revalidation schedules in policy YAML.

Approvals are checkpoints where designated stakeholders review evidence and sign off before a bundle can advance. They ensure that models, apps, and other assets meet organizational standards before progressing through the governance lifecycle. You need the **GovernanceAdmin** role to define approvals in policies. Roles and security has details on role assignment.

Each approval includes:

* A **name**

* A list of **approvers** (users or organizations)

* Optional **evidence** requirements which must be satisfied before approval can be granted

## Define approvals in a policy

Governance administrators define approvals within a stage using YAML.

**Example: Define an approval with evidence**\
In this example, the stage includes an approval named `Stage 4: validation sign off`, which includes a checklist prompt for the approver:

```yaml theme={null}
- name: 'Stage 4: validation sign off'
  approvers:
    - model-gov-org
  evidence:
    id: Local.validation-approval-body
    name: Sign-off
    description: The checklist for approvals
    definition:
      - artifactType: input
        details:
          label: "Have you read the model validation reports?"
          type: radio
          options:
            - Yes
            - No
```

## Conditional approval

Conditional approval lets practitioners proceed to the next stage while they track and address outstanding findings.

A reviewer can approve a stage conditionally by attaching findings to the approval. These findings form a remediation plan and create a formal feedback loop from the reviewer to the modeler.

If you resolve all findings by their due dates, the approval remains valid. If you don’t resolve the findings on time, Domino automatically revokes the approval.

## Periodic revalidation

Approvals can be configured to expire and renew on a recurring absolute schedule, which is anchored to a fixed date, not relative to when an individual approval was granted. This aligns with how organizations run validation programs: on a company-wide cadence rather than a per-model timeline.

Revalidation introduces two new approval states:

* **Pending Expiration**: The renewal window is open. The approver is notified and the bundle remains compliant.

* **Expired**: The window closed without a renewal decision. The bundle is non-compliant.

Governance Admins configure the schedule in YAML within an approval. See [Policy components](/cloud/platform-capabilities/features/governance/define-governance-policies/policy-components#revalidation) for the full YAML reference.

Bundle contributors and approvers can override the schedule at the bundle level without modifying the policy. When an approval is revoked, the schedule resets to none if set by a contributor, or to the policy default if set by a Governance Admin. Approvers receive a Domino notification and email when the window opens, and again when the approval expires. Renewal uses the existing approval workflow.

## Next steps

* [Gates](/cloud/platform-capabilities/features/governance/gating-domino-governance): control governed actions based on approval status

* [Policy components](/cloud/platform-capabilities/features/governance/define-governance-policies/policy-components): review YAML configuration details for policies

* [Review and approve bundles](/cloud/platform-capabilities/features/governance/work-with-bundles/review-bundles): understand how practitioners interact with approvals


# Audit Trail Data Glossary
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/audit-trail/audit-trail-glossary

Find Domino Audit Trail event names and definitions for Apps, files, data, environments, Flows, governance, models, Projects, users, and Workspaces.

The **Domino Audit Trail** provides definitions for all system events and Workspace File Access events including:

* Event categories

* Event names

* Event descriptions

## Application management

### App configuration

| Event Name | Description |
| - | - |
| Update App Custom URL | User updated the custom URL for an App. |

### App lifecycle

| Event Name | Description |
| - | - |
| Publish App | Logged when an App is published and visible to selected users. |
| Start App | Logged when an App is started. |
| Stop App | Logged when an App is no longer running. |

### App permission

| Event Name | Description |
| - | - |
| App Access Granted / Agent Access Granted / GenAI Endpoint Access Granted | Logged when a user or organization is granted access to an app, agent, or GenAI endpoint. |
| App Access Revoked / Agent Access Revoked / GenAI Endpoint Access Revoked | Logged when a previously allowed user or organization has their access removed. |
| App Access Requested / Agent Access Requested / GenAI Endpoint Access Requested | Logged when a user requests access to a grant-based app, agent, or GenAI endpoint. |
| App Access Request Denied / Agent Access Request Denied / GenAI Endpoint Access Request Denied | Logged when a pending access request is denied (explicitly set to Denied or removed from the request list). |
| App Visibility Changed / Agent Visibility Changed / GenAI Endpoint Visibility Changed | Logged when the visibility setting changes (for example, GrantBased, Authenticated, Public). |
| App Discoverability Changed / Agent Discoverability Changed / GenAI Endpoint Discoverability Changed | Logged when the discoverability setting is toggled on or off. |

## Code/artifact file management

### File operations

| Event Name | Description |
| - | - |
| Create Folder | User created a new folder in the project file system. |
| Download Files | Captured when a user downloads a repository file within a project without downloading it. |
| Download Repo File | User downloaded a file from a linked repository. |
| Edit File | Triggered when a user modifies a file within a project. |
| Remove File | Captured when a user deletes a non-git-based repository file. |
| Upload Files | Logged when new files are uploaded into a project or dataset. |
| View File | Captured when a user views a file within a project without downloading it. |
| View Repo File | Captured when a user views a repository file within a project without downloading it. |

### File versions

| Event Name | Description |
| - | - |
| Compare File Revisions | User compared two versions of a file in a repository. |

### Files

| Event Name | Description |
| - | - |
| Rename File / Directory | User renamed a file or directory in a project. |

### Filetasks

| Event Name | Description |
| - | - |
| Create Filetask Task to Delete Download Archive | System created a background task to delete a previously generated download archive. |
| Create Filetask Task to Make Download Archive For Multiple Files or Entire Snapshots | System created a background task to package files or snapshots into a downloadable archive. |

### Git branches

| Event Name | Description |
| - | - |
| Create Branch on Repo | User created a new branch in a linked repository. |

## Data Source management

### Credentials

| Event Name | Description |
| - | - |
| Add Datasource Credentials | User added credentials for accessing a data source. |

### Data Source operations

| Event Name | Description |
| - | - |
| Access Data Source from code execution | Logs when a data source is accessed during code execution (for example, Job, Workspace, App), capturing details of the user, environment, and timestamp of access. |
| Add Data Source Credentials | New credentials added to a data source. |
| Add Data Source to Project | Data source linked to a project. |
| Change Data Source Ownership | Ownership transferred to another user. |
| Change Data Source Permissions | Permissions modified for a data source. |
| Change Data Source Project IDs | Project associations for the data source changed. |
| Create Data Source | Data source created, capturing creator and type. |
| Delete Data Source | Data source permanently deleted. |
| Download Data Source Audit Data | Audit data downloaded for a data source. |
| Get Data Source Audit Data | This is a report. |
| Remove Data Source from Project | Data source removed from a project. |

### External volumes

| Event Name | Description |
| - | - |
| Mount External Data Volumes to Execution | User mounted external data volumes to an execution. |

### Project integration

| Event Name | Description |
| - | - |
| Add Datasource to Project | User linked a data source to a project. |
| Remove Datasource from Project | User removed a data source from a project. |

## Dataset management

### Dataset operations

| Event Name | Description |
| - | - |
| Add Tag | Logged when a tag is added to a dataset or snapshot, used for organizing or versioning. |
| Change Dataset Name | User changed the name of a dataset. |
| Create Dataset | Logged when a new dataset is created within the platform. |
| Delete Dataset | Logged when a dataset is permanently deleted from the system. |
| Delete Dataset Files | Logged when a dataset file is permanently deleted from the system. |
| Download Dataset File | Triggered when a user downloads a file from a dataset. |
| Finalize Delete Dataset | Captured when a dataset is permanently deleted after being marked for deletion. |
| Mark Dataset for Deletion | Captured when a dataset is flagged for deletion but not yet permanently removed. |
| Update Dataset Description | Logged when the metadata or description of a dataset is updated. |
| Upload Files to Dataset | Triggered when new files are added to an existing Domino Dataset folder. |
| View Dataset File | Captured when a user views a dataset file without downloading it. |

### Dataset permissions

| Event Name | Description |
| - | - |
| Add Grant For Dataset | Logged when specific access permissions (grants) are added to a dataset. |
| Associate Dataset To Project | Triggered when a dataset is linked to a project, making it accessible for analysis. |
| Dissociate Dataset From Project | Logged when a dataset is unlinked from a project, removing it from the project’s access. |

### Lifecycle

| Event Name | Description |
| - | - |
| Restore Dataset | User restored a dataset from a deleted state. |

### Permissions

| Event Name | Description |
| - | - |
| Remove Grant For Dataset | User revoked access grant for a dataset. |

### Sharing

| Event Name | Description |
| - | - |
| Add Shared Dataset | User added a shared dataset to a project. |
| Remove Shared Dataset | User removed a shared dataset from a project. |

### Snapshots

| Event Name | Description |
| - | - |
| Add Tag To Snapshot | User tagged a dataset snapshot. |
| Complete Snapshot Creation | Logged when the process of creating a snapshot is successfully completed, capturing the state of the project or dataset. |
| Create Dataset from Snapshot | User created a new dataset from an existing snapshot. |
| Create Snapshot | Captured when a snapshot is created to version data or code for reproducibility purposes. |
| Finalize Delete Snapshot | User finalized the deletion of a dataset snapshot. |
| Mark Snapshot for Deletion | User marked a dataset snapshot for deletion. |

## Environment management

### Configuration

| Event Name | Description |
| - | - |
| Edit Environment Description | User updated the description of an environment. |
| Edit Environment Subscription to Base | User changed the subscription of an environment to a base environment. |
| Set Default Environment | User set the default environment for projects. |

### Environment configuration

| Event Name | Description |
| - | - |
| Edit Environment Name | User changed the display name of an environment. |
| Edit Environment Visibility | User changed the visibility settings of an environment. |

### Environment lifecycle

| Event Name | Description |
| - | - |
| Duplicate Environment | User created a duplicate of an existing environment. |

### Environment operations

| Event Name | Description |
| - | - |
| Create Environment | Logged when a new compute environment is created for use in running jobs or workspaces. |
| Create Environment Revision | - |

### Feature flags

| Event Name | Description |
| - | - |
| Feature Flag changed | Captured when a feature flag is toggled, either enabling or disabling a particular feature. |
| Update Feature Flag for User | Logged when a feature flag is overridden for a specific user, granting them different access from the global setting. |
| Update Feature Flag Globally | Logged when a feature flag is updated globally for the entire organization, overriding default settings or user-level configurations. |

### Hardware

| Event Name | Description |
| - | - |
| Create Hardware Tier | User created a new hardware tier configuration. |

### Lifecycle

| Event Name | Description |
| - | - |
| Archive Environment | User archived an environment, making it inactive. |

### Resource configuration

| Event Name | Description |
| - | - |
| Get Resource Configurations | User retrieved available resource configuration options. |

### Revisions

| Event Name | Description |
| - | - |
| Set As Active Environment Revision | User set a specific environment revision as the active one. |

### Variables

| Event Name | Description |
| - | - |
| Unset Environment Variable | User removed an environment variable from a project or job. |

## Flows

### Dataset integration

| Event Name | Description |
| - | - |
| Flows Artifact Added to Dataset | Artifact produced by a Flow execution was added to a dataset. |

### NetApp Volume integration

| Event Name | Description |
| - | - |
| Flows Artifact Added to NetApp-backed Volume | Artifact produced by a Flow execution was added to a NetApp-backed volume. |

### Execution

| Event Name | Description |
| - | - |
| Flows Create Execution | User created a new Flow execution. |
| Flows Terminate Execution | User terminated a Flow execution. |

### Output

| Event Name | Description |
| - | - |
| Flows View/Download Output | User viewed or downloaded the output of a Flow. |

### Tasks

| Event Name | Description |
| - | - |
| Flows Create Task | User created a task within a Flow. |

### Workflow

| Event Name | Description |
| - | - |
| Flows Create Workflow | User created a new Flow workflow. |

## Governance events

### Audit

| Event Name | Description |
| - | - |
| Download Audit Trail | User downloaded the audit trail. |

### Bundle stages

| Event Name | Description |
| - | - |
| Update Governance Bundle Stage Assignee | User updated the assignee for a governance bundle stage. |

### Bundles

| Event Name | Description |
| - | - |
| Accept Governance Bundle Stage Approval Request | Logged when an approval request is accepted. |
| Add Attachment to Bundle | Triggered when an attachment is added to a bundled. |
| Change Governance Bundle Stage | Logged when the stage (as defined in the policy) of a bundle changes. |
| Change Governance Bundle State | User changed the current state of a governance bundle. |
| Copy Governance Bundle results from another Bundle | Triggered when a results from one bundle is copied to another bundle. |
| Create Governance Bundle | Triggered when a new governance bundle is created. |
| Create Governance Bundle Stage Approval Request | Logged when an approval is requested in a bundle. |
| Create Governance Bundle State | Logged when the state (active or archive) of a bundle changes. |
| Remove Attachment from Bundle | Triggered when an attachment is removed from a bundle. |
| Submit Results in a Bundle | Created when a governance result is added or when saving the notebook. |

### Findings and comments

| Event Name | Description |
| - | - |
| Change Governance Finding Status | Triggered when the status of a finding is changed. |
| Create Governance Finding | Logged when a finding is created. |
| Update Governance Comment events | Logged when a comment is added in a bundle. |

### Policies

| Event Name | Description |
| - | - |
| Add Policy to Governance Bundle | User added a governance policy to a governance bundle. |
| Archive Governance Policy | User archived a governance policy, making it inactive. |
| Change Governance Policy Status | Triggered when the policy changes status (draft or published). |
| Create Governance Policy | Logged when a new policy is created. |
| Deactivate Policy in Governance Bundle | User deactivated a governance policy in a governance bundle. |
| Delete Governance Policy | Logged when a policy is deleted. |
| Update Governance Policy Definition | Logged when any change to a policy is made. |

## Job management

### Jobs

| Event Name | Description |
| - | - |
| Archive Job | Logged when a job is archived, meaning it is no longer active or scheduled for future runs. |
| Change Results Branch Setting | Captured when the results branch setting for a project is changed, affecting where job results are stored or displayed. |
| Launch Job | Logged when a job is launched either manually or via a scheduled task. |
| Restart Job | User restarted an existing job after it failed or was stopped. |
| Stop Job | Logged when a running job is manually stopped before completion. |
| Update Job Name | User updated the display name of a job. |

### Runs

| Event Name | Description |
| - | - |
| Change Run Notification Setting | Captured when the notification settings for job run statuses are changed (for example, email alerts for job completion). |
| Edit Schedule Run | Captured when the details of a scheduled job are edited, such as adjusting the frequency or resource allocation. |
| Schedule Run | Logged when a job is scheduled to execute at a later time or on a recurring cadence. |
| Start Run | Triggered when a job or command is run with specific arguments or parameters. |
| Unschedule Run | Logged when a scheduled job is removed from the execution queue and no longer runs automatically. |

## Model management

### Audit

| Event Name | Description |
| - | - |
| Model Audit Log View | User viewed the audit log entries associated with a model. |

### Lifecycle

| Event Name | Description |
| - | - |
| Create Registered Model | User created a registered model in the model registry. |
| Model Archived | User archived a model, making it inactive but retained for reference. |
| New Model Create | User created a new model object. |
| New Model Publish | User published a model for broader visibility or use. |

### Permissions

| Event Name | Description |
| - | - |
| Model Collaborator Change: ChangeRole | User changed a collaborator’s role on a model. |

### Registered model info

| Event Name | Description |
| - | - |
| Get Registered Model Project Summary | User retrieved a project summary for a registered model. |
| Get Registered Model Version Info | User retrieved version information for a registered model. |

### Registered model settings

| Event Name | Description |
| - | - |
| Update Registered Model Discoverability | User updated whether a registered model is discoverable to others. |

### Settings

| Event Name | Description |
| - | - |
| Model Settings Add Model API Access Token Change | User added a model API access token in settings. |
| Model Settings Description Change | User updated the description in model settings. |
| Model Settings Model Visibility Changed Change | User changed model visibility in settings. |
| Model Settings Regenerate Model API Access Token Value Change | User regenerated a model API access token. |

### Version lifecycle

| Event Name | Description |
| - | - |
| Create Registered Model Version | User created a version of a registered model. |
| Model Version Start | User started a model version for serving or execution. |
| Model Version Stop | User stopped a running model version. |
| New Model Version Create | User created a new version of an existing model. |
| New Model Version Publish | User published a new version of a model. |

## NetApp Volume management

### Data transfer

| Event Name | Description |
| - | - |
| Create Data Transfer | User initiated a data transfer operation involving a NetApp-backed volume. |

### Execution integration

| Event Name | Description |
| - | - |
| Mount NetApp-backed Volumes to Execution | User mounted NetApp-backed volumes during an execution. |

### Files

| Event Name | Description |
| - | - |
| Download Files from NetApp-backed Volume | User downloaded files from a NetApp-backed volume. |
| Preview File from NetApp-backed Volume | User previewed a file from a NetApp-backed volume. |
| Rename File or Directory | User renamed a file or directory within a NetApp-backed volume. |
| Upload Files to NetApp-backed Volume | User uploaded files to a NetApp-backed volume. |

### Filesystem

| Event Name | Description |
| - | - |
| Create NetApp-backed Filesystem | User created a new NetApp-backed filesystem. |
| Delete FileSystem | User deleted a NetApp-backed filesystem. |

### Grants

| Event Name | Description |
| - | - |
| Add Grant for NetApp-backed Volume | User granted access permissions to a NetApp-backed volume. |
| Remove Grant for NetApp-backed Volume | User revoked an access grant for a NetApp-backed volume. |

### Project integration

| Event Name | Description |
| - | - |
| Add NetApp-backed Volume to Project | User linked a NetApp-backed volume to a project. |
| Remove volume from project | User removed a NetApp-backed volume from a project. |

### Snapshots

| Event Name | Description |
| - | - |
| Create NetApp-backed Snapshot | User created a snapshot of a NetApp-backed volume. |
| Delete Snapshot | User deleted a snapshot of a NetApp-backed volume. |
| Download files NetApp snapshot | User downloaded files from a NetApp-backed snapshot. |
| Preview file from NetApp Snapshot | User previewed a file stored in a NetApp-backed snapshot. |

### Volumes

| Event Name | Description |
| - | - |
| Create NetApp-backed Volume | User created a new NetApp-backed volume. |
| Delete Files from NetApp-backed Volume | User deleted files stored on a NetApp-backed volume. |
| Delete NetApp-backed Volume | User deleted a NetApp-backed volume. |
| Mark NetApp-backed Volume for Deletion | User marked a NetApp-backed volume for deletion. |
| Restore Volume | User restored a previously deleted or archived NetApp-backed volume. |
| Update NetApp-backed Volume Description | User updated the description of a NetApp-backed volume. |
| Update NetApp-backed Volume Name | User updated the name of a NetApp-backed volume. |

## Project and resource management

### Collaboration

| Event Name | Description |
| - | - |
| Remove Collaborator | User removed a collaborator from a project. |

### Deployment targets

| Event Name | Description |
| - | - |
| Get Deployment Targets | User retrieved a list of deployment targets. |
| Get Deployment Target | User retrieved details of a specific deployment target. |
| Get Deployment Target Types | User retrieved available deployment target types. |
| Get Selectable Deployment Targets | User retrieved a filtered list of selectable deployment targets. |

### External volumes

| Event Name | Description |
| - | - |
| Add External Data Volumes to Project | User added external data volumes to a project for use in execution. |
| Remove External Data Volume from Project | User removed external data volumes from a project. |
| Update External Data Mount | User updated an external data mount configuration. |
| Update External Data Volume | User updated details of an external data volume. |

### Git integration

| Event Name | Description |
| - | - |
| Add Imported Git Repo | User added an imported Git repository to a project. |
| Edit Imported Git Repo | User edited settings of an imported Git repository. |

### Launchers

| Event Name | Description |
| - | - |
| Create Run Launcher | User created a run launcher to enable recurring or parameterized execution. |
| Edit Run Launcher | User edited the configuration of a run launcher. |
| Start Launcher Run | User started a run using a launcher configuration. |

### Project environment

| Event Name | Description |
| - | - |
| Change Project Environment | User changed the compute environment associated with a project. |

### Project lifecycle

| Event Name | Description |
| - | - |
| Clone Project | User cloned an existing project into a new project. |
| Copy Project | User copied a project, including its resources and configuration. |
| Project stage changed | A project's stage was changed programmatically through the API or CLI. |

### Project management

| Event Name | Description |
| - | - |
| Archive Project | Logged when a project is archived, stopping all scheduled and in-progress runs and restricting further uploads/downloads. |
| Create Project | Logged when a new project is created within Domino, typically to organize new data science work. |
| Create Restricted Project | Captured when a restricted project is created, which limits access based on user roles or permissions. |
| Set Environment Variable | Captured when an environment variable is set for a specific environment, influencing the behavior of jobs or workspaces. |
| Update Project Setting Event | Captured when project settings are modified, such as changing visibility, the name of the project, or the description of the project. |

### Project tagging

| Event Name | Description |
| - | - |
| Add Tags To Project | User added one or more tags to a project. |
| Create Tag | User created a tag for use in projects. |
| Delete Tag | User deleted a project tag. |
| Merge Tags | User merged two or more tags. |
| Remove Tag From Project | User removed a tag from a project. |
| Update Tag | User updated a tag associated with a project. |

### Project templates

| Event Name | Description |
| - | - |
| Archive Custom Project Template | User archived a custom project template. |
| Create Custom Project Template | User created a custom project template for reuse. |
| Create Project from Template | User created a project using a template. |

### Resource management

| Event Name | Description |
| - | - |
| Generate Resource Usage Report | Logged when a resource usage report is generated, showing memory, CPU, and storage consumption for auditing or monitoring. |

### System configuration

| Event Name | Description |
| - | - |
| Create Central Config Setting | User created a configuration record setting. |
| Delete Central Config Setting | User deleted a configuration record setting. |
| Update Central Config Setting | User updated a configuration record setting. |

## System / Workspace management

### Queries

| Event Name | Description |
| - | - |
| Run Mongo Query | User executed a MongoDB query via the system. |

### Support

| Event Name | Description |
| - | - |
| Generate Support Bundle | User generated a support bundle for troubleshooting. |

## Tag and property management

These events record changes to [Tags](/cloud/platform-capabilities/core-concepts/knowledge-management/tags) and [Properties](/cloud/platform-capabilities/core-concepts/knowledge-management/properties). Create Tag, Delete Tag, Merge Tags, and Update Tag share names with [Project tagging](#project-tagging) events. The versions in this section record the tag's Namespace in the event metadata.

### Namespaces

| Event Name | Description |
| - | - |
| Bulk Delete Namespaces | User deleted multiple Namespaces in one request. The event records the deleted Namespace IDs and the number deleted. |
| Create Namespace | User created a Namespace. The event records whether the Namespace allows multiple tags per entity. |
| Delete Namespace | User deleted a Namespace. |
| Update Namespace | User changed a Namespace's label, description, status, or **Allow multiple tags per entity** setting. The event records the old and new value of each changed field. |

### Tags

| Event Name | Description |
| - | - |
| Bulk Delete Tags | User deleted multiple tags in one request. The event records the deleted tag IDs and the number deleted. |
| Create Tag | User created a tag in a Namespace, at the root or under a parent tag. The event records the Namespace and parent tag. |
| Delete Tag | User deleted a tag. The event records the Namespace, the parent tag, and the number of child tags deleted with it. |
| Import From File | User imported a tag structure from a file. The event records the number of Namespaces and tags created. |
| Merge Tags | User merged one or more source tags into a target tag. The event records the source and target tags, the number of tags merged, and the number of entities reassigned to the target tag. |
| Update Tag | User changed a tag's label, description, parent tag, or status, for example to deprecate it. The event records the old and new value of each changed field. |

### Tag assignments

| Event Name | Description |
| - | - |
| Copy Entity Tags | User copied all tags from one entity to another entity of the same type, replacing the target's existing tags. The event records the source and target entities and the number of tags copied. |
| Delete Entity Tags | User removed all tags from an entity. The event records the number of tags removed. |
| Tag Entity | User applied one or more tags to an entity, such as a Project, Dataset, or App. The event records the tags and their Namespaces. |
| Untag Entity | User removed one or more tags from an entity. The event records the tags removed. |

### Properties

| Event Name | Description |
| - | - |
| Create Property | User defined a property. The event records its type, applicable assets, group, and allowed values for select types. |
| Delete Property | User deleted a property definition. |
| Update Property | User changed a property's label, group, description, applicable assets, or allowed values. The event records the old and new value of each changed field. |

### Property values

| Event Name | Description |
| - | - |
| Clear Property Values | User removed every property value from an entity. The event records the number of values removed and, for a model version, the version. |
| Copy Property Values | User copied all property values from one entity to another entity of the same type, replacing the target's existing values. The event records the source and target entities and the number of values copied. |
| Updated Property Values | User set or cleared one or more property values on an entity. |

### Cleanup

| Event Name | Description |
| - | - |
| Reconcile Orphan Entity | Domino removed tags and property values that belonged to a Model or model version that no longer exists in the Model Registry. The event records the number of tags and property values deleted. |

## User management

### Authentication

| Event Name | Description |
| - | - |
| Rest Password Triggered for User | A user triggered a password reset process. |
| Save Kerberos Config For User | User saved Kerberos configuration details. |

### Global user roles and permissions

| Event Name | Description |
| - | - |
| Update User Global Roles | Logged when the role of a user in a project or workspace is updated. |
| User Activated | Captured when a previously deactivated user is reactivated, granting them access again to the organization and its resources. |
| User Deactivated | Captured when a user is deactivated from the system, removing their access. |
| User Marked As Domino Employee | A user has been designated as a Domino employee, indicating their internal role and permissions within the platform. |

### Project user roles and permissions

| Event Name | Description |
| - | - |
| Add Collaborator | Logged when a new collaborator is added to a project. |
| Change User Role In Project | Captured when a user’s role is changed within a specific project. |

### User activity report

| Event Name | Description |
| - | - |
| Generate User Activity Report | Logged when a report is generated that tracks user actions within the platform. |

### Organization

| Event Name | Description |
| - | - |
| Create Organization | User created a new organization in Domino. |

### User roles and permissions

| Event Name | Description |
| - | - |
| Add Users To Organization | Captured when one or more users are added to the organization, giving them access to projects and resources. This could be done individually or in bulk. |
| Change Organization Owner | Logged when the ownership of the organization is transferred to another user, giving them administrative control over all projects, environments, and users within the organization. |
| Remove User From Organization | Captured when a user is removed from the organization, revoking their access to all projects, datasets, and collaborative workspaces within the organization. |

## Workspace management

### Configuration

| Event Name | Description |
| - | - |
| Update Workspace | User updated workspace configuration or settings. |
| Update Workspace Title | User updated the title of a workspace. |

### Environment management

| Event Name | Description |
| - | - |
| Change Organization Default Environment | Logged when the default environment for all projects in the organization is changed, impacting the computational setup across multiple projects. |
| Restrict Environment Revision | Logged when environment revisions are restricted to ensure that only a specific version of the environment can be used. |

### File access

| Event Name | Description |
| - | - |
| Create | Logged when a user creates a NetApp Volume or Domino Dataset file within a Domino Workspace. Captures file path, workspace name, environment, and project association. |
| Delete | Logged when a user deletes NetApp Volume or Domino Dataset file within a Domino Workspace. Captures file path, workspace name, environment, and project association. |
| Read | Logged when a user reads or opens a file within a Domino Workspace, including NetApp Volume or Domino Datasets Files. Captures file path, workspace name, environment, and project association. |
| Rename | Logged when a user renames a file within a Domino Workspace, including NetApp Volume or Domino Datasets Files. Captures file path, workspace name, environment, and project association. |
| Write | Logged when a user modifies or saves a file in a workspace-mounted NetApp Volume or Domino Dataset. Includes file path, dataset name, and workspace context. |

### Hardware management

| Event Name | Description |
| - | - |
| Edit Hardware Tier | Tracks modifications to the hardware tier settings for a project, affecting the computational resources used for jobs or workspaces. |
| View Hardware Tier | Records when a user views the hardware tier configuration, detailing the compute resources allocated for project execution. |

### Logs

| Event Name | Description |
| - | - |
| Get Workspace Logs | User retrieved logs for a workspace session. |

### Workspace management

| Event Name | Description |
| - | - |
| Delete a Workspace | Logged when a workspace is deleted, removing all files and resources tied to it. |
| Start a Workspace Session | Logged when a workspace session is started, allowing interactive development work in Domino. |
| Start InteractiveSessionRunCommand Run | Captured when an interactive session command is executed within a workspace. |
| Stop a Workspace Session | Captured when a workspace session is stopped, either manually or due to inactivity. |
| Stopped Workspace Marked For Deletion | Triggered when a workspace that is no longer in use is marked for deletion. |
| Synchronize Working Directory | Logged when a working directory is synchronized with the latest version of the codebase, ensuring reproducibility. |

### Workspace volume

| Event Name | Description |
| - | - |
| Workspace Deleted by Volume Cleaner | Triggered when a workspace is automatically deleted by Domino’s volume cleaner to manage storage. |
| Workspace Volume Start Settings | Captured when the settings for workspace volume are initialized at the start of a session. |

## Next steps

* [System events](/cloud/platform-capabilities/features/governance/audit-trail/view-unified-audit-trail): View and export user and administrative actions

* [Workspace File Access events](/cloud/platform-capabilities/features/governance/audit-trail/workspace-file-access-events): How Domino captures file-level operations in workspaces

* [Audit Trail API](/api-reference/audit-trail/get-apiaudittrailv1auditevents): Programmatically query and export audit events

* [Workspace File Audit Trail API](/cloud/reference/api/workspace-file-audit-trail-api): Query and export workspace file access events


# Domino Audit Trail
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/audit-trail/index

Audit user actions, configuration changes, and Workspace file access across Domino through searchable UI views and programmatic APIs.

Domino records user activity across your Domino deployment for compliance verification, security investigations, and audit reporting. You can access audit events through purpose-built UI views and APIs. Domino retains audit data for 30 years to meet regulatory and governance requirements.

Access requires one of these roles: **CloudAdmin** or **GovernanceAdmin**.

Domino captures two categories of audit events:

[System events](/cloud/platform-capabilities/features/governance/audit-trail/view-unified-audit-trail) capture user-initiated actions executed via the UI or API. These actions either change system state (for example, editing a dataset) or access data (for example, viewing a dataset). Examples include:

* Project, dataset, and environment actions

* User, role, and permission changes

* Job, App, workflow, and model lifecycle events, including the start and stop of each App version

* File interactions that occur through the Domino interface

[Workspace File Access events](/cloud/platform-capabilities/features/governance/audit-trail/workspace-file-access-events) capture file-level access to Domino Datasets and NetApp Volumes that occur within Domino Workspaces, such as file reads and writes performed from code, terminals, or IDEs. It records when users view, download, or modify files in NetApp Volumes or Domino Datasets.

Together, these event categories capture all audit activity across Domino: user actions, configuration changes, and file-level data access.

## Next steps

* [System events](/cloud/platform-capabilities/features/governance/audit-trail/view-unified-audit-trail): View and export user and administrative actions

* [Enable Workspace File Audits](/cloud/platform-capabilities/features/governance/audit-trail/workspace-file-access-events/use-workspace-file-audit-app): Configure file-level access auditing

* [Audit Trail Data Glossary](/cloud/platform-capabilities/features/governance/audit-trail/audit-trail-glossary): Look up event names and definitions for Apps, files, data, environments, Flows, governance, models, Projects, users, and Workspaces

* [Audit Trail API](/api-reference/audit-trail/get-apiaudittrailv1auditevents): Programmatically query and export audit events

* [Workspace File Audit Trail API](/cloud/reference/api/workspace-file-audit-trail-api): Query and export workspace file access events


# System events
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/audit-trail/view-unified-audit-trail/index

Filter, inspect, and export Domino system events for Project, Dataset, user, Job, App, workflow, model, and governance activity.

System events track administrative and user actions that occur through the Domino platform interface and APIs. Domino records these actions as system events:

* Project creation, cloning, archiving, and configuration

* Dataset creation, updates, snapshots, and platform access

* Repository file operations via the UI

* User and role management

* Application, job, workflow, and model lifecycle activity

* Governance workflows, approvals, and findings

* NetApp-backed volume and external data source management

* Configuration record changes

## Filter and view system events

The Audit Trail table displays event name, timestamp, user, and target resource.

<img alt="Download event list" />

1. Go to **Admin** > **Audit Trail** to open the event table.

2. Select **Add filters**, choose an attribute, select an operator, and enter one or more values.

3. Add more filters as needed.

4. Select any event row to view full details.

## Export system events

Export filtered audit data for reporting or archival.

### From the UI

1. Filter events to the desired results.

2. Select **Download with current filter**.

3. Save the file locally.

### From the API

Use the [Audit Trail API](/api-reference/audit-trail/get-apiaudittrailv1auditevents) to programmatically access system events.

* Authenticate with standard Domino API credentials

* Filter by time range, user, event type, or resource

* Export data for external reporting or archival

## Next steps

* [Use the Domino Audit Trail](/cloud/platform-capabilities/features/governance/audit-trail): Overview of Domino’s unified audit capabilities

* [Workspace File Access events](/cloud/platform-capabilities/features/governance/audit-trail/workspace-file-access-events): How Domino captures file-level operations in workspaces

* [Audit Trail API](/api-reference/audit-trail/get-apiaudittrailv1auditevents): Programmatically query and export audit events

* [Workspace File Audit Trail API](/cloud/reference/api/workspace-file-audit-trail-api): Query and export workspace file access events


# Enable Workspace File Audits
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/audit-trail/workspace-file-access-events/enable-workspace-file-audits

Enable Workspace File Audits on AWS or Azure by provisioning working and archive storage, then configuring audit settings in agent.yaml.

Enable Workspace File Access auditing to capture file-level operations in workspaces. This feature is opt-in and requires enablement before use.

## Prerequisites

Verify that you have the following:

* Domino version 6.2 or later

* A supported cloud deployment: AWS or Azure

* Two object storage buckets in your cloud provider

### Enable the service

Enable Workspace File Audits by configuring object storage buckets and Domino settings. You need two buckets, one for temporary staging and an archive bucket for long-term storage:

* **Working bucket**: Temporary staging for raw audit events

* **Archive bucket**: Long-term storage for processed audit events

You can provision these buckets using the Domino Terraform module or create them manually.

#### Use the Domino Terraform module

The [Domino Terraform module](https://github.com/dominodatalab/terraform-aws-eks/blob/026e1e065e8bd6df194b51116e545959d8b696e0/modules/infra/submodules/storage/s3.tf#L608) contains the bucket configuration details.

1. Enable the `workspace_audit` option in the Terraform configuration.

2. Re-run the Terraform module to provision the required buckets (working and archive).

3. Contact your Domino administrator if you have questions or run into issues.

#### Without the Terraform module

Create and configure the buckets manually in your cloud provider.

1. Create the working bucket and archive bucket in your cloud blob storage provider.

2. Verify that both buckets have the required IAM permissions for Domino to read and write.

#### After you create the buckets

Update the `agent.yaml` file to enable workspace audit and provide the bucket details.

The setting `enabled: true` triggers the full infrastructure setup: enabling Falco, updating the Falco configuration, deploying the workspace audit service, and connecting to the provisioned buckets.

**AWS**:

```yaml theme={null}
workspace_audit:
  cloud_provider: aws
  enabled: true
  event_archive_storage_aws:
    bucket: <your-archive-bucket-name>
    region: <your-region>
    sse_kms_key_id: <your-kms-key-id>
  event_working_storage_aws:
    bucket: <your-working-bucket-name>
    region: <your-region>
    sse_kms_key_id: <your-kms-key-id>
```

**Azure**:

```yaml theme={null}
workspace_audit:
  cloud_provider: azure
  enabled: true
  event_archive_storage_azure:
    account_name: <your-azure-storage-account-name>
    account_key: <your-azure-storage-account-key>
    container: <your-archive-container-name>
  event_working_storage_azure:
    account_name: <your-azure-storage-account-name>
    account_key: <your-azure-storage-account-key>
    container: <your-working-container-name>
```

## Next steps

* [Use the Workspace File Audit App](/cloud/platform-capabilities/features/governance/audit-trail/workspace-file-access-events/use-workspace-file-audit-app): Query and export audit data

* [Monitoring and alerting](/cloud/platform-capabilities/features/governance/audit-trail/workspace-file-access-events): Set up Grafana alerts for audit pipeline health


# Workspace File Access events
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/audit-trail/workspace-file-access-events/index

Track read, write, create, delete, and rename operations on Domino Datasets and NetApp Volumes, then query archived Workspace audit events.

Workspace File Access events capture file-level access activity inside Domino Workspaces. Domino tracks these events only for Domino Datasets and NetApp Volumes. Domino records Workspace lifecycle actions as [system events](/cloud/platform-capabilities/features/governance/audit-trail/view-unified-audit-trail).

Domino records these file operations:

* Read

* Write

* Create

* Delete

* Rename

## How Workspace file access auditing works

Domino monitors file system activity inside active Workspaces and records relevant events.

<img alt="Workspace audit trail diagram" />

### File system monitoring

A Falco daemon runs alongside each workspace pod and observes file system calls. Falco monitors at the kernel level, so it captures file access consistently. This works regardless of access method: code execution, terminals, notebooks, or IDE tooling.

### Deduplication

Domino deduplicates events to reduce noise and storage overhead. The system records repeated access to the same file by the same user as a single event, using a deduplication time window that reduces event volumes while preserving meaningful access records.

### Event processing and storage

The system periodically processes staged events and writes them to object storage. The pipeline:

1. Cleans and deduplicates raw JSON events

2. Converts JSON to Parquet format for optimized querying

3. Archives events for long-term retention (up to 30 years)

### Processing delay

Domino processes events asynchronously in scheduled batches, not in real time. The system captures and stages file system events continuously. It then processes and writes them to object storage during each run (default: every 60 minutes).

Expect a delay between when a file access event occurs and when it appears in object storage, APIs, or the Workspace File Audit App.

You can configure processing frequency with `com.cerebro.domino.workspaceFileAudit.eventProcessingInMinutes`. The default is 60 minutes (minimum 60, maximum 360).

### Query and access

You can access events through:

* The Workspace Audit App

* The Audit Trail API

* Direct object storage queries

### Storage and retention

Storage location and retention depend on your deployment type:

| Deployment type | Storage location | Who manages | Retention |
| - | - | - | - |
| Customer-managed | Customer-owned object storage | Customer | Customer sets policy |
| Domino Cloud | Domino-owned object storage | Domino | 30 years |
| Domino Cloud for Life Sciences (DCLS) | Customer-owned S3 | Domino | 30 years |

### Performance considerations

Enabling Workspace File Access events adds processing overhead to active workspaces. Based on Domino performance testing, enabling file access auditing typically adds:

* \~ 15 percent CPU usage

* \~ 10 percent memory usage

Workloads with heavy workspace usage or high file I/O may see higher overhead. Plan for larger hardware tiers if you enable this feature.

**How hardware sizing is determined**

These estimates come from Domino scale and performance testing of the full audit pipeline, including kernel level event capture, event ingestion, and audit record processing. Customers should plan for slightly larger hardware tiers when this feature is enabled.

### Monitoring and alerting

In cloud deployments, Domino’s platform team monitors the Workspace Audit pipeline and manages all underlying audit infrastructure alerts on your behalf.

To monitor the health and activity of your own workloads, go to `/grafana-workload`.

#### If you suspect audit events are delayed or dropped

Contact Domino Support. Provide the approximate time the issue began and any relevant workspace details.

Normal audit capture resumes after Domino resolves the issue.

## Next steps

* [Enable Workspace File Audits](/cloud/platform-capabilities/features/governance/audit-trail/workspace-file-access-events/enable-workspace-file-audits): Configure file-level access auditing

* [Use the Workspace File Audit App](/cloud/platform-capabilities/features/governance/audit-trail/workspace-file-access-events/use-workspace-file-audit-app): Deploy the app, then filter and export file-level audit events by date, user, Project, or path

* [System events](/cloud/platform-capabilities/features/governance/audit-trail/view-unified-audit-trail): View and export user and administrative actions


# Use the Workspace File Audit App
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/audit-trail/workspace-file-access-events/use-workspace-file-audit-app

Deploy the Workspace File Audit App, filter events by date, user, Project, or path, and export results as Parquet, CSV, or API data.

Deploy the app, then query and export file-level audit data by date range, user, project, or file path.

## Deploy the application

Set up the **Workspace File Audit App** to query and export audit data. The app is available from the Domino GitHub repository and requires the Domino Standard Environment (DSE).

<Tip>
  Hardware requirements depend on query scope. For typical usage (up to one month of events), use at least 6 CPU cores and 8 GB memory. For larger time ranges, use more resources.
</Tip>

1. Go to **Projects** > **New Project**, then select **Import from Git** and enter the repository URL:

   ```markdown theme={null}
   https://github.com/dominodatalab/Workspace-File-Audit-Application
   ```

2. Complete the project creation and open the project.

3. Go to **Deployments** > **Apps & Agents** > **Publish** > **App**.

4. Configure the deployment:

   * **Name** and **Description** (example: "Workspace File Audit Query Tool").

   * **App file**: `start.sh`

   * **Environment**: Latest DSE

   * **Hardware Tier**: Select based on your organization’s tier names and resource needs

5. Select **Enable deep linking and query parameters** then click **Publish**.

6. Wait for the app status to show **Running**, then select **Open**.

**Note**: If the DSE is not available in your deployment, contact your Domino administrator.

## Filter and view events

Filter audit events by date range, event type, user, project, or file path.

<img alt="Workspace File Audit App" />

1. In the Workspace File Audit App, select a **Date Range** (recommended: 90 days or less; maximum: 1 year).

2. Select filters for **Event**, **Username**, **Project Name**, or **File path**.

3. Select **Submit**.

Results display in two views:

* **Events Over Time chart**: Total event counts across your timeframe

* **Events Details table**: Individual events with timestamp, user, event type, and file path

**Note**: New event data becomes available approximately every hour.

## Export events

Export system events from the UI or API for compliance reporting, security investigations, or long-term archival.

1. Filter events to the desired results.

2. Select **Download Parquet** or **Download CSV**.

3. Save the file locally.

The system automatically splits and packages CSV exports over 1 million records as a ZIP archive.

## Access events via API

Use the [Workspace File Audit Trail API](/cloud/reference/api/workspace-file-audit-trail-api): to programmatically access Workspace File Access events:

* Authenticate with standard Domino API credentials

* Filter by time range, user, event type, or resource

* Export data for external reporting or archival

## Access events in object storage

Domino stores processed events as Parquet files. Query them directly with cloud tools or analytics engines like DuckDB.

**Domino Cloud and DCLS**\
Domino stores audit events in Domino-hosted object storage within a single-tenant, fully managed environment. Domino manages storage infrastructure and access controls. Request credentials from Domino for authorized access.

## Customize the Workspace File Audit App

Clone the official repository to create a custom version for your organization. This lets you extend functionality, apply internal branding, or control rollout timing.

<Warning>
  Customized apps are not tested/validated for each new release; only our official Domino App is supported.
</Warning>

1. Clone the repository to your organization’s Git account.

2. Create a Git-based project in Domino pointing to your cloned repository.

3. Deploy your custom version.

## Next steps

* [Workspace File Access events](/cloud/platform-capabilities/features/governance/audit-trail/workspace-file-access-events): How Domino captures file-level operations in workspaces

* [Monitoring and alerting](/cloud/platform-capabilities/features/governance/audit-trail/workspace-file-access-events#monitoring-and-alerting): Set up Grafana alerts for audit pipeline failures

* [Workspace File Audit Trail API](/cloud/reference/api/workspace-file-audit-trail-api): Query and export workspace file access events


# Define policies
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/define-governance-policies/index

Create governance policies in Policy Builder or YAML, configure stages and rules, enforce them for Apps, and manage published versions.

Policies define review and approval workflows for deliverables, such as models, reports, and AI systems. Use policies to codify internal risk controls, regulatory frameworks, or corporate standards across your organization.

Only Governance administrators can create, edit, or publish policies. [Roles and security](/cloud/platform-capabilities/features/governance/roles-and-security) has details on role assignment.

You can build policies in two ways:

* **Policy Builder**: A visual UI for defining policies step-by-step

* **YAML**: A code-based method for advanced customization

Once published, policies enforce compliance at runtime when applied to governed bundles.

## Use cases for policies

Policies can:

* Require approvals before releasing models or reports

* Capture evidence and risk classifications during review

* Show or hide questions based on input

* Define multi-stage workflows with approvers and conditions

* Use gates to restrict specific actions, such as deploying a published App, an agent, or a model Endpoint, until all required approvals are complete

## Create a policy

You can create a policy either from scratch or by using an existing template. Both methods allow you to use the Policy Builder UI or the Code editor.

### Start a new policy

Follow these steps to begin defining your policy:

1. Go to **Govern** > **Policies**, then click **Create Policy**.

2. Choose **None (start from scratch)** or choose a policy template.

3. Enter a name and description and click **Create**.

### Choose a view

Once created, the policy opens in the **Policy Builder** by default. You can switch between these views at any time:

* **Policy Builder (UI editor)**: Guided form for building stages, rules, and questions

* **Graph view**: Visual overview of the policy’s workflow

* **Code editor**: Raw YAML editor for advanced configuration

**Note**: If editing YAML directly, remember to replace `user-organization-name` with your organization’s name to correctly assign approvers.

Policies are written in YAML and must be published before use. [Build governance policies](/cloud/platform-capabilities/features/governance/define-governance-policies/policy-components) in our documentation has more building blocks for advanced customization.

### Use the policy builder

The Policy Builder is a guided interface for defining policy components without writing YAML.

<img alt="Governance Policy Builder" />

In this UI editor, you can:

* Define policy stages such as validation or deployment

* Assign approvers to each stage

* Add evidence questions approvers must answer

* Add scripted checks for model validation

* Configure classification and visibility rules

Domino automatically generates YAML behind the scenes as you work in the builder. You can view or modify this YAML in the **Code editor** at any time.

## Define rules and logic

You can use rules to evaluate evidence, classify policy executions, and control which sections of a policy are shown to users. Classification and visibility rules often work together. For example, a classification rule can drive a visibility rule.

### Classification rules

You can use **classification rules** to apply tags or labels to a policy execution based on collected evidence. These outputs can:

* Trigger downstream rules

* Influence policy bundle matching

* Inform audit or compliance filters

[Policy components](/cloud/platform-capabilities/features/governance/define-governance-policies/policy-components) has YAML configuration details and examples for classification rules.

### Visibility rules

**Visibility rules** can be applied to sections of a policy, such as to show or hide specific stages or questions based on earlier responses. Use the eye icon in the section header to define a visibility condition.

<img alt="Visibility Rules Form" />

To configure a rule:

* Choose the input question to evaluate

* Specify the value that must be selected to make the section visible

Use a **custom visibility rule** if you need to:

* Reference more than one input question

* Use classification values to control visibility

Custom visibility rules can reference the output of a classification rule. This allows you to show or hide sections based on computed logic, not just raw inputs.

[Policy components](/cloud/platform-capabilities/features/governance/define-governance-policies/policy-components) has YAML configuration details and examples for visibility rules.

## About gating

Gating controls when certain operations are allowed based on policy status. For example, you can block a user from deploying a published App or an agent until all stages in the associated policy bundle are approved.

Gates must be defined in YAML. [Gates](/cloud/platform-capabilities/features/governance/gating-domino-governance) has details about how gates work and how to define them.

## Enforce required policies

Policy enforcement lets Governance admins define a set of policies that Domino attaches automatically to every governed asset of a given type, deployment-wide. This closes the gap where a practitioner could bypass governance by not attaching a policy. The current release supports policy enforcement for apps.

Enforced policies always attach at their latest version. Practitioners cannot remove an enforced policy from a bundle. Governance, System, and Cloud admins can remove one if needed.

### Configure the enforced policy set

1. Go to **Govern** > **Policies**.

2. Click **Policy enforcement**.

3. Select the policies to require for apps, then save. Use the toggle to turn enforcement on or off; changes take effect on the next app version.

### How enforcement applies to apps

Enforcement runs when an app version is created:

* **New app**: Domino creates the bundle with every currently enforced policy attached.

* **New version of an ungoverned app**: If the app has no bundle yet, Domino creates one with every enforced policy attached.

* **New version of a governed app**: Domino attaches any enforced policies that are missing from the bundle and upgrades already-attached enforced policies to their latest version.

If an admin removed an enforced policy from an earlier app version's bundle, the policy attaches again on the next app version as long as it is still in the enforced set.

## Create a new version of a policy

You can modify a published policy by creating a new version. The new version remains in draft until it is published.

When publishing the new version, you can:

* Choose whether the update is **mandatory** or **optional** for bundles using an earlier version

* Select which bundles are affected

* Decide whether to retain or revoke existing approvals

Domino displays a list of bundles using the current version. If you mark the update as mandatory, you must select which bundles it applies to. Those bundles are marked out of compliance until upgraded.

You can choose whether to retain or revoke existing approvals. Only revoke existing approvals if the new version includes significant or breaking changes.

### Policy diff

When drafting a new policy version or reviewing a published one, you can use the **Changes** tab to compare it against the previous version. The diff view shows both a side-by-side YAML comparison and a visual representation of structural changes, making it easy to spot what was added, removed, or modified.

<img alt="Policy Diff view showing YAML comparison between policy versions" />

### Mandatory version update

When publishing a mandatory update:

* Select the bundles that must upgrade during the publish step

* Affected bundles are blocked from further edits until upgraded

* To upgrade a bundle, open it and click **Upgrade Policy** in the warning message

### Optional version update

For minor or non-blocking changes, you can publish the new version as an optional update.

* Bundles governed by the previous version display a warning that a new version is available

* These bundles are not blocked and can continue to be edited

## Archive policies

You can archive policies to reduce clutter in the policy table. Archived policies:

* Cannot be restored

* Do not link to a detail page

* Only appear when the Archived filter is applied

To archive a policy:

1. Go to **Govern** > **Policies** and find the policy to archive.

2. Click the **…** menu next to the policy and select **Archive**.

3. If no active bundles use the policy, click **Confirm**.

4. If active bundles are using the policy, Domino shows a warning and prevents archival.

## Next steps

* [Policy components](/cloud/platform-capabilities/features/governance/define-governance-policies/policy-components): review YAML configuration details for policies

* [Gates](/cloud/platform-capabilities/features/governance/gating-domino-governance): control governed actions based on approval status

* [Roles and security](/cloud/platform-capabilities/features/governance/roles-and-security): understand permissions and role assignment for Governance


# Policy components
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/define-governance-policies/policy-components

Define governance policy YAML for stages, approvals, evidence, checks, input controls, classification, visibility rules, and deployment gates.

Governance lets you define how projects are reviewed, approved, and classified using policies written in YAML. These policies are managed in the **Governance Console** and applied to governed bundles to enforce consistent, auditable workflows.

Each policy defines what evidence needs to be collected, who must approve it, how risk is assessed, and whether any warnings or gates should apply. Once a policy is published, it can’t be edited. This guarantees that any bundle stays tied to the exact policy version it was approved under.

Only Governance administrators can create or edit policies. [Roles and security](/cloud/platform-capabilities/features/governance/roles-and-security) has details and role assignment steps.

## How policies are written

Policies are defined in YAML using a modular structure. Each policy is made up of components like stages, approvals, inputs, and checks. Each component is declared with an `artifactType` and a `details` section that sets its behavior.

Writing policies in YAML makes them easy to version, reuse, and review, whether you’re building one from scratch or editing an existing template.

## What this page covers

This page explains how to define each type of policy component using YAML. It includes examples for:

* Organizing stages and approvals

* Grouping and reusing evidence

* Running metrics checks and scripted checks

* Creating input fields and guidance elements

* Defining classification logic and visibility rules

* Gating high-risk actions

## Stages

Governance Admins define stages in YAML to organize evidence and approvals. Each stage can include:

* One or more evidence sets (for direct evidence)

* One or more approvals, each with a name, list of approvers, and optional evidence

You can assign multiple approvers to any stage or bundle, as long as they are already members of the project.

**Example: Define stages in YAML**\
This example defines two stages by name:

```yaml theme={null}
stages:
  - name: stage1
  - name: stage2
```

### Approvals

Approvals are defined within a stage. Each approval includes:

* A **name**

* A list of **approvers** (users or organizations)

* Optional **evidence**, which must be satisfied before approval can be granted

You can assign multiple approvers to any stage or bundle, as long as they are already members of the project.

Evidence may be used to support the approval process, but it is not required in all cases.

**Example: Define an approval with evidence**\
In this example, the stage includes an approval named `Stage 4: validation sign off`, which includes a checklist prompt for the approver:

```yaml theme={null}
- name: 'Stage 4: validation sign off'
  approvers:
    - model-gov-org
  evidence:
    id: Local.validation-approval-body
    name: Sign-off
    description: The checklist for approvals
    definition:
      - artifactType: input
        details:
          label: "Have you read the model validation reports?"
          type: radio
          options:
            - Yes
            - No
```

### Revalidation

Revalidation schedules are defined within an approval to enforce recurring renewal. See [Periodic Revalidation](/cloud/platform-capabilities/features/governance/approvals#periodic-revalidation) for the full configuration model.

**Example: Define a revalidation schedule**\
This example opens a 30-day renewal window, recurring every year starting from January 1, 2026:

```yaml theme={null}
revalidation:
  openWindowDaysBefore: 30
  recurring:
    every: 1
    unit: year    # or "month"
    startDate: 2026-01-01T00:00:00Z
```

### Sequential workflows

Sequential workflows define multi-stage approval processes where each stage must be completed before the next becomes available. Stages unlock in order, mirroring real-world review flows and helping maintain control throughout the governance process.

Progression is gated by required fields. All mandatory information must be provided before a bundle can move forward, supporting structured, auditable workflows and built-in compliance.

Sequential workflows are currently defined in YAML using the `enforceSequentialOrder` field.

**Example: Define a sequential approval workflow**\
The following YAML example enforces sequential approvals across designated approvers within the `model-gov-org` group:

```yaml theme={null}
enforceSequentialOrder: true
approvers:
  - name: model-gov-org
    showByDefault: true
    editable: false
```

## Evidence and evidence sets

Evidence defines the inputs, approvals, or checks required during a stage. You can define evidence directly in a stage, or organize it into an `evidenceSet` for reuse and clarity.

When you define local evidence, it must be declared in full the first time it’s used. Later references can reuse it by `id`.

**Example: Define evidence sets**\
This example defines a local evidence set named `sample local evidence`:

```yaml theme={null}
evidenceSet:
  - id: Local.sample
    name: sample local evidence
    description: Describe the sample local evidence
    definition: Define the evidence
```

### Metrics checks

Metrics checks use model metadata to validate performance against thresholds. These checks run automatically and reduce manual review.

Each check supports aliases to match metric names, as well as threshold logic to compare against expected values.

**Example: Define metrics checks**\
This check validates model accuracy using multiple `aliases` and a minimum threshold of `0.8`.

```yaml theme={null}
metrics:
  - id: Local.model-quality
    name: Model Quality
    description: Validates model accuracy meets minimum standards
    definition:
      - artifactType: metadata
        details:
          type: modelmetric
          metrics:
            - name: Acc
              label: Accuracy
              aliases:
                - acc
                - Correct Classification Rate
                - Percentage Correct
              threshold:
                operator: '>='
                value: 0.8
```

### Scripted checks

Scripted checks run custom validation logic as part of a policy. Each one defines a command, parameters, and expected outputs.

The script referenced in `command` must exist in the project's files before the check can run. In Git-based projects, commit and push the script to the repository first: Domino can't run the command until the script is pushed to Git.

Scripted checks can also run with user input. Define each input as a text parameter under `parameters`. When the check runs, Domino replaces the parameter's `${name}` placeholder in `command` with the value entered at run time, or with the parameter's `default`. The example below passes `model_hub` and `model_name` to the evaluation script this way.

Scripted checks require an `environmentID` and `hardwareTierID`.

* Environment IDs can be found under **Govern** > **Environments**. Select an environment, and the ID is shown as the last part of the URL.

* Hardware tier IDs are listed in the **Admin Portal**, under **Manage resources** > **Hardware tiers**.

Contact your administrator if you don’t have access to view these settings.

**Example: Define scripted checks**\
This scripted check runs a command-line model evaluation with input parameters and produces text and image output:

```yaml theme={null}
- artifactType: policyScriptedCheck
  details:
    name: Ethic and Fairness Evaluation
    label: Ethic and Fairness Evaluation
    command: evaluate_model.py create --model-hub ${model_hub} --model-name ${model_name}
    parameters:
      - name: model_hub
        type: text
        default: openai
      - name: model_name
        type: text
        default: gpt-4
    outputTypes:
      - txt
      - png
    environmentId: [your environmentID]
    hardwareTierId: small-k8s
    volumeSizeGiB: 4
```

### Monitoring checks

Monitoring checks connect governance to Domino Model Monitoring (DMM) to respond to drift, data quality, and other detection alerts.

When an alert triggers, governance executes actions like notifications and creates findings with severity levels and owners, with all activity logged for traceability and audit compliance.

**Example: Define a monitoring check**\
This monitoring check listens for drift and data quality alerts, then sends notifications and creates findings with assigned owners:

```yaml theme={null}
- artifactType: policyMonitoringCheck
  details:
    name: Model Drift Detection
    label: Model Drift Detection
    alertTypes:
      - drift
      - data_quality
    actions:
      - type: notification
        recipients:
          - model-gov-org
      - type: createFinding
        severity: high
        owner: model-gov-org
```

## Input artifacts

Input artifacts define form elements used to collect user input during policy execution. These inputs appear in the governed bundle approval interface and can be used as evidence, classification references, or dynamic visibility triggers.

### Radio buttons

Radio buttons present users with a single-select list of labeled options. Each is defined by a display label and submitted value.

**Example: Define radio buttons**\
This example defines a radio group with three labeled choices:

```yaml theme={null}
- artifactType: input
  details:
    type: radio
    label: "How would you rate the model risk?"
    options:
      - label: "High"
        value: "High"
      - label: "Medium"
        value: "Medium"
      - label: "Low"
        value: "Low"
    tooltip: "Guidance text"
```

### Text inputs

Text inputs collect short, freeform written responses from users.

**Example: Define text inputs**\
This example defines a single-line input field with placeholder and help text (`helpText`):

```yaml theme={null}
- artifactType: input
  details:
    type: textinput
    label: "What are the expected business benefits?"
    placeholder: "Explain the benefit"
    helpText: "The text under the input box to help the user"
```

### Text areas

Capture longer, multi-line responses. You can customize the visible height of the input.

**Example: Define text areas**\
This example defines a 10-line `textarea` input:

```yaml theme={null}
- artifactType: input
  details:
    type: textarea
    label: "What are the expected business benefits?"
    height: 10
    placeholder: "Explain the benefit"
    helpText: "The text under the input box to help the user"
```

### Select dropdowns

Select dropdowns provide a list of options where only one can be selected.

**Example: Define select dropdowns**\
This example defines a dropdown to select a base model template:

```yaml theme={null}
- artifactType: input
  details:
    type: select
    label: "Please select the base model template."
    options:
      - label: "base model1"
        value: "baseModel1"
      - label: "base model2"
        value: "baseModel2"
```

### Multi-select dropdowns

Multi-select dropdowns allow selection of multiple options from a list.

**Example: Define multi-select dropdowns**\
This example defines a multi-select input for selecting data sets:

```yaml theme={null}
- artifactType: input
  details:
    type: multiSelect
    label: "Please select the data sets used in the model."
    options:
      - label: "data set1"
        value: "dataset1"
      - label: "data set2"
        value: "dataset2"
      - label: "data set3"
        value: "dataset3"
```

### Checkbox groups

Checkbox groups display multiple options with checkboxes. Users can select any combination.

**Example: Define checkbox groups**\
This example defines checkboxes for department selection:

```yaml theme={null}
- artifactType: input
  details:
    type: checkbox
    label: "Please select the departments that will use the model."
    options:
      - label: "Sales"
        value: "DEPT001"
      - label: "Customer Success"
        value: "DEPT002"
```

### Date inputs

Date inputs allow users to select or enter a date. You can specify a start date and format.

**Example: Define date inputs**\
This example defines a date input in `ISO8601` format:

```yaml theme={null}
- artifactType: input
  details:
    type: date
    label: "What is the scheduled release date?"
    startDate: 20240612
    format: ISO8601
```

### Numeric inputs

Numeric inputs collect numbers within a defined range.

**Example: Define numeric inputs**\
This example defines an input for `F-score` with `min` and `max` constraints:

```yaml theme={null}
- artifactType: input
  details:
    type: numeric
    label: "What is the allowed F-score for the model to be deployed?"
    min: 0
    max: 1
```

When you assign an `aliasForClassification` to a numeric input, its answer is available to [visibility rules](#visibility-rules) as a number through the `numericInputs` map.

### File uploads

File upload artifacts let reviewers attach a file as evidence in the approval interface. The file is stored in the evidence notebook and automatically uploaded to the project directory.

Use file uploads when reviewers must submit supporting documents, such as validation reports or compliance forms. Unlike text inputs, file uploads capture and store the document itself.

**Example: Define file uploads**\
This example defines a file input field labeled `Model Validation Report`. The uploaded file becomes part of the governed evidence.

```yaml theme={null}
- artifactType: metadata
  details:
    type: file
    label: Model Validation Report
    description: Upload the model validation report. Make sure the file size does not exceed 500MB.
```

## Guidance artifacts

Guidance artifacts provide informational content to users, such as Markdown-formatted instructions or visual banners. These elements do not collect input but help orient users during policy execution.

### Text blocks

Text blocks display long-form guidance using Markdown syntax.

**Example: Define text blocks**\
This example shows how to include Markdown-formatted context as part of the policy:

```yaml theme={null}
- artifactType: guidance
  details:
    type: textblock
    text: >-
      [Map 1.4](https://ournistpolicyreferenceurl.com) The business value or
      context of business use has been clearly defined or - in the case of
      assessing existing AI systems - re-evaluated.
```

### Text banners

Text banners provide high-visibility messages to users.

**Example: Define text banners**\
This example shows how to display a banner with a policy reference:

```yaml theme={null}
- artifactType: guidance
  details:
    type: banner
    text: >-
      [Map 1.4](https://ournistpolicyreferenceurl.com) The business value or
      context of business use has been clearly defined or - in the case of
      assessing existing AI systems - re-evaluated.
```

## Classification

Classification is a top-level policy feature used to assign a risk tier (such as `low`, `medium`, or `high`) to a governed bundle. You can base classifications on user inputs, define rules to evaluate them, and display tooltips to guide decisions.

### Classification inputs

You can link input artifacts to a classification by assigning an alias. These inputs appear in stages like any other artifact but can be referenced in classification logic.

**Example: Define classification artifacts**\
This example defines a radio button input for model risk and links it to a classification alias:

```yaml theme={null}
classification:
  rule:
  artifacts:
    - model-risk

stages:
  - name: classificationExample
    artifacts:
      - id: Local.model-risk
        name: Model Risk
        description: Describe the risk of the model
        definition:
          - artifactType: input
            aliasForClassification: model-risk
            details:
              label: "How would you rate the model risk?"
              type: radio
              options:
                - High
                - Low
```

### Classification rules

Use rules to evaluate classification inputs and assign a final label. You can define simple conditions or write custom logic using scripts.

**Example: Define a classification rule**\
This example returns `"High"` if the sum of all input values is greater than or equal to 1:

```python theme={null}
func() string {
    var sum float64
    for _, value := range inputs {
        sum += value
    }
    if sum >= 1 {
        return "High"
    }
    return "Low"
}()
```

**Example: Complex classification rule**\
This example classification rule uses evidence values to define whether the classification for this bundle should be `"High"` or `"Low"`.

```yaml theme={null}
classification:
  rule: |
    var ans string
    ans = "Low"
    if classificationInputs["model-risk"].(string) == "High" || classificationInputs["change-type"].(string) == "Major" {
      ans = "High"
    }
    return ans
```

**Note**: You can reference this classification output in a custom visibility rule to conditionally show or hide a section.

### Visibility rules

Visibility rules control whether certain artifacts, such as evidence sets or individual questions, appear in the interface. You can base visibility on classification outcomes, user input, or any other evaluated condition.

Attach a `visibilityRule` in either of two places:

* Inside an artifact's `details` block, to show or hide a single question.

* On an `evidenceSet` item, at the same level as `definition`, to show or hide the entire evidence card.

A `visibilityRule` is a Go boolean expression. The artifact appears when the expression evaluates to `true`. Rules can reference four variables, keyed by each input's `aliasForClassification`:

| Variable | Type | Contains |
| :- | :- | :- |
| `classificationValue` | `string` | The bundle's computed classification. |
| `inputs` | `map[string]string` | Answers from `radio`, `select`, `textinput`, `textarea`, and `date` questions. |
| `arrayInputs` | `map[string][]string` | Answers from `checkbox` and `multiSelect` questions. |
| `numericInputs` | `map[string]float64` | Answers from `numeric` questions. |

Rules can also call these helper functions:

* `slices.Contains(list, value)` checks whether an array-typed answer includes a value.

* `atoi(value)` and `atof(value)` convert a string to an `int` or `float64`. Every question type except `numeric` answers with strings, so use these helpers to compare a numeric-looking string answer, such as a radio option whose value is `"3"`, as a number.

An unanswered question resolves to its zero value: `inputs["x"]` is `""`, `arrayInputs["x"]` is an empty list, and `numericInputs["x"]` is `0`.

**Example: Show a question based on a numeric threshold**\
This example shows a follow-up question only when the answer to a `numeric` question aliased as `drift-score` is greater than 0.75:

```yaml theme={null}
- artifactType: input
  details:
    type: textarea
    label: "Describe the drift remediation plan."
    visibilityRule: numericInputs["drift-score"] > 0.75
```

**Example: Show an evidence set based on classification or a numeric answer**\
This example hides an entire evidence card unless the bundle is classified `High` or the model has been in production for more than 24 months:

```yaml theme={null}
evidenceSet:
  - id: Local.escalation-review
    name: Escalation review
    visibilityRule: classificationValue == "High" || numericInputs["age"] > 24
    definition:
      - artifactType: input
        details:
          type: textarea
          label: "Document the escalation review."
```

**Example: Combine array membership and numeric conversion**\
This example shows a question only when the model is deployed in the USA and a radio question whose option values are numeric strings, aliased as `authority-level`, is at least 3:

```yaml theme={null}
visibilityRule: slices.Contains(arrayInputs["regions"], "USA") && atoi(inputs["authority-level"]) >= 3
```

For multi-step logic, write the rule as a block of Go statements that ends with an explicit `return`. If the word `return` appears anywhere in a rule, including inside a quoted string, Governance treats the rule as a full block and the rule must supply its own `return` statement.

**Example: Evaluate each element of an array-typed answer**\
This example shows a question when any selected option, converted to a number, is at least 500:

```yaml theme={null}
visibilityRule: |
  for _, v := range arrayInputs["exposure-tiers"] {
      if atoi(v) >= 500 {
          return true
      }
  }
  return false
```

## Gating

Gates define policy controls for high-impact actions, like deploying a published App, Agent, or model Endpoint. Each gate specifies:

* What action the gate applies to. The only supported action is `Deploy`, which covers deploying a published App, Agent, or model Endpoint. Preview deployments of App drafts are not gated.

* Which parameter values trigger it, like `hardwareTierId` or `dataPlaneId`

* What approvals are required for the action to proceed

<Note>
  The `CreateApp` and `CreateEndpoint` actions are deprecated. Use `Deploy` in new policies. Existing policies that use `CreateApp` or `CreateEndpoint` are still respected.
</Note>

Gates are optional but provide a way to prevent risky operations without manual oversight. Before defining a gate, you need to know the valid parameter values for your Domino instance:

* **Hardware Tier IDs** control compute resources like CPU and GPU. You can find available hardware tier IDs in Hardware tiers or by contacting your administrator.

* **Data plane IDs** control where workloads run. You can find available data plane IDs in Data planes or by contacting your administrator.

**Example: Define a gate for resource-heavy deployments**\
In this example, the gate blocks deployment of Apps, Agents, and model Endpoints that use the `large-k8s` or `gpu-small-k8s` hardware tiers unless the `Validation sign off` stage has been completed:

```yaml theme={null}
gates:
  - name: Prod
    rules:
      - action: Deploy
        parameters:
          hardwareTierId:
            - large-k8s
            - gpu-small-k8s
    approvals:
      - Validation sign off
```

Use the `any` keyword to apply a gate to all values of a parameter. This is useful when you want to gate an action regardless of which hardware tier or data plane is selected.

**Example: Gate all hardware tiers**\
This gate blocks deployment on any hardware tier until the Validation sign off stage is complete.

```yaml theme={null}
gates:
  - name: Prod
    rules:
      - action: Deploy
        parameters:
          hardwareTierId: "any"
    approvals:
      - Validation sign off
```

## Next steps

* [Define policies](/cloud/platform-capabilities/features/governance/define-governance-policies): create and configure policies in the Policy Builder

* [Gates](/cloud/platform-capabilities/features/governance/gating-domino-governance): control governed actions based on approval status

* [Roles and security](/cloud/platform-capabilities/features/governance/roles-and-security): understand permissions and role assignment in Governance


# Automated checks
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/evidence/automated-evidence-collection

Validate governed bundles with metrics thresholds and scripted checks that run custom code, capture outputs, and attach traceable evidence.

Automated checks run validation logic as bundles progress through approval stages. Policies can include two types of automated checks: metrics checks and scripted checks.

Results are recorded in the governance notebook and attached as evidence for approvers to review. All check results are logged and traceable, supporting compliance requirements and reproducibility.

## Prerequisites

You need the **GovernanceAdmin** role to define checks in policies. [Roles and security](/cloud/platform-capabilities/features/governance/roles-and-security) has details on role assignment.

For scripted checks, you also need:

* An environment with required dependencies

* Access to a hardware tier

* Sufficient volume storage for script execution

## Metrics checks

Metrics checks validate model performance using metadata captured during model registration or experimentation. Each check defines metric names, optional aliases, and threshold conditions.

Domino evaluates metrics checks automatically when a bundle enters a stage that requires them. If thresholds aren’t met, the system creates findings for approvers to review. Approvers see the results and can decide whether to proceed.

### How metrics checks work

When a bundle reaches a stage with metrics checks, Domino queries the model metadata for matching metric names or aliases. If a threshold is defined, Domino compares the actual value against the expected value using the specified operator.

* **Metric meets threshold**: Check passes

* **Metric below threshold**: Check fails, approver notified

* **Metric not found**: Check fails, missing metric logged

* **No threshold defined**: Metric value displayed for review

Governance administrators define these inputs when configuring policies. [Define metrics checks](/cloud/platform-capabilities/features/governance/define-governance-policies/policy-components#metrics-checks) in Governance policy components has YAML configuration details for metrics checks.

## Scripted checks

Scripted checks run custom validation logic as part of a policy. Use them to standardize complex evaluations like fairness assessments, bias detection, or compliance reporting. Scripts run in a specified environment and generate evidence that’s attached to the governance notebook.

Unlike metrics checks that evaluate existing metadata, scripted checks execute custom code in a controlled environment. You define the script command, input parameters, and expected outputs. When the check runs, Domino launches a job, executes your script, and captures the results as evidence in the governance notebook.

This approach lets you implement organization-specific validation logic while maintaining consistent execution and audit trails across all governed models.

### How scripted checks work

When a bundle enters a stage with scripted checks, Domino launches a job in the specified environment and hardware tier. The script runs with the provided parameters, generates output, and attaches results to the governance notebook. A link to the execution run is included for reproducibility.

* **Execution**: Job runs in specified environment with command-line parameters

* **Capture**: Output files (TXT, PNG, JSON, CSV) are uploaded to governance notebook

* **Evidence**: Results appear inline for approvers, with a link to reproduce the run

[Define scripted checks](/cloud/platform-capabilities/features/governance/define-governance-policies/policy-components#scripted-checks) in Policy components has YAML configuration details for scripted checks.

## Known issues and limitations

* Metrics checks support only numeric thresholds. Text-based metrics must be validated manually.

* Scripted checks run synchronously. Long-running scripts may delay bundle progression.

* Custom output types in scripted checks must be explicitly defined in the policy.

## Next steps

* [Manual evidence](/cloud/platform-capabilities/features/governance/evidence/manual-evidence): add text responses, files, and artifacts to bundles

* [Monitoring checks](/cloud/platform-capabilities/features/governance/evidence/monitoring-checks-in-governance): connect to Domino Model Monitoring

* [Policy components](/cloud/platform-capabilities/features/governance/define-governance-policies/policy-components): review YAML configuration details for policies


# Configure evidence
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/evidence/index

Configure manual evidence, metrics and scripted checks, and Domino Model Monitoring checks to support governed bundle review and audit trails.

Evidence captures information needed to evaluate and approve governed bundles. It documents decisions, validates model performance, and creates audit trails throughout the governance lifecycle.

Evidence takes many forms:

**Manual evidence** includes text responses, file uploads and structured selections that practitioners provide during review. These inputs answer questions defined in policy stages.

**Automated checks** run validation logic without manual intervention. Metrics checks evaluate model performance against thresholds. Scripted checks execute custom code for complex evaluations like fairness assessments.

**Monitoring checks** connect governance to Domino Model Monitoring (DMM). When drift or data quality alerts trigger, governance creates findings with severity levels and assigns owners for follow-up.

## Next steps

* [Manual evidence](/cloud/platform-capabilities/features/governance/evidence/manual-evidence): add text responses, files, and artifacts to bundles

* [Automated checks](/cloud/platform-capabilities/features/governance/evidence/automated-evidence-collection): run metrics and scripted validations

* [Monitoring checks](/cloud/platform-capabilities/features/governance/evidence/monitoring-checks-in-governance): connect to Domino Model Monitoring


# Manual evidence
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/evidence/manual-evidence

Collect text, selections, dates, numbers, files, models, Dataset snapshots, and Job outputs as evidence in governed bundles.

Manual evidence includes the responses, files, and artifacts that practitioners provide during governance review. Policies define which inputs are required at each stage, and practitioners complete them as bundles progress through approval.

## Input types

Policies can include various input types to collect evidence from practitioners:

* **Text inputs** collect short, freeform responses

* **Text areas** capture longer, multi-line responses

* **Radio buttons** present single-select options

* **Checkboxes** allow multiple selections

* **Dropdowns** provide single-select lists

* **Multi-select dropdowns** allow multiple selections from a list

* **Date inputs** capture dates in a specified format

* **Numeric inputs** collect numbers within defined ranges

* **File uploads** attach supporting documents as evidence

Governance administrators define these inputs when configuring policies. [Governance policy components](/cloud/platform-capabilities/features/governance/define-governance-policies/policy-components) has YAML configuration details for defining policies.

## Add evidence to a bundle

You can add files, artifacts, or job outputs to an existing governed bundle.

**From a File System project**:

1. Click the code link in the right navigation.

2. Select **Add to an existing Bundle**.

**From a Git-based project**:

1. Click the artifact link in the right navigation and select the artifact.

2. Select **Add to an existing Bundle** and choose a bundle from the menu.

**From a job**:

1. Navigate to the job and go to the **Output** tab.

2. Select the output, then select **Add to an existing Bundle** and choose a bundle from the menu.

To confirm the addition, click **Attachments** in the bundle.

## Add a model to a bundle

To add a model to an existing governed bundle:

1. Open the project that contains the model and select it from the list.

2. On the Model Details tab, select **Add to Bundles** > **Add to an existing Bundle**.

3. Choose a bundle from the menu and click **Add**.

## Add a dataset to a bundle

To add a dataset to a governed bundle, you need to [create a snapshot](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots) of the dataset first.

1. Navigate to **Data** > **Datasets**, and select the dataset.

2. Choose the snapshot of interest from the **Files** drop-down list.

3. Click the breadcrumbs next to the dataset, and select **Add to Bundle**.

4. Choose an existing bundle from the drop-down list, or create a new bundle instead, and click **Add**.

## Evidence notebooks

Evidence Notebooks **autosave** in real time as you make edits. Visibility rules and compliance checks trigger immediately when you update content.

## Next steps

* [Automated checks](/cloud/platform-capabilities/features/governance/evidence/automated-evidence-collection): run metrics and scripted validations

* [Monitoring checks](/cloud/platform-capabilities/features/governance/evidence/monitoring-checks-in-governance): connect to Domino Model Monitoring

* [Policy components](/cloud/platform-capabilities/features/governance/define-governance-policies/policy-components): review YAML configuration details for policies


# Monitoring checks
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/evidence/monitoring-checks-in-governance

Connect governance policies to Domino Model Monitoring alerts, then create assigned findings for drift and data quality issues without reprocessing data.

Monitoring checks connect governance to [Domino Model Monitoring](/cloud/admin/platform-configuration/manage-model-deployments) (DMM) and respond to alerts without reprocessing raw data. Governance listens for drift and data quality alerts. When an alert triggers, governance creates findings with severity levels and assigns owners for follow-up.

All alerts, actions, and results are logged in the governance notebook and dashboard for audit compliance.

## Prerequisites

You need the **GovernanceAdmin** role to define checks in policies. [Roles and security](/cloud/platform-capabilities/features/governance/roles-and-security) has details on role assignment.

For monitoring checks, you also need:

* Domino Model Monitoring (DMM) configured and running

* Model endpoints registered with DMM

## Monitoring checks

Monitoring checks listen for drift and data quality alerts from DMM without reprocessing raw data. Governance registers model endpoints with DMM and subscribes to alert notifications. When DMM detects an issue, governance evaluates the alert against policy rules and takes action:

* **Drift or data quality alert received**: Finding created with high severity

* **Alert threshold exceeded**: Owner assigned for investigation

All monitoring activity is recorded in the governance notebook with timestamps, alert details, and action history.

[Define monitoring checks](/cloud/platform-capabilities/features/governance/define-governance-policies/policy-components#monitoring-checks) in Policy components has YAML configuration details for monitoring checks.

## Known issues and limitations

* Monitoring checks require [DMM to be configured](/cloud/admin/platform-configuration/manage-model-deployments) before policies can reference endpoints.

* Endpoints must be attached to the bundle before the checks are set up: **Endpoint overview** > **Versions** > **Endpoint version** > **Add to bundle**

## Next steps

* [Manual evidence](/cloud/platform-capabilities/features/governance/evidence/manual-evidence): add text responses, files, and artifacts to bundles

* [Automated checks](/cloud/platform-capabilities/features/governance/evidence/automated-evidence-collection): run metrics and scripted validations

* [Add findings to a bundle](/cloud/platform-capabilities/features/governance/work-with-bundles/add-findings-to-bundle): document review decisions and track issues


# Gates
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/gating-domino-governance

Gate deployments of published Apps, Agents, and model Endpoints by Hardware Tier or Data Plane until required policy approvals are complete.

After bundles are approved through the governance lifecycle, gates enforce those approval requirements by controlling specific user actions in real time.

Gates help teams limit risk, automate compliance, and respond quickly to policy changes. Gating decisions are applied automatically and recorded for audit.

You need the **GovernanceAdmin** role to define gates in policies. [Roles and security](/cloud/platform-capabilities/features/governance/roles-and-security) has details on role assignment.

Domino evaluates gated actions when you deploy a published App, Agent, or model Endpoint, blocking users from completing these actions until the required approval is granted.

Common use cases include:

* Blocking GPU or high-cost hardware tiers until reviews are complete

* Controlling deployments to specific data planes

* Rechecking approval status on restart to maintain compliance

Gates apply only to governed actions in projects where a bundle with gating policies has been attached.

## Define gates in a policy

Governance Admins define gates as part of a policy using YAML. Each gate includes:

* One or more rules that define the governed action and its parameters

* A list of required approvals for the action to proceed

The only supported gate action is `Deploy`, which covers deploying a published App, Agent, or model Endpoint. Preview deployments of App drafts are not gated.

**Example: Define gates in YAML**\
In this example, the gate blocks deployment of Apps, Agents, and model Endpoints that use the `large-k8s` or `gpu-small-k8s` hardware tiers unless the `Validation sign off` stage has been completed:

```yaml theme={null}
gates:
  - name: Prod
    rules:
      - action: Deploy
        parameters:
          hardwareTierId:
            - large-k8s
            - gpu-small-k8s
    approvals:
      - Validation sign off
```

<Note>
  The `CreateApp` and `CreateEndpoint` actions are deprecated. Use `Deploy` in new policies. Existing policies that use `CreateApp` or `CreateEndpoint` are still respected.
</Note>

## Activate gating in a project

When a practitioner attaches a bundle with gating policies to a project, the system begins enforcing the gate rules. Only governed actions and parameters are affected. All other activity proceeds normally.

Gates remain active as long as the bundle is attached and the policy requires the relevant approvals.

## Manage gate approvals

Gates rely on the same approval stages used elsewhere in governance. Admins or approvers can grant or revoke approvals at any time.

When approval status changes, the system updates gate behavior automatically. This affects both new actions and restarts of governed resources.

## How gating enforcement works

When a user attempts a governed action, the system evaluates gate rules in real time. If the action and its parameters match a gated rule, the system checks for approval before allowing it to proceed.

* **During creation**: invalid configuration options are blocked in the UI

* **On restart**: users receive an error if a previously approved configuration no longer meets policy requirements

## How gate decisions are enforced and tracked

Gating decisions are enforced by two internal services:

* **Governance Service** manages policies, approvals, and gate definitions

* **Rules Service** applies gate logic and determines whether to allow or block actions in real time

Gate rules are stored in the shared governance database. When a policy is activated, the Rules Service sets all governed actions to `"deny"` by default. Actions are allowed only after the required approvals are granted.

All rule evaluations and gating outcomes are:

* Recorded in the **Audit Log**

* Tracked in **MixPanel** to support compliance reporting and usage analysis

## Next steps

* [Work with bundles](/cloud/platform-capabilities/features/governance/work-with-bundles): package models and artifacts for review

* [Automated checks](/cloud/platform-capabilities/features/governance/evidence/automated-evidence-collection): run validation checks during bundle progression

* [Monitoring checks](/cloud/platform-capabilities/features/governance/evidence/monitoring-checks-in-governance): respond to drift and data quality alerts from deployed models

* [Add findings to a bundle](/cloud/platform-capabilities/features/governance/work-with-bundles/add-findings-to-bundle): document review decisions and track issues

* [Send bundles for review](/cloud/platform-capabilities/features/governance/work-with-bundles/review-bundles): submit bundles for formal approval


# Glossary of governance terms
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/governance-glossary

Look up governance terms for policies, bundles, evidence, approvals, findings, gates, checks, stages, classification, and sequential workflows.

This glossary defines key concepts used throughout Governance. Use it as a reference when reading [policies](/cloud/platform-capabilities/features/governance/define-governance-policies), working with [governed bundles](/cloud/platform-capabilities/features/governance/work-with-bundles), or [reviewing bundles for approval](/cloud/platform-capabilities/features/governance/work-with-bundles/review-bundles).

The terms here establish a common language for understanding how Governance features fit together. The [Governance overview](/cloud/platform-capabilities/features/governance) provides more details on how these concepts are applied.

| Term | Definition |
| - | - |
| **Approvals** | Actions added to a policy to request approval from an organization or an individual. Anyone with project access can request approval, but only a `GovernanceApprover` can approve it. |
| **Attachments** | Any files from a project that needs to be governed by a policy in a specific context. |
| **Classification** | The top-level variable for a policy that is meant to carry the overall classification of the governed bundle. It could be used in a tiered approach, such as low, medium, or high risk. |
| **Classification rules** | Classification supports complex operations using the [Go programming language](https://go.dev/). It uses the values results from one or multiple evidence answers and expects an output of a string or a float. |
| **Evidence** | Information that is relevant to a policy. In most cases, evidence is a question with an answer and can include input, metric checks, and scripted checks. |
| **Evidence sets** | Groups of evidence that can be reused within a policy or across policies. Evidence sets are meant to promote consistency of governance. |
| **Findings** | These serve as a necessary step for capturing and addressing issues that prevent full compliance, ensuring a path to production readiness. |
| **Gates** | Prevent users from performing specific actions such as publishing apps or creating model endpoints, until required approvals are complete. Gates can be added to policies through YAML. |
| **Governed bundle** | A governed bundle can be a model, an application, a report, or any other asset developed within the context of a project. It stores all evidence related to the policy it governs and keeps the lineage to the relevant attachments. Bundles can now include multiple policies, which helps reduce duplication and group related governance rules in one place. |
| **Lifecycle** | A set of stages describing each step of building and maintaining a governed bundle. |
| **Metrics checks** | Automatically attach metrics to a governed bundle, eliminating the need for manual entry. |
| **Policy** | Definition of all stages, evidence, and approvals that need to be met for compliance. Bundles can now define compliance against multiple policies, making it easier to consolidate related requirements in a single place. |
| **Scripted checks** | These checks use centralized scripts to evaluate information on a Governed Bundle. Scripted checks can also be used to measure bias consistently in a dataset. |
| **Sequential workflows** | Require stages in a bundle’s lifecycle to be completed in order, preventing skipped steps and strengthening quality control of governance. |
| **Stages** | Key milestones or phases of a policy. For example, in a model risk management policy, the stages could be: develop business cases and define requirements; model development and testing; model validation and deployment; and model monitoring. |

## Next steps

* [Work with bundles](/cloud/platform-capabilities/features/governance/work-with-bundles): package models and artifacts for review

* [Evidence](/cloud/platform-capabilities/features/governance/evidence): learn about manual evidence, automated checks, and monitoring checks

* [Define policies](/cloud/platform-capabilities/features/governance/define-governance-policies): create and configure policies (for Governance administrators)


# Roles and security
Source: https://docs.domino.ai/cloud/platform-capabilities/features/governance/roles-and-security

Assign GovernanceAdmin access and compare how practitioners, approvers, and administrators create policies, submit evidence, and approve bundles.

Governance uses role-based access to control who can create policies, submit evidence, and approve bundles. Governance introduces one new Domino Global Role: **GovernanceAdmin**. For the remaining roles, Governance relies on existing project roles.

## GovernanceAdmin

**Governance Administrators** create, edit, and publish policies. They also have access to compliance views, audit trail, and Governance APIs. SysAdmins automatically have GovernanceAdmin permissions. All other users must be explicitly assigned the role.

## Project-based roles

The following roles are based on existing Domino project permissions:

* **Practitioners** are existing Domino users who gain access to governed bundles through their project permissions. They create bundles, add artifacts and evidence, and submit bundles for review. Practitioners can’t create policies or approve stages.

* **Approvers** review evidence and approve bundles at designated stages. When a practitioner requests a review, the approver receives a task assignment. Approvers can also create findings to document issues and track resolution. Any user listed in a policy or organization as an approver gains consumer-level access to projects with governed bundles.

## Assign the GovernanceAdmin role

To grant the **GovernanceAdmin** role:

1. In the Admin portal, go to **Manage Resources** > **Users**.

2. Select the user to update and click **Edit**.

3. Select **GovernanceAdmin (Admin role for Governance)** from the list of roles, then click **Save**.

## Role permissions summary

The following table summarizes what each role can do in Governance:

| Permission | Practitioner | Approver | GovernanceAdmin |
| - | - | - | - |
| Create governed bundles | Yes | No | Yes |
| Add evidence and artifacts | Yes | Yes | Yes |
| Submit bundles for review | Yes | No | Yes |
| Review and approve stages | No | Yes | Yes |
| Create findings | Yes | Yes | Yes |
| Create and publish policies | No | No | Yes |
| Access Governance APIs | Yes | No | Yes |

## Next steps

* [Work with bundles](/cloud/platform-capabilities/features/governance/work-with-bundles): package models and artifacts for review

* [Review and approve bundles](/cloud/platform-capabilities/features/governance/work-with-bundles/review-bundles): submit bundles for formal approval

* [Define policies](/cloud/platform-capabilities/features/governance/define-governance-policies): create and configure policies (for Governance administrators)

* [Audit Trail](/cloud/platform-capabilities/features/governance/audit-trail): review a log of user and system actions across the platform


# Integrations
Source: https://docs.domino.ai/cloud/platform-capabilities/features/integrations

Connect Apache Airflow or Kubeflow pipelines to Domino Jobs with python-domino, API credentials, and network access to automate workflows.

Data science projects often require multiple steps to go from raw data to useful data products. These steps tend to be sequential, and involve things like:

* Sourcing data

* Cleaning data

* Processing data

* Training models

After you understand the steps necessary to deliver results from your work, it's useful to automate them as a repeatable pipeline. Domino can [schedule Jobs](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job), but for more complex pipelines you can pair Domino with an external scheduling system like Apache Airflow or Kubeflow. Both integrations use the [python-domino](https://github.com/dominodatalab/python-domino) package.

To find out how to integrate an external scheduling system with Domino, select the relevant tab below:

<Tabs>
  <Tab title="Apache Airflow">
    You can pair Domino with [Apache Airflow](https://airflow.apache.org/) to automate complex pipelines as a repeatable pipeline.

    ## Get started with Airflow

    Airflow is an open source platform to author, schedule, and monitor pipelines of programmatic tasks. You can define pipelines with code and configure the Airflow scheduler to execute the underlying tasks. You can use the Airflow application to visualize, monitor, and troubleshoot pipelines.

    If you are new to Airflow, read the [Airflow QuickStart](https://airflow.apache.org/start.html) to set up your own Airflow server.

    There are many options for configuring your Airflow server, and for pipelines that can run parallel tasks, you must use Airflow's [LocalExecutor mode](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/local.html). In this mode, you can run tasks in parallel and execute multiple dependencies at the same time. Airflow uses a database to keep records of all the tasks it executes and schedules, so you must install and configure a SQL database for LocalExecutor mode.

    Read [A Guide On How To Build An Airflow Server/Cluster](http://stlong0521.github.io/20161023%20-%20Airflow.html) to learn more about setting up LocalExecutor mode.

    For more information about scheduling and triggers, notifications, and pipeline monitoring, read the [Airflow documentation](https://airflow.apache.org/index.html).

    ## Install python-domino on your Airflow workers

    To create Airflow tasks that work with Domino, you must install [python-domino](https://github.com/dominodatalab/python-domino) on your Airflow workers. Use this library to add tasks in your pipeline code that interact with the [Domino Platform API](/cloud/reference/api/domino-open-api) to start Jobs.

    Connect to your Airflow workers, and follow these steps to install and configure `python-domino`:

    1. Install from pip

       `pip install dominodatalab`

    2. Set up an [Airflow variable](https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#variables) to point to the Domino host. This is the URL where you load the Domino application in your browser.

       > **Key**: `DOMINO_API_HOST`\
       > **Value**: `<your-domino-url>`

    3. Set up an [Airflow variable](https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#variables) to store the [user API key](/cloud/reference/api/domino-api-authentication) you want to use with Airflow. This is the user Airflow with authentication to Domino to start Jobs.

       > **Key**: `DOMINO_API_KEY`\
       > **Value**: `<your-api-key>`

    ## How Airflow tasks map to Domino Jobs

    Airflow pipelines are defined with Python code. This fits in well with Domino's code-first philosophy. You can use [python-domino](https://github.com/dominodatalab/python-domino) in your pipeline definitions to create tasks that start Jobs in Domino.

    Architecturally, Airflow has its own server and worker nodes, and Airflow operates as an independent service that sits outside of your Domino deployment. Airflow needs network connectivity to Domino so its workers can access the [Domino Platform API](/cloud/reference/api/domino-open-api) to start Jobs in your Domino Project. All the code that performs the actual work in each step of the pipeline (code that fetches data, cleans data, and trains data science models) is maintained and versioned in your Domino Project. This way you have Domino's Reproducibility Engine working together with Airflow's scheduler.

    <img alt="Airflow pipeline" />
  </Tab>

  <Tab title="Kubeflow">
    You can pair Domino with [Kubeflow](https://www.kubeflow.org/docs/started/introduction/) to automate complex pipelines as a repeatable pipeline.

    ## Get started with Kubeflow

    Kubeflow is an open source platform designed for machine learning workflows on Kubernetes. It facilitates the orchestration of ML workflows, allowing you to define and manage machine learning pipelines.

    If you're new to Kubeflow, explore the Kubeflow documentation for [setting up and configuring your Kubeflow environment](https://www.kubeflow.org/docs/started/installing-kubeflow/).

    Configuring your Kubeflow environment might involve setting up a Kubernetes cluster, establishing the necessary networking, and defining resources for your pipeline execution.

    For more detailed information on setting up Kubeflow and managing your machine learning workflows, refer to the [Kubeflow documentation](https://www.kubeflow.org/docs/started/introduction/).

    ## Create pipelines

    To create Kubeflow pipelines that interact with Domino, you'll need to install the python-domino package on your Kubeflow cluster.

    To install the required package, follow these steps:

    1. Access your Kubeflow cluster.

    2. Install the domino-kubeflow package using the following command:

       ```bash theme={null}
       pip install dominodatalab
       ```

       Configure the connection between Kubeflow and Domino by setting up environment variables that point to the Domino host and store your [Domino API key](/cloud/reference/api/domino-api-authentication#authenticate-with-an-api-key-legacy).

    3. Set an environment variable to point to the Domino host:

       ```bash theme={null}
       export DOMINO_API_HOST=<your-domino-url>
       ```

    4. Store the user API key you want to use to authenticate into Domino:

       ```bash theme={null}
       export DOMINO_API_KEY=<your-api-key>
       ```

    ## Map Kubeflow pipelines to Domino Jobs

    Domino's code-first approach aligns well with defining tasks within Kubeflow pipelines to initiate Jobs in Domino.

    Architecturally, Kubeflow pipelines are defined, managed, and operated within a Kubernetes cluster, separate from your Domino Environment. Kubeflow requires network connectivity to access the Domino API for executing Jobs in your Domino Projects. The code for each step within the pipeline, such as data retrieval, cleaning, and model training, is stored and versioned in your Domino Project. This integration ensures collaboration between Kubeflow's orchestration capabilities and Domino's reproducibility features.
  </Tab>
</Tabs>

## Next steps

Learn how to [schedule Jobs](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job) and [view Job results](/cloud/platform-capabilities/core-concepts/jobs/monitor-jobs).


# Launchers
Source: https://docs.domino.ai/cloud/platform-capabilities/features/launchers/index

Build self-service Launcher forms that pass text, choices, dates, and files to Domino Jobs, then deliver reproducible results by email.

Domino launchers can accelerate collaboration between data scientists and non-technical stakeholders to gather information, propose new data, perform tasks, or propose different parameters. Launchers are self-serve web forms that a data scientist quickly builds to expose pre-canned tasks to less technical users.

Data scientists can automate analysis runs for others and empower business users to ask more questions and experiment independently. Launchers reduce the barrier of "asking a data scientist" and increase fruitful conversations between stakeholders.

## What is a Launcher?

A launcher is a web form. When data scientists build a launcher, they expose web UI controls (such as a textbox or a drop-down menu) to a consumer who runs the launcher.

When a consumer completes the web form and runs the launcher, Domino passes the inputs into a script created by the data scientist as command-line parameters that initiates a job in Domino.

The results from the launcher depend on the script. Data scientists have the flexibility to create any result they need: PDFs, images, rendered notebooks, HTML, and so on. See [tips and tricks](#advanced-use-cases-tips-and-tricks) for ideas to take advantage of this flexibility. Because launchers execute jobs under the hood, each run of a launcher is tracked, fully reproducible, and auditable.

The launcher user reviews the results through an email notification and the Jobs dashboard.

## Build a Launcher as a data scientist

In your project, go to **Deployments > Launchers > New Launcher** and fill in the fields in the launcher wizard to create a new launcher.

The essential component of a launcher is the command. The command runs as a job, with placeholder values for the command-line arguments.

Each argument you want to expose to the consumer must be written as `${arg}`. You can mix hard-coded parameters with consumer-input parameters:

```shell theme={null}
my_script.py -x=1 ${file} ${start_date}
```

Launchers support these input controls:

* **Text** - Input arbitrary text.

* **Select** - Select one item from a list.

* **Multi-select** - Select multiple items.

* **Date** - Select a date.

* **Checkbox** - Specify a binary yes/no value.

* **Upload files** - Upload one or more files that are passed to your script.

<Note>
  Domino captures results from the launcher run like it does from a job: any file your script creates or modifies. Commonly, launchers generate images or even HTML, which is included in email notifications sent to recipients and rendered in the results view for the associated job.
</Note>

### How Domino passes arguments to your script

When parameter values are executed in a job started by the launcher, the values are encased in single quotes. This quotation preserves spaces and other special shell characters in the values. However, such values cannot be used to expand to environment variables.

File parameters are passed into your file as the path to the file. Note that uploaded files are persisted as results in the project. Domino projects are not designed to handle files greater than several GB in size or more than thousands of files.

Multi-select parameters are passed in as the comma-separated list of all selected choices.

### Parse arguments in your script

<Tabs>
  <Tab title="R">
    For R, use the [commandArgs function](https://stat.ethz.ch/R-manual/R-patched/library/base/html/commandArgs.html).

    ```r theme={null}
    args <- commandArgs(trailingOnly=TRUE)
    p1 <- args[1]
    p2 <- args[2]

    # a file upload parameter
    print(readLines(args[3], n = 1))

    # a multi-select parameter
    for (each in strsplit(args[4],",")) {
      cat(each, sep="
    ")
    }
    ```
  </Tab>

  <Tab title="Python">
    For Python, use [sys.argv](https://www.tutorialspoint.com/python/python_command_line_arguments.htm).

    ```python theme={null}
    import sys
    p1 = sys.argv[1]
    p2 = sys.argv[2]
    # a file upload parameter
    with open(sys.argv[3], 'r') as f:
        print f.readline()

    # a multi-select parameter
    for part in sys.argv[4].split(","):
      print part
    ```
  </Tab>
</Tabs>

<Note>
  File parameters are passed into your file as the path to the file. Multi-select parameters are passed in as the comma-separated list of all selected choices.
</Note>

See the detailed examples below.

### Examples: Handle arguments in R and Python

<Tabs>
  <Tab title="R">
    Our simple R example lets users input two numbers, and behind the scenes, we run some R code that adds them and prints the sum. Our script is in a file called `launcher.R`, and we tell Domino to run `launcher.R 10 20`. So, our launcher’s command is `launcher.R ${A} ${B}`.

    ```r theme={null}
    args <- commandArgs(trailingOnly = TRUE)
    a <- as.integer(args[1])
    b <- as.integer(args[2])
    if (is.na(a)) {
      print("A is not a number")
    } else if (is.na(b)){
      print("B is not a number")
    } else {
      paste("The sum of", a, "and", b, "is:", a+b)
    }
    ```
  </Tab>

  <Tab title="Python">
    This Python example uses a script that creates an interactive scatter plot with [Bokeh](https://github.com/bokeh/bokeh) from a CSV file that anyone can upload with a web form.

    The user provides this information:

    * a file

    * what to put on the X- and Y- axes

    * some information about how to color the data points.

    The Python script generates an interactive Bokeh scatterplot.

    This code shows the complete Python script:

    ```python theme={null}
    from bokeh.plotting import show, output_file
    from bokeh.charts import Scatter
    import pandas as pd
    import sys

    output_file("scatter.html")

    data = pd.read_csv(sys.argv[1])
    scatter = Scatter(data, x = sys.argv[2], y = sys.argv[3], color = sys.argv[4], legend = "top_left")
    show(scatter)
    ```
  </Tab>
</Tabs>

## Run a Launcher as a consumer

Navigate to the **Deployments > Launchers** section of a Project and click a launcher.

Collaborators in the **Contributors** or **Launcher User** role can run a launcher. **Results Consumers** can view results from launchers but not run them.

An email with the results is sent to everyone on the notification list. A new job is also created in the Jobs dashboard, where the results and inputs are accessible.

## Copy Launcher definition

<Note>
  This feature is only available in private deployments of Domino Enterprise.
</Note>

Use the Advanced Launcher Editor to access the JSON representation of a Launcher. This is useful if you want to copy Launcher definitions between projects.

1. Go to the launcher.

2. Click the gear icon and then click **Edit**.

3. Click **Switch to JSON Edit Mode**.

## Advanced use cases, tips, and tricks

Combine with [knitr](https://cran.r-project.org/web/packages/knitr/index.html) for report generation\
Write an R script that knits a rich report in PDF, HTML, Markdown, etc., and flexibly expose parameters via a launcher to provide self-serve report generation to business users.

Combine with Domino’s customized email notifications, for example, [email.html](/cloud/platform-capabilities/core-concepts/jobs/customize-job-results)\
If your launcher script generates an `email.html` file, that becomes the body of the notification email sent to the launcher consumer. Launchers work great to create custom visual results that are sent directly as email notifications.

**Render Jupyter notebooks**\
If you run a Domino job with the name of an `ipynb` file, Domino calculates the notebook and saves it. This automation gives you an easy way to let end users update a notebook without opening Jupyter. For more advanced output formats, create a shell script that uses nbconvert to render a notebook as HTML or a PDF and email that to the launcher consumer.

**Process data**\
Launchers are great when a user wants to upload a file and process the data, for example, clean it and put it in a database.

**Score or infer with explainability**\
Launchers are great when a user wants to access input, run it through a model, and return the result with rich explainability data (provided via [shap](https://shap.readthedocs.io/en/latest/) or [lime](https://github.com/marcotcr/lime), for example).

## Launchers on remote data planes

You may want to launch Jobs in a separate cluster or data plane from where the model was created, either for scalability or data locality reasons. Domino enables this workflow through its Launchers feature, allowing a Launcher to be deployed near where it adds the most value. When launching, select the hardware tier linked to your preferred local or remote data plane.


# Create a Dataset
Source: https://docs.domino.ai/api-reference/datasets/create-a-dataset

/api-specs/cloud/public-api.json post /api/datasetrw/v1/datasets
Create a new Dataset. Requires access to the project the Dataset will originate from



# Delete Dataset
Source: https://docs.domino.ai/api-reference/datasets/delete-dataset

/api-specs/cloud/public-api.json delete /api/datasetrw/v1/datasets/{datasetId}
Delete a Dataset. Requires PermanentDelete access to the Dataset



# Deprecated - Get Datasets accessible to the user
Source: https://docs.domino.ai/api-reference/datasets/deprecated--get-datasets-accessible-to-the-user

/api-specs/cloud/public-api.json get /api/datasetrw/v1/datasets
Deprecated: Use getDatasetsV2. Get Datasets that a user has access to



# Get Dataset by ID
Source: https://docs.domino.ai/api-reference/datasets/get-dataset-by-id

/api-specs/cloud/public-api.json get /api/datasetrw/v1/datasets/{datasetId}
Get Dataset by ID. Requires List access to the Dataset



# Update Dataset metadata
Source: https://docs.domino.ai/api-reference/datasets/update-dataset-metadata

/api-specs/cloud/public-api.json patch /api/datasetrw/v1/datasets/{datasetId}
Update Dataset name or description. Requires Update access to the Dataset



# Archive an environment
Source: https://docs.domino.ai/api-reference/environments/archive-an-environment

/api-specs/cloud/public-api.json delete /api/environments/v1/environments/{environmentId}
Archive an Environment, removing it from the list of visible environments. Required permissions: `ManageEnvironments, EditEnvironment`



# Create a Revision of an Environment
Source: https://docs.domino.ai/api-reference/environments/create-a-revision-of-an-environment

/api-specs/cloud/public-api.json post /api/environments/beta/environments/{environmentId}/revisions
**Deprecated.** Use POST /api/environments/v2/environments/{environmentId}/revisions instead. This endpoint does not support specifying a CPU architecture and will not gain new capabilities.
Required permissions: `ManageEnvironments, EditEnvironment, UseFileStorage`.




# Create a Revision of an Environment (V2)
Source: https://docs.domino.ai/api-reference/environments/create-a-revision-of-an-environment-v2

/api-specs/cloud/public-api.json post /api/environments/v2/environments/{environmentId}/revisions
Create a revision of an environment, with support for selecting CPU architecture(s). Required permissions: `ManageEnvironments, EditEnvironment, UseFileStorage`.



# Create an environment
Source: https://docs.domino.ai/api-reference/environments/create-an-environment

/api-specs/cloud/public-api.json post /api/environments/beta/environments
**Deprecated.** Use POST /api/environments/v2/environments instead. This endpoint does not support specifying a CPU architecture and will not gain new capabilities.
Required permissions: `CreateEnvironment, EditEnvironment, UseFileStorage`.




# Create an environment (V2)
Source: https://docs.domino.ai/api-reference/environments/create-an-environment-v2

/api-specs/cloud/public-api.json post /api/environments/v2/environments
Create an environment, with support for selecting CPU architecture(s). Required permissions: `CreateEnvironment, EditEnvironment, UseFileStorage`.



# Get an environment
Source: https://docs.domino.ai/api-reference/environments/get-an-environment

/api-specs/cloud/public-api.json get /api/environments/v1/environments/{environmentId}
Get an Environment by its Id. Required permissions: `ViewEnvironment`



# Get Environments visible to a user
Source: https://docs.domino.ai/api-reference/environments/get-environments-visible-to-a-user

/api-specs/cloud/public-api.json get /api/environments/beta/environments
Get environments that a user can see. Required permissions: `None`. *Note:* This is a beta endpoint with known limitations.



# Rename an environment
Source: https://docs.domino.ai/api-reference/environments/rename-an-environment

/api-specs/cloud/public-api.json put /api/environments/v1/environments/{environmentId}/name
Change the display name of an existing compute environment. The new name must be
unique for the environment owner and cannot be blank.

Required permissions: `EditEnvironment`

Related:
- GET /api/environments/v1/environments/{environmentId} (read current name and metadata)
- PUT /api/environments/v1/environments/{environmentId}/description (update description)
- DELETE /api/environments/v1/environments/{environmentId} (archive environment)




# Update an environment description
Source: https://docs.domino.ai/api-reference/environments/update-an-environment-description

/api-specs/cloud/public-api.json put /api/environments/v1/environments/{environmentId}/description
Change the long-form description shown on the environment details page. Use this
to document purpose, base image notes, or usage guidance for collaborators.

Required permissions: `EditEnvironment`

Related:
- GET /api/environments/v1/environments/{environmentId} (read current description)
- PUT /api/environments/v1/environments/{environmentId}/name (rename environment)
- DELETE /api/environments/v1/environments/{environmentId} (archive environment)




# Update the restricted revision of an environment
Source: https://docs.domino.ai/api-reference/environments/update-the-restricted-revision-of-an-environment

/api-specs/cloud/public-api.json patch /api/environments/beta/environments/{environmentId}/revisions/{revisionId}
Update a revision of an environment to mark if isRestricted. Required permissions: `ClassifyEnvironments`. *Note:* This is a beta endpoint with known limitations.



# Archive a hardware tier
Source: https://docs.domino.ai/api-reference/hardware-tiers/archive-a-hardware-tier

/api-specs/cloud/public-api.json delete /api/hardwaretiers/v1/hardwaretiers/{hardwareTierId}
Archive a hardware tier by Id. Required permissions: `ManageHardwareTiers`



# Create a hardware tier
Source: https://docs.domino.ai/api-reference/hardware-tiers/create-a-hardware-tier

/api-specs/cloud/public-api.json post /api/hardwaretiers/v1/hardwaretiers
Create a hardware tier. Required permissions: `ManageHardwareTiers`



# Get a hardware tier by Id
Source: https://docs.domino.ai/api-reference/hardware-tiers/get-a-hardware-tier-by-id

/api-specs/cloud/public-api.json get /api/hardwaretiers/v1/hardwaretiers/{hardwareTierId}
Get a hardware tier by Id. Required permissions: `ViewHardwareTiers`



# Get all hardware tiers
Source: https://docs.domino.ai/api-reference/hardware-tiers/get-all-hardware-tiers

/api-specs/cloud/public-api.json get /api/hardwaretiers/v1/hardwaretiers
Get all hardware tiers. Required permissions: `ViewHardwareTiers`



# Update a hardware tier
Source: https://docs.domino.ai/api-reference/hardware-tiers/update-a-hardware-tier

/api-specs/cloud/public-api.json put /api/hardwaretiers/v1/hardwaretiers
Update a hardware tier. Required permissions: `ManageHardwareTiers`



# Cancel an HPC Job
Source: https://docs.domino.ai/api-reference/hpc-jobs/cancel-an-hpc-job

/api-specs/cloud/public-api.json post /api/hpc/v1/jobs/cancel
Cancel a running or queued HPC Job. Required permissions: `StartJob`



# Check whether Slurm submission is supported
Source: https://docs.domino.ai/api-reference/hpc-jobs/check-whether-slurm-submission-is-supported

/api-specs/cloud/public-api.json get /api/hpc/v1/jobs/submit-supported
Asks the target data plane whether it currently supports submitting Slurm jobs for the calling user and project. Required permissions: `StartJob, UseGlobalCompute`



# Get HPC Job details
Source: https://docs.domino.ai/api-reference/hpc-jobs/get-hpc-job-details

/api-specs/cloud/public-api.json get /api/hpc/v1/jobs/{jobId}
Retrieve an HPC Job's details by its Id. Required permissions: `ViewJobs`



# Get HPC Jobs for a project
Source: https://docs.domino.ai/api-reference/hpc-jobs/get-hpc-jobs-for-a-project

/api-specs/cloud/public-api.json get /api/hpc/v1/jobs
Retrieve all HPC Jobs that belong to a project. Required permissions: `ViewJobs`



# Get logs for an HPC Job
Source: https://docs.domino.ai/api-reference/hpc-jobs/get-logs-for-an-hpc-job

/api-specs/cloud/public-api.json get /api/hpc/v1/jobs/{jobId}/logs
Retrieve the logs for the HPC Job with the specified Id. Required permissions: `ViewJobs`



# Start an HPC Job
Source: https://docs.domino.ai/api-reference/hpc-jobs/start-an-hpc-job

/api-specs/cloud/public-api.json post /api/hpc/v1/jobs
Start a new HPC (Slurm) Job. Required permissions: `StartJob, UseGlobalCompute`



# Add a tag to a Job
Source: https://docs.domino.ai/api-reference/jobs/add-a-tag-to-a-job

/api-specs/cloud/public-api.json post /api/jobs/v1/jobs/{jobId}/tags
Add a Tag to the Job with the specified Id. Required permissions: `ViewJobs`



# Get Job details
Source: https://docs.domino.ai/api-reference/jobs/get-job-details

/api-specs/cloud/public-api.json get /api/jobs/beta/jobs/{jobId}
Retrieve a Job's details by its Id. Required permissions: `ViewJobs`. *Note:* This is a beta endpoint with known limitations.



# Get Jobs for a project
Source: https://docs.domino.ai/api-reference/jobs/get-jobs-for-a-project

/api-specs/cloud/public-api.json get /api/jobs/beta/jobs
Retrieve all Jobs that belong to a project. Required permissions: `ViewJobs.` *Note:* This is a beta endpoint with known limitations.



# Get linked goals for a job
Source: https://docs.domino.ai/api-reference/jobs/get-linked-goals-for-a-job

/api-specs/cloud/public-api.json get /api/jobs/v1/goals
Retrieve goals for a Job with the specified Id. Required permissions: `ViewJobs`



# Get logs for a Job
Source: https://docs.domino.ai/api-reference/jobs/get-logs-for-a-job

/api-specs/cloud/public-api.json get /api/jobs/beta/jobs/{jobId}/logs
Retrieve the logs for the Job with the specified Id. Required permissions: `ViewJobs`. *Note:* This is a beta endpoint with known limitations.



# Link a goal to a job
Source: https://docs.domino.ai/api-reference/jobs/link-a-goal-to-a-job

/api-specs/cloud/public-api.json post /api/jobs/v1/goals
Link the Goal with the specified Id to a Job. Required permissions: `ViewJobs, Edit`



# Remove a tag from a Job
Source: https://docs.domino.ai/api-reference/jobs/remove-a-tag-from-a-job

/api-specs/cloud/public-api.json delete /api/jobs/v1/jobs/{jobId}/tags/{tagId}
Remove a Tag from the Job with the specified Id. Required permissions: `ViewJobs`



# Resolve the default configuration a Job would run with
Source: https://docs.domino.ai/api-reference/jobs/resolve-the-default-configuration-a-job-would-run-with

/api-specs/cloud/public-api.json post /api/jobs/v1/projects/{projectId}/job-defaults
Resolve the configuration a Job in this project would run with, without starting anything. Every field in the request body is optional: what you send is taken as an override, and everything you leave out is filled in from the project's defaults. The response reports the fully resolved hardware tier, environment and revision, git ref, dataset and NetApp volume snapshots, external volumes and volume size, so a caller can show or record the exact configuration before creating a Job.

Seven fields are named exactly as `POST /api/jobs/v1/jobs` names them, so each resolved value can be copied straight into a start-Job request: `hardwareTier`, `environmentId`, `environmentRevisionSpec`, `mainRepoGitRef`, `computeCluster`, `externalVolumeMountIds` and `netAppVolumeIds`. `commitId` is named the same but carries over only for projects using Domino's internal file store; see the field description on `NewJobV1`. The remaining fields have no counterpart on `NewJobV1`: `datasetSnapshots`, `netAppVolumeSnapshots`, `volumeSizeGiB`, `resolvedMainRepoGitRef` and `resolvedImportedGitRepoRefs` are reported for inspection, not for starting a Job. Send the individual fields rather than the whole object.

Resolution is read-only and leaves no trace on the project. Required permissions: `ViewJobs`



# Start a Job
Source: https://docs.domino.ai/api-reference/jobs/start-a-job

/api-specs/cloud/public-api.json post /api/jobs/v1/jobs
Start a new Job. Required permissions: `StartJob, UseGlobalCompute`



# Unlink goal from job
Source: https://docs.domino.ai/api-reference/jobs/unlink-goal-from-job

/api-specs/cloud/public-api.json delete /api/jobs/v1/goals/{goalId}
Unlink the Goal with the specified Id from a Job. Required permissions: `ViewJobs, Edit`



# Create project template
Source: https://docs.domino.ai/api-reference/project-templates/create-project-template

/api-specs/cloud/public-api.json post /api/develop/v1/customer-project-templates



# Delete a customer project template by id.
Source: https://docs.domino.ai/api-reference/project-templates/delete-a-customer-project-template-by-id

/api-specs/cloud/public-api.json delete /api/develop/v1/customer-project-templates/{id}



# Get customer project template by id
Source: https://docs.domino.ai/api-reference/project-templates/get-customer-project-template-by-id

/api-specs/cloud/public-api.json get /api/develop/v1/customer-project-templates/{id}



# List project templates
Source: https://docs.domino.ai/api-reference/project-templates/list-project-templates

/api-specs/cloud/public-api.json get /api/develop/v1/project-templates



# List project templates (v2)
Source: https://docs.domino.ai/api-reference/project-templates/list-project-templates-v2

/api-specs/cloud/public-api.json get /api/develop/v2/project-templates
Lists project templates. Replaces the v1 `tagName` query parameter with `tagIds`, which accepts taxonomy tag IDs instead of legacy tag names.




# Update a customer project template
Source: https://docs.domino.ai/api-reference/project-templates/update-a-customer-project-template

/api-specs/cloud/public-api.json patch /api/develop/v1/customer-project-templates/{id}



# Add a collaborator to this project
Source: https://docs.domino.ai/api-reference/projects/add-a-collaborator-to-this-project

/api-specs/cloud/public-api.json post /api/projects/v1/projects/{projectId}/collaborators
Add a collaborator to this project. Required permissions: `ManageCollaborators`



# Add a goal to this project
Source: https://docs.domino.ai/api-reference/projects/add-a-goal-to-this-project

/api-specs/cloud/public-api.json post /api/projects/v1/projects/{projectId}/goals
Add a goal to this project. Required permissions: `Edit`



# Add an imported git repository to this project
Source: https://docs.domino.ai/api-reference/projects/add-an-imported-git-repository-to-this-project

/api-specs/cloud/public-api.json post /api/projects/v1/projects/{projectId}/repositories
Add an imported git repository to this project. Required permissions: `ChangeProjectSettings`



# Archive project by id
Source: https://docs.domino.ai/api-reference/projects/archive-project-by-id

/api-specs/cloud/public-api.json delete /api/projects/beta/projects/{projectId}
Archive a project by id. Required permissions: `ArchiveProject`. *Note:* This is a beta endpoint with known limitations.



# Create a new project by copying an existing project and providing optional overrides.
Source: https://docs.domino.ai/api-reference/projects/create-a-new-project-by-copying-an-existing-project-and-providing-optional-overrides

/api-specs/cloud/public-api.json post /api/projects/v1/projects/{projectId}/copy-project
Create a new project by copying an existing project and providing optional overrides. Specify a git repository to link to the copied project or copy the original project's git repository for the copied project.



# Create a project
Source: https://docs.domino.ai/api-reference/projects/create-a-project

/api-specs/cloud/public-api.json post /api/projects/beta/projects
Create a project. Required permissions: `CreateProject, UseFileStorage`. *Note:* This is a beta endpoint with known limitations.



# Create a project
Source: https://docs.domino.ai/api-reference/projects/create-a-project-1

/api-specs/cloud/public-api.json post /api/projects/v2/projects
Create a project. Required permissions: `CreateProject, UseFileStorage`.



# Delete a project goal
Source: https://docs.domino.ai/api-reference/projects/delete-a-project-goal

/api-specs/cloud/public-api.json delete /api/projects/v1/projects/{projectId}/goals/{goalId}
Delete a project goal. Required permissions: `Edit`



# Get a rendered view of a file in a project's git repository
Source: https://docs.domino.ai/api-reference/projects/get-a-rendered-view-of-a-file-in-a-projects-git-repository

/api-specs/cloud/public-api.json get /api/projects/v1/projects/{projectId}/repositories/{repositoryId}/rendered-file
Render a file from the project's main repository or from one of its imported repositories as HTML, for formats Domino knows how to render such as Markdown and notebooks. The HTML is produced from repository content and can carry active markup such as script tags, so treat it as untrusted: sanitize or sandbox it before inserting it into a page. Images and links in it point at Domino's internal raw file route, which is not part of the public contract and may change. Required permissions: `ListProject`



# Get all imported git repositories in this project
Source: https://docs.domino.ai/api-reference/projects/get-all-imported-git-repositories-in-this-project

/api-specs/cloud/public-api.json get /api/projects/v1/projects/{projectId}/repositories
Get all imported git repositories in this project. Required permissions: `ListProject`



# Get DFS head commit
Source: https://docs.domino.ai/api-reference/projects/get-dfs-head-commit

/api-specs/cloud/public-api.json get /api/projects/beta/projects/{projectId}/commits/latest
Returns the current DFS head commit ID for the project. Required permissions: `ListProject`.



# Get goals in this project
Source: https://docs.domino.ai/api-reference/projects/get-goals-in-this-project

/api-specs/cloud/public-api.json get /api/projects/v1/projects/{projectId}/goals
Get goals in this project. Required permissions: `ListProject`



# Get metadata about a file in a project's git repository
Source: https://docs.domino.ai/api-reference/projects/get-metadata-about-a-file-in-a-projects-git-repository

/api-specs/cloud/public-api.json get /api/projects/v1/projects/{projectId}/repositories/{repositoryId}/file-metadata
Get size, MIME type and path for a single file in the project's main repository or in one of its imported repositories, without fetching its contents. Required permissions: `ListProject`



# Get Project by Id
Source: https://docs.domino.ai/api-reference/projects/get-project-by-id

/api-specs/cloud/public-api.json get /api/projects/v1/projects/{projectId}
Get project by id. Required permissions: `ListProject`



# Get project result settings
Source: https://docs.domino.ai/api-reference/projects/get-project-result-settings

/api-specs/cloud/public-api.json get /api/projects/beta/projects/{projectId}/results-settings



# Get Projects visible to user
Source: https://docs.domino.ai/api-reference/projects/get-projects-visible-to-user

/api-specs/cloud/public-api.json get /api/projects/beta/projects
Get projects that a user can see. Required permissions: `ListProject`. *Note:* This is a beta endpoint with known limitations.



# Get the name of the readme file in a project's git repository
Source: https://docs.domino.ai/api-reference/projects/get-the-name-of-the-readme-file-in-a-projects-git-repository

/api-specs/cloud/public-api.json get /api/projects/v1/projects/{projectId}/repositories/{repositoryId}/readme
Find the readme file in a directory of the project's main repository or of one of its imported repositories. Returns the name of the readme as the repository spells it; the name is omitted when the directory has no readme. Required permissions: `ListProject`



# Remove a collaborator from project
Source: https://docs.domino.ai/api-reference/projects/remove-a-collaborator-from-project

/api-specs/cloud/public-api.json delete /api/projects/v1/projects/{projectId}/collaborators/{collaboratorId}
Remove a collaborator from the project. Required permissions: `ManageCollaborators`



# Remove an imported repository from project
Source: https://docs.domino.ai/api-reference/projects/remove-an-imported-repository-from-project

/api-specs/cloud/public-api.json delete /api/projects/v1/projects/{projectId}/repositories/{repositoryId}
Remove an imported repository from the project. Required permissions: `ChangeProjectSettings`



# Resolve a git reference to a commit SHA
Source: https://docs.domino.ai/api-reference/projects/resolve-a-git-reference-to-a-commit-sha

/api-specs/cloud/public-api.json post /api/projects/beta/projects/{projectId}/commits/resolveGitRef
Resolves a git reference (branch, tag, etc.) to a concrete commit SHA for the project's main repository. Required permissions: `ListProject`.



# Resolve a git reference to a commit SHA for an imported repository
Source: https://docs.domino.ai/api-reference/projects/resolve-a-git-reference-to-a-commit-sha-for-an-imported-repository

/api-specs/cloud/public-api.json post /api/projects/beta/projects/{projectId}/repositories/{repositoryId}/commits/resolveGitRef
Resolves a git reference (branch, tag, head, etc.) to a concrete commit SHA for the given imported git repository on this project. Required permissions: `ListProject`.



# Returns the contents of a file
Source: https://docs.domino.ai/api-reference/projects/returns-the-contents-of-a-file

/api-specs/cloud/public-api.json get /api/projects/v1/projects/{projectId}/files/{commitId}/{path}/content
Return the raw contents of a file in a project at given commit.



# Returns the contents of a file (deprecated, use v1 endpoint instead)
Source: https://docs.domino.ai/api-reference/projects/returns-the-contents-of-a-file-deprecated-use-v1-endpoint-instead

/api-specs/cloud/public-api.json get /api/projects/beta/projects/{projectId}/files/{commitId}/{path}/content
Return the raw contents of a file in a project at given commit.



# Update project goal status
Source: https://docs.domino.ai/api-reference/projects/update-project-goal-status

/api-specs/cloud/public-api.json patch /api/projects/v1/projects/{projectId}/goals/{goalId}
Update project goal status. Required permissions: `Edit`



# Update project result settings
Source: https://docs.domino.ai/api-reference/projects/update-project-result-settings

/api-specs/cloud/public-api.json put /api/projects/beta/projects/{projectId}/results-settings



# Update project status
Source: https://docs.domino.ai/api-reference/projects/update-project-status

/api-specs/cloud/public-api.json put /api/projects/v1/projects/{projectId}/status
Update the status of a project. Required permissions: `Edit`



# Update the default git reference of a repository in a project
Source: https://docs.domino.ai/api-reference/projects/update-the-default-git-reference-of-a-repository-in-a-project

/api-specs/cloud/public-api.json put /api/projects/v1/projects/{projectId}/repositories/{repositoryId}/default-ref
Set the default git reference Domino uses for the project's main repository or for one of its imported repositories. Runs and workspaces started after the update read the repository at this reference. Required permissions: `Edit`



# Create workspace session
Source: https://docs.domino.ai/api-reference/workspaces/create-workspace-session

/api-specs/cloud/public-api.json post /api/projects/v1/projects/{projectId}/workspaces/{workspaceId}/sessions
Creates a new session given an existing workspace. Required permissions: `OpenWorkspace`, `UseGlobalCompute`



# Get writable project mount paths for a Workspace
Source: https://docs.domino.ai/api-reference/workspaces/get-writable-project-mount-paths-for-a-workspace

/api-specs/cloud/public-api.json get /api/workspace/v1/workspace/{workspaceId}/writable-project-mounts
Retrieve the absolute writable project mount paths for a restartable Workspace. The response includes the main
Git repository, the main Domino File System (DFS) directory, and imported Git repositories. Required
permissions: `CreateWorkspace, UseGlobalCompute`.




# Disaster recovery
Source: https://docs.domino.ai/cloud/admin/operations/disaster-recovery

Review Domino's canonical data stores, backup methods, recovery targets, and support procedures for cluster, service, or application failures.

By default, Domino runs in a High Availability mode, replicating its core components across three or more nodes in the cluster. Data losses are rare and unlikely to happen. However, in the event of a service disruption which impacts the entire cluster, Domino recommends recovery point (RPO) targets of 0 to 12 hours depending on the service level requirements of your organization. Regardless of the failure scenario, Domino targets a recovery time objective (RTO) of 8 hours.

The following systems are canonical stores of critical Domino data, and they are stored and backed up in AWS as described in the following table.

| System | Purpose | Store | Backup |
| - | - | - | - |
| Projects store | Stores the contents of users' [project files](/cloud/platform-capabilities/core-concepts/projects#whats-in-a-project). | Stored in [S3](https://aws.amazon.com/s3/) in the bucket specified in the installer configuration (`blob_storage.projects.s3.bucket`). | Relies on the [inherent durability and automated replication and backups of S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html). |
| Log history | Stores execution logs from Domino jobs and workspaces. | Stored in [S3](https://aws.amazon.com/s3/) in the bucket specified in the installer configuration (`blob_storage.projects.s3.bucket`). | Relies on the [inherent durability and automated replication and backups of S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html). |
| Docker registry | Stores Docker images built by Domino to back users' [Domino environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments) and [Domino endpoints](/cloud/platform-capabilities/features/model-deployment). | Stored in [S3](https://aws.amazon.com/s3/) in the bucket specified in the installer configuration (`internal_docker_registry_s3_override_bucket`). | Relies on the [inherent durability and automated replication and backups of S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/DataDurability.html). |
| Datasets | Stores the contents of users’ Domino Datasets. | Stored in [EFS](https://aws.amazon.com/efs/) in the file system specified in the installer config (`storage_classes.shared.efs.filesystem_id`). | Relies on the [inherent durability and automated replication and backups of EFS](https://aws.amazon.com/efs/faq/#Data_protection_and_availability). |
| MongoDB | Stores Domino application object data and metadata. | Stored in EBS-backed Kubernetes persistent volumes attached to pods running the HA MongoDB service in the Domino cluster. | Domino performs a daily backup that writes an archive containing all MongoDB data to S3 in the bucket specified in the installer configuration (`blob_storage.backups.s3.bucket`). |
| Domino Git | Stores Domino project version history. | Stored in EBS-backed Kubernetes persistent volumes attached to pod running the Domino Git service in the Domino cluster. | Domino performs a daily backup that writes an archive containing all Git data to S3 in the bucket specified in the installer configuration (`blob_storage.backups.s3.bucket`). |
| Postgres | Stores information on users managed by the Keycloak authentication service. | Stored in EBS-backed Kubernetes persistent volumes attached to pods running the HA Postgres service in the Domino cluster. | Domino performs a daily backup that writes an archive containing all Postgres data to S3 in the bucket specified in the installer configuration (`blob_storage.backups.s3.bucket`). |

## Recovery

If you choose to run “hot” backup instances or additional storage facilities for disaster recovery, you can request an increase in quota limits for one or more of the AWS services if needed. You can check the [default service quotas](https://docs.aws.amazon.com/general/latest/gr/aws-service-information.html) and manage your quotas by logging in to the [AWS Service Quotas console](https://console.aws.amazon.com/servicequotas/home).

If you need to restore a deployment from backups, [submit a maximum severity support request](https://tickets.dominodatalab.com/hc/en-us/articles/360059918151-Domino-Support-Operating-Procedures) and the Domino operations team assists with the recovery procedure.

In the case of instance or service failure, [submit a maximum severity support request](https://tickets.dominodatalab.com/hc/en-us/articles/360059918151-Domino-Support-Operating-Procedures) and the Domino operations team assists with restoring the cluster to normal operation.

In the case of critical application failure, [submit a maximum severity support request](https://tickets.dominodatalab.com/hc/en-us/articles/360059918151-Domino-Support-Operating-Procedures) and the Domino operations team assists with restoring the cluster to normal operation.

By default, Domino runs in a “High Availability” mode, replicating its core components across three or more nodes in the cluster. Should a fault condition occur, [submit an appropriate severity support request ticket](https://tickets.dominodatalab.com/hc/en-us/articles/360059918151-Domino-Support-Operating-Procedures) and the Domino operations team assists with restoring the cluster to normal operation.

## Domino backups on-premises

The following systems are canonical stores of critical Domino data, and they are stored and backed on-premises as described below. These methods can also be applied to other clouds for which Domino does not have native storage integrations.

| System | Purpose | Store | Backup |
| - | - | - | - |
| Projects store | Stores the contents of users’ [project files](/cloud/platform-capabilities/core-concepts/projects#whats-in-a-project). | Stored in the shared storage class (`storage_classes.shared`), referenced by `blob_storage.projects`. The most common backing storage for this class is NFS. | Domino does not automatically back up this data. Domino recommends configuring a separate backup process that saves or syncs the entire contents of the NFS mount path provided to Domino to another storage system. The volume of project data grows linearly over time as users produce new projects, so you should account for the growing volume when setting backup frequency and retention policies. |
| Log history | Stores execution logs from Domino jobs and workspaces. | Stored in the shared storage class (`storage_classes.shared`), referenced by `blob_storage.logs`. | This is stored in the same volume as project data, and is therefore on the same filesystem path in the shared storage system. Any backups of the root path provided to Domino include log data. |
| Docker registry | Stores Docker images built by Domino to back users’ [Domino environments](/cloud/admin/platform-configuration/manage-compute-environments) and [Domino endpoints](/cloud/platform-capabilities/features/model-deployment). | Stored in a block storage volume provisioned from the storage class (`storage_classes.block`), mounted in the Docker registry container. | Domino does not automatically back up this data. Domino recommends configuring a separate backup process that saves or syncs the entire contents of the NFS mount path provided to Domino to another storage system. The volume of project data grows linearly over time as users produce new projects, so you should account for the growing volume when setting backup frequency and retention policies. |
| Domino Volumes for NetApp ONTAP | Stores the contents of users' [Domino Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes). | Defaults to the shared storage class (`storage_classes.shared`), but can be extended and configured by an admin. | Domino does not automatically back up Domino Volumes. Disaster recovery relies on NetApp’s native data protection features such as snapshot schedules, SnapMirror replication, or SnapVault. Administrators should configure policies that align with organizational data recovery requirements. |
| Datasets | Stores the contents of users’ [Domino Datasets](/cloud/platform-capabilities/core-concepts/data/datasets/create-and-manage-datasets). | Stored in the shared storage class (`storage_classes.shared`). | This is stored in the same volume as project data, and is therefore on the same filesystem path in the shared storage system. Any backups of the root path provided to Domino include log data. |
| MongoDB | Stores Domino application object data and metadata. | Stored in a block storage volume provisioned from the storage class (`storage_classes.block`), mounted in the Mongo containers. | Domino performs a daily backup that writes an archive containing all MongoDB data to the shared storage class (`storage_classes.shared`), referenced by `blob_storage.backups`. |
| Domino Git | Stores Domino project version history. | Stored in a block storage volume provisioned from the storage class (`storage_classes.block`), mounted in the Git container. | Domino performs a daily backup that writes an archive containing all Git data to the shared storage class (`storage_classes.shared`), referenced by `blob_storage.backups`. |
| Postgres | Stores information on users managed by the Keycloak authentication service. | Stored in a block storage volume provisioned from the storage class (`storage_classes.block`), mounted in the Postgres containers. | Domino performs a daily backup that writes an archive containing all Postgres data to the shared storage class (`storage_classes.shared`), referenced by (`blob_storage.backups`). |

## Support

Domino support coverage is outlined on our [Scope of Support Coverage page](https://tickets.dominodatalab.com/hc/en-us/articles/360059504652-Scope-of-Support-Coverage). You can learn more about our support priority codes, ticket statuses, and business hours on our [Support Definitions page](https://tickets.dominodatalab.com/hc/en-us/articles/360060058591-Support-Definitions).

In addition to the support tiers described above, some Domino customers may have unique contractual SLAs defined in the Master Service Agreement. All customers receive a standard [service level agreement](https://tickets.dominodatalab.com/hc/en-us/articles/360060058871-Service-Level-Agreement).


# Analyze costs using API
Source: https://docs.domino.ai/cloud/admin/operations/finops/3-analyze-costs-API

Use Domino Cost API endpoints to query allocation summaries, Project and user totals, asset costs, and accumulated cloud costs.

The Domino Cost Center APIs allow aggregation only by labels of Domino domain objects, not by arbitrary Kubernetes objects, such as clusters.

## Access Cost Center through Domino Cost API

You can use the Domino Cost API to access Cost Center features.

See the [Domino Platform API Reference](/cloud/reference/api/domino-open-api) documentation for the field options available with Cost Center APIs.

<Note>
  Only [CloudAdmins](/cloud/admin/identity-and-access/manage-users/roles) can use the `/allocation` and `/asset` API paths.
</Note>

### API samples

Domino recommends using [Service Accounts](/cloud/admin/identity-and-access/manage-users/service-accounts) to access the Domino API.

#### Today’s total cost with idle cost shared among allocations

```shell theme={null}
curl -H "Authorization: Bearer $TOKEN" -H "Accept: application/json" https://$DOMAIN/api/cost/v2/allocation/summary?window=7d&shareIdle=true -G
```

#### Today’s total for a particular Project

```shell theme={null}
curl -H "Authorization: Bearer $TOKEN" -H "Accept: application/json" https://$DOMAIN/api/cost/v2/allocation/summary?window=1d&&aggregate=projectId&aggregate=projectName&filter=projectId:%$PROJECT-ID%22 -G
```

#### Today’s total for a particular user

```shell theme={null}
curl -H "Authorization: Bearer $TOKEN" -H "Accept: application/json" https://$DOMAIN/api/cost/v2/allocation/summary?window=7d&shareIdle=true&filter=startingUserId:%$USER-ID%22 -G
```

#### Today’s asset costs

```shell theme={null}
curl -H "Authorization: Bearer $TOKEN" -H "Accept: application/json" https://$DOMAIN/api/cost/v2/asset?window=1d -G
```

#### Today’s cloud cost

```shell theme={null}
curl -H "Authorization: Bearer $TOKEN" -H "Accept: application/json" https://$DOMAIN/api/cost/v2/cloudCost/accumulated?window=7d&&aggregate=invoiceEntityID -G
```

## Next steps

See other ways Domino Cost Center can help you optimize the performance of your projects while managing cloud expenditures:

* Prevent overspending on compute and storage by [setting budgets and sending alerts](/cloud/admin/operations/finops/1-budgets-alerts) to team members approaching their limits.

* Optimize [Dataset storage usage](/cloud/admin/data-administration/datasets#set-limits-on-dataset-usage) to reduce cloud storage costs.


# Cost Calculation Methodology
Source: https://docs.domino.ai/cloud/admin/operations/finops/4-cost-calculation-methodology

Trace Cost Center calculations from hourly Kubecost data through CPU, GPU, memory, allocation, storage, and shared service totals.

Domino’s Cost Center integrates with Kubecost to collect and allocate cost data for your organization. This integration synchronizes hourly cost data and maps it to key Domino entities, including users, projects, and workloads.

## Benefits of using Kubecost

Kubecost provides deep integration with major cloud service providers including AWS, Azure, and GCP. This approach ensures:

* A standardized, transparent cost model across platforms.

* Hourly synchronization with complete billing data from your cloud provider.

* Automatic accounting for pricing discounts negotiated with your CSP.

## Cost allocation to Domino entities

Each hour, Kubecost ingests billing data and attributes it to Domino-specific entities using its allocation engine. These entities include:

* Users

* Projects

* Organizations

* Billing Tags

* Workloads

* Workload Types

This enables you to trace costs directly to specific users, workloads, and teams within Domino.

## Discount handling and cost computation

Compute pricing discounts, covering CPU, GPU, and RAM, are reconciled against the actual billing data from your cloud provider. This ensures accurate reflection of usage-based costs with the following formulas:

* `CPU COST` = `cpuCost` + `cpuCostAdjustment`

* `GPU COST` = `gpuCost` + `gpuCostAdjustment`

* `MEMORY COST` = `ramCost` + `ramCostAdjustment`

* `ALLOC COST` = `totalCost`

* `COMPUTE COST` = `CPU COST` + `GPU COST`

* `STORAGE COST` = `ALLOC COST` - `COMPUTE COST`

Adjustments in each formula represent discounts or credits applied by the CSP.

## Cost representation in the Domino UI

Kubecost also tracks the total cloud billing amount. In Domino, this is surfaced in the Cost Center’s **Total Cost** top-bar metric. Any cloud costs that cannot be mapped to compute or storage usage (such as shared infrastructure expenses) appear under **Cloud Services**.

## Data retention and reconciliation

Cost data is retained for up to one year. Be aware that recent cost data, usually from the past 36-48 hours, may be incomplete due to delays in cloud provider reporting. This data is automatically reconciled once full billing details become available.

## Next steps

* Prevent overspending by [setting budgets and sending alerts](/cloud/admin/operations/finops/1-budgets-alerts) to stakeholders, administrators, and team members approaching their limits.

* Learn more about [automatic allocation of usage-based costs](/cloud/admin/operations/finops/3-analyze-costs-API).

* Reduce cloud storage costs by optimizing [dataset storage usage](/cloud/admin/data-administration/datasets#set-limits-on-dataset-usage).


# Upgrades & maintenance
Source: https://docs.domino.ai/cloud/admin/operations/upgrades-maintenance

How Domino Cloud delivers product updates, and what a platform administrator needs to track or act on.

Domino Data Lab operates the Domino Cloud control plane as part of the managed service, delivering new features, fixes, and platform changes continuously as [Cloud Releases](/release-notes/cloud-release). Domino Cloud administrators can't trigger a service restart, search the configuration, or access the underlying infrastructure (see [Domino Cloud](/cloud/admin/architecture-and-deployment/domino-cloud#limitations)), so day-to-day control plane maintenance isn't something you schedule or perform.

If your deployment includes a remote Data Plane, such as a [Nexus hybrid](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture) region, you're responsible for keeping it in sync with the control plane.

## How it works

* **Cloud Release**: the unit of delivery for Domino Cloud. Cloud Releases ship on a regular cadence and are documented in the [Domino Cloud release notes](/release-notes/cloud-release). When a release includes something a platform administrator needs to act on, it's called out in a "Notes for platform administrators" section.
* **Control plane**: the core Domino services (UI, API, orchestration, metadata stores). In Domino Cloud, Domino operates the control plane; admins can't trigger a restart or search its configuration.
* **Data Plane**: a compute location where your workloads run. A Data Plane must stay within a supported version range of the control plane. Most Data Plane upgrades run from the admin portal; Domino Support performs the upgrade if the UI action isn't available for your Data Plane.

## Use upgrades and maintenance information

<Steps>
  <Step title="Track upcoming changes">
    Check the [Domino Cloud release notes](/release-notes/cloud-release) for each release, especially the "Notes for platform administrators" section. Most releases require no action; when one does, it's called out there.
  </Step>

  <Step title="Keep any Data Planes in sync">
    If you run a remote Data Plane, confirm it's within a [compatible version](/cloud/admin/platform-configuration/manage-data-planes#version-compatibility) of the control plane, then follow [Upgrade a Data Plane](/cloud/admin/platform-configuration/manage-data-planes#upgrade-a-data-plane).
  </Step>

  <Step title="Plan around the maintenance window">
    Before upgrading a Data Plane, notify users in the affected regions. Ongoing executions can typically continue through the upgrade, but stopping them first avoids lost work if reconciliation surfaces an issue.
  </Step>
</Steps>

<Note>
  **Best practice:** upgrade the control plane's supported version before any Data Plane. Running a Data Plane newer than the control plane isn't supported.
</Note>

## Related

* [Domino Cloud release notes](/release-notes/cloud-release)
* [Domino Cloud](/cloud/admin/architecture-and-deployment/domino-cloud), for what Domino operates on your behalf
* [Manage Data Planes](/cloud/admin/platform-configuration/manage-data-planes)
* [Kubernetes compatibility](/cloud/admin/architecture-and-deployment/kubernetes-compatibility)


# Domino QMS overview
Source: https://docs.domino.ai/cloud/admin/quality-management-system

How Domino's ISO 9001-certified Quality Management System assures customers of consistent quality, traceability, and support for regulated workflows, including life sciences validation requirements.

## Overview

Domino Data Lab operates a Quality Management System (QMS) certified to **ISO 9001:2015**, the International Organization for Standardization (ISO) standard for quality management. This certification reflects a formal, independently audited framework governing how we design, build, test, deploy, and support Domino's enterprise AI platform, with an emphasis on consistency, traceability, and continuous improvement.

Our QMS works alongside our information security program (ISO 27001:2022) and System and Organization Controls (SOC 2) Type II compliance to give customers a complete picture of how Domino manages quality, security, and risk. Certificates, audit reports, and supporting documentation are available on the [Domino Trust Center](https://trust.domino.ai/).

## Impact on customers

An ISO 9001-certified QMS translates into tangible assurances for customers running production workloads on Domino:

* **Consistent product quality.** Documented processes for development, testing, and release management reduce variability and defects across product updates.
* **Change control and traceability.** Formal change management processes ensure that platform changes are reviewed, tested, and recorded, supporting audit and validation needs.
* **Continuous improvement.** Regular internal audits, management reviews, and corrective action processes drive ongoing refinement of our engineering and support practices.
* **Vendor and third-party oversight.** The same quality discipline extends to how we evaluate and manage suppliers and subprocessors who touch the platform.
* **Independent verification.** Certification is maintained through periodic external audits, not just internal attestation, giving customers third-party assurance rather than self-reported claims.

Together, these controls reduce operational risk for customers and support their own internal quality and compliance obligations when Domino is part of a regulated workflow.

## Supporting life sciences customers

Life sciences organizations operate under some of the most stringent quality and validation requirements of any industry, and Domino's QMS is structured to support that context directly:

* **GxP-aligned processes.** Our documented change management, release validation, and traceability practices align with the principles underlying GxP and GAMP 5 expectations for computerized systems.
* **Regulatory framework support.** Domino's quality and security programs support customer compliance efforts under frameworks including 21 CFR Part 11, EudraLex Annex 11, and WHO Annex 5.
* **Model and data lineage.** Continuous monitoring and automated tracking of model lineage support the traceability and reproducibility requirements common in regulated life sciences environments.
* **Collaborative validation support.** Our Quality and Compliance team works directly with life sciences customers to provide the artifacts and evidence needed for their computer system validation (CSV) and internal audit programs.

For certificates, audit reports, and additional compliance documentation, visit the [Domino Trust Center](https://trust.domino.ai/).


# Administer identity and access control
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/auth-identity-access-control

The administrator's half of identity and access control on Domino Cloud: the SAML identity provider connection, the global roles that carry administrative capability, user provisioning, automation credentials, and the records that show who has access.

Domino authorizes every identity in two layers: a global role that applies across the whole deployment, and permissions on each individual Project, Dataset, Data Source, App, and Compute Environment. You own the first layer and the identity infrastructure behind it. Entity owners grant the second. [Sharing and access](/cloud/platform-capabilities/features/collaboration/sharing-and-access) covers that second layer from a practitioner's point of view.

Connect your identity provider first. Once it authenticates everyone, the rest of this page is what you do for the cases it does not cover.

## Connect an identity provider

Domino Cloud ships with a pre-built Security Assertion Markup Language (SAML) 2.0 single sign-on (SSO) provider, disabled by default. Enabling it is a joint task: you configure the application on your provider, and Domino Support completes the connection on the cluster.

<Steps>
  <Step title="Configure the SAML application in your provider">
    Point the application at `https://{your-cluster-name}.domino.tech/auth/realms/DominoRealm/broker/saml/endpoint`, which serves as both the recipient and the destination of the sign-on process. Domino identifies each user by email address and derives the username from it, so send an `email` attribute whenever the NameID is not an email address. Add `firstName` and `lastName` to spare users from entering their details at first sign-in.
  </Step>

  <Step title="Decide whether the provider also drives roles">
    To manage Domino roles in your provider, set a multi-valued `domino-system-roles` attribute on each user. Domino Cloud accepts `Practitioner`, `CloudAdmin`, `Librarian`, and `ProjectManager`. Comma-separated values are not supported, so pass each role as its own value.
  </Step>

  <Step title="Send the connection details to Domino Support">
    Domino Support needs the single sign-on service URL, the certificate for signature validation, whether roles are integrated, and the label you want on the login button. Domino Cloud does not support provider-initiated sign-in, so users start from the cluster login page either way.
  </Step>
</Steps>

<Warning>
  Role synchronization makes `domino-system-roles` the only source of a user's roles. Roles you set in the Admin application are overwritten the next time that user signs in, and a user whose attribute omits **Practitioner** loses it, along with the ability to create Projects, start Workspaces, and run Jobs. Send every role each user needs, including the ones Domino would otherwise assign by default.
</Warning>

[Authentication and SSO](/cloud/admin/identity-and-access/sso-for-domino-cloud) has the full attribute reference and the Okta and Microsoft Entra ID specifics, including how to drive `domino-system-roles` from provider groups.

## Assign global roles

A global role grants capabilities across the whole deployment, independent of any Project. Every user starts with **Practitioner**, which covers creating Projects, running Jobs, starting Workspaces, and publishing Apps. Roles are additive, so a user holding several gets the union of their permissions.

**CloudAdmin** is the administrative role on Domino Cloud. Assign roles in the Admin application under **Users**, or map them from your identity provider. Only a CloudAdmin can grant roles.

<Note>
  **CloudAdmin** cannot manage configuration records or feature flags, and **SysAdmin** is not available on Domino Cloud, so a change that depends on either goes to Domino Support.
</Note>

<Warning>
  Changing a user's roles revokes every Personal Access Token they hold. Their scripts, notebooks, and scheduled Jobs stop authenticating until they create replacement tokens, so tell the user before you change their roles.
</Warning>

| Role | What it administers | Where it stops |
| :- | :- | :- |
| CloudAdmin | Users and their roles, Admin application settings, global Compute Environments, Datasets and snapshots, Dataset permissions, usage reports, notifications, cost monitoring, and cost budgets and alerts. Can also delete Workspaces, stop executions, and archive Projects, which is what the deactivation steps below need | No configuration records, feature flags, email configuration, search index, API keys, MongoDB commands, Kubernetes dashboard, or Nucleus restart. Manages Hardware Tiers only when `com.cerebro.domino.dominoCloud.cloudAdmin.canManageHwTiers` is `true` |
| ProjectManager | Organizations and Project tags | Carries owner-level access to every Project owned by a fellow organization member, so treat it as a highly privileged role rather than a light delegation |
| GovernanceAdmin | Evidence templates for Governance policies | Views only for bundles, approvals, policy overviews, and audit events |
| LimitedAdmin | Admin application settings, feature flags, Hardware Tiers, usage reports, notifications, and executions | No user management, no configuration records, no global Compute Environments, and no permissions to Projects or data |
| SupportStaff | Compute-related functionality: starting and stopping Jobs, stopping Workspaces, Apps, and Domino endpoints, archiving Projects, and Datasets in the Admin application | Cannot create Projects, start Workspaces, or edit files |
| ReadOnlySupportStaff | Views compute-related configuration, Project settings, and Compute Environments | Read-only, and cannot start or stop anything |
| Librarian | The Project library, Project settings, and executions | No Admin application access |
| LicenseReviewer | License-related content and usage reports | No permissions to Projects or data |

[Work with roles](/cloud/admin/identity-and-access/manage-users/roles) has the full permission matrix for every role, including the ones with no administrative reach. To hand a user a narrower Practitioner, assign the [Data Analyst role](/cloud/admin/identity-and-access/manage-users/roles/data-analyst): the same access minus model features, Hardware Tier edits, and Compute Environment management. Enabling it in the first place takes a configuration record and a restart of Domino services, neither of which CloudAdmin can do.

### Project creation and ownership

No role grants Project creation on its own. Any **Practitioner** can create a Project, and whoever creates it becomes the Project owner. Ownership carries the authority that matters at the Project level: only the owner can change a Project's visibility or its name, and only the owner can transfer ownership or archive it from inside the Project.

Inviting a collaborator is the owner's action or a contributor's, not an administrative one. What you control is who holds Practitioner at all, who holds ProjectManager, and where ownership lands when someone leaves: CloudAdmin can archive a Project and transfer it to a new owner from the Admin application. See [Collaborate on Projects](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#collaborator-permissions) for what each collaborator role can do.

## Provision and deactivate users

Configure email services before you create users; user creation depends on them. Add a user from the Admin application by email address and assign roles at the same time. The user receives a notification and an invitation, and must follow the setup link to finish their profile.

Deactivating a user deletes all of their Personal Access Tokens for you, but leaves their Workspaces, executions, and Projects behind, so work through the cleanup in order.

<Steps>
  <Step title="Deactivate the account">
    Find the user under **Users** in the Admin application and deactivate them. Deactivation is reversible and you reactivate the same way, but the deleted tokens do not come back; the user creates replacements.
  </Step>

  <Step title="Stop and delete their Workspaces">
    Search **Reports** > **Workspaces** for Workspaces the user owns, and delete them.
  </Step>

  <Step title="Record what has to keep running">
    **Reports** > **Executions** lists the Domino endpoints and Apps the user owns. Note the ones that must stay available.
  </Step>

  <Step title="Transfer or archive their Projects">
    From **Reports** > **Projects**, transfer each Project to its new owner or archive it, then restart whatever you recorded in the previous step.
  </Step>
</Steps>

<Warning>
  Transferring a Project that contains running Domino endpoints, Apps, or other executions stops those executions until someone restarts them under the new owner. Record what is running before you transfer, and restart it afterward.
</Warning>

## Credentials for automation

Give shared automation a Domino Service Account rather than a person's Personal Access Token. A service account's tokens belong to no individual, so deactivating a departing employee never breaks a pipeline.

Only administrators create service accounts, and only through the Domino API. A service account inherits the roles of whoever created it, so check and adjust them before you issue a token: each token carries the roles the account held at the moment of creation, and later role changes do not reach tokens already issued. Tokens expire after four months by default, and you revoke them individually or by deactivating the account.

Personal Access Tokens stay with their owners. The Personal Access Tokens panel lists every token in the deployment with its owner, creation date, last use, and status, and you can revoke or delete any of them, in bulk if you need to. You cannot issue a token for another user, and you cannot read a token value: only its creator can retrieve it, only at the moment of creation.

## Audit who has access

| Review | Where |
| :- | :- |
| Who has an account, which roles they hold, whether they are active, and how much they run | [User activity report](/cloud/admin/identity-and-access/manage-users/run-user-activity-report), readable in the Admin application and downloadable as a comma-separated values (CSV) file |
| Every API credential in the deployment, its owner, and its last use | [Personal access tokens](/cloud/admin/identity-and-access/manage-users/personal-access-tokens) panel |
| Permission changes and Dataset events inside Projects | [Project logs](/cloud/admin/operations/audit-logs/monitor-project-logs) |
| Data Source creation, access, deletion, and permission changes | [Data Source logs](/cloud/admin/operations/audit-logs/monitor-data-source-logs) |

When a record of who approved something matters as much as the change itself, [require an e-signature](/cloud/admin/operations/audit-logs/esignature) on critical actions.

## Related

* [Sharing and access](/cloud/platform-capabilities/features/collaboration/sharing-and-access): how practitioners grant access to what they own, and how Project and entity permissions stack.
* [Work with roles](/cloud/admin/identity-and-access/manage-users/roles): every role and the exact actions each one permits.
* [Authentication and SSO](/cloud/admin/identity-and-access/sso-for-domino-cloud): SAML attributes, provider specifics, and role integration.
* [Manage organizations](/cloud/platform-capabilities/features/collaboration/manage-organizations): group users so Project access follows the team.
* [Reset password](/cloud/admin/identity-and-access/manage-users/password-reset): restore access for a user locked out of a local account.
* [Protecting tokens and API keys](/cloud/admin/security-and-compliance/protecting-tokens-api-keys): storage and rotation practices for credentials.


# Work with CA certificates
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/ca-certificates/index

Configure Istio to use Domino-provided or existing service meshes for intra-cluster encryption, then set up and update custom CA certificates.

Intra-cluster encryption in transit is implemented (if enabled) through a deployed service mesh called [Istio](https://istio.io). At installation, Domino can deploy Istio for Domino use only, or you can configure Domino to leverage an existing deployed Istio mesh on the Kubernetes cluster (potentially shared with other applications).

The following topics explain how to work with Certificate Authority (CA) certificates:

* [Set up custom CA certificates.](/cloud/admin/security-and-compliance/ca-certificates/set-up-custom-ca-certs)

* [Update custom CA certificates.](/cloud/admin/security-and-compliance/ca-certificates/update-custom-ca-certs)


# Set up custom Certificate Authority (CA) certificates
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/ca-certificates/set-up-custom-ca-certs

Set up Istio custom CA certificates by creating the CA certificates Kubernetes secret from X.509 PEM root, key, intermediate, and chain files.

<Warning>
  When Domino deploys Istio mesh, out of the box, Istio provides scalable [identity and X.509 certificate management](https://istio.io/latest/docs/concepts/security/#pki) for use with mTLS encryption, including periodic certificate and key rotation.

  Because all encrypted communication is internal,  these certificates are not exposed or required for communication to any external services, such as web browsers and clients.

  Domino acknowledges that enterprise policies might mandate the use of corporate public key infrastructure (PKI) and necessitate the use of certificate authority (CA) certificates.
</Warning>

<Note>
  All certificates must be X.509 PEM format and  keys must be passwordless.
</Note>

1. Obtain the certificate files, noting the file names for use in future commands.

   | Filename | Description |
   | - | - |
   | `root-cert.pem` | Root CA certificate for PKI. |
   | `ca-cert.pem` | Intermediate CA certificate from root CA. This is the Istio CA certificate. |
   | `ca-key.pem` | Private key for Istio CA certificate. |
   | `cert-chain.pem` | Full chain from `ca-cert.pem` to `root-cert.pem` (including both certificates). |

2. Use the code in the following sample.

   ```shell theme={null}
   # Concatenate all certificates into a certificate chain file
   # Assuming `N` intermediate certificates denoted as `int-ca-<i>.pem`, with `i = {1,...,N}`
   cat ca-cert.pem int-ca-1.pem ... int-ca-N.pem root-cert.pem > cert-chain.pem

   # Create new kubernetes secret with CA certificate files
   kubectl -n istio-system create secret generic cacerts
       --from-file=./ca-cert.pem
       --from-file=./ca-key.pem
       --from-file=./root-cert.pem
       --from-file=./cert-chain.pem
   ```

   In a new standard Domino install, if you follow the install process, the `fleetcommand-agent` (Domino installer) automatically picks up the [secret](https://kubernetes.io/docs/concepts/configuration/secret/) that you created and Istio uses the configured certificates.

   In an existing Domino installation, you must restart all the pods.


# Update custom Certificate Authority (CA) certificates
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/ca-certificates/update-custom-ca-certs

Update Istio custom CA certificates by recreating the CA certificates secret and restarting the required Istio or Domino pods.

This section describes how to update the custom CA certificates that Istio uses for intra-cluster encryption in transit in some scenarios. You must always create a new full chain certificate file (`cert-chain.pem`).

<Tip>
  Domino recommends backing up existing certificates and keys before updating new ones.
</Tip>

**Scenario 1: No changes were made to the private key and common name**

This assumes only `ca-cert.pem` is updated.

Create a secret with the new files and restart the Istio daemon (`istiod`).

```shell theme={null}
# Delete existing secret with CA cert files
kubectl -n istio-system delete secret cacerts

# Create new secret with CA cert files
kubectl -n istio-system create secret generic cacerts
    --from-file=./ca-cert.pem
    --from-file=./ca-key.pem
    --from-file=./root-cert.pem
    --from-file=./cert-chain.pem

# Restarting all istiod pods
kubectl -n istio-system delete po -l app=istiod
```

**Scenario 2: Updates were made to the private key, common name, or upstream certificates**

Changes made to the private key, common name (CN) or upstream certificates require that you recreate the `cacerts` secret and restart the Istio daemon.

```shell theme={null}
# Delete existing secret with CA cert files
kubectl -n istio-system delete secret cacerts

# Create new secret with CA cert files
kubectl -n istio-system create secret generic cacerts
    --from-file=./ca-cert.pem
    --from-file=./ca-key.pem
    --from-file=./root-cert.pem
    --from-file=./cert-chain.pem

# Full restart for all Istio pods
for NS in istio-system domino-platform domino-compute; do
    kubectl -n $NS get po --no-headers -o custom-columns=name:metadata.name | xargs kubectl -n $NS delete po
done
```


# Control plane security
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/control-plane-security

Secure Domino Nexus control plane ingress for Vault, RabbitMQ, and the Docker Registry with private networking, TLS, DNS, and Istio.

This guide outlines important security considerations that must be taken when configuring ingress for control plane services in [Domino Nexus](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture). Depending on the customer’s requirements and particular Domino configuration, the exact configuration of this ingress may vary significantly. This guide therefore discusses these security principles in the abstract, without describing solutions for a particular infrastructure vendor.

## Architecture

Domino Nexus control planes host several important services which must be accessible from data plane clusters:

**Domino API** - This is the API which is used by the Domino CLI, UI, and now data planes. Ingress is already configured on the main endpoint by which users access Domino. This endpoint should be routable from data plane clusters. Because this is already configured, this guide does not discuss it further.

**Vault** - Data plane clusters use HashiCorp’s Vault to authenticate with the control plane, encrypt secrets in transit, and authenticate with RabbitMQ. Communication with Vault happens over HTTPS on port 8200.

**RabbitMQ** - The primary communication channel between the Domino control plane and the data plane is based on RPC over RabbitMQ. Additionally, data planes publish execution state back to the control plane over RabbitMQ. This communication occurs using AMQP and RabbitMQ Stream protocols on ports 5672 and 5552.

**Docker Registry** - If you are using Domino’s internal Docker Registry then data plane Kubelets must be able to access the registry over HTTPS on port 443. This is used to pull environment images for user executions.

Vault and RabbitMQ in particular are sensitive services. Authentication and authorization for these services is tightly controlled. These services should not be generally available to the public Internet, to reduce the risk of compromised credentials or vulnerabilities in the services themselves.

<img alt="Control plane security" />

## Load balancers

Load balancers are typically used for Kubernetes ingress to route traffic from a stable DNS host to the variable set of cluster nodes hosting the service. Most infrastructure providers are able to configure Layer 4 (Network) and / or Layer 7 (Application) load balancers. Application load balancers can be used for Vault and Docker Registry because they use HTTPS, however a network load balancer is required for RabbitMQ because it is using AMQP.

### TLS termination

Data planes *must* communicate with the control plane services using TLS. These services do not support serving TLS connections themselves (unless Istio is in use, see below), and so load balancers must perform TLS termination. If your company issues non-public certificates, these can be used here, however see Domino’s administrator documentation for guidance on creating data planes when non-public certificates are in use.

### Network access

Load balancers should not be accessible from the public Internet. This can be achieved in two ways:

* The recommended way is to configure *internal* load balancers, which sit in a private subnet, do not have a public IP address, and are only accessible using private networking such as VPN or peering. Even with internal load balancers, you should limit source IP ranges to subnets which you expect to host data plane services.

* If your company does not have private networking between Nexus clusters, you *must* restrict the network CIDRs which can access your load balancers. This involves identifying the IP addresses for NAT gateways used by data plane services, and limiting load balancer access to these source ranges, either through load balancer configuration or network security groups.

### DNS

Data plane services must be able to reach load balancers using a stable DNS host.

Most cloud providers publish *public* DNS records for both *public* and *private* load balancers. This allows you to use public DNS to resolve private load balancers.

If you are hosting your own load balancers (such as on-premises), you must configure a DNS record (either public or private) to resolve to the load balancer’s IP. If the IP changes over time, you must keep the DNS record updated. If you are using private DNS, ensure that pods in data plane clusters are able to resolve such hosts.

## Network policies

In traditional Domino deployments, access to RabbitMQ and Vault by services inside the cluster is tightly controlled. Allowing ingress to these services from outside the cluster while limiting access from services inside the cluster is nontrivial because many types of access control operate at the network level and are not aware of cluster application level semantics.

In the architecture pictured above, it is recommended that load balancers are placed in a *separate* subnet from cluster nodes and pods. This allows network policies to be defined which only allow traffic from this subnet, and not other pods in the cluster. Some load balancer types forward client IP addresses (used for evaluating network policies). In this case you can allow only address space known to be used by data plane sources.

In addition to network policies, you configure security groups between load balancers and cluster nodes in accordance with best practice.

## Istio

Communication between load balancers and backend services is often not encrypted in standard Kubernetes configurations. If you are not using Istio in your cluster, such traffic is already not encrypted between local cluster pods and these services. To encrypt traffic between load balancers and Pods, you must have Istio enabled in your deployment.

If your company requires encryption in transit to be enabled, and you are running Istio in your Domino deployment, then you must take steps to allow the load balancer to connect to these services without using mTLS, while continuing to use TLS.

One way this can be accomplished is by using Istio’s Gateway. In this architecture, load balancers instead route traffic to Istio’s Gateway service, which originates mTLS and allows traffic to enter the service mesh. If you go this route, ensure that the load balancer configured by Istio conforms with the guidance above, and uses TLS to communicate with the Istio Gateway service backend.

Another option is to configure [Sidecar TLS termination](https://istio.io/latest/docs/tasks/traffic-management/ingress/ingress-sidecar-tls-termination/). With this method, *mutual* TLS is disabled for the services and ports that we wish to expose. The sidecar is configured to serve custom TLS certificates to the load balancer to provide encryption in transit between load balancer and service. Domino only relies on Istio for encryption in transit, not service identity, so *mutual* TLS is not necessarily required.

In either case, you can chose to have the Istio Gateway or Sidecar serve the external certificates for the control plane, or to leverage TLS termination in the load balancer.


# Add the PEM bundle to the ConfigMap
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/custom-certificates/add-pem-bundle

Add a PEM certificate bundle to the custom certificates section of the installer configuration so the installer creates the Domino certificate ConfigMap.

During Domino installation, you can add the contents of PEM bundle to the `domino.yml` configuration file with the top-level key `custom_certificates`.

The following is an example:

```
...
custom_certificates: |
  -----BEGIN CERTIFICATE-----
  MIICKzCCAbGgAwIBAgIKe3G2gla4EnycqDAKBggqhkjOPQQDAzBaMQswCQYDVQQG
  EwJVUzETMBEGA1UECxMKZW1TaWduIFBLSTEUMBIGA1UEChMLZU11ZGhyYSBJbmMx
  IDAeBgNVBAMTF2VtU2lnbiBFQ0MgUm9vdCBDQSAtIEMzMB4XDTE4MDIxODE4MzAw
  MFoXDTQzMDIxODE4MzAwMFowWjELMAkGA1UEBhMCVVMxEzARBgNVBAsTCmVtU2ln
  biBQS0kxFDASBgNVBAoTC2VNdWRocmEgSW5jMSAwHgYDVQQDExdlbVNpZ24gRUND
  IFJvb3QgQ0EgLSBDMzB2MBAGByqGSM49AgEGBSuBBAAiA2IABP2lYa57JhAd6bci
  MK4G9IGzsUJxlTm801Ljr6/58pc1kjZGDoeVjbk5Wum739D+yAdBPLtVb4Ojavti
  sIGJAnB9SMVK4+kiVCJNk7tCDK93nCOmfddhEc5lx/h//vXyqaNCMEAwHQYDVR0O
  BBYEFPtaSNCAIEDyqOkAB2kZd6fmw/TPMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMB
  Af8EBTADAQH/MAoGCCqGSM49BAMDA2gAMGUCMQC02C8Cif22TGK6Q04ThHK1rt0c
  3ta13FaPWEBaLd4gTCKDypOofu4SQMfWh0/434UCMBwUZOR8loMRnLDRWmFLpg9J
  0wD8ofzkpf9/rdcw0Md3f76BB1UwUCAU9Vc4CqgxUQ==
  -----END CERTIFICATE-----
  -----BEGIN CERTIFICATE-----
  MIICTjCCAdOgAwIBAgIKPPYHqWhwDtqLhDAKBggqhkjOPQQDAzBrMQswCQYDVQQG
  EwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNo
  bm9sb2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0g
  RzMwHhcNMTgwMjE4MTgzMDAwWhcNNDMwMjE4MTgzMDAwWjBrMQswCQYDVQQGEwJJ
  TjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNobm9s
  b2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0gRzMw
  djAQBgcqhkjOPQIBBgUrgQQAIgNiAAQjpQy4LRL1KPOxst3iAhKAnjlfSU2fySU0
  WXTsuwYc58Byr+iuL+FBVIcUqEqy6HyC5ltqtdyzdc6LBtCGI79G1Y4PPwT01xyS
  fvalY8L1X44uT6EYGQIrMgqCZH0Wk9GjQjBAMB0GA1UdDgQWBBR8XQKEE9TMipuB
  zhccLikenEhjQjAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAKBggq
  hkjOPQQDAwNpADBmAjEAvvNhzwIQHWSVB7gYboiFBS+DCBeQyh+KTOgNG3qxrdWB
  CUfvO6wIBHxcmbHtRwfSAjEAnbpV/KlK6O3t5nYBQnvI+GDZjVGLVTv7jHvrZQnD
  +JbNR6iC8hZVdyR+EhCVBCyj
  -----END CERTIFICATE-----
```

The installer copies the contents of the custom certificates into the `domino-custom-certificates` ConfigMap.


# Apply custom certificates
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/custom-certificates/apply-custom-certs

Apply a new custom certificate bundle by finding and restarting the Domino pods and services that mount the generated certificates.

To apply the new certificate bundle, restart the Domino services that use custom certificates.

<Warning>
  You might interrupt user workflows by restarting pods or deleting services. If downtime is acceptable in your environment, you can use the following command to delete all pods. This replaces the other commands in this section. Update the `namespace` to match the one used in your Domino installation.

  ```shell theme={null}
  kubectl get pods -n <namespace> -ojson | jq -r '.items[] | select(.spec.volumes[]? | select(.configMap.name == "domino-generated-certificates")) | .metadata.name' | xargs kubectl delete pod -n <namespace> --wait=false
  ```
</Warning>

1. Delete a pod to restart a service:

   ```shell theme={null}
   kubectl delete pod -n <namespace> <pod name>
   ```

2. Find the pods and services that use custom certificates. Update the `namespace` in each command to match the one used in your Domino installation.

   ```shell theme={null}
   kubectl get pods -n <namespace> -ojson | jq -r '.items[] | select(.spec.volumes[]? | select(.configMap.name == "domino-generated-certificates")) | .metadata.name'
   ```


# Domino-custom-certificates ConfigMap
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/custom-certificates/domino-custom-cert-configmap

Configure the Domino custom certificates ConfigMap with a PEM bundle, then reference its certificates from user sessions and applications.

Domino checks for `domino-custom-certificates` ConfigMap in the Kubernetes cluster’s `default` namespace. This ConfigMap must have a key named `bundle` with plaintext data of a certificate bundle in PEM format. If the bundle exists, then Domino uses certificates from this bundle to connect to the external services.

Domino supports the following certificate types:

Custom Certificate Authority (CA)\
Certificates that certify the other certificates issued under this authority.

Self-signed certificates\
Certificates that do not have a reference to the authority signing them.

The bundle is formatted as a series of concatenated certificates in PEM format. You must have the line breaks around the lines:

```shell theme={null}
-----BEGIN CERTIFICATE—--
```

and

```shell theme={null}
-----END CERTIFICATE—--
```

The bundle must contain all the certificates that you would typically use to connect to the private services, including intermediate and root certificates.

Domino includes public certificates, such as DigiCert root certificates, by default so you do not have to include them. Duplicate certificates do not cause any issues.

For user executions, all Domino certificates, including public and customer-provided, are stored in `/etc/ssl/certs/domino-custom`.

## Use custom certificates

If you must use custom certificates in a user session, such as to connect to internal Git servers from a Workspace command line, manually reference the certificates in `/etc/ssl/certs/domino-custom`. The following are some ways that you can do this:

* Copy the certs from `/etc/ssl/certs/domino-custom` to `/etc/ssl/certs` in a compute environment pre-run script.

* Add a Java truststore option such as: `-Djavax.net.ssl.trustStore=/etc/ssl/certs/domino-custom/cacerts.p12 -Djavax.net.ssl.trustStoreType=PKCS12 -Djavax.net.ssl.trustStorePassword=changeme`

* Add application-specific configurations such as setting the `GIT_SSL_CAINFO` variable for Git or passing the `--cacert` option for Curl.

  You can also set these options at runtime or modify the relevant configuration files in the compute environment.


# Work with custom certificates
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/custom-certificates/index

Configure Domino to trust custom certificates for private registries, Git servers, S3 services, identity providers, and Data Sources.

You can configure Domino to connect to services that use custom certificates that are external to the Domino cluster. In addition to public services like AWS S3, you might want to use private services in your security domain that are secured with custom certificates or a custom certificate authority.

The following are examples of private services:

* Docker registry

* Git server

* S3 service

* LDAPs

* OIDC

* Data sources

Domino recommends that you add certificates for private services to the installation configuration file (`domino.yml`) as described in this topic. This ensures that Domino propagates the certificates throughout the system and maintains them, even when you upgrade Domino.

<Note>
  If you add custom certificates to `etc/ssl/certs` in a [compute environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments), the system overwrites them at runtime. If you cannot use Domino’s recommended method or must configure the certificates for a specific compute environment, see [Configure Certificates for A Specific Compute Environment](https://tickets.dominodatalab.com/hc/en-us/articles/6665122897428-Configure-Certificates-for-A-Specific-Compute-Environment).
</Note>

The topics in this section explain how to work with custom certificates:

* [Domino-custom-certificates ConfigMap.](/cloud/admin/security-and-compliance/custom-certificates/domino-custom-cert-configmap)

* [Add the PEM bundle to the ConfigMap.](/cloud/admin/security-and-compliance/custom-certificates/add-pem-bundle)

* [Update the ConfigMap.](/cloud/admin/security-and-compliance/custom-certificates/update-configmap)

* [Apply custom certificates.](/cloud/admin/security-and-compliance/custom-certificates/apply-custom-certs)

* [Upgrade Domino.](/cloud/admin/security-and-compliance/custom-certificates/upgrade-domino)


# Update the ConfigMap
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/custom-certificates/update-configmap

Update custom certificates after installation by recreating the Domino custom certificates ConfigMap with a concatenated PEM bundle.

After Domino is installed and running, you can recreate the `domino-custom-certificates` ConfigMap to update it. To do this, run the following commands:

```shell theme={null}
kubectl delete configmap domino-custom-certificates
kubectl create configmap domino-custom-certificates --from-file=bundle
```

where `bundle` is the new certificate bundle in concatenated PEM format.


# Upgrade Domino
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/custom-certificates/upgrade-domino

Provide custom certificates in the installer configuration during an upgrade, or preserve and migrate existing certificate ConfigMaps when needed.

When upgrading a Domino deployment to a new version, you can use the `custom_certificates` key in `domino.yml` to provide a new certificate bundle.

If the key is not yet specified, the installer does the following:

* Any existing `domino-custom-certificates` bundle continues to be used, and can still be updated independently of the installer.

* If there is no existing `domino-custom-certificates` ConfigMap, `custom_certificates` is not set, and the legacy `domino-executor-certificates` ConfigMap exists, it is copied and upgraded to the new format/name. If you are using the `custom_certificates` field to configure certificate bundles and the `domino-executor-certificates` ConfigMap exists, make sure to copy the relevant contents of the `domino-executor-certificates` ConfigMap to the `domino.yml` as the ConfigMap is not used.


# Encryption & data security
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/encryption-data-security

How Domino encrypts data in transit at the cluster ingress and between services, and at rest for Project files and secrets.

Domino encrypts data both in transit within the cluster and at rest, using a mix of platform defaults and cloud-provider-managed encryption.

## Encryption in transit

### Domino web application

Domino serves its web application and API over HTTPS and terminates TLS at the ingress load balancer, using an SSL certificate applied to the load balancer and an NGINX ingress that enforces `TLSv1.2` and `TLSv1.3` with a FIPS-validated OpenSSL build.

### Intra-cluster communication

Intra-cluster encryption in transit is implemented, if enabled, through a deployed service mesh called [Istio](https://istio.io). At installation, Domino can deploy Istio for Domino use only, or you can configure Domino to use an existing Istio mesh on the Kubernetes cluster. See [Control plane security](/cloud/admin/security-and-compliance/control-plane-security) and [Work with CA certificates](/cloud/admin/security-and-compliance/ca-certificates) for configuration details.

## Encryption at rest

### Project files

Domino supports server-side encryption with customer-provided keys ([SSE-C](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html)) for Amazon S3, and [EBS file system encryption](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html) using the AES-256 algorithm on Elastic Block Store. Domino also supports default encryption keys for Amazon S3, Azure File Storage, and Google Cloud Filestore. Domino does not provide pre-write encryption for `nfs-client-provisioner` volumes. See [Manage Project file storage](/cloud/admin/data-administration/project-file-storage) for details.

### Secrets

Domino stores secrets in an instance of HashiCorp Vault. By default, Vault stores the following secrets encrypted at rest without additional configuration:

* User environment variables
* User API keys
* Data Source access secrets
* Project environment variables

## Related

* [Manage Project file storage](/cloud/admin/data-administration/project-file-storage)
* [Work with CA certificates](/cloud/admin/security-and-compliance/ca-certificates)
* [Control plane security](/cloud/admin/security-and-compliance/control-plane-security)


# Security at Domino
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/index

Use Domino security controls for identity, networks, containers, encryption, monitoring, and credentials, and access compliance records in Trust Center.

Domino is designed to give enterprise teams a platform they can trust with sensitive data, models, and intellectual property. Security is layered across the platform: identity and access, network and control plane, container and Kubernetes hardening, encryption, monitoring, and vulnerability management, all backed by a formal quality & compliance program that is continuously attested to in the [Domino Trust Center](https://trust.domino.ai/).

This page summarizes what the **Security & Compliance** section covers and takes a deeper look at what the Trust Center provides.

## What's in this section

<CardGroup>
  <Card title="Authentication, identity, and access control" icon="user-shield" href="/cloud/admin/security-and-compliance/auth-identity-access-control">
    Integrate with enterprise identity providers, enforce role-based access control (RBAC), and manage the user role hierarchy across Domino.
  </Card>

  <Card title="Control plane security" icon="shield-halved" href="/cloud/admin/security-and-compliance/control-plane-security">
    Guidance for securing ingress to Domino Nexus control plane services across on-premises, cloud, and hybrid architectures.
  </Card>

  <Card title="Encryption & data security" icon="lock" href="/cloud/admin/security-and-compliance/encryption-data-security">
    Encryption in transit and at rest, key management, and how Domino protects project files, datasets, and credentials.
  </Card>

  <Card title="Container & Kubernetes security" icon="cubes" href="/cloud/admin/security-and-compliance/pod-security-policies">
    Pod security contexts, least-privilege container configuration, and the Kubernetes-level hardening Domino applies.
  </Card>

  <Card title="Custom certificates" icon="certificate" href="/cloud/admin/security-and-compliance/custom-certificates">
    Configure Domino to trust internal or third-party services that present custom TLS certificates.
  </Card>

  <Card title="CA certificates" icon="file-certificate" href="/cloud/admin/security-and-compliance/ca-certificates">
    Manage the certificate authority bundle used by Domino, including Istio-based intra-cluster encryption in transit.
  </Card>

  <Card title="Platform monitoring" icon="chart-line" href="/cloud/admin/security-and-compliance/platform-monitoring">
    Audit logs and system telemetry that give admins and security teams visibility into user activity and cluster health.
  </Card>

  <Card title="Vulnerability management" icon="bug" href="/cloud/admin/security-and-compliance/vulnerability-management">
    How Domino identifies, triages, and remediates vulnerabilities in the platform and its dependencies.
  </Card>

  <Card title="Protect tokens and API keys" icon="key" href="/cloud/admin/security-and-compliance/protecting-tokens-api-keys">
    Best practices and Domino features for storing and using API keys, personal access tokens, and credentials without exposing them.
  </Card>

  <Card title="Reporting security concerns" icon="triangle-exclamation" href="/cloud/admin/security-and-compliance/reporting-security-concerns">
    How to report a vulnerability or security concern to the Domino Data Lab security team.
  </Card>

  <Card title="GDPR requests" icon="scale-balanced" href="/cloud/admin/security-and-compliance/submit-gdpr-requests">
    Submit General Data Protection Regulation requests such as user deletion or data access.
  </Card>

  <Card title="Trust Center" icon="shield-check" href="https://trust.domino.ai/">
    Live portal for Domino's certifications, audit reports, security posture, and compliance documentation.
  </Card>
</CardGroup>

## The Domino Trust Center

The [Domino Trust Center](https://trust.domino.ai/) is the authoritative, always-current source for how Domino Data Lab secures its products, its cloud services, and its corporate operations. It is the single place customers, prospects, and auditors can go to review Domino's security posture, request compliance documentation, and subscribe to updates as certifications are renewed or new controls are added.

### Compliance and certifications

The Trust Center provides direct access to Domino's third-party attestations and certifications, including:

* **SOC 2 Type II** report covering the security, availability, and confidentiality trust service criteria.
* **ISO/IEC 27001** certification for Domino's information security management system (ISMS).
* **HIPAA** attestation for handling protected health information in supported deployments.
* **GDPR** and **CCPA** program documentation describing how Domino processes personal data.
* **PCI DSS**, **NIST**, and other framework mappings where applicable to Domino's cloud offerings.

Reports and letters can be requested directly from the Trust Center, typically gated by an NDA click-through, so account teams and prospects can obtain them without a manual back-and-forth.

### Security program documentation

Beyond certifications, the Trust Center publishes the policies and program artifacts that describe how Domino operates:

* **Information security policies** covering acceptable use, access management, change management, incident response, and business continuity.
* **Data protection and privacy** documentation, including subprocessor lists, data processing addenda (DPA), and cross-border transfer mechanisms.
* **Product security overviews** describing Domino's secure development lifecycle (SDLC), penetration testing cadence, and vulnerability disclosure process.
* **Corporate security** artifacts such as background check policies, security awareness training, and physical security controls.

### How to use the Trust Center

<Steps>
  <Step title="Visit the portal">
    Go to [trust.domino.ai](https://trust.domino.ai/) to browse Domino's public security posture, certifications, and policies.
  </Step>

  <Step title="Request gated documents">
    Sign in or accept the NDA click-through to download SOC 2, ISO 27001, penetration test summaries, and other confidential reports.
  </Step>

  <Step title="Subscribe to updates">
    Follow the Trust Center to receive notifications when certifications are renewed, subprocessors change, or new advisories are published.
  </Step>

  <Step title="Share with your security team">
    Point internal reviewers, procurement, and auditors to the Trust Center as the canonical source, rather than emailing static PDFs.
  </Step>
</Steps>

## Related resources

* [Domino Trust Center](https://trust.domino.ai/)
* [Reporting security concerns](/cloud/admin/security-and-compliance/reporting-security-concerns)
* [Submit GDPR requests](/cloud/admin/security-and-compliance/submit-gdpr-requests)
* [Authentication, identity, and access control](/cloud/admin/security-and-compliance/auth-identity-access-control)


# Platform monitoring
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/platform-monitoring

How Domino's audit logs and system telemetry give admins and security teams visibility into platform activity and performance.

Platform monitoring gives admins and security teams visibility into who did what and how the cluster is performing. Monitoring in Domino covers two primary areas: audit logs for user and administrative events, and system telemetry for cluster performance and workload execution.

## How it works

* **Audit logs**: Domino records platform events to provide visibility into user activity and resource modifications. Audit logs let security teams conduct forensic investigations and verify compliance controls.
* **System and workload telemetry**: Monitoring cluster health and resource consumption ensures platform availability and helps prevent resource starvation during high-compute workloads.

## Audit logs

Domino automatically captures the following **event types** in the audit record:

* **Authentication and sessions:** user logins, logouts, password changes, failed login attempts, and API key regenerations.
* **Access control:** changes to user roles, organization memberships, project permissions, and administrative privilege assignments.
* **Secrets and credentials:** creation, modification, or deletion of user environment variables, project environment variables, and Git credentials.
* **Data sources:** creation, update, or deletion of data source configurations and data access grants.
* **Execution management:** launch, termination, and configuration changes for workspaces, jobs, apps, and Model APIs.

Platform administrators can **review audit events** directly within the Domino application.

To view audit records:

1. Log in to Domino as a platform administrator.
2. Go to **Admin** > **Audit Logs**.
3. Filter events by date range, user, event type, or target resource.

For enterprise log retention and automated threat detection, **export Domino audit logs** to your central Security Information and Event Management (SIEM) provider. Domino runs on Kubernetes and outputs structured log records to standard output streams. Configure your log collector running in your Kubernetes cluster to ship logs from the Domino control plane namespace to your external SIEM endpoint.

## System and workload telemetry

Domino exposes system metrics in **Prometheus** format. You can integrate these metrics with your existing monitoring stack or use Domino's built-in **Grafana** dashboards.

Key **monitoring metrics** include:

* **Node resource utilization:** CPU, memory, and disk usage across control plane and execution nodes.
* **GPU utilization:** memory consumption and compute load on GPU-enabled execution nodes.
* **Execution pod state:** status, restart counts, and resource limits for workspace and job containers.
* **Storage consumption:** persistent volume usage for datasets, project files, and system caches.

Every workspace, job, app, and Model API run creates dedicated **execution logs** that capture standard output (`stdout`) and standard error (`stderr`).

To view execution logs:

1. Open your project.
2. Select **Jobs**, **Workspaces**, or **Model APIs**.
3. Click the specific execution run.
4. Select the **Logs** tab to view or download the runtime output.

## Compliance best practices

When configuring platform monitoring for compliance requirements, follow these guidelines:

* **Restrict audit log access:** limit access to the **Admin** > **Audit Logs** interface to authorized security and compliance personnel.
* **Enforce external log immutability:** export audit records to an external, write-once-read-many (WORM) storage target to prevent tampering.
* **Set up threshold alerts:** configure alerts in your SIEM or Prometheus monitoring stack for high-risk events, such as repeated authentication failures, privilege escalations, or unusual spikes in data download activity.
* **Monitor API key rotation:** regularly review audit logs for API key regeneration events to verify that team members adhere to your organization's credential rotation schedules.

## Related

* [Administer identity and access control](/cloud/admin/security-and-compliance/auth-identity-access-control)
* [Vulnerability management](/cloud/admin/security-and-compliance/vulnerability-management)


# Container & Kubernetes security
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/pod-security-policies

Review Domino pod security contexts, privileged service exceptions, Kubernetes policy behavior, and settings for non-root user workloads.

Domino leverages Kubernetes [security contexts](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) to deploy pods with the least privileges necessary for them to function.

By default, pods created by Domino run with the following settings:

* Run as a non-root user with `runAsRoot: false`, `runAsNonRoot: true`, and `privileged: false` set.

* Acquisition of additional system-level `capabilities` must be prevented with `drop: ["ALL"]`.

Domino does not support a policy where `readOnlyRootFilesystem` is set.

In addition, due to inconsistency between SELinux support and volume relabeling, all Domino pods run with an SELinux type option `spc_t`. This setting purposefully bypasses volume relabeling due to the performance impact of volume relabeling while creating Domino user workloads.

Pods that do not adhere to the policies above, such as requiring root privileges or additional capabilities, are documented below with `privileged` indicating pods that need additional user or root privileges and `capabilities` indicated pods that may require additional system-level capabilities.

<table>
  <thead>
    <tr><th>Pod Name</th><th>Exceptions</th><th>Description</th></tr>
  </thead>

  <tbody>
    <tr>
      <td><code>aws-ebs-csi-driver-node</code></td>
      <td><code>capabilities</code><br /><code>privileged</code><br /><code>runAsRoot</code></td>
      <td>Requires access to host node resources to provision and manage AWS EBS volumes for Domino.
      Is not installed if AWS EBS is not configured as a storage provider during installation.</td>
    </tr>

    <tr>
      <td><code>aws-efs-csi-driver-controller</code></td>
      <td><code>capabilities</code><br /><code>runAsRoot</code></td>
      <td>Requires access to host node resources in order manage AWS EFS volumes for Domino.
      Is not installed if AWS EFS is not configured as a storage provider during installation.</td>
    </tr>

    <tr>
      <td><code>aws-efs-csi-driver-node</code></td>
      <td><code>capabilities</code><br /><code>privileged</code><br /><code>runAsRoot</code></td>
      <td>Requires access to host node resources in order manage AWS EFS volumes for Domino.
      Is not installed if AWS EFS is not configured as a storage provider during installation.</td>
    </tr>

    <tr>
      <td><code>csi-driver-smb-controller</code></td>
      <td><code>capabilities</code><br /><code>privileged</code><br /><code>runAsRoot</code></td>
      <td>Requires access to host node resources to manage SMB volumes for Domino.
      Is not installed if SMB volume management is not configured during installation.</td>
    </tr>

    <tr>
      <td><code>csi-driver-smb-linux</code></td>
      <td><code>capabilities</code><br /><code>privileged</code><br /><code>runAsRoot</code></td>
      <td>Requires access to host node resources to manage SMB volumes for Domino.
      Is not installed if SMB volume management is not configured during installation.</td>
    </tr>

    <tr>
      <td><code>dcgm-exporter</code></td>
      <td><code>capabilities</code><br /><code>runAsRoot</code></td>
      <td>Requires access to host node resources to communicate with NVIDIA host engine and expose GPU metrics.
      Only runs on nodes with NVIDIA GPUs. An option to disable this service is available by setting
      gpu\_monitoring\_enabled to false in deploy.yaml</td>
    </tr>

    <tr>
      <td><code>docker-registry-cert-mgr</code></td>
      <td><code>runAsRoot</code></td>
      <td>Requires access to host node resources to manage trusted certificates for Docker when Domino is configured to use a self-provided image registry.
      Is not installed if an external Docker registry is configured during installation.</td>
    </tr>

    <tr>
      <td><code>domino-workbench-backup</code></td>
      <td><code>capabilities</code><br /><code>runAsRoot</code></td>
      <td>Requires privileged access to filesystem resources for data backup.</td>
    </tr>

    <tr>
      <td><code>domino-workbench-backup-check</code></td>
      <td><code>capabilities</code><br /><code>runAsRoot</code></td>
      <td>Requires privileged access to filesystem resources for data backup.</td>
    </tr>

    <tr>
      <td><code>domino-data-importer</code></td>
      <td><code>capabilities</code><br /><code>runAsRoot</code></td>
      <td>Requires privileged access to filesystem resources for disaster recovery and data access. This service does not run by default and must be enabled by admins when needed.</td>
    </tr>

    <tr>
      <td><code>elasticsearch-tls</code></td>
      <td><code>capabilities</code><br /><code>privileged</code><br /><code>runAsRoot</code></td>
      <td>Requires privileged access to system to set a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.17/vm-max-map-count.html">mandatory system limit</a>. If the system setting can be managed outside of Domino, elevated privileges can be removed by setting the following:
      <pre><code>elasticsearch:
      enable\_sysctl: false</code></pre></td>
    </tr>

    <tr>
      <td><code>falco</code></td>
      <td><code>privileged</code><br /><code>runAsRoot</code></td>
      <td>Requires access to host node resources to audit system events using eBPF.</td>
    </tr>

    <tr>
      <td><code>fluentd</code></td>
      <td><code>capabilities</code><br /><code>runAsRoot</code></td>
      <td>Requires access to host node resources to follow container logs which are used to view workload output within Domino.</td>
    </tr>

    <tr>
      <td><code>hephaestus-buildkit</code></td>
      <td><code>capabilities</code><br /><code>privileged</code><br /><code>runAsRoot</code></td>
      <td>BuildKit powers container image building for the environments feature of Domino. By default, requires privileged access to utilize <code>overlay</code> mounts for performance during container image building.
      If your operating system supports <a href="https://rootlesscontaine.rs/how-it-works/overlayfs/">rootless overlayfs</a> natively, the privileges can be removed by setting the following:
      <pre><code>image\_building:
      rootless: true</code></pre></td>
    </tr>

    <tr>
      <td><code>image-caching-agent</code></td>
      <td><code>runAsRoot</code></td>
      <td>Requires access to host node resources to communicate with the container runtime when caching container images for faster workload scheduling. Can be disabled by setting the following:
      <pre><code>image\_caching:
      enabled: false</code></pre></td>
    </tr>

    <tr>
      <td><code>istio-cni-node</code></td>
      <td><code>privileged</code><br /><code>runAsRoot</code></td>
      <td>Requires access to host node resources to facilitate network setup for Istio. Is not installed if Istio is not enabled or the CNI option is disabled.</td>
    </tr>

    <tr>
      <td><code>newrelic-infrastructure</code></td>
      <td><code>capabilities</code><br /><code>privileged</code><br /><code>runAsRoot</code></td>
      <td>Requires access to host node resources to effectively monitor CPU, memory, network, and disk usage and report to New Relic.
      Is not installed if New Relic is not configured for Domino during installation.</td>
    </tr>

    <tr>
      <td><code>newrelic-logging</code></td>
      <td><code>capabilities</code><br /><code>runAsRoot</code></td>
      <td>Requires access to host node resources to follow container logs and report to New Relic.
      Is not installed if New Relic is not configured for Domino during installation.</td>
    </tr>

    <tr>
      <td><code>nfs-client-provisioner</code></td>
      <td><code>capabilities</code><br /><code>runAsRoot</code></td>
      <td>NFS Client Provisioner requires privileged access to filesystem resources to manage NFS volumes for Domino.
      Is not installed if NFS is not configured as a storage provider during installation.</td>
    </tr>

    <tr>
      <td><code>nvidia-device-plugin</code></td>
      <td><code>capabilities</code><br /><code>privileged</code><br /><code>runAsRoot</code></td>
      <td>Requires access to host node resources to communicate with the Kubernetes device API and expose NVIDIA GPU resources to the cluster.
      Is not installed if GPU capabilities are not configuring during installation.</td>
    </tr>

    <tr>
      <td><code>nvidia-device-plugin-mps-control-daemon</code></td>
      <td><code>capabilities</code><br /><code>privileged</code><br /><code>runAsRoot</code></td>
      <td>Requires access to host node resources to communicate with the Kubernetes device API and expose NVIDIA GPU resources to the cluster.
      Is not installed if GPU capabilities are not configuring during installation.</td>
    </tr>

    <tr>
      <td><code>nvidia-driver-installer</code></td>
      <td><code>capabilities</code><br /><code>privileged</code><br /><code>runAsRoot</code></td>
      <td>Requires access to host node resources to install and manage NVIDIA drivers for GPU-enabled instance types.
      Only required for installations on GKE clusters.</td>
    </tr>

    <tr>
      <td><code>openebs</code></td>
      <td><code>capabilities</code><br /><code>runAsRoot</code></td>
      <td>Requires access to host node resources to manage storage volumes.
      Is not installed if OpenEBS is not configured as a storage provider during installation.</td>
    </tr>

    <tr>
      <td><code>smarter-device-manager</code></td>
      <td><code>runAsRoot</code></td>
      <td>Requires access to host node resources to communicate with the Kubernetes device API and expose FUSE devices to the cluster.</td>
    </tr>
  </tbody>
</table>

All user-generated pods from workspaces, jobs, apps, or on-demand compute clusters run with a user and capabilities according to the Configuration record entry for `com.cerebro.domino.computegrid.kubernetes.nonRootExecutions.enabled`. See [non-root executions release notes](https://docs.dominodatalab.com/release_notes/5-8-0/#_non_root_executions) for details.

All user-generated pods from models run as a non-root user but with privilege-escalation capability.

Please see [Pod Security Policies](/cloud/admin/security-and-compliance/pod-security-policies) for an overview of how policies are enforced and specific information about the removal of Pod Security Policies in Kubernetes 1.25.

## Pod security policies

A [Pod Security Policy (PSP)](https://kubernetes.io/docs/concepts/security/pod-security-policy/) is a cluster-level resource that controls security-sensitive aspects of the pod specification. `PodSecurityPolicy` objects define a set of conditions that a pod must run with to be accepted into the system, as well as defaults for the related fields.

Pod security policies serve two purposes in helping secure a Domino installation:

* They prevent the admission (creation) of pods in the cluster if their security stance does not meet the defined policy.

* They mutate (modify) pods that have been allowed by the policy to set defaults matching the PSP (where no values are set).

### Pod security policy deprecation and removal

As of [Kubernetes 1.21](https://kubernetes.io/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/), PSPs are a deprecated feature. In Kubernetes 1.25, support for PSPs has been completely removed.

To support the removal of Pod Security Policies, Domino 5.4 and above do not create per-service PSPs as part of the installation. Instead, all pods define an explicit security context. This removes the need for duplicate modification of security contexts during pod creation by PSPs.

Domino does not support a replacement for runtime policy enforcement. Because Domino controls the definition of all pods created in its namespaces, PSP enforcement of security boundaries after installation (during “runtime”) is not necessary. Domino can control and report (during “build time”) the security context of every pod to ensure all services meet our security standards. See [Kubernetes Pod Security](/cloud/admin/security-and-compliance/pod-security-policies) for an overview with specifics and exemptions.

### OpenShift

OpenShift maintains its own version of pod security policies named "Security Context Constraints" and Red Hat has laid out a different evolution for that feature going forward from upstream Kubernetes.

Domino does not deploy any custom security contexts but makes use of OpenShift-provided SCCs (`anyuid` and `privileged`) bound to all Domino namespaces. Any additional SCCs cause the installation of Domino to fail.

In addition, UID remapping is not supported in Domino and all services run as the user ID defined in their container images.

### Kubernetes \< 1.25 support

Domino binds a blanket unrestricted policy for clusters where PSPs are still enabled. Many cluster providers, such as EKS, already do this. If Domino is the only application deployed in a cluster running Kubernetes \< 1.25, it is advisable to turn off pod security policy support entirely.


# Protecting tokens and API keys
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/protecting-tokens-api-keys

Best practices and Domino features for storing and using API keys, personal access tokens, and other credentials without exposing them.

API keys, personal access tokens, passwords, and similar credentials grant access to external resources. Exposing these credentials creates security risks for your organization.

Domino takes extensive measures to ensure that we do not have unauthorized access to your internal data. However, if secrets are mismanaged by users, sensitive data can still be exposed in unintended ways. Follow these guidelines and best practices to use sensitive credentials safely within Domino.

## Best practices for credential security

* **Never hardcode secrets in source code.** Do not paste raw tokens or passwords into Python scripts, R files, or Jupyter Notebook cells.
* **Keep secrets out of Git history.** Hardcoded credentials committed to Git remain in the commit history even after you delete them. Rotate any committed secret in the external service rather than removing it in a later commit. Add configuration files that contain keys to your `.gitignore` file to help prevent accidental commits.
* **Avoid printing secrets to logs.** Ensure your scripts do not write raw tokens or environment variables to standard output (`stdout`), standard error (`stderr`), or notebook cell outputs, as these can be captured in Domino run logs.

## Manage credentials in Domino

Domino provides multiple mechanisms to store and pass credentials securely, depending on whether the credential belongs to an individual user, a shared project, a Data Source, or an external vault.

### User environment variables

Use [user environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/4-setup-user-environment-variables) for individual access tokens or keys that follow your user account across every project you run.

**Scope:** Injected securely into all Workspaces, Jobs, and Apps started by your user account. Other users cannot see your user environment variables.

To configure user environment variables:

1. Go to **Account Settings** > **User Environment Variables**.
2. Enter the name and value of your variable.

### Project environment variables

Use [project environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/2-build-project-variables) when team members collaborating on a project need shared access to a set of operational credentials.

**Scope:** Accessible to all runs and executions within that specific project.

To configure project environment variables:

1. Open your project.
2. Go to **Settings** > **Environment Variables**.
3. Add the required keys and values.

### Git credentials

For authenticating with remote Git repositories, such as GitHub or Bitbucket, do not store Personal Access Tokens or private keys in project files. Manage them through [Git credentials](/cloud/platform-capabilities/core-concepts/workspaces/link-github-account-to-domino) instead.

**Scope:** Managed at the account level for authenticating code repositories.

To configure Git credentials:

1. Go to **Account Settings** > **Git Credentials**.
2. Add SSH keys or Personal Access Tokens.

### Data Source credentials

When connecting Domino to external data stores such as Snowflake or PostgreSQL, use [Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors) rather than storing database credentials in scripts. Domino stores Data Source credentials in a secret store backed by HashiCorp Vault.

**Scope:** Admin-configured or user-configured secure data connections.

To configure Data Source credentials:

1. Go to **Account Settings** > **Data Sources**, or open the **Data Sources** tab in your project.
2. Authenticate using OAuth or stored credentials.

### External secret managers

If your organization uses external secret management services, integrate them into your Domino workflows:

* **External HashiCorp Vault.** Domino administrators can configure the platform to connect directly to an enterprise HashiCorp Vault cluster to manage platform environment variables and credentials.
* **Cloud secret managers (AWS, Azure, GCP).** Query services such as AWS Secrets Manager or Azure Key Vault directly within your Python or R scripts using standard SDKs. Use cluster IAM roles, for example AWS IRSA, for passwordless authentication.

## Manage your Domino user API key

Your Domino user API key grants programmatic access to the Domino REST API and CLI. See [Authenticate with an API key](/cloud/reference/api/domino-api-authentication#authenticate-with-an-api-key-legacy) for details.

<Warning>
  The Domino user API key is scheduled for deprecation in future versions of Domino. Domino recommends using [Service Accounts](/cloud/admin/identity-and-access/manage-users/service-accounts) for programmatic access, which provide stable, token-based authentication, independent access management, and stronger security.
</Warning>

To view your key, go to **Account Settings** > **API Key**.

To revoke a compromised key:

1. Go to **Account Settings** > **API Key**.
2. Click **Re-generate API Key**. This immediately invalidates the previous key across all external services that use it.

## Related

* [Service Accounts](/cloud/admin/identity-and-access/manage-users/service-accounts)
* [Authenticate with an API key](/cloud/reference/api/domino-api-authentication#authenticate-with-an-api-key-legacy)
* [User environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/4-setup-user-environment-variables)
* [Project environment variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/2-build-project-variables)
* [Git credentials](/cloud/platform-capabilities/core-concepts/workspaces/link-github-account-to-domino)
* [Data Source connectors](/cloud/platform-capabilities/core-concepts/data/data-source-connectors)


# Reporting security concerns
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/reporting-security-concerns

Report Domino product or service vulnerabilities to the security team with reproduction steps, affected components, and potential impact.

If you discover a security vulnerability or have a security concern related to Domino's products or services, report it to the Domino Data Lab security team.

## Report a security concern

Send a message to [`security@dominodatalab.com`](mailto:security@dominodatalab.com) with as much detail as possible:

* Steps to reproduce the issue
* Affected components
* Potential impact

This detail helps the security team investigate and respond quickly.

## Related

* [Trust Center](/cloud/admin/security-and-compliance/trust-center)
* [Vulnerability management](/cloud/admin/security-and-compliance/vulnerability-management)


# Submit General Data Protection Regulation (GDPR) requests
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/submit-gdpr-requests

Submit user deletion, file access, Project reference, deletion, or substitution requests with the identifiers Domino needs to process them.

Domino Data Lab can assist you in your obligations as data controllers under GDPR. This topic describes how to submit requests to Domino, including the information required to process the request. Because Domino does not systematically access the contents of files, uploaded requests must either reference specific users or specific files.

* **User Deletion** - Domino can purge personal data about the name, email address and IP address of Domino users, if required. To process a request, provide:

  * The user account name.

  * A substitute user to inherit any owned projects and files.

* **File versions request** - Domino can provide the hash of all files in a version chain, as well as access to those files. To process a request, provide a text file with the username, project name, and file path. The file must be in this format: `username/file_path_1/file_name.csv`.

* **List of projects referencing a file** - Domino can provide the list of all projects referencing a specific file version in Domino. This helps identify the potential impact of changing a source file or version. To process a request, provide a text file with the username, project name, and file path. The file must be in this format: `username/file_path_1/file_name.csv`.

* **File deletion or substitution request** - To process a request, you must provide:

  * A text file with the username, project name, and file path that must be deleted or substituted. The file must be in this format: `username/file_path_1/file_name.csv`.

  * A text file with the username, project name, and file path to substitute, if applicable. The file must be in this format: `username/file_path_1/file_name.csv`.

After you make a GDPR request, Domino provides evidence of the completed actions, but can no longer access the data.

Most customers do not have to have data deleted or returned through the course of doing business. If you must, for GDPR or other reasons, this can impact your history for reproducibility or auditability.

Domino does not accept responsibility for identifying derived data from files, nor ensuring the stability of projects or other work referencing a user or file altered in a request.


# Trust Center
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/trust-center

Access the Domino Trust Center security and compliance portal through the documentation's external redirect to trust.domino.ai.



# Vulnerability management
Source: https://docs.domino.ai/cloud/admin/security-and-compliance/vulnerability-management

How Domino scans, prioritizes, and remediates vulnerabilities in its product container images and managed cloud infrastructure, for admins and security teams.

Domino runs a continuous vulnerability management program across its product container images, assessing every finding for real-world risk and remediating it before it reaches customer environments.

## How it works

* **Continuous scanning**: Domino continuously scans product container images using multiple vulnerability scanning tools, with scans run on a weekly cadence.
* **Risk validation**: Findings are reviewed to confirm applicability, exploitability, and potential impact within the Domino architecture, using the CVSS environmental score framework rather than relying on the base score alone.
* **Prioritization**: Vulnerabilities are prioritized based on severity, available fixes, exposure, and practical risk to customers.
* **Remediation**: Applicable vulnerabilities are addressed through package upgrades, configuration changes, or other security controls.
* **Release policy**: Domino does not ship a release with an open High or Critical CVE that has an available fix. Fixes for other findings are tracked across supported releases and rolled into future releases.
* **Customer transparency**: Domino provides point-in-time vulnerability reports, mitigation details, and remediation status to customers for each release at its GA date.

## Vulnerability transparency reports

After a release reaches general availability (GA), Domino publishes a transparency report for that release. The report lists every CVE found during scanning and states the risk disposition or exemption for each High and Critical CVE with a fix available.

## Cloud infrastructure

Domino runs a continuous vulnerability management program across its managed cloud infrastructure for Domino Cloud customers. The focus and methodology for these vulnerabilities is in line with product and container vulnerability management.

### How it works

* **Continuous scanning**: Domino continuously scans its cloud infrastructure accounts and assets on a weekly cadence.
* **Risk validation**: Findings are reviewed to confirm applicability, exploitability, and potential impact within the Domino environment, using the CVSS environmental score framework rather than relying on the base score alone.
* **Prioritization**: Vulnerabilities are prioritized based on severity, available fixes, exposure, and practical risk to customers.
* **Remediation**: Applicable vulnerabilities are addressed through package upgrades, configuration changes, OS updates, or other security controls.
* **Release policy**: Domino releases patched images for nodes regularly to ensure infrastructure is frequently patched.


# Domino API authentication
Source: https://docs.domino.ai/cloud/reference/api/domino-api-authentication

Authenticate Domino API requests with Personal Access Tokens, service accounts, the API proxy, or legacy API keys, and manage token lifecycle.

Every request to the [Domino API](/cloud/reference/api/domino-open-api) must
identify who is making it. Use a **Personal Access Token** when you are acting
as yourself, and a **Domino service account** when a pipeline or system acts on
its own behalf.

Domino also supports an older API key, which still works but is deprecated. If
you are choosing an approach now, start with a Personal Access Token.

## Authenticate with a Personal Access Token

Personal Access Tokens (PATs) are user-managed credentials that let you authenticate against the [Domino API](/cloud/reference/api). You create each token with a name, an optional description, and a custom expiration time. Once a token expires or is revoked, it can no longer be used.

PATs give you full control over your API credentials: you can create as many as you need, each with its own expiration, and revoke or delete them individually or all at once.

You can manage your PATs through the **Account** > **Account settings** > **Personal Access Token** panel in the Domino UI, or via the [Domino API](/cloud/reference/api/domino-open-api).

### Create a Personal Access Token

#### Using the UI

1. Go to **Account** > **Account settings** > **Personal Access Token**.

2. Click **Generate new Token**.

   <img alt="Personal Access Tokens panel with Generate new Token button" />

3. Fill in the token details:

   * **Token name**: A unique name for this token. May be visible to resource owners. Only alphanumeric characters, hyphens, and dots are allowed.

   * **Description**: Optional. Describe what this token is used for.

   * **Token Expiration**: Select how long the token should be valid from the dropdown.

     <img alt="Create Personal Access Token dialog" />

4. Click **Create token**.

<Warning>
  The token value is only shown once, immediately after creation. Copy and store it securely. You cannot retrieve it again.
</Warning>

#### Using the API

Token names must be unique, use only alphanumeric characters, hyphens, and dots, and are automatically lowercased. The `expiresIn` field is expressed in seconds.

```shell theme={null}
curl -X POST \
  -H "Authorization: Bearer $DOMINO_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "my-domino-token", "description": "My personal Domino API token", "expiresIn": 86400}' \
  $DOMINO_API_HOST/api/pat/v1/tokens
```

### Use a Personal Access Token

Once created, use the PAT as a bearer token in the `Authorization` header of any Domino API request:

```shell theme={null}
curl -H "Authorization: Bearer $PAT" \
  $DOMINO_API_HOST/v4/users/self
```

### Manage your Personal Access Tokens

The PAT panel lists all your tokens with their name, description, last used date, and current status. The **STATUS** column shows either a status badge (**Revoked**, **Expired**) or the token’s expiration date for active tokens.

| Status | Description |
| - | - |
| Active | Token is valid and not expiring within 7 days. |
| Expiring Soon | Token is valid but expires within 7 days. Rotate it proactively. |
| Expired | Token’s expiration date has passed. |
| Revoked | Token was manually invalidated. |

<Note>
  If your roles are modified by an administrator, your existing PATs are automatically revoked. You need to create tokens to reflect the updated permissions.
</Note>

To revoke or delete a token, click the three-dot menu at the end of the token row and select **Revoke** or **Delete**.

**Revoke** immediately invalidates the token. It remains visible in your list with a **Revoked** status until you delete it.

**Delete** permanently removes the token and all its metadata.

For full API details, see the [Domino REST API reference](/cloud/reference/api/domino-open-api).

Administrators can manage and revoke PATs across all users in the platform. See [Manage Personal Access Tokens](/cloud/admin/identity-and-access/manage-users/personal-access-tokens) for details.

## Authenticate with a Domino service account token

Domino provides a convenient way for external software and automation pipelines to access Domino APIs via Domino Service Accounts. These accounts are managed by the Domino Administrators and have long-lasting credentials, making them easy to use in other parts of the enterprise outside of Domino.

See [Domino Service Accounts documentation](/cloud/admin/identity-and-access/manage-users/service-accounts) for more information.

Once you receive your token from an admin, you can use the token to authenticate and run Domino API calls.

The following example starts a Domino Job using the `SA_TOKEN`. As an additional security measure, you should store the SA token as a [custom environment variable for secure credential storage](/cloud/platform-capabilities/core-concepts/projects/project-settings#store-credentials-as-environment-variables).

```shell theme={null}
$ curl -H "Authorization: Bearer $SA_TOKEN" \
  --header 'Content-Type: application/json' \
  --data '{ "projectId": "12398gg098", "runCommand": "main.py"}' \
  $DOMINO_API_HOST/api/jobs/v1/jobs
```

## Choosing between a Personal Access Token and a Domino service account

Both PATs and Domino Service Account tokens authenticate against the Domino API, but they serve different purposes.

Use a **Personal Access Token** when you are acting as yourself: running your own scripts, triggering jobs interactively, or accessing the API from your local machine. A PAT carries your identity and your permissions. It is the right choice for work that is tied to you as a user.

Use a **[Domino Service Account](/cloud/admin/identity-and-access/manage-users/service-accounts)** when the action should not be tied to a specific person, for example, a CI/CD pipeline, a scheduled automation, or a system integration. Service Accounts have a stable identity independent of any individual user, which means they are not affected by changes to a person’s account or roles, and they can be managed and audited by administrators separately.

As a rule of thumb: if you would be the one running it manually, use a PAT. If a system or pipeline would run it on its own, use a Service Account.

## Configure a service to trust Domino’s JWT

Each system has different steps for configuration but likely requires that you enter Domino’s JWKS endpoint which is `<Domino URL>/auth/realms/DominoRealm/protocol/openid-connect/certs>`. For offline services, retrieve and record the JWKS data to use offline with your library. Alternatively, you can use the public key or certificate which can be found at `<Domino URL>/auth/realms/DominoRealm`.

See [JSON Web Key Sets](https://auth0.com/docs/tokens/concepts/jwks) to learn more about JWKS.

## Token claims

By default, the Domino Token contains standard claims like name, username, and email, but you add mappers in the `domino-play` client in Keycloak to include additional custom claims. This can be useful if you intend to set up bounded claims in the system to which you are integrating. For example, you might want to only authorize users with a specific organization claim or any other SAML assertion that flows into Domino from your Identity Provider.

## Use the API proxy for Domino API authentication

You can use the API proxy to authenticate calls to Domino APIs using the address and port exposed by the proxy process. If you make a call to Domino and use this proxy, the proxy automatically adds the authentication information (the access JWT token) to the proxied request.

See [JWT.io](https://jwt.io/) for information about decoding and inspecting the content of a JWT token.

Authentication with the API proxy

1. The run starts immediately and the token is acquired when the first call is made through the proxy:

   ```shell theme={null}
   $ curl "$DOMINO_API_PROXY/v4/users/self"
   {"firstName":"Jack","lastName":"London","fullName":"Jack London","userName":"jack-london","email":"jack.london@oaklandpub.com","avatarUrl":"","id":"123456789"}
   ```

2. The API proxy runs and `$DOMINO_API_PROXY` contains the proxy address.

## Authenticate with an API key \[LEGACY]

To interact with secured endpoints of the [Model Monitoring API](/cloud/reference/api/model_monitoring/dmm-api) or [Domino Platform API](/cloud/reference/api/domino-open-api), you must send an API authentication token or key along with your request. This parameter identifies you as a specific Domino user so Domino can check for authorization.

<Warning>
  This feature is scheduled for deprecation in future versions of Domino. Domino recommends using [Personal Access Tokens](#authenticate-with-a-personal-access-token) or [Domino Service Accounts](/cloud/admin/identity-and-access/manage-users/service-accounts) instead, depending on your use case. See [Choosing between a Personal Access Token and a Domino Service Account](#choosing-between-a-personal-access-token-and-a-domino-service-account) for guidance on which option is right for you.
</Warning>

To authenticate your requests, include your API key with the header `X-Domino-Api-Key`.

### Get your API key

1. Go to **Account** > **Account settings**.

   <Frame>
     <img alt="The Domino Account settings page, with API Key in the left navigation" />
   </Frame>

2. Click **API Key**, then click **Regenerate** to produce a key. You can only
   view it once.

   <Frame>
     <img alt="The API Key panel showing the Regenerate control" />
   </Frame>

<Warning>
  Anyone with this key can authenticate to the Domino application as the user it represents and take any actions that the user is authorized to perform. Treat it like a sensitive password.
</Warning>


# Domino Platform API reference
Source: https://docs.domino.ai/cloud/reference/api/domino-open-api

Browse Domino's public REST API endpoints for core platform, Model Monitoring, Governance, NetApp Volumes, and Tags and Properties operations.

Reference information for the Domino platform's public REST API endpoints.

The sidebar groups endpoints by capability: core platform capabilities first,
then the Model Monitoring, Governance, NetApp Volumes, and Tags and Properties
groups, which are served by their own services but published inside the
platform's public API specification. The Domino Data API, Workspace File Audit
Trail API, and Flyte API are documented in their own sections alongside this
one.


# API
Source: https://docs.domino.ai/cloud/reference/api/index

Choose Domino APIs and libraries for platform resources, data, monitoring, governance, tags, audit events, NetApp Volumes, and Flows.

Much of Domino’s functionality is available programmatically through the [public REST API endpoints](/cloud/reference/api/domino-open-api). Domino also provides a [Python wrapper](/cloud/reference/python-sdk/python-domino).

The Domino API is different from our [Domino endpoint](/cloud/platform-capabilities/features/model-deployment) functionality:

<table>
  <thead>
    <tr>
      <th>Domino API</th>
      <th>Domino endpoints</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td><p>The Domino API lets you control Domino’s core functionality programmatically.</p> <p>See the topics in this guide to learn about the methods to access this functionality.</p></td>
      <td><p>Domino endpoints are REST API endpoints that provide programmatic access to your R and Python data science code.</p> <p>See <a href="/cloud/platform-capabilities/features/model-deployment">Domino endpoints</a> to learn how to publish and access your models as APIs.</p></td>
    </tr>
  </tbody>
</table>

See also the [Domino CLI](/cloud/reference/cli/cli-reference).

## The Domino API

The Domino API is a collection of public REST API endpoints that give you access to Domino’s core functionality. You can use the API to manage projects, datasets, environments, jobs, and more.

See [Domino Platform API reference](/cloud/reference/api/domino-open-api) for complete reference information about all of the public API endpoints.

Some endpoints require an [API key](/cloud/reference/api/domino-api-authentication#authenticate-with-an-api-key-legacy) to check for authorization to access specific resources.

## Domino Data API

This library for Python and R provides access to your data, in files or SQL-based tables, using consistent access patterns. The results are available as dataframe abstractions for popular libraries. You can query connectors interactively and you don’t have to restart a workload to install drivers.

See [Domino Data API](/cloud/reference/api/data_api) for details.

## The `python-domino` library

The `python-domino` library is a wrapper for many of the Domino API endpoints. You can install it in Domino and use it in your project code to manage your files, executions, datasets, the properties of the project, and more.

See [The python-domino Library](/cloud/reference/python-sdk/python-domino) for details. The public repository is available at [https://github.com/dominodatalab/python-domino](https://github.com/dominodatalab/python-domino).

## Domino R package

The Domino R package gives you access to the full power of Domino’s cloud compute and version control without leaving your R environment. To use this package, you must install [the Domino CLI](/cloud/reference/cli/cli-reference). See [Domino R package](/cloud/reference/r-package/domino-r-package) for details.

## Model Monitoring API

The Model Monitoring API gives you programmatic access to the [Model Monitor](/cloud/platform-capabilities/features/monitoring). See the [Model Monitoring API reference](/cloud/reference/api/model_monitoring/dmm-api) for more detail.

You can also use Domino’s [Model Monitoring custom metrics Python SDK](/cloud/reference/api/model_monitoring/custom-metrics) to define custom metrics and use them alongside drift and model quality monitored metrics.

## Domino Governance API

The Domino Governance API gives you programmatic access to the [Domino Governance](/cloud/platform-capabilities/features/governance) feature. See the [Domino Governance API reference](/cloud/reference/api/governance-api) for more detail.

## Domino Volumes for NetApp ONTAP API

The Domino NetApp Volumes API gives you programmatic access to the [Domino Volumes for NetApp ONTAP](/cloud/admin/data-administration/netapp-volumes) feature. See the [Domino NetApp Volumes API reference](/cloud/reference/api/domino-volumes-api) for more detail.

## Tags API

The Tags API manages the Domino Taxonomy and Properties features: namespaces, tags, and properties. See the [Tags API reference](/cloud/reference/api/tags-api) for more detail.

## Workspace File Audit Trail API

The Workspace File Audit Trail API lets you query and export Workspace file access events. See the [Workspace File Audit Trail API reference](/cloud/reference/api/workspace-file-audit-trail-api) for more detail.

## Flyte API

[Domino Flows](/cloud/platform-capabilities/features/flows) is built on the open source framework [Flyte](https://flyte.org/). Once flows are defined, they can be registered and launched using the Flyte CLI through a Workspace or Job.

See the [Flyte API reference](/cloud/reference/api/flyte) for more detail.


# Reference
Source: https://docs.domino.ai/cloud/reference/index

Find Domino references for API authentication, the Domino CLI, Projects, files, Datasets, and archived documentation versions.

[Domino API authentication](/cloud/reference/api/domino-api-authentication)\
Securely authenticate into Domino programmatically.

[Domino CLI](/cloud/reference/cli)\
Use the Domino Command Line Interface (CLI) to work with Projects, files, and Datasets.

[Documentation maintenance and archive](/help/help#documentation-maintenance-policy)\
Find out how Domino maintains the documentation and where to find the documentation for older, archived versions.


# Get evaluation histogram data
Source: https://docs.domino.ai/api-reference/ai-systems/get-evaluation-histogram-data

/api-specs/cloud/public-api.json get /api/model-services/v1/aisystem/{system_id}/version/{version_id}/chart/histogram/eval_name/{eval_name}
Get histogram data for the specified evaluation name



# Get filterable fields
Source: https://docs.domino.ai/api-reference/ai-systems/get-filterable-fields

/api-specs/cloud/public-api.json get /api/model-services/v1/aisystem/{system_id}/version/{version_id}/filters
Discover the fields that can be used in the server-side 'filter' query parameter, including numeric metrics, categorical label fields with their values, and free-text columns.



# Get labels for metric and label evaluations
Source: https://docs.domino.ai/api-reference/ai-systems/get-labels-for-metric-and-label-evaluations

/api-specs/cloud/public-api.json get /api/model-services/v1/aisystem/{system_id}/version/{version_id}/evaluation-labels
Get labels for metric and label evaluations



# Get metrics timeseries
Source: https://docs.domino.ai/api-reference/ai-systems/get-metrics-timeseries

/api-specs/cloud/public-api.json get /api/model-services/v1/aisystem/{system_id}/version/{version_id}/chart/timeseries/metric/{metric}
Get timeseries data for evaluation metrics over time



# Get trace count timeseries
Source: https://docs.domino.ai/api-reference/ai-systems/get-trace-count-timeseries

/api-specs/cloud/public-api.json get /api/model-services/v1/aisystem/{system_id}/version/{version_id}/chart/timeseries/trace-count
Get timeseries data for the number of traces made within a time period



# Get version timeline
Source: https://docs.domino.ai/api-reference/ai-systems/get-version-timeline

/api-specs/cloud/public-api.json get /api/model-services/v1/aisystem/{system_id}/versions
List all versions of an AI system with their first/last observed trace timestamps, used to draw version boundary markers.



# List traces
Source: https://docs.domino.ai/api-reference/ai-systems/list-traces

/api-specs/cloud/public-api.json get /api/model-services/v1/aisystem/{system_id}/version/{version_id}/traces
List traces for an AI system version, applying the same server-side filter grammar as the chart endpoints (including numeric range filters on eval-metric tags). Returns MLflow-trace-compatible rows with token-based pagination.



# Service Version
Source: https://docs.domino.ai/api-reference/ai-systems/service-version

/api-specs/cloud/public-api.json get /api/model-services/v1/aisystem/version
The service version.



# Get App Instance
Source: https://docs.domino.ai/api-reference/app-instances/get-app-instance

/api-specs/cloud/public-api.json get /api/apps/beta/apps/{appId}/versions/{versionId}/instances/{instanceId}
Retrieve an App Instance



# Get App Instance Logs
Source: https://docs.domino.ai/api-reference/app-instances/get-app-instance-logs

/api-specs/cloud/public-api.json get /api/apps/beta/apps/{appId}/versions/{versionId}/instances/{instanceId}/logs
Retrieve an App Instance Logs



# Get App Instance Real Time Logs
Source: https://docs.domino.ai/api-reference/app-instances/get-app-instance-real-time-logs

/api-specs/cloud/public-api.json get /api/apps/beta/apps/{appId}/versions/{versionId}/instances/{instanceId}/realTimeLogs
Retrieve an App Instance Real Time Logs



# List App Instances
Source: https://docs.domino.ai/api-reference/app-instances/list-app-instances

/api-specs/cloud/public-api.json get /api/apps/beta/apps/{appId}/versions/{versionId}/instances
List all app instances



# Stop App Instance
Source: https://docs.domino.ai/api-reference/app-instances/stop-app-instance

/api-specs/cloud/public-api.json delete /api/apps/beta/apps/{appId}/versions/{versionId}/instances/{instanceId}
Stop an App Instance



# Create App Version
Source: https://docs.domino.ai/api-reference/app-versions/create-app-version

/api-specs/cloud/public-api.json post /api/apps/beta/apps/{appId}/versions
Create an App Version



# Create App Version and Start (v1)
Source: https://docs.domino.ai/api-reference/app-versions/create-app-version-and-start-v1

/api-specs/cloud/public-api.json post /api/apps/v1/apps/{appId}/versions/createAndStart
Compound endpoint that creates a new version and initiates a run of it in a single request. The version is always persisted; if the start initiation fails the response still returns the persisted version with a `warnings` array describing the failure and the caller can retry via the regular start endpoint.
Stop-previous: if a different version of the same App is already running, it is stopped before the new version starts (uniform one-running-instance-per-App invariant; not draft-specific).
Draft-specific defaulting: when the parent App is a draft AND a previous running version exists on it, omitted content reference fields (`dfsCommitId`, `gitRef`, `entryScript`, `environmentId`, `environmentRevisionId`, `hardwareTierId`, `externalVolumeMountIds`, `netAppVolumeIds`) default to the previous running version's values. On the very first call (no previous running version) all required content fields must be supplied.
Governance only starts when a draft is published.




# Create App Version (v1)
Source: https://docs.domino.ai/api-reference/app-versions/create-app-version-v1

/api-specs/cloud/public-api.json post /api/apps/v1/apps/{appId}/versions
Create a new version. Does NOT start an instance.




# Get App Version
Source: https://docs.domino.ai/api-reference/app-versions/get-app-version

/api-specs/cloud/public-api.json get /api/apps/beta/apps/{appId}/versions/{versionId}
Retrieve an App Version



# Get App Version (v1)
Source: https://docs.domino.ai/api-reference/app-versions/get-app-version-v1

/api-specs/cloud/public-api.json get /api/apps/v1/apps/{appId}/versions/{versionId}
Get a single version by ID



# List App Versions
Source: https://docs.domino.ai/api-reference/app-versions/list-app-versions

/api-specs/cloud/public-api.json get /api/apps/beta/apps/{appId}/versions
List all app versions



# List App Versions (v1)
Source: https://docs.domino.ai/api-reference/app-versions/list-app-versions-v1

/api-specs/cloud/public-api.json get /api/apps/v1/apps/{appId}/versions
List all versions for an app



# Live Sync App Version (v1)
Source: https://docs.domino.ai/api-reference/app-versions/live-sync-app-version-v1

/api-specs/cloud/public-api.json post /api/apps/v1/apps/{appId}/versions/{versionId}/liveSync
Trigger a live sync for the active preview instance of a draft app version. The caller provides the current commit IDs; the server syncs only what has changed since the last sync. Returns 404 if no active preview instance exists. No-op (triggered=false) if all provided commits are already up to date.



# Start App Version
Source: https://docs.domino.ai/api-reference/app-versions/start-app-version

/api-specs/cloud/public-api.json post /api/apps/beta/apps/{appId}/versions/{versionId}/start
Start an existing App Version, creating a new instance for it



# Start App Version (v1)
Source: https://docs.domino.ai/api-reference/app-versions/start-app-version-v1

/api-specs/cloud/public-api.json post /api/apps/v1/apps/{appId}/versions/{versionId}/start
Launch an instance for a version. Optional deployment overrides (`hardwareTierId`, `autoscalingSpecification`) are mirrored back to the version so it always reflects the last launch.




# Stop App Version
Source: https://docs.domino.ai/api-reference/app-versions/stop-app-version

/api-specs/cloud/public-api.json post /api/apps/beta/apps/{appId}/versions/{versionId}/stop
Stop the running instance of an App Version



# Stop App Version (v1)
Source: https://docs.domino.ai/api-reference/app-versions/stop-app-version-v1

/api-specs/cloud/public-api.json post /api/apps/v1/apps/{appId}/versions/{versionId}/stop
Stop the running instance for this version.



# Update App Version
Source: https://docs.domino.ai/api-reference/app-versions/update-app-version

/api-specs/cloud/public-api.json patch /api/apps/beta/apps/{appId}/versions/{versionId}
Update an App Version



# Update App Version (v1)
Source: https://docs.domino.ai/api-reference/app-versions/update-app-version-v1

/api-specs/cloud/public-api.json patch /api/apps/v1/apps/{appId}/versions/{versionId}
Update version metadata. Only metadata fields (description, tags, bundleId) are accepted. Content and deployment fields return 422.




# Create App
Source: https://docs.domino.ai/api-reference/apps/create-app

/api-specs/cloud/public-api.json post /api/apps/beta/apps
Create an App



# Create App (v1)
Source: https://docs.domino.ai/api-reference/apps/create-app-v1

/api-specs/cloud/public-api.json post /api/apps/v1/apps
Create a published App with an initial version. The initialVersion field provides optional configuration for the version; a version is always created even if the field is omitted. Does NOT start an instance. (To create a draft App, use POST /apps/drafts.)




# Create or ensure a draft App for a workspace (v1)
Source: https://docs.domino.ai/api-reference/apps/create-or-ensure-a-draft-app-for-a-workspace-v1

/api-specs/cloud/public-api.json post /api/apps/v1/apps/drafts
Idempotent per workspace. Ensures a draft App exists for the caller's workspace: creates a new draft on first call (returns the new draft) or returns the existing draft on re-entry (with any supplied app-field overrides applied as a PATCH).
Always returns 200 with the resulting draft App. Status code is the same on first call and subsequent calls -- clients distinguish create-vs-existing by tracking whether they previously held this draft's id, not by the response status.




# Delete App
Source: https://docs.domino.ai/api-reference/apps/delete-app

/api-specs/cloud/public-api.json delete /api/apps/beta/apps/{appId}
Delete an App



# Delete App Thumbnail
Source: https://docs.domino.ai/api-reference/apps/delete-app-thumbnail

/api-specs/cloud/public-api.json delete /api/apps/beta/apps/{appId}/thumbnail
Delete an App's thumbnail



# Delete App (v1)
Source: https://docs.domino.ai/api-reference/apps/delete-app-v1

/api-specs/cloud/public-api.json delete /api/apps/v1/apps/{appId}
Delete an App



# Get App
Source: https://docs.domino.ai/api-reference/apps/get-app

/api-specs/cloud/public-api.json get /api/apps/beta/apps/{appId}
Retrieve an App



# Get App by Vanity URL
Source: https://docs.domino.ai/api-reference/apps/get-app-by-vanity-url

/api-specs/cloud/public-api.json get /api/apps/beta/apps/vanityUrls/{vanityUrl}
Retrieve an App by Vanity URL



# Get App Thumbnail
Source: https://docs.domino.ai/api-reference/apps/get-app-thumbnail

/api-specs/cloud/public-api.json get /api/apps/beta/apps/{appId}/thumbnail
Retrieve an App's thumbnail image



# Get App Thumbnail Metadata
Source: https://docs.domino.ai/api-reference/apps/get-app-thumbnail-metadata

/api-specs/cloud/public-api.json get /api/apps/beta/apps/{appId}/thumbnail/metadata
Retrieve an App's thumbnail metadata as JSON



# Get App (v1)
Source: https://docs.domino.ai/api-reference/apps/get-app-v1

/api-specs/cloud/public-api.json get /api/apps/v1/apps/{appId}
Get a single App by ID



# Get App Views
Source: https://docs.domino.ai/api-reference/apps/get-app-views

/api-specs/cloud/public-api.json get /api/apps/beta/apps/{appId}/views
Retrieve an App's Views



# Get Project Filter Options
Source: https://docs.domino.ai/api-reference/apps/get-project-filter-options

/api-specs/cloud/public-api.json get /api/apps/beta/apps/projectFilterOptions
Returns all projects that contain apps the user can access.



# Get Publisher Filter Options
Source: https://docs.domino.ai/api-reference/apps/get-publisher-filter-options

/api-specs/cloud/public-api.json get /api/apps/beta/apps/publisherFilterOptions
Returns all app publishers from all the apps the principal can access, optionally filtered by project.



# List Apps
Source: https://docs.domino.ai/api-reference/apps/list-apps

/api-specs/cloud/public-api.json get /api/apps/beta/apps
List all Apps



# List Apps (v1)
Source: https://docs.domino.ai/api-reference/apps/list-apps-v1

/api-specs/cloud/public-api.json get /api/apps/v1/apps
List all Apps



# Publish content from a draft App (v1)
Source: https://docs.domino.ai/api-reference/apps/publish-content-from-a-draft-app-v1

/api-specs/cloud/public-api.json post /api/apps/v1/apps/{appId}/publish
Publishes content from the draft App identified by `{appId}` into either a brand-new published App (publish-as-new) or an existing published App (publish-into-existing). The mode is selected implicitly by the request body: `targetAppId` present selects publish-into-existing, absent selects publish-as-new.
Both branches read the draft's *last-run* `AppVersion` (the version referenced by the most recent `AppInstance` on the draft) as the publish base, optionally apply content (reproducibility) overrides via `versionOverrides`, then write the result onto the destination App. For publish-as-new the destination is a new App with `isDraft=false` and `vanityUrlNamespace=Apps`; for publish-into-existing the destination is the target App identified by `targetAppId`. The draft App and its version history are NOT touched -- the draft persists post-publish so the same workspace can publish again.
Validation:
- The draft must exist and the caller must have publish permission on it. - The draft (and target App, when publishing into an existing app) must have
  configurationType=Standard.
- When publishing into an existing app, the draft and target must agree on
  configurationType and entryScript.
- If the draft has never been previewed and `versionOverrides` does not supply
  the required content fields (commitId, entryScript, environmentId,
  hardwareTierId), returns 422. Clients can use `versionOverrides` to publish
  without first previewing.
- If neither `versionOverrides.entryScript` nor the publish base supplies an
  entryScript, the published version uses `app.sh`.
- `appOverrides` is rejected with 422 when `targetAppId` is present because
  `appOverrides` is only applied when publish inserts a new App. Note that
  `startOverrides` can still write deployment fields such as `vanityUrl` /
  `renderIFrame` back to the target App when `start=true`.

Bundle continuity:
- Publish-as-new: the new version's `bundleId` is whatever the caller supplies,
  otherwise unset.
- Publish-into-existing: the new version inherits the target App's last
  `bundleId` unless the caller supplies a different one.




# Replace all subject (user or organization) role assignments for the given app
Source: https://docs.domino.ai/api-reference/apps/replace-all-subject-user-or-organization-role-assignments-for-the-given-app

/api-specs/cloud/public-api.json put /api/apps/beta/apps/{appId}/roles



# Request Access to App
Source: https://docs.domino.ai/api-reference/apps/request-access-to-app

/api-specs/cloud/public-api.json post /api/apps/beta/apps/{appId}/access/requests
Request access to an App



# Return all subjects (users and organizations) and their corresponding assigned roles for the given app
Source: https://docs.domino.ai/api-reference/apps/return-all-subjects-users-and-organizations-and-their-corresponding-assigned-roles-for-the-given-app

/api-specs/cloud/public-api.json get /api/apps/beta/apps/{appId}/roles



# Update App
Source: https://docs.domino.ai/api-reference/apps/update-app

/api-specs/cloud/public-api.json patch /api/apps/beta/apps/{appId}
Update an App



# Update App (v1)
Source: https://docs.domino.ai/api-reference/apps/update-app-v1

/api-specs/cloud/public-api.json patch /api/apps/v1/apps/{appId}
Update app metadata and defaults. Does not touch versions or instances. entryPoint is immutable and not accepted.




# Upsert App Thumbnail
Source: https://docs.domino.ai/api-reference/apps/upsert-app-thumbnail

/api-specs/cloud/public-api.json post /api/apps/beta/apps/{appId}/thumbnail
Create or replace an App's thumbnail



# Create a Data Source
Source: https://docs.domino.ai/api-reference/data-sources/create-a-data-source

/api-specs/cloud/public-api.json post /api/datasource/v1/datasources
Create a Data Source



# Delete Data Source with specified ID
Source: https://docs.domino.ai/api-reference/data-sources/delete-data-source-with-specified-id

/api-specs/cloud/public-api.json delete /api/datasource/v1/datasources/{dataSourceId}
Delete Data Source with specified ID. Requires Data Source ownership privileges



# Delete monitoring Data Source
Source: https://docs.domino.ai/api-reference/data-sources/delete-model-monitorv2apidatasource

/api-specs/cloud/public-api.json delete /model-monitor/v2/api/datasource



# Get all active Data Sources the user has access to
Source: https://docs.domino.ai/api-reference/data-sources/get-all-active-data-sources-the-user-has-access-to

/api-specs/cloud/public-api.json get /api/datasource/v1/datasources
Get Data Sources that a user has access to based on Data Source permissions and input filters



# Get Data Source Audit Data
Source: https://docs.domino.ai/api-reference/data-sources/get-data-source-audit-data

/api-specs/cloud/public-api.json get /api/datasource/v1/audit
Gets Data Source audit data given filter input parameters



# Get Data Source by ID
Source: https://docs.domino.ai/api-reference/data-sources/get-data-source-by-id

/api-specs/cloud/public-api.json get /api/datasource/v1/datasources/{dataSourceId}
Gets Data Source by ID. Requires access to Data Source



# List monitoring Data Sources
Source: https://docs.domino.ai/api-reference/data-sources/get-model-monitorv2apidatasources

/api-specs/cloud/public-api.json get /model-monitor/v2/api/datasources



# Create monitoring Data Source
Source: https://docs.domino.ai/api-reference/data-sources/put-model-monitorv2apidatasource

/api-specs/cloud/public-api.json put /model-monitor/v2/api/datasource



# Update Data Source with specified ID
Source: https://docs.domino.ai/api-reference/data-sources/update-data-source-with-specified-id

/api-specs/cloud/public-api.json patch /api/datasource/v1/datasources/{dataSourceId}
Update Data Source with specified ID. If the current user is not an admin, then only their individual credentials are updated. Otherwise, the shared credentials are updated. If updating a Starburst-powered Data Source, please remember to restart the Starburst cluster in the UI for the changes to take effect. Requires Data Source management privileges



# Add a grant to a Dataset's existing sequence of grants
Source: https://docs.domino.ai/api-reference/datasets/add-a-grant-to-a-datasets-existing-sequence-of-grants

/api-specs/cloud/public-api.json post /api/datasetrw/v1/datasets/{datasetId}/grants
Add a grant to a Dataset's existing sequence of grants. Requires EditSecurity access to the Dataset.



# Attempts to move the uploaded files to the Dataset
Source: https://docs.domino.ai/api-reference/datasets/attempts-to-move-the-uploaded-files-to-the-dataset

/api-specs/cloud/public-api.json get /datasetrw/datasets/{datasetId}/snapshot/file/end/{uploadKey}
If upload destination consistent, moves uploaded files to specified Dataset directory



# Attempts to rename file or directory
Source: https://docs.domino.ai/api-reference/datasets/attempts-to-rename-file-or-directory

/api-specs/cloud/public-api.json put /datasetrw/dataset/{datasetId}/rename-file
Attempts to rename Dataset file or directory corresponding to the specified datasetId. The request body should contain a path to the file/directory and the intended new name



# Cancels the upload process to a Dataset
Source: https://docs.domino.ai/api-reference/datasets/cancels-the-upload-process-to-a-dataset

/api-specs/cloud/public-api.json get /datasetrw/datasets/{datasetId}/snapshot/file/cancel/{uploadKey}
Cancels the upload process to the specified Dataset and upload key, deleting the corresponding session and directory



# Create a Dataset Snapshot
Source: https://docs.domino.ai/api-reference/datasets/create-a-dataset-snapshot

/api-specs/cloud/public-api.json post /api/datasetrw/v1/datasets/{datasetId}/snapshots
Create a new Snapshot in a Dataset. Requires Read access to the Dataset and project access



# Deletes files from snapshot
Source: https://docs.domino.ai/api-reference/datasets/deletes-files-from-snapshot

/api-specs/cloud/public-api.json delete /datasetrw/snapshot/{snapshotId}/files
Deletes specified files from snapshot corresponding to specified snapshotId. The request body should contain the relative paths of the files to delete



# Download archive file from Domino to local
Source: https://docs.domino.ai/api-reference/datasets/download-archive-file-from-domino-to-local

/api-specs/cloud/public-api.json get /datasetrw/snapshot/{snapshotId}/download-local/{taskKey}
Download archive file created from a download task by filetask from Domino to your local machine



# Get a Dataset Snapshot
Source: https://docs.domino.ai/api-reference/datasets/get-a-dataset-snapshot

/api-specs/cloud/public-api.json get /api/datasetrw/v1/snapshots/{snapshotId}
Fetch a Dataset Snapshot by ID. Requires List access to the Dataset



# Get Dataset grants by ID
Source: https://docs.domino.ai/api-reference/datasets/get-dataset-grants-by-id

/api-specs/cloud/public-api.json get /api/datasetrw/v1/datasets/{datasetId}/grants
Get Dataset grants by ID. Requires List access to the Dataset



# Get Datasets the user has access to
Source: https://docs.domino.ai/api-reference/datasets/get-datasets-the-user-has-access-to

/api-specs/cloud/public-api.json get /api/datasetrw/v2/datasets
Get Datasets that a user has access to based on Dataset permissions and input filters



# Get file names in a snapshot at a specific subpath recursively
Source: https://docs.domino.ai/api-reference/datasets/get-file-names-in-a-snapshot-at-a-specific-subpath-recursively

/api-specs/cloud/public-api.json get /datasetrw/snapshot/{snapshotId}/files/recursive
Get file names in a snapshot at a specific subpath recursively



# Get file names in snapshot
Source: https://docs.domino.ai/api-reference/datasets/get-file-names-in-snapshot

/api-specs/cloud/public-api.json get /datasetrw/files/{snapshotId}
Get file names in the specified snapshot



# Get shared datasets that a project uses
Source: https://docs.domino.ai/api-reference/datasets/get-shared-datasets-that-a-project-uses

/api-specs/cloud/public-api.json get /api/projects/v1/projects/{projectId}/shared-datasets
List shared datasets used by a project. Required permissions: `GetDatasetsRw`



# Get snapshot file metadata at path
Source: https://docs.domino.ai/api-reference/datasets/get-snapshot-file-metadata-at-path

/api-specs/cloud/public-api.json get /datasetrw/snapshot/{snapshotId}/file/meta
Get snapshot file metadata like MIME type and size at specified path



# Get snapshot file raw content at specified path
Source: https://docs.domino.ai/api-reference/datasets/get-snapshot-file-raw-content-at-specified-path

/api-specs/cloud/public-api.json get /datasetrw/snapshot/{snapshotId}/file/raw
Get snapshot file raw content at specified path either to be downloaded or not



# Get Snapshots belonging to a Dataset
Source: https://docs.domino.ai/api-reference/datasets/get-snapshots-belonging-to-a-dataset

/api-specs/cloud/public-api.json get /api/datasetrw/v1/datasets/{datasetId}/snapshots
Get Snapshots belonging to a Dataset. Requires List access to the Dataset



# Link a shared dataset to this project
Source: https://docs.domino.ai/api-reference/datasets/link-a-shared-dataset-to-this-project

/api-specs/cloud/public-api.json post /api/projects/v1/projects/{projectId}/shared-datasets
Link a shared dataset to this project. Required permissions: `ManageDatasetsRw,UseFileStorage`



# Remove a grant from a Dataset's existing sequence of grants
Source: https://docs.domino.ai/api-reference/datasets/remove-a-grant-from-a-datasets-existing-sequence-of-grants

/api-specs/cloud/public-api.json delete /api/datasetrw/v1/datasets/{datasetId}/grants
Remove a grant from a Dataset's existing sequence of grants. Requires EditSecurity access to the Dataset



# Remove a tag from a Dataset
Source: https://docs.domino.ai/api-reference/datasets/remove-a-tag-from-a-dataset

/api-specs/cloud/public-api.json delete /api/datasetrw/v1/datasets/{datasetId}/tags/{tagName}
Remove a tag from a Dataset. Requires Update access to the Dataset



# Returns whether or not this upload chunk has already been sent
Source: https://docs.domino.ai/api-reference/datasets/returns-whether-or-not-this-upload-chunk-has-already-been-sent

/api-specs/cloud/public-api.json get /datasetrw/datasets/{datasetId}/snapshot/file/test
Returns whether or not this upload chunk has already been sent, which is useful for resuming uploads



# Send file upload chunk
Source: https://docs.domino.ai/api-reference/datasets/send-file-upload-chunk

/api-specs/cloud/public-api.json post /datasetrw/datasets/{datasetId}/snapshot/file
Send file upload chunk for the specified Dataset, writing the chunk information to the SQLite database



# Start process to upload a file(s) to the Dataset
Source: https://docs.domino.ai/api-reference/datasets/start-process-to-upload-a-files-to-the-dataset

/api-specs/cloud/public-api.json post /datasetrw/datasets/{datasetId}/snapshot/file/start
Start process to upload a file(s) to the read-write snapshot of the specified Dataset.



# Tag a Snapshot in this Dataset
Source: https://docs.domino.ai/api-reference/datasets/tag-a-snapshot-in-this-dataset

/api-specs/cloud/public-api.json post /api/datasetrw/v1/datasets/{datasetId}/tags
Tag a Snapshot in this Dataset with the given tagName. Requires Update access to the Dataset



# Unlink a shared dataset from this project
Source: https://docs.domino.ai/api-reference/datasets/unlink-a-shared-dataset-from-this-project

/api-specs/cloud/public-api.json delete /api/projects/v1/projects/{projectId}/shared-datasets/{datasetId}
Unlink the dataset with the specified Id from a project. Required permissions: `ManageDatasetsRw,UseFileStorage`



# Check if a vanity URL is available
Source: https://docs.domino.ai/api-reference/genai/check-if-a-vanity-url-is-available

/api-specs/cloud/public-api.json get /api/gen-ai/beta/vanity-url-availability/{vanityUrl}
Check if a vanity URL is available



# Create a new Gen AI Endpoint
Source: https://docs.domino.ai/api-reference/genai/create-a-new-gen-ai-endpoint

/api-specs/cloud/public-api.json post /api/gen-ai/beta/endpoints
Create a new Gen AI Endpoint



# Delete a Gen AI Endpoint
Source: https://docs.domino.ai/api-reference/genai/delete-a-gen-ai-endpoint

/api-specs/cloud/public-api.json delete /api/gen-ai/beta/endpoints/{endpointId}
Delete a Gen AI Endpoint



# Get a Gen AI Endpoint Version
Source: https://docs.domino.ai/api-reference/genai/get-a-gen-ai-endpoint-version

/api-specs/cloud/public-api.json get /api/gen-ai/beta/endpoints/{endpointId}/versions/{versionNumber}
Get a Gen AI Endpoint Version



# Get all Gen AI endpoints accessible by the user
Source: https://docs.domino.ai/api-reference/genai/get-all-gen-ai-endpoints-accessible-by-the-user

/api-specs/cloud/public-api.json get /api/gen-ai/beta/endpoints
Get all Gen AI endpoints accessible by the user



# Get endpoint by id
Source: https://docs.domino.ai/api-reference/genai/get-endpoint-by-id

/api-specs/cloud/public-api.json get /api/gen-ai/beta/endpoints/{endpointId}
Get endpoint by id



# Get Gen AI Endpoint Version Real Time Logs
Source: https://docs.domino.ai/api-reference/genai/get-gen-ai-endpoint-version-real-time-logs

/api-specs/cloud/public-api.json get /api/gen-ai/beta/endpoints/{endpointId}/versions/{versionNumber}/realTimeLogs
Retrieve a Gen AI Endpoint Version Real Time Logs



# Get pod resource usage metrics for a model endpoint version
Source: https://docs.domino.ai/api-reference/genai/get-pod-resource-usage-metrics-for-a-model-endpoint-version

/api-specs/cloud/public-api.json get /api/gen-ai/beta/endpoints/{endpointId}/versions/{versionNumber}/pod-resource-metrics
Get pod resource usage metrics time series data for a specific model endpoint version from Prometheus. Returns memory usage, CPU usage, GPU utilization, and GPU memory usage.



# Get request metrics for a model endpoint version
Source: https://docs.domino.ai/api-reference/genai/get-request-metrics-for-a-model-endpoint-version

/api-specs/cloud/public-api.json get /api/gen-ai/beta/endpoints/{endpointId}/versions/{versionNumber}/request-metrics
Get request metrics time series data for a specific model endpoint version from Prometheus. Returns successful requests, failed requests, and total requests.



# Get the collaborators for a Gen AI Endpoint
Source: https://docs.domino.ai/api-reference/genai/get-the-collaborators-for-a-gen-ai-endpoint

/api-specs/cloud/public-api.json get /api/gen-ai/beta/endpoints/{endpointId}/collaborators
Get the collaborators for a Gen AI Endpoint



# Get the default environment for Gen AI endpoints
Source: https://docs.domino.ai/api-reference/genai/get-the-default-environment-for-gen-ai-endpoints

/api-specs/cloud/public-api.json get /api/gen-ai/beta/default-endpoint-environment
Returns the default environment for Gen AI endpoints



# Get the versions for a Gen AI Endpoint
Source: https://docs.domino.ai/api-reference/genai/get-the-versions-for-a-gen-ai-endpoint

/api-specs/cloud/public-api.json get /api/gen-ai/beta/endpoints/{endpointId}/versions
Get the versions for a Gen AI Endpoint



# Get vLLM latency metrics for a model endpoint version
Source: https://docs.domino.ai/api-reference/genai/get-vllm-latency-metrics-for-a-model-endpoint-version

/api-specs/cloud/public-api.json get /api/gen-ai/beta/endpoints/{endpointId}/versions/{versionNumber}/vllm-latency-metrics
Get vLLM latency metrics time series data for a specific model endpoint version from Prometheus. Returns smoothed averages for time to first token, end-to-end latency, and time per output token.



# Get vLLM token metrics for a model endpoint version
Source: https://docs.domino.ai/api-reference/genai/get-vllm-token-metrics-for-a-model-endpoint-version

/api-specs/cloud/public-api.json get /api/gen-ai/beta/endpoints/{endpointId}/versions/{versionNumber}/vllm-token-metrics
Get vLLM token metrics time series data for a specific model endpoint version from Prometheus. Returns tokens per interval for prompt tokens and generation tokens.



# Patch a Gen AI Endpoint version
Source: https://docs.domino.ai/api-reference/genai/patch-a-gen-ai-endpoint-version

/api-specs/cloud/public-api.json patch /api/gen-ai/beta/endpoints/{endpointId}/versions/{versionNumber}
Patch a Gen AI Endpoint version

This can be used to patch the endpoints properties and version label and description in place,
or to generate a new version based on the given version.
Changing the value of any of the following properties will stop the current version and start a new one
with the patched values:

- vanityUrl
- modelSource
- environment
- hardwareTierId
- configuration




# Start a Gen AI Endpoint Version
Source: https://docs.domino.ai/api-reference/genai/start-a-gen-ai-endpoint-version

/api-specs/cloud/public-api.json post /api/gen-ai/beta/endpoints/{endpointId}/versions/{versionNumber}/start
Start a Gen AI Endpoint Version



# Stop a Gen AI Endpoint Version
Source: https://docs.domino.ai/api-reference/genai/stop-a-gen-ai-endpoint-version

/api-specs/cloud/public-api.json post /api/gen-ai/beta/endpoints/{endpointId}/versions/{versionNumber}/stop
Stop a Gen AI Endpoint Version



# Record App Instance View
Source: https://docs.domino.ai/api-reference/app-instances/record-app-instance-view

/api-specs/cloud/public-api.json post /api/apps/beta/apps/{appId}/versions/{versionId}/instances/{instanceId}/views
Record an App Instance View



# Deactivate billing tag
Source: https://docs.domino.ai/api-reference/cost-and-billing/deactivate-billing-tag

/api-specs/cloud/public-api.json delete /api/cost/v1/billingtags/{tag}
Deactivate a  billing tag.



# Get billing tag by tag name
Source: https://docs.domino.ai/api-reference/cost-and-billing/get-billing-tag-by-tag-name

/api-specs/cloud/public-api.json get /api/cost/v1/billingtags/{tag}
Get billing tag by tag name.



# Get  billing tags setting
Source: https://docs.domino.ai/api-reference/cost-and-billing/get-billing-tags-setting

/api-specs/cloud/public-api.json get /api/cost/v1/billingtagSettings
Get  billing tags setting



# Get mode setting of the billing tags
Source: https://docs.domino.ai/api-reference/cost-and-billing/get-mode-setting-of-the-billing-tags

/api-specs/cloud/public-api.json get /api/cost/v1/billingtagSettings/mode
Get mode setting of the billing tags



# Get notifications setting of the billing tags
Source: https://docs.domino.ai/api-reference/cost-and-billing/get-notifications-setting-of-the-billing-tags

/api-specs/cloud/public-api.json get /api/cost/v1/billingtagSettings/notifications
Get notifications setting of the billing tags



# Get the activation status of the cost feature
Source: https://docs.domino.ai/api-reference/cost-and-billing/get-the-activation-status-of-the-cost-feature

/api-specs/cloud/public-api.json get /api/cost/v1/costActivationStatus
Get the activation status of the cost feature



# Get the asset cost over a time window
Source: https://docs.domino.ai/api-reference/cost-and-billing/get-the-asset-cost-over-a-time-window

/api-specs/cloud/public-api.json get /api/cost/v1/asset
Retrieve asset cost



# Get the cloud cost data
Source: https://docs.domino.ai/api-reference/cost-and-billing/get-the-cloud-cost-data

/api-specs/cloud/public-api.json get /api/cost/v1/cloudCost
Retrieve cloud cost



# Get the cost allocation over a time window
Source: https://docs.domino.ai/api-reference/cost-and-billing/get-the-cost-allocation-over-a-time-window

/api-specs/cloud/public-api.json get /api/cost/v1/allocation
Retrieve cost allocation



# Get the cost allocation summary over a time window
Source: https://docs.domino.ai/api-reference/cost-and-billing/get-the-cost-allocation-summary-over-a-time-window

/api-specs/cloud/public-api.json get /api/cost/v1/allocation/summary
Retrieve cost allocation summary. Faster than the full allocation, but may include less details.



# Get usable billing codes
Source: https://docs.domino.ai/api-reference/cost-and-billing/get-usable-billing-codes

/api-specs/cloud/public-api.json get /api/cost/v1/billingtags
Get billing codes that a user can see.



# Set AWS Billing API Configuration
Source: https://docs.domino.ai/api-reference/cost-and-billing/set-aws-billing-api-configuration

/api-specs/cloud/public-api.json put /api/cost/v1/athenaConfigs
Set AWS Billing API Configuration



# Update billing tags setting
Source: https://docs.domino.ai/api-reference/cost-and-billing/update-billing-tags-setting

/api-specs/cloud/public-api.json put /api/cost/v1/billingtagSettings
Update billing tags setting



# Update mode setting of the billing tags
Source: https://docs.domino.ai/api-reference/cost-and-billing/update-mode-setting-of-the-billing-tags

/api-specs/cloud/public-api.json put /api/cost/v1/billingtagSettings/mode
Update mode setting of the billing tags



# Update notifications setting of the billing tags
Source: https://docs.domino.ai/api-reference/cost-and-billing/update-notifications-setting-of-the-billing-tags

/api-specs/cloud/public-api.json put /api/cost/v1/billingtagSettings/notifications
Update notifications setting of the billing tags



# Upsert billing codes
Source: https://docs.domino.ai/api-reference/cost-and-billing/upsert-billing-codes

/api-specs/cloud/public-api.json post /api/cost/v1/billingtags
Upsert billing codes.



# Cancel a Domino Official Extension install or upgrade
Source: https://docs.domino.ai/api-reference/extensions/cancel-a-domino-official-extension-install-or-upgrade

/api-specs/cloud/public-api.json post /api/extensions/beta/official-installs/{id}/cancel
Cancels an in-progress install or upgrade. This is a destructive operation. The installer will stop at the earliest opportunity, revert any changes made to managed resources during this operation, and clean up the incomplete snapshot. The snapshot is deleted after revert. For an upgrade, if the previously installed app version was running when the upgrade started, that app version is restarted. If the installer completes the operation before processing the cancel request, the install will appear as completed.




# Create an Extension
Source: https://docs.domino.ai/api-reference/extensions/create-an-extension

/api-specs/cloud/public-api.json post /api/extensions/beta/extensions
Create an Extension given an App reference



# Get a Domino Official Extension install by id
Source: https://docs.domino.ai/api-reference/extensions/get-a-domino-official-extension-install-by-id

/api-specs/cloud/public-api.json get /api/extensions/beta/official-installs/{id}
Returns the official extension install entry for the given id.



# Get an Extension by id
Source: https://docs.domino.ai/api-reference/extensions/get-an-extension-by-id

/api-specs/cloud/public-api.json get /api/extensions/beta/extensions/{extensionId}
Get an Extension by id



# Get an Extension for UI consumption
Source: https://docs.domino.ai/api-reference/extensions/get-an-extension-for-ui-consumption

/api-specs/cloud/public-api.json get /api/extensions/beta/extensions-ui/{extensionId}
Get an Extension by id with admin-only fields omitted. Intended for non-admin UI consumption. Returns 404 for globally-disabled or soft-deleted Extensions.




# Get the live diff for a completed extension install snapshot
Source: https://docs.domino.ai/api-reference/extensions/get-the-live-diff-for-a-completed-extension-install-snapshot

/api-specs/cloud/public-api.json get /api/extensions/beta/official-installs/{installId}/snapshots/{snapshotId}/liveDiff
Compares the install snapshot's manifest against the current live state of each managed entity (e.g. project, app, etc) for the given completed install snapshot. Only complete snapshots may be diffed. Returns 400 if the snapshot is incomplete, scheduled for uninstall, or uses an unsupported manifest schema version.




# Increment views for an Extension by one
Source: https://docs.domino.ai/api-reference/extensions/increment-views-for-an-extension-by-one

/api-specs/cloud/public-api.json post /api/extensions/beta/extensions/{extensionId}/views
Increment the view count for an Extension by one. If appVersionId is provided, uses that for per-version tracking; otherwise falls back to the Extension's stored appVersionId.



# Initiate a Domino Official Extension install or upgrade
Source: https://docs.domino.ai/api-reference/extensions/initiate-a-domino-official-extension-install-or-upgrade

/api-specs/cloud/public-api.json post /api/extensions/beta/official-installs
Creates an official extension install record if one does not exist for the given repo, then creates an install/upgrade snapshot and begins the install/upgrade process. Returns 404 if the repo is not a known Domino Official Extension. Returns 400 if there is an existing incomplete install or upgrade for this repo.



# List Domino Official Extension install options
Source: https://docs.domino.ai/api-reference/extensions/list-domino-official-extension-install-options

/api-specs/cloud/public-api.json get /api/extensions/beta/official-install-menu
Returns the available Domino Official Extension installs and upgrades.



# List Extensions
Source: https://docs.domino.ai/api-reference/extensions/list-extensions

/api-specs/cloud/public-api.json get /api/extensions/beta/extensions
List Extensions



# List Extensions for UI consumption
Source: https://docs.domino.ai/api-reference/extensions/list-extensions-for-ui-consumption

/api-specs/cloud/public-api.json get /api/extensions/beta/extensions-ui
List Extensions with admin-only fields omitted. Intended for non-admin UI consumption. Only returns globally-enabled, non-deleted Extensions whose mount config for the requested `mount_point_type` is enabled.




# Patch an Extension by id
Source: https://docs.domino.ai/api-reference/extensions/patch-an-extension-by-id

/api-specs/cloud/public-api.json patch /api/extensions/beta/extensions/{extensionId}
Partially update an Extension. Absent fields are left unchanged.
Any mount points within any `uiMountPointTypeConfig` are upserted.




# Retry a Domino Official Extension install or upgrade
Source: https://docs.domino.ai/api-reference/extensions/retry-a-domino-official-extension-install-or-upgrade

/api-specs/cloud/public-api.json post /api/extensions/beta/official-installs/{id}/retry
Retries a failed or incomplete official extension install. Can resume from the first failed/pending step or restart from the beginning.



# Soft delete an Extension by id
Source: https://docs.domino.ai/api-reference/extensions/soft-delete-an-extension-by-id

/api-specs/cloud/public-api.json delete /api/extensions/beta/extensions/{extensionId}
Soft delete an Extension by id



# Uninstall a Domino Official Extension
Source: https://docs.domino.ai/api-reference/extensions/uninstall-a-domino-official-extension

/api-specs/cloud/public-api.json post /api/extensions/beta/official-installs/{id}/uninstall
Synchronously deletes or archives all Domino resources (app, environment, project, extension record) associated with any install or upgrade for this official extension. If any exception happens during this synchronous operation, the 202 response body includes details about the failure. The remaining cleanup work then happens asynchronously in the background, and the client is immediately able (if desired) to try a fresh install of the extension.




# Ask whether your Bearer token may invoke this Model API
Source: https://docs.domino.ai/api-reference/model-apis/ask-whether-your-bearer-token-may-invoke-this-model-api

/api-specs/cloud/public-api.json get /api/modelServing/beta/modelApis/{modelApiId}/consumer/access
Confirm that the caller in `Authorization: Bearer` may invoke this Model API.
Use this from the Classic Model API sidecar (or any headless client) before
forwarding an inference request. A successful check is HTTP 200 with an empty
body. There is no JSON `canInvoke` field and no login or OAuth redirect.

This is a beta route behind an install-wide identity-invocation flag. When
that flag is off, this endpoint rejects the request. Legacy Model Access
Tokens stay on the sidecar local allowlist and do not use this route.

Public visibility allows every authenticated Domino user to invoke.
Discoverable and Private require an assigned Owner, Editor, or Viewer.
Discoverable alone does not grant invoke. Public still requires a valid
Domino identity; callers are never anonymous.

Related:
- GET /api/modelServing/v1/modelApis/{modelApiId} (read Model API metadata)
- GET /api/modelServing/beta/modelApis/{modelApiId}/visibility (read visibility)
- GET /api/modelServing/beta/modelApis/{modelApiId}/collaborators (list assigned roles)




# Change a Model API collaborator's role
Source: https://docs.domino.ai/api-reference/model-apis/change-a-model-api-collaborators-role

/api-specs/cloud/public-api.json put /api/modelServing/beta/modelApis/{modelApiId}/collaborators/{collaboratorId}
Set an existing collaborator to Owner, Editor, Operator, or Viewer. Path collaboratorId must
equal body.id. isOrganization must match whether the stored principal is an
organization. You cannot demote the last Owner. Public vs private visibility is
not managed here. Only current Owners can change roles.

Related:
- GET /api/modelServing/beta/modelApis/{modelApiId}/collaborators/{collaboratorId} (read the current role)
- POST /api/modelServing/beta/modelApis/{modelApiId}/collaborators (grant access if not assigned)
- DELETE /api/modelServing/beta/modelApis/{modelApiId}/collaborators/{collaboratorId} (remove)




# Change who can discover and invoke a Model API
Source: https://docs.domino.ai/api-reference/model-apis/change-who-can-discover-and-invoke-a-model-api

/api-specs/cloud/public-api.json put /api/modelServing/beta/modelApis/{modelApiId}/visibility
Set visibility to Public, Discoverable, or Private by sending
`{ "visibility": "Private" }` (or Public / Discoverable). Model API Owners and
users with ManageModelCollaborators can change visibility. This updates the
first-class `visibility` field only. It does not write Everyone collaborator
rows or change `access.isPublic` on the v1 payload.

Public still requires every caller to authenticate. Central config
`com.cerebro.domino.modelmanager.allowPublicModels` blocks Public only;
Discoverable and Private remain available when that setting is false.

Related:
- GET /api/modelServing/beta/modelApis/{modelApiId}/visibility (read current visibility)
- GET /api/modelServing/v1/modelApis/{modelApiId} (read Model API metadata)
- GET /api/modelServing/beta/modelApis/{modelApiId}/collaborators (list assigned roles)




# Delete Model API
Source: https://docs.domino.ai/api-reference/model-apis/delete-apimodelservingv1modelapis

/api-specs/cloud/public-api.json delete /api/modelServing/v1/modelApis/{modelApiId}
Archives a Model API.



# List Model APIs
Source: https://docs.domino.ai/api-reference/model-apis/get-apimodelservingv1modelapis

/api-specs/cloud/public-api.json get /api/modelServing/v1/modelApis
Lists Model APIs based on the query filters. Returns only Model APIs visible to the requesting user.



# Get Model API
Source: https://docs.domino.ai/api-reference/model-apis/get-apimodelservingv1modelapis-1

/api-specs/cloud/public-api.json get /api/modelServing/v1/modelApis/{modelApiId}
Retrieves a single Model API.



# List Model API versions
Source: https://docs.domino.ai/api-reference/model-apis/get-apimodelservingv1modelapis-versions

/api-specs/cloud/public-api.json get /api/modelServing/v1/modelApis/{modelApiId}/versions
Lists the Model API Versions for a Model API.



# Get Model API version
Source: https://docs.domino.ai/api-reference/model-apis/get-apimodelservingv1modelapis-versions-1

/api-specs/cloud/public-api.json get /api/modelServing/v1/modelApis/{modelApiId}/versions/{modelApiVersionId}
Retrieves a single Model API Version.



# Get Model API version build logs
Source: https://docs.domino.ai/api-reference/model-apis/get-apimodelservingv1modelapis-versions-buildlogs

/api-specs/cloud/public-api.json get /api/modelServing/v1/modelApis/{modelApiId}/versions/{modelApiVersionId}/buildLogs
Retrieves the build logs for a Model API Version.



# Get Model API version export logs
Source: https://docs.domino.ai/api-reference/model-apis/get-apimodelservingv1modelapis-versions-exportlogs

/api-specs/cloud/public-api.json get /api/modelServing/v1/modelApis/{modelApiId}/versions/{modelApiVersionId}/exportLogs
Retrieves the export logs for a Model API Version.



# Get Model API version instance logs
Source: https://docs.domino.ai/api-reference/model-apis/get-apimodelservingv1modelapis-versions-instancelogs

/api-specs/cloud/public-api.json get /api/modelServing/v1/modelApis/{modelApiId}/versions/{modelApiVersionId}/instanceLogs
Retrieves the instance logs for a Model API Version.



# Grant a user or organization Model API access
Source: https://docs.domino.ai/api-reference/model-apis/grant-a-user-or-organization-model-api-access

/api-specs/cloud/public-api.json post /api/modelServing/beta/modelApis/{modelApiId}/collaborators
Add a user or organization as Owner, Editor, Operator, or Viewer. Organizations may be
Owners. Set isOrganization to true for an organization id and false for a user id;
Domino does not look up the id to confirm the type or that the principal exists.

If that id is already assigned (user or organization), the request fails. Use PUT
to change an existing role. Do not use this endpoint to change public vs private
visibility. Only current Owners can add collaborators.

Related:
- GET /api/modelServing/beta/modelApis/{modelApiId}/collaborators (list after the change)
- PUT /api/modelServing/beta/modelApis/{modelApiId}/collaborators/{collaboratorId} (change an existing role)
- DELETE /api/modelServing/beta/modelApis/{modelApiId}/collaborators/{collaboratorId} (revoke access)




# Create Model API
Source: https://docs.domino.ai/api-reference/model-apis/post-apimodelservingv1modelapis

/api-specs/cloud/public-api.json post /api/modelServing/v1/modelApis
Creates a new Model API with a single Model API version.



# Create Model API version
Source: https://docs.domino.ai/api-reference/model-apis/post-apimodelservingv1modelapis-versions

/api-specs/cloud/public-api.json post /api/modelServing/v1/modelApis/{modelApiId}/versions
Creates a Model API Version for a Model API.



# Update Model API
Source: https://docs.domino.ai/api-reference/model-apis/put-apimodelservingv1modelapis

/api-specs/cloud/public-api.json put /api/modelServing/v1/modelApis/{modelApiId}
Updates a Model API.



# Read one Model API collaborator's role
Source: https://docs.domino.ai/api-reference/model-apis/read-one-model-api-collaborators-role

/api-specs/cloud/public-api.json get /api/modelServing/beta/modelApis/{modelApiId}/collaborators/{collaboratorId}
Return id, role, and isOrganization for one assigned user or organization.
Match is by id only (user or organization). Everyone / visibility is not returned.
You need permission to view the Model API.

Related:
- GET /api/modelServing/beta/modelApis/{modelApiId}/collaborators (list all)
- PUT /api/modelServing/beta/modelApis/{modelApiId}/collaborators/{collaboratorId} (change role)
- DELETE /api/modelServing/beta/modelApis/{modelApiId}/collaborators/{collaboratorId} (remove)




# Read who can discover and invoke a Model API
Source: https://docs.domino.ai/api-reference/model-apis/read-who-can-discover-and-invoke-a-model-api

/api-specs/cloud/public-api.json get /api/modelServing/beta/modelApis/{modelApiId}/visibility
Return Public, Discoverable, or Private for this Model API. Use this beta route;
GET /api/modelServing/v1/modelApis/{modelApiId} does not include a `visibility` field.

Any authenticated user can read visibility when it is Public or Discoverable.
Private requires an assigned Owner, Editor, Operator, or Viewer.

Public: every authenticated Domino user can discover and invoke (JWT or session
required; callers are never anonymous). Discoverable: every authenticated user can
discover overview metadata; invoke requires an assigned Owner, Editor, Operator, or Viewer.
Private: assigned collaborators only for discover and invoke.

Related:
- GET /api/modelServing/v1/modelApis/{modelApiId} (read Model API metadata)
- PUT /api/modelServing/beta/modelApis/{modelApiId}/visibility (change visibility)
- GET /api/modelServing/beta/modelApis/{modelApiId}/collaborators (list assigned roles)




# Request a prediction from an Async model
Source: https://docs.domino.ai/api-reference/model-apis/request-a-prediction-from-an-async-model

/api-specs/cloud/public-api.json post /api/modelApis/async/v1/{asyncModelId}
Request a prediction from an Async Model



# Retrieve the result of an Async Model prediction
Source: https://docs.domino.ai/api-reference/model-apis/retrieve-the-result-of-an-async-model-prediction

/api-specs/cloud/public-api.json get /api/modelApis/async/v1/{asyncModelId}/{asyncPredictionId}
Retrieve the result of an Async Model prediction



# Revoke a collaborator's access to a Model API
Source: https://docs.domino.ai/api-reference/model-apis/revoke-a-collaborators-access-to-a-model-api

/api-specs/cloud/public-api.json delete /api/modelServing/beta/modelApis/{modelApiId}/collaborators/{collaboratorId}
Remove a user or organization collaborator. You cannot remove the last Owner.
Public vs private visibility is not managed here. Only current Owners can remove
collaborators.

Related:
- GET /api/modelServing/beta/modelApis/{modelApiId}/collaborators (confirm who remains)
- POST /api/modelServing/beta/modelApis/{modelApiId}/collaborators (grant access again)
- PUT /api/modelServing/beta/modelApis/{modelApiId}/collaborators/{collaboratorId} (demote instead of remove)




# Save Model API visibility and named collaborators in one request
Source: https://docs.domino.ai/api-reference/model-apis/save-model-api-visibility-and-named-collaborators-in-one-request

/api-specs/cloud/public-api.json put /api/modelServing/beta/modelApis/{modelApiId}/access
Persist Public, Discoverable, or Private together with the full list of user and
organization collaborators. Use this for a single Access Save so both halves apply
or neither does. Load the current values from the related GET routes, send every
collaborator that should remain, and keep at least one Owner. Anyone omitted loses
the assigned role, including the caller. Omitting yourself is allowed when another
Owner remains; omitting the last Owner is rejected.

This route does not change `access.isPublic` on
GET /api/modelServing/v1/modelApis/{modelApiId}. Public still requires an
authenticated caller. `com.cerebro.domino.modelmanager.allowPublicModels` blocks
Public only; Discoverable and Private stay available when that setting is false.
PUT /api/modelServing/v1/modelApis/{modelApiId} does not accept these fields.
Incremental writes still work: PUT .../visibility and POST/PUT/DELETE .../collaborators.

Related:
- GET /api/modelServing/beta/modelApis/{modelApiId}/visibility (read current visibility)
- GET /api/modelServing/beta/modelApis/{modelApiId}/collaborators (list current collaborators)
- GET /api/modelServing/v1/modelApis/{modelApiId} (confirm the Model API id)




# See who can access a Model API
Source: https://docs.domino.ai/api-reference/model-apis/see-who-can-access-a-model-api

/api-specs/cloud/public-api.json get /api/modelServing/beta/modelApis/{modelApiId}/collaborators
Return each user and organization collaborator with role Owner, Editor, Operator, or Viewer.
Use this to audit sharing before you grant, change, or revoke access.

This list does not include the Everyone visibility entry. Public vs private access
is not managed here. GET /api/modelServing/v1/modelApis/{modelApiId} still embeds
collaborators as {collaborator, role} strings; this beta list uses {id, role,
isOrganization}.

Related:
- GET /api/modelServing/v1/modelApis/{modelApiId} (read the Model API)
- POST /api/modelServing/beta/modelApis/{modelApiId}/collaborators (grant access)
- GET /api/modelServing/beta/modelApis/{modelApiId}/collaborators/{collaboratorId} (read one collaborator)




# Create a new Model Deployment
Source: https://docs.domino.ai/api-reference/model-deployment/create-a-new-model-deployment

/api-specs/cloud/public-api.json post /api/modelServing/v1/modelDeployments
Creates a new Model Deployment entity.



# Creates a new Deployment Target
Source: https://docs.domino.ai/api-reference/model-deployment/creates-a-new-deployment-target

/api-specs/cloud/public-api.json post /api/admin/v1/deploymentTargets
Creates a new Deployment Target.



# Creates a new Resource Configuration
Source: https://docs.domino.ai/api-reference/model-deployment/creates-a-new-resource-configuration

/api-specs/cloud/public-api.json post /api/admin/v1/deploymentTargets/{deploymentTargetId}/resourceConfigurations
Creates a new Resource Configuration.



# Delete model deployment
Source: https://docs.domino.ai/api-reference/model-deployment/delete-apimodelservingv1modeldeployments

/api-specs/cloud/public-api.json delete /api/modelServing/v1/modelDeployments/{modelDeploymentId}
Archives a Model Deployment.



# Deletes a Deployment Target
Source: https://docs.domino.ai/api-reference/model-deployment/deletes-a-deployment-target

/api-specs/cloud/public-api.json delete /api/admin/v1/deploymentTargets/{deploymentTargetId}
Deletes a Deployment Target.



# Deletes a Resource Configuration
Source: https://docs.domino.ai/api-reference/model-deployment/deletes-a-resource-configuration

/api-specs/cloud/public-api.json delete /api/admin/v1/deploymentTargets/{deploymentTargetId}/resourceConfigurations/{resourceConfigurationId}
Deletes a Resource Configuration.



# Gets a Deployment Target
Source: https://docs.domino.ai/api-reference/model-deployment/gets-a-deployment-target

/api-specs/cloud/public-api.json get /api/admin/v1/deploymentTargets/{deploymentTargetId}
Gets a Deployment Target.



# Gets a Deployment Target Type
Source: https://docs.domino.ai/api-reference/model-deployment/gets-a-deployment-target-type

/api-specs/cloud/public-api.json get /api/admin/v1/deploymentTargetTypes/{deploymentTargetTypeId}
Gets a Deployment Target Type.



# Gets a Resource Configuration
Source: https://docs.domino.ai/api-reference/model-deployment/gets-a-resource-configuration

/api-specs/cloud/public-api.json get /api/admin/v1/deploymentTargets/{deploymentTargetId}/resourceConfigurations/{resourceConfigurationId}
Gets a Resource Configuration.



# Gets all Deployment Target Types
Source: https://docs.domino.ai/api-reference/model-deployment/gets-all-deployment-target-types

/api-specs/cloud/public-api.json get /api/admin/v1/deploymentTargetTypes
Gets all Deployment Target Types.



# Gets all non-archived Deployment Targets based on the provided filters
Source: https://docs.domino.ai/api-reference/model-deployment/gets-all-non-archived-deployment-targets-based-on-the-provided-filters

/api-specs/cloud/public-api.json get /api/admin/v1/deploymentTargets
Gets all non-archived Deployment Targets based on the provided filters.



# Gets all non-archived Resource Configurations based on the provided filters
Source: https://docs.domino.ai/api-reference/model-deployment/gets-all-non-archived-resource-configurations-based-on-the-provided-filters

/api-specs/cloud/public-api.json get /api/admin/v1/deploymentTargets/{deploymentTargetId}/resourceConfigurations
Gets all non-archived Resource Configurations based on the provided filters.



# Gets all selectable non-archived Deployment Targets based on the provided filters
Source: https://docs.domino.ai/api-reference/model-deployment/gets-all-selectable-non-archived-deployment-targets-based-on-the-provided-filters

/api-specs/cloud/public-api.json get /api/admin/v1/deploymentTargets/selectable
Gets all selectable non-archived Deployment Targets based on the provided filters.



# Gets the effective configuration for model deployments using the resource configuration
Source: https://docs.domino.ai/api-reference/model-deployment/gets-the-effective-configuration-for-model-deployments-using-the-resource-configuration

/api-specs/cloud/public-api.json get /api/admin/v1/deploymentTargets/{deploymentTargetId}/resourceConfigurations/{resourceConfigurationId}/effectiveModelDeploymentSchema
Gets the effective configuration for model deployments using the resource configuration.



# Start model deployment
Source: https://docs.domino.ai/api-reference/model-deployment/post-apimodelservingv1modeldeployments-start

/api-specs/cloud/public-api.json post /api/modelServing/v1/modelDeployments/{modelDeploymentId}/start
Starts a Stopped or Failed Model Deployment.



# Stop model deployment
Source: https://docs.domino.ai/api-reference/model-deployment/post-apimodelservingv1modeldeployments-stop

/api-specs/cloud/public-api.json post /api/modelServing/v1/modelDeployments/{modelDeploymentId}/stop
Stops a Running Model Deployment.



# Retrieve a specific Model Deployment
Source: https://docs.domino.ai/api-reference/model-deployment/retrieve-a-specific-model-deployment

/api-specs/cloud/public-api.json get /api/modelServing/v1/modelDeployments/{modelDeploymentId}
Retrieves a specific Model Deployment entity by id.



# Retrieve a specific version of a Model Deployment
Source: https://docs.domino.ai/api-reference/model-deployment/retrieve-a-specific-version-of-a-model-deployment

/api-specs/cloud/public-api.json get /api/modelServing/v1/modelDeployments/{modelDeploymentId}/versions/{version}
Retrieves a specific historical version (Model Deployment entity) of a Model Deployment.



# Retrieve all Model Deployments
Source: https://docs.domino.ai/api-reference/model-deployment/retrieve-all-model-deployments

/api-specs/cloud/public-api.json get /api/modelServing/v1/modelDeployments
Retrieves all Model Deployments filtered by optional query arguments.



# Retrieve all temporary credentials for a specific Model Deployment
Source: https://docs.domino.ai/api-reference/model-deployment/retrieve-all-temporary-credentials-for-a-specific-model-deployment

/api-specs/cloud/public-api.json get /api/modelServing/v1/modelDeployments/{modelDeploymentId}/credentials
Retrieves summary information for all available temporary credentials of a Model Deployment.



# Retrieve all versions of a specific Model Deployment
Source: https://docs.domino.ai/api-reference/model-deployment/retrieve-all-versions-of-a-specific-model-deployment

/api-specs/cloud/public-api.json get /api/modelServing/v1/modelDeployments/{modelDeploymentId}/versions
Retrieves summary information for all historical versions of a Model Deployment.



# Retrieve temporary credentials of a specific Model Deployment for a specific operation type
Source: https://docs.domino.ai/api-reference/model-deployment/retrieve-temporary-credentials-of-a-specific-model-deployment-for-a-specific-operation-type

/api-specs/cloud/public-api.json get /api/modelServing/v1/modelDeployments/{modelDeploymentId}/credentials/{operationType}
Retrieves information for available temporary credentials of a Model Deployment for a specific operation type.



# Retrieve the logs for a specific Model Deployment
Source: https://docs.domino.ai/api-reference/model-deployment/retrieve-the-logs-for-a-specific-model-deployment

/api-specs/cloud/public-api.json get /api/modelServing/v1/modelDeployments/{modelDeploymentId}/logs/{logSuffix}
Retrieves the logs for a specific Model Deployment entity by id.



# Update a Model Deployment
Source: https://docs.domino.ai/api-reference/model-deployment/update-a-model-deployment

/api-specs/cloud/public-api.json patch /api/modelServing/v1/modelDeployments/{modelDeploymentId}
Update fields of a Model Deployment entity by id.



# Updates a Deployment Target
Source: https://docs.domino.ai/api-reference/model-deployment/updates-a-deployment-target

/api-specs/cloud/public-api.json patch /api/admin/v1/deploymentTargets/{deploymentTargetId}
Updates a Deployment Target.



# Updates a Resource Configuration
Source: https://docs.domino.ai/api-reference/model-deployment/updates-a-resource-configuration

/api-specs/cloud/public-api.json patch /api/admin/v1/deploymentTargets/{deploymentTargetId}/resourceConfigurations/{resourceConfigurationId}
Updates a Resource Configuration.



# [DEPRECATED] Create a new Registered Model from an experiment run
Source: https://docs.domino.ai/api-reference/registered-models/[deprecated]-create-a-new-registered-model-from-an-experiment-run

/api-specs/cloud/public-api.json post /api/registeredmodels/v1
**DEPRECATED**: This endpoint is deprecated as of version 6.2.1.
Please migrate to POST /api/registeredmodels/v2 for enhanced functionality.




# [DEPRECATED] Create a new version of a Registered Model
Source: https://docs.domino.ai/api-reference/registered-models/[deprecated]-create-a-new-version-of-a-registered-model

/api-specs/cloud/public-api.json post /api/registeredmodels/v1/{modelName}/versions
**DEPRECATED**: This endpoint is deprecated as of version 6.2.1.
Please migrate to POST /api/registeredmodels/v2 for enhanced functionality.




# Download model image build logs
Source: https://docs.domino.ai/api-reference/registered-models/download-model-image-build-logs

/api-specs/cloud/public-api.json get /api/registeredmodels/v1/{modelName}/versions/{modelVersion}/builds/{buildId}/logs/download
Download model image build logs as new-line-separated JSON



# Get a list of Registered Models' names visible to user
Source: https://docs.domino.ai/api-reference/registered-models/get-a-list-of-registered-models-names-visible-to-user

/api-specs/cloud/public-api.json get /api/registeredmodels/v1/names
Get a list of Registered Models' names visible to user.



# Get a specific Registered Model
Source: https://docs.domino.ai/api-reference/registered-models/get-a-specific-registered-model

/api-specs/cloud/public-api.json get /api/registeredmodels/v1/{modelName}
Get a specific Registered Model



# Get a specific version of a Registered Model
Source: https://docs.domino.ai/api-reference/registered-models/get-a-specific-version-of-a-registered-model

/api-specs/cloud/public-api.json get /api/registeredmodels/v1/{modelName}/versions/{version}
Get a specific version of a Registered Model



# Get a specific version of a Registered Model (V2)
Source: https://docs.domino.ai/api-reference/registered-models/get-a-specific-version-of-a-registered-model-v2

/api-specs/cloud/public-api.json get /api/registeredmodels/v2/{modelName}/versions/{version}
Get a specific version of a Registered Model with optional experiment run ID



# Get all versions of a Registered Model
Source: https://docs.domino.ai/api-reference/registered-models/get-all-versions-of-a-registered-model

/api-specs/cloud/public-api.json get /api/registeredmodels/v1/{modelName}/versions
Get all versions of a Registered Model



# Get model image build logs
Source: https://docs.domino.ai/api-reference/registered-models/get-model-image-build-logs

/api-specs/cloud/public-api.json get /api/registeredmodels/v1/{modelName}/versions/{modelVersion}/builds/{buildId}/logs
Get model image build logs for streaming



# Get Registered Models visible to user
Source: https://docs.domino.ai/api-reference/registered-models/get-registered-models-visible-to-user

/api-specs/cloud/public-api.json get /api/registeredmodels/v1
Get registered models that a user can see. Returns only ML models.



# Get Registered Models visible to user
Source: https://docs.domino.ai/api-reference/registered-models/get-registered-models-visible-to-user-1

/api-specs/cloud/public-api.json get /api/registeredmodels/v1/ui
Get registered models that a user can see. Returns only ML models.



# Get Registered Models visible to user (v2)
Source: https://docs.domino.ai/api-reference/registered-models/get-registered-models-visible-to-user-v2

/api-specs/cloud/public-api.json get /api/registeredmodels/v2
Get registered models that a user can see, with support for filtering by model category. Returns all models by default.



# Get Registered Models visible to user (v2)
Source: https://docs.domino.ai/api-reference/registered-models/get-registered-models-visible-to-user-v2-1

/api-specs/cloud/public-api.json get /api/registeredmodels/v2/ui
Get registered models that a user can see, with support for filtering by model category. Returns all models by default.



# Get unregistered models for an experiment
Source: https://docs.domino.ai/api-reference/registered-models/get-unregistered-models-for-an-experiment

/api-specs/cloud/public-api.json get /api/registeredmodels/v1/experiments/{experimentId}/unregistered-models
Returns models logged from all runs in an experiment that are not yet registered in the model registry. This includes MLflow 2 artifact paths (directories) and MLflow 3 logged model IDs with their names. The response is organized by run ID.



# Get versions of a Registered Model (V2)
Source: https://docs.domino.ai/api-reference/registered-models/get-versions-of-a-registered-model-v2

/api-specs/cloud/public-api.json get /api/registeredmodels/v2/{modelName}/versions
Get versions of a Registered Model with optional experiment run ID



# Register a model from various sources
Source: https://docs.domino.ai/api-reference/registered-models/register-a-model-from-various-sources

/api-specs/cloud/public-api.json post /api/registeredmodels/v2
Register a model from various sources using a unified API



# Request access to a project
Source: https://docs.domino.ai/api-reference/registered-models/request-access-to-a-project

/api-specs/cloud/public-api.json post /api/registeredmodels/v1/access/{projectId}
Request access to a project



# Returns list of Model APIs deployed from a specific Registered Model
Source: https://docs.domino.ai/api-reference/registered-models/returns-list-of-model-apis-deployed-from-a-specific-registered-model

/api-specs/cloud/public-api.json get /api/registeredmodels/v1/{modelName}/modelapis
Gets all active model Apis that were deployed from a given Registered Model



# Returns list of Model APIs deployed from a specific Registered Model Version
Source: https://docs.domino.ai/api-reference/registered-models/returns-list-of-model-apis-deployed-from-a-specific-registered-model-version

/api-specs/cloud/public-api.json get /api/registeredmodels/v1/{modelName}/versions/{version}/modelapis
Gets all active model Apis that were deployed from a given Registered Model Version



# Update a Registered Model
Source: https://docs.domino.ai/api-reference/registered-models/update-a-registered-model

/api-specs/cloud/public-api.json patch /api/registeredmodels/v1/{modelName}
Update a Registered Model



# List audit events
Source: https://docs.domino.ai/api-reference/audit-trail/get-apiaudittrailv1auditevents

/api-specs/cloud/public-api.json get /api/audittrail/v1/auditevents



# Get distinct field values
Source: https://docs.domino.ai/api-reference/audit-trail/get-apiaudittrailv1fieldvalues

/api-specs/cloud/public-api.json get /api/audittrail/v1/fieldValues
Get distinct values for a specific field with optional search filtering. Supports fileName extraction from multiple document sources.



# Search audit events
Source: https://docs.domino.ai/api-reference/audit-trail/post-apiaudittrailv1search

/api-specs/cloud/public-api.json post /api/audittrail/v1/search
Search audit events using JSON filter conditions with support for complex queries including fileName searches



# Add kubecost license key
Source: https://docs.domino.ai/api-reference/cost-and-billing/add-kubecost-license-key

/api-specs/cloud/public-api.json put /api/cost/v1/licenseKey
Add kubecost license key



# Get accumulated asset cost over a time window
Source: https://docs.domino.ai/api-reference/cost-and-billing/get-accumulated-asset-cost-over-a-time-window

/api-specs/cloud/public-api.json get /api/cost/v2/asset/accumulated
Retrieve accumulated asset cost v2



# Get accumulated cost allocation over a time window
Source: https://docs.domino.ai/api-reference/cost-and-billing/get-accumulated-cost-allocation-over-a-time-window

/api-specs/cloud/public-api.json get /api/cost/v2/allocation/accumulated
Retrieve cost allocation accumulated v2



# Get accumulated cost allocation summary over a time window
Source: https://docs.domino.ai/api-reference/cost-and-billing/get-accumulated-cost-allocation-summary-over-a-time-window

/api-specs/cloud/public-api.json get /api/cost/v2/allocation/summary/accumulated
Retrieve accumulated cost allocation summary. Faster than the full allocation, but may include less details.



# Get the accumulated cloud cost data
Source: https://docs.domino.ai/api-reference/cost-and-billing/get-the-accumulated-cloud-cost-data

/api-specs/cloud/public-api.json get /api/cost/v2/cloudCost/accumulated
Retrieve accumulated cloud cost V2



# Get the asset cost over a time window
Source: https://docs.domino.ai/api-reference/cost-and-billing/get-the-asset-cost-over-a-time-window-1

/api-specs/cloud/public-api.json get /api/cost/v2/asset
Retrieve asset cost v2



# Get the cloud cost data
Source: https://docs.domino.ai/api-reference/cost-and-billing/get-the-cloud-cost-data-1

/api-specs/cloud/public-api.json get /api/cost/v2/cloudCost
Retrieve cloud cost V2



# Get the cost allocation over a time window
Source: https://docs.domino.ai/api-reference/cost-and-billing/get-the-cost-allocation-over-a-time-window-1

/api-specs/cloud/public-api.json get /api/cost/v2/allocation
Retrieve cost allocation v2



# Get the cost allocation summary over a time window
Source: https://docs.domino.ai/api-reference/cost-and-billing/get-the-cost-allocation-summary-over-a-time-window-1

/api-specs/cloud/public-api.json get /api/cost/v2/allocation/summary
Retrieve cost allocation summary. Faster than the full allocation, but may include less details.



# Log metric values
Source: https://docs.domino.ai/api-reference/custom-metrics/log-metric-values

/api-specs/cloud/public-api.json post /api/metricValues/v1
Log metric values. Required Permissions: `RegisterMonitoringDataset`



# Retrieve metric values
Source: https://docs.domino.ai/api-reference/custom-metrics/retrieve-metric-values

/api-specs/cloud/public-api.json get /api/metricValues/v1/{modelMonitoringId}/{metric}
Retrieve metric values. Required Permissions: `UpdateMonitoringSettings`



# Send a metric alert
Source: https://docs.domino.ai/api-reference/custom-metrics/send-a-metric-alert

/api-specs/cloud/public-api.json post /api/metricAlerts/v1
Send a metric out of range alert for a monitored model. Required Permissions: `ViewMonitoringResults`



# Get dataset timeline
Source: https://docs.domino.ai/api-reference/model-monitoring-drift-and-quality/get-model-monitorv2apimodel-dataset-timeline

/api-specs/cloud/public-api.json get /model-monitor/v2/api/model/{model_id}/dataset-timeline



# Get drift check history
Source: https://docs.domino.ai/api-reference/model-monitoring-drift-and-quality/get-model-monitorv2apimodel-drift-schedule-checks-history

/api-specs/cloud/public-api.json get /model-monitor/v2/api/model/{model_id}/drift-schedule-checks-history



# Get drift scheduler query
Source: https://docs.domino.ai/api-reference/model-monitoring-drift-and-quality/get-model-monitorv2apimodel-drift-scheduler-query

/api-specs/cloud/public-api.json get /model-monitor/v2/api/model/{model_id}/drift-scheduler-query



# Get column drift trend
Source: https://docs.domino.ai/api-reference/model-monitoring-drift-and-quality/get-model-monitorv2apimodel-drift-trend

/api-specs/cloud/public-api.json get /model-monitor/v2/api/model/{model_id}/{column_id}/drift-trend



# Get drift user query
Source: https://docs.domino.ai/api-reference/model-monitoring-drift-and-quality/get-model-monitorv2apimodel-drift-user-query

/api-specs/cloud/public-api.json get /model-monitor/v2/api/model/{model_id}/drift-user-query



# Get quality user query
Source: https://docs.domino.ai/api-reference/model-monitoring-drift-and-quality/get-model-monitorv2apimodel-model-quality-user-query

/api-specs/cloud/public-api.json get /model-monitor/v2/api/model/{model_id}/model-quality-user-query



# Get scheduled drift check data
Source: https://docs.domino.ai/api-reference/model-monitoring-drift-and-quality/get-model-monitorv2apimodel-scheduled-check-drift-data

/api-specs/cloud/public-api.json get /model-monitor/v2/api/model/{model_id}/{scheduler_id}/scheduled-check-drift-data



# List dataset jobs
Source: https://docs.domino.ai/api-reference/model-monitoring-drift-and-quality/post-model-monitorv2api-dataset-jobs

/api-specs/cloud/public-api.json post /model-monitor/v2/api/{model_id}/dataset-jobs



# Analyze drift
Source: https://docs.domino.ai/api-reference/model-monitoring-drift-and-quality/post-model-monitorv2apimodel-analyse-drift

/api-specs/cloud/public-api.json post /model-monitor/v2/api/model/{model_id}/analyse-drift



# Analyze model quality
Source: https://docs.domino.ai/api-reference/model-monitoring-drift-and-quality/post-model-monitorv2apimodel-analyze-model-quality

/api-specs/cloud/public-api.json post /model-monitor/v2/api/model/{model_id}/analyze-model-quality



# Save drift scheduler query
Source: https://docs.domino.ai/api-reference/model-monitoring-drift-and-quality/post-model-monitorv2apimodel-save-scheduler-query

/api-specs/cloud/public-api.json post /model-monitor/v2/api/model/{model_id}/save-scheduler-query



# Delete model
Source: https://docs.domino.ai/api-reference/model-monitoring-models/delete-model-monitorv2apimodel

/api-specs/cloud/public-api.json delete /model-monitor/v2/api/model



# Remove model collaborator
Source: https://docs.domino.ai/api-reference/model-monitoring-models/delete-model-monitorv2apimodel-collaborators

/api-specs/cloud/public-api.json delete /model-monitor/v2/api/model/{model_id}/collaborators
Deletes an existing collaborator for a standalone DMM model



# Remove model tag
Source: https://docs.domino.ai/api-reference/model-monitoring-models/delete-model-monitorv2apimodel-tags

/api-specs/cloud/public-api.json delete /model-monitor/v2/api/model/{model_id}/tags
Remove a tag from a DMM model



# Get model summary
Source: https://docs.domino.ai/api-reference/model-monitoring-models/get-model-monitorv2apiget_model_summary

/api-specs/cloud/public-api.json get /model-monitor/v2/api/get_model_summary
Gets a DMM model summary either via a unique model identifier or via a unique combination of workbench model id and workbench model version id



# Get model
Source: https://docs.domino.ai/api-reference/model-monitoring-models/get-model-monitorv2apimodel

/api-specs/cloud/public-api.json get /model-monitor/v2/api/model
Gets a DMM model either via a unique model identifier or via a unique combination of workbench model id and workbench model version id



# List model collaborators
Source: https://docs.domino.ai/api-reference/model-monitoring-models/get-model-monitorv2apimodel-collaborators

/api-specs/cloud/public-api.json get /model-monitor/v2/api/model/{model_id}/collaborators
Fetches all collaborators for a standalone DMM model



# List model tags
Source: https://docs.domino.ai/api-reference/model-monitoring-models/get-model-monitorv2apimodel-tags

/api-specs/cloud/public-api.json get /model-monitor/v2/api/model/{model_id}/tags
List tags that can be added to a DMM model



# Get ground truth traffic
Source: https://docs.domino.ai/api-reference/model-monitoring-models/get-model-monitorv2apimodel-trafficground-truth

/api-specs/cloud/public-api.json get /model-monitor/v2/api/model/{model_id}/traffic/ground-truth



# Get model variables summary
Source: https://docs.domino.ai/api-reference/model-monitoring-models/get-model-monitorv2apimodel-variablessummary

/api-specs/cloud/public-api.json get /model-monitor/v2/api/model/{model_id}/variables/summary



# Get prediction traffic
Source: https://docs.domino.ai/api-reference/model-monitoring-models/get-model-monitorv2apimodelget-prediction-traffic

/api-specs/cloud/public-api.json get /model-monitor/v2/api/model/get-prediction-traffic



# List models
Source: https://docs.domino.ai/api-reference/model-monitoring-models/get-model-monitorv2apimodels

/api-specs/cloud/public-api.json get /model-monitor/v2/api/models



# Get summary for workbench model versions
Source: https://docs.domino.ai/api-reference/model-monitoring-models/get-summary-for-workbench-model-versions

/api-specs/cloud/public-api.json get /model-monitor/v2/api/workbench/model/version/summary



# Update model
Source: https://docs.domino.ai/api-reference/model-monitoring-models/patch-model-monitorv2apimodel

/api-specs/cloud/public-api.json patch /model-monitor/v2/api/model
Update one or more editable attributes of a DMM model.



# Archive model tag
Source: https://docs.domino.ai/api-reference/model-monitoring-models/patch-model-monitorv2apimodel-tag-archive

/api-specs/cloud/public-api.json patch /model-monitor/v2/api/model-tag/{tag_id}/archive
Archive a model tag



# Unarchive model tag
Source: https://docs.domino.ai/api-reference/model-monitoring-models/patch-model-monitorv2apimodel-tag-unarchive

/api-specs/cloud/public-api.json patch /model-monitor/v2/api/model-tag/{tag_id}/unarchive
Unarchive a model tag



# Add model collaborators
Source: https://docs.domino.ai/api-reference/model-monitoring-models/post-model-monitorv2apimodel-collaborators

/api-specs/cloud/public-api.json post /model-monitor/v2/api/model/{model_id}/collaborators
Adds collaborators to a standalone DMM model



# Add model tags
Source: https://docs.domino.ai/api-reference/model-monitoring-models/post-model-monitorv2apimodel-tags

/api-specs/cloud/public-api.json post /model-monitor/v2/api/model/{model_id}/tags
Adds tags to a DMM model



# Register model
Source: https://docs.domino.ai/api-reference/model-monitoring-models/put-model-monitorv2apimodel

/api-specs/cloud/public-api.json put /model-monitor/v2/api/model



# Update collaborator role
Source: https://docs.domino.ai/api-reference/model-monitoring-models/put-model-monitorv2apimodel-collaborators

/api-specs/cloud/public-api.json put /model-monitor/v2/api/model/{model_id}/collaborators
Updates an existing collaborator for a standalone DMM model



# Register dataset config
Source: https://docs.domino.ai/api-reference/model-monitoring-models/put-model-monitorv2apimodel-register-dataset

/api-specs/cloud/public-api.json put /model-monitor/v2/api/model/{model_id}/register-dataset/{dataset_type}



# Update model visibility
Source: https://docs.domino.ai/api-reference/model-monitoring-models/put-model-monitorv2apimodel-visibility

/api-specs/cloud/public-api.json put /model-monitor/v2/api/model/{model_id}/visibility
Updates the visibility of a standalone DMM model



# Delete a list of personal access tokens
Source: https://docs.domino.ai/api-reference/personal-access-tokens-admin/delete-a-list-of-personal-access-tokens

/api-specs/cloud/public-api.json post /api/pat/v1/users/tokens/delete/bulk



# Delete a personal access token of a User
Source: https://docs.domino.ai/api-reference/personal-access-tokens-admin/delete-a-personal-access-token-of-a-user

/api-specs/cloud/public-api.json delete /api/pat/v1/users/{userId}/tokens/{patId}



# Delete all personal access token of a User
Source: https://docs.domino.ai/api-reference/personal-access-tokens-admin/delete-all-personal-access-token-of-a-user

/api-specs/cloud/public-api.json delete /api/pat/v1/users/{userId}/tokens



# Invalidate a list of personal access tokens
Source: https://docs.domino.ai/api-reference/personal-access-tokens-admin/invalidate-a-list-of-personal-access-tokens

/api-specs/cloud/public-api.json post /api/pat/v1/users/tokens/invalidate/bulk



# Invalidate a personal access token of a User
Source: https://docs.domino.ai/api-reference/personal-access-tokens-admin/invalidate-a-personal-access-token-of-a-user

/api-specs/cloud/public-api.json post /api/pat/v1/users/{userId}/tokens/{patId}/invalidate



# Invalidate all personal access tokens of a User
Source: https://docs.domino.ai/api-reference/personal-access-tokens-admin/invalidate-all-personal-access-tokens-of-a-user

/api-specs/cloud/public-api.json post /api/pat/v1/users/{userId}/tokens/invalidate



# Retrieves a list of personal access tokens
Source: https://docs.domino.ai/api-reference/personal-access-tokens-admin/retrieves-a-list-of-personal-access-tokens

/api-specs/cloud/public-api.json get /api/pat/v1/users/tokens



# Retrieves a list of personal access tokens
Source: https://docs.domino.ai/api-reference/personal-access-tokens-admin/retrieves-a-list-of-personal-access-tokens-1

/api-specs/cloud/public-api.json get /api/pat/v1/users/{userId}/tokens



# Create a personal access token
Source: https://docs.domino.ai/api-reference/personal-access-tokens/create-a-personal-access-token

/api-specs/cloud/public-api.json post /api/pat/v1/tokens



# Delete a list of personal access tokens
Source: https://docs.domino.ai/api-reference/personal-access-tokens/delete-a-list-of-personal-access-tokens

/api-specs/cloud/public-api.json post /api/pat/v1/tokens/delete/bulk



# Delete a personal access token
Source: https://docs.domino.ai/api-reference/personal-access-tokens/delete-a-personal-access-token

/api-specs/cloud/public-api.json delete /api/pat/v1/tokens/{patId}



# Delete all their personal access token
Source: https://docs.domino.ai/api-reference/personal-access-tokens/delete-all-their-personal-access-token

/api-specs/cloud/public-api.json delete /api/pat/v1/tokens



# Invalidate a list of personal access tokens
Source: https://docs.domino.ai/api-reference/personal-access-tokens/invalidate-a-list-of-personal-access-tokens

/api-specs/cloud/public-api.json post /api/pat/v1/tokens/invalidate/bulk



# Invalidate a personal access token
Source: https://docs.domino.ai/api-reference/personal-access-tokens/invalidate-a-personal-access-token

/api-specs/cloud/public-api.json post /api/pat/v1/tokens/{patId}/invalidate



# Invalidate all their personal access token
Source: https://docs.domino.ai/api-reference/personal-access-tokens/invalidate-all-their-personal-access-token

/api-specs/cloud/public-api.json post /api/pat/v1/tokens/invalidate



# Retrieves a list of personal access tokens
Source: https://docs.domino.ai/api-reference/personal-access-tokens/retrieves-a-list-of-personal-access-tokens

/api-specs/cloud/public-api.json get /api/pat/v1/tokens



# Retrieves the min and max expiration time allowed for a personal access token
Source: https://docs.domino.ai/api-reference/personal-access-tokens/retrieves-the-min-and-max-expiration-time-allowed-for-a-personal-access-token

/api-specs/cloud/public-api.json get /api/pat/v1/tokens/expiration-policy



# Retrieves the scopes that can be granted to a personal access token
Source: https://docs.domino.ai/api-reference/personal-access-tokens/retrieves-the-scopes-that-can-be-granted-to-a-personal-access-token

/api-specs/cloud/public-api.json get /api/pat/v1/tokens/grantable-scopes



# Create Admin Config
Source: https://docs.domino.ai/api-reference/ppm/create-admin-config

/api-specs/cloud/public-api.json post /api/v4/ppm/admin/config
Create new PPM admin configuration



# Delete Admin Config
Source: https://docs.domino.ai/api-reference/ppm/delete-admin-config

/api-specs/cloud/public-api.json delete /api/v4/ppm/admin/config/{configId}
Delete a specific PPM admin configuration by ID



# Get Admin Config
Source: https://docs.domino.ai/api-reference/ppm/get-admin-config

/api-specs/cloud/public-api.json get /api/v4/ppm/admin/config
Get PPM admin configuration



# Get Workspace PPM Config with Repository Detection
Source: https://docs.domino.ai/api-reference/ppm/get-workspace-ppm-config-with-repository-detection

/api-specs/cloud/public-api.json get /api/v4/ppm/projects/{projectId}/workspaces/{workspaceId}/config
Get PPM configuration including repositories detected from workspace renv.lock file



# Render PPM Installation Snippet
Source: https://docs.domino.ai/api-reference/ppm/render-ppm-installation-snippet

/api-specs/cloud/public-api.json post /api/v4/ppm/snippet
Generate a copyable Rscript command for PPM package installation with automatic biocVersion detection when needed



# Search PPM Snapshots with Workspace Repository Targeting
Source: https://docs.domino.ai/api-reference/ppm/search-ppm-snapshots-with-workspace-repository-targeting

/api-specs/cloud/public-api.json post /api/v4/ppm/search
Search for ascending snapshot candidates that satisfy the provided package constraints. When workspace repositories are provided, the search is targeted to matching server repositories.



# Update Admin Config
Source: https://docs.domino.ai/api-reference/ppm/update-admin-config

/api-specs/cloud/public-api.json patch /api/v4/ppm/admin/config/{configId}
Partially update a specific PPM admin configuration by ID (only provided fields will be updated)



# Remove Git credentials from service account
Source: https://docs.domino.ai/api-reference/service-accounts/delete-apiserviceaccountsv1serviceaccounts-gitcredentials

/api-specs/cloud/public-api.json delete /api/serviceAccounts/v1/serviceAccounts/{serviceAccountIdpId}/gitcredentials/{credentialId}



# Delete service account token
Source: https://docs.domino.ai/api-reference/service-accounts/delete-apiserviceaccountsv1serviceaccounts-tokens

/api-specs/cloud/public-api.json delete /api/serviceAccounts/v1/serviceAccounts/{serviceAccountIdpId}/tokens/{tokenName}



# List service accounts
Source: https://docs.domino.ai/api-reference/service-accounts/get-apiserviceaccountsv1serviceaccounts

/api-specs/cloud/public-api.json get /api/serviceAccounts/v1/serviceAccounts



# List service account Git credentials
Source: https://docs.domino.ai/api-reference/service-accounts/get-apiserviceaccountsv1serviceaccounts-gitcredentials

/api-specs/cloud/public-api.json get /api/serviceAccounts/v1/serviceAccounts/{serviceAccountIdpId}/gitcredentials



# List service account tokens
Source: https://docs.domino.ai/api-reference/service-accounts/get-apiserviceaccountsv1serviceaccounts-tokens

/api-specs/cloud/public-api.json get /api/serviceAccounts/v1/serviceAccounts/{serviceAccountIdpId}/tokens



# Get service account token
Source: https://docs.domino.ai/api-reference/service-accounts/get-apiserviceaccountsv1serviceaccounts-tokens-1

/api-specs/cloud/public-api.json get /api/serviceAccounts/v1/serviceAccounts/{serviceAccountIdpId}/tokens/{tokenName}



# Create service account
Source: https://docs.domino.ai/api-reference/service-accounts/post-apiserviceaccountsv1serviceaccounts

/api-specs/cloud/public-api.json post /api/serviceAccounts/v1/serviceAccounts



# Activate service account
Source: https://docs.domino.ai/api-reference/service-accounts/post-apiserviceaccountsv1serviceaccounts-activate

/api-specs/cloud/public-api.json post /api/serviceAccounts/v1/serviceAccounts/{serviceAccountIdpId}/activate



# Deactivate service account
Source: https://docs.domino.ai/api-reference/service-accounts/post-apiserviceaccountsv1serviceaccounts-deactivate

/api-specs/cloud/public-api.json post /api/serviceAccounts/v1/serviceAccounts/{serviceAccountIdpId}/deactivate



# Add password Git credentials to service account
Source: https://docs.domino.ai/api-reference/service-accounts/post-apiserviceaccountsv1serviceaccounts-passwordgitcredentials

/api-specs/cloud/public-api.json post /api/serviceAccounts/v1/serviceAccounts/{serviceAccountIdpId}/passwordgitcredentials



# Reset service account password
Source: https://docs.domino.ai/api-reference/service-accounts/post-apiserviceaccountsv1serviceaccounts-resetpassword

/api-specs/cloud/public-api.json post /api/serviceAccounts/v1/serviceAccounts/{serviceAccountIdpId}/resetpassword



# Add SSH Git credentials to service account
Source: https://docs.domino.ai/api-reference/service-accounts/post-apiserviceaccountsv1serviceaccounts-sshgitcredentials

/api-specs/cloud/public-api.json post /api/serviceAccounts/v1/serviceAccounts/{serviceAccountIdpId}/sshgitcredentials



# Add token Git credentials to service account
Source: https://docs.domino.ai/api-reference/service-accounts/post-apiserviceaccountsv1serviceaccounts-tokengitcredentials

/api-specs/cloud/public-api.json post /api/serviceAccounts/v1/serviceAccounts/{serviceAccountIdpId}/tokengitcredentials



# Create service account token
Source: https://docs.domino.ai/api-reference/service-accounts/post-apiserviceaccountsv1serviceaccounts-tokens

/api-specs/cloud/public-api.json post /api/serviceAccounts/v1/serviceAccounts/{serviceAccountIdpId}/tokens



# Invalidate service account token
Source: https://docs.domino.ai/api-reference/service-accounts/post-apiserviceaccountsv1serviceaccounts-tokens-invalidate

/api-specs/cloud/public-api.json post /api/serviceAccounts/v1/serviceAccounts/{serviceAccountIdpId}/tokens/{tokenName}/invalidate



# Add a user to an org
Source: https://docs.domino.ai/api-reference/users-and-organizations/add-a-user-to-an-org

/api-specs/cloud/public-api.json put /api/organizations/v1/organizations/{organizationId}/user
Add a new user to an Organization. Required permissions: `EditMembers`



# Create an organization
Source: https://docs.domino.ai/api-reference/users-and-organizations/create-an-organization

/api-specs/cloud/public-api.json post /api/organizations/v1/organizations
Create a new Organization. Required permissions: `Must be logged in user`



# Get a user by ID
Source: https://docs.domino.ai/api-reference/users-and-organizations/get-a-user-by-id

/api-specs/cloud/public-api.json get /api/users/v1/user/{userId}
Retrieve a single user by their ID. Required permissions: `None` (subject to org-scoped visibility, same as the list-users endpoint).



# Get all organizations. Only accessible to admin users.
Source: https://docs.domino.ai/api-reference/users-and-organizations/get-all-organizations-only-accessible-to-admin-users

/api-specs/cloud/public-api.json get /api/organizations/v1/organizations/all
Get all organizations. Required permissions: `ManageOrganizations`



# Get all users visible to the current user
Source: https://docs.domino.ai/api-reference/users-and-organizations/get-all-users-visible-to-the-current-user

/api-specs/cloud/public-api.json get /api/users/v1/users
Retrieves all users visible to the current user. Required permissions: `None`



# Get an organization by Id
Source: https://docs.domino.ai/api-reference/users-and-organizations/get-an-organization-by-id

/api-specs/cloud/public-api.json get /api/organizations/v1/organizations/{organizationId}
Retrieve an Organization by its Id. Required permissions: `ViewOrganization`



# Get git credential accessor for a User
Source: https://docs.domino.ai/api-reference/users-and-organizations/get-git-credential-accessor-for-a-user

/api-specs/cloud/public-api.json get /api/users/beta/credentials/{userId}
Retrieve a users git credentials. Required permissions: `UpdateUser`. *Note:* This is a beta endpoint with known limitations.



# Get roles assigned to a user
Source: https://docs.domino.ai/api-reference/users-and-organizations/get-roles-assigned-to-a-user

/api-specs/cloud/public-api.json get /api/users/v1/user/{userId}/roles
Retrieve the roles assigned to a user. Required permissions: `SetUserSystemRoles`



# Get the current user
Source: https://docs.domino.ai/api-reference/users-and-organizations/get-the-current-user

/api-specs/cloud/public-api.json get /api/users/v1/self
Retrieve the current user. Required permissions: `None`



# Get the Organizations for a user
Source: https://docs.domino.ai/api-reference/users-and-organizations/get-the-organizations-for-a-user

/api-specs/cloud/public-api.json get /api/organizations/v1/organizations
Retrieve all Organizations of which this user is a member. Required permissions: `None`



# Remove a user from an org
Source: https://docs.domino.ai/api-reference/users-and-organizations/remove-a-user-from-an-org

/api-specs/cloud/public-api.json delete /api/organizations/v1/organizations/{organizationId}/user
Remove a user from an Organization. Required permissions: `EditMembers`



# Set roles for a user
Source: https://docs.domino.ai/api-reference/users-and-organizations/set-roles-for-a-user

/api-specs/cloud/public-api.json post /api/users/v1/user/{userId}/roles
Set the complete list of roles for a user, replacing any previously assigned roles. Required permissions: `SetUserSystemRoles`



# Update git credential accessor for a User
Source: https://docs.domino.ai/api-reference/users-and-organizations/update-git-credential-accessor-for-a-user

/api-specs/cloud/public-api.json put /api/users/v1/user/{userId}/tokenCredentials/{credentialId}
Update a user's git credential. Required permissions: `UpdateUser`. *Note:* This only supports updating GitHub and GitHub Enterprise PATs.



# Add a policy to govern a bundle
Source: https://docs.domino.ai/api-reference/governance-bundles/add-a-policy-to-govern-a-bundle

/api-specs/cloud/public-api.json post /api/governance/v1/rpc/add-policies-to-bundle
Add a policy to govern a bundle



# Archive all monitored models for a bundle
Source: https://docs.domino.ai/api-reference/governance-bundles/archive-all-monitored-models-for-a-bundle

/api-specs/cloud/public-api.json post /api/governance/v1/bundles/{id}/archive-endpoints
Archive all monitored models for a bundle. This is typically done when a bundle is archived.



# Attach a Domino object to a bundle
Source: https://docs.domino.ai/api-reference/governance-bundles/attach-a-domino-object-to-a-bundle

/api-specs/cloud/public-api.json post /api/governance/v1/bundles/{id}/attachments
Attach a Domino object to a bundle, as supporting evidence or as the governed entity itself. `type` and `identifier` together determine what's referenced; the required `identifier` shape depends on `type`:
| type | identifier shape |
|---|---|
| `ModelVersion` | `{"name": string, "version": number}` |
| `Report` (a file in the project's filesystem, not a generic report document) | `{"filename": string, "commit": string, "source": "DFS" / "GBP", "branch": string}` |
| `DatasetSnapshotFile` | `{"datasetId": string, "snapshotId": string, "filename": string}` |
| `Endpoint` | `{"endpointVersionId": string, "endpointId": string, "name": string, "version": number}` |
| `NetAppVolumeSnapshotFile` | `{"volumeId": string, "snapshotId": string, "filename": string}` |
| `FlowArtifactFile` | `{"artifactId": string, "filename": string, "executionName": string, "executionWorkflowName": string, "executionWorkflowVersion": string}` |
`id` is the bundle ID, from `POST /bundles` or `GET /bundles`.
Returns 409 if the bundle is already completed or archived (read-only).
Related:
- GET /attachment-overviews (list existing attachments across bundles)
- POST /rpc/delete-attachment-and-results (remove an attachment along with drafts and results on them)
- POST /rpc/bulk-add-attachments (attach to multiple bundles in one call)



# Bulk add attachments to bundles
Source: https://docs.domino.ai/api-reference/governance-bundles/bulk-add-attachments-to-bundles

/api-specs/cloud/public-api.json post /api/governance/v1/rpc/bulk-add-attachments
Add attachments to multiple bundles in a single request.
Validation (duplicate bundleId, bundle existence, project authorization, read-only state)
is performed up front and any failure rejects the entire request before any write.
The database insert of the resulting attachments is a single batched write, so the
attachment rows are atomic. Rules-gateway propagation runs after the insert and is
best-effort: if it fails the attachments remain persisted and the caller should retry —
deduplication makes retries safe (already-attached items are skipped). This matches the
behavior of the single-attach endpoint.



# Bulk reassign bundle stages
Source: https://docs.domino.ai/api-reference/governance-bundles/bulk-reassign-bundle-stages

/api-specs/cloud/public-api.json post /api/governance/v1/bundle-stages/bulk-reassign
Reassign one or more bundle stages to a single assignee. Each item targets a specific stage ({bundleId, stageId}) or just a bundle ({bundleId}), in which case its current stage is used. Returns a per-item result list. Requires manage permission on every project the bundles belong to.



# Bulk update bundles
Source: https://docs.domino.ai/api-reference/governance-bundles/bulk-update-bundles

/api-specs/cloud/public-api.json post /api/governance/v1/bundles/bulk-update
Bulk update bundles



# Create a bundle to collect evidence for a governed entity
Source: https://docs.domino.ai/api-reference/governance-bundles/create-a-bundle-to-collect-evidence-for-a-governed-entity

/api-specs/cloud/public-api.json post /api/governance/v1/bundles
Create a bundle — the container that holds all evidence collected for a governed entity — under a project and governed by one or more policies. `policyId` must reference an existing Published policy — get it from `GET /policy-overviews` (filter by `status=Published`) or `GET /policies/{id}`. `projectId` is the Domino project ID the bundle belongs to.
If `resolveNameOnConflict` is true and the requested name is already taken in the project, the next available variant (`<name> 2`, `<name> 3`, ...) is used instead of failing.
If `enforcePoliciesFor` is set (e.g. `App`, `Model`), the mandatory policies configured for that target are also attached and cannot later be deactivated. When set, `policyId` becomes optional: if omitted, the primary policy is derived from the target's PolicyEnforcer; if supplied, that `policyId` is always used as the primary.
Returns 400 with `"code": "BUNDLE_REQUIRES_POLICY"` when no `policyId` is supplied and the target has no enabled enforcer with a published policy.
Related:
- POST /rpc/compute-policy (read the bundle's evidence/approval state for a given policy after creation)
- GET /policy-overviews (find a published policyId to attach)
- GET /policies/{id}/definition (read the policy's evidence/approval structure)



# Create revalidation config for an approval
Source: https://docs.domino.ai/api-reference/governance-bundles/create-revalidation-config-for-an-approval

/api-specs/cloud/public-api.json post /api/governance/v1/bundles/{id}/approvals/{approvalId}/revalidation
Create revalidation config for an approval



# Delete an attachment along with its drafts and results
Source: https://docs.domino.ai/api-reference/governance-bundles/delete-an-attachment-along-with-its-drafts-and-results

/api-specs/cloud/public-api.json post /api/governance/v1/rpc/delete-attachment-and-results
Deletes the attachment and cleans up any drafts and results tied to its evidence in the same transaction.



# Delete attachment
Source: https://docs.domino.ai/api-reference/governance-bundles/delete-attachment

/api-specs/cloud/public-api.json delete /api/governance/v1/bundles/{id}/attachments/{attachmentID}
Deprecated: use `POST /rpc/delete-attachment-and-results` instead. This endpoint only detaches the attachment and does not delete drafts or results tied to its evidence, which can leave orphaned data behind.



# Delete bundle by ID
Source: https://docs.domino.ai/api-reference/governance-bundles/delete-bundle-by-id

/api-specs/cloud/public-api.json delete /api/governance/v1/bundles/{id}
Delete bundle by ID



# Delete revalidation config from an approval
Source: https://docs.domino.ai/api-reference/governance-bundles/delete-revalidation-config-from-an-approval

/api-specs/cloud/public-api.json delete /api/governance/v1/bundles/{id}/approvals/{approvalId}/revalidation
Delete revalidation config from an approval



# Download a PDF report for a bundle
Source: https://docs.domino.ai/api-reference/governance-bundles/download-a-pdf-report-for-a-bundle

/api-specs/cloud/public-api.json get /api/governance/v1/bundles/{id}/report
Download a PDF report for a bundle. Includes all stages, evidence, and results



# Get bundle by ID
Source: https://docs.domino.ai/api-reference/governance-bundles/get-bundle-by-id

/api-specs/cloud/public-api.json get /api/governance/v1/bundles/{id}
Get bundle by ID



# Get bundle gates
Source: https://docs.domino.ai/api-reference/governance-bundles/get-bundle-gates

/api-specs/cloud/public-api.json get /api/governance/v1/bundles/{id}/gates
Get bundle gates



# Get the current status of every approval in a bundle
Source: https://docs.domino.ai/api-reference/governance-bundles/get-the-current-status-of-every-approval-in-a-bundle

/api-specs/cloud/public-api.json get /api/governance/v1/bundles/{id}/approvals
Returns each `Approval` in the bundle with its current `status` (`PendingSubmission`, `PendingReview`, `Approved`, `ConditionallyApproved`, `PendingExpiration`, `Expired`), across all policies attached to the bundle. This is the lightest-weight way to poll for approval status — use `POST /rpc/compute-policy` instead if you also need results or findings for a specific policy in the same response.
`Approved` and `ConditionallyApproved` are the two "check passed" states; `ConditionallyApproved` may still have `findingsInfo` findings to resolve. There is no absolutely terminal status — even `Expired` can transition back to `PendingReview` via revalidation.
`id` is the bundle ID from `POST /bundles`.
Related:
- POST /rpc/compute-policy (fuller snapshot including results/findings, scoped to one policy)
- POST /rpc/publish-approval-event (advance an approval returned here to its next status)



# List attachment overviews
Source: https://docs.domino.ai/api-reference/governance-bundles/list-attachment-overviews

/api-specs/cloud/public-api.json get /api/governance/v1/attachment-overviews
Retrieve a list of attachment overviews with optional filters, search, and pagination



# List bundle stages (Stage View)
Source: https://docs.domino.ai/api-reference/governance-bundles/list-bundle-stages-stage-view

/api-specs/cloud/public-api.json get /api/governance/v1/bundle-stages
Returns one row per stage of each bundle's policy. Non-admin callers see only stages in projects they can access; admins see all. Each row's status (NotStarted, InProgress, Done) reflects the stage's current evidence, approvals, and findings.



# List bundles
Source: https://docs.domino.ai/api-reference/governance-bundles/list-bundles

/api-specs/cloud/public-api.json get /api/governance/v1/bundles
List bundles



# Replace a bundle's primary policy with a different policy
Source: https://docs.domino.ai/api-reference/governance-bundles/replace-a-bundles-primary-policy-with-a-different-policy

/api-specs/cloud/public-api.json post /api/governance/v1/rpc/replace-bundle-primary-policy
Replaces the bundle's primary policy with a completely different policy. Set disableOldPolicy=true to deactivate the old primary (default: false).



# Replace a policy version in a bundle
Source: https://docs.domino.ai/api-reference/governance-bundles/replace-a-policy-version-in-a-bundle

/api-specs/cloud/public-api.json post /api/governance/v1/rpc/upgrade-bundle-policy
Atomically deactivates the current policy version and activates a new version. Set preview=true to get the impact without making changes.



# Set monitored model as governed
Source: https://docs.domino.ai/api-reference/governance-bundles/set-monitored-model-as-governed

/api-specs/cloud/public-api.json post /api/governance/v1/bundles/{id}/register-endpoint
Add monitor model entry and set it as governed



# Update approval by ID
Source: https://docs.domino.ai/api-reference/governance-bundles/update-approval-by-id

/api-specs/cloud/public-api.json post /api/governance/v1/rpc/update-approval
Update approval by ID



# Update bundle by ID
Source: https://docs.domino.ai/api-reference/governance-bundles/update-bundle-by-id

/api-specs/cloud/public-api.json patch /api/governance/v1/bundles/{id}
Update bundle by ID. Exactly one field must be provided. When evidenceRestricted is set to true, evidence answers, policy additions, removals, and upgrades are all blocked on the bundle (409). It is a one-way toggle and cannot be unset.



# Update bundle stage by ID
Source: https://docs.domino.ai/api-reference/governance-bundles/update-bundle-stage-by-id

/api-specs/cloud/public-api.json patch /api/governance/v1/bundles/{id}/stages/{stageID}
Update bundle stage by ID



# Update monitored model entry
Source: https://docs.domino.ai/api-reference/governance-bundles/update-monitored-model-entry

/api-specs/cloud/public-api.json put /api/governance/v1/bundles/{id}/update-endpoint/{endpointID}
Update monitored model entry



# Update revalidation config for an approval
Source: https://docs.domino.ai/api-reference/governance-bundles/update-revalidation-config-for-an-approval

/api-specs/cloud/public-api.json patch /api/governance/v1/bundles/{id}/approvals/{approvalId}/revalidation
Update revalidation config for an approval



# Update the policy of a bundle
Source: https://docs.domino.ai/api-reference/governance-bundles/update-the-policy-of-a-bundle

/api-specs/cloud/public-api.json patch /api/governance/v1/rpc/update-bundle-policy
Update the policy of a bundle



# Bulk update finding severity or due date
Source: https://docs.domino.ai/api-reference/governance-evidence-and-results/bulk-update-finding-severity-or-due-date

/api-specs/cloud/public-api.json put /api/governance/v1/rpc/bulk-edit-findings
Bulk update finding severity or due date given a list of finding IDs



# Create new finding
Source: https://docs.domino.ai/api-reference/governance-evidence-and-results/create-new-finding

/api-specs/cloud/public-api.json post /api/governance/v1/findings
Create new finding



# Delete evidence template by ID
Source: https://docs.domino.ai/api-reference/governance-evidence-and-results/delete-evidence-template-by-id

/api-specs/cloud/public-api.json delete /api/governance/v1/evidence-templates/{id}
Delete evidence template by ID



# Get evidence template by ID
Source: https://docs.domino.ai/api-reference/governance-evidence-and-results/get-evidence-template-by-id

/api-specs/cloud/public-api.json get /api/governance/v1/evidence-templates/{id}
Get evidence template by ID



# Get evidence template definition YAML by ID
Source: https://docs.domino.ai/api-reference/governance-evidence-and-results/get-evidence-template-definition-yaml-by-id

/api-specs/cloud/public-api.json get /api/governance/v1/evidence-templates/{id}/definition
Get evidence template definition YAML by ID



# Get finding by ID
Source: https://docs.domino.ai/api-reference/governance-evidence-and-results/get-finding-by-id

/api-specs/cloud/public-api.json get /api/governance/v1/findings/{id}
Get finding by ID



# List evidence template paginated
Source: https://docs.domino.ai/api-reference/governance-evidence-and-results/list-evidence-template-paginated

/api-specs/cloud/public-api.json get /api/governance/v1/evidence-templates
List evidence template paginated



# List Findings
Source: https://docs.domino.ai/api-reference/governance-evidence-and-results/list-findings

/api-specs/cloud/public-api.json get /api/governance/v1/bundles/{id}/findings
List Findings



# List results
Source: https://docs.domino.ai/api-reference/governance-evidence-and-results/list-results

/api-specs/cloud/public-api.json get /api/governance/v1/results
List results



# Read back the latest submitted result(s) for a bundle
Source: https://docs.domino.ai/api-reference/governance-evidence-and-results/read-back-the-latest-submitted-results-for-a-bundle

/api-specs/cloud/public-api.json get /api/governance/v1/results/latest
Returns the latest `Result` per artifact for the given bundle/policy.
`bundleId` comes from `POST /bundles`. `artifactID` (repeatable) narrows to specific artifacts; omit it to get every artifact's latest result.
Related:
- POST /rpc/submit-result-to-policy (write the result this endpoint reads, and recompute classification)
- POST /rpc/compute-policy (read results alongside approvals and findings in one call)



# Submit a final result for an artifact (does not recompute classification)
Source: https://docs.domino.ai/api-reference/governance-evidence-and-results/submit-a-final-result-for-an-artifact-does-not-recompute-classification

/api-specs/cloud/public-api.json post /api/governance/v1/results
Submit a final result for an artifact. This endpoint does not recompute the bundle's classification value.
If the artifact belongs to the policy's classification rule or simple classification question, submitting it here will leave `classificationValue` stale and inconsistent with the latest results. Use `/rpc/submit-result-to-policy` instead for any artifact that may affect classification.
`evidenceId` comes from the policy's evidence definition (`GET /policies/{id}`, `stages[].evidenceSet[].id`). `bundleId` comes from `POST /bundles`.
`content` is a map of `artifactId` → value; the required shape depends on the artifact's `artifactType`/`details.type` — see `POST /rpc/submit-result-to-policy` for the full content-shape table.
Related:
- POST /rpc/submit-result-to-policy (recommended — same content contract, plus recomputes classification and returns the current approvals/findings in one call; full content-shape table)
- GET /policies/{id} (read each artifact's `details` to determine its content shape before submitting)
- GET /results/latest (read back results for a bundle)
- POST /rpc/compute-policy (read all approvals, results, and findings for a given policy on the bundle in one call)



# Update evidence template definition by ID
Source: https://docs.domino.ai/api-reference/governance-evidence-and-results/update-evidence-template-definition-by-id

/api-specs/cloud/public-api.json put /api/governance/v1/evidence-templates/{id}/definition
Update evidence template definition by ID



# Update evidence template status by ID
Source: https://docs.domino.ai/api-reference/governance-evidence-and-results/update-evidence-template-status-by-id

/api-specs/cloud/public-api.json put /api/governance/v1/evidence-templates/{id}/status
Update evidence template status by ID



# Update the finding by ID
Source: https://docs.domino.ai/api-reference/governance-evidence-and-results/update-the-finding-by-id

/api-specs/cloud/public-api.json put /api/governance/v1/findings/{id}
Update the finding by ID



# Upsert evidence template
Source: https://docs.domino.ai/api-reference/governance-evidence-and-results/upsert-evidence-template

/api-specs/cloud/public-api.json put /api/governance/v1/evidence-templates/{id}
Upsert evidence template



# [BETA] Analyze model monitor metrics
Source: https://docs.domino.ai/api-reference/governance-operations/[beta]-analyze-model-monitor-metrics

/api-specs/cloud/public-api.json post /api/governance/v1/rpc/analyze-monitor-model
[BETA] Analyze model monitor metrics. Endpoint is subject to change, should only be used internally.



# Attach each App's mandatory policies to its governed bundle
Source: https://docs.domino.ai/api-reference/governance-operations/attach-each-apps-mandatory-policies-to-its-governed-bundle

/api-specs/cloud/public-api.json post /api/governance/v1/rpc/attach-app-enforced-policies
For each App, attaches mandatory policies at their latest version to its governed bundle, creating one if none exists. Per-app outcomes are reported in the 200 body; an unknown App is a per-app 404.



# Clone a bundle into a new, standalone bundle
Source: https://docs.domino.ai/api-reference/governance-operations/clone-a-bundle-into-a-new-standalone-bundle

/api-specs/cloud/public-api.json post /api/governance/v1/rpc/clone-bundle
Clones a bundle in its own project at the same policy versions. Copies attachments, evidence, findings, approvals (including revalidation schedules), stage assignees, classification value, evidence-restricted flag, and state. Source is unchanged.
Deactivated policies, comments, and bundle metadata (createdAt/audit trail) are not copied. Set `resolveNameOnConflict` to auto-append a numeric suffix on name conflicts.



# Create a policy
Source: https://docs.domino.ai/api-reference/governance-policies/create-a-policy

/api-specs/cloud/public-api.json post /api/governance/v1/policies
Create a policy



# Create a policy enforcer
Source: https://docs.domino.ai/api-reference/governance-policies/create-a-policy-enforcer

/api-specs/cloud/public-api.json post /api/governance/v1/policy-enforcers
Create the mandatory policy list for a given product type. Only one policy enforcer may exist per enforcedOn target. The first ID in policyIds is treated as the optimistic primary policy; subsequent IDs are additional mandatory policies.



# Create a policy version
Source: https://docs.domino.ai/api-reference/governance-policies/create-a-policy-version

/api-specs/cloud/public-api.json post /api/governance/v1/policies/{id}/versions
Create a policy version



# Create or update a policy
Source: https://docs.domino.ai/api-reference/governance-policies/create-or-update-a-policy

/api-specs/cloud/public-api.json put /api/governance/v1/policies
Create or update a policy



# Create or update a policy definition through yaml
Source: https://docs.domino.ai/api-reference/governance-policies/create-or-update-a-policy-definition-through-yaml

/api-specs/cloud/public-api.json put /api/governance/v1/policies/{id}/definition
Create or update a policy definition through yaml



# Create or update a policy version definition through yaml
Source: https://docs.domino.ai/api-reference/governance-policies/create-or-update-a-policy-version-definition-through-yaml

/api-specs/cloud/public-api.json put /api/governance/v1/policy-versions/{id}/definition
Create or update a policy version definition through yaml



# Delete policy by ID
Source: https://docs.domino.ai/api-reference/governance-policies/delete-policy-by-id

/api-specs/cloud/public-api.json delete /api/governance/v1/policies/{id}
Delete policy by ID



# Diff two policy versions
Source: https://docs.domino.ai/api-reference/governance-policies/diff-two-policy-versions

/api-specs/cloud/public-api.json get /api/governance/v1/policy-versions/{id}/diff/{targetId}
Compute the diff between two policy versions (source and target)



# Draft a policy version
Source: https://docs.domino.ai/api-reference/governance-policies/draft-a-policy-version

/api-specs/cloud/public-api.json post /api/governance/v1/rpc/draft-latest-published-policy-version
Draft a policy version



# Get a policy's full structure, including evidence and the content shape each artifact expects
Source: https://docs.domino.ai/api-reference/governance-policies/get-a-policys-full-structure-including-evidence-and-the-content-shape-each-artifact-expects

/api-specs/cloud/public-api.json get /api/governance/v1/policies/{id}
Returns the policy's stages, evidence sets, and approval slots. Set `hydrateTaxonomy=false` to skip taxonomy tag/property hydration if you don't need it.
Each evidence artifact's `details` object describes what to submit later as a `Result` (via `POST /rpc/submit-result-to-policy`) — its shape depends on `artifactType`:
- `artifactType: "input"` — `details.type` is one of `textinput`, `textarea`, `numeric`, `date`, `radio`, `select`, `checkbox`, `multiselect`. For `radio`/`select`/`checkbox`/`multiselect`, `details.options[].value` lists the only valid content values.
- `artifactType: "guidance"` — `details.text` is a display-only note; no content submission is expected.
- `artifactType: "metadata"` — `details.type` is `file`, `modelmetric`, or `monitorcheck`; `monitorcheck` is populated via `POST /rpc/analyze-monitor-model`, not by submitting content directly.
- `artifactType: "policyScriptedCheck"` — `details` describes a Domino Job to run: `command` (optionally with named `parameters[]` referenced as `${<name>}` placeholders), and optionally `environmentId`/`hardwareTierId`. If either is absent, list available options via `GET /api/environments/beta/environments` or `GET /api/hardwaretiers/v1/hardwaretiers` (Domino's public APIs).
Related:
- POST /rpc/submit-result-to-policy (recommended submission path; full content-shape table)
- GET /policies/{id}/definition (the same structure as YAML, for human review or version control)
- POST /bundles (attach this policy to a new bundle)



# Get the policy definition YAML by ID
Source: https://docs.domino.ai/api-reference/governance-policies/get-the-policy-definition-yaml-by-id

/api-specs/cloud/public-api.json get /api/governance/v1/policies/{id}/definition
Get the policy definition YAML by ID



# Get the policy version by version tag
Source: https://docs.domino.ai/api-reference/governance-policies/get-the-policy-version-by-version-tag

/api-specs/cloud/public-api.json get /api/governance/v1/policy-versions/{id}
Get the policy version by version tag



# Get the policy version definition by version tag
Source: https://docs.domino.ai/api-reference/governance-policies/get-the-policy-version-definition-by-version-tag

/api-specs/cloud/public-api.json get /api/governance/v1/policy-versions/{id}/definition
Get the policy version definition by version tag



# List policy enforcers
Source: https://docs.domino.ai/api-reference/governance-policies/list-policy-enforcers

/api-specs/cloud/public-api.json get /api/governance/v1/policy-enforcers
List policy enforcers, optionally filtered by enforced_on target.



# List policy overviews
Source: https://docs.domino.ai/api-reference/governance-policies/list-policy-overviews

/api-specs/cloud/public-api.json get /api/governance/v1/policy-overviews
List policy overviews



# List Policy Versions
Source: https://docs.domino.ai/api-reference/governance-policies/list-policy-versions

/api-specs/cloud/public-api.json get /api/governance/v1/policies/{id}/versions
List Policy Versions



# Publish a policy version
Source: https://docs.domino.ai/api-reference/governance-policies/publish-a-policy-version

/api-specs/cloud/public-api.json post /api/governance/v1/rpc/publish-policy-version
Publish a policy version



# Publish, draft, or archive a policy
Source: https://docs.domino.ai/api-reference/governance-policies/publish-draft-or-archive-a-policy

/api-specs/cloud/public-api.json put /api/governance/v1/policies/{id}/status
Sets the policy's status to `Draft`, `Published`, or `Archived`. Only `Published` policies can be attached to new bundles via `POST /bundles`.
If `status` already equals the policy's current status, this returns `200` and makes no update. `Archived` → `Draft` is rejected with `400`. `Published` → `Draft` is rejected with `400` if the policy has ever been applied to a bundle.
An enforced policy cannot move to any non-`Published` status — that returns `400`.
Related:
- POST /policies (create a new policy, which starts in Draft)
- POST /bundles (attach a Published policy to a bundle)
- GET /policy-overviews (check a policy's current status before calling this)



# Update a policy enforcer
Source: https://docs.domino.ai/api-reference/governance-policies/update-a-policy-enforcer

/api-specs/cloud/public-api.json patch /api/governance/v1/policy-enforcers/{id}
Update the policy list and/or enabled flag of an existing policy enforcer. enforcedOn cannot be changed.



# Update the policy meta by ID
Source: https://docs.domino.ai/api-reference/governance-policies/update-the-policy-meta-by-id

/api-specs/cloud/public-api.json put /api/governance/v1/policies/{id}
Update the policy meta by ID



# Get API key report
Source: https://docs.domino.ai/api-reference/model-monitoring-authentication/get-model-monitorv2apiauthapi_key_report

/api-specs/cloud/public-api.json get /model-monitor/v2/api/auth/api_key_report



# Get OIDC details
Source: https://docs.domino.ai/api-reference/model-monitoring-authentication/get-model-monitorv2apiauthget_oidc_details

/api-specs/cloud/public-api.json get /model-monitor/v2/api/auth/get_oidc_details



# Log in
Source: https://docs.domino.ai/api-reference/model-monitoring-authentication/get-model-monitorv2apiauthlogin

/api-specs/cloud/public-api.json get /model-monitor/v2/api/auth/login



# Log out
Source: https://docs.domino.ai/api-reference/model-monitoring-authentication/get-model-monitorv2apiauthlogout

/api-specs/cloud/public-api.json get /model-monitor/v2/api/auth/logout



# Get user API key
Source: https://docs.domino.ai/api-reference/model-monitoring-authentication/get-model-monitorv2apiauthuserapi_key

/api-specs/cloud/public-api.json get /model-monitor/v2/api/auth/user/api_key



# Get current user
Source: https://docs.domino.ai/api-reference/model-monitoring-authentication/get-model-monitorv2apiauthusercurrent

/api-specs/cloud/public-api.json get /model-monitor/v2/api/auth/user/current



# Generate user API key
Source: https://docs.domino.ai/api-reference/model-monitoring-authentication/get-model-monitorv2apiauthusergenerate_api_key

/api-specs/cloud/public-api.json get /model-monitor/v2/api/auth/user/generate_api_key



# Ping
Source: https://docs.domino.ai/api-reference/model-monitoring-authentication/get-model-monitorv2apiping

/api-specs/cloud/public-api.json get /model-monitor/v2/api/ping



# Revoke user API key
Source: https://docs.domino.ai/api-reference/model-monitoring-authentication/put-model-monitorv2apiauthuserrevoke_api_key

/api-specs/cloud/public-api.json put /model-monitor/v2/api/auth/user/revoke_api_key



# Get scheduled quality check data
Source: https://docs.domino.ai/api-reference/model-monitoring-drift-and-quality/get-model-monitorv2apimodel-model-quality-scheduled-check-data

/api-specs/cloud/public-api.json get /model-monitor/v2/api/model/{scheduled_check_id}/model-quality-scheduled-check-data



# Get quality check history
Source: https://docs.domino.ai/api-reference/model-monitoring-drift-and-quality/get-model-monitorv2apimodel-model-quality-scheduled-checks-history

/api-specs/cloud/public-api.json get /model-monitor/v2/api/model/{model_id}/model-quality-scheduled-checks-history



# Get quality scheduler query
Source: https://docs.domino.ai/api-reference/model-monitoring-drift-and-quality/get-model-monitorv2apimodel-model-quality-scheduler-query

/api-specs/cloud/public-api.json get /model-monitor/v2/api/model/{model_id}/model-quality-scheduler-query



# Get model quality trend
Source: https://docs.domino.ai/api-reference/model-monitoring-drift-and-quality/get-model-monitorv2apimodel-model-quality-trend

/api-specs/cloud/public-api.json get /model-monitor/v2/api/model/{model_id}/model-quality-trend



# Save quality scheduler query
Source: https://docs.domino.ai/api-reference/model-monitoring-drift-and-quality/post-model-monitorv2apimodel-model-quality-scheduler-query

/api-specs/cloud/public-api.json post /model-monitor/v2/api/model/{model_id}/model-quality-scheduler-query



# Create or Update existing channel config for notifications in DMM.
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/create-or-update-existing-channel-config-for-notifications-in-dmm

/api-specs/cloud/public-api.json post /model-monitor/v2/api/notification/channel-config



# Create or Update the existing model alert config.
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/create-or-update-the-existing-model-alert-config

/api-specs/cloud/public-api.json post /model-monitor/v2/api/notification/model-config/{model_id}



# Creates a schedule check for a model drift
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/creates-a-schedule-check-for-a-model-drift

/api-specs/cloud/public-api.json post /model-monitor/api/scheduler/create_drift_schedule_check



# Creates a schedule check for model quality.
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/creates-a-schedule-check-for-model-quality

/api-specs/cloud/public-api.json post /model-monitor/api/scheduler/create_quality_schedule_check



# Fetch alert config for the given model.
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/fetch-alert-config-for-the-given-model

/api-specs/cloud/public-api.json get /model-monitor/v2/api/notification/model-config/{model_id}



# Fetch channel config for notifications in DMM.
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/fetch-channel-config-for-notifications-in-dmm

/api-specs/cloud/public-api.json get /model-monitor/v2/api/notification/channel-config



# Get drift default settings
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/get-drift-default-settings

/api-specs/cloud/public-api.json get /model-monitor/v2/api/setting/default/drift



# Get global settings.
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/get-global-settings

/api-specs/cloud/public-api.json get /model-monitor/v2/api/setting/global



# Get model quality metric default settings
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/get-model-quality-metric-default-settings

/api-specs/cloud/public-api.json get /model-monitor/v2/api/setting/default/metric



# Get the existing user setting for the given user.
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/get-the-existing-user-setting-for-the-given-user

/api-specs/cloud/public-api.json get /model-monitor/v2/api/setting/user



# Gets schedule check data by schedule job ID
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/gets-schedule-check-data-by-schedule-job-id

/api-specs/cloud/public-api.json get /model-monitor/api/scheduler/{scheduleCheckId}/get_schedule_check_by_id



# Gets schedule check job for model drift.
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/gets-schedule-check-job-for-model-drift

/api-specs/cloud/public-api.json get /model-monitor/api/scheduler/{modelId}/get_drift_schedule_check



# Gets schedule check job for model quality
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/gets-schedule-check-job-for-model-quality

/api-specs/cloud/public-api.json get /model-monitor/api/scheduler/{modelId}/get_quality_schedule_check



# Pauses schedule job
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/pauses-schedule-job

/api-specs/cloud/public-api.json post /model-monitor/api/scheduler/{schedulerJobId}/unschedule_scheduler_job



# Sends test mail
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/sends-test-mail

/api-specs/cloud/public-api.json post /model-monitor/api/scheduler/test_email



# Update global settings.
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/update-global-settings

/api-specs/cloud/public-api.json post /model-monitor/v2/api/setting/global



# Update the drift default settings.
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/update-the-drift-default-settings

/api-specs/cloud/public-api.json put /model-monitor/v2/api/setting/default/drift



# Update the existing user setting for the user.
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/update-the-existing-user-setting-for-the-user

/api-specs/cloud/public-api.json post /model-monitor/v2/api/setting/user



# Update the model quality metric default settings.
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/update-the-model-quality-metric-default-settings

/api-specs/cloud/public-api.json put /model-monitor/v2/api/setting/default/metric



# Updates scheduler job of a model
Source: https://docs.domino.ai/api-reference/model-monitoring-settings/updates-scheduler-job-of-a-model

/api-specs/cloud/public-api.json post /model-monitor/api/scheduler/update_scheduler_config



# Copy the transferable results from one bundle to another
Source: https://docs.domino.ai/api-reference/governance-operations/copy-the-transferable-results-from-one-bundle-to-another

/api-specs/cloud/public-api.json post /api/governance/v1/rpc/copy-bundle
Copy the transferable results from one bundle to another



# Diff policies between two bundles
Source: https://docs.domino.ai/api-reference/governance-operations/diff-policies-between-two-bundles

/api-specs/cloud/public-api.json post /api/governance/v1/rpc/diff-bundle-policies
Compare the policies of two bundles, returning structural diffs and optionally evidence result diffs



# Get a bundle's evidence, results, and approval progress against one of its applied policies
Source: https://docs.domino.ai/api-reference/governance-operations/get-a-bundles-evidence-results-and-approval-progress-against-one-of-its-applied-policies

/api-specs/cloud/public-api.json post /api/governance/v1/rpc/compute-policy
Returns the full state of a bundle for a specific policy attached to it: the policy's structure (stages, evidence, approval slots) combined with the bundle's current progress against it (results, approval statuses, findings, comments). This is the read used to render a bundle's policy view in the Domino UI after a policy is selected. Never mutates data, so it's safe to poll on an interval.
`bundleId` comes from `POST /bundles` or `GET /bundles`; `policyId` must be one of the policies already attached to the bundle. If `policyVersionId` is omitted, the bundle's currently active version of that policy is used.
Related:
- GET /bundles/{id}/approvals (lighter-weight poll if you only need approval status, not results/comments)
- POST /rpc/publish-approval-event (moves an approval to its next status)



# Reset bundle results, drafts, approvals, stage assignees, and optionally detach attachments
Source: https://docs.domino.ai/api-reference/governance-operations/reset-bundle-results-drafts-approvals-stage-assignees-and-optionally-detach-attachments

/api-specs/cloud/public-api.json post /api/governance/v1/rpc/reset-bundle
Resets all results, drafts, approvals, stage assignees, and stage/classification for the specified policy versions within a bundle. Optionally detach specified attachments. Accepts reason and relatedFindingId for audit purposes.



# Snapshot a bundle into an evidence-restricted copy and revoke the source's gated approvals
Source: https://docs.domino.ai/api-reference/governance-operations/snapshot-a-bundle-into-an-evidence-restricted-copy-and-revoke-the-sources-gated-approvals

/api-specs/cloud/public-api.json post /api/governance/v1/rpc/snapshot-bundle
Create an evidence-restricted copy of source named `<source.Name><nameSuffix>`, appending a number (e.g. ` 2`) on conflict. Attachments, drafts, results, stage assignees, and approvals are copied; findings and comments are not.
The source's gated approvals are revoked (reset to PendingSubmission) so it can begin a new approval cycle.
If `sourceEnforcePoliciesFor` is set (e.g. `App`), the source (not the snapshot) is synced to that target's current mandatory policies after the snapshot is taken.



# Submit final evidence for one artifact and get the bundle's updated state
Source: https://docs.domino.ai/api-reference/governance-operations/submit-final-evidence-for-one-artifact-and-get-the-bundles-updated-state

/api-specs/cloud/public-api.json post /api/governance/v1/rpc/submit-result-to-policy
Records `content` as the final `Result` for `evidenceId` on this bundle/policy, recomputes classification if the artifact affects it, then returns the same response shape as `POST /rpc/compute-policy`. Use this — not `POST /results` — when you also want the recomputed classification and the bundle's current results, approvals, and findings back in the same response.
`evidenceId` comes from the policy's evidence definition (`GET /policies/{id}`, `stages[].evidenceSet[].id`). `bundleId` comes from `POST /bundles`. Returns `409` if the bundle is read-only or has `evidenceRestricted` set — either case rejects before any write happens.
`content` is a map of `artifactId` → value; the required shape depends on the artifact's `artifactType`/`details.type` (read via `GET /policies/{id}`):
| artifactType | details.type | content shape |
|---|---|---|
| `input` | `textinput` / `textarea` | `string` |
| `input` | `numeric` | `number` |
| `input` | `date` | `string` |
| `input` | `radio` / `select` | `string`, must equal one of `details.options[].value` |
| `input` | `checkbox` / `multiselect` | `string[]`, each must equal one of `details.options[].value` |
| `guidance` | — | no content needed; `details.text` is display-only |
| `metadata` | `file` | `{"commit": string, "files": [{"name","path","sizeLabel"}, ...]}` |
| `metadata` | `modelmetric` | `[{"<metricName>": "<value>"}, ...]`, one entry per row, a value for every metric named in `details.metrics[].name` |
| `metadata` | `monitorcheck` | not submitted here — populated via `POST /rpc/analyze-monitor-model` |
| `policyScriptedCheck` | — | `{"jobId": string, "parameters"?: {...}, "noDetails": boolean}` |
For `policyScriptedCheck`: run the job described in `details` via `POST /api/jobs/v1/jobs` (Domino's public Jobs API) first, then submit its `jobId` here. That API's field names differ from `details`: send `details.command` as `runCommand`, `details.environmentId` as `environmentId`, `details.hardwareTierId` as `hardwareTier`; `projectId` is required and comes from `GET /bundles/{id}` (`.projectId`), not from `details`. `runCommand` is a shell or Python file that must already exist in the project's file system (e.g. `app.py`). If `details` defines named `parameters[]`, `command` contains literal placeholders `${<param.name>}` for each — `runCommand` is `details.command` with each `${<param.name>}` placeholder substituted by its actual value; the Jobs API only ever sees this fully-substituted string, not the parameter names/values separately. Then include the same values under `parameters` (keyed by name) in this content, as a record of what was actually run. Set `noDetails` to `true` only if you ran an ad-hoc command with no defined `parameters`; set it to `false` when running the check's configured `command`/`parameters` as-is. Job output isn't included in this content — fetch it separately via `GET /api/jobs/beta/jobs/{jobId}`.
`command`, `environmentId`, and `hardwareTierId` are all optional on `details` — a policy can define a scripted check without any of them. If `command` is absent, there's no fixed command to run; supply your own `runCommand` in the `POST /api/jobs/v1/jobs` call — it still must be a shell or Python file that already exists in the project's file system, the same as a configured `command`. If `environmentId`/`hardwareTierId` are absent, you can omit them from that call too: it defaults `environmentId` to the project's default environment and `hardwareTier` to the project's default hardware tier. To use a specific one instead, list the options yourself first — `GET /api/environments/beta/environments` or `GET /api/hardwaretiers/v1/hardwaretiers` — and pass the chosen ID explicitly.
Related:
- POST /results (submit a result without recomputing classification)
- GET /policies/{id} (read each artifact's `details` to determine its content shape before submitting)



# Transition an approval to its next status (approve, reject, submit, cancel)
Source: https://docs.domino.ai/api-reference/governance-operations/transition-an-approval-to-its-next-status-approve-reject-submit-cancel

/api-specs/cloud/public-api.json post /api/governance/v1/rpc/publish-approval-event
Publishes an `eventType` against an approval and returns its updated status.
Each `eventType` is only valid from specific current statuses — calling it from the wrong status is rejected, so check the approval's current status first:
| From status | eventType | Resulting status |
|---|---|---|
| `PendingSubmission` | `RequestSubmitted` | `PendingReview` |
| `PendingReview` | `RequestApproved` | `Approved` |
| `PendingReview` | `RequestConditionalApproved` | `ConditionallyApproved` |
| `PendingReview` | `RequestCancelled` | `PendingSubmission` |
| `ConditionallyApproved` | `RequestApproved` / `FindingsApproved` | `Approved` |
| `ConditionallyApproved` | `RequestRejected` / `RequestCancelled` | `PendingSubmission` |
| `Approved` | `RequestRejected` | `PendingSubmission` |
| `Approved` | `RevalidationRequired` | `PendingExpiration` |
| `PendingExpiration` | `RequestApproved` | `Approved` |
| `PendingExpiration` | `RequestRejected` | `PendingSubmission` |
| `PendingExpiration` | `RevalidationExpired` | `Expired` |
| `Expired` | `RequestSubmitted` | `PendingReview` |
| `Expired` | `RequestApproved` | `Approved` |
| `Expired` | `RequestRejected` | `PendingSubmission` |
Calling an `eventType` that isn't valid for the approval's current status currently returns a generic `500` with a message like `"invalid event RequestApproved for approval status PendingSubmission"` — check status via `GET /bundles/{id}/approvals` first to avoid this.
`approvalId` comes from `GET /bundles/{id}/approvals`. If omitted, `stageApprovalId` is used to resolve it instead. `stageApprovalId` identifies the approval slot defined by the policy; each bundle that has the policy applied gets its own `Approval` instance for that slot, and `stageApprovalId` resolves to that bundle-specific `approvalId`. Get it from `GET /policy-versions/{id}` (`stages[].approvals[].id`), using the bundle's actual `policyVersionId` from `GET /bundles/{id}/approvals`.
`meta` shape depends on `eventType`:
| eventType | meta shape |
|---|---|
| `RequestApproved` | `{"result"?: {"evidenceId": string, "content": {...}}}` — optionally submit the final evidence in the same call as approving |
| `RequestConditionalApproved` | `{"result"?: {...same as above...}, "requiredFindings": string[]}` — `requiredFindings` (finding IDs) is required |
| all other event types | no defined fields — send `{}` |
Related:
- GET /bundles/{id}/approvals (get the current status before choosing an eventType)
- POST /rpc/submit-result-to-policy (submit the evidence this approval depends on)
- POST /rpc/update-approval (change who is assigned to approve, not the approval's status)



# Create new filesystem
Source: https://docs.domino.ai/api-reference/netapp-volumes-filesystems/create-new-filesystem

/api-specs/cloud/public-api.json post /remotefs/v1/filesystems
Create new filesystem



# Delete filesystem by ID
Source: https://docs.domino.ai/api-reference/netapp-volumes-filesystems/delete-filesystem-by-id

/api-specs/cloud/public-api.json delete /remotefs/v1/filesystems/{id}
Delete a filesystem by ID



# Get filesystem by ID
Source: https://docs.domino.ai/api-reference/netapp-volumes-filesystems/get-filesystem-by-id

/api-specs/cloud/public-api.json get /remotefs/v1/filesystems/{id}
Get filesystem by ID



# List available PVCs in the domino-compute namespace with the netapp-storage label
Source: https://docs.domino.ai/api-reference/netapp-volumes-filesystems/list-available-pvcs-in-the-domino-compute-namespace-with-the-netapp-storage-label

/api-specs/cloud/public-api.json get /remotefs/v1/root-pvcs
List available PVCs in the domino-compute namespace with the netapp-storage label



# List filesystem
Source: https://docs.domino.ai/api-reference/netapp-volumes-filesystems/list-filesystem

/api-specs/cloud/public-api.json get /remotefs/v1/filesystems
List filesystem



# Update filesystem
Source: https://docs.domino.ai/api-reference/netapp-volumes-filesystems/update-filesystem

/api-specs/cloud/public-api.json put /remotefs/v1/filesystems/{id}
Update a filesystem. Can be updated if there are volumes associated with the filesystem



# Add snapshot tag
Source: https://docs.domino.ai/api-reference/netapp-volumes-snapshots/add-snapshot-tag

/api-specs/cloud/public-api.json post /remotefs/v1/snapshots/{id}/tags
Add a tag to a snapshot



# Create new snapshot
Source: https://docs.domino.ai/api-reference/netapp-volumes-snapshots/create-new-snapshot

/api-specs/cloud/public-api.json post /remotefs/v1/snapshots
Create new snapshot with optional tags, preserving the order provided in the input



# Create new snapshot from a run
Source: https://docs.domino.ai/api-reference/netapp-volumes-snapshots/create-new-snapshot-from-a-run

/api-specs/cloud/public-api.json post /remotefs/v1/rpc/create-snapshot-from-run
Create new snapshot from a volumes mounted in a run, must be called by super user



# Delete snapshot by ID
Source: https://docs.domino.ai/api-reference/netapp-volumes-snapshots/delete-snapshot-by-id

/api-specs/cloud/public-api.json post /remotefs/v1/rpc/delete-snapshot
Delete a snapshot by ID and associated snapshot tags



# Delete snapshot tag
Source: https://docs.domino.ai/api-reference/netapp-volumes-snapshots/delete-snapshot-tag

/api-specs/cloud/public-api.json delete /remotefs/v1/snapshots/{id}/tags/{name}
Delete a tag from a snapshot



# Get file names in a snapshot recursively
Source: https://docs.domino.ai/api-reference/netapp-volumes-snapshots/get-file-names-in-a-snapshot-recursively

/api-specs/cloud/public-api.json get /remotefs/v1/snapshots/{id}/files/recursive
Get file names in a snapshot at a specific subpath recursively



# Get file names in a volume snapshot
Source: https://docs.domino.ai/api-reference/netapp-volumes-snapshots/get-file-names-in-a-volume-snapshot

/api-specs/cloud/public-api.json get /remotefs/v1/snapshots/{id}/files
Get file names in a volume snapshot at a specific subpath



# Get non-directory volume snapshot file raw content at specified path
Source: https://docs.domino.ai/api-reference/netapp-volumes-snapshots/get-non-directory-volume-snapshot-file-raw-content-at-specified-path

/api-specs/cloud/public-api.json get /remotefs/v1/snapshots/{id}/files/raw
Get non-directory volume snapshot file raw content at specified path either to be downloaded or not



# Get snapshot by ID
Source: https://docs.domino.ai/api-reference/netapp-volumes-snapshots/get-snapshot-by-id

/api-specs/cloud/public-api.json get /remotefs/v1/snapshots/{id}
Get snapshot by ID



# Get snapshot by volume ID and snapshot version
Source: https://docs.domino.ai/api-reference/netapp-volumes-snapshots/get-snapshot-by-volume-id-and-snapshot-version

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}/snapshots/{version}
Get snapshot by volume ID and snapshot version



# Get volume snapshot file metadata content at specified path
Source: https://docs.domino.ai/api-reference/netapp-volumes-snapshots/get-volume-snapshot-file-metadata-content-at-specified-path

/api-specs/cloud/public-api.json get /remotefs/v1/snapshots/{id}/files/metadata
Get volume snapshot file metadata content at specified path



# Get volume snapshot file raw content at subpath
Source: https://docs.domino.ai/api-reference/netapp-volumes-snapshots/get-volume-snapshot-file-raw-content-at-subpath

/api-specs/cloud/public-api.json get /remotefs/v1/snapshots/{id}/files/raw/{path}
Get volume snapshot file raw content at subpath either to be downloaded or not



# Get volume snapshot object
Source: https://docs.domino.ai/api-reference/netapp-volumes-snapshots/get-volume-snapshot-object

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}/snapshots/{version}/keys/{key}
Get volume snapshot object



# Get volume snapshot object URL
Source: https://docs.domino.ai/api-reference/netapp-volumes-snapshots/get-volume-snapshot-object-url

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}/snapshots/{version}/urls/{key}
Get volume snapshot object URL



# List keys in a volume snapshot
Source: https://docs.domino.ai/api-reference/netapp-volumes-snapshots/list-keys-in-a-volume-snapshot

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}/snapshots/{version}/keys
List keys in a volume snapshot



# List snapshots
Source: https://docs.domino.ai/api-reference/netapp-volumes-snapshots/list-snapshots

/api-specs/cloud/public-api.json get /remotefs/v1/snapshots
List snapshots



# Restore snapshot to active state
Source: https://docs.domino.ai/api-reference/netapp-volumes-snapshots/restore-snapshot-to-active-state

/api-specs/cloud/public-api.json post /remotefs/v1/rpc/restore-snapshot
Restore snapshot to active state



# Update Snapshot by ID
Source: https://docs.domino.ai/api-reference/netapp-volumes-snapshots/update-snapshot-by-id

/api-specs/cloud/public-api.json put /remotefs/v1/snapshots/{id}
Update Snapshot by ID



# Callback for data transfer from filetask
Source: https://docs.domino.ai/api-reference/netapp-volumes-transfers/callback-for-data-transfer-from-filetask

/api-specs/cloud/public-api.json post /remotefs/v1/data-transfers/{id}/callback
Callback for data transfer from filetask



# Create download archive
Source: https://docs.domino.ai/api-reference/netapp-volumes-transfers/create-download-archive

/api-specs/cloud/public-api.json post /remotefs/v1/downloads
Create download archive



# Create new data transfer
Source: https://docs.domino.ai/api-reference/netapp-volumes-transfers/create-new-data-transfer

/api-specs/cloud/public-api.json post /remotefs/v1/data-transfers
Create new data transfer



# Get data transfer by ID
Source: https://docs.domino.ai/api-reference/netapp-volumes-transfers/get-data-transfer-by-id

/api-specs/cloud/public-api.json get /remotefs/v1/data-transfers/{id}
Get data transfer by ID



# Get download task status
Source: https://docs.domino.ai/api-reference/netapp-volumes-transfers/get-download-task-status

/api-specs/cloud/public-api.json get /remotefs/v1/downloads/{id}
Get download task status



# Get import blobs task update
Source: https://docs.domino.ai/api-reference/netapp-volumes-transfers/get-import-blobs-task-update

/api-specs/cloud/public-api.json get /remotefs/v1/tasks/import-blobs/{key}/update
Get import blobs task update



# Gets all permissions for requesting user
Source: https://docs.domino.ai/api-reference/netapp-volumes-transfers/gets-all-permissions-for-requesting-user

/api-specs/cloud/public-api.json post /remotefs/v1/account/authz/permissions/authorizedactions
Gets all permissions for requesting user with optional volume ID or project ID in context field



# List data transfers
Source: https://docs.domino.ai/api-reference/netapp-volumes-transfers/list-data-transfers

/api-specs/cloud/public-api.json get /remotefs/v1/data-transfers
List data transfers



# Attach volume to project
Source: https://docs.domino.ai/api-reference/netapp-volumes/attach-volume-to-project

/api-specs/cloud/public-api.json post /remotefs/v1/rpc/attach-volume-to-project
Attach volume to project



# Attempts to rename volume file or directory
Source: https://docs.domino.ai/api-reference/netapp-volumes/attempts-to-rename-volume-file-or-directory

/api-specs/cloud/public-api.json patch /remotefs/v1/volumes/{id}/files
Attempts to rename volume file or directory corresponding to the specified volume ID



# Bulk attaches volumes to target project and allows for authorized user impersonation
Source: https://docs.domino.ai/api-reference/netapp-volumes/bulk-attaches-volumes-to-target-project-and-allows-for-authorized-user-impersonation

/api-specs/cloud/public-api.json post /remotefs/v1/rpc/bulk-attach-volumes-to-project
Bulk attaches volumes to target project and allows for authorized user impersonation,
returns success/failure per volume



# Bulk copies volumes and contents to target project and allows for authorized user impersonation
Source: https://docs.domino.ai/api-reference/netapp-volumes/bulk-copies-volumes-and-contents-to-target-project-and-allows-for-authorized-user-impersonation

/api-specs/cloud/public-api.json post /remotefs/v1/rpc/bulk-copy-volumes-to-project
Bulk copies volumes and contents to target project and allows for authorized user impersonation,
does not copy snapshots and returns success/failure per volume



# Cancels the upload process to a volume
Source: https://docs.domino.ai/api-reference/netapp-volumes/cancels-the-upload-process-to-a-volume

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}/uploads/{uploadID}
Cancels the upload process to the specified volume and upload key, deleting the corresponding session and directory



# Check if a file(s) already exist in a Volume
Source: https://docs.domino.ai/api-reference/netapp-volumes/check-if-a-files-already-exist-in-a-volume

/api-specs/cloud/public-api.json post /remotefs/v1/volumes/{id}/files/check-collision
Check if a file(s) already exist in a Volume and return the conflicting files, if any, as an error



# Check whether requesting user is over allowed volume limit
Source: https://docs.domino.ai/api-reference/netapp-volumes/check-whether-requesting-user-is-over-allowed-volume-limit

/api-specs/cloud/public-api.json post /remotefs/v1/rpc/check-is-over-limit
Check whether requesting user is over allowed volume limit



# Completes the volume file upload session
Source: https://docs.domino.ai/api-reference/netapp-volumes/completes-the-volume-file-upload-session

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}/uploads/{uploadID}/complete
Completes the volume file upload session



# Create new volume
Source: https://docs.domino.ai/api-reference/netapp-volumes/create-new-volume

/api-specs/cloud/public-api.json post /remotefs/v1/volumes
Create new volume



# Delete volume by ID
Source: https://docs.domino.ai/api-reference/netapp-volumes/delete-volume-by-id

/api-specs/cloud/public-api.json post /remotefs/v1/rpc/delete-volume
Delete a volume by ID



# Deletes files from a volume
Source: https://docs.domino.ai/api-reference/netapp-volumes/deletes-files-from-a-volume

/api-specs/cloud/public-api.json delete /remotefs/v1/volumes/{id}/files
Deletes files from a volume



# Detach volume from project
Source: https://docs.domino.ai/api-reference/netapp-volumes/detach-volume-from-project

/api-specs/cloud/public-api.json post /remotefs/v1/rpc/detach-volume-from-project
Detach volume from project



# Download volume archive file from Domino to local
Source: https://docs.domino.ai/api-reference/netapp-volumes/download-volume-archive-file-from-domino-to-local

/api-specs/cloud/public-api.json post /remotefs/v1/rpc/download-archive
Download volume archive file created from a download task by filetask from Domino to your local machine



# Get a volume by unique name
Source: https://docs.domino.ai/api-reference/netapp-volumes/get-a-volume-by-unique-name

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/unique-name/{uniqueName}
Get a volume by unique name



# Get domino staging volume name
Source: https://docs.domino.ai/api-reference/netapp-volumes/get-domino-staging-volume-name

/api-specs/cloud/public-api.json get /remotefs/v1/staging-volume
Get domino staging volume name



# Get file names in a volume
Source: https://docs.domino.ai/api-reference/netapp-volumes/get-file-names-in-a-volume

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}/files
Get file names in a volume at a specific subpath



# Get file names in a volume recursively
Source: https://docs.domino.ai/api-reference/netapp-volumes/get-file-names-in-a-volume-recursively

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}/files/recursive
Get file names in a volume at a specific subpath recursively



# Get grants by Volume ID
Source: https://docs.domino.ai/api-reference/netapp-volumes/get-grants-by-volume-id

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}/grants
Get grants by Volume ID



# Get if user has all provided volume permissions
Source: https://docs.domino.ai/api-reference/netapp-volumes/get-if-user-has-all-provided-volume-permissions

/api-specs/cloud/public-api.json post /remotefs/v1/account/authz/permissions/volume/:id
Get if user has all provided volume permissions



# Get non-directory volume file raw content at specified path
Source: https://docs.domino.ai/api-reference/netapp-volumes/get-non-directory-volume-file-raw-content-at-specified-path

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}/files/raw
Get non-directory volume file raw content at specified path either to be downloaded or not



# Get the connection snippets for a given volume to access as a datasource
Source: https://docs.domino.ai/api-reference/netapp-volumes/get-the-connection-snippets-for-a-given-volume-to-access-as-a-datasource

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}/connection-snippets
Get the connection snippets for a given volume to access as a datasource



# Get volume by ID
Source: https://docs.domino.ai/api-reference/netapp-volumes/get-volume-by-id

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}
Get volume by ID



# Get volume file metadata content at specified path
Source: https://docs.domino.ai/api-reference/netapp-volumes/get-volume-file-metadata-content-at-specified-path

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}/files/metadata
Get volume file metadata content at specified path



# Get volume file raw content at subpath
Source: https://docs.domino.ai/api-reference/netapp-volumes/get-volume-file-raw-content-at-subpath

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}/files/raw/{path}
Get volume file raw content at subpath either to be downloaded or not



# Get volume file raw content at subpath for previewing with truncation
Source: https://docs.domino.ai/api-reference/netapp-volumes/get-volume-file-raw-content-at-subpath-for-previewing-with-truncation

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}/files/preview/{path}
Get volume file raw content at subpath for previewing, truncating files larger than max preview config



# Get volume mounts by project ID
Source: https://docs.domino.ai/api-reference/netapp-volumes/get-volume-mounts-by-project-id

/api-specs/cloud/public-api.json post /remotefs/v1/rpc/get-volume-mounts
Get volume and snapshot mounts by project ID and optional data plane ID or volume ID



# Get volume object URL
Source: https://docs.domino.ai/api-reference/netapp-volumes/get-volume-object-url

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}/urls/{key}
Get volume object URL



# Get volume snapshot file raw content at subpath for previewing with truncation
Source: https://docs.domino.ai/api-reference/netapp-volumes/get-volume-snapshot-file-raw-content-at-subpath-for-previewing-with-truncation

/api-specs/cloud/public-api.json get /remotefs/v1/snapshots/{id}/files/preview/{path}
Get volume snapshot file raw content at subpath for previewing, truncating files larger than max preview config



# List accessible volume IDs
Source: https://docs.domino.ai/api-reference/netapp-volumes/list-accessible-volume-ids

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/ids
Lightweight list of volume IDs accessible to the caller, without hydrating volumes or projects



# List keys in a volume
Source: https://docs.domino.ai/api-reference/netapp-volumes/list-keys-in-a-volume

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}/keys
List keys in a volume



# List volumes
Source: https://docs.domino.ai/api-reference/netapp-volumes/list-volumes

/api-specs/cloud/public-api.json get /remotefs/v1/volumes
List volumes



# Mark volume for deletion
Source: https://docs.domino.ai/api-reference/netapp-volumes/mark-volume-for-deletion

/api-specs/cloud/public-api.json post /remotefs/v1/rpc/mark-volume-for-deletion
Mark volume for deletion



# Restore volume to active state
Source: https://docs.domino.ai/api-reference/netapp-volumes/restore-volume-to-active-state

/api-specs/cloud/public-api.json post /remotefs/v1/rpc/restore-volume
Restore volume to active state



# Retrieves the content of a key in a volume
Source: https://docs.domino.ai/api-reference/netapp-volumes/retrieves-the-content-of-a-key-in-a-volume

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}/keys/{key}
Retrieves the content of a key in a volume



# Returns whether this upload chunk has already been sent
Source: https://docs.domino.ai/api-reference/netapp-volumes/returns-whether-this-upload-chunk-has-already-been-sent

/api-specs/cloud/public-api.json get /remotefs/v1/volumes/{id}/uploads/status
Returns whether this upload chunk has already been sent, which is useful for resuming uploads



# Send file upload chunk
Source: https://docs.domino.ai/api-reference/netapp-volumes/send-file-upload-chunk

/api-specs/cloud/public-api.json post /remotefs/v1/volumes/{id}/uploads
Send file upload chunk for the specified Volume, writing the chunk information to the SQLite database



# Start process to upload a file(s) to a Volume
Source: https://docs.domino.ai/api-reference/netapp-volumes/start-process-to-upload-a-files-to-a-volume

/api-specs/cloud/public-api.json post /remotefs/v1/volumes/{id}/uploads/start
Start process to upload a file(s) to the read-write snapshot of the specified Volume.



# Update grants by Volume ID
Source: https://docs.domino.ai/api-reference/netapp-volumes/update-grants-by-volume-id

/api-specs/cloud/public-api.json put /remotefs/v1/volumes/{id}/grants
Update grants by Volume ID



# Update Volume
Source: https://docs.domino.ai/api-reference/netapp-volumes/update-volume

/api-specs/cloud/public-api.json patch /remotefs/v1/volumes/{id}
Update Volume



# Write content to a specified volume key
Source: https://docs.domino.ai/api-reference/netapp-volumes/write-content-to-a-specified-volume-key

/api-specs/cloud/public-api.json put /remotefs/v1/volumes/{id}/keys/{key}
Write content to a specified volume key



# Autocomplete tags
Source: https://docs.domino.ai/api-reference/tags/autocomplete-tags

/api-specs/cloud/public-api.json get /api/taxonomy/v1/tags/autocomplete
Search tags by prefix and return path strings



# Bulk delete namespaces
Source: https://docs.domino.ai/api-reference/tags/bulk-delete-namespaces

/api-specs/cloud/public-api.json post /api/taxonomy/v1/namespaces/bulk-delete
Delete multiple namespaces by ID



# Bulk delete tags
Source: https://docs.domino.ai/api-reference/tags/bulk-delete-tags

/api-specs/cloud/public-api.json post /api/taxonomy/v1/tags/bulk-delete
Delete multiple tags by ID



# Copy all tags from one entity to another
Source: https://docs.domino.ai/api-reference/tags/copy-all-tags-from-one-entity-to-another

/api-specs/cloud/public-api.json post /api/taxonomy/v1/rpc/copy-entity-tags
Copy all tag assignments from a source entity ID to a target entity ID of the same type, replacing the target's existing tags. The source is retained. Both entities must exist. Internal, service-account only.



# Create a new namespace
Source: https://docs.domino.ai/api-reference/tags/create-a-new-namespace

/api-specs/cloud/public-api.json post /api/taxonomy/v1/namespaces
Create a new namespace for organizing tags



# Create a new tag
Source: https://docs.domino.ai/api-reference/tags/create-a-new-tag

/api-specs/cloud/public-api.json post /api/taxonomy/v1/tags
Create a new tag within a namespace



# Delete a namespace
Source: https://docs.domino.ai/api-reference/tags/delete-a-namespace

/api-specs/cloud/public-api.json delete /api/taxonomy/v1/namespaces/{namespaceId}
Delete a namespace and all its tags (fails if tags have entity assignments)



# Delete all tags for an entity
Source: https://docs.domino.ai/api-reference/tags/delete-all-tags-for-an-entity

/api-specs/cloud/public-api.json delete /api/taxonomy/v1/entity-tags
Remove all tag assignments for a given entity ID and type. Idempotent — returns success even if no tags exist.



# Export taxonomy to file
Source: https://docs.domino.ai/api-reference/tags/export-taxonomy-to-file

/api-specs/cloud/public-api.json post /api/taxonomy/v1/rpc/export-to-file
Downloads the full taxonomy tree as a file. Currently supports CSV format.



# Get a namespace by ID
Source: https://docs.domino.ai/api-reference/tags/get-a-namespace-by-id

/api-specs/cloud/public-api.json get /api/taxonomy/v1/namespaces/{namespaceId}
Retrieve a single namespace by its UUID



# Get configuration limits
Source: https://docs.domino.ai/api-reference/tags/get-configuration-limits

/api-specs/cloud/public-api.json get /api/taxonomy/v1/config
Retrieve the current configuration limit settings



# Get entities by multiple tags
Source: https://docs.domino.ai/api-reference/tags/get-entities-by-multiple-tags

/api-specs/cloud/public-api.json get /api/taxonomy/v1/entities
Get a paginated list of entities that have ALL of the specified tags assigned (AND semantics)



# Get tags for multiple entities
Source: https://docs.domino.ai/api-reference/tags/get-tags-for-multiple-entities

/api-specs/cloud/public-api.json get /api/taxonomy/v1/entity-tags
Batch-fetch tags grouped by entity ID



# Import from file
Source: https://docs.domino.ai/api-reference/tags/import-from-file

/api-specs/cloud/public-api.json post /api/taxonomy/v1/rpc/import-from-file
Bulk import namespaces and tags from a CSV file



# List namespaces
Source: https://docs.domino.ai/api-reference/tags/list-namespaces

/api-specs/cloud/public-api.json get /api/taxonomy/v1/namespaces
List namespaces with optional search, filtering, sorting, and pagination



# List tags
Source: https://docs.domino.ai/api-reference/tags/list-tags

/api-specs/cloud/public-api.json get /api/taxonomy/v1/tags
List tags with optional search, filtering, sorting, and pagination



# Merge tags
Source: https://docs.domino.ai/api-reference/tags/merge-tags

/api-specs/cloud/public-api.json post /api/taxonomy/v1/rpc/merge-tags
Merge source tags into a target tag, reassigning entity assignments



# Tag an entity
Source: https://docs.domino.ai/api-reference/tags/tag-an-entity

/api-specs/cloud/public-api.json post /api/taxonomy/v1/rpc/tag-entity
Assign one or more tags to an entity



# Untag an entity
Source: https://docs.domino.ai/api-reference/tags/untag-an-entity

/api-specs/cloud/public-api.json post /api/taxonomy/v1/rpc/untag-entity
Remove one or more tag assignments from an entity



# Update a namespace
Source: https://docs.domino.ai/api-reference/tags/update-a-namespace

/api-specs/cloud/public-api.json put /api/taxonomy/v1/namespaces/{namespaceId}
Update a namespace's label, description, assignment policy, or status



# Validate import file
Source: https://docs.domino.ai/api-reference/tags/validate-import-file

/api-specs/cloud/public-api.json post /api/taxonomy/v1/rpc/validate-file
Validates a CSV import file without persisting any data



# Fetch a list of NodeExecution launched by the reference TaskExecution.
Source: https://docs.domino.ai/api-reference/children/fetch-a-list-of-nodeexecution-launched-by-the-reference-taskexecution

/api-specs/cloud/flyte-native-api.json get /api/v1/children/task_executions/{task_execution_id.node_execution_id.execution_id.project}/{task_execution_id.node_execution_id.execution_id.domain}/{task_execution_id.node_execution_id.execution_id.name}/{task_execution_id.node_execution_id.node_id}/{task_execution_id.task_id.project}/{task_execution_id.task_id.domain}/{task_execution_id.task_id.name}/{task_execution_id.task_id.version}/{task_execution_id.retry_attempt}



# Fetches input and output data for a Execution.
Source: https://docs.domino.ai/api-reference/data/fetches-input-and-output-data-for-a-execution

/api-specs/cloud/flyte-native-api.json get /api/v1/data/executions/{id.project}/{id.domain}/{id.name}



# Fetches input and output data for a NodeExecution.
Source: https://docs.domino.ai/api-reference/data/fetches-input-and-output-data-for-a-nodeexecution

/api-specs/cloud/flyte-native-api.json get /api/v1/data/node_executions/{id.execution_id.project}/{id.execution_id.domain}/{id.execution_id.name}/{id.node_id}



# Fetches input and output data for a TaskExecution.
Source: https://docs.domino.ai/api-reference/data/fetches-input-and-output-data-for-a-taskexecution

/api-specs/cloud/flyte-native-api.json get /api/v1/data/task_executions/{id.node_execution_id.execution_id.project}/{id.node_execution_id.execution_id.domain}/{id.node_execution_id.execution_id.name}/{id.node_execution_id.node_id}/{id.task_id.project}/{id.task_id.domain}/{id.task_id.name}/{id.task_id.version}/{id.retry_attempt}
Retrieve input and output data from an existing task execution.



# Fetch a DescriptionEntity object.
Source: https://docs.domino.ai/api-reference/description-entities/fetch-a-descriptionentity-object

/api-specs/cloud/flyte-native-api.json get /api/v1/description_entities/{id.resource_type}/{id.project}/{id.domain}/{id.name}/{id.version}
Retrieve an existing description entity description.



# Fetch a list of DescriptionEntity definitions.
Source: https://docs.domino.ai/api-reference/description-entities/fetch-a-list-of-descriptionentity-definitions

/api-specs/cloud/flyte-native-api.json get /api/v1/description_entities/{resource_type}/{id.project}/{id.domain}
Fetch existing description entity definitions matching input filters.



# Fetch a list of DescriptionEntity definitions (by name)
Source: https://docs.domino.ai/api-reference/description-entities/fetch-a-list-of-descriptionentity-definitions-by-name

/api-specs/cloud/flyte-native-api.json get /api/v1/description_entities/{resource_type}/{id.project}/{id.domain}/{id.name}
Fetch existing description entity definitions matching input filters.



# Get domains
Source: https://docs.domino.ai/api-reference/domains/get-domains

/api-specs/cloud/flyte-native-api.json get /api/v1/domains



# Get event download URLs
Source: https://docs.domino.ai/api-reference/events/get-event-download-urls

/api-specs/cloud/workspace-file-audit-trail-api.json get /events/download-urls
Retrieves pre-signed download URLs for aggregated event files within the specified timestamp range



# Get processing status
Source: https://docs.domino.ai/api-reference/events/get-processing-status

/api-specs/cloud/workspace-file-audit-trail-api.json get /process/latest
Retrieves the status of the latest processing result



# Get processing status
Source: https://docs.domino.ai/api-reference/events/get-processing-status-1

/api-specs/cloud/workspace-file-audit-trail-api.json get /process/{id}
Retrieves the status of a specific processing run by its UUID



# Indicates a NodeExecutionEvent has occurred.
Source: https://docs.domino.ai/api-reference/events/indicates-a-nodeexecutionevent-has-occurred

/api-specs/cloud/flyte-native-api.json post /api/v1/events/nodes
Create a node execution event recording a phase transition.



# Indicates a TaskExecutionEvent has occurred.
Source: https://docs.domino.ai/api-reference/events/indicates-a-taskexecutionevent-has-occurred

/api-specs/cloud/flyte-native-api.json post /api/v1/events/tasks
Create a task execution event recording a phase transition.



# Indicates a WorkflowExecutionEvent has occurred.
Source: https://docs.domino.ai/api-reference/events/indicates-a-workflowexecutionevent-has-occurred

/api-specs/cloud/flyte-native-api.json post /api/v1/events/workflows
Create a workflow execution event recording a phase transition.



# Process workspace audit events
Source: https://docs.domino.ai/api-reference/events/process-workspace-audit-events

/api-specs/cloud/workspace-file-audit-trail-api.json put /process
Processes any new workspace audit events that have been created in the storage location since the last
processing run and updates the aggregated event parquet files.



# Fetch a list of Execution.
Source: https://docs.domino.ai/api-reference/executions/fetch-a-list-of-execution

/api-specs/cloud/flyte-native-api.json get /api/v1/executions/{id.project}/{id.domain}



# Fetches a Execution.
Source: https://docs.domino.ai/api-reference/executions/fetches-a-execution

/api-specs/cloud/flyte-native-api.json get /api/v1/executions/{id.project}/{id.domain}/{id.name}
Retrieve an existing workflow execution.



# Recreates a previously-run workflow execution that will only start executing from the last known
Source: https://docs.domino.ai/api-reference/executions/recreates-a-previously-run-workflow-execution-that-will-only-start-executing-from-the-last-known

/api-specs/cloud/flyte-native-api.json post /api/v1/executions/recover
Recreates a previously-run workflow execution that will only start executing from the last known failure point. In Recover mode, users cannot change any input parameters or update the version of the execution. This is extremely useful to recover from system errors and byzantine faults like - Loss of K8s cluster, bugs in platform or instability, machine failures, downstream system failures (downstream services), or simply to recover executions that failed because of retry exhaustion and should complete if tried again.



# Terminates an in-progress Execution.
Source: https://docs.domino.ai/api-reference/executions/terminates-an-in-progress-execution

/api-specs/cloud/flyte-native-api.json delete /api/v1/executions/{id.project}/{id.domain}/{id.name}



# Triggers the creation of a Execution
Source: https://docs.domino.ai/api-reference/executions/triggers-the-creation-of-a-execution

/api-specs/cloud/flyte-native-api.json post /api/v1/executions
Create a workflow execution.



# Triggers the creation of an identical Execution
Source: https://docs.domino.ai/api-reference/executions/triggers-the-creation-of-an-identical-execution

/api-specs/cloud/flyte-native-api.json post /api/v1/executions/relaunch
Relaunch a workflow execution.



# Update execution belonging to project domain Execution.
Source: https://docs.domino.ai/api-reference/executions/update-execution-belonging-to-project-domain-execution

/api-specs/cloud/flyte-native-api.json put /api/v1/executions/{id.project}/{id.domain}/{id.name}



# Create and upload a LaunchPlan definition
Source: https://docs.domino.ai/api-reference/launch-plans/create-and-upload-a-launchplan-definition

/api-specs/cloud/flyte-native-api.json post /api/v1/launch_plans
Create and register a launch plan definition.



# Fetch a LaunchPlan definition.
Source: https://docs.domino.ai/api-reference/launch-plans/fetch-a-launchplan-definition

/api-specs/cloud/flyte-native-api.json get /api/v1/launch_plans/{id.project}/{id.domain}/{id.name}/{id.version}
Retrieve an existing launch plan definition.



# Fetch a list of LaunchPlan definitions.
Source: https://docs.domino.ai/api-reference/launch-plans/fetch-a-list-of-launchplan-definitions

/api-specs/cloud/flyte-native-api.json get /api/v1/launch_plans/{id.project}/{id.domain}
Fetch existing launch plan definitions matching input filters.



# Fetch a list of LaunchPlan definitions (by name)
Source: https://docs.domino.ai/api-reference/launch-plans/fetch-a-list-of-launchplan-definitions-by-name

/api-specs/cloud/flyte-native-api.json get /api/v1/launch_plans/{id.project}/{id.domain}/{id.name}
Fetch existing launch plan definitions matching input filters.



# Fetch a list of NamedEntityIdentifier of launch plan objects.
Source: https://docs.domino.ai/api-reference/launch-plans/fetch-a-list-of-namedentityidentifier-of-launch-plan-objects

/api-specs/cloud/flyte-native-api.json get /api/v1/launch_plan_ids/{project}/{domain}
Fetch existing launch plan definition identifiers matching input filters.



# Fetch the active version of a LaunchPlan.
Source: https://docs.domino.ai/api-reference/launch-plans/fetch-the-active-version-of-a-launchplan

/api-specs/cloud/flyte-native-api.json get /api/v1/active_launch_plans/{id.project}/{id.domain}/{id.name}
Retrieve the active launch plan version specified by input request filters.



# List active versions of LaunchPlan.
Source: https://docs.domino.ai/api-reference/launch-plans/list-active-versions-of-launchplan

/api-specs/cloud/flyte-native-api.json get /api/v1/active_launch_plans/{project}/{domain}
Fetch the active launch plan versions specified by input request filters.



# Updates the status of a registered LaunchPlan.
Source: https://docs.domino.ai/api-reference/launch-plans/updates-the-status-of-a-registered-launchplan

/api-specs/cloud/flyte-native-api.json put /api/v1/launch_plans/{id.project}/{id.domain}/{id.name}/{id.version}
Update the status of an existing launch plan definition. At most one launch plan version for a given {project, domain, name} can be active at a time. If this call sets a launch plan to active and existing version is already active, the result of this call will be that the formerly active launch plan will be made inactive and specified launch plan in this request will be made active. In the event that the formerly active launch plan had a schedule associated it with it, this schedule will be disabled. If the reference launch plan in this request is being set to active and has a schedule associated with it, the schedule will be enabled.



# Lists custom MatchableAttributesConfiguration for a specific resource type.
Source: https://docs.domino.ai/api-reference/matchable-attributes/lists-custom-matchableattributesconfiguration-for-a-specific-resource-type

/api-specs/cloud/flyte-native-api.json get /api/v1/matchable_attributes
Retrieve a list of MatchableAttributesConfiguration objects.



# Fetches runtime metrics for a Execution.
Source: https://docs.domino.ai/api-reference/metrics/fetches-runtime-metrics-for-a-execution

/api-specs/cloud/flyte-native-api.json get /api/v1/metrics/executions/{id.project}/{id.domain}/{id.name}
Retrieve metrics from an existing workflow execution.



# Returns a list of NamedEntity objects.
Source: https://docs.domino.ai/api-reference/named-entities/returns-a-list-of-namedentity-objects

/api-specs/cloud/flyte-native-api.json get /api/v1/named_entities/{resource_type}/{project}/{domain}
Retrieve a list of NamedEntity objects sharing a common resource type, project, and domain.



# Returns a NamedEntity object.
Source: https://docs.domino.ai/api-reference/named-entities/returns-a-namedentity-object

/api-specs/cloud/flyte-native-api.json get /api/v1/named_entities/{resource_type}/{id.project}/{id.domain}/{id.name}
Retrieve a NamedEntity object.



# Updates a NamedEntity object.
Source: https://docs.domino.ai/api-reference/named-entities/updates-a-namedentity-object

/api-specs/cloud/flyte-native-api.json put /api/v1/named_entities/{resource_type}/{id.project}/{id.domain}/{id.name}
Update the fields associated with a NamedEntity



# Fetch a list of NodeExecution.
Source: https://docs.domino.ai/api-reference/node-executions/fetch-a-list-of-nodeexecution

/api-specs/cloud/flyte-native-api.json get /api/v1/node_executions/{workflow_execution_id.project}/{workflow_execution_id.domain}/{workflow_execution_id.name}



# Fetches a DynamicNodeWorkflowResponse.
Source: https://docs.domino.ai/api-reference/node-executions/fetches-a-dynamicnodeworkflowresponse

/api-specs/cloud/flyte-native-api.json get /api/v1/node_executions/{id.execution_id.project}/{id.execution_id.domain}/{id.execution_id.name}/{id.node_id}/dynamic_workflow



# Fetches a NodeExecution.
Source: https://docs.domino.ai/api-reference/node-executions/fetches-a-nodeexecution

/api-specs/cloud/flyte-native-api.json get /api/v1/node_executions/{id.execution_id.project}/{id.execution_id.domain}/{id.execution_id.name}/{id.node_id}



# Creates or updates custom MatchableAttributesConfiguration at the project level
Source: https://docs.domino.ai/api-reference/project-attributes/creates-or-updates-custom-matchableattributesconfiguration-at-the-project-level

/api-specs/cloud/flyte-native-api.json put /api/v1/project_attributes/{attributes.project}
Update the customized resource attributes associated with a project



# Deletes custom MatchableAttributesConfiguration for a project and domain.
Source: https://docs.domino.ai/api-reference/project-attributes/deletes-custom-matchableattributesconfiguration-for-a-project-and-domain

/api-specs/cloud/flyte-native-api.json delete /api/v1/project_attributes/{project}
Delete the customized resource attributes associated with a project



# Fetches custom MatchableAttributesConfiguration for a project and domain.
Source: https://docs.domino.ai/api-reference/project-attributes/fetches-custom-matchableattributesconfiguration-for-a-project-and-domain

/api-specs/cloud/flyte-native-api.json get /api/v1/project_attributes/{project}
Retrieve the customized resource attributes associated with a project



# Creates or updates custom MatchableAttributesConfiguration for a project and domain.
Source: https://docs.domino.ai/api-reference/project-domain-attributes/creates-or-updates-custom-matchableattributesconfiguration-for-a-project-and-domain

/api-specs/cloud/flyte-native-api.json put /api/v1/project_domain_attributes/{attributes.project}/{attributes.domain}
Update the customized resource attributes associated with a project-domain combination



# Deletes custom MatchableAttributesConfiguration for a project and domain.
Source: https://docs.domino.ai/api-reference/project-domain-attributes/deletes-custom-matchableattributesconfiguration-for-a-project-and-domain

/api-specs/cloud/flyte-native-api.json delete /api/v1/project_domain_attributes/{project}/{domain}
Delete the customized resource attributes associated with a project-domain combination



# Fetches custom MatchableAttributesConfiguration for a project and domain.
Source: https://docs.domino.ai/api-reference/project-domain-attributes/fetches-custom-matchableattributesconfiguration-for-a-project-and-domain

/api-specs/cloud/flyte-native-api.json get /api/v1/project_domain_attributes/{project}/{domain}
Retrieve the customized resource attributes associated with a project-domain combination



# Fetches a list of Project
Source: https://docs.domino.ai/api-reference/projects/fetches-a-list-of-project

/api-specs/cloud/flyte-native-api.json get /api/v1/projects
Fetch registered projects.



# Fetches a Project
Source: https://docs.domino.ai/api-reference/projects/fetches-a-project

/api-specs/cloud/flyte-native-api.json get /api/v1/projects/{id}
Fetch a registered project.



# Registers a Project with the Flyte deployment.
Source: https://docs.domino.ai/api-reference/projects/registers-a-project-with-the-flyte-deployment

/api-specs/cloud/flyte-native-api.json post /api/v1/projects



# Updates an existing Project
Source: https://docs.domino.ai/api-reference/projects/updates-an-existing-project

/api-specs/cloud/flyte-native-api.json put /api/v1/projects/{id}
Update a project.



# Clear all property values for an entity
Source: https://docs.domino.ai/api-reference/properties/clear-all-property-values-for-an-entity

/api-specs/cloud/public-api.json delete /api/taxonomy/v1/property-values/{entityType}/{entityId}
Remove every property value assigned to the given entity.



# Copy all property values from one entity to another
Source: https://docs.domino.ai/api-reference/properties/copy-all-property-values-from-one-entity-to-another

/api-specs/cloud/public-api.json post /api/taxonomy/v1/rpc/copy-property-values
Copy all property values from a source entity ID to a target entity ID of the same type, replacing the target's existing values. The source is retained. Both entities must exist. Internal, service-account only.



# Create a new property
Source: https://docs.domino.ai/api-reference/properties/create-a-new-property

/api-specs/cloud/public-api.json post /api/taxonomy/v1/properties
Create a new property definition



# Delete a property
Source: https://docs.domino.ai/api-reference/properties/delete-a-property

/api-specs/cloud/public-api.json delete /api/taxonomy/v1/properties/{propertyId}
Soft-delete a property. Existing property values are retained but no longer surfaced.



# Get a property by ID
Source: https://docs.domino.ai/api-reference/properties/get-a-property-by-id

/api-specs/cloud/public-api.json get /api/taxonomy/v1/properties/{propertyId}
Retrieve a single property by its UUID



# Get property values for an entity
Source: https://docs.domino.ai/api-reference/properties/get-property-values-for-an-entity

/api-specs/cloud/public-api.json get /api/taxonomy/v1/property-values/{entityType}/{entityId}
Return the property values assigned to the given entity, grouped by group name. Active permitted fields with no stored value are still listed, with `value` omitted.



# List properties
Source: https://docs.domino.ai/api-reference/properties/list-properties

/api-specs/cloud/public-api.json get /api/taxonomy/v1/properties
List properties with optional filtering, sorting, and pagination



# List property groups
Source: https://docs.domino.ai/api-reference/properties/list-property-groups

/api-specs/cloud/public-api.json get /api/taxonomy/v1/property-groups
Return the distinct group names across active properties



# Set or clear property values for an entity
Source: https://docs.domino.ai/api-reference/properties/set-or-clear-property-values-for-an-entity

/api-specs/cloud/public-api.json patch /api/taxonomy/v1/property-values/{entityType}/{entityId}
Apply a batch of set/clear operations to the entity's property values. Per-item failures do not abort the batch. Multi-select properties take their selection via the `values` array; all other types use `value`.



# Update a property
Source: https://docs.domino.ai/api-reference/properties/update-a-property

/api-specs/cloud/public-api.json put /api/taxonomy/v1/properties/{propertyId}
Update a property's label, group, description, allowed entities, or allowed values



# Delete a tag
Source: https://docs.domino.ai/api-reference/tags/delete-a-tag

/api-specs/cloud/public-api.json delete /api/taxonomy/v1/tags/{tagId}
Delete a tag and unassign all its entity assignments



# Get a tag by ID
Source: https://docs.domino.ai/api-reference/tags/get-a-tag-by-id

/api-specs/cloud/public-api.json get /api/taxonomy/v1/tags/{tagId}
Retrieve a single tag by its UUID



# Get entities by tag
Source: https://docs.domino.ai/api-reference/tags/get-entities-by-tag

/api-specs/cloud/public-api.json get /api/taxonomy/v1/tags/{tagId}/entities
Get a paginated list of entities assigned to a tag



# Get full taxonomy tree
Source: https://docs.domino.ai/api-reference/tags/get-full-taxonomy-tree

/api-specs/cloud/public-api.json get /api/taxonomy/v1/taxonomy
Returns all non-deleted namespaces with their tags arranged in a hierarchy.



# Update a tag
Source: https://docs.domino.ai/api-reference/tags/update-a-tag

/api-specs/cloud/public-api.json put /api/taxonomy/v1/tags/{tagId}
Update a tag's label, description, parent, or status



# Fetches a list of TaskExecution.
Source: https://docs.domino.ai/api-reference/task-executions/fetches-a-list-of-taskexecution

/api-specs/cloud/flyte-native-api.json get /api/v1/task_executions/{node_execution_id.execution_id.project}/{node_execution_id.execution_id.domain}/{node_execution_id.execution_id.name}/{node_execution_id.node_id}
Fetch existing task executions matching input filters.



# Fetches a TaskExecution.
Source: https://docs.domino.ai/api-reference/task-executions/fetches-a-taskexecution

/api-specs/cloud/flyte-native-api.json get /api/v1/task_executions/{id.node_execution_id.execution_id.project}/{id.node_execution_id.execution_id.domain}/{id.node_execution_id.execution_id.name}/{id.node_execution_id.node_id}/{id.task_id.project}/{id.task_id.domain}/{id.task_id.name}/{id.task_id.version}/{id.retry_attempt}
Retrieve an existing task execution.



# Create and upload a Task definition
Source: https://docs.domino.ai/api-reference/tasks/create-and-upload-a-task-definition

/api-specs/cloud/flyte-native-api.json post /api/v1/tasks
Create and register a task definition.



# Fetch a list of NamedEntityIdentifier of task objects.
Source: https://docs.domino.ai/api-reference/tasks/fetch-a-list-of-namedentityidentifier-of-task-objects

/api-specs/cloud/flyte-native-api.json get /api/v1/task_ids/{project}/{domain}
Fetch existing task definition identifiers matching input filters.



# Fetch a list of Task definitions.
Source: https://docs.domino.ai/api-reference/tasks/fetch-a-list-of-task-definitions

/api-specs/cloud/flyte-native-api.json get /api/v1/tasks/{id.project}/{id.domain}
Fetch existing task definitions matching input filters.



# Fetch a list of Task definitions (by name)
Source: https://docs.domino.ai/api-reference/tasks/fetch-a-list-of-task-definitions-by-name

/api-specs/cloud/flyte-native-api.json get /api/v1/tasks/{id.project}/{id.domain}/{id.name}
Fetch existing task definitions matching input filters.



# Fetch a Task definition.
Source: https://docs.domino.ai/api-reference/tasks/fetch-a-task-definition

/api-specs/cloud/flyte-native-api.json get /api/v1/tasks/{id.project}/{id.domain}/{id.name}/{id.version}
Retrieve an existing task definition.



# Get version
Source: https://docs.domino.ai/api-reference/version/get-version

/api-specs/cloud/flyte-native-api.json get /api/v1/version
Retrieve the Version (including the Build  information) for FlyteAdmin service



# Creates or updates custom MatchableAttributesConfiguration for a project, domain and workflow.
Source: https://docs.domino.ai/api-reference/workflow-attributes/creates-or-updates-custom-matchableattributesconfiguration-for-a-project-domain-and-workflow

/api-specs/cloud/flyte-native-api.json put /api/v1/workflow_attributes/{attributes.project}/{attributes.domain}/{attributes.workflow}
Update the customized resource attributes associated with a project, domain and workflow combination



# Deletes custom MatchableAttributesConfiguration for a project, domain and workflow.
Source: https://docs.domino.ai/api-reference/workflow-attributes/deletes-custom-matchableattributesconfiguration-for-a-project-domain-and-workflow

/api-specs/cloud/flyte-native-api.json delete /api/v1/workflow_attributes/{project}/{domain}/{workflow}
Delete the customized resource attributes associated with a project, domain and workflow combination



# Fetches custom MatchableAttributesConfiguration for a project, domain and workflow.
Source: https://docs.domino.ai/api-reference/workflow-attributes/fetches-custom-matchableattributesconfiguration-for-a-project-domain-and-workflow

/api-specs/cloud/flyte-native-api.json get /api/v1/workflow_attributes/{project}/{domain}/{workflow}
Retrieve the customized resource attributes associated with a project, domain and workflow combination



# Fetch a list of NamedEntityIdentifier of workflow objects.
Source: https://docs.domino.ai/api-reference/workflows/fetch-a-list-of-namedentityidentifier-of-workflow-objects

/api-specs/cloud/flyte-native-api.json get /api/v1/workflow_ids/{project}/{domain}



# Authentication
Source: https://docs.domino.ai/cloud/reference/api/data_api/authentication

Authenticate Domino Data API clients with credential propagation or an API key, and check support across Workspaces, Jobs, Apps, and endpoints.

## Methods

These are the authentication methods available when using the Domino Data API:

* **Credential propagation**: A token is periodically refreshed and stored in a file. The file location is defined in the **DOMINO\_TOKEN\_FILE** env variable

* **API Key**: If enabled, the user API key is exported in the **DOMINO\_USER\_API\_KEY** env variable

## Availability

<Note>
  Those methods are not available in all run types. The following table lists the availability for these methods.
</Note>

| Run type | Who? | Token | API Key |
| - | - | - | - |
| Workspace | Creator | Yes | Yes |
| Job | Creator | Yes | Yes |
| Scheduled Job | Creator | Yes | Yes |
| Launcher | User | Yes | Yes |
| App | Creator | Yes | Yes |
| Domino endpoint | N/A | No | No |

<Note>
  Anonymous users don’t have a token or API key, and are therefore not allowed to access data sources through public launchers automatically. If you still want to support anonymous execution, the launcher author would need to configure the launcher following the instructions in the [custom authentication](/cloud/reference/api/data_api/datasource_usecases/custom_auth) section or pass the credentials as a parameter.
</Note>

<Warning>
  As described in the matrix, Domino endpoint does not support automatic authentication. You can set the **DOMINO\_USER\_API\_KEY** environment variable for Domino endpoint or follow the instructions in the [custom authentication](/cloud/reference/api/data_api/datasource_usecases/custom_auth) section to use the *DataSourceClient* or the *TrainingSetClient* in this type of run.
</Warning>


# Access Datasets via Datasource proxy
Source: https://docs.domino.ai/cloud/reference/api/data_api/datasource_usecases/access-datasets-via-proxy

Access Domino Datasets through the Datasource Proxy with a user's authorization token, then list and retrieve files with the Python client.

Domino provides secure, programmatic access to datasets through the Datasource Proxy. This allows your application to treat Domino datasets as standard datasources.

When a user interacts with your app, Domino automatically includes an authorization token in the request header. Your app uses this token to access datasets on behalf of the user, ensuring that data access adheres to their permissions.

Although you can call the Datasource Proxy API directly, we recommend using the Domino client libraries for Python. These libraries simplify integration and result in cleaner, more maintainable code.

Note: Example code snippets are available on each dataset’s detail page in the Domino UI.

```python theme={null}
from domino_data.datasets import DatasetClient, DatasetConfig

# instantiate a client
dataset = DatasetClient(token = request.headers.get('Authorization', '')[7:]).get_dataset("dataset-quick-start-67bca8d95df4c35e79db40f5")

# select a specific snapshot, if a specific snapshot is not set, the read/write snapshot is used
dataset.update(config=DatasetConfig(snapshot_id="[your own snapshotID]"))

# list files in the dataset
dataset.list_files()

# get files in the dataset
dataset.get("/path/to/file")
```


# Configuration override
Source: https://docs.domino.ai/cloud/reference/api/data_api/datasource_usecases/config_override

Override Data Source connection settings and datetime precision temporarily in Python or R, then reset the client to its configured defaults.

Data Source configurations are set in the Data / Data Sources panel of the Domino UI. You can update it there for any permanent change.

Some configuration attributes can be overridden locally in the API. A sample of Data Source type configuration is described in the [Classes](#classes) section below.

## Usage

You can create a configuration override with any of the config classes and update your Data Source entity:

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    from domino.data_sources import DataSourceClient, SnowflakeConfig

    snowflake = DataSourceClient().get_datasource("snowflake-prod")

    # Build a override config with a different warehouse than configured in Domino
    config_xxl = SnowflakeConfig(warehouse="compute-xxl")

    # Local update with no permanent change
    snowflake.update(config=config_xxl)
    res = snowflake.query("SELECT COUNT(*) FROM very_large_table")

    # Override can also be used for temporary credentials
    snowflake.update(config=SnowflakeConfig(username="admin", password="<password>"))
    res = snowflake.query("SELECT secret_data FROM secret_table LIMIT 10")
    ```
  </Tab>

  <Tab title="R">
    ```r theme={null}
    library(DominoDataR)
    client <- DominoDataR::datasource_client()

    # Query with a different warehouse than configured in Domino
    table <- DominoDataR::query(
      client,
      "snowflake-prod",
      "SELECT COUNT(*) FROM very_large_table",
      DominoDataR::add_override(warehouse = "compute-xxl")
    )

    # Query with temporary credentials
    table <- DominoDataR::query(
      client,
      "snowflake-prod",
      "SELECT secret_data FROM secret_table LIMIT 10",
      DominoDataR::add_override(username = "admin", password="<password>")
    )
    ```
  </Tab>
</Tabs>

To remove the configuration override, simply reset it:

```python theme={null}
from domino.data_sources import DataSourceClient, SnowflakeConfig

snowflake = DataSourceClient().get_datasource("snowflake-prod")
# Update to dev database
snowflake.update(SnowflakeConfig(database="dev"))

# Reset to default values
snowflake.reset_config()
res = snowflake.query("SELECT * FROM prod_table")
```

Relational Data Sources support an optional datetime precision field, which is overridable in the Data API for version 6.3.0 and above. Adjusting the precision prevents extreme datetimes from overflowing. The precision can be set to "seconds", "milliseconds", "microseconds", or "nanoseconds", with "nanoseconds" as the default.

<Tabs>
  <Tab title="Python">
    ```python theme={null}
    from domino.data_sources import DataSourceClient, SnowflakeConfig

    snowflake = DataSourceClient().get_datasource("snowflake-prod")

    # Update to datetime precision to milliseconds to support extremely large datetime values
    snowflake.update(SnowflakeConfig(datetime_precision="milliseconds"))

    res = snowflake.query("SELECT * FROM prod_table")
    ```
  </Tab>

  <Tab title="R">
    ```r theme={null}
    library(DominoDataR)
    client <- DominoDataR::datasource_client()

    # Query with a different datetime precision than the default
    table <- DominoDataR::query(
      client,
      "snowflake-prod",
      "SELECT * FROM prod_table",
      DominoDataR::add_override(datetime_precision = "milliseconds")
    )
    ```
  </Tab>
</Tabs>

## Classes

[Work with Data Source Connectors](/cloud/platform-capabilities/core-concepts/data/data-source-connectors) has instructions on how to set up the Data Sources.

Below are examples of Data Sources and their configurable override options for **6.7.4** of the Domino Data API.

The source code for these class definitions is available on [GitHub](https://github.com/dominodatalab/domino-data/blob/v6.7.4/domino_data/configuration_gen.py).

### ADLS

```python theme={null}
class domino_data.data_sources.ADLSConfig(
    *, container=None, access_key=None
)
```

ADLS datasource configuration.

| Parameter | Value |
| - | - |
| `access_key` | `Optional[str]` |
| `container` | `Optional[str]` |

### AzureBlobStorage

```python theme={null}
class domino_data.data_sources.AzureBlobStorageConfig(
    *, container=None, access_key=None
)
```

AzureBlobStorage datasource configuration.

| Parameter | Value |
| - | - |
| `access_key` | `Optional[str]` |
| `container` | `Optional[str]` |

### BigQuery

```python theme={null}
class domino_data.data_sources.BigQueryConfig(
    *, gcp_project_id=None, datetime_precision=None, private_key_json=None
)
```

BigQuery datasource configuration.

| Parameter | Value |
| - | - |
| `private_key_json` | `Optional[str]` |
| `gcp_project_id` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### ClickHouse

```python theme={null}
class domino_data.data_sources.ClickHouseConfig(
    *, datetime_precision=None, username=None, password=None
)
```

ClickHouse datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### Databricks

```python theme={null}
class domino_data.data_sources.DatabricksConfig(
    *, datetime_precision=None, catalog=None, schema=None,
    personal_access_token=None
)
```

Databricks datasource configuration.

| Parameter | Value |
| - | - |
| `personal_access_token` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |
| `catalog` | `Optional[str]` |
| `schema` | `Optional[str]` |

### Dataset

```python theme={null}
class domino_data.data_sources.DatasetConfig(
    *, snapshot_id=None, subfolder=None
)
```

Dataset datasource configuration.

| Parameter | Value |
| - | - |
| `snapshot_id` | `Optional[str]` |
| `subfolder` | `Optional[str]` |

### DB2

```python theme={null}
class domino_data.data_sources.DB2Config(
    *, datetime_precision=None, username=None, password=None
)
```

DB2 datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### DB2Native

```python theme={null}
class domino_data.data_sources.DB2NativeConfig(
    *, database=None, datetime_precision=None, username=None,
    password=None, dsn=None
)
```

DB2Native datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `dsn` | `Optional[str]` |
| `database` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### Druid

```python theme={null}
class domino_data.data_sources.DruidConfig(
    *, datetime_precision=None, username=None, password=None
)
```

Druid datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### GCS

```python theme={null}
class domino_data.data_sources.GCSConfig(
    *, bucket=None, private_key_json=None
)
```

GCS datasource configuration.

| Parameter | Value |
| - | - |
| `private_key_json` | `Optional[str]` |
| `bucket` | `Optional[str]` |

### GenericJDBC

```python theme={null}
class domino_data.data_sources.GenericJDBCConfig(
    *, datetime_precision=None, username=None, password=None
)
```

GenericJDBC datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### GenericS3

```python theme={null}
class domino_data.data_sources.GenericS3Config(
    *, bucket=None, subfolder=None, host=None,
    region=None, aws_access_key_id=None, aws_secret_access_key=None
)
```

GenericS3 datasource configuration.

| Parameter | Value |
| - | - |
| `aws_access_key_id` | `Optional[str]` |
| `aws_secret_access_key` | `Optional[str]` |
| `bucket` | `Optional[str]` |
| `subfolder` | `Optional[str]` |
| `host` | `Optional[str]` |
| `region` | `Optional[str]` |

### Greenplum

```python theme={null}
class domino_data.data_sources.GreenplumConfig(
    *, datetime_precision=None, username=None, password=None
)
```

Greenplum datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### Ignite

```python theme={null}
class domino_data.data_sources.IgniteConfig(
    *, datetime_precision=None, username=None, password=None
)
```

Ignite datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### MariaDB

```python theme={null}
class domino_data.data_sources.MariaDBConfig(
    *, datetime_precision=None, username=None, password=None
)
```

MariaDB datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### MongoDB

```python theme={null}
class domino_data.data_sources.MongoDBConfig(
    *, datetime_precision=None, username=None, password=None
)
```

MongoDB datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### MySQL

```python theme={null}
class domino_data.data_sources.MySQLConfig(
    *, database=None, datetime_precision=None, region=None,
    username=None, password=None, aws_access_key_id=None, aws_secret_access_key=None,
    aws_session_token=None
)
```

MySQL datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `aws_access_key_id` | `Optional[str]` |
| `aws_secret_access_key` | `Optional[str]` |
| `aws_session_token` | `Optional[str]` |
| `database` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |
| `region` | `Optional[str]` |

### Netezza

```python theme={null}
class domino_data.data_sources.NetezzaConfig(
    *, datetime_precision=None, username=None, password=None
)
```

Netezza datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### Oracle

```python theme={null}
class domino_data.data_sources.OracleConfig(
    *, database=None, datetime_precision=None, username=None,
    password=None, client_cert=None, client_key=None, ca_cert=None
)
```

Oracle datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `client_cert` | `Optional[str]` |
| `client_key` | `Optional[str]` |
| `ca_cert` | `Optional[str]` |
| `database` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### Palantir

```python theme={null}
class domino_data.data_sources.PalantirConfig(
    *, datetime_precision=None, client_id=None, client_secret=None,
    o_auth_token=None
)
```

Palantir datasource configuration.

| Parameter | Value |
| - | - |
| `client_id` | `Optional[str]` |
| `client_secret` | `Optional[str]` |
| `o_auth_token` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### PostgreSQL

```python theme={null}
class domino_data.data_sources.PostgreSQLConfig(
    *, database=None, datetime_precision=None, region=None,
    username=None, password=None, aws_access_key_id=None, aws_secret_access_key=None,
    aws_session_token=None, client_cert=None, client_key=None, ca_cert=None
)
```

PostgreSQL datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `aws_access_key_id` | `Optional[str]` |
| `aws_secret_access_key` | `Optional[str]` |
| `aws_session_token` | `Optional[str]` |
| `client_cert` | `Optional[str]` |
| `client_key` | `Optional[str]` |
| `ca_cert` | `Optional[str]` |
| `database` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |
| `region` | `Optional[str]` |

### Redshift

```python theme={null}
class domino_data.data_sources.RedshiftConfig(
    *, database=None, datetime_precision=None, region=None,
    username=None, password=None, aws_access_key_id=None, aws_secret_access_key=None,
    aws_session_token=None
)
```

Redshift datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `aws_access_key_id` | `Optional[str]` |
| `aws_secret_access_key` | `Optional[str]` |
| `aws_session_token` | `Optional[str]` |
| `database` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |
| `region` | `Optional[str]` |

### S3

```python theme={null}
class domino_data.data_sources.S3Config(
    *, bucket=None, subfolder=None, region=None,
    aws_access_key_id=None, aws_secret_access_key=None, aws_session_token=None
)
```

S3 datasource configuration.

| Parameter | Value |
| - | - |
| `aws_access_key_id` | `Optional[str]` |
| `aws_secret_access_key` | `Optional[str]` |
| `aws_session_token` | `Optional[str]` |
| `bucket` | `Optional[str]` |
| `subfolder` | `Optional[str]` |
| `region` | `Optional[str]` |

### SAPHana

```python theme={null}
class domino_data.data_sources.SAPHanaConfig(
    *, datetime_precision=None, username=None, password=None
)
```

SAPHana datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### SingleStore

```python theme={null}
class domino_data.data_sources.SingleStoreConfig(
    *, datetime_precision=None, username=None, password=None
)
```

SingleStore datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### SQLServer

```python theme={null}
class domino_data.data_sources.SQLServerConfig(
    *, database=None, datetime_precision=None, username=None,
    password=None
)
```

SQLServer datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `database` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### Snowflake

```python theme={null}
class domino_data.data_sources.SnowflakeConfig(
    *, datetime_precision=None, database=None, role=None,
    schema=None, warehouse=None, username=None, private_key=None,
    passphrase=None, password=None, token=None
)
```

Snowflake datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `private_key` | `Optional[str]` |
| `passphrase` | `Optional[str]` |
| `password` | `Optional[str]` |
| `token` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |
| `database` | `Optional[str]` |
| `role` | `Optional[str]` |
| `schema` | `Optional[str]` |
| `warehouse` | `Optional[str]` |

### Synapse

```python theme={null}
class domino_data.data_sources.SynapseConfig(
    *, datetime_precision=None, username=None, password=None
)
```

Synapse datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### TabularS3Glue

```python theme={null}
class domino_data.data_sources.TabularS3GlueConfig(
    *, database=None, datetime_precision=None, region=None
)
```

TabularS3Glue datasource configuration.

| Parameter | Value |
| - | - |
| `database` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |
| `region` | `Optional[str]` |

### Teradata

```python theme={null}
class domino_data.data_sources.TeradataConfig(
    *, datetime_precision=None, username=None, password=None
)
```

Teradata datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### Trino

```python theme={null}
class domino_data.data_sources.TrinoConfig(
    *, catalog=None, datetime_precision=None, schema=None,
    username=None, password=None
)
```

Trino datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `catalog` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |
| `schema` | `Optional[str]` |

### Vertica

```python theme={null}
class domino_data.data_sources.VerticaConfig(
    *, datetime_precision=None, username=None, password=None
)
```

Vertica datasource configuration.

| Parameter | Value |
| - | - |
| `username` | `Optional[str]` |
| `password` | `Optional[str]` |
| `datetime_precision` | `Optional[str]` |

### NetAppVolume

```python theme={null}
class domino_data.data_sources.NetAppVolumeConfig(
    *, snapshot_version=None, subfolder=None
)
```

NetAppVolume datasource configuration.

| Parameter | Value |
| - | - |
| `snapshot_version` | `Optional[str]` |
| `subfolder` | `Optional[str]` |


# Custom authentication
Source: https://docs.domino.ai/cloud/reference/api/data_api/datasource_usecases/custom_auth

Override Domino DataSourceClient authentication in Python or R by supplying a custom API key or changing the token file location.

See [Authentication](/cloud/reference/api/data_api/authentication) for the default behavior of the DataSourceClient.

## Override the API key

```python theme={null}
from domino.data_sources import DataSourceClient

custom_api_key = "VALID_API_KEY"

client = DataSourceClient(api_key=custom_api_key)
db = client.get_datasource("my-db")
```

```r theme={null}
library(DominoDataR)
client <- DominoDataR::datasource_client(api_key = custom_api_key)
```

## Override the location of the token file

```python theme={null}
from domino.data_sources import DataSourceClient

custom_token_file = "/valid/token/file/location"

client = DataSourceClient(token_file=custom_token_file)
db = client.get_datasource("my-db")
```

```r theme={null}
library(DominoDataR)
client <- DominoDataR::datasource_client(token_file = "/valid/token/file/location")
```


# Data Source use cases
Source: https://docs.domino.ai/cloud/reference/api/data_api/datasource_usecases/index

Use Domino Data Sources to query tabular stores, read and write object stores, export files, customize authentication, and override configuration.

Domino data sources offer a mechanism to create and manage connection properties to a supported external data service. Both administrators and users create data sources, and share them among collaborators. Connection properties are stored securely and there is no need to install data source specific drivers or libraries. A tightly coupled library provides a consistent access pattern for both tabular and file based data. To learn more about data sources, see [Domino data sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors).

Here is a list of Data Source operations:

* [Access Datasets via Datasource Proxy](/cloud/reference/api/data_api/datasource_usecases/access-datasets-via-proxy)

* [Tabular Store](/cloud/reference/api/data_api/datasource_usecases/simple_query)

* [Object Store](/cloud/reference/api/data_api/datasource_usecases/object_store)

* [Write to a Local File](/cloud/reference/api/data_api/datasource_usecases/write_to_file)

* [Custom Authentication](/cloud/reference/api/data_api/datasource_usecases/custom_auth)

* [Configuration Override](/cloud/reference/api/data_api/datasource_usecases/config_override)


# Read/Write to an object store
Source: https://docs.domino.ai/cloud/reference/api/data_api/datasource_usecases/object_store

Use the Domino Data API in Python or R to list, read, download, write, and upload objects in S3 and other object store Data Sources.

The API supports object store type datasources (such as S3) and allows for easy retrieval and upload of objects.

<Note>
  The following APIs are only available when using this type of datasource.
</Note>

## List

Get the datasource from the client:

```python theme={null}
from domino.data_sources import DataSourceClient

s3_dev = DataSourceClient().get_datasource("s3-dev")
```

```r theme={null}
library(DominoDataR)
client <- DominoDataR::datasource_client()
```

You can list objects available in the datasource. You can also specify a prefix:

```python theme={null}
objects = s3_dev.list_objects()

objects_under_path = s3_dev.list_objects("path_prefix")
```

```r theme={null}
keys <- DominoDataR::list_keys(client, "s3-dev")

keys_under_path <- DominoDataR::list_keys(client, "s3-dev", "path_prefix")
```

By default the number of returned objects is limited by the underlying datasource. You can specify how many keys you want as an optional parameter:

```python theme={null}
objects = s3_dev.list_objects(page_size = 1500)
```

```r theme={null}
keys = DominoDataR::list_keys(client, "s3-dev", page_size = 1500)
```

## Read

You can get object content, without having to create object entities, by using the datasource API and specifying the `Object` key name:

```python theme={null}
# Get content as binary
content = s3_dev.get("key")

# Download content to file
s3_dev.download_file("key", "./path/to/local/file")

# Download content to file-like object
f = io.BytesIO()
s3_dev.download_fileobj("key", f)
```

```r theme={null}
# Get content as raw vector
content <- DominoDataR::get_object(client, "s3-dev", "key")

# Get content as character vector
content <- DominoDataR::get_object(client, "s3-dev", "key", as = "text")

# Download content to file
DominoDataR::save_object(client, "s3-dev", "key", "./path/to/local/file")
```

You can also get the datasource entity content from an object entity (Python only):

```python theme={null}
# Key object
my_key = s3_dev.Object("key")

# Get content as binary
content = my_key.get()

# Download content to file
my_key.download_file("./path/to/local/file")

# Download content to file-like object
f = io.BytesIO()
my_key.download_fileobj(f)
```

## Write

Similar to the read/get APIs, you can also write data to a specific object key. From the datasource:

```python theme={null}
# Put binary content to given object key
s3_dev.put("key", b"content")

# Upload file content to specified object key
s3_dev.upload_file("key", "./path/to/local/file")

# Upload file-like content to specified object key
f = io.BytesIO(b"content")
s3_dev.upload_fileobj("key", f)
```

```r theme={null}
# Put raw or character vector to object key
DominoDataR::put_object(client, "s3-dev", "key", what = "content")

# Upload file content to specified object key
DominoDataR::upload_object(client, "s3-dev", "key", file = "./path/to/local/file")
```

You can also write from the object entity (Python only).

```python theme={null}
# Key object
my_key = s3_dev.Object("key")

# Put content as binary
my_key.put(b"content")

# Upload content from file
my_key.upload_file("./path/to/local/file")

# Upload content from file-like object
f = io.BytesIO()
my_key.upload_fileobj(f)
```


# Query a tabular store
Source: https://docs.domino.ai/cloud/reference/api/data_api/datasource_usecases/simple_query

Authenticate the Domino DataSource client and query a configured tabular Data Source from Python or R, then load results into a dataframe.

<Note>
  Ensure that the API is available in your workspace environment; see [Install the Data API](/cloud/reference/api/data_api/install_the_data_api) for setup information.
</Note>

## Authentication

The Datasource client uses environment variables available in the workspace to automatically authenticate your identity.

To override this behavior, see [Custom Authentication](/cloud/reference/api/data_api/datasource_usecases/custom_auth).

## Usage

Assuming a data source named `redshift-test` is configured with valid credentials for the current user:

```python theme={null}
from domino.data_sources import DataSourceClient

# instantiate a client and fetch the datasource instance
redshift = DataSourceClient().get_datasource("redshift-test")

query = """
     SELECT
         firstname,
         lastname,
         age
     FROM
         employees
     LIMIT 1000
 """

 # res is a simple wrapper of the query result
 res = redshift.query(query)
 # to_pandas() loads the result into a pandas dataframe
 df = res.to_pandas()
 # check the first 10 rows
 df.head(10)
```

```r theme={null}
library(DominoDataR)
client <- DominoDataR::datasource_client()

query <- "
  SELECT
    firstname,
    lastname,
    age
  FROM
    employees
  LIMIT 1000"

# table is a https://arrow.apache.org/docs/r/reference/Table-class.html
table <- DominoDataR::query(client, "redshift-test", query)
#> Table
#> 1000 rows x 3 columns
#> $firstname <string not null>
#> $lastname <string not null>
#> $age <int32 not null>
```


# Write to a local file
Source: https://docs.domino.ai/cloud/reference/api/data_api/datasource_usecases/write_to_file

Write Domino Data Source query results to local Parquet or CSV files from Python or R with PyArrow, pandas, or Arrow library functions.

## Parquet

Because Domino uses [PyArrow](https://arrow.apache.org/docs/python/) to serialize and transport data, the query result is easily written to a local parquet file. You can also use pandas as shown in the CSV example.

```python theme={null}
redshift = DataSourceClient().get_datasource("redshift-test")

res = redshift.query("SELECT * FROM wines LIMIT 1000")

# to_parquet() accepts a path or file-like object
# the whole result is loaded and written once
res.to_parquet("./wines_1000.parquet")
```

```r theme={null}
client <- DominoDataR::datasource_client()
table <- DominoDataR::query(client, "redshift-test", "SELECT* FROM wines LIMIT 1000")

# We can use https://arrow.apache.org/docs/r/reference/write_parquet.html since we leverage the arrow library
arrow::write_parquet(table, "./wines_1000.parquet")
```

## CSV

Because serializing to a CSV is lossy, Domino recommends using the [Pandas.to\_csv](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_csv.html#pandas-dataframe-to-csv) API so you can leverage the multiple options that it provides.

```python theme={null}
redshift = DataSourceClient().get_datasource("redshift-test")

res = redshift.query("SELECT * FROM wines LIMIT 1000")

# See Pandas.to_csv documentation for all options
csv_options = {header: True, quotechar: "'"}

res.to_pandas().to_csv("./wines_1000.csv", **csv_options)
```

```r theme={null}
client <- DominoDataR::datasource_client()
table <- DominoDataR::query(client, "redshift-test", "SELECT* FROM wines LIMIT 1000")

# We can use https://arrow.apache.org/docs/r/reference/write_csv_arrow.html since we leverage the arrow library
arrow::write_csv_arrow(table, "./wines_1000.csv")
```


# Domino Data API
Source: https://docs.domino.ai/cloud/reference/api/data_api/index

Use the Domino Data API libraries for Python and R to query supported file and SQL Data Sources and work with versioned TrainingSets.

The [Python library](https://pypi.org/project/dominodatalab-data/) and [R library](https://github.com/dominodatalab/r-domino-data) provides access to file-based data and SQL-based tabular data. The results are available as dataframe abstractions for popular libraries. You can query connectors interactively, and you don’t have to restart a workload to install drivers.

## Compatibility

This Domino release is compatible with these versions of the Domino Data API:

* Python: `dominodatalab-data>=6.7.4`

* R: `dominodatalab/DominoDataR>=v0.2.3`

## Supported data access methods

* Snowflake

* AWS Redshift

* S3

* Generic-S3

* MySQL

* PostgreSQL

* MSSQL

* Oracle DB

* GCS

* ADLS

* GCP BigQuery

* Databricks

* Azure Blob Storage

For the full list of supported data access methods, see the [Data Source Connectors](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/index) page.

## Additional documentation

* Find out how to [install the Data API](/cloud/reference/api/data_api/install_the_data_api) so that you can use it with the correct Domino environment.

* Learn about the [authentication methods](/cloud/reference/api/data_api/authentication) available when using the Domino Data API.

* Learn how to create and manage connection properties to a supported external data service with Domino’s [Data Source operations](/cloud/reference/api/data_api/datasource_usecases).

* Learn how you can [use Domino TrainingSets](/cloud/reference/api/data_api/trainingsets_usecases) to persist dataframes for model training and other analysis.


# Install the Data API
Source: https://docs.domino.ai/cloud/reference/api/data_api/install_the_data_api

Install compatible Domino Data API libraries for Python and R in a Domino Standard Environment or a custom environment with pip and GitHub.

Use the Data API with the right Domino environment. If you want to install it manually, with `pip`, Domino recommends you install the Data API as an extra dependency of the main Python Domino package.

## Domino standard environment

The Data API comes pre-packaged in the Domino Standard Environment (DSE) starting with version 5.0 for Python and version 5.3 for R.

If you want to use your own environment, you can install the API by adding the following to the Dockerfile Instructions section:

Please refer to the [Index](/cloud/reference/api/data_api) page to find which version of the packages you need to install on your version of Domino.

**To install the Python library**

```dockerfile theme={null}
USER root

## Install Domino and Data API Python packages
RUN python -m pip install dominodatalab[data]

## Alternatively if you need to specify the version of the Data library
RUN python -m pip install dominodatalab dominodatalab-data=={x.y.z}

USER ubuntu
```

**To install the R library**

```dockerfile theme={null}
USER root

## Install Domino Data API as dependency of R library
RUN python -m pip install dominodatalab-data=={x.y.z}

## Install R library via github (using https://remotes.r-lib.org/)
RUN R --no-save -e "install.packages('remotes')"
RUN R --no-save -e "remotes::install_github('dominodatalab/DominoDataR@{a.b.c}')"

USER ubuntu
```

## Python Package Index (PyPI)

<Note>
  If you use the API as a standalone library, the package name is domino\_data. We recommend installing it as part of the main domino package for easier and consistent import see [Tabular store](/cloud/reference/api/data_api/datasource_usecases/simple_query) for an example.
</Note>

```python theme={null}
python -m pip install dominodatalab[data]
```

<Note>
  The Data API is published in [PyPI](https://pypi.org/):
</Note>

```python theme={null}
python -m pip install dominodatalab-data=={x.y.z}
```


# Create TrainingSets
Source: https://docs.domino.ai/cloud/reference/api/data_api/trainingsets_usecases/create_trainingset

Create a versioned TrainingSet from a pandas dataframe with key, target, excluded, monitoring, and metadata columns for model training.

A `TrainingSet` is a versioned set of data, column information, and other metadata. `TrainingSets` are created implicitly when the first `TrainingSetVersion` with a particular `training_set_name` are added using the `create_training_set_version` function.

A `TrainingSet` can include versions from the same project. Attempting to add a version from a different project can result in an error.

`TrainingSet` names are strings containing only alphanumeric characters in the basic Latin alphabet including dash and underscore: `[-A-Za-z_-]`

```python theme={null}
from domino.training_sets import TrainingSetClient, model

training_set_version = TrainingSetClient.create_training_set_version(
    training_set_name=training_set_name,
    df=my_pandas_dataframe,
    key_columns=["user_id", "transaction_id"],
    target_columns=["is_fraud"],
    exclude_columns=["extra_column1", "extra_column2"],
    monitoring_meta=model.MonitoringMeta(
        timestamp_columns=["ts"],
        categorical_columns=["categorical_column1", "categorical_column2"],
        ordinal_columns=["ordinal_column1"],
    ),
    meta={"year": "2021"}
)
```

<Note>
  To use a `TrainingSet` for model monitoring the `monitoring_meta` keyword argument must have a value for classification models. You can create a `TrainingSet` without this argument, but it can not be used until the argument has a value. Trying to register a Domino endpoint for monitoring, from the **Grafana Monitoring** tab for that Domino endpoint, with a null `monitoring_meta` value displays the following error: `The selected Feature Set Version cannot currently be used for monitoring because it does not contain a schema definition.`
</Note>


# Custom authentication
Source: https://docs.domino.ai/cloud/reference/api/data_api/trainingsets_usecases/custom-auth

Authenticate TrainingSetClient with your own user API key by setting DOMINO_USER_API_KEY and, for older clients, DOMINO_API_HOST.

The TrainingSetClient is a Python module wrapping a set of API methods. See [Authentication](/cloud/reference/api/data_api/authentication) for the default behavior of the TrainingSetClient.

To override authentication, you need to set the right environment variable with your own user API key:

```python theme={null}
import os

os.environ["DOMINO_USER_API_KEY"] = "<your-own-api-key>"

# In Domino endpoint if the client version is <0.1.8
os.environ["DOMINO_API_HOST"] = os.getenv("DOMINO_USER_HOST")
```

```shell theme={null}
export DOMINO_USER_API_KEY=<your-own-api-key>

# In Domino endpoint if the client version is <0.1.8
export DOMINO_API_HOST=$DOMINO_USER_HOST
```


# Delete TrainingSets
Source: https://docs.domino.ai/cloud/reference/api/data_api/trainingsets_usecases/delete_trainingset

Delete individual TrainingSetVersions, then remove an empty TrainingSet or iterate through every version before deleting the set.

You can delete `TrainingSets` and `TrainingSetVersions`. The following examples show how to delete `TrainingSets` and `TrainingSetVersions`.

## Delete TrainingSetVersion

```python theme={null}
TrainingSetClient.delete_training_set_version("my-training-set", 2)
```

## Delete TrainingSet

<Note>
  `TrainingSets` may only be deleted if they have no versions.
</Note>

```python theme={null}
TrainingSetClient.delete_training_set("my-training-set")
```

The following example shows how to delete the `TrainingSet` and all versions:

```python theme={null}
found_tsvs = TrainingSetClient.list_training_set_versions(
    training_set_name="my-training-set",
)

for tsv in found_tsvs:
    TrainingSetClient.delete_training_set_version(tsv.training_set_name, tsv.number)

TrainingSetClient.delete_training_set("my-training-set")
```


# Retrieve TrainingSets
Source: https://docs.domino.ai/cloud/reference/api/data_api/trainingsets_usecases/get_trainingset

Retrieve TrainingSets and TrainingSetVersions by name or number, load raw or training data, and search versions or sets by metadata.

You can get `TrainingSets` using their version number or using an explicit search.

## Get TrainingSetVersion

`TrainingSetVersions` can be retrieved by name and number.

```python theme={null}
tsv_by_num = TrainingSetClient.get_training_set_version(
    training_set_name="my-training-set",
    number=2,
)
```

You can also access the raw data from the `TrainingSetVersion`.

```python theme={null}
raw_df = tsv_by_num.load_raw_pandas()
```

Training data can also be accessed.

```python theme={null}
training_df = tsv_by_num.load_training_pandas()
```

## Find TrainingSetVersions

You can search for `TrainingSetVersions` for the current project using the training set name, `TrainingSet` metadata, and `TrainingSetVersion` metadata. Results matching all search fields are returned.

```python theme={null}
versions = TrainingSetClient.list_training_set_versions(
    training_set_name="my-training-set",
    meta={"year": "2021"},
    training_set_meta={"category": "widgets"}
)
```

## Get TrainingSet

You can retrieve `TrainingSets` by name.

```python theme={null}
ts = TrainingSetClient.get_training_set("my-training-set")
```

## Find TrainingSets

You can search `TrainingSets` by metadata. Results matching all metadata fields are returned.

```python theme={null}
ts = TrainingSetClient.list_training_sets(
    meta={"category": "widgets"},
)
```


# TrainingSets use cases
Source: https://docs.domino.ai/cloud/reference/api/data_api/trainingsets_usecases/index

Use TrainingSets to persist versioned dataframes and monitoring metadata within a Project, then create, retrieve, update, or delete them.

<Note>
  This feature is accessible only through the [Domino Data API](/cloud/reference/api/data_api/index).
</Note>

Domino training sets contain the initial data used for training your machine learning models. A `TrainingSet` is a versioned set of data, column information, and other metadata.

Use Domino Training Sets to persist dataframes for model training and other analysis. You can store and load multiple versions of a given dataframe from a training set so you can connect a model to the specific version of a dataframe that was used to train it.

The dataframe used as the basis for a Training Set can be constructed with the result of a Domino Data Source query (as described previously) or through any other construction method.

In addition to storing the underlying dataframe, training sets can be used to capture additional monitoring metadata. When this additional data is present, Training Set versions are available as sources of baseline training data when publishing Domino endpoints in Domino.

Training sets are only scoped to the projects in which they are created. Users with project contributor permissions can create, load, and delete Training Set versions in that project.

Training sets are available as an API-only feature.

The following is a list of `TrainingSet` operations:

* [Create TrainingSets](/cloud/reference/api/data_api/trainingsets_usecases/create_trainingset)

* [Retrieve TrainingSets](/cloud/reference/api/data_api/trainingsets_usecases/get_trainingset)

* [Update TrainingSets](/cloud/reference/api/data_api/trainingsets_usecases/update_trainingset)

* [Delete TrainingSets](/cloud/reference/api/data_api/trainingsets_usecases/delete_trainingset)

You can also override authentication. For more information, see [custom authentication](/cloud/reference/api/data_api/trainingsets_usecases/custom-auth).


# Update TrainingSet
Source: https://docs.domino.ai/cloud/reference/api/data_api/trainingsets_usecases/update_trainingset

Update a TrainingSet or TrainingSetVersion description and metadata with TrainingSetClient, then access the returned updated object.

The following example shows how to update your `TrainingSet`:

```python theme={null}
ts = TrainingSetClient.get_training_set("my-training-set")

ts.description = "widget transactions"
ts.metadata.update({"region": "west"})

updated = TrainingSetClient.update_training_set(ts)
```

## Update TrainingSetVersion

The following example shows how to update your TrainingSetVersion:

```python theme={null}
tsv = TrainingSetClient.get_training_set_version("my-training-set", 2)

tsv.description = "2020 transactions"
tsv.metadata.update({"status": "final"})

updated = TrainingSetClient.update_training_set_version(tsv)
```


# Domino NetApp Volumes API
Source: https://docs.domino.ai/cloud/reference/api/domino-volumes-api

Use the Domino NetApp Volumes API at /remotefs/v1 to manage filesystems and transfers through endpoints in the Domino Platform API reference.

The Domino NetApp Volumes API gives you programmatic access to
[Domino Volumes for NetApp ONTAP](/cloud/admin/data-administration/netapp-volumes).
It is served at `/remotefs/v1`.

Its endpoints are documented in the
[Domino Platform API reference](/cloud/reference/api/domino-open-api), under the
[NetApp Volumes](/api-reference/netapp-volumes/list-volumes),
[NetApp Volumes snapshots](/api-reference/netapp-volumes-snapshots/create-new-snapshot-from-a-run),
[NetApp Volumes filesystems](/api-reference/netapp-volumes-filesystems/list-filesystem),
and [NetApp Volumes transfers](/api-reference/netapp-volumes-transfers/list-data-transfers)
groups.


# Flyte native API
Source: https://docs.domino.ai/cloud/reference/api/flyte/flyte-native-api

Use the Flyte native API for workflows, tasks, launch plans, executions, projects, domains, and metadata that support Domino Flows.

The Flyte native API is the upstream Flyte control plane API, which backs
[Domino Flows](/cloud/platform-capabilities/features/flows). It covers workflows,
tasks, launch plans, node and task executions, and the project and domain
metadata around them.

Domino serves it under `/flows`, so every path in the reference is relative to
`https://<your-domino-hostname>/flows`.

<Note>
  Domino vendors Flyte rather than authoring it, so these endpoint descriptions
  come from the upstream Flyte project. Two parts of the REST surface Flyte
  actually serves are absent from the specification it publishes, and so from
  this reference: `/api/v1/dataproxy/artifact_urn`, and the SignalService
  operations.
</Note>


# Flyte API
Source: https://docs.domino.ai/cloud/reference/api/flyte/index

Find the Flyte native and extensions API references for the endpoints and models that control Domino Flows and manage execution artifacts.

[Flyte native API](/cloud/reference/api/flyte/flyte-native-api)\
A reference guide with detailed information about the native Flyte API endpoints and models.

[Flyte extensions API](/cloud/reference/api/flyte/flyte-extensions-api)\
A reference guide with detailed information about the Flyte API extensions endpoints and models.


# Governance API
Source: https://docs.domino.ai/cloud/reference/api/governance-api

Use the Governance API at /api/governance/v1 to manage Domino Governance bundles, policies, evidence, results, approvals, and operations.

The Governance API manages Domino Governance bundles, policies, evidence, and
approvals. It is served at `/api/governance/v1`.

Its endpoints are documented in the
[Domino Platform API reference](/cloud/reference/api/domino-open-api), under the
[Governance bundles](/api-reference/governance-bundles/list-bundle-stages-stage-view),
[Governance policies](/api-reference/governance-policies/create-or-update-a-policy),
[Governance evidence and results](/api-reference/governance-evidence-and-results/list-findings),
and [Governance operations](/api-reference/governance-operations/get-a-bundles-evidence-results-and-approval-progress-against-one-of-its-applied-policies)
groups.


# Model Monitoring custom metrics Python SDK
Source: https://docs.domino.ai/cloud/reference/api/model_monitoring/custom-metrics

Use the Model Monitoring custom metrics Python SDK to register and log metrics, trigger threshold alerts, and retrieve values by time range.

Use Domino’s Model Monitoring custom metrics Python SDK to define custom metrics and use them alongside out-of-the-box drift and model quality metrics that are monitored in Domino Model Monitor. With this SDK, you can register new metrics and define the logic to compute them. You can author this logic and evaluate it from within a Domino project.

For every model that you register for monitoring, you can select a registered metric, associate the data sources from which the metric is computed, and set up the execution environment to compute this metric on a periodic basis. You are notified by email when a metric behaves abnormally based on threshold definitions.

For end-to-end working code with a description of the workflow, see the `/custom metrics example` folder in the quick-start project.

## Use custom metrics

Use the following library functions from the custom metrics Python SDK to build and deploy your own monitoring metrics.

1. Instantiate the client. Enter your `DMM model ID`:

   ```python theme={null}
   import domino
   dmm_model_id = "XXXXXXXXXXXXX"
   d = domino.Domino("integration-test/quick-start")
   metrics_client = d.custom_metrics_client()
   ```

2. Log the custom metrics:

   ```python theme={null}
   metrics_client.log_metric(dmm_model_id, "accuracy", 7.1234, "2022-10-08T00:00:00Z", { "example_tag1" : "value1", "example_tag2" : "value2" })

   metrics_client.log_metrics([
   { "modelMonitoringId" : dmm_model_id, "metric" : "accuracy", "value" : 7.1234,
   "timestamp" : "2022-10-08T00:00:00Z",
   "tags" : { "example_tag1" : "value1", "example_tag2" : "value2" }
   ]
   },
   { "modelMonitoringId" : dmm_model_id, "metric" : "accuracy", "value" : 8.4567,
   "timestamp" : "2022-10-09T00:00:00Z" }
   ])
   ```

   * `modelMonitoringId`: ID of the monitored model to send metric alerts for

   * `metric`: Name of the metric to send alert for

   * `value`: Value of the metric

   * `timestamp`: Timezone is in UTC in ISO 8601 format.

   * `tags`: Custom metadata for metric represented as key-value string pairs

3. Send a custom metrics alert:

   ```python theme={null}
   metrics_client.trigger_alert(dmm_model_id, "metric_name", 3.14,
   condition = metrics_client.BETWEEN,
   lower_limit=2.0, upper_limit=3.0,
   description = "Breached 2.0-3.0 range." )
   ```

   * `modelMonitoringId`: ID of the monitored model for which to send metric alerts.

   * `metric`: Name of the metric for which to send the alert.

   * `value`: Value of the metric.

   * `condition`: Target range for the metric defined by lower and upper limit bounds.\
     The following are potential values for the `condition` argument:

     * `metrics_client.LESS_THAN = "lessThan"`

     * `metrics_client.LESS_THAN_EQUAL = "lessThanEqual"`

     * `metrics_client.GREATER_THAN = "greaterThan"`

     * `metrics_client.GREATER_THAN_EQUAL = "greaterThanEqual"`

     * `metrics_client.BETWEEN = "between"`

   * `lower_limit`: The lower limit for the `condition`.

   * `upper_limit`: The upper limit for the `condition`.

   * `description`: Optional message included in the alert.

4. Retrieve the custom metrics:

   ```python theme={null}
   res = metrics_client.read_metrics(dmm_model_id, "accuracy",
   "2022-10-08T00:00:00Z", "2022-10-10T00:00:00Z" )
   ```

   * `modelMonitoringId`: ID of the monitored model for which to retrieve the metric values.

   * `metric`: Name of the metric for which to retrieve the metric values.

   * `start_timestamp`: The start timestamp of the range when the metrics were logged. The timezone is in UTC in ISO 8601 format.

   * `end_timestamp`: The end timestamp of the range when the metrics were logged. The timezone is in UTC in ISO 8601 format.

## Sample output

```
{
  'metricValues': [
      {'timestamp': '2022-10-08T00:00:00Z', 'value': 3.123, 'tags': []},
      {'timestamp': '2022-10-08T00:00:00Z', 'value': 3.123, 'tags': []},
      {'timestamp': '2022-10-08T00:00:00Z', 'value': 3.123, 'tags': []},
      {'timestamp': '2022-10-08T00:00:00Z', 'value': 3.123, 'tags': []},
{'timestamp': '2022-10-08T00:00:00Z', 'value': 7.1234, 'tags': {'example_tag1': 'value1', 'example_tag2': 'value2'}},
      {'timestamp': '2022-10-09T00:00:00Z', 'value': 8.4567, 'tags': []}
   ],
   'metadata': {'requestId': '97e61f63-3be5-4766-a07a-06c931d91268', 'notices': []}
}
```

Results:

* `timestamp`: The time this metric was logged.

* `value`: The value of the metric.

* `tags`: Custom metadata for metric represented as key-value string pairs.


# Model Monitoring API reference
Source: https://docs.domino.ai/cloud/reference/api/model_monitoring/dmm-api

Use Model Monitoring API endpoints to manage monitored models, drift and quality, settings, authentication, and prediction data ingestion.

The Model Monitoring API gives you programmatic access to the [Model Monitor](/cloud/platform-capabilities/features/monitoring).

To invoke these APIs, you must pass the API token in the header. See [API Token](/cloud/platform-capabilities/features/monitoring/7-monitor-settings/1-api) for more information.

Domino Model Monitoring can only ingest prediction data for Domino endpoints in the form of [Domino Datasets](/cloud/admin/data-administration/datasets) captured using [Domino’s prediction capture library](/cloud/platform-capabilities/features/monitoring/3-set-up-domino-endpoints/1-set-up-prediction-capture). To ingest different data types, create a new model in Domino Model Monitor and use a Model Monitor Data Source.

The endpoints are documented in the [Domino Platform API
reference](/cloud/reference/api/domino-open-api), under the [Model Monitoring
models](/api-reference/model-monitoring-models/get-model-monitorv2apimodelget-prediction-traffic),
[Model Monitoring drift and quality](/api-reference/model-monitoring-drift-and-quality/post-model-monitorv2apimodel-analyse-drift),
[Model Monitoring settings](/api-reference/model-monitoring-settings/get-global-settings),
and [Model Monitoring authentication](/api-reference/model-monitoring-authentication/get-model-monitorv2apiauthlogin)
groups.


# Model Monitoring API
Source: https://docs.domino.ai/cloud/reference/api/model_monitoring/index

Find the Model Monitoring API reference and Python SDK for accessing monitored models, defining custom metrics, and managing metric data.

[Model Monitoring API reference](/cloud/reference/api/model_monitoring/dmm-api)\
A reference guide with detailed information about the Model Monitoring API endpoints and models.

[Model Monitoring custom metrics Python SDK](/cloud/reference/api/model_monitoring/custom-metrics)\
A guide on how to define custom metrics and use them alongside out-of-the-box drift and model quality metrics that are monitored in Domino.


# Tags API
Source: https://docs.domino.ai/cloud/reference/api/tags-api

Use the Tags API at /api/taxonomy/v1 to manage Domino Taxonomy namespaces, tags, properties, and their associated operations.

The Tags API manages the Domino Taxonomy and Properties features: namespaces,
[tags](/cloud/platform-capabilities/core-concepts/knowledge-management/tags), and
[properties](/cloud/platform-capabilities/core-concepts/knowledge-management/properties).
It is served at `/api/taxonomy/v1`.

Its endpoints are documented in the
[Domino Platform API reference](/cloud/reference/api/domino-open-api), under the
[Tags](/api-reference/tags/list-tags) and
[Properties](/api-reference/properties/list-properties) groups.


# Workspace File Audit Trail API
Source: https://docs.domino.ai/cloud/reference/api/workspace-file-audit-trail-api

Use the Workspace File Audit Trail API to query and export Workspace file access events for governance review and audit workflows.

The Workspace File Audit Trail API lets you query and export Workspace file
access events. To see what the API records, read
[Use the Workspace File Audit app](/cloud/platform-capabilities/features/governance/audit-trail/workspace-file-access-events/use-workspace-file-audit-app).


# Generate Client Code
Source: https://docs.domino.ai/api-reference/artifacts/generate-client-code

/api-specs/cloud/flyte-extensions-api.json get /artifacts/v1/artifacts/{artifact-id}/versions/{execution-name}/code
Generates Python client code to download the Artifact version.



# List
Source: https://docs.domino.ai/api-reference/artifacts/list

/api-specs/cloud/flyte-extensions-api.json get /artifacts/v1/artifacts
Lists all of the artifacts associated with a Flyte execution. It reads metadata from Flyte APIs. The response will not contain the blob sizes of any files.



# Retrieve contents
Source: https://docs.domino.ai/api-reference/blob/retrieve-contents

/api-specs/cloud/flyte-extensions-api.json get /artifacts/v1/blobs/render
Given the blob storage uri for an output in Flyte, streams the content (up to a max configured limit)



# Retrieve size
Source: https://docs.domino.ai/api-reference/blob/retrieve-size

/api-specs/cloud/flyte-extensions-api.json get /artifacts/v1/blobs/size
Given the blob storage uri for an output in Flyte, retrieves the size of the content.



# Bulk update
Source: https://docs.domino.ai/api-reference/bookmarks/bulk-update

/api-specs/cloud/flyte-extensions-api.json post /artifacts/v1/bookmarks/bulk-update
Bulk update a list of bookmarks.



# Create
Source: https://docs.domino.ai/api-reference/bookmarks/create

/api-specs/cloud/flyte-extensions-api.json post /artifacts/v1/bookmarks
Create a bookmark for an artifact version.



# Delete
Source: https://docs.domino.ai/api-reference/bookmarks/delete

/api-specs/cloud/flyte-extensions-api.json delete /artifacts/v1/artifacts/{artifact-id}/bookmarks/{execution-name}
Delete a bookmark for an artifact version.



# Get a single artifact file
Source: https://docs.domino.ai/api-reference/bookmarks/get-a-single-artifact-file

/api-specs/cloud/flyte-extensions-api.json get /artifacts/v1/artifacts/{artifact-id}/bookmarks/{execution-name}/files/{file-name}
Get an artifact file from a bookmarked artifact version from an execution.



# Get a single bookmark
Source: https://docs.domino.ai/api-reference/bookmarks/get-a-single-bookmark

/api-specs/cloud/flyte-extensions-api.json get /artifacts/v1/artifacts/{artifact-id}/bookmarks/{execution-name}
Get a single bookmarked Flyte artifact version from an execution.



# List
Source: https://docs.domino.ai/api-reference/bookmarks/list

/api-specs/cloud/flyte-extensions-api.json get /artifacts/v1/bookmarks
Lists all of the bookmarks associated with a project.



# Re run with overrides
Source: https://docs.domino.ai/api-reference/execution/re-run-with-overrides

/api-specs/cloud/flyte-extensions-api.json post /executions/v1/execution/{execution-id}/rerunWithOverrides
Re-run a Flyte execution with the same inputs but with overridden parameters



# Create
Source: https://docs.domino.ai/api-reference/exports/create

/api-specs/cloud/flyte-extensions-api.json post /artifacts/v1/exports-callback
Create an export for an artifact version as part of a callback from special clients. This currently only supports the Filetask client, as a callback from the creation of dataset-export types, in order to track the lineage. It is the requesting client's responsibility to ensure the correctness of the export lineage.



# List
Source: https://docs.domino.ai/api-reference/exports/list

/api-specs/cloud/flyte-extensions-api.json get /artifacts/v1/exports
Lists all of the artifact versions exports associated with an export target.



# Prepare artifact version for export
Source: https://docs.domino.ai/api-reference/exports/prepare-artifact-version-for-export

/api-specs/cloud/flyte-extensions-api.json get /artifacts/v1/artifacts/{artifact-id}/versions/{execution-name}/export-types/{export-type}/exports/{export-target-id}/prepare
Prepare an artifact version to be exported by returning a signed payload containing presigned URLs for the artifact files.



# Create and upload a Workflow definition
Source: https://docs.domino.ai/api-reference/workflows/create-and-upload-a-workflow-definition

/api-specs/cloud/flyte-native-api.json post /api/v1/workflows
Create and register a workflow definition.



# Fetch a list of Workflow definitions.
Source: https://docs.domino.ai/api-reference/workflows/fetch-a-list-of-workflow-definitions

/api-specs/cloud/flyte-native-api.json get /api/v1/workflows/{id.project}/{id.domain}
Fetch existing workflow definitions matching input filters.



# Fetch a list of Workflow definitions (by name)
Source: https://docs.domino.ai/api-reference/workflows/fetch-a-list-of-workflow-definitions-by-name

/api-specs/cloud/flyte-native-api.json get /api/v1/workflows/{id.project}/{id.domain}/{id.name}
Fetch existing workflow definitions matching input filters.



# Fetch a Workflow definition.
Source: https://docs.domino.ai/api-reference/workflows/fetch-a-workflow-definition

/api-specs/cloud/flyte-native-api.json get /api/v1/workflows/{id.project}/{id.domain}/{id.name}/{id.version}
Retrieve an existing workflow definition.



# Blueprints
Source: https://docs.domino.ai/cloud/examples/blueprints

Implementation guides from the Domino Professional Services team, published in the Blueprints catalog on domino.ai.

Blueprints are guides from the Domino Professional Services team: quick starts, code, and examples for configuration and integration work such as propagating cloud credentials into workloads, mounting object storage as a local filesystem, or running coding assistants against Domino. Some blueprints ship runnable code in the [domino-blueprints](https://github.com/dominodatalab/domino-blueprints) repository.

Every blueprint is published on domino.ai, and the [Blueprints catalog](https://domino.ai/resources/blueprints) is the authoritative list, carrying anything published since this page was last updated.

<Columns>
  <Card title="Using Cline on Domino" href="https://domino.ai/resources/blueprints/cline-on-domino">
    Coding assistants
  </Card>

  <Card title="Integrating Amazon Bedrock AgentCore with Domino" href="https://domino.ai/resources/blueprints/integrating-amazon-bedrock-agentcore-with-domino">
    Agentic AI
  </Card>

  <Card title="A minimum privilege guide to Domino Volumes on Azure with NetApp ONTAP" href="https://domino.ai/resources/blueprints/minimum-privilege-guide-to-domino-volumes-on-azure-with-netapp-ontap">
    Security
  </Card>

  <Card title="Deploying the Domino LLM Gateway" href="https://domino.ai/resources/blueprints/deploying-the-domino-llm-gateway">
    Coding assistants
  </Card>

  <Card title="Claude Code in RStudio: bringing AI-assisted clinical programming to pharma" href="https://domino.ai/resources/blueprints/claude-code-in-r-studio">
    Coding assistants
  </Card>

  <Card title="Composable guardrails for agentic AI using a defense-in-depth approach" href="https://domino.ai/resources/blueprints/composable-guardrails-for-agentic-ai">
    Agentic AI
  </Card>

  <Card title="Governed FSI onboarding workflows with coding assistants" href="https://domino.ai/resources/blueprints/coding-assistants-to-build-governed-fsi-onboarding-workflows">
    Coding assistants
  </Card>

  <Card title="Building an agentic RAG pipeline with multi-model orchestration on Domino" href="https://domino.ai/resources/blueprints/building-agentic-rag-pipeline-with-multi-model-orchestration">
    Agentic AI
  </Card>

  <Card title="Claude Code on Domino: AI-assisted development with Domino skills" href="https://domino.ai/resources/blueprints/claude-code-on-domino">
    Coding assistants
  </Card>

  <Card title="Getting started with the Domino VS Code Extension" href="https://domino.ai/resources/blueprints/getting-started-domino-vs-code-extension">
    Getting started
  </Card>

  <Card title="Integrate SCE workflows with Domino" href="https://domino.ai/resources/blueprints/integrate-sce-workflows-with-domino">
    Statistical Compute Environment (SCE)
  </Card>

  <Card title="Deploying self-hosted LLMs in Domino" href="https://domino.ai/resources/blueprints/deploying-self-hosted-llms">
    Agentic AI
  </Card>

  <Card title="Cut cloud costs by enabling spot instances on Domino" href="https://domino.ai/resources/blueprints/cut-cloud-cost-by-enabling-spot-instances">
    Compute
  </Card>

  <Card title="Deploying agentic AI systems in Domino" href="https://domino.ai/resources/blueprints/deploying-agentic-ai-systems">
    Agentic AI
  </Card>

  <Card title="Turn cloud object storage into local filesystems for Domino: Azure and Google" href="https://domino.ai/resources/blueprints/turn-cloud-object-storage-into-local-filesystems">
    Filesystem in Userspace (FUSE)
  </Card>

  <Card title="Automated GenAI tracing for agent and LLM experimentation" href="https://domino.ai/resources/blueprints/automated-genai-tracing-for-agent-and-llm-experimentation">
    GenAI
  </Card>

  <Card title="React app deployment with CI/CD on Domino" href="https://domino.ai/resources/blueprints/react-app-deployment-with-ci-cd-on-domino">
    Apps
  </Card>

  <Card title="How to ensure reliable model endpoints in Domino" href="https://domino.ai/resources/blueprints/production-monitoring-for-mlops">
    MLOps
  </Card>

  <Card title="Data-driven model performance optimization" href="https://domino.ai/resources/blueprints/how-to-monitor-and-optimize-endpoints">
    MLOps
  </Card>

  <Card title="Using Domino Experiment Manager from an external client" href="https://domino.ai/resources/blueprints/using-domino-experiment-manager-from-external-client">
    MLOps
  </Card>

  <Card title="On-demand distributed clusters: API-driven scaling" href="https://domino.ai/resources/blueprints/on-demand-distributed-clusters">
    MLOps
  </Card>

  <Card title="Azure authorization" href="https://domino.ai/resources/blueprints/azure-authorization">
    Security
  </Card>

  <Card title="Vibe modeling with Domino" href="https://domino.ai/resources/blueprints/vibe-modeling">
    Vibe modeling
  </Card>

  <Card title="Transforming S3 into local storage: Mountpoint solutions for Domino data science workflows" href="https://domino.ai/resources/blueprints/transforming-s3-into-local-storage">
    Filesystem in Userspace (FUSE)
  </Card>

  <Card title="Tuning SAS for the cloud: How to run high-performance SAS workloads on Domino" href="https://domino.ai/resources/blueprints/run-high-performance-sas-workloads-on-domino">
    Cloud
  </Card>

  <Card title="Efficient and scalable GPU model inference on Domino with NVIDIA Triton inference framework" href="https://domino.ai/resources/blueprints/efficient-scalable-gpu-inference-on-domino-with-nvidia-triton">
    MLOps
  </Card>

  <Card title="LLMOps in Domino: integrating LLMs with model registry and Domino Model API" href="https://domino.ai/resources/blueprints/integrating-llms-with-model-registry-and-domino-model-api">
    MLOps
  </Card>

  <Card title="Azure Entra user-based credential propagation for Domino" href="https://domino.ai/resources/blueprints/domino-azure-credential-propagation">
    Security
  </Card>

  <Card title="Adopting IAM Role for Service Accounts (IRSA) for credential propagation in Domino" href="https://domino.ai/resources/blueprints/aws-irsa">
    Security
  </Card>

  <Card title="Domino advanced credential propagation for AWS with EKS Pod Identity" href="https://domino.ai/resources/blueprints/advanced-credential-propagation-aws-eks">
    Security
  </Card>
</Columns>


# Code examples
Source: https://docs.domino.ai/cloud/examples/code-examples

Domino's runnable example code: the quick-start Project created for every new user, and the public example repositories on GitHub.

Domino publishes runnable example code in two places: the `quick-start` Project that Domino creates inside your own deployment, and a set of public GitHub repositories you can open as Projects of your own. Start with `quick-start` for the platform basics, then pick a repository when you need a specific capability such as an agent, distributed training, or a flow.

## The `quick-start` Project

Domino creates a `quick-start` Project for every new user at signup. It has example files that show how to take advantage of Domino's features, and a detailed `README`. Because Domino provisions it for you, it's the quickest example to open. Some features ship their example code there: the `/custom metrics example` folder holds end-to-end code for [custom Model Monitoring metrics](/cloud/platform-capabilities/features/monitoring/5-use-monitoring/2-custom-metrics).

To start new users on Projects your own team maintains instead, ask Domino Support to change the [default Project configuration record](/cloud/admin/platform-configuration/configure-projects#default-projects-for-new-users).

## Example repositories on GitHub

Open any of these as a Git-based Project. Each group links to the documentation for the capability; the repositories hold the code.

### Agents

[Agents](/cloud/platform-capabilities/features/agents) explains how you build an agentic system in Domino, evaluate it against real data, and ship it.

`simple_domino_agent` is a minimal agent with tool calls, tracing, evaluation, and deployment, plus a production app with a chat UI in `chat_app.py`.

<GitHub.Repo />

`rag-agent-demo` is a retrieval-augmented generation (RAG) agent built on ChromaDB, using Domino's tracing instrumentation.

<GitHub.Repo />

`simple_agent_api_only` is a REST API and agent-to-agent (A2A) example, with instructions for pairing it with a Domino-hosted agent registry and orchestrator.

<GitHub.Repo />

### Distributed computing

[Scale distributed workloads](/cloud/platform-capabilities/features/development/distributed-training) covers the on-demand Spark, Dask, Ray, and MPI clusters these examples run on.

`domino-quickstart-ray` covers Ray environment setup and multiple types of training algorithms.

<GitHub.Repo />

`domino-quickstart-dask` covers Dask environment setup and multiple types of training algorithms.

<GitHub.Repo />

`reference-project-dask` is a Dask pipeline covering environment setup, data pipelines, and model training. Start with `dask_ml.ipynb`.

<GitHub.Repo />

`spark-quickstart-winequality` covers Spark environment setup, Project creation, and model training.

<GitHub.Repo />

### Model Registry

[Model Registry](/cloud/platform-capabilities/features/models/register-and-govern/model-registry) covers how Domino tracks and manages the models these notebooks register.

`reference-project-domino-mlflow-supported-models` holds notebooks that register XGBoost, Sklearn, and Python-function MLflow models, two of them with integrated model monitoring.

<GitHub.Repo />

### Flows

[Get started with Flows](/cloud/platform-capabilities/features/flows/get-started-with-flows) builds a basic flow and then a training flow.

`domino-ai-flows` is the template repository you fork to build a training flow.

<GitHub.Repo />

## Run an example in Domino

<Steps>
  <Step title="Fork the repository if you plan to change it">
    Fork it to your own Git provider so you have somewhere to push commits. Skip this if you only want to read and run the code as published.
  </Step>

  <Step title="Add Git credentials if you'll write commits back">
    Domino reads a public repository without credentials. Writing commits back to the remote, using a private repository, or connecting over SSH each need [Git credentials](/cloud/platform-capabilities/core-concepts/projects/create-projects#add-git-credentials) stored in your Domino account.
  </Step>

  <Step title="Create a Git-based Project that points at it">
    Follow [Create a Project](/cloud/platform-capabilities/core-concepts/projects/create-projects#create-a-project) and give Domino the repository URL.
  </Step>

  <Step title="Open a Workspace or run a Job">
    Work through the code interactively in a Workspace, or run one of its scripts as a Job.
  </Step>
</Steps>

## Related

* [Getting started with Domino](/cloud/platform-overview/getting-started-with-domino): a guided tutorial that builds and deploys a transaction review App, from a Git-based Project through an authenticated model endpoint.
* [The python-domino library](/cloud/reference/python-sdk/python-domino): the Python bindings for the Domino APIs, whose repository carries its own example code.
* [Blueprints](/cloud/examples/blueprints)


# Flyte extensions API
Source: https://docs.domino.ai/cloud/reference/api/flyte/flyte-extensions-api

Use Domino's Flyte extensions API to retrieve, render, size, bookmark, and export artifacts or rerun Flows executions with overrides.

The Flyte extensions API covers the operations Domino adds to Flyte in support of
[Domino Flows](/cloud/platform-capabilities/features/flows): retrieving artifact
metadata, rendering and sizing artifact blobs, bookmarking the artifacts an
execution produced, exporting them, and rerunning an execution with overrides.

Domino serves it under `/flows/api`, so every path in the reference is relative
to `https://<your-domino-hostname>/flows/api`.


# Domino CLI reference
Source: https://docs.domino.ai/cloud/reference/cli/cli-reference

Use the Domino CLI command reference to run code, create and restore Projects, sync files, manage Datasets, get help, and check versions.

## Install and get started with the CLI

[How to install and get started with the Domino CLI](/cloud/reference/cli/install-the-cli)

## Run your code

`run [--direct][--wait][--no-sync][--tier][--local] <file> [args…​]`

"run" copies all files in your project folder to the cloud, where it executes the specified file with any arguments you have specified, unless you have specified the `--local` flag.

Use `domino sync` before `domino run <script>` if receiving an error like “blob file not found”.

The optional `--wait` flag runs your script synchronously, that is, the command does not return until your job finishes.

The optional `--direct` flag lets you run a shell command directly on the Domino machines.

The optional `--no-sync` flag runs the latest version of code on the server, without uploading any local changes on your computer.

The optional `--tier` flag lets you select a specific hardware tier to execute the specified file on.

The optional `--local` flag lets you run your script on your local machine.

The optional `--title` flag lets you supply a title for the run.

Local runs are useful if your desired compute environment is not immediately available, but you would like to record your results in Domino. When a local run is invoked, Domino commits a snapshot of project files to the server and then runs the given command (for example, `python main.py`) on the local machine. When the command has finished running, Domino detects results that have been produced and commits those back to the server, tracking them as “results” of the run as though the command ran on the server.

Viewing the “Results” for Run #16 brings up a page showing the output of the `main.py` where colleagues can view and discuss the output.

Examples

```shell theme={null}
domino run calculateStats.R
domino run runModel.m model1 .05
domino run --wait runModel.m model1 .05
domino run --direct "pip freeze | grep pandas"
domino run --tier "Large" calculateStats.R
domino run --local "python main.py"
```

## Projects

### Create a new Project

`create [projectName]`

`create` creates a new Project by making a new folder on your computer and telling the Domino server about your new Project. Optionally, you can specify the name of the Project. If you don’t specify a name, Domino asks you for one. To create a Project on behalf of an organization you belong to, use the `--owner` flag.

Examples

```shell theme={null}
domino create
domino create myProject
domino create --owner myOrg myProject
```

### Create a Project in a directory that already has files

`init [projectName]`

`init` creates a new Domino Project *inside your current folder*. This is useful if you already have a working folder and you’d like to convert it to a Domino Project. While the "create" command creates a new folder inside your current directory, `init` initializes a Project from your current folder. The `--owner` flag can be used to set an organization you belong to as the Project owner.

<Note>
  `init` does **not** upload any of your files. You’ll have to use `upload` or `run` after you run `init`.
</Note>

Examples

```shell theme={null}
domino init
domino init myProject
domino init --owner myOrg myProject
```

### Get a Project from the server onto your computer for the first time

`get [username]/projectName`

`get` finds an existing Project on the server and copies it to your computer. This is useful to get a project you created on a different computer or to get a Project that someone else created, and you are collaborating on.

If you are getting someone else’s Project, specify their username and the Project name. If you are getting one of your own Projects, you can omit the username.

Examples

```shell theme={null}
domino get otherUser/someProject
domino get myProject
```

### Connect a folder to an existing Project

`restore`

`restore` connects your current directory to an existing Domino Project on the server. Restore looks for evidence of a Project name inside `.domino/config.json`. If it doesn’t find such a file, it prompts you for the name of an existing Project to use.

This is particularly useful if you are using Domino and Git to track the same folder. If you clone a Project with Git, Git likely ignores the hidden files that identify it as a Domino Project. So you can run `git clone` and then `domino restore` to re-connect the folder to its Domino Project.

Example

```shell theme={null}
domino restore
```

## Files

### Synchronize the files on your computer with the server

`sync`

`sync` synchronizes the Project folder on your computer with the Project stored on the server. This is equivalent to running a download followed by an upload.

Example

```shell theme={null}
domino sync
```

<Tip>
  If `sync` fails with an error like "too many open files" on Projects with many files, consider using a [.noLock](/cloud/platform-capabilities/core-concepts/projects/files-artifacts-and-results#special-files) file.
</Tip>

### Download the latest version of your files

`download`

`download` downloads the latest copy of your files from the cloud into your current project folder. If you have made changes that conflict with changes in the cloud, you see both versions of the conflicting file side-by-side.

There are two reasons files in the cloud might change: first, your collaborators on a Project might make changes; second, you might have executed a run that produced new output files.

Example

```shell theme={null}
domino download
```

### Upload your files without starting a run

`upload [-m "message"]`

`upload` uploads your current project folder contents to the cloud, but does not begin a new run. The optional message flag lets you record a message, which is displayed when browsing past commits on the **Code** tab of your project.

<Note>
  When using the CLI to upload more than 1000 files at once, a timeout error might occur during the upload process. To avoid this, upload smaller batches of files.
</Note>

Example

```shell theme={null}
domino upload -m "upload data files"
```

## Datasets

<Note>
  The following Dataset CLI actions may cause you to exceed your quota threshold if your admin has created one. See [Dataset quotas](/cloud/platform-capabilities/core-concepts/data/datasets/manage-large-data#dataset-quotas-and-limits) for more information.
</Note>

### Upload folder to a Dataset

Upload all the files in a folder from your local machine to an existing dataset.

<Warning>
  This command overwrites existing files unless you use the `‑‑fileUploadSetting` option.
</Warning>

Syntax:

```shell theme={null}
domino [--fileUploadSetting Ignore|Overwrite|Rename] [--targetRelativePath <destination path>] <project-owner>/<project-name>/<dataset-name> <folder path>
```

The optional `fileUploadSetting` flag handles path collisions:

* `Ignore`: If a file already exists in the Dataset, ignore the new file.

* `Overwrite`: If a file already exists in the Dataset, overwrite the existing file with the new file.

* `Rename`: If a file already exists in the Dataset, append `_1` to the uploaded filename. For example, if `/Users/myUser/data/file.txt` already exists then the newly uploaded file becomes `/Users/myUser/data/file_1.txt`.

The optional `targetRelativePath` flag enables uploading to a subdirectory in the Dataset:

* `<destination path>`: This is the path, relative to the root directory of the Dataset, where files are uploaded. Note that other users with access to the Dataset can alter the file structure, which may require you to modify the destination path accordingly.

  If `targetRelativePath` is not specified, files are uploaded to the root directory of the Dataset.

<Note>
  `fileUploadSetting`, `targetRelativePath`, and their values are case-sensitive.
</Note>

Example:

```shell theme={null}
domino upload-dataset --fileUploadSetting Overwrite --targetRelativePath Experiments/Run1/Analysis jsmith/global-predictions/global-data /Users/myUser/data
```

You can take advantage of the Datasets parallel upload capability, which allows for configurable chunk sizes and multi-threading. The two tuning parameters are:

1. `DOMINO_UPLOAD_CHUNK_BYTES` (Default: 3145728 (3 MB)): This configures the chunk size. Larger chunks mean less chunk setup overhead. For a fast network, the setup overhead is a larger percentage of the overall time needed to upload a chunk.

2. `DOMINO_UPLOAD_THREADS` (Default: 8): This configures the number of parallel threads for multi-threaded uploads and parallel chunk uploads. The number of threads is proportional to the number of cores in your operating system.

Set the Environment variable for Linux:

```shell theme={null}
export DOMINO_UPLOAD_CHUNK_BYTES=<new-value>
export DOMINO_UPLOAD_THREADS=<new-value>
```

Set the Environment variable for Windows (Powershell):

```shell theme={null}
set [DOMINO_UPLOAD_CHUNK_BYTES=[<string>]]
set [DOMINO_UPLOAD_THREADS=[<string>]]
```

### Create a snapshot

Use the following command to create a snapshot that is visible and populated after the backend finishes copying the files in the Dataset.

<Note>
  Do not modify the files in the Dataset until the snapshot is done.
</Note>

Syntax:

```shell theme={null}
domino create-snapshot <project-owner>/<project-name>/<dataset-name>
```

### Create a Dataset from a snapshot

Use the following command to create a Dataset from a snapshot where `<snapshot-integer>` maps to a valid snapshot number in the Dataset.

<Note>
  The new Dataset can take time to populate.
</Note>

Syntax:

```shell theme={null}
domino create-dataset-from-snapshot <project-owner>/<project-name>/<dataset-name> <snapshot-number> <new-dataset-name>
```

## Get help

`help`

`help` prints out a list of commands you can run, with information about each one, similar to this page.

Examples

`domino help` `domino help run`

## Get the version of your CLI

`version`

`version` outputs your CLI version.

Examples

`domino --version`


# Download files with the CLI
Source: https://docs.domino.ai/cloud/reference/cli/download-files-with-the-cli

Use domino download-results with a run ID or number and an optional --filter pattern to download selected Project output files.

If you must download individual files from a project, you typically use the Files page. However, if you have too many files to manage from the application, you can use a CLI command:

`domino download-results`.

This command accepts a filter that allows you to download a subset of the project. Run the following to see documentation of this command:

```shell theme={null}
domino help download-results
```

As an example, you can download the contents of a folder with a command like this:

```shell theme={null}
domino download-results --filter 'myfolder/*' 548fda01e4b0a8f06b7e8d99
```

* The `--filter` option is what limits the downloaded files to the pattern that follows it

* `548fda01e4b0a8f06b7e8d99` is the run ID

You can also use a run number instead of an ID:

```shell theme={null}
domino download-results --filter 'myfolder/*' 1
```

This downloads the output files from a run, so you must have submitted at least one run in the project. Files are listed in the local directory under `results/run-<runId or runNumber>`.


# Force-restore a local Project
Source: https://docs.domino.ai/cloud/reference/cli/force-restore-a-local-project

Repair local Domino Project metadata by removing the .domino directory, running domino restore, and reconciling local and server files.

If you encounter errors with the [Domino CLI](/cloud/reference/cli/cli-reference), it might be due to a problem with your local project metadata. This can happen if a file is modified by another program during the sync and [file locking](/cloud/platform-capabilities/core-concepts/projects/files-artifacts-and-results#special-files) is turned off. You might be able to resolve this issue by performing a force-restore. To do so, follow these steps:

1. Go to your local copy and remove the Domino project information.

   ```shell theme={null}
   cd <project folder>
   rm -rf .domino
   ```

2. To restore the Domino project information, run the following command and then enter the project name.

   ```shell theme={null}
   domino restore
   ```

This downloads files on the server that are not in your local project. Delete them if you do not need them. If any of your local files are different from the server version, the local version is pushed the next time you run `domino sync`.


# Domino CLI
Source: https://docs.domino.ai/cloud/reference/cli/index

Use the Domino CLI to manage Projects, files, and Datasets from a command line, including installation, downloads, restores, moves, and proxy setup.

The Domino Command Line Interface (CLI) is a tool that allows you to manage your Domino Environment from the command line. It is different from the Dom CLI tool kit used to [Streamline Local Development with SSH](/cloud/platform-capabilities/core-concepts/workspaces/launch-a-workspace/streamline-local-development-ssh).

These guides provide instructions to work with the Domino CLI on your system. Instructions are provided for both graphical and headless installations.

[Install the Domino Command Line Interface (CLI)](/cloud/reference/cli/install-the-cli)\
Install the Domino Command Line to your machine.

[Domino CLI Reference](/cloud/reference/cli/cli-reference)\
Commands to use with the CLI.

[Download Files with the CLI](/cloud/reference/cli/download-files-with-the-cli)\
Download all files, or a subset of files, from a project.

[Force Restore a Local Project](/cloud/reference/cli/force-restore-a-local-project)\
Fix errors with the CLI.

[Move a Project Between Deployments](/cloud/reference/cli/move-project-between-deployments)\
Move a project from one deployment to another, such as from a trial deployment to production.

[Use the Domino CLI Behind a Proxy](/cloud/reference/cli/use-the-cli-behind-a-proxy)\
Configure the Domino CLI to use your proxy.


# Install the Domino Command Line Interface (CLI)
Source: https://docs.domino.ai/cloud/reference/cli/install-the-cli

Install the Domino CLI in graphical or headless mode on Linux, Windows, or macOS, then sign in through a local browser or device code flow.

The Domino Command Line Interface (CLI) is a tool that allows you to manage your Domino Environment from the command line. It can be used to perform various tasks, such as creating and deleting Projects, backup and restoring Projects, and downloading files.

This guide provides instructions to install the Domino CLI on your system. Instructions are provided for both graphical and headless installations.

## Headless vs. graphical installation

Depending on your use case, you can either go for a Headless or Graphical installation:

* Headless installation is an installation of the Domino CLI on a server or machine that does not have a graphical user interface (GUI).

  This is commonly done for machines that are used to automate tasks or run Domino CLI tasks in a server environment or on a Domino Workspace.

* Graphical installation is an installation of the Domino CLI on a machine that has a GUI. This is the best option for users who want to use the Domino CLI with a graphical interface.

### Install the Domino CLI using graphical mode

#### Step 1: Download the CLI installer

In the top navigation, click **Account > Download CLI**. Click the applicable operating system to download the CLI installer.

#### Step 2: Run the CLI installer

Run the installer application depending on your operating system. See [Operating System Support](#operating-system-support) for more information.

#### Step 3: Log in to the Domino CLI

1. In a command prompt, type `domino login https://<your-domino-url>`.

   The CLI opens your default browser to the Domino sign-in page.

   <Note>
     After downloading and installing the Domino CLI, you have to close out of the terminal fully for it to recognize the install.
   </Note>

2. Sign in from the browser tab that opens. You might be required to log in.

   If a browser doesn't open automatically, or `Desktop` isn't supported on your machine, the terminal prints a URL.
   Open it in a browser on the *same machine* you're running the CLI from, since sign-in redirects back to a local address on that machine.

   On a machine with no usable local browser (for example, a remote server accessed over SSH), run `domino login --device https://<your-domino-url>` instead to use the [device code sign-in flow](#sign-in-from-a-machine-without-a-local-browser), which prints a verification URL and short user code you can open on any other device.

3. Once you've signed in, the browser tab shows a message that authentication succeeded and you can close the tab. The CLI receives the authorization automatically.

   <Note>
     CLI versions older than 6.2.3 use a different authentication client and can no longer log in. If `domino login` fails after upgrading Domino, download and reinstall the latest Domino CLI.
   </Note>

### Sign in from a machine without a local browser

When the CLI can't open or receive a redirect from a local browser, it uses the OAuth 2.0 Device Authorization Grant flow (RFC 8628) so you can complete sign-in from a browser on any other device, such as your laptop or phone.

Use this flow when:

* You're running the CLI over SSH on a remote server or Domino Workspace with no browser installed.
* Your machine is headless and the `Desktop` integration isn't available.
* Sign-in redirects to a local callback URL that a browser on a different machine can't reach.

To sign in with the device code flow:

1. Run `domino login --device https://<your-domino-url>` on the machine you want to authenticate.

   The CLI prints a verification URL and a short user code, for example:

   ```text theme={null}
   To sign in, use a web browser to open the page https://<your-domino-url>/auth/realms/DominoRealm/device
   and enter the code ABCD-EFGH.
   ```

   The CLI then polls Domino for the token in the background.

2. On any device with a browser, open the verification URL, enter the user code, and sign in to Domino.

3. On the consent screen, review the requested access and select **Grant Access**.

   Once you approve, the CLI receives the token automatically and prints `Login succeeded`. You can then run any other `domino` command on that machine.

The user code expires after a short period. If it expires before you sign in, run `domino login --device` again to get a new code.

### Install the Domino CLI on a headless Linux machine

A headless installation is an installation of the Domino CLI on a server or machine that does not have a GUI. This is commonly done for machines that are used to automate tasks or run Domino CLI tasks in a server environment or on a Domino Workspace.

#### Install CLI from the command line

**Step 1: Install headless Java**

```bash theme={null}
apt install openjdk-8-jre-headless
```

**Step 2: Download the CLI installer**

Replace `<domino-url>` in the link below with the Domino URL that is currently linked to the client.

```bash theme={null}
wget https://<domino-url>/download/client/unix -O /tmp/domino-install-unix.sh
```

**Step 3: Run the CLI installer**

```bash theme={null}
sh /tmp/domino-install-unix.sh
```

<Note>
  If you encounter an `AtkWrapper` error, resolve it by commenting out `/etc/java-8-openjdk/accessibility.properties assistive_technologies` and re-running the installer.
</Note>

```bash theme={null}
sed -i 's/^assistive_technologies=/#assistive_technologies=/g' /etc/java-8-openjdk/accessibility.properties
```

**Step 4: Log in to the Domino CLI**

Run `domino login \https://<your-domino-url>` as described in [Step 3: Log in to the Domino CLI](#step-3-log-in-to-the-domino-cli) above.

<Note>
  On a headless machine with no browser installed, run `domino login --device https://<your-domino-url>` to use the [device code sign-in flow](#sign-in-from-a-machine-without-a-local-browser) instead of a local callback. The CLI prints a verification URL and short user code, which you can open in a browser on any other device (for example, your laptop) to complete sign-in.
</Note>

## Use the CLI

Check out the [Domino CLI Reference](/cloud/reference/cli/cli-reference) for documentation about using the CLI.

## Operating system support

Domino supports recent versions of the following operating systems:

* Ubuntu Linux

* Windows 10

* macOS

### Linux

The Domino CLI can be installed on Linux machines using either the GUI or headless method.

If you chose to install the CLI using the GUI:

1. Navigate to the directory where you downloaded the CLI installer.

2. Run the installer:

   ```bash theme={null}
   sh domino-install-unix.sh
   ```

### Windows

Domino recommends that you use the version of the Domino CLI that comes bundled with a 64-bit JRE. The 32-bit version is deprecated.

### macOS

You must have root privileges to run the Domino CLI for macOS.

On Mac OS X, the installer prompts you to enter administrator credentials. If you do not have administrator privileges, you can click **Cancel**. When the installer prompts you to manually update your `PATH`, click **Ignore** to continue the installation process, then add `export PATH="$PATH:/Applications/domino"` (assuming the default installation folder) to your `bash_profile`.

Having trouble? [Email us](mailto:support@dominodatalab.com).


# Move a Project between deployments
Source: https://docs.domino.ai/cloud/reference/cli/move-project-between-deployments

Move a Project's latest files between Domino deployments with the CLI, then verify access and collaboration settings on the destination.

If you have a project on a deployment, such as a Trial deployment, and want to move it to a deployment, such as a Production deployment, you can use the CLI.

<Warning>
  This process only moves the latest files. It doesn’t move the history and revisions.
</Warning>

For more information about the CLI, see [Install the Domino Client (CLI)](/cloud/reference/cli/install-the-cli).

## Prerequisites

You must have the latest local copy of your project on your workstation. If you don’t have a local copy of the project, review the [Domino CLI Reference](/cloud/reference/cli/cli-reference#get-a-project-from-the-server-onto-your-computer-for-the-first-time) to get it.

## Move the Project between deployments

1. Go to the project on your local system that you’d like to migrate:

   ```shell theme={null}
   cd <your-project>
   ```

2. `.domino` is what identifies it as a domino project.

   To remove this (UNIX):

   ```shell theme={null}
   rm -rf .domino
   ```

   To remove this (Windows):

   ```shell theme={null}
   rmdir /s/q .domino and del .domino*
   ```

3. Log into the target deployment, initialize, and upload your project:

   ```shell theme={null}
   domino login <target URL>
   ```

   ```shell theme={null}
   domino init
   ```

   ```shell theme={null}
   domino upload
   ```

4. Log into the new Domino system where your project is hosted and verify the access and collaboration settings are correct. You might have to update the access and collaboration settings because they do not transfer with the project.


# Use the Domino CLI behind a proxy
Source: https://docs.domino.ai/cloud/reference/cli/use-the-cli-behind-a-proxy

Configure the Domino CLI for HTTP and HTTPS proxies with user-level domino.vmoptions settings, including Basic or NTLM authentication.

To configure the Domino CLI to use your proxy, you must modify files in the client installation directory:

* On Mac, the directory is: `/Applications/domino`

* On Windows, the directory is: `C:\Users\<your_username>\AppData\Local\Programs\domino`

In the root of the installation directory, there is a file named `domino.vmoptions` with some configuration options available. Add the line `-include-options ~/.domino/domino.vmoptions` at the *end* of the file for any \*nix based system (Windows users MUST specify the appropriate path where `domino.vmoptions` is created):

```shell theme={null}
-Ddomino.defaultHost=
-Dhttp.proxyHost=
-Dhttp.proxyPort=
-Dhttp.nonProxyHosts=
-Dhttps.proxyHost=
-Dhttps.proxyPort=
-Dhttps.nonProxyHosts=
-include-options ~/.domino/domino.vmoptions
```

You must create the user level configuration file `domino.vmoptions` in the appropriate location/folder. The folder `~/.domino/` may not exist, in which case you should run `mkdir ~/.domino` to create it. You must have a user level configuration file to ensure your settings are never modified if the CLI is reinstalled or updated. In the new `~/.domino/domino.vmoptions`, assign your proxy’s port and host configuration:

```shell theme={null}
-Dhttp.proxyHost=YOURPROXYHOST
-Dhttp.proxyPort=1234
-Dhttps.proxyHost=YOURPROXYHOST
-Dhttps.proxyPort=1234
```

Replace `YOURPROXYHOST` and `1234` with your actual proxy connection information.

<Warning>
  Do not include `http://` or `https://` in the proxyHost parameters.
</Warning>

## Proxy authentication with the Domino CLI

CLI supports Basic and NTLM proxy authentication. Add the following parameters to your `~/.domino/domino.vmoptions` file for proxy authentication:

**Basic**

```shell theme={null}
# Make sure you have disabled the system proxy via this setting
-Djava.net.useSystemProxies=false
# This is required to enable basic authentication for Java versions >= Java 8u111
# More info here -> http://www.oracle.com/technetwork/java/javase/8u111-relnotes-3124969.html
-Djdk.http.auth.tunneling.disabledSchemes=
# (https) Do not include quotes
-Dhttps.proxyUser=YOURPROXYUSER
# (https) Do not include quotes, even if your password contains spaces
-Dhttps.proxyPassword=YOURPROXYPASSWORD
# (http) Do not include quotes
-Dhttp.proxyUser=YOURPROXYUSER
# (http) Do not include quotes, even if your password contains spaces
-Dhttp.proxyPassword=YOURPROXYPASSWORD
# Make sure to leave a blank line at the end
```

**NTLM**

```shell theme={null}
# This is an extension of Basic Proxy settings
# (http) Make sure to specify the NTLM domain for this Proxy
-Dhttp.auth.ntlm.domain=YOURNTLMDOMAIN
# (https) Make sure to specify the NTLM domain for this Proxy.
-Dhttps.auth.ntlm.domain=YOURNTLMDOMAIN
# Make sure to leave a blank line at the end
```

<Note>
  The proxy username is case-sensitive when using NTLM proxy authentication.
</Note>


# API reference for the Agents Python SDK
Source: https://docs.domino.ai/cloud/reference/experiment-agent-sdk/python-sdk-agents

Use the Agents Python SDK reference to trace and evaluate agentic systems, search development or production traces, and log evaluation results.

The `domino.agents` package provides tracing, evaluation, and search capabilities for agentic systems. This is part of the python-domino library.

<Note>
  The canonical import path is `from domino.agents.tracing import …​`\
  Some documentation may reference `domino.aisystems.tracing`. This is an older alias. Both paths resolve to the same module.
</Note>

## add\_tracing

Decorator that starts an MLflow span for the decorated function. If an existing trace is in progress, appends a span to it; otherwise creates a new trace.

```python theme={null}
domino.agents.tracing.add_tracing(
    name: str,
    autolog_frameworks: list[str] | None = [],
    evaluator: Callable | None = None,
    trace_evaluator: Callable | None = None,
    eagerly_evaluate_streamed_results: bool = True,
    allow_tracing_evaluator: bool = False
)
```

| Parameter | Type | Description | | | |
| - | - | - | - | - | - |
| `name` | `str` | Name of the span (or new trace). | | | |
| `autolog_frameworks` | \`list\[str] | None\` | MLflow-supported frameworks to autolog (for instance, `["langchain"]`, `["openai"]`). | | |
| `evaluator` | \`Callable | None\` | Function that receives the span and returns evaluation results as \`dict\[str, int | float | str]\`. Runs on every invocation. |
| `trace_evaluator` | \`Callable | None\` | Function that receives the complete trace. Only fires when the trace was started and finished by this decorator. Useful for end-to-end quality checks. | | |
| `eagerly_evaluate_streamed_results` | `bool` | When `True` (default), aggregates all yielded values into a single span for evaluation. When `False`, each yielded value creates a separate span with a shared `group_id`. | | | |
| `allow_tracing_evaluator` | `bool` | Default `False`. When `True`, inline evaluators are also traced by MLflow autolog. | | | |

## init\_tracing

Initialize MLflow autologging and set the active experiment to enable tracing. Used to initialize logging in both development and production modes.

```python theme={null}
domino.agents.tracing.init_tracing(
    autolog_frameworks: list[str] | None = None
)
```

* In production mode, the environment variables `DOMINO_AGENT_IS_PROD` and `DOMINO_APP_ID` must be set.

* Call `init_tracing()` before your app starts serving requests.

## search\_traces

Search for traces in a development-mode evaluation run. Returns a paginated response of trace summaries.

```python theme={null}
domino.agents.tracing.search_traces(
    run_id: str,
    trace_name: str | None = None,
    start_time: datetime | None = None,
    end_time: datetime | None = None,
    page_token: str | None = None,
    max_results: int | None = None
) → SearchTracesResponse
```

## search\_agent\_traces

Search for traces from a deployed production agent. Filter by agent version, trace name, and time range. If `agent_version` is not provided, searches across all versions.

```python theme={null}
domino.agents.tracing.search_agent_traces(
    agent_id: str,
    agent_version: str | None = None,
    trace_name: str | None = None,
    start_time: datetime | None = None,
    end_time: datetime | None = None,
    page_token: str | None = None,
    max_results: int | None = None
) → SearchTracesResponse
```

## Data classes

### EvaluationResult

```python theme={null}
class domino.agents.tracing.EvaluationResult(
    name: str,
    value: float | str
)
```

| Parameter | Type | Description | |
| - | - | - | - |
| `name` | `str` | Name of the evaluation. | |
| `value` | \`float | str\` | Evaluation result value. |

### TraceSummary

```python theme={null}
class domino.agents.tracing.TraceSummary(
    name: str,
    id: str,
    spans: list[SpanSummary],
    evaluation_results: list[EvaluationResult]
)
```

| Parameter | Type | Description |
| - | - | - |
| `name` | `str` | Name of the trace. |
| `id` | `str` | MLflow trace ID. |
| `spans` | `list[SpanSummary]` | Child spans of this trace. |
| `evaluation_results` | `list[EvaluationResult]` | Evaluation results for this trace. |

### SpanSummary

```python theme={null}
class domino.agents.tracing.SpanSummary(
    id: str,
    name: str,
    trace_id: str,
    inputs: Any,
    outputs: Any
)
```

| Parameter | Type | Description |
| - | - | - |
| `id` | `str` | MLflow span ID. |
| `name` | `str` | Span name. |
| `trace_id` | `str` | Parent trace ID. |
| `inputs` | `Any` | Inputs to the function that created the span. |
| `outputs` | `Any` | Outputs of the function. |

### SearchTracesResponse

```python theme={null}
class domino.agents.tracing.SearchTracesResponse(
    data: list[TraceSummary],
    page_token: str | None
)
```

| Parameter | Type | Description | |
| - | - | - | - |
| `data` | `list[TraceSummary]` | List of trace summaries. | |
| `page_token` | \`str | None\` | Token for the next page of results. |

## log\_evaluation

Log an evaluation result against an existing trace. Used for ad-hoc or production evaluations where you evaluate traces after they’ve been collected such as in a scheduled Job.

```python theme={null}
domino.agents.logging.log_evaluation(
    trace_id: str,
    name: str,
    value: float | str
)
```

| Parameter | Type | Description | |
| - | - | - | - |
| `trace_id` | `str` | The MLflow trace ID to attach the evaluation to. | |
| `name` | `str` | Name of the evaluation metric. | |
| `value` | \`float | str\` | Evaluation result value (numeric score or string label). |

## DominoAgentContext

The `DominoAgentContext` wrapper from `domino.agents.logging` creates an MLflow agent version that stores traces and logged parameters.

```python theme={null}
from domino.agents.logging import DominoAgentContext

with DominoAgentContext(
    agent_config_path="config.yaml",
    aggregated_metrics=[("toxicity", "mean"), ("bleu", "median")]
) as run:
    run_agent(...)
```

| Parameter | Description |
| - | - |
| `agent_config_path` | Path to a YAML config file. Logged as parameters in the Experiment Manager. |
| `aggregated_metrics` | List of `(metric_name, aggregation)` tuples. Aggregation types: `mean`, `median`, `stdev`, `min`, `max`. Defaults to `mean` for all metrics. |

## Environment variables

| Variable | Required for | Description |
| - | - | - |
| `MLFLOW_TRACKING_URI` | All modes | Set automatically in Domino executions. For local testing, use `http://localhost:5000`. |
| `DOMINO_AGENT_IS_PROD` | Production | Set to indicate production mode for `init_tracing()`. |
| `DOMINO_APP_ID` | Production | The ID of the deployed agent app. |


# Glossary
Source: https://docs.domino.ai/cloud/reference/glossary

Look up Domino terms such as Project, Workspace, Job, Data Plane, bundle, and policy, with links to the pages that explain each one.

<Note>
  For terms specific to one area, see the [Governance glossary](/cloud/platform-capabilities/features/governance/governance-glossary), the [Audit Trail Data Glossary](/cloud/platform-capabilities/features/governance/audit-trail/audit-trail-glossary), and [Default Domino Environment Variables](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/manage-environment-variables/5-glossary-default-variables).
</Note>

## AI Gateway (legacy)

A proxy between Domino and external LLM providers, such as OpenAI or AWS Bedrock. It routes requests through central credentials and logs every request. In Domino 6.3, AI Gateway is off by default and works only in deployments that enable it. Domino Cloud doesn't include it. [LLM Gateway 2.0](#llm-gateway-2-0) replaces it. See [AI Gateway (legacy)](/6.3/platform-capabilities/features/llms/ai-gateway).

## API endpoint

Another name for a [Domino endpoint](#domino-endpoint). See [Model deployment](/cloud/platform-capabilities/features/model-deployment).

## App

An interactive web application that runs code from your Project. Each App runs in its own container, and Domino handles its routing, authentication, and resources. See [Apps](/cloud/platform-capabilities/features/apps).

## App & Agent Hub

Where people in your organization find and use published Apps and agents. See the [Domino overview](/cloud/platform-overview/domino-overview).

## App instance

One run of an [App version](#app-version), from start to stop. An App has at most one running instance at a time. Domino records logs and resource use for each instance. See [App versioning and reproducibility](/cloud/platform-capabilities/features/apps/app-versioning-and-reproducibility).

## App Preview

A private run of a [Draft App](#draft-app), started from a Workspace, that only the App author can open. It doesn't change the published App, but it isn't sandboxed: it runs as the author, with real access to Domino APIs and shared data. See [Develop and test an App](/cloud/platform-capabilities/features/apps/develop-and-test-an-app).

## App version

An immutable snapshot of everything that defines an App: pinned source commits, Compute Environment and revision, Data Plane, mounted storage, and identity propagation. Publishing a Draft or changing any of these creates a version. Starting the App doesn't. See [App versioning and reproducibility](/cloud/platform-capabilities/features/apps/app-versioning-and-reproducibility).

## Approval

A sign-off on evidence that a governance stage requires before it's complete. Approvals can also control moves between stages. See [Governance](/cloud/platform-capabilities/features/governance).

## Audit Trail

Domino's record of user activity across the deployment, for compliance, security investigations, and audit reporting. You can view it in the UI or query it through APIs. See [Audit Trail](/cloud/platform-capabilities/features/governance/audit-trail).

## Bundle

A group of work items, such as models, files, and Apps, that you govern together. Policies apply to bundles, so anything you govern must be in one. Each bundle belongs to a Project. See [Work with bundles](/cloud/platform-capabilities/features/governance/work-with-bundles).

## Checkpoint

A commit in a Workspace's history that you can return to. Domino creates one each time you sync code or artifact changes. Recreating a Workspace from a checkpoint starts a new branch. See [Create a Workspace from a checkpoint](/cloud/platform-capabilities/core-concepts/workspaces/create-workspace-from-checkpoint).

## Compute cluster

A Spark, Ray, Dask, or MPI cluster that Domino provisions on demand and attaches to a Workspace or Job. It runs on a Domino Cluster Environment. See [On-demand distributed computing](/cloud/platform-capabilities/features/compute-clusters-hpc).

## Compute Environment

A versioned, shareable Docker image that defines the software every Workspace, Job, App, and model runs with. Domino starts a separate container from it for each execution. See [Compute Environments](/cloud/platform-capabilities/core-concepts/compute-environments).

## Control Plane

The Domino services and user interface that schedule and manage work across every [Data Plane](#data-plane). See [Data Plane](/cloud/platform-capabilities/core-concepts/data-plane).

## Data Plane

A Kubernetes namespace where Domino runs your workloads. The local Data Plane is in the same cluster as the Domino Control Plane; remote Data Planes are in other clusters, regions, or clouds. See [Data Plane](/cloud/platform-capabilities/core-concepts/data-plane).

## Data Source

A saved connection to an external data system, such as a database, data warehouse, or data lake, that you can reuse in any Project. See [Domino Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors).

## Dataset

Versioned, high-performance file storage in Domino for curating data and sharing it across Projects. See [Datasets](/cloud/platform-capabilities/core-concepts/data/datasets).

## Domino endpoint

A model deployed from a Project as an HTTP API. Clients send inputs and get predictions back, either in the response or, for long-running work, asynchronously by polling. See [Model deployment](/cloud/platform-capabilities/features/model-deployment).

## Domino File System (DFS)

Domino-managed file storage for Projects that don't use a Git provider. Domino versions DFS files automatically and records a new Project version on every sync. See [DFS Projects](/cloud/platform-capabilities/core-concepts/projects/dfs-projects).

## Domino Nexus

The Domino capability to run workloads on multiple clusters, in any cloud, region, or on-premises location, by attaching remote Data Planes to one Domino Control Plane. See [Data Plane](/cloud/platform-capabilities/core-concepts/data-plane).

## Domino skill

A packaged set of instructions that teaches a coding assistant a Domino task, such as submitting a Job, registering a model, or deploying an App. The Domino Standard Environment includes the skills from the public `domino-claude-plugin` repository, and coding assistants load them when they start in a Workspace. Skills are optimized for Claude Code. Anyone can contribute one through a pull request. See [Domino skills reference](/cloud/reference/skills).

## Domino Standard Environment (DSE)

A [Compute Environment](#compute-environment) included in every Domino installation, with preconfigured data science tools and libraries. You can use it for any Workspace, Job, App, or Domino endpoint. Domino updates it with each major release. See [Domino Standard Environments](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/2-domino-standard-environments).

## Draft App

A private, unpublished App linked to one Workspace, for development and testing. Domino creates it the first time you use [App Preview](#app-preview) in that Workspace and deletes it with the Workspace. Draft Apps don't appear in App lists or search. See [Develop and test an App](/cloud/platform-capabilities/features/apps/develop-and-test-an-app).

## Evidence

The information a governance policy requires, usually answers to questions, collected as inputs, metric checks, or scripted checks. Each policy defines the evidence for each stage. See [Governance](/cloud/platform-capabilities/features/governance).

## Experiment Manager

Domino's experiment tracking, built on MLflow Tracking. It logs parameters, metrics, and artifacts, and stores them in the Project for reproducibility. See [Track and monitor experiments](/cloud/platform-capabilities/features/development/track-monitor).

## Extension

An entry point in the Domino UI that opens a Domino App. Extensions can appear in the Project sidebar, on Dataset or Model details pages, in the three-dot menu for Dataset files, or as a page in the Admin panel. One App can power several Extensions. See [Create and manage Extensions](/cloud/platform-capabilities/features/extensions/create-manage-extensions).

## External data volume (EDV)

A network-attached storage volume that Domino mounts into every execution in the Projects you attach it to. See [External data volumes](/cloud/platform-capabilities/core-concepts/data/external-volumes).

## Finding

An issue raised while reviewing governance evidence. A finding belongs to one bundle and usually points to a specific piece of required evidence. See [Governance](/cloud/platform-capabilities/features/governance).

## FinOps

Cost management for AI work, part of the [Governance Center](#governance-center). See [FinOps](/cloud/platform-capabilities/features/finops).

## Flow

A multi-step pipeline with full lineage and reproducibility. Each step, or task, runs as a Domino Job, and the tasks form a directed acyclic graph (DAG). See [Flows](/cloud/platform-capabilities/features/flows).

## Gate

A governance control that blocks an action, such as deploying a published App, an agent, or a model endpoint, until required approvals are complete. See [Governance](/cloud/platform-capabilities/features/governance).

## Git-based Project

A Project whose code lives in a repository at your Git provider. You branch, commit, push, and pull from your Workspace, and keep your provider's review and collaboration features. Use a Git-based Project whenever you have a Git provider. See [Git-based Projects](/cloud/platform-capabilities/core-concepts/projects/git-based-projects).

## Governance Center

Domino's governance capabilities: policies that enforce lifecycle standards, [FinOps](#finops) for costs, and the [Audit Trail](#audit-trail) of platform activity. See the [Domino overview](/cloud/platform-overview/domino-overview) and [Governance](/cloud/platform-capabilities/features/governance).

## Hardware Tier

A named amount of CPU, memory, and GPU, on a specific node type, that you choose when you start an execution. See [Hardware Tiers](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers).

## HPC Job

A batch job that Slurm schedules on autoscaled worker nodes from inside Domino. Submit one from a Workspace with the Slurm CLI, or from the HPC tab on the Project's Jobs page if an administrator enabled UI submission. See [Run HPC workloads with Slurm](/cloud/platform-capabilities/features/compute-clusters-hpc/slurm).

## Job

A non-interactive, reproducible run of your code for batch and headless work. For interactive work, use a Workspace. See [Jobs](/cloud/platform-capabilities/core-concepts/jobs).

## Launcher

A web form that lets less technical users run a predefined task. Each form control, such as a text box or menu, supplies a parameter to the run. See [Launchers](/cloud/platform-capabilities/features/launchers).

## LLM Gateway 2.0

A gateway that centralizes access to external LLM providers, with usage tracking across users, Projects, and models, per-team and per-model budgets, guardrails that inspect prompts and responses, and broad model and vendor coverage. It replaces [AI Gateway](#ai-gateway-legacy). Domino distributes LLM Gateway 2.0 as a binary that you deploy as a Domino App; contact your Domino field representative for access. See [LLM Gateway 2.0](/cloud/platform-capabilities/features/llms/llm-gateway).

## Model Monitoring

Tracking of deployed models for drift in input features and predictions, with alerts when a metric crosses a threshold you set. See [Model Monitoring](/cloud/platform-capabilities/features/monitoring).

## Model Registry

Domino's MLflow-based catalog of models, at Project and deployment scope. It records versions, metadata, lineage, and deployments, and controls access with Role-Based Access Control (RBAC). See [Manage models with model registry](/cloud/platform-capabilities/features/models/register-and-govern/model-registry).

## Namespace

A category in Domino's tag structure, such as `business-domain` or `data-sensitivity`. Each Namespace holds the [Tags](#tag) allowed as its values. See [Tags](/cloud/platform-capabilities/core-concepts/knowledge-management/tags).

## NetApp Volume

NetApp ONTAP storage that Domino mounts directly into workloads through its native NetApp integration. See [NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes).

## Organization

A group of users that you can grant access to as one unit. Domino treats an organization like a user that contains other users. The user who creates an organization owns it and manages its members. See [Manage organizations](/cloud/platform-capabilities/features/collaboration/manage-organizations).

## Policy

The governance lifecycle a bundle must complete: its stages, evidence, and approvals. A bundle can follow several policies, and a policy can apply to many bundles. See [Define policies](/cloud/platform-capabilities/features/governance/define-governance-policies).

## Policy version

A published revision of a policy. When you publish a new version, you choose whether bundles on an earlier version must upgrade and whether existing approvals carry over. See [Define policies](/cloud/platform-capabilities/features/governance/define-governance-policies).

## Project

Domino's unit of work: the code, data, results, and collaborators for one effort. Apps, models, and Domino endpoints publish from a Project, and governance bundles track its work. See [Projects](/cloud/platform-capabilities/core-concepts/projects).

## Project template

A reusable starting point made from selected assets of an existing Git-based Project. See [Create Projects](/cloud/platform-capabilities/core-concepts/projects/create-projects#create-a-project-from-a-template).

## Property

A typed field that adds custom information to an entity. Unlike Tags, Properties are flat and accept any value of their type, except select types, which offer fixed choices. See [Properties](/cloud/platform-capabilities/core-concepts/knowledge-management/properties).

## Scheduled Job

A Job that runs on a recurring schedule. See [Schedule a Job](/cloud/platform-capabilities/core-concepts/jobs/schedule-a-job).

## Service account

A non-human account that administrators create for API automation, such as CI/CD pipelines. It can't sign in to the web UI or CLI, and it's the only account type that can use long-lived API tokens. See [Service accounts](/cloud/admin/identity-and-access/manage-users/service-accounts).

## Snapshot

An immutable copy of a Dataset at a point in time. You can tag, download, or create versions from snapshots. See [Version data with snapshots](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots).

## Stage

One phase of a policy's lifecycle, such as Use Case Definition, Data Selection, Deployment, or Ongoing Monitoring. Each stage holds the evidence the policy requires at that point. See [Governance](/cloud/platform-capabilities/features/governance).

## Tag

A value from a controlled, hierarchical list, used to categorize and find assets. Each Tag belongs to a [Namespace](#namespace). See [Tags](/cloud/platform-capabilities/core-concepts/knowledge-management/tags).

## Workspace

An interactive development session in an IDE such as Jupyter, RStudio, or VS Code, running on Domino compute. See [Workspaces](/cloud/platform-capabilities/core-concepts/workspaces).


# The python-domino library
Source: https://docs.domino.ai/cloud/reference/python-sdk/python-domino

Use the python-domino reference to configure authentication and manage Projects, Jobs, Apps, Datasets, files, traces, evaluations, and budgets.

This library provides bindings for the Domino APIs.  It ships with the Domino Standard Environment (DSE).

See this documentation for details about the APIs:

* [Latest public Domino
  APIs](https://docs.dominodatalab.com/en/latest/api_guide/8c929e/domino-public-apis/)

* [Legacy APIs](https://dominodatalab.github.io/api-docs/)

The latest released version of `python-domino` is `2.2.0`.

# Version compatibility matrix

The `python-domino` library is compatible with different versions of
Domino:

| Domino Versions | Python-Domino |
| - | :-: |
| 3.6.x or lower | [0.3.5](https://github.com/dominodatalab/python-domino/archive/0.3.5.zip) |
| 4.1.0 or higher | [1.0.0](https://github.com/dominodatalab/python-domino/archive/refs/tags/1.0.0.zip) or Higher |
| 5.3.0 or higher | [1.2.0](https://github.com/dominodatalab/python-domino/archive/refs/tags/Release-1.2.0.zip) or Higher |
| 5.5.0 or higher | [1.2.2](https://github.com/dominodatalab/python-domino/archive/refs/tags/Release-1.2.2.zip) or Higher |
| 5.10.0 or higher | [1.3.1](https://github.com/dominodatalab/python-domino/archive/refs/tags/Release-1.3.1.zip) or Higher |
| 5.11.0 or higher | [1.4.1](https://github.com/dominodatalab/python-domino/archive/refs/tags/Release-1.4.1.zip) or Higher |
| 6.0.0 or higher | [1.4.8](https://github.com/dominodatalab/python-domino/archive/refs/tags/Release-1.4.8.zip) or Higher |
| 6.2.0 or higher | [2.0.0](https://github.com/dominodatalab/python-domino/archive/refs/tags/Release-2.0.0.zip) or Higher |
| 6.3.0 or higher | [2.1.0](https://github.com/dominodatalab/python-domino/archive/refs/tags/Release-2.1.0.zip) or Higher |

# Development

The current `python-domino` is based on Python 3.10, which is therefore recommended for development. `Pipenv` is also recommended to manage the dependencies.

To use the Python binding in a Domino workbook session, include `dominodatalab` in your project's requirements.txt file.
This makes the Python binding available for each new workbook session (or batch run) started within the project.

To install dependencies from `setup.py` for development:

```
pipenv --python 3.10 install -e ".[dev]"
```

Use the same process for Airflow, agents, and data:

```
pipenv --python 3.10 install -e ".[data]" ".[airflow]" ".[agents]"
```

# Set up the connection

You can set up the connection by creating a new instance of `Domino`:

```
_class_ Domino(project, api_key=None, host=None, domino_token_file=None, auth_token=None)
```

* *project:* A project identifier (in the form of ownerusername/projectname).

* *api\_proxy:* (Optional) Location of the Domino API reverse proxy as host:port.

  If set, this proxy is used to intercept any Domino API requests and insert an authentication token.
  *This is the preferred method of authentication*.
  Alternatively, set the `DOMINO_API_PROXY` environment variable.
  In Domino 5.4.0 or later, this variable is set inside a Domino run container.

  NOTE: This mechanism does not work when connecting to an HTTPS endpoint; it is meant to be used inside Domino runs.

* *api\_key:* (Optional) An API key to authenticate with.

  If not provided, the library expects to find one in the `DOMINO_USER_API_KEY` environment variable.
  If you are using the Python package in code that is already running in Domino, the `DOMINO_API_USER_KEY` variable is set automatically to be the key for the user who started the run.

* *host:* (Optional) A host URL.

  If not provided, the library expects to find one in the `DOMINO_API_HOST` environment variable.

* *domino\_token\_file:* (Optional) Path to the Domino token file
  containing the auth token.

  If not provided, the library expects to find one in the `DOMINO_TOKEN_FILE` environment variable.
  If you are using Python package in code that is already running in Domino, the `DOMINO_TOKEN_FILE` is set automatically to be the token file for the user who started the run.

* *auth\_token:* (Optional) Authentication token.

## Authentication

Domino looks for the authentication method in the following order and uses the first one it finds:

1. `api_proxy`
2. `auth_token`
3. `domino_token_file`
4. `api_key`
5. `DOMINO_API_PROXY`
6. `DOMINO_TOKEN_FILE`
7. `DOMINO_USER_API_KEY`

The API proxy is the preferred method of authentication.
See
[Use the API Proxy to Authenticate Calls to the Domino API](https://docs.dominodatalab.com/en/latest/user_guide/40b91f/domino-api/).

## Additional environment variables

* `DOMINO_LOG_LEVEL`

  The default log level is `INFO`.
  You can change the log level by setting `DOMINO_LOG_LEVEL`, for example to `DEBUG`.

* `DOMINO_VERIFY_CERTIFICATE`

  For testing purposes and issues with SSL certificates, set `DOMINO_VERIFY_CERTIFICATE` to `false`.
  Be sure to unset this variable when not in use.

* `DOMINO_MAX_RETRIES`

  Default Retry is set to 4
  Determines the number of attempts for the request session in case of a ConnectionError
  Get more info on request max timeout/error durations based on Retry and backoff factors [here](https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#module-urllib3.util.retry)

* `MLFLOW_TRACKING_URI`

  Must be set for the `domino.agents` package to work properly. This will be set automatically in executions, e.g. Jobs, Workspaces, Scheduled Jobs, etc.
  For testing, the value must be set on the command line to "[http://localhost:5000](http://localhost:5000)".

# Working with docs

## building locally

* Install the app in dev mode and all dependencies and pandoc: `brew install pandoc ghc cabal-install haskell-stack` and `cabal update && cabal install --lib pandoc-types --package-env .`
* install dependencies in this order: `pipenv --python 3.10 install -e ".[dev]" && pipenv --python 3.10 install -e ".[docs]" && pipenv --python 3.10 install -e ".[data]" ".[airflow]" ".[agents]"`
* Build: `./docs/build.sh`
* View: open docs/build/html/index.html
* Manually pick the changes you want and add to `README.adoc` and `README.md`. Update the styling of the docs as needed.

## transform agents docs for README

* Run:
  `./docs/transform_agents_docs_for_readme.sh`
* copy/paste the outputs for asciidoc and markdown into respective README files

# Methods

# Budgets and Billing Tags

See
[`example_budget_manager.py`](https://github.com/dominodatalab/python-domino/blob/Release-2.2.0/examples/example_budget_manager.py)
for example code.

### budget\_defaults\_list()

Get a list of the available default budgets with the assigned (if any) limits
Requires Admin permission

### budget\_defaults\_update(budget\_label, budget\_limit)

Update default budgets by BudgetLabel
Requires Admin roles

* *budget\_label:* (required) label of budget to be updated ex: `BillingTag`, `Organization`

* *budget\_limit:* (required) new budget quota to assign to default label

### budget\_overrides\_list()

Get a list of the available budgets overrides with the assigned limits.
Requires Admin permission

### budget\_override\_create(budget\_label, budget\_id, budget\_limit)

Create Budget overrides based on BudgetLabels, ie BillingTags, Organization, or Projects
the object id is used as budget ids
Requires Admin roles

* *budget\_label:* label of budget to be updated

* *budget\_id:* id of project or organization to be used as new budget override id.

* *budget\_limit:* budget quota to assign to override

### budget\_override\_update(budget\_label, budget\_id, budget\_limit)

Update Budget overrides based on BudgetLabel and budget id
Requires Admin roles

* *budget\_label:* label of budget to be updated

* *budget\_id:* id of budget override to be updated.

* *budget\_limit:* new budget quota to assign to override

### budget\_override\_delete(budget\_id)

Delete an existing budget override
Requires Admin roles

* *budget\_id:* id of budget override to be deleted.

### budget\_alerts\_settings()

Get the current budget alerts settings
Requires Admin permission

### budget\_alerts\_settings\_update(alerts\_enabled, notify\_org\_owner)

Update the current budget alerts settings to enable/disable budget notifications
and whether to notify org owners on projects notifications
Requires Admin permission

* *alerts\_enabled:* whether to enable or disable notifications.

* *notify\_org\_owner:* whether to notify organizations owners on projects reaching threshold.

### budget\_alerts\_targets\_update(targets)

Update the current budget alerts settings with additional email targets per budget label
Requires Admin permission

* *targets:* dictionary of budget labels and list of email addresses

### billing\_tags\_list\_active()

Get a list of active billing tags
Requires Admin permission

### billing\_tags\_create(tags\_list)

Create a list of active billing tags
Requires Admin permission

* *tags\_list:* list of billing tags names to be created

### active\_billing\_tag\_by\_name(name)

Get detailed info on active or archived billing tag
Requires Admin permission

* *name:* name of existing billing tag

### billing\_tag\_archive(name)

Archive an active billing tag
Requires Admin permission

* *name:* name of existing billing tag to archive

### billing\_tag\_settings()

Get the current billing tag settings
Requires Admin permission

### billing\_tag\_settings\_mode()

Get the current billing tag settings mode
Requires Admin permission

### billing\_tag\_settings\_mode\_update(mode)

Update the current billing tag settings mode
Requires Admin permission

* *mode:* new mode to set the billing tag settings (see BillingTagSettingMode)

### project\_billing\_tag(project\_id)

Get a billing tag assigned to a particular project by project id
Requires Admin permission

* *project\_id:* id of the project to find assigned billing tag

### project\_billing\_tag\_update(billing\_tag, project\_id)

Update project's billing tag with new billing tag.
Requires Admin permission

* *billing\_tag:* billing tag to assign to a project

* *project\_id:* id of the project to assign a billing tag

### project\_billing\_tag\_reset(project\_id)

Remove a billing tag from a specified project
Requires Admin permission

* *project\_id:* id of the project to reset billing tag field

### projects\_by\_billing\_tag( billing\_tag, offset, page\_size, name\_filter, sort\_by, sort\_order, missing\_tag\_only=False)

Remove a billing tag from a specified project
Requires Admin permission

* *billing\_tag:* billing tag string to filter projects by

* *offset:* The index of the start of the page, where checkpointProjectId is index 0.
  If the offset is negative the project it points to will be the end of the page.

* *page\_size:* The number of records to return per page.

* *name\_filter:* matches projects by name substring

* *sort\_by:* (Optional) field to sort the projects on

* *sort\_order:* (Optional) Whether to sort in asc or desc order

* *missing\_tag\_only:* (Optional) determine whether to only return projects with missing tag

### project\_billing\_tag\_bulk\_update(projects\_tag)

Update project's billing tags in bulk
Requires Admin permission

* *projects\_tag:* dictionary of project\_id and billing\_tags

## Projects

See
[`example_projects_usage.py`](https://github.com/dominodatalab/python-domino/blob/Release-2.2.0/examples/example_projects_usage.py)
for example code.

### project\_create\_v4(project\_name, owner\_id, owner\_username, description, collaborators, tags, billing\_tag, visibility=PUBLIC)

Newer version of projects creation using the v4 endpoints which allows more optional fields.

* *project\_name:* (required) The name of the project.

* *owner\_id:* (Optional) user id of the owner of the new project to be created (must be admin to create projects for other users)
  owner\_id or owner\_username can be used, both are not needed (Defaults to current owner\_username)

* *owner\_username:* (Optional) username of the owner of the new project to be created (must be admin to create projects for other users)
  owner\_id or owner\_username can be used, both are not needed (Defaults to current owner\_username)

* *description:* (Optional) description of the project

* *collaborators:* (Optional) list of collaborators to be added to the project

* *tags:* (Optional) list of tags to add to project

* *billing\_tag:* (Optional unless billingTag settings mode is Required) active billing tag to be added to projects for governance

* *visibility:* (Optional) (Defaults to Public) project visibility

### project\_create(project\_name, owner\_username=None)

Create a new project with given project name.

* *project\_name:* The name of the project.

* *owner\_username:* (Optional) The owner username for the project.
  This parameter is useful when you need to create a project under an
  organization.

### collaborators\_get()

Get the list of collaborators on a project.

### collaborators\_add(username\_or\_email, message="")

Add collaborators to a project.

* *username\_or\_email:* Name or email of the Domino user to add as
  collaborator to the current project.

* *message:* Optional - Message related to the user’s role or purpose
  to the project.

## Project tags

Project tags are an easy way to add freeform metadata to a project. Tags
help colleagues and consumers organize and find the Domino projects that
interest them. Tags can be used to describe the subject explored by a
project, the packages and libraries it uses, or the source of the data
within.

See
[`example_projects_usage.py`](https://github.com/dominodatalab/python-domino/blob/Release-2.2.0/examples/example_projects_usage.py)
for example code.

### tags\_list(\*project\_id)

List a project’s tags.

* *project\_id:* The project identifier.

### tag\_details(tag\_id)

Get details about a tag.

* *tag\_id:* The tag identifier.

### tags\_add(tags, \*project\_id)

Create a tag, if it does not exist, and add it to a project.

* *tags (list):* One or more tag names.

* *project\_id:* (Defaults to current project ID) The project
  identifier.

### tag\_get\_id(tag\_name, \*project\_id)

Get the tag ID using the tag string name.

* *tag\_name (string):* The tag name.

* *project\_id:* (Defaults to current project id) The project ID.

### tags\_remove(tag\_name, project\_id=None)

Remove a tag from a project.

* *tag\_name (string):* The tag name.

* *project\_id:* (Defaults to current project id) The project ID.

## Executions

> **`runs_start` vs `job_start` — which should I use?**
>
> The SDK exposes two ways to start an execution:
>
> | | `runs_start` / `runs_start_blocking` | `job_start` / `job_start_blocking` |
> | - | - | - |
> | **API version** | v1 (legacy) | v4 (current) |
> | **Command format** | List of strings: `["main.py", "arg1"]` | Single string: `"main.py arg1"` |
> | **Hardware tier** | By name (`tier`) | By name (`hardware_tier_name`) or ID (`hardware_tier_id`) |
> | **Compute clusters** | Not supported | Spark, Ray, Dask, MPI via `compute_cluster_properties` |
> | **Git ref targeting** | By commit ID only | By commit ID, branch, or raw `main_repo_git_ref` dict |
> | **External volumes** | Not supported | Supported via `external_volume_mounts` |
> | **Recommended for** | Legacy scripts and simple use cases | All new work |
>
> Use `job_start` for all new development. `runs_start` is retained for backwards compatibility.

See these code example files:

* [`start_run_and_check_status.py`](https://github.com/dominodatalab/python-domino/blob/Release-2.2.0/examples/start_run_and_check_status.py)

* [`export_runs.py`](https://github.com/dominodatalab/python-domino/blob/Release-2.2.0/examples/export_runs.py)

### runs\_list()

List the executions on the selected project.

### runs\_start(command, isDirect, commitId, title, tier, publishApiEndpoint)

Start a new execution on the selected project.

* *command:* The command to execution as an array of strings where
  members of the array represent arguments of the command. For
  example: `["main.py", "hi mom"]`

* *isDirect:* (Optional) Whether this command should be passed
  directly to a shell.

* *commitId:* (Optional) The `commitId` to launch from. If not
  provided, the project launches from the latest commit.

* *title:* (Optional) A title for the execution.

* *tier:* (Optional) The hardware tier to use for the execution. This
  is the human-readable name of the hardware tier, such as "Free",
  "Small", or "Medium". If not provided, the project’s default tier is
  used.

* *publishApiEndpoint:* (Optional) Whether to publish an API endpoint
  from the resulting output.

### runs\_start\_blocking(command, isDirect, commitId, title, tier, publishApiEndpoint, poll\_freq=5, max\_poll\_time=6000)

Start a new execution on the selected project and make a blocking
request that waits until job is finished.

* *command:* The command to execution as an array of strings where
  members of the array represent arguments of the command. For
  example: `["main.py", "hi mom"]`

* *isDirect:* (Optional) Whether this command should be passed
  directly to a shell.

* *commitId:* (Optional) The `commitId` to launch from. If not
  provided, the project launches from the latest commit.

* *title:* (Optional) A title for the execution.

* *tier:* (Optional) The hardware tier to use for the execution. Will
  use project’s default tier if not provided. If not provided, the
  project’s default tier is used.

* *publishApiEndpoint:* (Optional) Whether to publish an API endpoint
  from the resulting output.

* *poll\_freq:* (Optional) Number of seconds between polling of the
  Domino server for status of the task that is running.

* *max\_poll\_time:* (Optional) Maximum number of seconds to wait for
  a task to complete. If this threshold is exceeded, an exception is
  raised.

* *retry\_count:* (Optional) Maximum number of polling retries (in
  case of transient HTTP errors). If this threshold is exceeded, an
  exception is raised.

### run\_stop(runId, saveChanges=True):

Stop an existing execution in the selected project.

* *runId:* String that identifies the execution.

* *saveChanges:* (Defaults to True) If false, execution results are
  discarded.

### runs\_stdout(runId)

Get `stdout` emitted by a particular execution.

* *runId:* string that identifies the execution

## Files and blobs

See these code example files:

* [`upload_file.py`](https://github.com/dominodatalab/python-domino/blob/Release-2.2.0/examples/upload_file.py)

* [`upload_and_run_file_and_download_results.py`](https://github.com/dominodatalab/python-domino/blob/Release-2.2.0/examples/upload_and_run_file_and_download_results.py)

### files\_list(commitId, path)

List the files in a folder in the Domino project.

* *commitId:* The `commitId` to list files from.

* *path:* (Defaults to "/") The path to list from.

### files\_upload(path, file)

Upload a Python file object into the specified path inside the project.
See `examples/upload_file.py` for an example. All parameters are
required.

* *path:* The path to save the file to. For example, `/README.md`
  writes to the root directory of the project while
  `/data/numbers.csv` saves the file to a sub folder named `data`. If
  the specified folder does not yet exist, it is created.

* *file:* A Python file object. For example:
  `f = open("authors.txt","rb")`

### blobs\_get(key)

**Deprecated** Retrieve a file from the Domino server by blob key. Use blobs\_get\_v2(path, commit\_id, project\_id) instead.

* *key:* The key of the file to fetch from the blob server.

### blobs\_get\_v2(path, commit\_id, project\_id)

Retrieve a file from the Domino server in a project from its path and commit id.

* *path:* The path to the file in the Domino project.
* *commit\_id:* ID of the commit to retrieve the file from.
* *project\_id:* ID of the project to retrieve the file from.

### files\_download(path, commit\_id=None)

Convenience wrapper around `blobs_get_v2` that downloads a file by path without
needing to look up a commit ID first.

* *path (string):* Path to the file within the project, e.g. `"/README.md"`.

* *commit\_id (string):* (Optional) The commit to download from. Defaults to
  the latest commit in the project.

Returns a raw file content stream (urllib3 response). Example:

```python theme={null}
content = d.files_download("/results/output.csv").read()
```

## Apps

### app\_publish(unpublish\_running\_apps=True, hardware\_tier\_id=None, environment\_id=None, external\_volume\_mount\_ids=None, commit\_id=None, branch=None, app\_id=None)

Publish an app within a project, or republish an existing app.

* *unpublish\_running\_apps:* (Defaults to `True`) Check for an active
  app instance in the current project and stop it before
  re/publishing.

* *hardware\_tier\_id:* (Optional) Launch the app on the specified
  hardware tier ID.

* *environment\_id:* (Optional) Launch the app with the specified
  environment ID.

* *external\_volume\_mount\_ids:* (Optional) List of external volume
  mount IDs to attach to the app.

* *commit\_id:* (Optional) Launch the app from a specific commit.
  Cannot be combined with `branch`.

* *branch:* (Optional) Launch the app from the tip of a specific
  branch. Cannot be combined with `commit_id`.

* *app\_id:* (Optional) The ID of the app to publish. If omitted, the
  project's default app is used (or a new one is created if none
  exists).

```python theme={null}
# Publish from a specific branch
d.app_publish(branch="my-feature-branch")

# Publish from a specific commit
d.app_publish(commit_id="abc123def456")

# Publish a specific app by ID
d.app_publish(app_id="aabbccddeeff001122334457")
```

> **Note:** The parameters `unpublishRunningApps`, `hardwareTierId`,
> `environmentId`, `externalVolumeMountIds`, `commitId`, and `appId`
> are deprecated and will be removed in the next major version.
> Use the `snake_case` equivalents listed above.
>
> These renames are Python-side only — the JSON keys sent to the
> Domino HTTP API are unchanged (the SDK still emits `commitId`,
> `hardwareTierId`, etc. on the wire). The HTTP API contract is
> not affected.

### app\_unpublish(app\_id=None)

Stop the running app in the project.

* *app\_id:* (Optional) The ID of the app to stop. If omitted, the
  project's default app is used.

### app\_get\_status(app\_id)

Return the current status of an app.

* *app\_id (string):* The ID of the app to query.

Returns the status string (e.g. `"Running"`, `"Stopped"`, `"Failed"`), or
`None` if the app does not exist.

### app\_id

Read-only property. Returns the ID of the first app in the current project,
or `None` if no app exists. Useful when you need the app ID to pass to
`app_get_status()` or `app_publish()`.

```python theme={null}
print(d.app_id)  # e.g. "aabbccddeeff001122334457"
```

## Jobs

> **Prefer `job_start` over `runs_start` for all new work.** See the [Executions](#executions) section for a full comparison.

### job\_start(command, commit\_id=None, branch=None, hardware\_tier\_name=None, environment\_id=None, on\_demand\_spark\_cluster\_properties=None, compute\_cluster\_properties=None, external\_volume\_mounts=None, title=None, main\_repo\_git\_ref=None):

Start a new job (execution) in the project using the v4 Jobs API.

* *command (string):* Command to execute in Job. For example:
  `domino.job_start(command="main.py arg1 arg2")`

* *commit\_id (string):* (Optional) The commit ID to launch from. If
  not provided, the job launches from the latest commit. Mutually
  exclusive with `branch`.

  This always pins the project's internal DFS commit, regardless of
  project type. For git-based projects, that is not the same as a
  commit on the main git repository: a valid DFS commit SHA still
  works here, but a commit SHA from the main git repository fails.
  To pin a commit on a git-based project's main repository, use
  `main_repo_git_ref={"type": "commitId", "value": <sha>}` instead.

* *branch (string):* (Optional) The branch name to launch from. If
  not provided, the job launches from the latest commit on the default
  branch. Mutually exclusive with `commit_id` and `main_repo_git_ref`.

* *hardware\_tier\_name (string):* (Optional) The hardware tier NAME
  to launch job in. If not provided, the project’s default tier is
  used.

* *environment\_id (string):* (Optional) The environment ID with which
  to launch the job. If not provided, the project’s default
  environment is used.

* *on\_demand\_spark\_cluster\_properties (dict):* (Optional) On
  demand spark cluster properties. The following properties can be
  provided in the Spark cluster:

```
{
    "computeEnvironmentId": "<Environment ID configured with spark>"
    "executorCount": "<Number of Executors in cluster>"
     (optional defaults to 1)
    "executorHardwareTierId": "<Hardware tier ID for Spark Executors>"
     (optional defaults to last used historically if available)
    "masterHardwareTierId":  "<Hardware tier ID for Spark master"
     (optional defaults to last used historically if available)
    "executorStorageMB": "<Executor's storage in MB>"
     (optional defaults to 0; 1GB is 1000MB Here)
}
```

* *param compute\_cluster\_properties (dict):* (Optional) The
  compute-cluster properties definition contains parameters for
  launching any Domino supported compute cluster for a job. Use this
  to launch a job that uses a compute-cluster instead of the
  deprecated `on_demand_spark_cluster_properties` field. If
  `on_demand_spark_cluster_properties` and
  `compute_cluster_properties` are both present,
  `on_demand_spark_cluster_properties` is ignored.
  `compute_cluster_properties` contains the following fields:

```
{
    "clusterType": <string, one of "Ray", "Spark", "Dask", "MPI">,
    "computeEnvironmentId": <string, The environment ID for the cluster's nodes>,
    "computeEnvironmentRevisionSpec": <one of "ActiveRevision", "LatestRevision",
    {"revisionId":"<environment_revision_id>"} (optional)>,
    "masterHardwareTierId": <string, the Hardware tier ID for the cluster's master node (required unless clusterType is MPI)>,
    "workerCount": <number, the total workers to spawn for the cluster>,
    "workerHardwareTierId": <string, The Hardware tier ID for the cluster workers>,
    "workerStorage": <{ "value": <number>, "unit": <one of "GiB", "MB"> },
    The disk storage size for the cluster's worker nodes (optional)>
    "maxWorkerCount": <number, The max number of workers allowed. When
    this configuration exists, autoscaling is enabled for the cluster and
    "workerCount" is interpreted as the min number of workers allowed in the cluster
    (optional)>
}
```

* *external\_volume\_mounts (List\[string]):* (Optional) External
  volume mount IDs to mount to execution. If not provided, the job
  launches with no external volumes mounted.

* \*title (string): (Optional) Title for Job.

* *main\_repo\_git\_ref (dict):* (Optional) For git-based projects,
  specifies the branch, tag, or commit to run from. Must contain
  `"type"` and `"value"` keys. Supported types are `"branches"`,
  `"tags"`, and `"commitId"`. For example:

```
{"type": "branches", "value": "my-feature-branch"}
{"type": "tags", "value": "v1.2.3"}
{"type": "commitId", "value": "abc123def456"}
```

Use `"commitId"` to pin a specific commit SHA on the main git
repository (`commit_id` cannot be used for that purpose). Cannot
be combined with `branch`. If not provided, the job launches from
the latest commit on the default branch.

### job\_stop(job\_id, commit\_results=True):

Stop the Job (execution) in the project.

* *job\_id (string):* Job identifier.

* *commit\_results (boolean):* (Defaults to `true`) If `false`, the
  job results are not committed.

### jobs\_list(self, project\_id: str, order\_by: str = "number", sort\_by: str = "desc", page\_size: Optional\[int] = None, page\_no: int = 1, show\_archived: str = "false", status: str = "all", tag: Optional\[str] = None):

Lists job history for a given project\_id

* *project\_id (string):* The project to query

* *order\_by (string):* Field on which sort has to be applied– e.g. "title" (default "number")

* *sort\_by (string):* Sort "desc" (default) or "asc"

* *page\_size (integer):* The number of jobs to return (default: 3)

* *page\_no (integer):* Page number to fetch (default: 1).

* *show\_archived (string):* Show archived jobs in results (default: false)

* *status (string):* Status of jobs to fetch– e.g. "completed" (default: "all")

* *tag (string):* Optional tag filter

### job\_status(job\_id):

Get the status of a job.

* *job\_id (string):* Job identifier.

### job\_restart(job\_id, should\_use\_original\_input\_commit=True):

Restart a previous job

* *job\_id (string):* ID of the original job. This can be obtained with `jobs_list()`.

* *should\_use\_original\_input\_commit (bool):* Should the new job run use the original code, or the current version?

### job\_start\_blocking(poll\_freq=5, max\_poll\_time=6000, \*\*kwargs):

Start a job and poll until the job is finished. Additionally, this
method supports all the parameters in the `job_start` method.

* *poll\_freq:* Poll frequency interval in seconds.

* *max\_poll\_time:* Max poll time in seconds.

## Datasets

A Domino dataset is a collection of files that are available in user
executions as a filesystem directory. A dataset always reflects the most
recent version of the data. You can modify the contents of a dataset
through the Domino UI or through workload executions.

See [Domino
Datasets](https://docs.dominodatalab.com/en/latest/user_guide/0a8d11/datasets-overview/)
for more details, and
[`example_dataset.py`](https://github.com/dominodatalab/python-domino/blob/Release-2.2.0/examples/example_dataset.py)
for example code.

### datasets\_list(project\_id=None)

Provide a JSON list of all the available datasets.

* *project\_id (string):* (Defaults to None) The project identifier.
  Each project can hold up to 5 datasets.

### datasets\_ids(project\_id)

List the IDs the datasets for a particular project.

* *project\_id:* The project identifier.

### datasets\_names(project\_id)

List the names the datasets for a particular project.

* *project\_id:* The project identifier.

### datasets\_details(dataset\_id)

Provide details about a dataset.

* *dataset\_id:* The dataset identifier.

### datasets\_create(dataset\_name, dataset\_description)

Create a new dataset.

* *dataset\_name:* Name of the new dataset. NOTE: The name must be
  unique.

* *dataset\_description:* Description of the dataset.

### datasets\_update\_details(dataset\_id, dataset\_name=None, dataset\_description=None)

Update a dataset’s name or description.

* *dataset\_id:* The dataset identifier.

* *dataset\_name:* (Optional) New name of the dataset.

* *dataset\_description:* (Optional) New description of the dataset.

### datasets\_remove(dataset\_ids)

Delete a set of datasets.

* *dataset\_ids (list\[string]):* List of IDs of the datasets to
  delete. NOTE: Datasets are first marked for deletion, then deleted
  after a grace period (15 minutes, configurable). A Domino admin may
  also need to complete this process before the name can be reused.

### datasets\_upload\_files(dataset\_id, local\_path\_to\_file\_or\_directory, file\_upload\_setting, max\_workers, target\_chunk\_size, target\_relative\_path)

Uploads a file or entire directory to a dataset.

* *dataset\_id:* The dataset identifier.
* *local\_path\_to\_file\_or\_directory:* The path to the file or directory in local machine.
* *file\_upload\_setting:* The setting to resolve naming conflict, must be one of `Overwrite`, `Rename`, `Ignore` (default).
* *max\_workers:* The max amount of threads (default: 10).
* *target\_chunk\_size:* The max chunk size for multipart upload (default: 8MB).
* *target\_relative\_path:* The path on the dataset to upload the file or directory to. Note that the path must exist or the upload will fail.

# Agents

## domino.agents.environment\_variables

DOMINO\_AGENT\_CONFIG\_PATH:\
For configuring the location of the agent\_config.yaml file. If not set,
defaults to ‘./agent\_config.yaml’.

type:\
str

DOMINO\_AGENT\_IS\_PROD:\
Indicates if the Agent is running in production mode. Set to ‘true’ to
optimize for production.

type:\
str

DOMINO\_APP\_ID:\
Indicates the ID of the Agent application. Must be set in production
mode.

type:\
str

MLFLOW\_TRACKING\_URI:\
Used to configure Mlflow functionality. It is required in order for
library to work and will be set automatically when running in Domino.

type:\
str

## domino.agents.logging

Functions

| | |
| - | - |
| `log_evaluation`(trace\_id, name, value) | This logs evaluation data and metadata to a parent trace. |

Classes

| | |
| - | - |
| `DominoRun`(\[experiment\_name, run\_id, ...]) | DominoRun is a context manager that starts an Mlflow run and attaches the user's Agent configuration to it, create a Logged Model with the Agent configuration, and computes summary metrics for evaluation traces made during the run. |

### *class* domino.agents.logging.DominoRun(*experiment\_name: str | None = None*, *run\_id: str | None = None*, *agent\_config\_path: str | None = None*, *custom\_summary\_metrics: list\[str, Literal\['mean', 'median', 'stdev', 'max', 'min']] | None = None*)

Bases: `object`

DominoRun is a context manager that starts an Mlflow run and attaches
the user’s Agent configuration to it, create a Logged Model with the
Agent configuration, and computes summary metrics for evaluation traces
made during the run. Average metrics are computed by default, but the
user can provide a custom list of evaluation metric aggregators. This is
intended to be used in development mode for Agent evaluation. Context
manager docs: [https://docs.python.org/3/library/contextlib.html](https://docs.python.org/3/library/contextlib.html)
When running an agentic workflow, use DominoAgentContext instead to indicate the run in the context of an agent.

Parallelism: DominoRun is not thread-safe. Runs in different threads
will work correctly. This is due to Mlflow’s architecture. Parallelizing
operations within a single DominoRun context however, is supported.

Example

```python theme={null}
import mlflow

mlflow.set_experiment(“my_experiment”)

with DominoRun():
train_model()
```

Parameters:

* **experiment\_name** – the name of the mlflow experiment to log the run
  to.

* **run\_id** – optional, the ID of the mlflow run to continue logging
  to. If not provided a new run will start.

* **agent\_config\_path** – the optional path to the Agent configuration
  file. If not provided, defaults to the DOMINO\_AGENT\_CONFIG\_PATH
  environment variable.

* **custom\_summary\_metrics** – an optional list of tuples that define
  what summary statistic to use with what evaluation metric. Valid
  summary statistics are: “mean”, “median”, “stdev”, “max”, “min” e.g.
  \[(“hallucination\_rate”, “max”)]

Returns: DominoRun context manager

 

### *class* domino.agents.logging.DominoAgentContext(*experiment\_name: str | None = None*, *run\_id: str | None = None*, *agent\_config\_path: str | None = None*, *custom\_summary\_metrics: list\[str, Literal\['mean', 'median', 'stdev', 'max', 'min']] | None = None*)

Bases: `DominoRun`

DominoAgentContext behaves identically to DominoRun but it is used to indicate that the run originated from an agent.
Use DominoAgentContext instead of DominoRun when the run is part of an agentic workflow.

Example

```python theme={null}
import mlflow

mlflow.set_experiment("my_experiment")

with DominoAgentContext():
agent.run()
```

Parameters:

* **experiment\_name** – the name of the mlflow experiment to log the run
  to.

* **run\_id** – optional, the ID of the mlflow run to continue logging
  to. If not provided a new run will start.

* **agent\_config\_path** – the optional path to the Agent configuration
  file. If not provided, defaults to the DOMINO\_AGENT\_CONFIG\_PATH
  environment variable.

* **custom\_summary\_metrics** – an optional list of tuples that define
  what summary statistic to use with what evaluation metric. Valid
  summary statistics are: "mean", "median", "stdev", "max", "min" e.g.
  \[("hallucination\_rate", "max")]

Returns: DominoAgentContext context manager

 

### domino.agents.logging.log\_evaluation(*trace\_id: str*, *name: str*, *value: float | str*)

This logs evaluation data and metadata to a parent trace. This is used
to log the evaluation of a span after it was created. This is useful for
analyzing past performance of an Agent component.

Parameters:

* **trace\_id** – the ID of the trace to evaluate

* **name** – a label for the evaluation result. This is used to identify
  the evaluation result

* **value** – the evaluation result to log. This must be a float or
  string

Modules

| | |
| - | - |
| `dominorun` | |
| `logging` | |

## domino.agents.tracing

Functions

| | |
| - | - |
| `add_tracing`(name\[, autolog\_frameworks, ...]) | This is a decorator that starts an mlflow span for the function it decorates. |
| `init_tracing`(\[autolog\_frameworks]) | Initialize Mlflow autologging for various frameworks and sets the active experiment to enable tracing in production. |
| `search_agent_traces`(agent\_id\[, ...]) | This allows searching for traces that have a certain name and returns a paginated response of trace summaries that include the spans that were requested. |
| `search_traces`(run\_id\[, trace\_name, ...]) | This allows searching for traces that have a certain name and returns a paginated response of trace summaries that inclued the spans that were requested. |

Classes

| | |
| - | - |
| `EvaluationResult`(name, value) | An evaluation result for a trace. |
| `SearchTracesResponse`(data, page\_token) | The response from searching for traces. |
| `SpanSummary`(id, name, trace\_id, inputs, outputs) | A span in a trace. |
| `TraceSummary`(name, id, spans, evaluation\_results) | A summary of a trace. |

### *class* domino.agents.tracing.EvaluationResult(*name: str*, *value: float | str*)

Bases: `object`

An evaluation result for a trace.

name\*: str\*\
The name of the evaluation

value\*: float | str\*\
The value of the evaluation

 

### *class* domino.agents.tracing.SearchTracesResponse(*data: list\[TraceSummary]*, *page\_token: str | None*)

Bases: `object`

The response from searching for traces.

data\*: list\[TraceSummary]\*\
The list of trace summaries

page\_token\*: str | None\*\
The token for the next page of results

 

### *class* domino.agents.tracing.SpanSummary(*id: str*, *name: str*, *trace\_id: str*, *inputs: Any*, *outputs: Any*)

Bases: `object`

A span in a trace.

id\*: str\*\
the mlflow ID of the span

inputs\*: Any\*\
The inputs to the function that created the span

name\*: str\*\
The name of the span

outputs\*: Any\*\
The outputs of the function that created the span

trace\_id\*: str\*\
The parent trace ID

 

### *class* domino.agents.tracing.TraceSummary(*name: str*, *id: str*, *spans: list\[SpanSummary]*, *evaluation\_results: list\[EvaluationResult]*)

Bases: `object`

A summary of a trace.

evaluation\_results\*: list\[EvaluationResult]\*\
The evaluation results for this trace

id\*: str\*\
The mlflow ID of the trace

name\*: str\*\
The name of the trace

spans\*: list\[SpanSummary]\*\
The child spans of this trace

 

### domino.agents.tracing.add\_tracing(*name: str*, *autolog\_frameworks: list\[str] | None = \[]*, *evaluator: Callable\[\[mlflow\.entities.Span], dict\[str, int | float | str]] | None = None*, *trace\_evaluator: Callable\[\[mlflow\.entities.Trace], dict\[str, int | float | str]] | None = None*, *eagerly\_evaluate\_streamed\_results: bool = True*, *span\_type: str | None = SpanType.UNKNOWN*, *attributes: dict\[str, Any] | None = None*)

This is a decorator that starts an mlflow span for the function it
decorates. If there is an existing trace a span will be appended to it.
If there is no existing trace, a new trace will be created.

It also enables the user to run evaluators when the code is run in
development mode. Evaluators can be run on the span and/or trace
generated for the wrapped function call. The trace evaluator will run if
the parent trace was started and finished by the related decorator call.
The trace will contain all child span information. The span evaluator
will always run. The evaluation results from both evaluators will be
combined and saved to the trace.

This decorator must be used directly on the function to be traced
without any intervening decorators, because it must have access to the
arguments.

@add\_tracing(\
name=”assistant\_chat\_bot”, evaluator=evaluate\_helpfulness,

) def ask\_chat\_bot(user\_input: str) -> dict:

> …

Parameters:

* **name** – the name of the span to add to existing trace or create if
  no trace exists yet.

* **autolog\_frameworks** – an optional list of mlflow supported
  frameworks to autolog

* **evaluator** – an optional function that takes the span created for
  the wrapped function and returns a dictionary of evaluation results.
  The evaluation results will be saved to the trace

* **trace\_evaluator** – an optional function that takes the trace for
  this call stack and returns a dictionary of evaluation results. This
  evaluator will be triggered if the trace was started and finished by
  the add tracing decorator. The evaluation results will be saved to the
  trace

* **eagerly\_evaluate\_streamed\_results** – optional boolean, defaults to
  true, this determines if all yielded values should be aggregated and
  set as outputs to a single span. This makes evaluation easier, but
  will impact performance if you expect a large number of streamed
  values. If set to false, each yielded value will generate a new span
  on the trace, which can be evaluated post-hoc. Inline evaluators won’t
  be executed. Each span will have a group\_id set in their attributes to
  indicate that they are part of the same function call. Each span will
  have an index to indicate what order they arrived in.

* **allow\_tracing\_evaluator** – optional boolean, defaults to false.
  This determines if inline evaluators will be traced by mlflow autolog.

* **span\_type** – optional string specifying the type of span, passed
  directly to `mlflow.start_span()`. Defaults to `SpanType.UNKNOWN`.
  Common types from `mlflow.entities.SpanType` include LLM, AGENT,
  TOOL, CHAIN, RETRIEVER, CHAT\_MODEL, PARSER, EMBEDDING, RERANKER.
  Consult the [MLflow documentation](https://mlflow.org/docs/latest) for more information.

* **attributes** – optional dictionary of additional attributes to set
  on the span, passed directly to `mlflow.start_span()`. These can be
  used to attach metadata and context to the span. For example:
  \{"model": "gpt-4", "temperature": 0.7}.
  Consult the [MLflow documentation](https://mlflow.org/docs/latest) for more information.

Returns:\
A decorator that wraps the function to be traced.

 

### domino.agents.tracing.init\_tracing(*autolog\_frameworks: list\[str] | None = None*)

Initialize Mlflow autologging for various frameworks and sets the active
experiment to enable tracing in production. This may be used to
initialize logging and tracing for the Agent in dev and prod modes.

In prod mode, environment variables DOMINO\_AGENT\_IS\_PROD, DOMINO\_APP\_ID
must be set. Call init\_tracing before your app starts up to start
logging traces to Domino.

Parameters:\
**autolog\_frameworks** – list of frameworks to autolog

 

### domino.agents.tracing.search\_agent\_traces(*agent\_id: str*, *agent\_version: str | None = None*, *trace\_name: str | None = None*, *start\_time: datetime | None = None*, *end\_time: datetime | None = None*, *page\_token: str | None = None*, *max\_results: int | None = None*) → SearchTracesResponse

This allows searching for traces that have a certain name and returns a
paginated response of trace summaries that include the spans that were
requested.

Parameters:

* **agent\_id** – string, the ID of the Agent to filter by

* **agent\_version** – string, the version of the Agent to filter by, if
  not provided will search throuh all versions

* **trace\_name** – the name of the traces to search for

* **start\_time** – python datetime

* **end\_time** – python datetime, defaults to now

* **page\_token** – page token for pagination. You can use this to
  request the next page of results and may find a page\_token in the
  response of the previous search\_traces call.

* **max\_results** – defaults to 100

Returns:\
a token based pagination response that contains a list of trace summaries\
data: list of TraceSummary page\_token: the next page’s token

Return type:\
SearchTracesResponse

 

### domino.agents.tracing.search\_traces(*run\_id: str*, *trace\_name: str | None = None*, *start\_time: datetime | None = None*, *end\_time: datetime | None = None*, *page\_token: str | None = None*, *max\_results: int | None = None*) → SearchTracesResponse

This allows searching for traces that have a certain name and returns a
paginated response of trace summaries that inclued the spans that were
requested.

Parameters:

* **run\_id** – string, the ID of the development mode evaluation run to
  search for traces.

* **trace\_name** – the name of the traces to search for

* **start\_time** – python datetime

* **end\_time** – python datetime, defaults to now

* **page\_token** – page token for pagination. You can use this to
  request the next page of results and may find a page\_token in the
  response of the previous search\_traces call.

* **max\_results** – defaults to 100

Returns:\
a token based pagination response that contains a list of trace summaries\
data: list of TraceSummary page\_token: the next page’s token

Return type:\
SearchTracesReponse

Modules

| | |
| - | - |
| `inittracing` | |
| `tracing` | |

# Airflow

The `python-domino` client comes bundled with an
[Operator](https://airflow.apache.org/docs/apache-airflow/stable/concepts/operators.html)
for use with [Apache Airflow](https://airflow.apache.org/) as an extra.

When installing the client from PyPI, add the `airflow` flag to extras:

```
pip install "dominodatalab[airflow]"
```

Similarly, when installing the client from GitHub, use the following
command:

```
pip install -e git+https://github.com/dominodatalab/python-domino.git@1.0.6#egg="dominodatalab[airflow]"
```

See also
[example\_airflow\_dag.py](https://github.com/dominodatalab/python-domino/blob/Release-2.2.0/examples/example_airflow_dag.py)
for example code.

## DominoOperator

```
from domino.airflow import DominoOperator
```

Allows a user to schedule Domino executions via Airflow. Follows the
same function signature as `domino.runs_start` with two extra arguments:

* `startup_delay: Optional[int] = 10` | Add a startup delay to your job, useful if you want to delay execution until after other work finishes.
* `include_setup_log: Optional[bool] = True` | Determine whether or not to publish the setup log of the job as the log prefix before `stdout`.

## DominoSparkOperator

```
from domino.airflow import DominoSparkOperator
```

Allows a user to schedule Domino executions via the v4 API, which
supports `onDemandSparkClusters`. Follows the same function signature as
`domino.job_start`, with the addition of `startup_delay` from above.

# Example

```
from domino import Domino

# By and large your commands will run against a single project,
# so you must specify the full project name
domino = Domino("chris/canon")

# List all runs in the project, most-recently queued first
all_runs = domino.runs_list()['data']

latest_100_runs = all_runs[0:100]

print(latest_100_runs)

# all runs have a commitId (the snapshot of the project when the
# run starts) and, if the run completed, an "outputCommitId"
# (the snapshot of the project after the run completed)
most_recent_run = all_runs[0]

commitId = most_recent_run['outputCommitId']

# list all the files in the output commit ID -- only showing the
# entries under the results directory.  If not provided, will
# list all files in the project.  Or you can say path=“/“ to
# list all files
files = domino.files_list(commitId, path='results/')['data']

for file in files:
print file['path'], '->', file['url']

print(files)

# Get the content (i.e. blob) for the file you're interested in.
# blobs_get returns a connection rather than the content, because
# the content can get quite large and it's up to you how you want
# to handle it
print(domino.blobs_get(files[0]['key']).read())

# Start a run of file main.py using the latest copy of that file
domino.runs_start(["main.py", "arg1", "arg2"])

# Start a "direct" command
domino.runs_start(["echo 'Hello, World!'"], isDirect=True)

# Start a run of a specific commit
domino.runs_start(["main.py"], commitId="aabbccddee")
```

# Manual installation

Because `python-domino` ships with the DSE, normally you do not need to install it.\
This section provides instructions for installing it in another environment or updating it to a newer version.

Starting from version `1.0.6`, `python-domino` is available on PyPI as `dominodatalab`:

```
pip install dominodatalab
```

If you are adding install instructions for `python-domino` to your [Domino Environment](https://support.dominodatalab.com/hc/en-us/articles/115000392643-Compute-Environment-Management) Dockerfile Instructions field, you must add `RUN` to the beginning:

```
RUN pip install dominodatalab
```

To install a specific version of the library from PyPI, such as `1.0.6`:

```
pip install dominodatalab==1.0.6
```

To install a specific version of the library from GitHub, such as
`1.0.6`:

```
pip install https://github.com/dominodatalab/python-domino/archive/1.0.6.zip
```

# License

This library is made available under the Apache 2.0 License. This is an
open-source project of [Domino Data Lab](https://www.dominodatalab.com).

## Visit the `python-domino` repository

You can find the complete library, with documentation and example code, in the public repository at [https://github.com/dominodatalab/python-domino](https://github.com/dominodatalab/python-domino).


# Python Wrapper for Domino API
Source: https://docs.domino.ai/cloud/reference/python-sdk/python-wrapper-for-domino-api

Install and authenticate the Python wrapper for the Domino API, then list runs and files, download blobs, and start executions from code.

The Python binding for the Domino API is available in PyPI:

```shell theme={null}
pip install dominodatalab
```

If you want to use the Python binding in a Domino workbook session include `dominodatalab` in your project’s [requirements.txt file](/cloud/platform-capabilities/core-concepts/compute-environments/add-packages-to-environments/use-requirements-txt). This makes the Python binding available for each new workbook session (or batch run) started within the project.

<Tip>
  Full documentation for this API is forthcoming. For more information now, see [Domino’s public Python bindings project](https://github.com/dominodatalab/python-domino).
</Tip>

**In existing deployments**

After installation, you can instantiate a library either with your API Key or Auth Token File.

**Token File:** To use Token File to instantiate Python Domino, pass the path to this token file, either through class constructor (`domino_token_file=<Path to Token file>`) or through an environment variable.

```python theme={null}
export DOMINO_TOKEN_FILE=PATH_TO_DOMINO_TOKEN_FILE
```

If you are using Python package in code that is already running in Domino, the `DOMINO_TOKEN_FILE` is set automatically to be the token file for the user who started the run.

**API Key:** You must get your API key from your account page. To get your API Key, log into Domino and click your name in the menu. Click **Account Settings** and select **API Key**. Copy the API key to your clipboard.

The Python library reads this key from environment variables, so set it as follows in your shell:

```python theme={null}
export DOMINO_USER_API_KEY=YOUR_API_KEY
```

If you are using the Python package in code that is already running in Domino, the `DOMINO_API_USER_KEY` variable is set automatically to be the key for the user who started the run.

<Note>
  * If both the API Key and Token files are present, default preference is given to the Token file. To use the API Key file instead, clear the `DOMINO_TOKEN_FILE` environment variable.

  * See the [Domino Platform API Reference](/cloud/reference/api/domino-open-api) for more detail.
</Note>

The following is an example of usage:

```python theme={null}
from domino import Domino

# By and large your commands will run against a single project,
# so you must specify the full project name
domino = Domino("chris/canon")

# List all runs in the project, most-recently queued first
all_runs = domino.runs_list()['data']

latest_100_runs = all_runs[0:100]

print(latest_100_runs)

# all runs have a commitId (the snapshot of the project when the
# run starts) and, if the run completed, an "outputCommitId"
# (the snapshot of the project after the run completed)
most_recent_run = all_runs[0]

commitId = most_recent_run['outputCommitId']

# list all the files in the output commit ID -- only showing the
# entries under the results directory.  If not provided, will
# list all files in the project.  Or you can say path=“/“ to
# list all files
files = domino.files_list(commitId, path='results/')['data']

for file in files:
print file['path'], '->', file['url']

print(files)

# Get the content (i.e. blob) for the file you're interested in.
# blobs_get returns a connection rather than the content, because
# the content can get quite large and it's up to you how you want
# to handle it
print(domino.blobs_get(files[0]['key']).read())

# Start a run of file main.py using the latest copy of that file
domino.runs_start(["main.py", "arg1", "arg2"])

# Start a "direct" command
domino.runs_start(["echo 'Hello, World!'"], isDirect=True)

# Start a run of a specific commit
domino.runs_start(["main.py"], commitId="aabbccddee")
```

**In new deployments**

Domino updated the Python library to work with the API Proxy inside the execution. If the library is instantiated without any authentication arguments, it detects the API Proxy (see the `DOMINO_API_PROXY` environment variable) and routes the calls through it automatically. If you explicitly specify other means of authentication (`domino_token`, `domino_token_file`, `api_key`), it works exactly like it did previously.

See [Use API Proxy to Authenticate Calls to the Domino API](/cloud/reference/api/domino-api-authentication).


# R package
Source: https://docs.domino.ai/cloud/reference/r-package/domino-r-package

Install the Domino R package to retrieve and initialize Projects, run code with arguments, download results, and synchronize local and server files.

The Domino R package gives you access to the full power of Domino’s cloud compute and version control without leaving your R environment.

<Note>
  This information may be outdated. For additional functionality, consider writing your own R wrapper around the platform API.
</Note>

## Install

The package is [hosted on CRAN](http://cran.r-project.org/web/packages/domino/), like any standard R package.

Install the Domino [command line client](/cloud/reference/cli/install-the-cli). You won’t use it directly, but it does have to be installed on your computer.

From R, run the following:

```r theme={null}
install.packages("domino", repos='http://cran.r-project.org', type="source")
```

After it’s installed you can import the package as you would any other R package:

```r theme={null}
library(domino)
```

## Log in

You must login again from R to verify your identity to Domino. To do that type:

```r theme={null}
domino.login("yourUsername", host="https://app.dominodatalab.com")
```

You should get a success message. If you’re running in a private deployment, your host is the address of your internal Domino server.

## Get a Project

<Note>
  If you already have a copy of your project on your computer, skip this step. Set your working directory to the root of that project folder and continue to the next section.
</Note>

To download an existing project, run this command:

```r theme={null}
domino.get("quick-start")
```

You should see something like this:

```r theme={null}
> domino.get("quick-start")
Getting project chris/quick-start...
Project get complete. Make sure to navigate to your new project folder by running:
    cd quick-start
To run a file on Domino, type:
    domino run [command-to-run]
[1] "Changed working directory to new project's director"
```

This command also changes your working directory to the root of the project folder. In general, these Domino commands only work if your working directory is set to be inside your project folder.

## Run code

To start the execution of a script on Domino, type:

```r theme={null}
domino.run("myScript.R")
```

You should see something like this:

```r theme={null}
> domino.run("myScript.R")
Determining which files are out of date...

No changes to upload to server, because you are up to date.
Run for project chris/quick-start started. You can view progress here:
  https://app.dominodatalab.com/chris/quick-start/run/53051188e4b0b5aeedd6a73b
```

You can also pass in arguments into the scripts:

```r theme={null}
domino.run("myScript.R", 1, "someString")
```

You can read in those parameters in your script:

```r theme={null}
> firstParameter  1
> secondParameter  "someString"
```

You can launch several jobs:

```r theme={null}
for (alpha <- 1:3) {
    for (beta <- 1:3) {
        domino.run("myScript.R", alpha, beta)
    }
}
```

This queues 9 jobs, each with a unique setting of the values alpha and beta.

## Download results

After your runs have completed, you get download the results back to your computer by running:

```r theme={null}
domino.download()
```

This downloads all the new and modified files in your project to your local computer. Once this command completes, these files are available immediately to be loaded into your R workspace:

```r theme={null}
> domino.download()
Downloading latest changes to project...

Determining which files are out of date...

Changes from the server:
------------------------

  x Modifying file results/stdout.txt
  x Modifying file someData.RData

Starting download of new files:
-------------------------------
    Downloading "someData.RData" ...
      complete!
    Downloading "results/stdout.txt" ...
      complete!
Download complete.

> load("someData.RData") # -> load the R objects
```

## Two-way sync

You can perform a two-way sync with the command:

```r theme={null}
domino.sync()
```

This runs a download followed by an upload. If there is a conflict between the local and server versions of a file, you get both versions so that you can fix the conflict and then re-sync.

## Initialize a Project

To initialize a new project inside an existing directory, go to the directory and then run:

```r theme={null}
domino.init("projectName")
```

This creates a Domino project named `projectName`, with its root at the current working directory. You can then sync this project to the server:

```r theme={null}
domino.sync()
```

## Other commands

You can also run other commands from the [CLI](/cloud/reference/cli/cli-reference) with methods named according to their CLI counterparts. For additional details, see the documentation [hosted on CRAN](https://cran.r-project.org/web/packages/domino/domino.pdf).


# Domino skills reference
Source: https://docs.domino.ai/cloud/reference/skills

Packaged instructions that let a coding assistant carry out Domino platform tasks, preinstalled in the Domino Standard Environment and published in the public domino-claude-plugin repository.

A Domino skill is a packaged set of instructions that teaches a coding assistant how to carry out a specific Domino platform task, such as submitting a Job, registering a model, or deploying an App. You describe the task and the assistant selects the matching skill, so you work in terms of the capability rather than the API.

Skills are optimized for Claude Code. Copilot and Codex run in Domino as general-purpose coding agents, but they do not reliably pick up Domino-specific skills.

## What the plugin ships

Skills are published in the public [`domino-claude-plugin`](https://github.com/dominodatalab/domino-claude-plugin) repository, alongside several other Claude Code components:

| Component | What it provides |
| :- | :- |
| Skills | Roughly 23 skills covering Jobs, Flows, Datasets, environments, model endpoints, monitoring, governance, tracing, and the Domino SDKs. |
| Slash commands | `/domino-app-init`, `/domino-debug-proxy`, `/domino-experiment-setup`, and `/domino-trace-setup`. Invoked by name, unlike skills. |
| Agent definitions | Preconfigured agents for debugging, deployment, and Project setup. |
| Output styles | Alternative response styles for MLOps and for learning Domino. |
| Example hooks | Sample Claude Code hooks. Nothing is active until you copy one into your own configuration. |
| Bundled MCP server | A Model Context Protocol (MCP) server that runs Domino Jobs, checks Job status and results, and syncs files with DFS-based Projects. |

The repository is public, carries an MIT license, and accepts pull requests. See [Contribute a skill](#contribute-a-skill).

## Preinstalled in Workspaces

The Domino Standard Environment (DSE) ships Claude Code, Codex, and GitHub Copilot, together with the Domino Claude plugin and its skills, so a Workspace needs no setup:

| Agent | Loads Domino skills |
| :- | :- |
| Claude Code | Yes |
| Codex | Not by default |
| GitHub Copilot | Not by default |

Codex and Copilot handle general coding work in a Workspace, and [Troubleshoot coding assistants](/cloud/platform-capabilities/features/coding-assistants/troubleshoot) records the current limitation on Domino-specific skills for both. Which agents are available in which Workspace IDE is covered by [Use AI coding assistants](/cloud/platform-capabilities/features/coding-assistants). To authenticate any of the three once and reuse the credential in every Workspace, see [Reuse credentials](/cloud/platform-capabilities/features/coding-assistants/reuse-credentials).

Skills load when a coding assistant starts inside a Domino Workspace, so there is nothing to install or configure. Describe what you want done:

```
Train a model on the diabetes dataset and track the experiment.
```

<Note>
  Plugin skills do not appear in the `/skills` list. Claude Code activates them from the context of your request and reports the loaded set in its startup message.
</Note>

For the catalog of skills the DSE provides, see [the skills catalog](/cloud/platform-capabilities/features/coding-assistants/domino-skills).

## Use the skills elsewhere

Each skill is a directory under `skills/` holding a `SKILL.md` file and, where a skill carries more detail than one file should, supporting Markdown files beside it. That content is plain Markdown, so it travels to places Claude Code's plugin mechanism does not reach:

| Component | Outside Claude Code |
| :- | :- |
| Skills, meaning `SKILL.md` and its supporting Markdown | Portable. Any agent that reads instruction files from your Project or home directory can consume the text as written. |
| Bundled MCP server | Portable. It is a standalone Python program that speaks the Model Context Protocol, so any MCP client can run it. The `.mcp.json` file that registers it is a Claude Code convention. |
| Marketplace and plugin installation | Claude Code only. |
| Slash commands, agent definitions, output styles, and hooks | Claude Code only. Each one depends on a Claude Code feature that has no cross-agent equivalent. |

Portable is not automatic. Nothing loads the skill files into Codex, Copilot, or a third-party agent for you, and the mechanism that matches your request to a skill belongs to Claude Code rather than to the Markdown, so another agent needs the file placed where it already looks for instructions.

To standardize a team on a different agent, pin agent versions, or bundle extra command-line tools and MCP servers into an image, build a [custom Compute Environment](/cloud/platform-capabilities/features/coding-assistants/custom-environments). Clone the plugin repository in the environment's Dockerfile instructions to put the skill files in every Workspace that uses the environment, and pin a commit when you need the set to stay fixed.

## Contribute a skill

Anyone can propose a skill by opening a pull request against [`domino-claude-plugin`](https://github.com/dominodatalab/domino-claude-plugin). `CONTRIBUTING.md` in the repository is the authority on the process, which runs as follows.

<Steps>
  <Step title="Fork the repository and branch">
    Fork the repository, clone your fork, and create a feature branch.
  </Step>

  <Step title="Create the skill directory">
    Add a directory under `skills/` and write a `SKILL.md` file in it. The file opens with a YAML block carrying `name` and a `description` that states what the skill does and includes the keywords that should trigger it. Keep `SKILL.md` under 500 lines and move the detail into supporting files in the same directory.
  </Step>

  <Step title="Register the skill">
    Add the skill to `plugin.json`.
  </Step>

  <Step title="Follow the skill authoring standards">
    `CONTRIBUTING.md` sets five rules that reviewers enforce:

    * Fetch a short-lived bearer token from `http://localhost:8899/access-token`. Never use `DOMINO_USER_API_KEY`, not even as a fallback, because Domino API keys are deprecated.
    * Read hosts and identifiers from the environment variables Domino injects, such as `DOMINO_API_HOST`, `DOMINO_REMOTE_FILE_SYSTEM_HOSTPORT`, `DOMINO_PROJECT_ID`, and `DOMINO_RUN_ID`, rather than writing a hostname or a placeholder into an example.
    * Show REST calls instead of the `python-domino` SDK. The `domino-data-sdk` and `python-sdk` skills are the exceptions, since they document the SDKs.
    * Verify every endpoint against the cluster's own API spec at `$DOMINO_API_HOST/assets/public-api.json` before you document it.
    * Run each documented payload against a live deployment and confirm a 2xx response.
  </Step>

  <Step title="Test the plugin locally">
    Load your branch with `claude --plugin-dir /path/to/domino-claude-plugin`, confirm the skill triggers, and check that every file and link it references resolves.
  </Step>

  <Step title="Open the pull request">
    Update the repository's `README.md` when you add a feature, record your smoke-test results in the pull request description, and request review from the maintainers.
  </Step>
</Steps>

## Install the plugin yourself

Install the plugin directly when you run a coding assistant outside a Domino Workspace, when you need to pin a version, or when you build a [custom Compute Environment](/cloud/platform-capabilities/features/coding-assistants/custom-environments) that standardizes on different tooling.

Prerequisites:

* Claude Code 1.0.33 or later.
* The [`uv`](https://github.com/astral-sh/uv) package manager, for the bundled MCP server.
* `DOMINO_API_KEY` and `DOMINO_HOST` set in your shell. Outside a Workspace, the MCP server has no other way to reach your deployment.

Use the marketplace install unless you are evaluating the plugin or developing against it.

<AccordionGroup>
  <Accordion title="Marketplace, recommended">
    Registers the plugin through Claude Code's marketplace so it survives across sessions.

    <Steps>
      <Step title="Clone the repository into a marketplace directory">
        Run the following:

        ```bash theme={null}
        git clone https://github.com/dominodatalab/domino-claude-plugin.git
        mkdir -p ~/.claude/marketplaces/domino/.claude-plugin
        mkdir -p ~/.claude/marketplaces/domino/plugins
        mv domino-claude-plugin ~/.claude/marketplaces/domino/plugins/domino-claude-plugin
        ```
      </Step>

      <Step title="Write the marketplace manifest">
        Save this as `~/.claude/marketplaces/domino/.claude-plugin/marketplace.json`. The `owner` object is required, and `source` must be a string path rather than a nested object:

        ```json theme={null}
        {
          "name": "domino-marketplace",
          "owner": { "name": "Domino Data Lab", "email": "support@dominodatalab.com" },
          "plugins": [
            {
              "name": "domino-claude-plugin",
              "description": "Domino Data Lab plugin for Claude Code",
              "version": "1.0.0",
              "source": "./plugins/domino-claude-plugin",
              "category": "development"
            }
          ]
        }
        ```
      </Step>

      <Step title="Register the marketplace and install the plugin">
        From inside Claude Code, run the following. Use an absolute path, because `~` may not expand:

        ```
        /plugin marketplace add /home/your-username/.claude/marketplaces/domino
        /plugin install domino-claude-plugin@domino-marketplace
        ```
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="Plugin directory, for evaluation and development">
    Loads the plugin from a cloned directory without registering a marketplace:

    ```bash theme={null}
    git clone https://github.com/dominodatalab/domino-claude-plugin.git
    claude --plugin-dir ./domino-claude-plugin
    ```

    To keep it loaded, alias the command:

    ```bash theme={null}
    echo 'alias claude="claude --plugin-dir /path/to/domino-claude-plugin"' >> ~/.bashrc
    ```
  </Accordion>

  <Accordion title="Project settings, to share with a team">
    Declare the marketplace and enable the plugin in the repository's `.claude/settings.json`. The `path` is the marketplace directory you created, not the marketplace name:

    ```json theme={null}
    {
      "extraKnownMarketplaces": {
        "domino-marketplace": {
          "source": {
            "source": "directory",
            "path": "/home/your-username/.claude/marketplaces/domino"
          }
        }
      },
      "enabledPlugins": { "domino-claude-plugin@domino-marketplace": true }
    }
    ```

    Claude Code prompts each team member to install the marketplace and plugin once they trust the repository folder.
  </Accordion>
</AccordionGroup>

Restart Claude Code after installing, then confirm the plugin appears under the **Installed** tab of `/plugin`.

To update, pull the latest commit and restart Claude Code. After a marketplace install the plugin lives at `~/.claude/marketplaces/domino/plugins/domino-claude-plugin`, not where you first cloned it. After a `--plugin-dir` install, pull in the cloned directory.

## Next steps

* [Get started with coding assistants](/cloud/platform-capabilities/features/coding-assistants/get-started): launch a Workspace and authenticate your agent.
* [Skills catalog](/cloud/platform-capabilities/features/coding-assistants/domino-skills): the skills available in the DSE.
* [Modeling assistant walkthrough](/cloud/platform-capabilities/features/coding-assistants/modeling-assistant): an end-to-end example of a skill doing real work.
* [Custom environments](/cloud/platform-capabilities/features/coding-assistants/custom-environments): pin versions, add tools, or standardize on a different agent.
* [Troubleshoot coding assistants](/cloud/platform-capabilities/features/coding-assistants/troubleshoot): known issues, including skill availability for agents other than Claude Code.


# Adding a new Git Repository to your project using Personal Access Token (PAT)
Source: https://docs.domino.ai/knowledge-base/adding-a-new-git-repository-to-your-project-using-personal-access-token-pat



#### **Version/Environment (if relevant):**

Domino Versions 5.x and later

## **What:**

Connecting a new Git repository to projects using a Personal Access Token (PAT) generated from any Git service account.

## **How:**

To link a Git repository to your Domino project using a Personal Access Token (PAT), follow these steps:

## Creating credentials

If you are adding a private repository, want to write commits to remote, or are using SSH, you will need to add Git credentials to your Domino account. Domino will use these credentials to authenticate with the service hosting your repository when you start a [run](https://docs.dominodatalab.com/en/latest/user_guide/af97b7/create-and-run-jobs/).

### Personal Access Token creation

You will need a personal access token to access a private repository via HTTPS. You will need a personal access token if the URI you want to use to interact with a repository is formatted as:

`https://<domain>/<user>/<repository>.git`

Personal Access Tokens are supported by the following Git services.

* [GitHub Personal Access Tokens](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/)
* [GitLab Personal Access Tokens](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html)

The personal access token you generate needs to have read and write access to your private repositories.

To generate a personal access token in your **GitHub**

1. Navigate to your Git account settings, then Developer Settings. Click the Personal access tokens menu, then click Generate new token.
2. Select repo as the scope. The token will be applicable for all the specified actions in your repositories.
3. Click Generate Token.

To generate a personal access token in your **GitLab**

1. From your home page, open your user settings, and then select Personal access tokens.
2. And then select + New Token.
3. Name your token, select the organization where you want to use the token, and then choose a lifespan for your token.
4. Select the scopes for this token to authorize for your specific tasks.

After generating a personal access token in your Git service, use these steps to add it to Domino:

1. In Domino, click your username in the upper right, then click **Account Settings**.
2. Scroll down to the panel labeled *Git Credentials*, then click **Add a New Credential**.
3. In the *Domain field*, enter the exact domain of the service hosting your repositories, such as github.com, **gitlab.com**, or **your-internal-gitlab-url.com**.
4. For *Authentication Credential Type*, click to select **Personal Access Token**.
5. Enter your personal access token, then click **Add Credential**.
6. You should now see your credential listed in the Git *Credentials* panel. You can also delete it from this panel if desired.

## [Add a repository to a project.](https://docs.dominodatalab.com/en/latest/user_guide/314004/import-git-repositories/)

1. Open the project you want to add a repository to, then click **Files** from the left navigation bar.
2. Click to open the **Git Repositories** tab, then click **Add a New Repository**.
3. Enter an optional directory name and the HTTPS or SSH URI of the repository you want to add. The directory name will be the directory in that `/repos` this repository clones into. It defaults to the name of the repository.
4. Use the dropdown menu to choose which branch of the repository you want Domino to check out when it clones this repository into a run or workspace. If you leave this setting at *Use default branch*, Domino will check out the branch specified as default by your Git service, typically `master`. You can also specify a different branch name, tag name, or commit ID, or supply a custom Git ref.
5. Click **Add Repository**.

**You're All Set!**

## **Notes/Information:**

### **References/Internal Records:**

[https://docs.dominodatalab.com/en/latest/user\_guide/314004/import-git-repositories/](https://docs.dominodatalab.com/en/latest/user_guide/314004/import-git-repositories/)


# Administration
Source: https://docs.domino.ai/knowledge-base/administration-overview

Account and user management tasks for admins.

Account and user management tasks for admins.

## User Management

* [Updating Your Domino Account Details](/knowledge-base/changing-domino-account-information)


# App URLs and URL parameters
Source: https://docs.domino.ai/knowledge-base/app-urls-and-url-parameters

For Domino versions < 6.1.0: This article provides an overview of the different URLs you can use to access your Web App, as well as some options if you'd like to append parameters in your URL.

**NOTE:** The [release of Domino 6.1.0](https://docs.dominodatalab.com/release_notes/6-1-0/#_domino_apps_enhancements) included a significant update and overhaul of the Domino Apps feature. This article is mainly written for Domino Apps on versions previous to Domino 6.1.0 and may not be applicable to Domino 6.1.0 and newer.

This article provides an overview of the different URLs you can use to access your Web App, as well as some options if you'd like to append parameters in your URL.

### App URLs

There are essentially 3 different URLs you can use to access an App:

1. The official app URL:

```
http://<your-domino-url>/modelproducts/<app_id>
```

This URL will not ever change for an app. Going to this URL in a browser gives you your app with a Domino banner at the top.

2. The "static" internal app URL:

```
http://<your-domino-url>/u/<user>/<project>/app
```

Like the first URL, this URL will not change if you stop and start the app, but it will change if you rename the project or change ownership, since it has the project owner and project name in the URL.

3. The current session URL:

```
http://<your-domino-url>/<user>/<project>/r/notebookSession/<session_id>
```

This URL will change every time you start and stop the app.

### Using parameters in URLs

What if you want to use a parameter in your URL? For example, what if you want your landing page to be

```
<app url>/page.html?key1=value1?
```

Currently one of the three URL options will allow URL parameters. To summarize:

1. The official app URL will not ever change for an app. Currently, you can't append a parameter to it, as it just gets ignored/thrown out.

2. The "static" internal app UR: will not change if you stop and start the app, but it will change if you rename the project or change ownership. Currently, you can't append a parameter to it, as it just gets ignored/thrown out. Going to this URL in a browser immediately redirects to the next URL, your app without the Domino banner:

3. The current session URL will change every time you start and stop the app. You can append parameters or paths to other pages and the app will parse them.

### Example App with URL parameters

As we see above, there is a URL we can use for our App where we can append URL parameters. Great! But this URL is going to change any time the App is stopped/restarted. One way to address this is to make your initial app page a landing page, which can add your parameters before going to your main app page, or, if appropriate, prompt your app user for the parameters.

Let's look at a couple of examples! For both Apps, we will use the same app.sh and app.py:

**app.sh:**

```
#!/usr/bin/env bash
python ./app.py
```

**app.py:**

```
import http.server
import socketserver
PORT = 8888
Handler = http.server.SimpleHTTPRequestHandler
httpd = socketserver.TCPServer(("", PORT), Handler)
print(f'serving at port {PORT}')
httpd.serve_forever()
```

Both Apps will also use the same main App page. This page simply greets the user by name:

**appPage.html:**

```
<html>
<head>
 <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.6.0/jquery.min.js"></script>
</head>

<body onload="greetings()">
 <h1>Welcome to the App!</h1>
 <p id="greet"></p>
 <script>
  function greetings() {
   let urlParams = new URLSearchParams(window.location.search);
   let username = urlParams.get('name');
   let timeperiod = urlParams.get('period');
   if (username != null && timeperiod != null) {
    document.getElementById("greet").innerHTML =
    "Hello " + username + "! How are you to" + timeperiod + "?";
   }
  }
 </script>

</body>
</html>
```

**Example App #1**

For our first example, we'll just pass coded variables as URL parameters.

**Index.html:**

```
<html>
<head>
 <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.6.0/jquery.min.js"></script>
</head>

<body>
 <script>
  let person = "Domino";
  const d = new Date();
  let theHour = d.getHours();
  if(4<=theHour<=20){
   var period= "day";
  }
  else{
   period = "night";
  }
  location.href="appPage.html?name="+person+"&period="+period;
 </script>

</body>
</html>
```

With this example, we can use any of the URLs to get to our main App.

The official app URL (http\://`<your-domino-url>`/modelproducts/`<app\_id>`) will automatically go to appPage.html.

The "static" internal app URL (http\://`<your-domino-url>`/u/`<user>`/`<project>`/app) will automatically redirect to the current session URL and append the parameters: http\://`<your-domino-url>`/`<user>`/`<project>`/r/notebookSession/`<session\_id>`appPage.html?name=Domino\&period=day

**Example App #2**

For our second example, we'll actually prompt the user for some input.

**Index.html:**

```
<html>

<head>
 <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.6.0/jquery.min.js"></script>
</head>

<body onload="myFunction()">
 <script>
  function myFunction() {
   const d = new Date();
   let theHour = d.getHours();
   if(4<=theHour<=20){
    var period= "day";
   }
   else{
    period = "night";
   }

   let person = prompt("Please enter your name", "");
   if (person != null) {
    location.href="appPage.html?name="+person+"&period="+period;
   }
  }
 </script>
</body>
</html>
```

The end result of this is the same (we can use any of the URLs to get to our main App). The only difference is, this App is going to stop and prompt the user for input before moving on to our appPage.html.

Once the user adds their input - which will become a URL parameter - the official app URL (http\://`<your-domino-url>`/modelproducts/`<app\_id>`) will automatically go to appPage.html.

The "static" internal app URL (http\://`<your-domino-url>`/u/`<user>`/`<project>`/app) will automatically redirect to the current session URL and append the parameters: http\://`<your-domino-url>`/`<user>`/`<project>`/r/notebookSession/`<session\_id>`appPage.html?name=Domino\&period=day

### Summary

The examples provided above are incredibly basic, but give you an idea of some of the ways you can utilise URL parameters for your Apps!

\*Note: There is a feature request to allow users to append paths and query parameters to published apps URLs without needing alternatives such as the options described above. This request is currently tracked under DOM-26880.


# Build error for nvidia GPG key
Source: https://docs.domino.ai/knowledge-base/build-error-for-nvidia-gpg-key



**Version/Environment (if relevant):**

This issue affects Ubuntu-based environments (Ubuntu 16.04, 18.04, 20.04, 22.04) that use NVIDIA CUDA repositories in Dockerfiles for custom compute environments. This is relevant to all Domino versions where users build custom environments with NVIDIA/CUDA dependencies.

**Issue:**

Build failures occur during `apt-get update` commands with GPG signature verification errors. Common symptoms include:

```
Reading package lists... Done
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
```

**Root Cause:**

NVIDIA's GPG signing key for their CUDA repositories expired, causing package managers to reject repository updates for security reasons. The old GPG key (7fa2af80) is no longer valid and must be replaced with the updated keyring package.

**Resolution:**

Install the new NVIDIA CUDA keyring package in your Dockerfile before any `apt-get update` operations. Add the following commands early in your Dockerfile:

```
# Fix NVIDIA GPG key - place before any apt-get update commands
# Remove old repository files if they exist (optional cleanup)
RUN sudo rm /etc/apt/sources.list.d/cuda.list || echo "cuda.list not found, skipping"
RUN sudo rm /etc/apt/sources.list.d/nvidia-ml.list || echo "nvidia-ml.list not found, skipping"

# Install new NVIDIA keyring (adjust Ubuntu version as needed)
# For Ubuntu 18.04:
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb && \
    sudo apt-key del 7fa2af80 && \
    sudo dpkg -i cuda-keyring_1.0-1_all.deb

# For Ubuntu 20.04, use:
# RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb && \
#     sudo apt-key del 7fa2af80 && \
#     sudo dpkg -i cuda-keyring_1.0-1_all.deb

# For Ubuntu 22.04, use:
# RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.0-1_all.deb && \
#     sudo apt-key del 7fa2af80 && \
#     sudo dpkg -i cuda-keyring_1.0-1_all.deb
```

**Notes/Information:**

* **Ubuntu Version Identification**: Check your Ubuntu version in a workspace terminal using: `lsb_release -a`
* **URL Pattern**: Update the URL to match your Ubuntu version: `ubuntu1804` (18.04), `ubuntu2004` (20.04), `ubuntu2204` (22.04)
* **Execution Order**: These commands must run before any other `apt-get update` or package installation operations
* **GPG/PGP/SBT Fixes**: If you have other GPG-related fixes in your Dockerfile, place the NVIDIA fix before them
* **Error Handling**: The `|| echo` statements prevent build failures if repository list files don't exist
* **Alternative Approach**: For environments that don't require specific CUDA versions, consider using Domino's pre-built GPU environments from quay.io/repository/domino/

**References/Internal Records:**

For the most current NVIDIA guidance, refer to: [https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/](https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/)


# Can Not Connect to KeyServer Caused by GPG Error
Source: https://docs.domino.ai/knowledge-base/can-not-connect-to-keyserver-caused-by-gpg-error

When attempting to refresh a CRAN GPG key during a Domino compute environment build, the build may fail if the environment cannot reach the keyserver. This article explains why this happens and how ow to resolve it without requiring keyserver connectivity

**Affected Versions:** All Domino versions building Ubuntu-based compute environments

## Issue

A compute environment build fails with a keyserver connection timeout similar to the following:

```
gpg: keyserver receive failed: Connection timed out

The command 'gpg --keyserver hkp://keyserver.ubuntu.com:80 \
  --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9 \
  && gpg -a --export E298A3A825C0D65DFD57CBB651716619E084DAB9 \
  | sudo apt-key add -' returned a non-zero code: 2
```

This typically occurs in environments where outbound access to keyserver infrastructure is restricted — for example, on-premises deployments, GovCloud environments, or deployments with strict egress ACLs.

## Root Cause

The build is attempting to fetch the CRAN repository signing key from the Ubuntu keyserver (keyserver.ubuntu.com) using the HKP protocol (port 11371 or port 80). Egress rules or firewall policies in the deployment environment block this outbound connection, causing the GPG keyserver lookup to time out and the build to fail.

**Deprecated:** The gpg … | apt-key add - pattern used in older Dockerfile instructions is deprecated as of Ubuntu 20.10 and removed in Ubuntu 24.04+. Even where it still functions, it should not be used in new or updated compute environments.

## Resolution

The recommended fix is to **bypass the keyserver entirely** by fetching the CRAN signing key directly from CRAN's own HTTPS endpoint at cloud.r-project.org. This avoids any dependency on HKP keyserver access and uses the modern gpg --dearmor approach.

Add the following to your compute environment's **Dockerfile Instructions**:

```
# Step 1 — Install prerequisites (if not already present)
RUN apt-get update && apt-get install -y --no-install-recommends \
    curl \
    ca-certificates \
    gnupg \
    && rm -rf /var/lib/apt/lists/*

# Step 2 — Fetch the CRAN signing key directly from CRAN (no keyserver needed)
RUN curl -fsSL https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc \
    | gpg --dearmor -o /usr/share/keyrings/cran_ubuntu_key.gpg

# Step 3 — Verify the key fingerprint (optional but recommended)
# Expected: E298 A3A8 25C0 D65D FD57 CBB6 5171 6619 E084 DAB9  [expires: 2027-09-30]
RUN gpg --show-keys /usr/share/keyrings/cran_ubuntu_key.gpg
```

**Tip:** This approach works in any environment that allows outbound HTTPS to cloud.r-project.org — no keyserver port access required.

## Fully Air-Gapped Environments

If your environment cannot reach any external endpoint during the build, the key must be pre-staged:

1. On a machine with internet access, download the key:

   ```
   curl -fsSL https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc \
     -o marutter_pubkey.asc
   ```
2. Upload the .asc file to your Domino project files.
3. Reference the key file in your Dockerfile Instructions using the Domino project file access pattern, or embed the key as a base64-encoded string in your Dockerfile.

For detailed guidance on accessing project files during environment builds, see:  [Access Project Files During Docker Build](https://support.domino.ai/support/s/article/Access-project-files-during-docker-build-using-Dockerfile-Instructions-1718868028462)

## Legacy Proxy Workaround (Not Recommended)

An older workaround involved routing keyserver traffic through an HTTP proxy:

```
# NOT RECOMMENDED — apt-key is deprecated on Ubuntu 22.04+
RUN apt-key adv \
  --keyserver-options http-proxy=http://PROXY_URL:PORT/ \
  --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9
```

**Not Recommended:** This approach is no longer recommended because:

* apt-key is deprecated since Ubuntu 20.10 and removed in Ubuntu 24.04+
* It still requires keyserver connectivity even with a proxy
* The direct CRAN HTTPS fetch described above is simpler and more reliable


# Updating Your Domino Account Details
Source: https://docs.domino.ai/knowledge-base/changing-domino-account-information



## **Version/Environment (if relevant):**

All Domino Version

## **What:**

The user intended to modify the Domino account information to verify that the listed users are correct.

## **How:**

To update your account information in your Domino deployment, follow these steps:

Sign in to your Domino deployment.

Navigate to Account Settings.

Select Manage Login Profile.

Update the desired fields (e.g., username, email, first name, or last name).

Click Change to save your updates.

**You're all set!**

## **Notes/Information:**

### **References/Internal Records:**


# Clearing Git Cache
Source: https://docs.domino.ai/knowledge-base/clearing-git-cache



There are security measures set on each deployment. They are placed in such a way they can perform a scan on each users script. When users sync codes to linked repository (github , gitlab etc) , the script maybe tagged as\*\*"sensitive information"\*\*and will be denied to the push request.

Even after completely wiping the script and starting over with the code to avoid this error, they are still unable to push to the github, gitlab etc. This would tell us there are still remnants of the code initially scanned by the organizations security measures .

We can address this by simply removing git cache.

The easiest way to clear your Git cache is to use the *git rm* command with the --*cached* option.

We can also use :

```
> git rm -r --cached .
> git add .
> git commit -m 'git cache cleared'
> git push
```


# Creating and Publishing React Apps
Source: https://docs.domino.ai/knowledge-base/creating-and-publishing-react-apps

Walks through creating a React app in a Domino Workspace with Create React App, configuring app.sh to set the dynamic app URL, publishing it as a Domino App, and managing node_modules sync via .dominoignore.

### ***Version/Environment (if relevant):***

Applies to creating a React app in a Domino Workspace and deploying it as a Domino App. This example uses [Create React App](https://create-react-app.dev/), though an existing React app can also be uploaded directly to a project instead.

## **Resolution:**

### Create the app structure

If you're using a [Domino Standard Environment](https://docs.dominodatalab.com/en/latest/user_guide/0d73c6/domino-standard-environments/) on Domino 5.9.x, Node.js and npm may already be included. If not present in your environment, add your preferred version by following the [Node.js installation instructions](https://nodejs.org/en/download/package-manager) (using nvm or fnm is recommended for Linux).

Once Node.js and npm are available, open a Terminal in your Domino Workspace and run the following, substituting your own app name for `domino-react`:

`npx create-react-app domino-react`

![Running npx create-react-app in a Domino Workspace terminal](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr\&ids=068UY00000HroMg\&d=%2Fa%2FUY0000018nnl%2FStZVxFtcPJdPpRU2GYb.8PTMT4bbmjydIPlsbiFxAdQ\&asPdf=false)

You may be prompted to confirm installing the `create-react-app` package — confirm to proceed. Once installed, it will create your React app. On success, you'll see output similar to:

![Successful create-react-app output in terminal](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr\&ids=068UY00000HroMh\&d=%2Fa%2FUY0000018nnm%2FJ2qwecN3RISYbOrmDuAn9W1zzIWBb37Po4HIiJj7l74\&asPdf=false)

This creates the React project's files and folder structure inside your `/mnt` folder:

![React app folder structure in /mnt](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr\&ids=068UY00000HroMi\&d=%2Fa%2FUY0000018nnn%2FFn2SRy4sc8O6u85HMP7NyXtG40zmf5HSxQ6OEYmPMDg\&asPdf=false)

A large number of files will now be available to sync from your Workspace. All of these need to be synced at least once for the app to run — given the file count, this initial sync may take a while.

![Syncing React app files from Workspace](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr\&ids=068UY00000HroMj\&d=%2Fa%2FUY0000018nno%2FfEtpE8Mjjpw.0byOVMNXV.ENjBllEEMT45MV4M_8g70\&asPdf=false)

### Starting the app

Once the React app's folder/file structure has synced from your Workspace to your project, add an `app.sh` to start the app. App URLs are dynamic — they include the run ID — so the app's `homepage` field in `package.json` must be set at runtime. The sample `app.sh` below does this, assuming a React app named `domino-react`:

```
URL_BASE="https://<platform-domino-url>/$DOMINO_PROJECT_OWNER/$DOMINO_PROJECT_NAME/r/notebookSession/$DOMINO_RUN_ID"
HOME_STRING="\"homepage\": \"$URL_BASE\","
cd domino-react/

#Set hostname to the base url for the app

if grep -F '"homepage"' package.json; then
#replace homepage if exists
sudo sed -i "s|\"homepage\":.*|$HOME_STRING|" package.json
else
#add homepage if line doesn't exist
sudo sed -i "4i $HOME_STRING" package.json
fi

#install necessary packages
sudo npm install create-react-app
sudo npm i web-vitals --save-dev
sudo npm install -g serve

#Below lines run a Production Build (does not auto refresh upon changes to src)
sudo npm run build
serve -s build -l 8888
```

This script writes a `homepage` line to `package.json` if one doesn't already exist, or overwrites the existing line with the current run's app URL.

With this `app.sh` in place, publish your App. It will take a few minutes to prepare; once ready, you can view it:

![Published React app running in Domino](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr\&ids=068UY00000HroMk\&d=%2Fa%2FUY0000018nnp%2Fssle5igej16aSbvLu9GcSTFMPJgrGSgW3OTn07MPDMY\&asPdf=false)

### Managing the app's resources

Creating the app with `create-react-app` generates a large number of subfolders and files in `node_modules`. These are required for the app to run, but you may want to exclude them from syncing on future runs or Workspace startups. Add this folder to your `.dominoignore` file to do so:

![Adding node\_modules to .dominoignore](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr\&ids=068UY00000HroMl\&d=%2Fa%2FUY0000018nnq%2F7fcZxYNfL8TRtVrcFrNW_6u6jPl2W2xu.V47AGHqlhs\&asPdf=false)

### **Notes/Information:**

The `sed -i "4i ..."` fallback in `app.sh` assumes the `homepage` line should be inserted at line 4 of `package.json`. This works for the default file layout produced by Create React App, but if your `package.json` has a different structure, adjust the line number accordingly.


# Database Locking
Source: https://docs.domino.ai/knowledge-base/database-locking

Database locks occur when queries or transactions conflict. Use the database-specific fix steps or Linux tools (ps, lsof, strace, fuser) to identify and resolve them.

Version/Environment (if relevant):
Applicable to all databases used within Domino deployments.

## Issue:

A database lock occurs when a query or transaction blocks or stalls due to a lock conflict. This may appear in code execution or log messages, for example:

Raw code

`sqlite3.OperationalError: database is locked`

## Root Cause:

Common causes of database locks include:

* Concurrent data access — multiple processes try to read and write the same data simultaneously, requiring the database to serialize access
* Long-running transactions — a transaction holds locks for an extended period, blocking other processes
* Resource contention — multiple processes compete for limited disk or memory resources
* Deadlocks — two or more processes each wait for the other to release a lock, creating a cycle that never resolves
* Hardware or software failures — locks persist beyond a failure even after the failure itself resolves

## Resolution:

MySQL

Error messages:

Raw code

`Deadlock found when trying to get lock
Lock wait timeout exceeded; try restarting transaction`

Fix steps:

1. Run `SHOW PROCESSLIST` to identify the process holding the lock.

Raw code

`SHOW PROCESSLIST;`

2. Terminate the blocking process using its process ID.

Raw code

`KILL <process_id>;`

3. If an open transaction is causing the lock, roll it back.

Raw code

`ROLLBACK;`

SQL Server

Error message:

Raw code

`Transaction (Process ID) was deadlocked on lock resources with another process
and was chosen as the victim of the deadlock. Restart the transaction.`

Fix steps:

1. Run `sp_who2` to identify the blocking process.

Raw code

`EXEC sp_who2;`

2. Terminate the blocking process using its process ID.

Raw code

`KILL <process_id>;`

3. If an open transaction is causing the lock, roll it back.

Raw code

`ROLLBACK;`

SQLite

Error message:

Raw code

`sqlite3.OperationalError: database is locked`

Fix steps:

1. Commit any open transaction.

Raw code

`conn.commit()`

2. Close the connection to release the lock.

Raw code

`conn.close()`

3. If multiple connections exist, close each one before reopening. Use the `sqlite3` module in Python to manage open connections.

Raw code

`# Example: close all connections before re-querying
for conn in open_connections:
conn.close()`

MongoDB

Error messages:

Raw code

`Error: couldn't execute operation: couldn't execute command.
CommandFailed: Transaction 12 is being rolled back - lock was not available`

Fix steps:

1. If a long-running operation holds the lock, wait for it to finish.
2. If the operation is stuck, terminate it using its operation ID.

Raw code

`db.killOp(<op_id>)`

3. As a last resort, restart the MongoDB server if a bug or system issue is causing the lock.

Linux tooling

Outside the database, use these Linux tools to identify signs of locking:

* ps — lists running processes and shows which processes hold locks
* lsof — lists all open files, including locked ones, and shows which process locks each file
* strace — traces system calls and signals for a process, helping identify the root cause of a lock
* fuser — shows which processes use a specific file or file system, including those holding locks

### Notes/Information:

* Design your schema and queries to minimize contention and reduce the need for locks.
* Experiment with different isolation levels to prevent transactions from conflicting with each other.
* Address locks promptly — persistent locks degrade database performance.

### References/Internal Records:

* Public KB Articles:
  * [Database Locking (KB-000001303)](https://support.domino.ai/support/s/article/Database-Locking-1718868034992)


# Docker Layers - failed to register layer: max depth exceeded
Source: https://docs.domino.ai/knowledge-base/docker-layers-failed-to-register-layer-max-depth-exceeded

Resolve the "failed to register layer: max depth exceeded" Docker error in Domino by reducing image layers via combined RUN commands or multi-stage builds.

## **Version:**

All docker versions

## **Error:**

When attempting to launch a workspace you are presented with this error: "***failed to register layer: max depth exceeded"*** in the logs. The error is also visible on the describe of a new pod (Under Events)

```
Warning Failed 12m (x7 over 26m) kubelet Failed to pull image "100.97.188.95:5000/dominodatalab/environment:63a051030e277d167bec6bfd-35": rpc error: code = Unknown desc = failed to register layer: max depth exceeded
```

## **Cause:**

Docker images consist of stacked layers. Each instruction in a Dockerfile creates one new layer. When using the overlay2 storage driver, Docker enforces a maximum of 127 layers per image.

This error occurs when your image exceeds that limit. Note that the base image your compute environment builds on may already use many layers. Your own Dockerfile instructions stack on top of those existing layers.

## **Resolution:**

Reduce the total number of layers in your Dockerfile. The most effective approaches are combining multiple `RUN` instructions and using multi-stage builds.

### **1. Combine RUN instructions**

Each separate **`RUN`** instruction creates a new layer. Combine related commands into a single `RUN` instruction using **`&&`**. Install all packages in one step:

```
RUN apt-get update && \
    apt-get install -y package1 && \
    apt-get install -y package2 && \
    apt-get install -y package3 && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*
```

### **2. Use multi-stage builds**

If your build process requires multiple tools, use multi-stage builds. Build artifacts in one stage, then copy only what you need into the final image. This avoids carrying intermediate layers into the final image:

```
FROM builder AS build
# Build commands go here

FROM base_image
# Copy artifacts from the build stage
COPY --from=build /path/to/artifacts /app
```

### **3. Create a new compute environment**

If your layer count still exceeds 127 after combining instructions, the base image itself may account for too many layers. In that case, create a new compute environment in Domino and copy your existing Dockerfile instructions into it, confirming the total layer count stays within the 127-layer limit.

For help editing Dockerfile instructions in a Domino compute environment, see [Use Dockerfile instructions](https://docs.dominodatalab.com/en/latest/user_guide/813d89/use-dockerfile-instructions/).

## **Additional resources**

* [Dockerfile best practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) — guidance on reducing layers and optimizing image size
* [Dockerfile reference](https://docs.docker.com/engine/reference/builder/) — full reference for all Dockerfile instructions


# Domino Product Version Lifecycle and Support Policies
Source: https://docs.domino.ai/knowledge-base/domino-product-version-lifecycle-and-support-policies



The following page provides information related to the lifecycle of each Domino product release. Timelines for both Product Patching and Technical Support are covered.

## **Domino Versions: Naming Conventions and Version Types**

Domino's product versions follow a common Semantic Version ([SemVer](https://semver.org/)) numbering scheme, **Ma.Mi.Pa**, where:

* **Ma (Major version)** - Contains new features, fixes, potential platform/technology changes, and may include breaking changes or interface updates that are not backward compatible

* **Mi (Minor version)** - Contains new features, fixes, and potential platform/technology support

* **Pa (Patch version)** - Contains only security updates and critical bug fixes

* **Example:** Version 6.0.2 is on the 6.x major release line, and 6.0.2 is the latest patch for the 6.0 minor version.

## **Domino Release Schedule**

### **Domino Cloud**

Domino Cloud targets weekly updates with new features, bug fixes, and platform/technology support.

### **Domino Enterprise (Self-Hosted)**

Self-hosted releases follow this schedule:

* **Major/Minor releases** - Released roughly every 6 months with new features, fixes, and possible platform/technology support
* **Patch releases** - Released between 30-50 days of the last major/minor/patch release, and includes only bug fixes and security updates (no new features)
* **Patch availability** - Patches are only released for the current minor version. If a bug fix or security issue necessitates a patch release, only supported versions of Domino will be patched.

***

## **End of Patching and End of Support Policies**

These policies apply to all self-hosted Domino versions.

### **Policy Definitions**

**End of Patching (EOP)** After this date, we will not release any more security or bug fix patchesfor a given version.

\*\*End of Life (EOL)\*\*After this date, Domino Support will no longer accept tickets, provide proactive monitoring, or answer technical questions about this version. For support on EOL versions of Domino, customers can purchase extended support.

***

## **Support Timelines**

### **End of Patching (EOP)**

**LATEST MAJOR RELEASE:**

* Security patches are provided between 30-50 days of release for the current minor version of the latest major release.

**PREVIOUS MAJOR RELEASE:**

* Domino will only patch previous major releases as needed AND
* Patches will only be released for supported versions of Domino (see EOL policy for supported Domino versions)

***

### **End of Life (EOL)**

**ALL VERSIONS:**

* **12 months** after the next major or minor version is released

**Examples:**

* Version 5.10.x reaches EOL 12 months after 5.11 is released
* Version 5.11.x reaches EOL 12 months after 6.0 is released
* Version 6.0.x reaches EOL 12 months after 6.1 is released


# Environment build fails with "The method driver /usr/lib/apt/methods/https could not be found"
Source: https://docs.domino.ai/knowledge-base/environment-build-fails-with-the-method-driver-usr-lib-apt-methods-https-could-not-be-found

Fix a compute environment build failure caused by missing apt-transport-https by adding a RUN apt-get install command to your Dockerfile before other apt commands.

The following error in a compute environment build

"*The method driver /usr/lib/apt/methods/https could not be found.*"

indicates that the "apt-transport-https" package is not installed. To correct this in your compute environment add the following line to your docker definition in the compute image prior to any other apt commands.

```
RUN apt-get install apt-transport-https -y
```


# ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied:
Source: https://docs.domino.ai/knowledge-base/error-could-not-install-packages-due-to-an-environmenterror-errno-13-permission-denied



Often times, users customize their environments to tailor their analytic needs. Package installation on the docker file is the most common path a user takes.

But , during the package installation process users sometimes encounter an error :

```
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied:
```

followed by the package name. This usually indicates you lack the necessary permissions to continue the install.

Luckily fixing the error can be simple. Just try installing the package by adding '**--user**' option.

```
 ! pip install (package) --user
```


# VSCode Error: ENOSPC: System limit for number of file watchers reached
Source: https://docs.domino.ai/knowledge-base/error-enospc-system-limit-for-number-of-file-watchers-reached

VSCode workspaces fail with "ENOSPC: System limit for number of file watchers reached" when large directories exhaust inotify handles. Fix by excluding folders like .venv from files.watcherExclude in VSCode settings, or increase fs.inotify.max_user_watches.

### **Version:**

All supported versions of Domino using VSCode as the workspace IDE.

## **Issue:**

An error prompt appears in a VSCode workspace with the following message:

```
Error: ENOSPC: System limit for number of file watchers reached
```

This error is also visible in the workspace support bundle logs:

```
[File Watcher (node.js)] Failed to watch /home/ubuntu/.local/share/code-server/Machine/settings.json
for changes using fs.watch() (Error: ENOSPC: System limit for number of file watchers reached, watch
'/home/ubuntu/.local/share/code-server/Machine/settings.json')
```

## **Root Cause:**

VSCode uses the Linux `inotify` subsystem to watch files for changes. Each file or directory being watched consumes one inotify handle. When a workspace contains a large number of files — for example, due to a Python virtual environment (`.venv`), `node_modules`, or a large project directory — VSCode exhausts the system's available file watcher handles, triggering this error.

## **Resolution:**

There are two approaches to resolve this issue. Try **Option 1** first before increasing system limits.

**Option 1 – Exclude large directories from file watching (Recommended)**
Configure VSCode to exclude large folders from the file watcher by adding them to the `files.watcherExclude` setting. This reduces the number of handles consumed without changing system limits.

Open your VSCode settings and add the following, expanding the list to include any additional large directories in your workspace (e.g., `.venv`):

```
"files.watcherExclude": {
  "**/.git/objects/**": true,
  "**/.git/subtree-cache/**": true,
  "**/node_modules/*/**": true,
  "**/.hg/store/**": true,
  "**/.venv/**": true
}
```

This setting can be applied at two levels:

* **User Settings** — applies globally to all VSCode instances for the user.
* **Workspace Settings** — stored in the workspace and applies only when that workspace is open.

To edit settings via the UI, first enable the split JSON editor by searching for and enabling the following setting:

```
workbench.settings.useSplitJSON
```

This opens a split pane view with a read-only reference on the left and a writeable editor on the right. Make your changes in the right pane and save.

**Option 2 – Increase the system inotify file watcher limit**
If excluding directories does not resolve the issue, a Domino Admin can increase the maximum number of file watchers at the OS level. Run the following commands on the node hosting the workspace:

```
# Set the new value (maximum is 524288)
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

# Verify the new value was applied
cat /proc/sys/fs/inotify/max_user_watches
```

The maximum supported value is **524288**. Set the value appropriate to the size of the workspaces on the deployment.

### **Notes/Information:**

Option 1 is preferred as it resolves the issue without requiring system-level changes or admin access. Common high-file-count directories to exclude include `.venv`, `node_modules`, and `.git/objects`.

For more information on this error in VSCode, see the official VSCode documentation: [Visual Studio Code – Unable to watch for file changes (ENOSPC)](https://code.visualstudio.com/docs/setup/linux#_visual-studio-code-is-unable-to-watch-for-file-changes-in-this-large-workspace-error-enospc).


# Errors during model build : Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/prometheus-client/
Source: https://docs.domino.ai/knowledge-base/errors-during-model-build-failed-to-establish-a-new-connection-errno-2-name-or-service-not-known-simple-prometheus-client



#### ***Version/Environment (if relevant):***

4.x, 5.x

## **Issue:**

When deploying a Model API the build is very slow and the build logs list repeating network related errors like:

```
Jan 23 2023 18:14:08 +0100 #13 4.807 WARNING: Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f081026bd90>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/prometheus-client/
```

The model eventually deploys, but the build process is slow.

## **Root Cause:**

Model API initialization code tries to install the python package prometheus-client 0.11.0 (among others), a client for metrics & monitoring tools. If your system is air-gapped or public/internet access is tightly controlled your system might not be able to fetch that python package from [https://pypi.org](https://pypi.org), resulting in the error above.

## **Resolution:**

Install prometheus-client in your project's Compute Environment via a dockerfile instruction:

**RUN pip install prometheus-client==0.11.0**

If this isn't possible another idea is to modify pip's config files to make it not retry and "fail fast", like:

```
pip config --quiet --user set global.retries 0
pip config --quiet --user set global.timeout 1
pip config --quiet --user set global.disable-pip-version-check True
```

## **Notes/Information:**

If the above isn't possible please reach out to Domino Technical Support referencing DOM #36045 and DOM #41738


# failed to pull image 602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5
Source: https://docs.domino.ai/knowledge-base/failed-to-pull-image-602401143452-dkr-ecr-us-east-1-amazonaws-com-eks-pause-3-5



#### ***Version/Environment:***

This applies to deployments hosted in Amazon AWS.

## **Issue:**

A customer's pod was failing to transition to a running state, it's describe showed this event:

```
Warning FailedCreatePodSandBox 7s (x15 over 3m11s) kubelet Failed to create pod sandbox: rpc error: code= Unknown desc = failed to get sandbox image "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5": failed to pull image "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5": failed to pull and unpack image "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5": failed to resolve reference "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5": pulling from host 602401143452.dkr.ecr.us-east-1.amazonaws.com failed with status code [manifests 3.5]: 401 Unauthorized
```

The issue impacted their DEV environment, but not their PROD environment and in some ways didn't seem 100% reproducible. The customer found that this problem did NOT occur when a new NODE was spun-up to host the pod.

## **Root Cause:**

Container image pause is regularly used by kubernetes as a temporary placeholder for future containers during pod creation. Due to a problem in EKS there is a broken process in which a garbage collector is pruning images on nodes and removing the pause container image that is initially added as part of the node provisioning. Apparently it is no longer possible to fetch that "pause" from the default referenced location also due to a change on the EKS side. The EKS bug is partly discussed here: [https://github.com/aws/amazon-vpc-cni-k8s/issues/2030#issuecomment-1372204463](https://github.com/aws/amazon-vpc-cni-k8s/issues/2030#issuecomment-1372204463)

The support for Container Runtime Interface (CRI) for Docker (Dockershim) has been discontinued with the introduction of Kubernetes 1.24 in EKS. Official Amazon EKS Amazon Machine Images (AMIs) now use containerd as the runtime exclusively.

The issue occurs when kubelet, a key component of Kubernetes, seeks to garbage collect these "pause" containers. While Kubernetes offers credentials for obtaining pod images, containerd does not use these credentials when requesting the sandbox image. As a result, users who attempt to pull these images may experience authentication issues due to a lack of saved credentials in containerd. This drawback necessitates the re-deployment of nodes.

To address this, kubelet offers the '--pod-infra-container-image' flag which is specific to the EKS "pause" image. This flag has a two fold purpose: it guarantees that the image is not pruned by the image garbage collector and so helps prevent the occurrence of such issues. Ordinarily, kubelet should not undertake the garbage collection of the "pause" container. If this occurs, it may potentially result from various underlying causes, such as the "pause" being garbage collected due to disk pressure (evidenced by commands like ('df -h /var/lib..').

For further insights and discussions related to these challenges and solutions, you may refer to the following resources:

* Kubernetes GitHub issue discussing pause container image: [https://github.com/kubernetes/kubernetes/issues/81756](https://github.com/kubernetes/kubernetes/issues/81756)
* Rancher GitHub issue regarding Kubernetes and pause container: [https://github.com/rancher/rke2/issues/1830](https://github.com/rancher/rke2/issues/1830)
* Kubernetes GitHub issue about kubelet's behavior and pause container: [https://github.com/kubernetes/kubernetes/issues/62388](https://github.com/kubernetes/kubernetes/issues/62388)

## **Resolution:**

If you are witnessing this problem you can attempt to

* Ensure the nodes have policy AmazonEC2ContainerRegistryReadOnly, if not add it. The following can check.

  ```
  aws iam list-entities-for-policy --policy-arn "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly" --entity-filter Role --query 'PolicyRoles[].RoleName'
  ```
* You can also try the bootstrap args listed in the link above.
* Check the root volume is sufficiently sized (#df -h | column -t) , also check that AMI versions used by the nodes is recent and compatible with EKS 1.24
* Check the config.toml on these nodes looks good, especially for sandbox\_image.

```
root@ip-10-0-86-99 ~]# cd /etc/containerd/
[root@ip-10-0-86-99 containerd]# ls
config.toml
[root@ip-10-0-86-99 containerd]# cat config.toml
version = 2
root = "/var/lib/containerd"
state = "/run/containerd"

[grpc]
address = "/run/containerd/containerd.sock"

[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
discard_unpacked_layers = true

[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pause:3.5"
```

* Check kubelet entry for --pod-infra-container-image

```
root   17634 2.3 0.3 2817932 119732 ?   Ssl 23:11  0:07
/usr/bin/kubelet --config /etc/kubernetes/kubelet/kubelet-config.json
--kubeconfig /var/lib/kubelet/kubeconfig --container-runtime-endpoint
unix:///run/containerd/containerd.sock --image-credential-provider-config
/etc/eks/image-credential-provider/config.json --image-credential-provider-bin-dir
/etc/eks/image-credential-provider --node-ip=10.0.86.99 --pod-infra-container-image=602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pause:3.5 --v=2 --cloud-provider=aws --container-runtime=remote --node-labels lifecycle=OnDemand --node-labels=dominodatalab.com/node-pool=default,dominodatalab.com/domino-node=true
```

* If urgent you can cordon the broken node and cycle it out with a newer node.
* If the above does not work you can also attempt to cordon the node, restart kubelet on the effected node and uncordon the node:

```
 On the kubernetes environment, cordon the node
```

```
#kubectl cordon <node name>
```

```
Logon and run on the node itself
```

```
#sudo systemctl restart kubelet && sudo systemctl status kubelet
```

```
On the kubernetes environment, uncordon the node
```

```
#kubectl uncordon <node name>
```

* If the above does not help, you should contact AWS Support.

**Caveat:**

In Domino 5.4.x. An issue has been identified under DOM-44673, where the Image-cache-agent is pruning the pause image from EKS nodes in ECR. This has been fixed under Domino 5.5. If you identify this to be the issue, then there is the option of disabling the Image-cache-agent, doing this however will have a negative impact on performance since the agent will no longer be able to prune cached images on compute nodes.

To disable:
This command patches the image-cache-agent daemonset in the domino-platform namespace by adding a nodeSelector that will not match any nodes.

```
kubectl -n domino-platform patch daemonset image-cache-agent -p '{"spec": {"template": {"spec": {"nodeSelector": {"non-existing": "true"}}}}}'
```

To re-enable:

This command patches the image-cache-agent daemonset in the domino-platform namespace to remove the nodeSelector that was added in the previous example.

```
kubectl -n domino-platform patch daemonset image-cache-agent --type json -p=[{"op": "remove", "path": "/spec/template/spec/nodeSelector/non-existing"}]
```


# FailedMount event "efs.csi.aws.com not found in the list of registered CSI drivers"
Source: https://docs.domino.ai/knowledge-base/failedmount-event-efs-csi-aws-com-not-found-in-the-list-of-registered-csi-drivers

Troubleshoot the FailedMount "efs.csi.aws.com not found in registered CSI drivers" warning on EKS pods, including how to verify driver installation with kubectl.

## Issue

Admins may see the following warning when describing a pod:

```
Warning FailedMount 55s (x2 over 1m2s) kubelet MountVolume.MountDevice failed for volume "domino-shared-store-domino-compute" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name efs.csi.aws.com not found in the list of registered CSI drivers
```

## Root cause

This warning typically appears only in Elastic Kubernetes Service (EKS) clusters. It's usually temporary — the Container Storage Interface (CSI) driver is still exposing the file system to the node's pods.

If the `FailedMount` warnings stop and the volume mounts successfully, the issue is transient and requires no action.

## Resolution

If `FailedMount` warnings persist and the pod does not mount successfully, follow the steps below.

### Step 1: Confirm the EFS CSI driver is registered

Run the following command to check whether the `efs.csi.aws.com` driver appears in the list of registered CSI drivers:

```
kubectl get csidrivers.storage.k8s.io
```

Verify that `efs.csi.aws.com` appears in the output.

If the driver is not listed, the node may not have been provisioned correctly. Review your EKS node provisioning configuration to confirm the EFS CSI add-on was enabled during cluster or node group setup.

### Step 2: Inspect CSI driver pods

If `efs.csi.aws.com` is listed but the mount failure continues, inspect the CSI-related pods across all namespaces:

```
kubectl get po -A | grep csi
```

For any pods returned, review their descriptions and logs. Include the `kube-system` namespace in your search. Check both the CSI controller pods and the CSI node pod assigned to the same node as your failing pod.

If you need help, contact Technical Support and include your screenshots, pod descriptions, and logs.


# FastAPI App with Multiple Endpoints
Source: https://docs.domino.ai/knowledge-base/fastapi-app-with-multiple-endpoints



**Version/Environment (if relevant):** Domino Version 6.1 and Above

## **What:**

Domino's [Simple FastAPI example](https://support.domino.ai/support/s/article/Simple-FastAPI-example-as-a-Published-App-1718868040509) is a great way to get started with FastAPI in Domino, but what if you're ready to go a step further and set up a FastAPI app to take multiple GET and POST requests?

This guide assumes you've already installed FastAPI in your environment or set it up to install via *requirements.txt*.

## **How:**

## app.sh

Like many of our other App examples, this simply calls the Python script:

```
python app.py
```

## Python script

First, we need to import our packages and collect some environment variables:

```
from fastapi import FastAPI
from pydantic import BaseModel
import uvicorn
import os

runID = os.environ['DOMINO_RUN_ID']
userID = os.environ['DOMINO_PROJECT_OWNER']
projectID = os.environ['DOMINO_PROJECT_NAME']
```

Domino utilizes a reverse proxy, and App URLs have path prefixes that need to be included (e.g. `/username/app-name/app`). Therefore, we need to set a root\_path for our FastAPI app (see [https://fastapi.tiangolo.com/advanced/behind-a-proxy/#setting-the-root\_path-in-the-fastapi-app](https://fastapi.tiangolo.com/advanced/behind-a-proxy/#setting-the-root_path-in-the-fastapi-app)). With that example in mind, I was able to get to the swagger UI by modifying our code example as follows:

```
path_prefix = f"/{userID}/{projectID}/r/notebookSession/{runID}"

app = FastAPI(root_path=path_prefix)
```

We can then add routes as desired for our API:

```
class Item(BaseModel):
  name: str
  description: str
  price: float

@app.get("/")
async def root():
  return {"message": "Hello World"}

@app.get("/items/{item_id}")
async def read_item(item_id):
  return {"item_id": item_id}

@app.post("/items/")
async def create_item(item: Item):
  return item
```

Lastly, run the app:

```
if __name__ == "__main__":
  uvicorn.run("app:app",host="0.0.0.0",port=8888,reload = True)
```

Putting it all together, the complete app.py looks like:

```
from fastapi import FastAPI
from pydantic import BaseModel
import uvicorn
import os

runID = os.environ['DOMINO_RUN_ID']
userID = os.environ['DOMINO_PROJECT_OWNER']
projectID = os.environ['DOMINO_PROJECT_NAME']

path_prefix = f"/{userID}/{projectID}/r/notebookSession/{runID}"

print(f"App URL is https://<domino_url>/{path_prefix}")

app = FastAPI(root_path=path_prefix)

class Item(BaseModel):
  name: str
  description: str
  price: float

@app.get("/")
async def root():
  return {"message": "Hello World"}

@app.get("/items/{item_id}")
async def read_item(item_id):
  return {"item_id": item_id}

@app.post("/items/")
async def create_item(item: Item):
  return item

if __name__ == "__main__":
  uvicorn.run("app:app",host="0.0.0.0",port=8888,reload = True)
```

Similar to our Simple FastAPI example, hitting "View App" will bring up your FastAPI app with our "Hello World" message.

While our root page is accessible via the "official" app URL, you aren't able to append paths to this. For this, you need to use the session URL: `http://<your-domino-url>/<user>/<project>/r/notebookSession/<session_id>` (see [App URLS and URL parameters](https://support.domino.ai/support/s/article/App-URLs-and-URL-parameters-1718868036753)). To find this easily, you might add a print statement to your app.py, such as:

```
print(f"App URL is https://<domino_url>/{path_prefix}")
```

...which would print in your User Output when you start your App.

With that URL, you can utilize your other endpoints. For example:

`http://<your-domino-url>/<userID>/<projectID>/r/notebookSession/<runID>/item/foo`

You can also view the Swagger UI for your FastAPI app at:

`http://<your-domino-url>/<userID>/<projectID>/r/notebookSession/<runID>/docs`

**You're all set.**

## **Notes/Information:**

[https://support.domino.ai/support/s/global-search/fastAPI](https://support.domino.ai/support/s/global-search/fastAPI)

### **References/Internal Records:**


# Getting started with Streamlit in Domino
Source: https://docs.domino.ai/knowledge-base/getting-started-with-streamlit-in-domino

Install Streamlit in a Compute Environment, build your app in /mnt, live-test via Jupyter Server Proxy, then publish using an app.sh that binds to port 8888 on 0.0.0.0 with headless mode enabled.

### **Version/Environment (if relevant):**

All versions. Requires a Compute Environment with Streamlit installed. Domino Standard Environments are recommended.

## **How To:**

### **Step 1 — Install Streamlit**

If you are using a [Domino Standard Environment](https://docs.dominodatalab.com/en/latest/user_guide/0d73c6/domino-standard-environments/), add the Streamlit package to your Compute Environment via one of the following methods:

**Dockerfile instruction:**

```
USER root
RUN pip install -U streamlit starlette
```

**requirements.txt:**

`streamlit
starlette`
**Note:** Some older versions of streamlit and starlette have compatibility issues, make sure you are using the latest versions.

### **Step 2 — Build your Streamlit app**

Create your Streamlit app file (e.g. `streamlit_app.py`) in your project's `/mnt` directory. The example below is a working demo adapted from [Streamlit's NYC Uber Ridesharing demo](https://github.com/streamlit/demo-uber-nyc-pickups) (updated for Streamlit 1.3.8):

```
import os
import altair as alt
import numpy as np
import pandas as pd
import pydeck as pdk
import streamlit as st

st.set_page_config(layout="wide", page_title="NYC Ridesharing Demo", page_icon=":taxi:")

@st.cache_resource
def load_data():
    path = "uber-raw-data-sep14.csv.gz"
    if not os.path.isfile(path):
        path = f"https://github.com/streamlit/demo-uber-nyc-pickups/raw/main/{path}"
    data = pd.read_csv(
        path,
        nrows=100000,
        names=["date/time", "lat", "lon"],
        skiprows=1,
        usecols=[0, 1, 2],
        parse_dates=["date/time"],
    )
    return data

def map(data, lat, lon, zoom):
    st.write(pdk.Deck(
        map_style="mapbox://styles/mapbox/light-v9",
        initial_view_state={"latitude": lat, "longitude": lon, "zoom": zoom, "pitch": 50},
        layers=[pdk.Layer("HexagonLayer", data=data, get_position=["lon", "lat"],
            radius=100, elevation_scale=4, elevation_range=[0, 1000],
            pickable=True, extruded=True)],
    ))

@st.cache_data
def filterdata(df, hour_selected):
    return df[df["date/time"].dt.hour == hour_selected]

@st.cache_data
def mpoint(lat, lon):
    return (np.average(lat), np.average(lon))

@st.cache_data
def histdata(df, hr):
    filtered = df[(df["date/time"].dt.hour >= hr) & (df["date/time"].dt.hour < (hr + 1))]
    hist = np.histogram(filtered["date/time"].dt.minute, bins=60, range=(0, 60))[0]
    return pd.DataFrame({"minute": range(60), "pickups": hist})

data = load_data()
row1_1, row1_2 = st.columns((2, 3))

if not st.session_state.get("url_synced", False):
    try:
        pickup_hour = int(st.query_params["pickup_hour"][0])
        st.session_state["pickup_hour"] = pickup_hour
        st.session_state["url_synced"] = True
    except KeyError:
        pass

def update_query_params():
    st.query_params["pickup_hour"] = st.session_state["pickup_hour"]

with row1_1:
    st.title("NYC Uber Ridesharing Data")
    hour_selected = st.slider("Select hour of pickup", 0, 23,
        key="pickup_hour", on_change=update_query_params)

with row1_2:
    st.write("""
## Examining how Uber pickups vary over time in New York City
    and at its major regional airports.""")

row2_1, row2_2, row2_3, row2_4 = st.columns((2, 1, 1, 1))
la_guardia = [40.7900, -73.8700]
jfk = [40.6650, -73.7821]
newark = [40.7090, -74.1805]
zoom_level = 12
midpoint = mpoint(data["lat"], data["lon"])

with row2_1:
    st.write(f"**All NYC from {hour_selected}:00 to {(hour_selected + 1) % 24}:00**")
    map(filterdata(data, hour_selected), midpoint[0], midpoint[1], 11)
with row2_2:
    st.write("**La Guardia Airport**")
    map(filterdata(data, hour_selected), la_guardia[0], la_guardia[1], zoom_level)
with row2_3:
    st.write("**JFK Airport**")
    map(filterdata(data, hour_selected), jfk[0], jfk[1], zoom_level)
with row2_4:
    st.write("**Newark Airport**")
    map(filterdata(data, hour_selected), newark[0], newark[1], zoom_level)

chart_data = histdata(data, hour_selected)
st.write(f"**Rides per minute between {hour_selected}:00 and {(hour_selected + 1) % 24}:00**")
st.altair_chart(
    alt.Chart(chart_data).mark_area(interpolate="step-after")
    .encode(x=alt.X("minute:Q", scale=alt.Scale(nice=False)),
            y=alt.Y("pickups:Q"), tooltip=["minute", "pickups"])
    .configure_mark(opacity=0.2, color="red"),
    use_container_width=True,
)
```

### **Step 3 — Live test from a workspace**

### JupyterLab

You can live test your Streamlit app from a JupyterLab workspace using Jupyter Server Proxy. Ensure that [JupyterLab](https://jupyterlab.readthedocs.io/en/stable/getting_started/installation.html#pip) and [jupyter-server-proxy](https://jupyter-server-proxy.readthedocs.io/en/latest/install.html) are installed in your Compute Environment. Both are pre-installed in Domino Standard Environments.

1. Launch a JupyterLab workspace and open a new terminal.
2. Run your app from the `/mnt` directory:

   ```
   streamlit run streamlit_app.py --server.port 8501
   ```
3. Access the app in your browser using the following URL pattern (default Streamlit port is `8501`):

   ```
   https://<your-domino-url>/<username>/<project-name>/notebookSession/<run-id>/proxy/8501/
   ```

**Note:** The trailing `/` at the end of the URL is required. Omitting it will prevent the app from loading. The `<run-id>` can be found in the URL of the active workspace session.

### VSCode

Use the same process to start the app as above, but for live testing from a **VSCode workspace**, the URL structure differs — insert `/r/` before `notebookSession`:
ex.
`https://<your-domino-url>/<username>/<project-name>/r/notebookSession/<run-id>/proxy/8501/`

### **Step 4 — Publish the app in Domino**

Once development is complete, publish the app using an `app.sh` file. Create `app.sh` in the root of your project with the following contents:

```
#!/bin/bash
set -x
streamlit run streamlit_app.py \
    --server.port 8888 \
    --server.address 0.0.0.0 \
    --server.headless true

streamlit run streamlit_app.py
```

This file configures Streamlit to bind to the host and port Domino expects for published Apps. Once saved:

1. Navigate to **Deployment > Apps & Agents** in your project.
2. Click **Publish** and select **App**
3. Complete the form and click **Publish Domino App**.
4. Once the App is running, click **View App** to open it.

**Notes/Information:**

* See: [Live testing Streamlit in Domino on VSCode workspace](https://support.domino.ai/support/s/article/Live-testing-the-Streamlit-in-Domino-on-VSCode-workspace-1718868026350) (KB 000001036).
* For full Domino App publishing documentation, see: [Publish Apps — Domino Docs](https://docs.dominodatalab.com/en/latest/user_guide/71635d/publish-apps/).


# How to connect and authenticate to Gitlab & GitHub with ssh keys using the native bash terminal
Source: https://docs.domino.ai/knowledge-base/how-to-connect-and-authenticate-to-gitlab-github-with-ssh-keys-using-the-native-bash-terminal

Set up SSH keys for GitLab and GitHub on a Linux bash terminal, test the connection, clone a repo, and push/pull changes

Use these steps to set up a Secure Shell (SSH) key using the native Linux bash terminal, test the connection, clone a repository, and perform basic Git operations like pushing and pulling changes.

## Step 1: Install package and generate SSH key

Run the following commands to install Git and generate an SSH key:

```
sudo apt install git
ssh-keygen -t ed25519 -C "your_email@test.com"
```

Follow the prompts to save the key (typically to `~/.ssh/id_ed25519`) and optionally add a passphrase.

## Step 2: Start the SSH agent and add key to the agent

```
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519
```

## Step 3: Copy the public SSH key to GitLab

Run the following command to print your public key:

```
cat ~/.ssh/id_ed25519.pub
```

Copy the full output, then add it to GitLab:

1. Go to your GitLab profile settings.
2. Navigate to **SSH Keys**.
3. Paste your SSH key into the **Key** field.
4. Add a descriptive title in the **Title** field.
5. Click **Add key**.

## Step 4: Test the connection

```
ssh -T git@gitlab.com
```

A successful connection returns:

```
Welcome to GitLab, @your_username!
```

## Step 5: Clone a repository, commit, push, and pull

Clone the repository and create a test file:

```
git clone git@gitlab.com:username/repo.git
cd repo
echo "Hello, GitLab!" > testfile.txt
git add testfile.txt
git commit -m "Add a test file"
```

**Error → Fix:** If Git prompts you to set up a global identity, run the following commands, then retry the commit:

```
git config --global user.name "Your Name"
git config --global user.email "you@test.com"
```

### Push changes

```
git push origin main
```

### Pull changes

```
git pull origin main
```

> **Note:** Replace `main` with the appropriate branch name if it differs.

## References

* [Adding a new SSH key to your GitHub account](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account)
* [Add an SSH key to your GitLab account](https://docs.gitlab.com/ee/user/ssh.html#add-an-ssh-key-to-your-gitlab-account)


# How-to guides
Source: https://docs.domino.ai/knowledge-base/how-to-guides-overview

Step-by-step guides for apps, environments, CLI tools, and Git integration.

Step-by-step guides for apps, environments, CLI tools, and Git integration.

## Apps

* [FastAPI App with Multiple Endpoints](/knowledge-base/fastapi-app-with-multiple-endpoints)
* [Getting started with Streamlit in Domino](/knowledge-base/getting-started-with-streamlit-in-domino)
* [Use Gradio in a Jupyter workspace](/knowledge-base/use-gradio-in-a-jupyter-workspace)
* [Creating and Publishing React Apps](/knowledge-base/creating-and-publishing-react-apps)
* [Increasing the timeout for Shiny Server](/knowledge-base/increasing-the-timeout-for-shiny-server)
* [App URLs and URL parameters](/knowledge-base/app-urls-and-url-parameters)

## CLI

* [The lsof utility - 'list open files'](/knowledge-base/the-lsof-utility-list-open-files)
* [htop: Interactive process viewer, a brief overview](/knowledge-base/htop-interactive-process-viewer-a-brief-overview)

## Environments

* [Installing Archived R Packages No Longer Available on CRAN](/knowledge-base/installing-r-packages-from-archive)
* [Installing Packages And Dependencies](/knowledge-base/installing-packages-and-dependencies)
* [Configuring and Sending Emails in Domino Using Python smtplib](/knowledge-base/sending-emails-python-smtplib)

## Git Integration

* [Adding a new Git Repository using a Personal Access Token (PAT)](/knowledge-base/adding-a-new-git-repository-to-your-project-using-personal-access-token-pat)
* [How to connect and authenticate to GitLab & GitHub with SSH keys](/knowledge-base/how-to-connect-and-authenticate-to-gitlab-github-with-ssh-keys-using-the-native-bash-terminal)

## Infrastructure and Kubernetes

* [Resolve internal (on-premise) DNS Server in AWS EKS via CoreDNS](/knowledge-base/resolve-internal-on-premise-dns-server-in-aws-eks-via-coredns)

## Jobs

* [Preventing Jupyter timeout errors when running notebooks as a batch job](/knowledge-base/preventing-jupyter-timeout-errors-when-running-notebooks-as-a-batch-scheduled-job)


# How to prevent the permission denied error when running "pip install" in a workspace
Source: https://docs.domino.ai/knowledge-base/how-to-prevent-the-permission-denied-error-when-running-pip-install-in-a-workspace



If you are attempting to install or upgrade packages using "pip install" in a workspace, then you may receive a permission denied error.

For example:

```
 !pip install --upgrade scikit-learn

OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/scikit-learn'
```

It's possible to install/upgrade packages in a workspace session, but you need to include the --user flag to avoid the permissions error.

For example:

```
!pip install --upgrade --user scikit-learn
```

However, this installation will not persist from run to run. If you consistently need this library upgrade, you can install this in a custom environment by following the instructions [here](https://support.domino.ai/support/s/article/Installing-Packages-And-Dependencies-1723748824009).


# How to Troubleshoot Kubernetes Autoscaling (ASG) Issues in Domino
Source: https://docs.domino.ai/knowledge-base/how-to-troubleshoot-autoscaling-asg-issues

Pods fail to schedule due to autoscaling issues such as ASG misconfiguration, AZ mismatches, insufficient resources, or node label conflicts. Check execution.log, pod events, and ASG settings to diagnose.

### **Version/Environment (if relevant):**

All versions of Domino; applies to AWS-based deployments using Kubernetes Cluster Autoscaler and AutoScaling Groups (ASGs)

## **Issue:**

Executions fail to start or pods fail to schedule due to autoscaling issues. Common symptoms include pods stuck in Pending state, scale-up not triggering, or volume node affinity conflicts. Errors appear in the support bundle's execution.log or events.json, for example:

```
2023-06-26T15:06:39.204Z: pod didn't trigger scale-up:2node(s) had volume node affinity conflict,26node(s) didn't match Pod's node affinity/selector
Please check if the hardware tier'p3-Large-GPU'has a valid nodePool
```

```
"resourceKind":"Pod",
"reason":"NotTriggerScaleUp",
"description":"pod didn't trigger scale-up: 26 node(s) didn't match Pod's node affinity/selector, 2 node(s) had volume node affinity conflict",
```

## **Root Cause:**

Autoscaling failures in Domino are typically caused by one or more of the following:

* Autoscaling is not enabled in the cluster yaml
* Pod resource requests (CPU/memory) exceed what any node in the group can provide
* Mismatch between ASG configuration in AWS and the Kubernetes autoscaler deployment settings
* ASG configured across multiple Availability Zones, causing volume node affinity conflicts
* Node labels not matching the Hardware Tier's node selector or affinity rules
* ASG max node group size has been reached

## **Resolution:**

**1. Verify autoscaling is enabled in the cluster yaml:**

```
autoscaler:
 enabled: true
```

**2. Describe the unscheduled pod and inspect its Events section:**

Common event messages and their meanings:

* **Autoscaler intentionally not triggering scale-up** — pods request more CPU/memory than any node can provide:

```
Normal NotTriggerScaleUp 2m24s (x18 over 27m) cluster-autoscaler pod didn't trigger scale-up (it wouldn't fit if a new node is added): 45 Insufficient memory, 3 max node group size reached, 45 Insufficient cpu
```

* **Autoscaler not triggering when it should** — mismatch between AWS ASGs and Kubernetes settings:

```
Warning FailedScheduling <unknown> default-scheduler 0/12 nodes are available: 1 Insufficient memory, 1 node(s) didn't match node selector, 10 Insufficient cpu.
```

* **Volume node affinity conflict** — ASG spans multiple AZs that don't match storage volume AZs:

```
pod didn't trigger scale-up (it wouldn't fit if a new node is added): 1 node(s) had volume node affinity conflict
```

**3. Verify and reconcile ASG settings between AWS and Kubernetes:**

```
kubectl get configmap cluster-autoscaler-status -n <install namespace> -o yaml
kubectl edit deployment -n <platform namespace> <cluster autoscaler deployment name>
```

Ensure each ASG has only one Availability Zone, both in the AWS console and in the configmap. Storage volume AZs must match ASG instance AZs.

**4. If scale-up is intentionally not triggering due to insufficient resources, reduce pod resource requests** via Hardware Tier settings. Refer to: [https://admin.dominodatalab.com/en/4.4.1/compute/hardware-tiers.html?highlight=hardware%20tier](https://admin.dominodatalab.com/en/4.4.1/compute/hardware-tiers.html?highlight=hardware%20tier)

**5. Review min, max, and currently running nodes:**

```
kubectl describe cm -n <could be platform namespace, could be something like kube-system> cluster-autoscaler-status
```

### **Notes/Information:**

Other less common scheduling messages and their meanings:

* `2 node(s) didn't match pod affinity rules, 2 node(s) didn't match pod affinity/anti-affinity rules` — Pod labels or affinity rules don't match the available nodes. Describe the pod and check the affinity section.
* `11 node(s) didn't match Pod's node affinity` or `4 node(s) didn't match node selector` — Nodes exist but don't carry the label required by the Hardware Tier's Node Selector (e.g., dominodatalab.com/node-pool=xlarge-memory). Check pod affinity, Node Selector, and node labels.
* `3 max node group size reached` — The ASG's maximum node count in AWS has been reached. Increase the ASG maxSize to allow additional nodes to spin up.

If the Kubernetes autoscaler deployment is edited directly, also update the autoscaler section of the cluster yaml (deploy manifest) to persist changes across upgrades and maintenance.


# How to use the apt-get install command in environments
Source: https://docs.domino.ai/knowledge-base/how-to-use-the-apt-get-install-command-in-environments

Fix the apt-get install non-zero exit code error in environment builds by adding the -y flag to run in non-interactive mode.

# How to use apt-get install in environments

When building an environment, running apt-get install without the -y flag causes the following error:

```
The command '/bin/sh -c sudo apt-get install <package>' returned a non-zero code: 100
ERROR: ProgressMessage{id=null, status=null, stream=null,
error=The command '/bin/sh -c sudo apt-get install <package>' returned a non-zero code: 100, progress=null,
progressDetail=null}
```

The error message suggests a connectivity issue, but the actual cause is the missing -y confirmation switch. Without -y, apt-get waits for interactive input during the install prompt, which causes the non-zero exit code.

## Fix: add the -y flag to apt-get install

Add the -y confirmation switch to your command so apt-get runs in non-interactive mode:

```
USER root
RUN apt-get install -y <package>
```


# htop – Interactive process viewer: a brief overview
Source: https://docs.domino.ai/knowledge-base/htop-interactive-process-viewer-a-brief-overview

Understand and use htop's interface sections, color coding, process columns, and installation steps for Linux system monitoring.

htop is a terminal-based system monitoring tool that displays process statistics and resource usage in a color-coded, human-readable format. It complements the standard `top` utility by adding mouse support, direct process termination, and improved readability. Use htop to monitor running processes and their full command lines to identify the source of system load.

## **Installation**

htop is available in most major distributions' standard repositories.

On any Debian-based distribution:
`sudo apt-get install htop -y`

On any Red Hat-based distribution:
`sudo yum install htop -y`

## **Run htop**

To launch htop, type the following command and press **Enter**:
`htop`

## **The user interface explained**

The htop interface contains four sections:

* **Section 1** — Interactive bar gauges for processors and memory, which you can customize
* **Section 2** — A quick overview of running tasks, load average, and uptime
* **Section 3** — A detailed view of running processes
* **Section 4** — A list of common keyboard shortcuts, which you can also click with a mouse

## **Section 1: Processors and memory**

### **CPU utilization**

This section displays CPU and memory resources. The number of bars indicates the number of CPU cores — a 4-core server shows bars labeled 0 through 3, a 5-core server shows five bars, and so on. Each bar represents one CPU core. The percentage on the right side of each bar shows current CPU utilization.

htop uses color coding to categorize CPU utilization by process type:

* **Blue** — Low-priority processes
* **Green** — User processes
* **Red** — System/kernel processes
* **Yellow/Orange** — Hardware Interrupt Request (IRQ) time
* **Magenta** — Soft IRQ time
* **Grey** — I/O wait time
* **Cyan** — Steal time

In the example above, the bars are mostly empty, indicating the CPU is largely idle. Sustained high CPU usage — not occasional spikes — warrants investigation, as it affects overall server performance.

### **RAM and swap utilization**

This section shows current memory usage. RAM is system memory; swap is a portion of disk storage the server uses as overflow memory. In the example, only 383 MB of RAM is in use.

htop uses the following colors for memory:

* **Green** — Used memory pages
* **Blue** — Buffer pages
* **Yellow** — Cache pages

When RAM is exhausted, the system falls back to swap space. High usage of both RAM and swap degrades the server's ability to handle tasks effectively. In the example, swap shows 0K/0K, indicating no swap space is configured.

## **Section 2: Processes and load average**

This section summarizes running tasks. In the example, 33 total processes are present, with 1 currently running. The value labeled thr (18 in the example) represents the number of kernel threads.

Load average (also known as the processor queue length) shows the average system load over the past 1, 5, and 15 minutes — displayed as three values such as `0.14`, `0.07`, and `0.04`. To assess whether load is problematic, apply the formula `4 × CPU cores`. If the load average exceeds this threshold, investigate further. Sustained high CPU load is a stronger indicator of a problem than high CPU usage alone. Tools such as `sysstat` can collect and analyze these metrics over time.

This section also displays **server uptime** — the total time the server has been running since its last restart.

## **Section 3: Process information**

When you notice high resource utilization, review the process list to identify the source. Per the `htop` man page, the default columns are:

* **PID** — The process ID
* **USER** — The process owner's username, or user ID if the name can't be determined
* **PRI** — The kernel's internal priority for the process, usually the NI value plus 20; differs for real-time processes
* **NI** — The nice value, from 19 (low priority) to -20 (high priority); only root can lower this value
* **VIRT** — Total program size in memory
* **RES** — Resident set size: the text and data sections plus stack usage
* **SHR** — Size of the process's shared pages
* **S** — Process state:
  * **S** — Sleeping (idle)
  * **R** — Running
  * **D** — Disk sleep (uninterruptible)
  * **Z** — Zombie (waiting for parent to read its exit status)
  * **T** — Traced or suspended (for example, by SIGTSTP)
  * **W** — Paging
* **CPU%** — Percentage of CPU time the process currently uses
* **MEM%** — Percentage of memory the process currently uses, based on its resident memory size
* **TIME+** — Total time the process has spent in user and system time, measured in clock ticks
* **Command** — Full path of the command

## **Section 4: Keyboard shortcuts**

The shortcuts displayed at the bottom of the htop interface are self-explanatory, configurable, and mouse-clickable.

## **Further reading**

For full details on htop's configuration options, flags, and features, consult the man page:
`man htop`

The project's documentation is also available at [https://htop.dev/](https://htop.dev/).


# I cannot install the pywin32 package on Domino using pip install
Source: https://docs.domino.ai/knowledge-base/i-cannot-install-the-pywin32-package-on-domino-using-pip-install

pywin32 cannot be installed on Domino because it is a Windows-only package and Domino workspaces run on Linux. Use Linux-compatible alternatives such as python-xlib, subprocess, psutil, or DB-API libraries depending on your use case.

Version/Environment (if relevant):
All Domino deployments running Linux-based workspaces.

## Issue:

Running `pip install pywin32` fails with the following error:

Raw code

`ERROR: Could not find a version that satisfies the requirement pywin32 (from versions: none)
ERROR: No matching distribution found for pywin32
Note: you may need to restart the kernel to use updated packages.`

## Root Cause:

`pywin32` provides Python bindings exclusively for the Windows API. Domino workspaces run on Linux, where the following Windows-specific dependencies do not exist:

* Windows API — system calls, GUI manipulation, and registry operations
* COM (Component Object Model) objects and Active Directory services
* WMI (Windows Management Instrumentation) components
* DLLs (Dynamic Link Libraries) and other Windows system resources

## Resolution:

Use one of the following Linux-compatible libraries as an alternative. Install the one that matches your use case.

python-xlib — GUI and window management Interact with the Linux X Window System, including windows, events, and input devices.

Raw code

`pip install python-xlib`

subprocess — run command-line tools Run Linux command-line tools and utilities from within your Python script. `subprocess` is part of the Python standard library — no installation required.

Raw code

`import subprocess
result = subprocess.run(['ls', '-l'], capture_output=True, text=True)
print(result.stdout)`

psutil — system and process monitoring Retrieve and monitor system resources such as CPU, memory, disk usage, and network data.

Raw code

`pip install psutil`

DB-API — database connectivity Connect Python to databases using a standardized interface. Use a database-specific library for your platform:

Raw code

\`# PostgreSQL
pip install psycopg2

# MySQL

pip install mysql-connector-python

# Oracle Database

pip install cx\_Oracle\`

For more information about `pywin32`, see the [PyPI project page](https://pypi.org/project/pywin32/).

### Notes/Information:

* Domino Support focuses on Domino's core product. For environment issues, contact your local admins as a first step.

### References/Internal Records:

* Public KB Articles:
  * [I Cannot Install the pywin32 Package on Domino Using pip install (KB-000001194)](https://support.domino.ai/support/s/article/I-cannot-install-the-pywin32-package-on-Domino-using-pip-install-1718868031408)


# 415 Unsupported Media Type Error When Calling the Domino REST API
Source: https://docs.domino.ai/knowledge-base/i-keep-getting-a-415-error-unsupported-media-type-when-i-call-domino-s-rest-api-what-am-i-doing-wrong

A 415 Unsupported Media Type error from the Domino REST API is caused by a missing or incorrect Content-Type header. Add Content-Type: application/json to your request headers to resolve.

### **Version/Environment (if relevant):**

All Domino Versions

## **Issue:**

A user receives a **415 Unsupported Media Type** error when calling the Domino REST API.

## **Root Cause:**

The API request is missing the Content-Type header, or it is set to an unsupported value. The Domino REST API requires requests to specify application/json as the content type.

## **Resolution:**

Add the Content-Type header to your API request:

```
Content-Type': 'application/json'
```

Since the header should also include your API key, the full header section will look like this:

```
headers = {
'Content-Type': 'application/json',
'X-Domino-Api-Key': '<api key>'
}
```

If you continue to experience issues, use [Postman](https://www.postman.com/) to test your API calls. Postman is a free application that allows you to test different media types and headers with minimal overhead.

Additional API examples can be found here: [API Examples for Domino's 5.x REST API](https://support.domino.ai/support/s/article/API-Examples-for-Domino-s-5-x-REST-API-1718868032320).

### **Notes/Information:**


# Increasing the timeout for Shiny Server
Source: https://docs.domino.ai/knowledge-base/increasing-the-timeout-for-shiny-server



When you access a published Shiny app, the app times out after 60 seconds of inactivity by default. Use one of the two methods below to increase this timeout.

## Prerequisites

Before applying the non-Pro workaround, confirm you have:

* A published Shiny app with separate UI and server files
* Access to edit and republish the app in Domino

## Method 1: Shiny Server Pro

If your team has a Shiny Server Pro license, you can configure application timeouts directly within your Domino installation. For configuration details, see the application timeouts section in the Shiny Server Pro documentation at [https://docs.posit.co/shiny-server/#application-timeouts.](https://docs.posit.co/shiny-server/#application-timeouts.)

If you have a license and need assistance setting this up, contact Domino support.

## Method 2: WebSocket keep-alive workaround (non-Pro)

This workaround sends a ping through the WebSocket every 15 seconds to keep the connection open, preventing the default 60-second inactivity timeout.

### Step 1: Add the keep-alive script to your UI file

Add the following code block to your UI file:

```
tags$head(
    HTML(
     "
     <script>
     var socket_timeout_interval
     var n = 0
     $(document).on('shiny:connected', function(event) {
     socket_timeout_interval = setInterval(function(){
     Shiny.onInputChange('count', n++)
     }, 15000)
     });
     $(document).on('shiny:disconnected', function(event) {
     clearInterval(socket_timeout_interval)
     });
     </script>
     "
    )
)
```

### Step 2: Add the output element to your UI file

Anywhere after the script added in Step 1, add the following line to the same UI file:

```
textOutput("keepAlive")
```

### Step 3: Add the render function to your server file

In your server file, add:

```
output$keepAlive <- renderText({
   req(input$count)
   paste("keep alive ", input$count)
 })
```

### Step 4: Verify the changes

Republish your Shiny app and confirm it loads without errors. To verify the new timeout is in effect, leave the app idle for longer than 60 seconds and confirm the session remains active.

## Still timing out?

If the workaround above doesn't resolve the issue, see [Shiny app shows a gray screen after some time or times out](https://support.domino.ai/support/s/article/Shiny-App-shows-a-gray-screen-after-sometime-or-timeouts-1718868032640) .


# Domino Knowledge Base
Source: https://docs.domino.ai/knowledge-base/index

Support articles and troubleshooting guides from the Domino Support team, covering known issues, workarounds, and diagnostic steps.

The Knowledge Base collects support articles written by the Domino Support team: known issues, workarounds, and diagnostic steps that fall outside the product documentation.

<CardGroup>
  <Card title="How-to guides" icon="book-open" href="/knowledge-base/how-to-guides-overview">
    Step-by-step guides for apps, environments, CLI tools, and Git integration.
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/knowledge-base/troubleshooting-overview">
    Fixes for common errors across environments, workspaces, models, and infrastructure.
  </Card>

  <Card title="Administration" icon="shield" href="/knowledge-base/administration-overview">
    Account and user management tasks for admins.
  </Card>

  <Card title="Support Policies" icon="headset" href="/knowledge-base/support-policies-overview">
    What's covered under Domino Support, and what isn't.
  </Card>
</CardGroup>

For everything else, use the [Domino documentation](/knowledge-base/cloud/platform-overview/domino-overview) for product guides and reference, or contact [Domino Support](https://support.domino.ai) directly.


# Installing Packages And Dependencies
Source: https://docs.domino.ai/knowledge-base/installing-packages-and-dependencies



# Installing packages and dependencies

Domino is pre-installed with many common libraries, so we recommend that you first try running your code before customizing your configuration. If you need to, there are specific ways to add your own dependencies for...

* [Python](https://tickets.dominodatalab.com/hc/en-us/articles/28352932497044-Installing-Packages-And-Dependencies#python)
* [R](https://tickets.dominodatalab.com/hc/en-us/articles/28352932497044-Installing-Packages-And-Dependencies#r)

If you have more specialized dependencies, please [let us know](https://support.domino.ai/support/s).

# Persistent installations

By default, Domino runs each of your scripts (or interactive sessions) from a fresh environment. I.e., any packages you install, will install each time. We have more advanced functionality for managing your own compute environments with software changes that stay permanently installed. Learn more about [custom environments](https://docs.dominodatalab.com/en/latest/user_guide/e46d54/customize-environments/).

# Python

## Checking what's already installed

Many common modules are installed by default. To get a list of pre-installed modules, you can include help('modules') at the start of your script or in a new IPython Notebook session. Or using the Domino CLI tool, you can run: domino run --direct "pip freeze"

## Adding your own packages

To specify your own additional dependencies, you can use '[pip](http://www.pip-installer.org/en/latest/)' with Domino. To specify module dependencies for your project, add a '[pip requirements file](https://pip.pypa.io/en/latest/user_guide/#requirements-files)' named 'requirements.txt' to the root of your project folder.

The requirements file specifies which libraries and any version requirements for them. An example:

```
pandas
lxml==3.2.3
numpy>=1.7.1
```

For a full reference on the syntax of the requirements file, read [this](https://pip.pypa.io/en/latest/user_guide/#requirements-files).

If you're using pip on your local machine, the easiest way to generate the 'requirements.txt' file is to run the following command in the root of your project folder:

```
~/domino/myProject $ pip freeze > requirements.txt
```

For performance reasons, you should prune that file so that it includes only the libraries you need for your actual analysis.

Alternatively, if you're working in a Jupyter Notebook, you can also use pip to install dependencies interactively. In a notebook cell, you can run

```
! pip install --user <package>
```

(The '!' tells the notebook to execute the cell as a shell command)

## Installing Packages Hosted from a Git Repository

**Warning:** this is an advanced topic!

Pip can install Python packages from source by cloning a public Git repository over https. For full reference, see [here](https://docs.readthedocs.io/en/stable/guides/private-python-packages.html). To specify this, you will need to add something like the following line to your 'requirements.txt' file:

```
-e git+https://git.yourproject.org/you/Project.git#egg=YourProject
```

The most common host of Git projects is GitHub. If the package you wish to install is publicly accessible, then the instructions above will work. However, if you need to install any private repositories, Domino can securely integrate with GitHub to access those private repositories.

**Never embed your GitHub credentials** directly in the requirements.txt file. Instead, integrate Domino with GitHub securely.

# R

If you're using Domino for R scripts, you may also want to check out our R Package for controlling Domino in your R IDE.

Most common packages are installed by default (you can include "installed.packages()" at the start of your script to print out a list of installed packages). If you need additional packages, you can use R's built-in package manager to install and load them: simply add "install.packages" calls for any packages your code needs to the top of your R scripts, e.g.,

```
install.packages("rpart", dependencies=TRUE, repos='http://cran.us.r-project.org')
# install other packages...

library('rpart')

# rest of your script...
```

These package installation calls must be at the top of **every** R script that Domino runs. If your packages take a long time to install, we recommend that you create a [custom compute environment](https://docs.dominodatalab.com/en/latest/user_guide/e46d54/customize-environments/) for efficiency.


# Installing Archived R Packages No Longer Available on CRAN
Source: https://docs.domino.ai/knowledge-base/installing-r-packages-from-archive

R packages removed from CRAN can be installed from the archive. Download the .tar.gz directly and use install.packages(pkgs=pkgFile, type="source", repos=NULL) to install.

### **Version/Environment (if relevant):**

All Domino versions

## **Issue:**

RStudio CRAN repository has removed a number of packages, which are now only available in the CRAN archive at [https://cran.r-project.org/src/contrib/Archive/](https://cran.r-project.org/src/contrib/Archive/). When attempting to install such a package using the standard `install.packages()` command, R returns a warning that the package is no longer available. For example:

```
> install.packages('RSQLServer')
Installing package into "/usr/local/lib/R/site-library"
(as "lib" is unspecified)
Warning in install.packages :
package "RSQLServer" is not available for this version of R

A version of this package for your version of R might be available elsewhere,
see the ideas at
https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installing-packages
```

## **Root Cause:**

The package has been removed from the active CRAN repository and is no longer returned by the standard `install.packages()` function. The package still exists in the CRAN archive but must be installed directly from the archive URL using a source-based install method.

## **Resolution:**

Install the package directly from the CRAN archive using the following code (replace the URL and filename with the specific package version needed):

```
> url <- "https://cran.r-project.org/src/contrib/Archive/RSQLServer/RSQLServer_0.3.0.tar.gz"
> pkgFile <- "RSQLServer_0.3.0.tar.gz"
> download.file(url = url, destfile = pkgFile)
trying URL 'https://cran.r-project.org/src/contrib/Archive/RSQLServer/RSQLServer_0.3.0.tar.gz'
Content type 'application/x-gzip' length 633887 bytes (619 KB)
==================================================
downloaded 619 KB

> install.packages(pkgs=pkgFile, type="source", repos=NULL)
Installing package into "/usr/local/lib/R/site-library"
(as "lib" is unspecified)
* installing *source* package "RSQLServer" ...
** package "RSQLServer" successfully unpacked and MD5 sums checked
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (RSQLServer)
> library("RSQLServer")
```

### **Notes/Information:**

The full CRAN archive is available at [https://cran.r-project.org/src/contrib/Archive/](https://cran.r-project.org/src/contrib/Archive/). Browse to the package directory to find available versions and their corresponding `.tar.gz` download URLs.


# Invalid Client Credentials When Generating A Service Account Token
Source: https://docs.domino.ai/knowledge-base/invalid-client-credential-when-generating-service-account-token



## Overview

When creating a service account in Domino, you may encounter the following error while generating a token:

```
{"message":"Request unauthorized: {\"error\":\"unauthorized_client\",\"error_description\":\"Invalid client or Invalid client credentials\"}","success":false}
```

This typically indicates a mismatch between the **client secret** stored in Kubernetes and the one configured in **Keycloak** for the `domino-service-accounts` client.

## Symptoms

* Service account creation succeeds, but token generation fails.
* Error response shows `unauthorized_client` or `Invalid client credentials`.
* Nucleus frontend logs display `OidcUnauthorizedException` related to the `domino-service-accounts` client.

```
2024-11-12T09:32:09.048739313-05:00 domino.helpers.oidc.OidcUnauthorizedException: Request unauthorized: {"error":"unauthorized_client","error_description":"Invalid client or Invalid client credentials"}
```

* Issue may appear after making changes to Keycloak configuration (e.g., adding or modifying JWT keys).

## Root Cause

The `domino-service-accounts` client credentials in Keycloak and the Kubernetes secret may become out of sync. Since **Keycloak client secrets cannot be directly edited**, and **Kubernetes secrets are auto-generated**, reconciliation is required to realign both values.

## How to Verify Credentials

1. **Fetch values from the cluster:**

```
# Get clientId
kubectl -n domino-platform get secret nucleus-domino-service-accounts-client -o jsonpath='{.data.clientId}' | base64 -d; echo

# Get clientSecret
kubectl -n domino-platform get secret nucleus-domino-service-accounts-client -o jsonpath='{.data.clientSecret}' | base64 -d; echo
```

* The `clientId` should be `domino-service-accounts`.
* Note down the `clientSecret`.

**2. Check Keycloak configuration:**

* Log in to [Keycloak](https://docs.dominodatalab.com/en/latest/admin_guide/1a7dfa/set-up-keycloak/#tr1).
* Go to: **Domino Realm > Clients > domino-service-accounts > Credentials**.
* Reveal the **Client Secret** and compare it with the one from Kubernetes.

!

If they **do not match**, reconciliation is needed.

## Reconciliation Procedure

Since neither Keycloak nor Kubernetes allows direct editing of these secrets, the recommended approach is to **re-import the `domino-service-accounts` client** into Keycloak:

**1. Export the client configuration:**

* In Keycloak UI, go to:
  **Domino Realm > Clients > domino-service-accounts > Action > Export**.

!

* Save the JSON file.

**2. Update the exported file:**

* Locate the `"secret"` property in the JSON.
* Replace its value with the `clientSecret` obtained from Kubernetes.

```
"secret": "your-cluster-secret-value"
```

!

**3. Delete the old client:**

* In Keycloak UI, navigate to **Domino Realm > Clients > domino-service-accounts > Action > Delete**.

**4. Re-import the updated client:**

* Go to **Domino Realm > Clients > Clients List > Import Client**.
* Upload the modified JSON file.

!

* Click **Save**

**5. Verify the reconciliation:**

* Navigate back to **Credentials** for the `domino-service-accounts` client.
* Confirm that the secret matches the value in Kubernetes.
* Retry generating the service account token.

!

## Important Notes

* **Avoid** **toggling** the **Client Authentication** setting in Keycloak unnecessarily. Doing so may regenerate the secret and cause mismatches.
* If new JWT keys are required (e.g., for S3 connectivity), ensure existing Domino-related client configurations are not modified.
* After reconciliation, service account token generation should work again without impacting other Keycloak configurations.

## Related Documentation

* [Domino Service Accounts Guide](https://docs.dominodatalab.com/en/cloud/admin_guide/6921e5/manage-domino-service-accounts/#_what_are_domino_service_accounts)


# "Invalid host" error when trying to connect MiniO server as a Generic S3 data source
Source: https://docs.domino.ai/knowledge-base/invalid-host-error-when-trying-to-connect-minio-server-as-a-generic-s3-data-source

After a Chrome update, Chromium-based browsers rejected  incomplete SSL certificate chain with a NET::ERR_CERT_COMMON_NAME_INVALID error. Fix involved converting the .p7b chain file to PEM using OpenSSL and re-importing it into AWS Certificate Manager

**Version/Environment (if relevant):**

Applies to all version of Domino after browser update.

## Issue:

After a Chrome update, Chromium-based browsers rejected the SSL certificate chain for the deployment, making it unreachable. The following error was displayed in the browser:

```
NET::ERR_CERT_COMMON_NAME_INVALID
```

## Root Cause:

Per AWS documentation, NET::ERR\_CERT\_COMMON\_NAME\_INVALID on an Application Load Balancer can have the following causes:

* The domain name used in the HTTPS request does not match the alternate name specified in the listener's associated AWS Certificate Manager (ACM) certificate.
* The load balancer's default DNS name is being used — public certificates cannot be issued for the \*.amazonaws.com domain.

This error occurs when the browser cannot verify a website's SSL certificate and is unable to establish a secure connection. It typically results from a misconfigured or incomplete certificate chain on the server.

Per MinIO docs:

* [https://min.io/docs/minio/linux/index.html](https://min.io/docs/minio/linux/index.html):

> While the port 9000 is used for connecting to the API, MinIO automatically redirects browser access to the MinIO Console.

* [https://min.io/docs/minio/linux/administration/minio-console.html#static-vs-dynamic-port-assignment](https://min.io/docs/minio/linux/administration/minio-console.html#static-vs-dynamic-port-assignment)

> respond to S3 API operations on the default MinIO server port :9000 and browser access on the MinIO Console port :9090

## **Resolution:**

In the **Host** field, replace the port number with **9000**:

**Step 1: Extract the private key and certificate body from the .pfx file**

If you need to obtain the private key and certificate body from a .pfx file, run the following commands:

```
openssl pkcs12 -in domino.p12 -nocerts -nodes -out privatekey.pem
openssl pkcs12 -in domino.p12 -clcerts -nokeys -out certificate.pem
```

The first command outputs the private key to privatekey.pem. The second outputs the certificate body to certificate.pem.

**Step 2: Convert the certificate chain file to PEM format**

Run the following command to convert the .p7b file to PEM format:

```
openssl pkcs7 -inform der -in DSNYCA_Chain-7-28-23.p7b -print_certs -out certificates.pem
```

This outputs the full certificate chain to certificates.pem.

**Step 3: Re-import the certificate into ACM**

1. Open the **AWS Certificate Manager (ACM)** console.
2. Locate the existing certificate and select **Reimport certificate**.
3. Paste the contents of certificates.pem into the **Certificate chain** field. Supply the certificate body and private key from Step 1 if prompted.
4. Complete the import.

## **Notes/Information:**

If changing the port to 9000 doesn't resolve the issue, please open a Support ticket and send in the following output for the datasource-proxy pod. This requires the Admin to have kubectl access:

1. Data Source Pod Status: kubectl -n `<platform-namespace>` get po --selector=app.kubernetes.io/name=datasource-proxy
2. Data Source Logs: kubectl -n `<platform-namespace>` logs --selector=app.kubernetes.io/name=datasource-proxy -c datasource-proxy
3. Data Source Proxy Pod describe: kubectl describe pod -n domino-platform `<datasource-proxy-name>`

Depending on the issue, Support may also ask for:

1. Data Source Proxy Deployment: kubectl get deployment -n domino-platform datasource-proxy -o yaml
2. Data Source Proxy Configure Map: kubectl get configmap -n domino-platform datasource-proxy -o yaml
3. Data Source Proxy Service: kubectl get service datasource-proxy -n domino-platform -o yaml

Since the certificate was originally imported into ACM manually, ACM provides the option to re-import it with an updated certificate chain.

### References/Internal Records:

* New York Dept-Sanitation Zendesk ticket #80270


# javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
Source: https://docs.domino.ai/knowledge-base/javax-net-ssl-sslexception-unsupported-or-unrecognized-ssl-message



#### **Version/Environment (if relevant):**

Not version specific

## **Issue:**

After specifying new SMTP settings for email notifications in Central Config a customer failed to receive any notifications. To find more detail we triggered a quick job and tailed the nucleus-dispatcher log. We found an exception occurring. The "caused by" provided a clue:

```
Caused by: javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
at java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(SSLSocketInputRecord.java:451)
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:175)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:110)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1418)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1324)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411)
at com.sun.mail.util.SocketFetcher.configureSSLSocket(SocketFetcher.java:626)
at com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:400)
at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:238)
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2175)
... 22 common frames omitted
```

## **Root Cause:**

`javax.net.ssl.SSLException: Unsupported or unrecognized SSL message` can occur if protocols are incorrect. For example if your specify a port that is **NOT** SSL but you chose settings that trying to connect via SSL, this can occur. It is the equivalent of trying to browse https protocol on a webserver port that is only listening for http protocol.

## **Resolution:**

You need to correct either the port or the protocol. Double-check your values for `com.cerebro.domino.email.smtp.ssl` and `com.cerebro.domino.email.channel` and ensure they're aligned with the mailserver's specifications on the port you've designated.

For example if your mailserver administrator told you to use port 25 then you probably need `com.cerebro.domino.email.smtp.ssl=false` rather than true since 25 is usually is not SSL-based.

If your mailserver requires TLS then use:
`com.cerebro.domino.email.smtp.ssl`: false
`com.cerebro.domino.email.smtp.channel`: starttls
and your port would likely be 587 (but confirm with your mailserver admin).

## **Notes/Information:**

Tail dispatcher with timestamps:
kubectl logs deployment.apps/nucleus-dispatcher -n `<your namespace>` --container=nucleus-frontend --follow --timestamps=true


# pip install throws SSL certificate error
Source: https://docs.domino.ai/knowledge-base/pip-install-throws-ssl-certificate-error

A pip install SSL certificate verification failure is typically caused by a firewall or proxy stripping SSL certificates, and can be resolved by using the --trusted-host flag either inline or persistently via pip.conf.

# pip install throws SSL certificate error

Version: Not specific to Domino version

## Issue

A `pip install` fails with:

`connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed`

## Root cause

By default, pip doesn't use system certificates. For details, see the [pip HTTPS certificates documentation](https://pip.pypa.io/en/stable/topics/https-certificates/). This error can occur when you're behind a firewall or proxy that strips the SSL certificate.

## Resolution

Use the `--trusted-host` flag to bypass certificate verification for the affected hosts.

**Option 1 — CLI flag**

Pass `--trusted-host` directly to the install command. Replace `<package>` with your package name:

`pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org <package>`

**Option 2 — pip.conf (persistent)**

Add the following to your `pip.conf` file to apply the setting to all pip commands.

The location of `pip.conf` varies by environment:

* **Linux/macOS (user-level):** `~/.config/pip/pip.conf`
* **Linux/macOS (virtual environment):** `<venv>/pip.conf`
* **Windows (user-level):** `%APPDATA%\pip\pip.ini`

```
[global]
trusted-host = pypi.python.org
               pypi.org
               files.pythonhosted.org
```

## Notes

This error can also appear in Dockerfile instructions used in Domino Compute Environments. The same `--trusted-host` workaround applies.

Add `--trusted-host` for each host your `RUN pip install` command accesses. For example:

```
USER root
RUN pip install --trusted-host <hostname> --trusted-host <cdn-hostname> <package-or-url>
```


# Pods stuck in ContainerCreating with an event mentioning errno 524
Source: https://docs.domino.ai/knowledge-base/pods-stuck-in-containercreating-with-an-event-mentioning-errno-524



#### ***Version/Environment (if relevant):***

All versions on AWS/EKS

## **Issue:**

Customers find Pods stuck in ContainerCreating and/or liveness probes fail with an event in the kubectl describe which mentions **`errno 524`**:

```
Warning FailedCreatePodSandBox 4m9s (x384 over 89m) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524: unknown
```

##

## **Root Cause:**

There is a linux kernel memory leak bug in the EKS optimized AMI based on linux kernel version 5.10.x. The linux bug can be verified by using *`cat /proc/sys/net/core/bpf_jit_limit`* and seeing what the limit currently is and by using *`cat /proc/vmallocinfo | grep bpf_jit | awk '{s+=$2} END {print s}'`*, to see if the limit has been exceeded.

Example:

```
[root@ip-10-0-35-23 /]# cat /proc/sys/net/core/bpf_jit_limit
264241152
[root@ip-10-0-35-23 /]# cat /proc/vmallocinfo | grep bpf_jit | awk '{s+=$2} END {print s}'
352337920
```

More details: [https://github.com/awslabs/amazon-eks-ami/issues/1219#issuecomment-1533797378](https://github.com/awslabs/amazon-eks-ami/issues/1219#issuecomment-1533797378)

"This issue is more likely to be encountered with kernel versions *`kernel-5.10.176-157.645.amzn2`thru*`kernel-5.10.177-158.645.amzn` where the rate of the memory leak is higher."

***

## **Resolution:**

AWS corrected the problem by patching by a new AMI [Release AMI Release v20230501 awslabs/amazon-eks-ami](https://github.com/awslabs/amazon-eks-ami/releases/tag/v20230501), released May 3rd 2023.

Temporary workaround is to run ***`sysctl net.core.bpf_jit_limit=452534528`*** on all **platform** nodes by ssh'ing directly onto them.

The permanent fix is to get your EKS to use this AMI: [Release AMI Release v20230501 awslabs/amazon-eks-ami](https://github.com/awslabs/amazon-eks-ami/releases/tag/v20230501)

## **Notes/Information:**

Note the *"temporary workaround"* is short-term only, since the kernel bug can continue leaking.


# Preventing Jupyter timeout errors when running notebooks as a batch/scheduled job
Source: https://docs.domino.ai/knowledge-base/preventing-jupyter-timeout-errors-when-running-notebooks-as-a-batch-scheduled-job



If you are running a Jupyter notebook as a batch job, and it contains a long-running cell, you may encounter the following error:

```
[NbConvertApp] ERROR | Timeout waiting for execute reply (30s). If your cell should take longer than this, you can increase the timeout with: c.ExecutePreprocessor.timeout = SECONDS in jupyter_nbconvert_config.py
```

This indicates that the cell caused Jupyter to timeout. You can avoid this by configuring a longer timeout. Here's how to do it in the pre-setup script of a [custom environment](https://docs.dominodatalab.com/en/latest/reference/environments/Environment_management.html) definition:

```
echo "c.ExecutePreprocessor.timeout = $SECONDS" >> /home/ubuntu/.ipython/profile_default/ipython_nbconvert_config.py
chown ubuntu:ubuntu /home/ubuntu/.ipython/profile_default/ipython_nbconvert_config.py
```

Substitute an appropriate value for **\$SECONDS**. Note the *chown* command - this is needed because the pre-setup script runs as the root user, so the file ownership must be changed back to the ubuntu user in order to avoid a permissions error during your run. Keep in mind this is for batch runs.

**Note:** later versions of Jupyter require you to modify a different config file *(`/home/ubuntu/.jupyter/jupyter_nbconvert_config.py`* instead of `/home/ubuntu/.ipython/profile_default/ipython_nbconvert_config.py`).

If you are running a recent Jupyter version and the above commands do not work, use this instead:

```
echo "c.ExecutePreprocessor.timeout = $SECONDS" >> /home/ubuntu/.jupyter/jupyter_nbconvert_config.py
chown ubuntu:ubuntu /home/ubuntu/.jupyter/jupyter_nbconvert_config.py
```


# Resolve internal (on-premise) DNS Server in AWS EKS via CoreDNS
Source: https://docs.domino.ai/knowledge-base/resolve-internal-on-premise-dns-server-in-aws-eks-via-coredns

Configure CoreDNS on AWS EKS and Domino environments to resolve on-premise DNS hostnames from Domino Workspaces via a VPN tunnel.

# Resolve internal (on-premise) DNS server in AWS EKS via CoreDNS

Domino users often need to access on-premise services from their Domino workspace. This article explains how to configure Amazon Web Services (AWS) Elastic Kubernetes Service (EKS) CoreDNS to forward DNS queries to an on-premise DNS server, then update a Domino environment to resolve the corresponding search domain.

This configuration assumes you have already established a VPN tunnel or equivalent network connectivity between AWS and your on-premise network.

## Prerequisites

* CoreDNS running on your EKS cluster (default since EKS v1.12). If CoreDNS isn't installed, follow the [AWS CoreDNS documentation](https://docs.aws.amazon.com/eks/latest/userguide/coredns.html) to install it.
* `kubectl` access to the EKS cluster running your Domino deployment. Update your local kubeconfig or connect directly in AWS before proceeding.

## Update the CoreDNS configmap

1. Open the CoreDNS configmap for editing:

   ```
   kubectl -n kube-system edit configmap coredns
   ```
2. Locate the `Corefile` section. It looks similar to this:

   ```
   Corefile: |
     .:53 {
       log
       errors
       health
       kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        upstream
        fallthrough in-addr.arpa ip6.arpa
       }
       prometheus :9153
       forward . /etc/resolv.conf
       cache 30
       loop
       reload
       loadbalance
     }
   ```
3. Add a new forwarding block after the closing `}` of the existing stanza. Replace `google.com` with your on-premise domain and `8.8.8.8` with the IP address of your DNS server (for example, the load balancer IP for your VPN tunnel). Keep the period (`.`) between `forward` and the IP address.

   The updated `Corefile` section should look like this:

   ```
   Corefile: |
     .:53 {
       log
       errors
       health
       kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        upstream
        fallthrough in-addr.arpa ip6.arpa
       }
       prometheus :9153
       forward . /etc/resolv.conf
       cache 30
       loop
       reload
       loadbalance
     }
     google.com:53 {
      forward . 8.8.8.8
     }
   ```
4. Save the file. The editor displays a confirmation message when the configmap saves successfully.
5. Restart the CoreDNS pods to apply the change immediately:

   ```
   kubectl get pods -n kube-system -oname |grep coredns |xargs kubectl delete -n kube-system
   ```
6. Verify that the CoreDNS pods return to a **Running** status:

   ```
   kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
   ```

   All listed pods should show `Running` in the **STATUS** column before you proceed.

## Configure the Domino environment

Add a pre-setup script to the Domino environment. This script appends the search domain to `/etc/resolv.conf` inside each workspace container.

1. In Domino, navigate to **Environments** and select the environment you want to update.
2. Click **Edit Definition**.
3. Scroll down and expand the **Advanced** section.
4. Enter the following script in the **Pre Setup Script** field, replacing `google.com` with your on-premise domain:

   ```
   cat /etc/resolv.conf >> /tmp/resolv.conf.tmp
   sed -i '2s/$/ google.com/' /tmp/resolv.conf.tmp
   cat /tmp/resolv.conf.tmp > /etc/resolv.conf
   ```
5. Save and build the environment.

## Verify DNS resolution

1. Start a new Jupyter workspace and open a terminal.
2. Run an `nslookup` for a hostname in your domain. Using the Google example:

   ```
   nslookup maps
   ```
3. Confirm that the output resolves the short hostname to a fully qualified domain name (FQDN). Expected output:

   ```
   ubuntu@run-5ea9d22e84e7430006080fdb-7bfkr:/mnt$ nslookup maps
   Server:     10.100.0.10
   Address:    10.100.0.10#53

   Non-authoritative answer:
   Name:  maps.google.com
   Address: 172.217.14.206
   Name:  maps.google.com
   Address: 2607:f8b0:400a:808::200e
   ```

   Successful output shows the short name resolving to the FQDN (for example, `maps` → `maps.google.com`), confirming that on-premise DNS resolution is working


# Scope of Support Coverage
Source: https://docs.domino.ai/knowledge-base/scope-of-support-coverage



## **Scope of Service**

The following information relates to the Support services Domino bundles with our customer/user licenses, at no additional cost, via Domino Support's Customer Portal ([https://support.domino.ai](https://support.domino.ai) and [support@dominodatalab.com](mailto:support@dominodatalab.com)).

Domino Support provides world-class Technical Support and an outstanding customer experience by ensuring timely, thorough resolution to the issues we investigate. We maintain a constant feedback loop with consumers of our services, acting quickly and frequently on their input to achieve continuous improvement.

Domino Support's area of expertise is in core Domino products. We maintain very high standards for customer satisfaction and responsiveness when investigating product-related issues on behalf of our customers. Please see the information below for details.

Domino Support provides support for:

* Usage of our core products
* Diagnosis of issues within our core products
* Configuration of core product features

Domino Support does not provide expertise related to:

* Modified installers or deployment packages
* Building custom environments and dependency resolution
* Product customizations or code development
* System or network design (for systems and networks not owned by Domino)
  * Customers on Domino Cloud may escalate issues to Domino Cloud Operations through Domino Support
* Implementation of development or security policies/practices
* Undocumented product features or functionality
* Beta releases, release candidates, or feature previews
* Training modules
* Untested platforms
* Installation, upgrades, or migration of Domino
* Disaster recovery:
  * This includes data recovery exercises and data migration

This applies to the commercial releases of Domino products, including Domino Workbench and Domino Model Monitor

### **Domino APIs: Scope of Coverage**

In addition to the information above, the following information applies to Domino Support's service coverage for our APIs (example: REST, Model API).

Domino Support offers the following assistance for APIs:

* Confirmation that Domino's proprietary methods and inputs return results as expected, based on our product documentation

Domino Support does not provide assistance for:

* Customers' scripts, code, application, customized system configuration, or issues that result from conflicts therein


# Impact of Secure Identity Propagation Feature Flag on Domino Apps
Source: https://docs.domino.ai/knowledge-base/secureidentitypropagationtoappsenabled-feature-flag-impacts-app-behavior

In Domino 6.1.1+, SecureIdentityPropagationToAppsEnabled defaults to true, breaking apps on restart if they manually construct requests_pathname_prefix. Fix by replacing the manual path with os.environ.get("DOMINO_RUN_HOST_PATH").

# Apps Break When SecureIdentityPropagationToAppsEnabled Is True

## Issue

A Domino App stops working after it is restarted on a deployment where the SecureIdentityPropagationToAppsEnabled feature flag is set to true. The app may have been functioning correctly before the restart or before the flag was enabled. The most common symptom is a routing or 404 error caused by a stale, manually constructed requests\_pathname\_prefix path that no longer matches the new routing structure.

**Typical symptoms include:**

* App loads a blank page or returns a 404/502 after restart
* Dash (Plotly) app fails to serve assets or callbacks
* The app was working before the flag was enabled or before the last restart
* The app code manually constructs the path prefix from DOMINO\_PROJECT\_OWNER, DOMINO\_PROJECT\_NAME, and DOMINO\_RUN\_ID environment variables

## Cause

When SecureIdentityPropagationToAppsEnabled is true, Domino changes the URL routing structure used to proxy requests to a running app. Apps that manually construct the requests\_pathname\_prefix from individual environment variables (DOMINO\_PROJECT\_OWNER, DOMINO\_PROJECT\_NAME, DOMINO\_RUN\_ID) produce a path that no longer matches the new routing structure, causing request routing to fail.

An app that is **already running** when the feature flag is enabled will continue to work until it is restarted. The breakage occurs on the next restart because the new routing structure takes effect at that point.

The legacy path construction pattern that causes the issue:

```
user    = os.environ.get("DOMINO_PROJECT_OWNER")
project = os.environ.get("DOMINO_PROJECT_NAME")
run_id  = os.environ.get("DOMINO_RUN_ID")

requests_pathname_prefix = f"/{user}/{project}/r/notebookSession/{run_id}/"
```

## Resolution

Replace the manually constructed path prefix with the Domino-provided DOMINO\_RUN\_HOST\_PATH environment variable. This variable is automatically set by Domino and always reflects the correct routing path for the current session, regardless of whether SecureIdentityPropagationToAppsEnabled is true or false.

### Step 1 — Locate the path prefix in your app code

Search your app's entry point for any reference to requests\_pathname\_prefix that manually constructs the path from environment variables, for example:

```
user    = os.environ.get("DOMINO_PROJECT_OWNER")
project = os.environ.get("DOMINO_PROJECT_NAME")
run_id  = os.environ.get("DOMINO_RUN_ID")
runurl  = f'/{user}/{project}/r/notebookSession/{run_id}/'

app = Dash(__name__, routes_pathname_prefix='/', requests_pathname_prefix=runurl)
```

### Step 2 — Replace with DOMINO\_RUN\_HOST\_PATH

Update the code to read the path directly from the Domino-provided environment variable:

```
runurl = os.environ.get("DOMINO_RUN_HOST_PATH")

app = Dash(__name__, routes_pathname_prefix='/', requests_pathname_prefix=runurl)
```

### Step 3 — Restart the app

Publish the updated code to your Domino project and restart the app. Verify that the app loads correctly and that callbacks and assets resolve without errors.

## Additional Notes

**Important:** If SecureIdentityPropagationToAppsEnabled is set to false and you need to allow anonymous (unauthenticated) access to Domino Apps, ensure the Central Config parameter com.cerebro.domino.launchpad.allowPublicModelProducts is also set to true. See KB article [000002243](https://support.domino.ai/support/s/article/We-are-no-longer-seeing-the-option-to-allow-anonymous-users-to-view-apps) for details.

* **Domino Cloud / 6.1.1+:** The flag defaults to true. All new and existing apps should be updated to use DOMINO\_RUN\_HOST\_PATH.
* **Domino 6.0.x (self-managed):** The flag defaults to false. No immediate action is required, but migrating to DOMINO\_RUN\_HOST\_PATH is strongly recommended to avoid breakage during future upgrades.
* **Non-Dash frameworks:** Any Python web framework (Streamlit, Flask, etc.) that accepts an explicit base path or prefix should also be updated to read from DOMINO\_RUN\_HOST\_PATH.

## Related Articles

* [000002243 — Anonymous user access to Domino Apps](https://support.domino.ai/support/s/article/We-are-no-longer-seeing-the-option-to-allow-anonymous-users-to-view-apps)
* [000001814 — Feature Flags in Domino](https://support.domino.ai/support/s/article/Feature-flags-1718868051921)
* [000002017 — Changing a specific user's feature flag](https://support.domino.ai/support/s/article/Changing-a-specific-user-s-feature-flag-1724943001691)


# Configuring and Sending Emails in Domino Using Python smtplib
Source: https://docs.domino.ai/knowledge-base/sending-emails-python-smtplib

Guide to sending emails from Domino using Python's smtplib. Covers secure SMTP setup via SMTPS or STARTTLS, sending plain-text emails, and HTML-formatted emails using MIME.

## Issue

Users need guidance on how to send emails from Domino using Python's `smtplib` library, including how to set up a secure SMTP connection, send plain-text emails, and send HTML-formatted emails using MIME.

## Root cause

Sending emails from Domino requires configuring a secure SMTP connection and using the correct Python libraries. Without proper setup of connection security, authentication, and MIME formatting, emails may fail to send or render incorrectly.

## Resolution

This article covers three topics:

* Setting up a secure SMTP connection using SMTPS or STARTTLS
* Sending plain-text emails using `smtplib`
* Sending HTML-formatted emails using the `email.mime` module

### 1. Setting up a local SMTP debug server

To test your code locally, use the Python `smtpd` binary bound to port 1025. Messages will not leave localhost and will be output to stdout as a byte stream:

```bash theme={null}
python -m smtpd -c DebuggingServer -n localhost:1025
```

### 2. Starting a secure SMTP connection

There are two ways to secure your SMTP connection. Both examples use Gmail's SMTP server and Python's `ssl` package to load the system's default trusted CA bundles via `create_default_context()`.

**Option A: SMTPS (secured from the outset)**

```python theme={null}
# Using SMTPS to secure the connection from the outset
import smtplib, ssl

port = 465  # For SSL
password = input("Input password and press enter: ")

# Create a secure SSL context, loads system's default trusted CAs
context = ssl.create_default_context()

with smtplib.SMTP_SSL("smtp.gmail.com", port, context=context) as server:
    server.login("my@gmail.com", password)
```

**Option B: STARTTLS (connection secured after initial handshake)**

```python theme={null}
# Using STARTTLS to secure the connection from a point onwards
import smtplib, ssl

smtp_server = "smtp.gmail.com"
port = 587  # For STARTTLS
sender_email = "my@gmail.com"  # Enter your address
password = input("Input password and press enter: ")

# Create a secure SSL context, loads system's default trusted CAs
context = ssl.create_default_context()

try:
    server = smtplib.SMTP(smtp_server, port)
    server.ehlo()  # Can be omitted, called implicitly by the method if required
    server.starttls(context=context)  # Secure the connection
    server.ehlo()  # Can be omitted, called implicitly by the method if required
    server.login(sender_email, password)
    # TODO: Send email here
except Exception as e:
    # Print any error messages to stdout
    print(e)
finally:
    server.quit()
```

### 3. Sending a plain-text email

Define the sender, receiver, and message body at the top of your script. Use `server.sendmail` to send the email.

**Using SMTPS:**

```python theme={null}
import smtplib, ssl

port = 465  # For SSL
smtp_server = "smtp.gmail.com"
sender_email = "my@gmail.com"  # Enter your address
receiver_email = "your@gmail.com"  # Enter receiver address
password = input("Input password and press enter: ")
message = """\
Subject: Hi there

Message body after 2 new lines."""

context = ssl.create_default_context()
with smtplib.SMTP_SSL(smtp_server, port, context=context) as server:
    server.login(sender_email, password)
    server.sendmail(sender_email, receiver_email, message)
```

**Using STARTTLS:**

```python theme={null}
import smtplib, ssl

port = 587  # For STARTTLS
smtp_server = "smtp.gmail.com"
sender_email = "my@gmail.com"  # Enter your address
receiver_email = "your@gmail.com"  # Enter receiver address
password = input("Input password and press enter: ")
message = """\
Subject: Hi there

Message body after 2 new lines."""

context = ssl.create_default_context()
with smtplib.SMTP(smtp_server, port) as server:
    server.ehlo()  # Can be omitted, called implicitly by the method if required
    server.starttls(context=context)
    server.ehlo()  # Can be omitted, called implicitly by the method if required
    server.login(sender_email, password)
    server.sendmail(sender_email, receiver_email, message)
```

### 4. Sending HTML-formatted emails using MIME

Use the `email.mime` module to send emails combining HTML and plain-text content. Since email clients render the last MIME part first, the HTML part must be added after the plain-text part.

```python theme={null}
import smtplib, ssl
from email.mime.text import MIMEText  # Contains the HTML and plain-text content
from email.mime.multipart import MIMEMultipart  # Combines the two

sender_email = "my@gmail.com"  # Enter your address
receiver_email = "your@gmail.com"  # Enter receiver address
password = input("Input password and press enter: ")

message = MIMEMultipart("alternative")
message["Subject"] = "multipart test"
message["From"] = sender_email
message["To"] = receiver_email

# Create the plain-text and HTML version of your message
text = """\
Hi,
How are you?"""
html = """\
<html>
  <body>
    <p>Hi,<br>
      How are you?<br>
    </p>
  </body>
</html>
"""

# Convert these into MIMEText objects
text1 = MIMEText(text, "plain")
html1 = MIMEText(html, "html")

# Add the objects to the MIMEMultipart object
# The email client renders the last part first, hence we add the HTML part last
message.attach(text1)
message.attach(html1)

# Create secure connection with server and send email
context = ssl.create_default_context()
with smtplib.SMTP_SSL("smtp.gmail.com", 465, context=context) as server:
    server.login(sender_email, password)
    server.sendmail(
        sender_email, receiver_email, message.as_string()
    )
```

## Notes

When configuring for a production Domino deployment (assumed to use AWS SES as the SMTP server), the following values are required:

* **SMTP host:** The server name from the SMTP settings page, accessible via the AWS SES service page.
* **SMTP username and password:** Generate these from the SMTP settings page using the **Create My SMTP Credentials** button.
* These values should be available in Domino Central Config. If not, they must be added as Central Config key-value pairs. Contact your internal Domino admin for assistance.
* The code examples in this article are provided as a reference. Some adjustments and indentation corrections may be needed to suit your specific application.


# Spark Executor Pods Unable to Find Kerberos Credentials Cache
Source: https://docs.domino.ai/knowledge-base/spark-on-domino-klist-no-credentials-cache-found-error-when-trying-to-use-kerberos-to-authenticate-into-sql-server

Kerberos klist: No credentials cache found on Spark executors. Fix: set KRB5CCNAME and KRB5_CONFIG via ENV in the Dockerfile, run kinit in Pre-Run, and concatenate all extraJavaOptions in one line.

### ***Version/Environment (if relevant):***

Applies to all versions of Domino.

## **Issue:**

When using Kerberos authentication with PySpark in a Domino Spark cluster to connect to a SQL Server, the following errors are encountered:

```
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1) (12.34.567.89 executor 0): com.microsoft.sqlserver.jdbc.SQLServerException: Integrated authentication failed. ClientConnectionId:1234d56a-78c9-1234-beda-eedc1234b567
```

The [Spark configuration](https://docs.dominodatalab.com/en/latest/user_guide/f11f6a/work-with-your-cluster/#_spark_executors_versus_workers) has been set at both the code level and on the Spark Web UI's Project Integration page. Running `kinit` in the Spark driver (workspace) succeeds, and the token is visible via `klist` in the workspace logs:

```
I have no name!@spark-123456cf7c89123b456fa789-spark-worker-2:/tmp$ klist
Ticket cache: FILE:/tmp/krb5cc_12345
Default principal: sys-data@AD.COMPANY.COM

Valid starting   Expires       Service principal
03/29/23 15:28:44  03/29/23 23:28:44  krbtgt/AD.COMPANY.COM@AD.COMPANY.COM
   renew until 04/03/23 15:28:44
```

However, `klist` fails on the executor Spark worker pods with the following error:

```
I have no name!@spark-1234567f1c23456b789faed1-spark-worker-0:/opt/bitnami/spark$ klist
klist: No credentials cache found (filename: /tmp/krb5cc_12345)
```

## **Root Cause:**

There are two root causes:

1. Setting environment variables via `spark.executorEnv.ENVNAME` does not work. Environment variables must be set using `ENV` in the cluster environment's Dockerfile.
2. When setting multiple `extraJavaOptions` parameters with one line per option, only the last parameter is applied at runtime — each new line overwrites the previous one.

## **Resolution:**

**1. Dockerfile Instructions**

Set the following environment variables in the executor cluster environment's Dockerfile:

* `ENV KRB5CCNAME=FILE:/mnt/data/workspace_name/krb5cc_12345`
* `ENV KRB5_CONFIG=/mnt/data/workspace_name/krb5.conf`

This changes the default `ccache` directory to a shared directory accessible by both the driver and executors, allowing executors to read the `ccache` generated by the driver during pre-run. Setting `KRB5CCNAME` via `spark.executorEnv.KRB5CCNAME` or any other method will not work — executors will always fall back to the default location `/tmp/krb5cc_12345`. Use `ENV` instead of `ARG` when setting `KRB5_CONFIG`.

**2. Pre-Run Script**

Run `kinit` in the Spark driver's environment Pre-Run script to generate and cache the Kerberos token on the dataset so that executors can use it for authentication.

**3. `extraJavaOptions` Parameters**

Under **Project Settings > Integrations > Apache Spark mode > Spark Configuration Options**, setting multiple `extraJavaOptions` with one line per parameter will not work, as each line overwrites the previous:

```
spark_conf.set("spark.executor.extraJavaOptions","-Djavax.security.auth.useSubjectCredsOnly=false")
spark_conf.set("spark.executor.extraJavaOptions","-Djava.security.auth.login.config=/mnt/data/workspace_name/SQLJDBCDriver2.conf")
spark_conf.set("spark.executor.extraJavaOptions","-Djava.security.krb5.conf=/mnt/data/workspace_name/krb5.conf")
spark_conf.set("spark.executor.extraJavaOptions","-Dsun.security.krb5.debug=true")
![Project_settings_-_Integrations_-_Apache_Spark_mode.png](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr&ids=068UY00000Hru0q&d=%2Fa%2FUY0000018p1k%2FT0zoitoAOVJ0hrugfPbV4jh50.msvIGPLAufFbN3Yto&asPdf=false)
```

Instead, concatenate all `spark.driver.extraJavaOptions` parameters into a single line, separated by spaces:

```
spark_conf.set("spark.executor.extraJavaOptions","-Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.auth.login.config=/mnt/data/workspace_name/SQLJDBCDriver2.conf -Djava.security.krb5.conf=/mnt/data/workspace_name/krb5.conf -Dsun.security.krb5.debug=true")
```

![Spark\_Web\_UI\_-Environment-Spark\_Properties-\_all\_params.png](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr\&ids=068UY00000Hru0s\&d=%2Fa%2FUY0000018p1m%2FT8Et6o60_ob2jCkG.RVRJ9ufONxIQFEYytdAYaz6AMM\&asPdf=false)

You can verify all parameters are correctly applied by checking **Spark Web UI > Environment > Spark Properties**.

![Spark\_Web\_UI\_-Env-Spark\_Properties-\_only\_last\_param.png](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr\&ids=068UY00000Hru0r\&d=%2Fa%2FUY0000018p1l%2FrjC1BtmO6uZkRsyiSCYp8yG8pzZs6_0.3q_rqBYq8FQ\&asPdf=false)

### **Notes/Information:**

Additional resources:

* [Apache Spark 3.0 — Kerberos ticket cache](https://spark.apache.org/docs/3.0.0-preview/security.html#using-a-ticket-cache): By setting `spark.kerberos.renewal.credentials` to `ccache`, the local Kerberos ticket cache will be used for authentication. The location can be customized via the `KRB5CCNAME` environment variable.
* [Spark on Domino — Configure Prerequisites](https://docs.dominodatalab.com/en/latest/user_guide/1962f3/configure-prerequisites/): Configure PySpark compute environments for workspaces and/or jobs connecting to your cluster.
* [Domino Kerberos Authentication](https://docs.dominodatalab.com/en/latest/user_guide/ed5f5a/kerberos-authentication/): Domino supports keytab-based Kerberos authentication at the project or user level. Once configured, runs will automatically execute `kinit` and retrieve the ticket for authentication.

### **References/Internal Records:**


# Domino Support Service Information
Source: https://docs.domino.ai/knowledge-base/support-policies-overview

What's covered under Domino Support, and what isn't.

What's covered under Domino Support, and what isn't.

## Domino Support Policies and Procedures

* [Scope of Support Coverage](/knowledge-base/scope-of-support-coverage)
* [Domino Product Version Lifecycle and Support Policies](/knowledge-base/domino-product-version-lifecycle-and-support-policies)


# The lsof utility - 'list open files'
Source: https://docs.domino.ai/knowledge-base/the-lsof-utility-list-open-files

Troubleshoot open files, network ports, and process locks on Linux using lsof basic and advanced command examples.

**Version/Environment (if relevant):** Applicable to any Linux-based system including Domino deployments.

## Issue:

Administrators and engineers need to identify processes holding file locks, investigate socket activity, or troubleshoot process ID (PID) conflicts on a Linux-based system.

## Root Cause:

This is a reference article for the `lsof` utility.

## Resolution:

### What is lsof?

`lsof` lists open files on Linux-based systems. It displays information about file descriptors and the processes that hold them open, including network connections and listening ports. Use `lsof` to identify processes holding file locks, investigate socket activity, and troubleshoot process ID (PID) conflicts.

> ⚠️ **Prerequisite:** `lsof` may require privilege elevation to access files belonging to other users. Prepend `sudo` to any command when running as a non-root user.

**Basic usage**

Use the following commands to troubleshoot open files on a Linux system:

Find all open files for a user:

```
lsof -u <username>
```

Find all open files associated with a particular process using a PID:

```
lsof -p <PID>
```

List and locate all open socket files:

```
lsof +D / -a -d sock
```

Locate all open files listening on a specific port:

```
lsof -i :<port number>
```

Identify all open files linked to an application by program name:

```
lsof -c <program name>
```

**Advanced usage**

Use these commands for more targeted investigation across processes, file descriptors, and time intervals:

Display the process name, PID, and file name for all open files at a set time interval:

```
lsof -n -r <time interval> +L -F pn -F f
```

Locate all open files for a specific process at a set time interval and sort by PID:

```
lsof -p <PID> -n -r <time interval> +L -s
```

Find all open files and the command line that opened them:

```
lsof -c <command name> -a -c <command name> +L1
```

Find all open files linked to a specific file descriptor:

```
lsof -a -d <file descriptor>
```

### Notes/Information:

* `lsof` can also help identify processes blocking a filesystem unmount. Running `lsof <filesystem_name>` will display all open files on the named filesystem.
* `lsof +L1` can be used to find unlinked open files on local disks — files that have been deleted but are still held open by a process, which can cause unexpected disk usage.


# Troubleshooting
Source: https://docs.domino.ai/knowledge-base/troubleshooting-overview

Fixes for common errors across environments, workspaces, models, and infrastructure.

Fixes for common errors across environments, workspaces, models, and infrastructure.

## Administration

* [javax.net.ssl.SSLException: Unsupported or unrecognized SSL message](/knowledge-base/javax-net-ssl-sslexception-unsupported-or-unrecognized-ssl-message)

## Apps

* [Impact of Secure Identity Propagation Feature Flag on Domino Apps](/knowledge-base/secureidentitypropagationtoappsenabled-feature-flag-impacts-app-behavior)

## Authentication

* [Unexpected error when authenticating with identity provider (duplicate key constraint)](/knowledge-base/unexpected-error-when-authenticating-with-identity-provider-duplicate-key-value-violates-unique-constraint-constraint-40)
* [Invalid Client Credentials When Generating A Service Account Token](/knowledge-base/invalid-client-credential-when-generating-service-account-token)

## Data Sources & Datasets

* [Database Locking](/knowledge-base/database-locking)
* ["Invalid host" error connecting MiniO server as a Generic S3 data source](/knowledge-base/invalid-host-error-when-trying-to-connect-minio-server-as-a-generic-s3-data-source)

## Environments

* [Preventing the permission denied error when running pip install](/knowledge-base/how-to-prevent-the-permission-denied-error-when-running-pip-install-in-a-workspace)
* [Build error for nvidia GPG key](/knowledge-base/build-error-for-nvidia-gpg-key)
* [Docker Layers: failed to register layer, max depth exceeded](/knowledge-base/docker-layers-failed-to-register-layer-max-depth-exceeded)
* [pip install throws SSL certificate error](/knowledge-base/pip-install-throws-ssl-certificate-error)
* [Unable to install R package: package not available for your version of R](/knowledge-base/unable-to-install-r-package-error-package-is-not-available-for-your-version-of-r)
* [Yarn Package invalid key](/knowledge-base/yarn-package-invalid-key)
* [Could not install packages: EnvironmentError Errno 13 Permission denied](/knowledge-base/error-could-not-install-packages-due-to-an-environmenterror-errno-13-permission-denied)
* [Cannot install the pywin32 package using pip install](/knowledge-base/i-cannot-install-the-pywin32-package-on-domino-using-pip-install)
* [Environment build fails: method driver /usr/lib/apt/methods/https not found](/knowledge-base/environment-build-fails-with-the-method-driver-usr-lib-apt-methods-https-could-not-be-found)
* [Can Not Connect to KeyServer Caused by GPG Error](/knowledge-base/can-not-connect-to-keyserver-caused-by-gpg-error)
* [How to use the apt-get install command in environments](/knowledge-base/how-to-use-the-apt-get-install-command-in-environments)

## Git Integration

* [Clearing Git Cache](/knowledge-base/clearing-git-cache)

## Infrastructure and Kubernetes

* [FailedMount event: efs.csi.aws.com not found in registered CSI drivers](/knowledge-base/failedmount-event-efs-csi-aws-com-not-found-in-the-list-of-registered-csi-drivers)
* [Pods stuck in ContainerCreating with an event mentioning errno 524](/knowledge-base/pods-stuck-in-containercreating-with-an-event-mentioning-errno-524)
* [Spark Executor Pods Unable to Find Kerberos Credentials Cache](/knowledge-base/spark-on-domino-klist-no-credentials-cache-found-error-when-trying-to-use-kerberos-to-authenticate-into-sql-server)
* [vm\_memory\_high\_watermark error in RabbitMQ service logs](/knowledge-base/vm-memory-high-watermark-error-in-rabbitmq-service-logs)
* [How to Troubleshoot Kubernetes Autoscaling (ASG) Issues](/knowledge-base/how-to-troubleshoot-autoscaling-asg-issues)
* [Failed to pull EKS pause image from ECR](/knowledge-base/failed-to-pull-image-602401143452-dkr-ecr-us-east-1-amazonaws-com-eks-pause-3-5)

## Models

* [Unable to deploy Model: Errno 13 Permission denied /.cache](/knowledge-base/unable-to-deploy-model-errno-13-permission-denied-cache)
* [uwsgi error loading shared libraries: libpcre.so.1 not found](/knowledge-base/uwsgi-error-while-loading-shared-libraries-libpcre-so-1-cannot-open-shared-object-file-no-such-file-or-directory)
* [Model build errors: failed to establish connection to prometheus-client](/knowledge-base/errors-during-model-build-failed-to-establish-a-new-connection-errno-2-name-or-service-not-known-simple-prometheus-client)

## REST API

* [415 Unsupported Media Type Error When Calling the Domino REST API](/knowledge-base/i-keep-getting-a-415-error-unsupported-media-type-when-i-call-domino-s-rest-api-what-am-i-doing-wrong)

## Workspaces

* [VSCode Error: ENOSPC, system limit for file watchers reached](/knowledge-base/error-enospc-system-limit-for-number-of-file-watchers-reached)
* [VSCode Failed to get latest version: getaddrinfo ENOTFOUND](/knowledge-base/vscode-failed-to-get-latest-version-error-getaddrinfo-enotfound-api-github-com)
* [Why did my execution fail with "No space left on device"?](/knowledge-base/why-did-my-execution-fail-with-the-error-no-space-left-on-device)


# Unable to deploy Model - [Errno 13] Permission denied: '/.cache'
Source: https://docs.domino.ai/knowledge-base/unable-to-deploy-model-errno-13-permission-denied-cache



## Issue

When deploying a model as a Model API, the build completes successfully but the model fails to start, with the following error:

```
Jan 05 2023 14:21:05 -0500 There was a problem when trying to write in your cache folder (/.cache/huggingface/hub). You should set the environment variable TRANSFORMERS_CACHE to a writable directory.
Jan 05 2023 14:21:05 -0500 PermissionError: [Errno 13] Permission denied: '/.cache'
```

The model's Python code uses the Transformers package. By default, Transformers sets its cache directory to `/.cache`, which may not be writable in the Model API environment.

## Solution

Override the default cache path by setting `TRANSFORMERS_CACHE` to a writable directory.

**Python:**

```
import os
os.environ['TRANSFORMERS_CACHE'] = '/path/to/writable/cache/'
```

**Bash:**

```
export TRANSFORMERS_CACHE=/path/to/writable/cache/
```

Replace `/path/to/writable/cache/` with a directory your model environment can write to, such as a path under your project or home directory.

## Set HF\_HOME for broader cache coverage

The `TRANSFORMERS_CACHE` variable controls only the Transformers cache. The Hugging Face (HF) `HF_HOME` variable controls a broader set of caches, including the datasets cache. Set `HF_HOME` to cover both:

```
export HF_HOME=/path/to/writable/cache/
```

To also protect against cases where the environment variable isn't set, create a symlink from the default cache location to your writable directory:

```
ln -s /path/to/writable/cache/ ~/.cache/huggingface
```

### How the default cache directory is determined

The Transformers documentation describes [cache setup](https://huggingface.co/docs/transformers/installation?highlight=transformers_cache#cache-setup) and the order of priority for environment variables:

> Pretrained models are downloaded and locally cached at: `~/.cache/huggingface/transformers/`. This is the default directory given by the shell environment variable `TRANSFORMERS_CACHE`. On Windows, the default directory is given by `C:\Users\username\.cache\huggingface\transformers`. You can change the shell environment variables shown below — in order of priority — to specify a different cache directory:
>
> 1. Shell environment variable (default): `TRANSFORMERS_CACHE`
> 2. Shell environment variable: `HF_HOME` + `transformers/`
> 3. Shell environment variable: `XDG_CACHE_HOME` + `/huggingface/transformers`

`HF_HOME` defaults to `$XDG_CACHE_HOME/huggingface`. If `XDG_CACHE_HOME` is not set, it defaults to `$HOME/.cache`, making the usual default `$HOME/.cache/huggingface`.


# Unable to install R package | Error : Package is not available for your version of R
Source: https://docs.domino.ai/knowledge-base/unable-to-install-r-package-error-package-is-not-available-for-your-version-of-r

Fix the "package is not available for this version of R" CRAN error by specifying an alternate mirror inline or updating your default mirror in .Rprofile or rsession.conf.

## **Issue**

When building an environment or installing a package inside a workspace, R raises an error similar to the following:

```
Warning in install.packages :
  package 'renv' is not available for this version of R

Warning in install.packages :
  unable to access index for repository https://cran.us.r-project.org/bin/...:
  cannot open URL 'https://cran.us.r-project.org/bin/.../PACKAGES'
```

The package isn't available from the default Comprehensive R Archive Network (CRAN) mirror ([https://cran.us.r-project.org/](https://cran.us.r-project.org/)) configured for this environment. The same package may be available from a different CRAN mirror.

## **Quick fix: specify a different CRAN mirror inline**

Pass a different repo argument directly to install.packages():

`install.packages("renv", repo = "https://cran.r-project.org/")`

This overrides the default mirror for this one installation call without changing any configuration files.

## **Permanent fix: update the default CRAN mirror**

To avoid specifying the `repo` argument every time, update the default CRAN mirror using one of the options below.

### **Option 1: edit `rsession.conf` in your compute environment**

Add the r-cran-repos setting to your rsession.conf file and point it to the correct CRAN mirror. For example:

`r-cran-repos=https://cran.r-project.org/`

Locate rsession.conf in your compute environment's R session configuration directory.

### **Option 2: edit .Rprofile or Rprofile.site**

Set the default CRAN mirror in your .Rprofile or Rprofile.site file. Find both files under your R home directory. Add the following:

`options(repos = c(CRAN = "https://cran.r-project.org/"))`

* .Rprofile — applies settings for your user session only

* Rprofile.site — applies settings for all users on the system

After saving either file, restart your R session for the changes to take effect.


# Unexpected error when authenticating with identity provider: duplicate key value violates unique constraint "constraint_40"
Source: https://docs.domino.ai/knowledge-base/unexpected-error-when-authenticating-with-identity-provider-duplicate-key-value-violates-unique-constraint-constraint-40

Resolve the Keycloak "duplicate key value violates unique constraint constraint_40" SSO login error by removing and re-linking the affected user's Identity Provider record.

**Applies to:** All versions through 6.x with Single Sign-On (SSO) (such as Okta, WebSSO, etc.)

## **Issue:**

An individual user attempts to log in and sees "Unexpected error when authenticating with Identity provider":

Clearing the browser cache and cookies or switching to a different browser does not resolve the error. Search for `exception` in the Keycloak logs using the following command:

```
kubectl logs -n domino-platform keycloakpodname --timestamps | grep -i exception
```

The logs return an entry similar to this:

```
2023-04-25T13:06:37.247217108Z ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default task-3858) ERROR: duplicate key value violates unique constraint "constraint_40"
2023-04-25T13:06:37.247219975Z Detail: Key (identity_provider, user_id)=(domino-credentials, 39a47782-8c3e-46d1-9091-a7105ff795ea) already exists.
```

## **Root Cause:**

Domino requests that Keycloak verify authentication against the SSO provider. Based on the SSO response, Keycloak attempts to create a new user record for the combination of **Provider Username** and email address.

Keycloak creates a new record when it detects a change in the casing of the username, email address, or another attribute. For example, if the SSO Security Assertion Markup Language (SAML) response returns an email address with an uppercase letter, Keycloak treats it as a new record.

The duplicate key violation occurs because the username or **Provider User ID** already exists in the database.

## **Resolution:**

1. In the Keycloak Admin Console, open the individual user record.
2. On the **Identity Provider Links** tab, select **Remove**.
3. Have the user re-attempt login through the Domino UI.

### **Notes**

To understand why Keycloak is treating the user as a new record, review the SAML requests and responses exchanged between Keycloak and the SSO provider.


# Use Gradio in a Jupyter workspace
Source: https://docs.domino.ai/knowledge-base/use-gradio-in-a-jupyter-workspace



#### **Version/Environment:**

This applies to all versions of Domino.

## **Issue:**

[Gradio's "Hello, World" demo](https://pypi.org/project/gradio/) fails to render in a Jupyter workspace and/or a browser window.

## **Root Cause:**

There is a bug in the Gradio package which strips the trailing slash from URLs being loaded in-line, i.e. in the notebook.

## **Resolution:**

Download the attached example Jupyter notebook to get started using Gradio in Domino. This runs on the current Domino Standard Environment (DSE) and DSE with GPU images. The app renders both in the Jupyter workspace and in a browser tab. In this line, replace the URL demo2.dominodatalab.com with your deployment's URL (without https\://):

* host = os.environ.get('GRADIO\_PROXY\_HOST') or "demo2.dominodatalab.com"

Please note that you'll need access to jupyter-server-proxy >=3.2.2 and access to the Pypi repositories to download the Gradio packages.

* To check the version in a Jupyter Terminal, run pip list | grep -i jupyter-server-proxy

Please note "show\_api=False" is not supported in newer version of Gradio 6.x, you will need to remove it from launch,

```
# Launch gradio app in the Domino workspace (behind proxy)
        interface.launch(root_path=application_path, server_name=server, server_port=port, quiet=True, inline=False, share=False)
```

### References/Internal Records:

* PTE-666


# uwsgi: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory
Source: https://docs.domino.ai/knowledge-base/uwsgi-error-while-loading-shared-libraries-libpcre-so-1-cannot-open-shared-object-file-no-such-file-or-directory

Fix the uWSGI libpcre.so.1 missing shared library error in Domino model deployments using ldd diagnostics and symlink or LD_LIBRARY_PATH remediation steps.

# uWSGI error: cannot open shared object file libpcre.so.1

## Error

`uwsgi: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory`

## Cause

This error occurs in model builds when uWSGI can't find one of its dynamically linked libraries. The model builds successfully, but fails to start. Instance logs show the uWSGI error.

## Solution

Use the `ldd` (list dynamic dependencies) command to identify the missing library and its expected location, then create a symlink or update the library path to resolve it.

### Step 1: Reproduce the error in a workspace

`$ uwsgi
uwsgi: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory`

### Step 2: Locate the uWSGI binary

`$ which uwsgi
/usr/local/anaconda/bin/uwsgi`

### Step 3: Identify the missing library

Run `ldd` against the uWSGI binary to see all dynamically linked libraries:

`$ ldd /usr/local/anaconda/bin/uwsgi
linux-vdso.so.1 (0x00007ffe9d5ec000)
libpthread.so.0 (0x00007f95e83ad000)
libm.so.6 (0x00007f95e800f000)
libdl.so.2 (0x00007f95e7e0b000)
libz.so.1 (0x00007f95e7bee000)
libpcre.so.1 => not found
libuuid.so.1 (0x00007f95e79e7000)
libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f95e777d000)
libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f95e7305000)
libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f95e6f44000)
libutil.so.1 (0x00007f95e6d41000)
libcrypt.so.1 (0x00007f95e6b09000)
libc.so.6 (0x00007f95e6718000)
/lib64/ld-linux-x86-64.so.2 (0x00007f95e85cc000)
libicuuc.so.60 => /usr/lib/x86_64-linux-gnu/libicuuc.so.60 (0x00007f95e6361000)
liblzma.so.5 (0x00007f95e613b000)
libicudata.so.60 => /usr/lib/x86_64-linux-gnu/libicudata.so.60 (0x00007f95e4592000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f95e4209000)
libgcc_s.so.1 (0x00007f95e3ff1000)`

Two things to note in this output:

* `libpcre.so.1` shows `not found`
* Most resolved libraries point to `/usr/lib/x86_64-linux-gnu/`, which is where uWSGI searches for its dependencies

Confirm the search path by checking the `LD_LIBRARY_PATH` environment variable:

`$ echo $LD_LIBRARY_PATH
/usr/lib/x86_64-linux-gnu/:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/opt/oracle/instantclient_12_1:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/opt/oracle/instantclient_12_1:`

### Step 4: Resolve the missing library

Choose one of the following options:

1. **Create a symlink from an existing versioned library in `/usr/lib/x86_64-linux-gnu/`.** If `libpcre.so.3` exists there, run:

   `ln -s /usr/lib/x86_64-linux-gnu/libpcre.so.3 /usr/lib/x86_64-linux-gnu/libpcre.so.1`
2. **Create a symlink from a library found elsewhere on the system.** Use `find` to locate available copies:

   `$ find / -name libpcre.so.1
   /usr/local/anaconda/pkgs/pcre-8.43-he6710b0_0/lib/libpcre.so.1
   /usr/local/anaconda/pkgs/pcre-8.44-he6710b0_0/lib/libpcre.so.1
   /usr/local/anaconda/lib/libpcre.so.1
   /libpcre.so`

   Then link one of those paths into the expected directory:

   `ln -s /usr/local/anaconda/lib/libpcre.so.1 /usr/lib/x86_64-linux-gnu/libpcre.so.1`
3. **Update `LD_LIBRARY_PATH`** to include the directory containing `libpcre.so.1`:

   `export LD_LIBRARY_PATH=/usr/local/anaconda/lib/:$LD_LIBRARY_PATH`

### Step 5: Verify the fix

1. Re-run `ldd` and confirm `libpcre.so.1` now resolves to a path instead of showing `not found`:

   `$ ldd /usr/local/anaconda/bin/uwsgi
   linux-vdso.so.1 (0x00007ffffc181000)
   libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f30d6d35000)
   libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f30d6997000)
   libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f30d6793000)
   libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f30d6576000)
   libpcre.so.1 => /lib/x86_64-linux-gnu/libpcre.so.1 (0x00007f30d6304000)
   libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f30d60fd000)
   libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f30d5e93000)
   libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f30d5a1b000)
   libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f30d565a000)
   libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f30d5457000)
   libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f30d521f000)
   libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f30d4e2e000)
   /lib64/ld-linux-x86-64.so.2 (0x00007f30d6f54000)
   libicuuc.so.60 => /usr/lib/x86_64-linux-gnu/libicuuc.so.60 (0x00007f30d4a77000)
   liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f30d4851000)
   libicudata.so.60 => /usr/lib/x86_64-linux-gnu/libicudata.so.60 (0x00007f30d2ca8000)
   libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f30d291f000)
   libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f30d2707000)`

   `libpcre.so.1` now resolves to `/lib/x86_64-linux-gnu/libpcre.so.1`.
2. Start uWSGI and confirm it runs without the shared library error:

   `$ uwsgi
   Starting uWSGI 2.0.19.1 (64bit) on [Fri Oct 9 22:49:20 2020] ***
   compiled with version: 4.8.5 on 09 October 2020 19:29:50
   os: Linux-4.4.0-1075-aws #85-Ubuntu SMP Thu Jan 17 17:15:12 UTC 2019
   nodename: domino-run-5f80e2df4cedfd0008f5d7d4
   machine: x86_64
   clock source: unix
   pcre jit disabled
   detected number of CPU cores: 4
   current working directory: /lib/x86_64-linux-gnu
   detected binary path: /usr/local/anaconda/bin/uwsgi
   WARNING: you are running uWSGI without its master process manager ***
   your memory page size is 4096 bytes
   detected max file descriptor number: 4096
   lock engine: pthread robust mutexes
   thunder lock: disabled (you can enable it with --thunder-lock)
   The s/-socket option is missing and stdin is not a socket.`

   uWSGI starts successfully without the shared library error.

If this solution doesn't resolve your issue, please contact Domino Support.


# vm_memory_high_watermark" error in RabbitMQ service logs
Source: https://docs.domino.ai/knowledge-base/vm-memory-high-watermark-error-in-rabbitmq-service-logs

Resolve RabbitMQ vm_memory_high_watermark memory alarms in Domino on Kubernetes by restarting the affected pod or permanently raising the absolute memory threshold in the ConfigMap.

## **Issue:**

Examining the logs for the RabbitMQ StatefulSet/rabbitmq-ha-XXX shows the following error:

```
[info] <0.449.0> vm_memory_high_watermark set. Memory used:1858117632 allowed:1717986918
[warning] <0.447.0> memory resource limit alarm set on node 'rabbit@rabbitmq-ha-311-0.rabbitmq-ha-discovery.example.svc.cluster.local'.
[warning] <0.447.0>
[warning] <0.447.0> **********************************************************
[warning] <0.447.0> *** Publishers will be blocked until this alarm clears ***
[warning] <0.447.0> **********************************************************
[warning] <0.447.0>
[warning] <0.447.0> memory resource limit alarm cleared on node 'rabbit@rabbitmq-ha-311-0.rabbitmq-ha-discovery.example.svc.cluster.local'
[warning] <0.447.0> memory resource limit alarm cleared across the cluster
[warning] <0.447.0> memory resource limit alarm set on node 'rabbit@rabbitmq-ha-311-0.rabbitmq-ha-discovery.example.svc.cluster.local'.
```

This condition typically surfaces in the Domino frontend as an error similar to:

> "Workspace could not be started. Data plane Local is not in a healthy state: DataPlaneStatus(Disconnected, Not responding, Some(1700470196563))"

## **Root Cause:**

When RabbitMQ reaches the vm\_memory\_high\_watermark threshold, it sets a memory resource limit alarm on the affected node and stops accepting new connections. The data-plane-agent service detects its connections to RabbitMQ are refused and marks itself as unhealthy, which causes the frontend error above.

To confirm this in the logs, retrieve the RabbitMQ container logs using one of the following methods.

**Recent versions of Domino** — stream logs directly from the Kubernetes container:

```
kubectl logs -n <your-domino-platform-namespace> rabbitmq-ha-311-0 --follow
```

> **Note:** The pod name rabbitmq-ha-311-0 is illustrative. Replace it with the name of the affected pod in your environment.

**Older versions of Domino** — exec into the container and read the log file:

```
kubectl exec -it -n <your-domino-platform-namespace> rabbitmq-ha-311-0 -c rabbitmq-ha -- /bin/bash
cat /var/log/rabbitmq/*log
```

Look for vm\_memory\_high\_watermark set and memory resource limit alarm set entries to confirm the root cause.

## **Resolution:**

**Temporary fix**

Restart the affected RabbitMQ pod to clear the alarm and restore operations. Use the following command, replacing the pod name with the affected pod in your environment:

```
kubectl delete pod -n <your-domino-platform-namespace> rabbitmq-ha-311-0
```

The pod restarts automatically. However, memory usage typically increases over time and the alarm recurs without a permanent configuration change.

**Permanent fix**

Increase the vm\_memory\_high\_watermark setting in the rabbitmq-ha ConfigMap to set an absolute memory limit appropriate for your container's resource limits.

The default value is vm\_memory\_high\_watermark.relative = 0.4, which allocates 40% of available memory. This default protects against out-of-memory (OOM) situations. Where container memory limits allow, replacing the relative value with an explicit absolute value provides more predictable behavior. For example, on a container with a 6 GB memory limit, 3500MB is an acceptable threshold.

**Step 1: Edit the ConfigMap.**

```
kubectl edit cm -n <your-domino-platform-namespace> rabbitmq-ha
```

**Step 2: Replace the relative setting with an absolute value.**

```
## Memory Threshold
# vm_memory_high_watermark.relative = 0.4
vm_memory_high_watermark.absolute = 3500MB
```

**Step 3: Restart the RabbitMQ pods to apply the change.** Run the following command to perform a rolling restart of the StatefulSet:

```
kubectl rollout restart statefulset -n <your-domino-platform-namespace> rabbitmq-ha
```

**Step 4: Verify the new watermark is active.** After the pods restart, check the logs for confirmation that the updated value loaded:

```
kubectl logs -n <your-domino-platform-namespace> <rabbitmq-pod-name> | grep vm_memory_high_watermark
```

Confirm the output references the absolute value you set (for example, 3500MB) rather than the previous relative value.


# VSCode Failed to get latest version {"error":"getaddrinfo ENOTFOUND api.github.com"}
Source: https://docs.domino.ai/knowledge-base/vscode-failed-to-get-latest-version-error-getaddrinfo-enotfound-api-github-com

Explains VSCode "getaddrinfo ENOTFOUND api.github.com" and unknown IP log errors in air-gapped Domino environments, with settings to reduce network calls.

# VSCode error: "Failed to get latest version — getaddrinfo ENOTFOUND api.github.com"

When you run VSCode (code-server) in a closed or air-gapped environment, you may see error messages in your logs. These errors are benign — the Domino Standard Environment (DSE) doesn't rely on internet resources to function.

## Failed to get latest version

The error `Failed to get latest version {"error":"getaddrinfo ENOTFOUND api.github.com"}` occurs because code-server attempts to update itself on startup. You can see the relevant code in [code-server/update.ts](https://github.com/cdr/code-server/blob/e07ae004b73823483e9f7015548dc7d38334558a/src/node/update.ts#L64).

Apply the following network-related settings to suppress these update attempts.

### Apply the settings

1. Open `$DOMINO_WORKING_DIR/.vscode/User/settings.json` in your editor.
2. Add or merge the following configuration into the file:

   ```
   {
     "update.mode": "none",
     "extensions.autoUpdate": false,
     "workbench.enableExperiments": false,
     "extensions.autoCheckUpdates": false,
     "json.schemaDownload.enable": false,
     "update.showReleaseNotes": false,
     "npm.fetchOnlinePackageInfo": false,
     "workbench.settings.enableNaturalLanguageSearch": false
   }
   ```
3. Save the file and restart VSCode.

> **Note:** This workaround has not been verified in a fully air-gapped environment. Results may vary.

## Unknown IP addresses in logs

The IP addresses `142.250.179.176` and `216.58.214.16` appear in the logs because code-server tries to update the extensions that ship with the DSE. This behavior originates from the `downloadInstallableExtension` function in code-server.


# Why did my execution fail with "No space left on device"?
Source: https://docs.domino.ai/knowledge-base/why-did-my-execution-fail-with-the-error-no-space-left-on-device

Resolve 'No space left on device' execution failures in Domino by freeing disk space, moving files to Datasets, or increasing the allocated disk size (v4.4+).

**Applies to:** All Domino versions. Disk size configuration in project settings is available from Domino v4.4 and later. The default disk size is 10 GB for versions before v4.4 and 15 GB for v4.4 and later.

## **Issue**

An execution fails with the following error in the logs:

```
java.lang.IllegalStateException: Blob transfer (download) failed because 24 operations failed:
Download blob -> java.io.IOException: No space left on device
```

The execution has exceeded the available disk space on the compute instance.

## **Root Cause**

Each execution runs inside a compute instance with a finite amount of disk space. Total disk usage combines project files, external Git repository contents, the compute environment image, and any data generated during the run. When combined usage exceeds the available disk size, the execution fails.

Crash or dump files left behind by integrated development environments (IDEs) or libraries can also consume large amounts of disk space without being immediately visible.

## **Resolution**

### **For users**

1. **Switch to a different hardware tier.** This moves the execution to a different compute instance with available disk space. It's the quickest fix if you don't have admin privileges.
2. **Reduce project data.** Remove unnecessary files from the project. Git repository contents count toward disk usage.
3. **Move large files to Datasets.** Move large project files to a Domino Dataset rather than keeping them in the project directory.
4. **Check for crash or dump files.** Jupyter and other IDEs can crash and leave large files in the workspace that aren't immediately visible.
5. **Increase the disk size (v4.4 and later).** In your project, go to **Settings → Hardware & Environment → Workspace and Jobs Volume Size** and increase the allocated disk size. Changes apply to new workspaces only and don't affect currently running sessions.

### **For admins**

1. Place the affected executor into **Maintenance Mode** to take it out of service.
2. In the Domino Dispatcher UI, locate the executor and verify that no executions are currently running on it. Then terminate the executor to release the storage. Because executors are ephemeral, terminating one is a safe and fast resolution.

> **Note:** Garbage collection may also recover disk space on the executor without requiring termination, but termination is the fastest option. If this issue recurs, consider increasing the default disk size using the Central Config setting: com.cerebro.domino.computegrid.kubernetes.volume.volumesSizeInGB

### **Additional information**

**Domino v4.0 and later:** Each execution gets its own dedicated volume and is no longer affected by shared resource contention from other executions. Users on v4.4 and later can adjust their individual disk size in project settings.

**Executor maintenance mode:** Domino polls executors for disk status and automatically places them into Maintenance Mode when a partition has less than 10% free space. A common scenario is that the disk sits just above this threshold and an execution fills the remaining space. When this happens, new executions on that executor can fail repeatedly until an admin intervenes.

**Domino v3.6 and earlier:** On startup, Domino copies the project files and the compute environment Docker image to a partition on the executor. If this data exceeds the available disk space, the execution fails on startup. Long-lived executors can also accumulate cached project data and environment images over time, gradually filling the disk.


# Yarn Package invalid key
Source: https://docs.domino.ai/knowledge-base/yarn-package-invalid-key



### **Version/Environment (if relevant):**

Not Domino version specific. Applies to any Domino compute environment using the yarn package.

## **Issue:**

A Domino environment build fails with a GPG signature error related to the yarn package, similar to the following:

```
GPG error: https://dl.yarnpkg.com/debian stable InRelease: The following signatures
were invalid: EXPKEYSIG 23E7166788B63E1E Yarn Packaging <yarn@dan.cx>
```

## **Root Cause:**

The GPG key used to sign the yarn package repository has expired or become invalid. The environment's Dockerfile is using a stale key that can no longer be verified, causing the build to fail.

## **Resolution:**

Add one of the following to the top of the environment's Dockerfile to refresh the yarn package key. Try each option in order if the previous one does not work, as the correct approach may vary depending on your specific environment version.

**Option 1 — Standard key refresh (try this first):**

```
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
```

**Option 2 — Alternative using gpg dearmor:**

```
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg|gpg --dearmor|sudo tee /usr/share/keyrings/yarnkey.gpg>/dev/null
```

**Option 3 — Recommended if apt-key is deprecated in your environment:**

```
ENV YARNKEY=yarn-keyring.gpg
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo gpg --dearmour -o /usr/share/keyrings/$YARNKEY && \
 echo "deb [signed-by=/usr/share/keyrings/$YARNKEY] https://dl.yarnpkg.com/debian stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
```

### **Notes/Information:**

* These are best-effort suggestions that have been verified to work in a standard Domino environment. Results may vary depending on your specific environment configuration.
* `apt-key` is becoming deprecated in newer Linux distributions. If Options 1 and 2 do not work, use Option 3 which uses the modern `gpg --dearmour` approach with a signed-by keyring.
* If you are seeing a similar error for an Nvidia GPG key rather than yarn, refer to the related article: **Build error for Nvidia GPG key**.
* While Domino Support will assist where possible, environment build issues are primarily the responsibility of local admins. If these steps do not resolve the issue, the best first step is to engage your local environment administrators.


# Core concepts at a glance
Source: https://docs.domino.ai/platform-capabilities/core-concepts

The building blocks Domino provides for AI and data science work

## Ways to build

| Concept | What it is |
| :- | :- |
| <Icon icon="laptop-code" /> [Workspaces](/cloud/platform-capabilities/core-concepts/workspaces) | A self-serve interactive development environment using the IDE of your choice, designed to feel like a “laptop in the cloud.” Intended for exploratory analysis and interactive development. |
| <Icon icon="play" /> [Jobs](/cloud/platform-capabilities/core-concepts/jobs) | A batch run of a script or command, fully reproducible, for training, simulation, and data processing. Can be run manually, via APIs, or on a schedule. |
| <Icon icon="flask" /> [Experiments](/cloud/platform-capabilities/features/development/track-monitor) | Useful primarily when developing machine learning models, Experiments provide a way to track, organize, and compare different model configurations you're testing. |
| <Icon icon="diagram-project" /> [Flows](/cloud/platform-capabilities/features/flows) | Multi-step computation graphs. A good fit for data engineering tasks or MLOps tasks. |
| <Icon icon="server" /> [Compute clusters](/cloud/platform-capabilities/features/compute-clusters-hpc) | Frameworks for running highly parallelizable workloads, including Spark, Ray, Dask, and Slurm. Domino can attach clusters to Workspaces, Jobs, and other workloads and manages the necessary infrastructure, including autoscaling. |

## Use data

| Concept | What it is |
| :- | :- |
| <Icon icon="database" /> [Datasets](/cloud/platform-capabilities/core-concepts/data/datasets) | Versioned filesystem storage for large data, shared across Projects, with strong features for reproducibility. |
| <Icon icon="hard-drive" /> [NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes) | Like Datasets but backed by NetApp ONTAP storage, with near-instant snapshots regardless of size. Use them wherever your organization runs ONTAP. |
| <Icon icon="plug" /> [Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors) | Connectors to external databases, data warehouses, data lakes, and object stores such as Amazon S3. |
| <Icon icon="network-wired" /> [External Data Volumes](/cloud/platform-capabilities/core-concepts/data/external-volumes) | Network-attached storage, such as network file system (NFS) shares or Windows shares, mounted into the workloads you run. |

## Productize what you build

| Concept | What it is |
| :- | :- |
| <Icon icon="rocket" /> [Domino endpoints](/cloud/platform-capabilities/features/model-deployment) | An HTTP API wrapped around your code, meant for real-time inference tasks such as classification and scoring. |
| <Icon icon="window-maximize" /> [Apps](/cloud/platform-capabilities/features/apps) | Interactive web applications built with frameworks such as Streamlit, Dash, Shiny, Flask, or any other web framework. Domino handles routing, authentication, and scaling. |
| <Icon icon="robot" /> [Agents](/cloud/platform-capabilities/features/agents) | Programs that use a large language model (LLM) to plan and act across multiple steps. Domino hosts and serves them alongside Apps and traces them for evaluation and monitoring. |
| <Icon icon="wand-magic-sparkles" /> [Launchers](/cloud/platform-capabilities/features/launchers) | Self-serve web forms that run a predefined Job with the parameters a less technical user fills in. Useful for turning scripts into simple web interfaces. |

## Govern

| Concept | What it is |
| :- | :- |
| <Icon icon="route" /> [LLM Gateway](/cloud/platform-capabilities/features/llms/llm-gateway) | A router that lets you direct LLM calls to any underlying model, with usage tracking, cost controls, and guardrails. Contact your Domino field team for access. |
| <Icon icon="clipboard-check" /> [Governance policies](/cloud/platform-capabilities/features/governance) | Policies define the stages, required evidence, and required approvals that a work product, such as an App or model, must go through as it's built. Used to enforce standards for internal workflows and compliance policies. |
| <Icon icon="coins" /> [FinOps](/cloud/platform-capabilities/features/finops) | Manage and monitor infrastructure spend. |
| <Icon icon="clock-rotate-left" /> [Audit Trail](/cloud/platform-capabilities/features/governance/audit-trail) | A record of user activity for compliance verification, security investigations, and audit reporting. |

## Collaborate, reproduce, and manage knowledge

| Concept | What it is |
| :- | :- |
| <Icon icon="folder-open" /> [Projects](/cloud/platform-capabilities/core-concepts/projects) | A shared space for people to collaborate with access to the same set of code, data, results, and history. The primary way of organizing work in Domino. |
| <Icon icon="rotate" /> [Reproducibility](/cloud/platform-capabilities/core-concepts/reproducibility) | Domino keeps records of all the "inputs" for most deliverables and results you create in the platform: the code, data, and Compute Environment that produced a given result or Model or App. |
| <Icon icon="magnifying-glass" /> [Search](/cloud/platform-capabilities/features/collaboration/domino-search) | Domino indexes contents in the platform so you can find and reuse past work easily. |
| <Icon icon="copy" /> [Project templates](/cloud/platform-capabilities/core-concepts/projects/create-projects#create-a-project-from-a-template) | A Project pre-configured with code, data, and other materials, to be used as a starting point for new Projects to help people use best practices and avoid re-inventing the wheel. |
| <Icon icon="tags" /> [Tags and Properties](/cloud/platform-capabilities/core-concepts/knowledge-management) | Tags are controlled, hierarchical labels for categorizing and finding assets. Properties are typed custom fields that enrich them. |

## Architecture and administration

| Concept | What it is |
| :- | :- |
| <Icon icon="cubes" /> [Compute orchestration](/cloud/admin/architecture-and-deployment/architecture) | Domino runs every workload as a container on Kubernetes. A cluster autoscaler adds compute capacity as demand grows. |
| <Icon icon="cloud" /> [Hybrid and multicloud compute](/cloud/platform-capabilities/core-concepts/data-plane) | One Domino deployment can run workloads in several regions, clouds, or on-premises data centers, each a Data Plane attached to the same Control Plane. |
| <Icon icon="box" /> [Compute Environments](/cloud/platform-capabilities/core-concepts/compute-environments) | A versioned, shareable definition of the tools and packages a workload runs in, built from a Docker image, so everyone runs the same software. |
| <Icon icon="microchip" /> [Hardware Tiers](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers) | Administrator-defined compute profiles (CPU, memory, GPU) that you pick when launching a workload. Access can be limited to an organization. |


# Domino 6.3 release notes
Source: https://docs.domino.ai/release-notes/6-3-0

Domino 6.3 release notes: on-demand Slurm HPC compute, a full App SDLC, coding assistants in Workspaces, tags and properties, governance gates, and Extensions.

Domino 6.3 brings on-demand Slurm compute for high-performance computing (HPC) workloads, a full [App](/cloud/reference/glossary#app) software development lifecycle (SDLC), coding assistants in [Workspaces](/cloud/reference/glossary#workspace), [tags](/cloud/reference/glossary#tag) and [Properties](/cloud/reference/glossary#property) for knowledge management, governance enforcement for Apps and [agents](/6.3/platform-capabilities/features/agents), and reusable [Extensions](/cloud/reference/glossary#extension).

## Highlights

### On-demand Slurm compute for HPC workloads

Run existing HPC job scripts from a Workspace while Domino provisions Slurm workers on demand and tracks [HPC Jobs](/cloud/reference/glossary#hpc-job) with other [Project](/cloud/reference/glossary#project) [Jobs](/cloud/reference/glossary#job). Domino deploys Slurm on Kubernetes with [Slinky](https://slinky.schedmd.com/), SchedMD's Slurm operator, so each [Data Plane](/cloud/reference/glossary#data-plane) runs a single persistent Slurm cluster with worker capacity that scales with demand. See [Run HPC workloads with Slurm](/6.3/platform-capabilities/features/compute-clusters-hpc/slurm).

### Full App SDLC

Take Apps from private draft to an immutable published [App version](/cloud/reference/glossary#app-version) with distinct create, publish, and deploy stages. Pinned commits make versions reproducible, restartable, and easier to roll back; see the [App lifecycle](/6.3/platform-capabilities/features/apps/apps-in-domino/app-structure-lifecycle#app-lifecycle) and [App versioning and reproducibility](/6.3/platform-capabilities/features/apps/app-versioning-and-reproducibility).

### Coding assistants in Workspaces

Start with GitHub Copilot, Claude Code, and Codex pre-installed in Domino Standard Environment (DSE) Workspaces, together with Domino skills for common platform tasks. See [Coding assistants](/6.3/platform-capabilities/features/coding-assistants) for availability by IDE and customization options.

### Knowledge management with tags and properties

Give teams a shared way to organize and discover Domino assets. SysAdmins and [Librarians](/6.3/admin/identity-and-access/manage-users/roles) can define tags and typed Properties, while users can find classified assets through search, filters, and the Tag Browser; see [Knowledge management](/6.3/platform-capabilities/core-concepts/knowledge-management).

### Governance enforcement for Apps & Agents

Turn governance requirements into deployment [gates](/cloud/reference/glossary#gate) for Apps and agents by linking them to [governance bundles](/cloud/reference/glossary#bundle). Required [policies](/cloud/reference/glossary#policy) can block deployment, while App versions carry [Evidence](/cloud/reference/glossary#evidence) forward with version diffs; see [Govern models, Apps, agents, and files](/6.3/platform-capabilities/features/governance/work-with-bundles/govern-models-apps-files).

### Extensions

Extensions bring Apps into Project, [Dataset](/cloud/reference/glossary#dataset), [Model](/6.3/platform-capabilities/features/models), file, and Admin pages. [Build your own Extensions](/6.3/platform-capabilities/features/extensions/create-manage-extensions), or install AutoML Studio, Auto Model Documentation, and Clinical Data Explorer from the [official Extensions catalog](/6.3/platform-capabilities/features/extensions/install-domino-official-extensions).

<hr />

## Flows

### Runtime configuration for flow runs

Rerun a registered [Flow](/cloud/reference/glossary#flow) with different inputs, [Compute Environments](/cloud/reference/glossary#compute-environment), [Hardware Tiers](/cloud/reference/glossary#hardware-tier), or Git references without editing its definition. Apply the same overrides from the UI, API, or CLI; see [Launch Flows](/6.3/platform-capabilities/features/flows/launch-flows#relaunch-an-existing-flow).

### Consistent paths and secondary repositories in Flow tasks

Use the same input and output paths across Flows, Workspaces, and Jobs. Flow tasks can also pin a branch, tag, or commit from an imported Git repository; see [Define Flows](/6.3/platform-capabilities/features/flows/define-flows#write-task-code).

<hr />

## Governance

### Governance bundle lifecycle and review controls

See compliance status at a glance and move completed bundles to **Completed** or **Archived**. Reviewers can reset sequential bundles, replace their primary policy, and select eligible Project collaborators as approvers; see [Work with bundles](/6.3/platform-capabilities/features/governance/work-with-bundles).

### Governance stage notifications and reassignment

Keep reviews moving with email and in-product notifications when stages start. A global **Stages** view shows status and assignees and supports bulk reassignment; see [Manage stages across bundles](/6.3/platform-capabilities/features/governance/work-with-bundles#manage-stages-across-bundles).

### Flow artifacts as governance attachments

Capture Flow outputs directly in governance bundles. Flows can attach files automatically, or users can attach them from a run's **Artifacts** tab; see [Flow artifact files](/6.3/platform-capabilities/features/governance/work-with-bundles/attachments#flow-artifact-files).

### Periodic revalidation for Governance

Keep [Approvals](/cloud/reference/glossary#approval) current with a recurring, organization-wide renewal schedule. Bundles remain compliant during the renewal window and become non-compliant if approval expires; see [Periodic revalidation](/6.3/platform-capabilities/features/governance/approvals#periodic-revalidation).

### Policy versioning for Governance

Develop changes to a published policy in a draft, then choose whether the update is mandatory and whether existing Approvals remain valid. Compare versions from the **Changes** tab; see [Create a new version of a policy](/6.3/platform-capabilities/features/governance/define-governance-policies#create-a-new-version-of-a-policy).

<hr />

## Identity and API access

### Personal Access Tokens replace user API keys

Move user-context API access to expiring personal access tokens (PATs) that administrators can audit and revoke. Starting in 6.3, PATs are the supported replacement for legacy user API keys, which remain functional but are deprecated; see [Personal access tokens](/6.3/admin/identity-and-access/manage-users/personal-access-tokens).

### Cluster-specific API documentation

Explore cluster-specific API documentation directly at `https://<your-domino-domain>/docs`. See the [API browser](/6.3/reference/api/domino-open-api#api-browser) for details.

<hr />

## Notes for platform administrators

* **Slurm support is enabled when a Data Plane is deployed.** Set `compute_clusters.slurm.enabled` to `true` when deploying the [Data Plane](/cloud/reference/glossary#data-plane); adding Slurm later requires a redeployment. Create the required Compute Environments, worker Hardware Tiers, and shared Dataset, and enable Data Plane storage and Workspaces for remote Data Planes; deployments that leave Slurm disabled require no action. See [Administer Slurm](/6.3/admin/infrastructure-and-compute/administer-slurm).

- **HPC Job submission from the UI and API is off by default.** Set [`ShortLived.EnableHPCJobs`](/6.3/admin/configuration/feature-flags#param-short-lived-enable-hpc-jobs) to `true` to add a **Run** button to the **HPC** tab. Terminal submission needs no flag; see [Administer Slurm](/6.3/admin/infrastructure-and-compute/administer-slurm).

* **Configuration records have Basic and Advanced views.** **Basic** provides validated controls for common settings, while **Advanced** retains the full editor. The new interface is off by default; set [`EnableNewCentralConfigsEditor`](/6.3/admin/configuration/feature-flags#param-enable-new-central-configs-editor) to `true` to enable it without changing existing values, or leave it off with no action required. See [Basic and Advanced views](/6.3/admin/configuration/central-configuration#basic-and-advanced-views).

- **Administrators can start another user's stopped Workspace through the API.** The `adminStartWorkspace` operation requires `ManageWorkspaces` and preserves the Workspace owner's previous mounts. Pair it with the existing stop operation to automate restarts; see [Start a user's Workspace as an administrator](/6.3/platform-capabilities/core-concepts/workspaces/manage-workspaces#start-a-users-workspace-as-an-administrator).

* **The platform Grafana instance is operator-managed.** Manage custom dashboards, alerts, contact points, and notification policies at `/grafana` as Kubernetes resources. Don't edit Domino-managed resources because upgrades overwrite them; deployments without custom Grafana content require no action. See [Manage Grafana dashboards and alerts as custom resources](/6.3/admin/operations/monitoring/grafana/manage-with-custom-resources).

- **Legacy Grafana remains available during the 6.3.x migration window.** Migrate custom content from `/grafana-classic` to `/grafana` before upgrading beyond 6.3.x; see [Access the legacy Grafana instance during migration](/6.3/admin/operations/monitoring/grafana/access_grafana#access-the-legacy-grafana-instance-during-migration) and [Migrate Grafana dashboards and alerts to the operator-managed instance in Domino 6.3](https://support.domino.ai/support/s/article/Migrate-Grafana-dashboards-and-alerts-to-the-operator-managed-instance-in-Domino-6-3-1789393086479467000) on the Support site (requires sign-in).

* **Deployment identification for product analytics.** Set the optional `customer_name` and `deployment_name` values to identify your organization and environment in product analytics and Support cases. Changes take effect after Domino services restart; see [Deployment identification](/6.3/admin/configuration/deployment-identification).

## Bug fixes

## Known issues

<AccordionGroup>
  <Accordion title="Coding assistants">
    * When working with Workspace coding assistants, if you are authenticating a coding agent for the first time, the link in the popup dialog does not work. To complete authentication, copy the URL displayed in the terminal and open it in your browser.

    * Domino skills might not work by default for coding agents other than Claude Code.

    * In some cases, the VS Code Claude Code extension panel appears blank. To fix this, refresh your browser.

    * Claude Code can default to a model tier other than the one you prefer. Use the model selector at the top of the Claude Code panel to switch tiers.
  </Accordion>

  <Accordion title="Workspaces">
    * When restarting an RStudio workspace with home directory persistence enabled, the user may see a notice stating that the previous session was abnormally terminated due to a crash. This notice is only due to how the new feature interacts with RStudio and does not result in any data loss.
  </Accordion>

  <Accordion title="Flows">
    * [Flows](/cloud/reference/glossary#flow) are supported in Domino Cloud and self-managed deployments on AWS and Azure. Google Cloud and on-premises deployments aren't supported.

    - Some Flyte features are not yet available in Flows, such as [dynamic workflows](https://docs.flyte.org/en/latest/user_guide/advanced_composition/dynamic_workflows.html) and [map\_task](https://docs.flyte.org/en/latest/user_guide/advanced_composition/map_tasks.html).
  </Accordion>

  <Accordion title="Azure Blob Storage">
    * In Azure Blob Store deployments, Projects with many files may fail to sync through the Domino CLI. To work around this issue, do not disable file locking when prompted by Domino.

    - When uploading a large file to the Azure blob store by [syncing a Workspace](/6.3/platform-capabilities/core-concepts/workspaces/sync-changes-in-a-workspace), you may encounter a Java Out of Memory error from Azure if the file/blob already exists. To work around this issue, use the [Domino CLI](/6.3/reference/cli) to upload the file to the Project.

    * Domino instances that make use of Azure Blob Storage may experience stalled Jobs within Projects with many large files.
  </Accordion>

  <Accordion title="Projects and files">
    * If you attach a Git repository to a DFS Project that points to a tagged release, the tag is not honored when building a Domino endpoint in that Project. The build log shows an error similar to the following, and the model is built using the default branch of your Git repository instead of the tagged branch:

      ```
      Jul 05 2023 14:36:27 -0500 #10 6.481 WARN [d.r.d.GitRepoUpdater] could not parse ref: v1.3.0 checking out default branch correlationId="iA2qWrYSLQ" thread="main"
      ```

      To work around this issue, use the branch name when building Domino endpoints instead of the release tag.
  </Accordion>

  <Accordion title="Sessions and authentication">
    * If idle timeout is configured on a deployment, Domino may not recognize pages opened in spawned tabs (for example, via `Ctrl+Click`) or outside of an iFrame as being active. Users may be unintentionally logged out despite activity in the spawned pages.

    - If an admin resets a user’s password, it invalidates all the user’s authentication tokens, including tokens used for long-running tasks like Jobs, Workspaces, or Apps. The user must create a new password, log back into Domino, and restart all executions. This also applies to CLI authentication; the user must re-login to their Domino CLI.
  </Accordion>

  <Accordion title="Monitoring">
    * [Model Monitoring](/cloud/reference/glossary#model-monitoring) data sources aren’t validated. If you enter an invalid bucket name and attempt to save, the entry goes through. However, you can’t see metrics for that entry, because the name points to an invalid bucket.

    - Agent monitoring dashboard retains only the current day’s data (charts have missing data and traces are gone). Historical data from previous days are lost, regardless of what time window is selected.
  </Accordion>

  <Accordion title="Platform and infrastructure">
    * During a 48-hour soak test, the `nginx-ingress-controller` pods exhibited a gradual increase in memory usage while handling heavy [Domino endpoint](/cloud/reference/glossary#domino-endpoint) workloads. The cause is under investigation.
  </Accordion>
</AccordionGroup>

## Validated frameworks

Domino 6.3.0 validates the following framework versions. Other versions might be compatible but aren't guaranteed.

| Framework | Version |
| :- | :- |
| [Kubernetes](/6.3/admin/architecture-and-deployment/kubernetes-infrastructure#compatibility-matrix) | See the compatibility matrix |
| [Apache Spark](/6.3/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/validated-spark-version) | 4.1.2 |
| [Dask](/6.3/platform-capabilities/features/compute-clusters-hpc/dask/validated-dask-version) | 2026.3.0 |
| [Ray](/6.3/platform-capabilities/features/compute-clusters-hpc/ray/validated-ray-version) | 2.55.1 |

## Upgrade notes

<Warning>
  Domino upgrades its PostgreSQL service databases from version 11 to 17 during the 6.3.0 upgrade. Before starting the upgrade, take a full filesystem-level backup of each PostgreSQL persistent volume. The migration changes data in place and can't automatically roll back after it starts. Coordinate the backup with your Domino representative and plan a longer maintenance window than usual for the database upgrade.
</Warning>

<Warning>
  NGINX ingress now runs a Federal Information Processing Standards (FIPS)-validated OpenSSL that drops the legacy static-RSA TLS ciphers. If you set an explicit `ssl-ciphers` override and front NGINX with an AWS Classic Load Balancer over TLS, update it before upgrading or the installer stops with a configuration error. Default installs and Network Load Balancer (NLB) or Application Load Balancer (ALB) deployments are unaffected. See [Deploy Domino on AWS EKS](/6.3/admin/installation/manage-deployments/aws/deploy).
</Warning>

This release supports the following versions in the [Kubernetes compatibility matrix](/6.3/admin/architecture-and-deployment/kubernetes-infrastructure#compatibility-matrix):

| Platform | Kubernetes version |
| :- | :- |
| Red Hat OpenShift | 1.34 (OpenShift 4.21) |
| Google GKE | 1.34, 1.35 |
| Azure AKS | 1.34, 1.35 |
| Amazon EKS | 1.34, 1.35 |
| Nexus data plane (non-vCluster on Amazon EKS) | 1.34, 1.35 |

## Breaking changes & deprecations

* **Legacy Domino user API keys are deprecated.** Existing keys still work in 6.3, but migrate existing integrations and build new ones with a Personal Access Token or [Domino Service Account](/6.3/admin/identity-and-access/manage-users/service-accounts); see [Domino API authentication](/6.3/reference/api/domino-api-authentication).

- **The `project.tag.approved=` search operator is removed.** Filter by tag from the search bar or filter panel instead; `project.tag=` is unchanged. See [Use Domino search](/6.3/platform-capabilities/features/collaboration/domino-search#filter-results-by-tag).

* **Project Tasks, Goals, and the Project Stages and Status UI are removed.** Use your organization's issue tracker for task-level milestones; the read-only Project Status badge, [Activity feed](/6.3/platform-capabilities/core-concepts/projects/collaborate-on-projects#view-project-activity), and backend Goal APIs remain available. Domino migrates existing Project goal comments to bundle comments during the upgrade.

- **The Project-level Jira integration UI is removed.** Project-to-ticket links and task synchronization are no longer shown, while the backend Jira APIs and deployment-wide OAuth registration remain unchanged.

* **The legacy platform Grafana instance at `/grafana-classic` is deprecated.** Migrate custom content to `/grafana` before upgrading beyond 6.3.x; see [Access Grafana](/6.3/admin/operations/monitoring/grafana/access_grafana) and [Migrate Grafana dashboards and alerts to the operator-managed instance in Domino 6.3](https://support.domino.ai/support/s/article/Migrate-Grafana-dashboards-and-alerts-to-the-operator-managed-instance-in-Domino-6-3-1789393086479467000) on the Support site (requires sign-in).

Additional component updates are available in [ddlctl and fleetcommand-agent Release Notes](/6.3/admin/installation/releases).


# Domino Cloud release notes
Source: https://docs.domino.ai/release-notes/cloud-release

What shipped in each Domino Cloud release, for practitioners and platform administrators.

This page includes release notes and updates for [Domino Cloud](/cloud/platform-overview/deployment-options#domino-cloud). Keep track of upcoming changes, new features, and improvements.

Releases are listed newest first. Scroll down for previous Cloud releases. For self-managed release notes, see [Domino 6.3](/release-notes/6-3-0).

## September 2026

September 30, 2026

Cloud Release 93 lets Governance visibility rules read numeric answers, sends in-app notifications for Governance events, adds a **Trigger** column to the [Jobs](/cloud/reference/glossary#job) table, and redesigns the Git reference picker. It also reorganizes access settings for [Domino endpoints](/cloud/reference/glossary#domino-endpoint) and narrows what endpoint Viewers can see.

### Governance

#### Numeric answers in visibility rules

[Policy](/cloud/reference/glossary#policy) visibility rules can now read answers to `numeric` questions through a `numericInputs` map, so a rule can show a question or evidence set only above a threshold, such as `numericInputs["drift-score"] > 0.75`. Previously, answers to `numeric` questions weren't available to visibility rules. The existing `atoi` and `atof` helpers still convert numeric-looking string answers from other question types. See [Visibility rules](/cloud/platform-capabilities/features/governance/define-governance-policies/policy-components#visibility-rules).

#### In-app notifications for Governance events

Governance now sends an in-app Domino notification alongside the existing email when an [approval](/cloud/reference/glossary#approval) is requested, a [finding](/cloud/reference/glossary#finding) is assigned to you, a finding review is requested, a finding reaches a final state, or a [bundle](/cloud/reference/glossary#bundle) [stage](/cloud/reference/glossary#stage) is assigned to you. The navigation bar alerts you to new notifications. See [View event notifications](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#view-event-notifications).

<hr />

### Jobs, Projects, and development

#### Trigger column in the Jobs table

The Jobs table in a [Project](/cloud/reference/glossary#project) has a new **Trigger** column that shows **Scheduled** or **Manual** for each Job. Previously, a clock icon and "Scheduled by" hover text in the **User** column marked a [Scheduled Job](/cloud/reference/glossary#scheduled-job) run; the **User** column now shows only the user's name. See [Jobs Dashboard in a Project](/cloud/platform-capabilities/core-concepts/jobs/create-and-run-jobs#jobs-dashboard-in-a-project).

#### Git tag in Job details

When a Job runs from a Git tag, the code panel in its details shows a **Tag** field alongside **Commit** and **Branch**. The field shows `--` when the Job didn't use a tag.

#### Redesigned Git reference picker

The Git reference picker, used when you create an [App](/cloud/reference/glossary#app), run a Job, submit a high-performance computing ([HPC](/cloud/reference/glossary#hpc-job)) Job, or run a [flow](/cloud/reference/glossary#flow) or override a flow node, is redesigned. Choose a reference type, then enter or search for the matching branch, commit, or tag. The options are renamed: **Git repo default branch** (formerly **Use default branch** or **Use repo default git branch**), **Other branch** (formerly **Branch**), **Commit ID**, **Tag**, **Custom Git reference** (formerly **Custom Git Ref**), and **Project default branch** (formerly **Use project default git branch**).

#### Spaces become hyphens in Project names

When you type a Project name while creating a Project or copying a [Git-based Project](/cloud/reference/glossary#git-based-project), Domino replaces spaces with a hyphen, and a run of spaces becomes one hyphen. Other invalid characters still show a validation error. See [Create Projects](/cloud/platform-capabilities/core-concepts/projects/create-projects).

<hr />

### Apps, endpoints, data, and environments

#### Endpoint access settings on one tab

The **Access & Sharing** tab in a Domino endpoint's **Settings** is renamed **Access** and now sets default access and named collaborators in one form with a single **Save** button. Default access options are **Anyone in Domino**, **Discoverable**, and **Restricted**, and collaborators get the **Viewer**, **Editor**, or **Owner** role. An endpoint must keep at least 1 Owner. The **Invocation** tab is marked **Deprecated**, but you still use it to choose **Unrestricted** or **Restricted** invocation and to add endpoint access tokens. See [Domino endpoint security](/cloud/platform-capabilities/features/model-deployment/security).&#x20;

#### Dataset file list keeps filters and pagination

The file list of a [Dataset](/cloud/reference/glossary#dataset) now keeps its filter, sort order, page, and page size when you open a file preview and go back.

#### Read-only toolbar for NetApp Volume snapshots

When you view a snapshot of a [NetApp Volume](/cloud/reference/glossary#netapp-volume), the **Files** toolbar offers only **Download selected files** and **Delete Snapshot**, because snapshots are read-only. The latest view keeps **Take Snapshot**, **Upload files**, and the **Actions** menu. See [View and edit NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/view-and-edit-netapp-volumes).

#### Environment properties in the details sidebar

On a [Compute Environment's](/cloud/reference/glossary#compute-environment) details page, [Property](/cloud/reference/glossary#property) values move from their own tab into the details sidebar, next to the description, environment base, visibility, and [tags](/cloud/reference/glossary#tag). The remaining tabs are **Overview**, **Revisions**, **Projects**, **Endpoints**, and **Dependency Tree**.

#### Request access from App and agent tabs

If you can't access an App's or [agent's](/cloud/platform-capabilities/features/agents) inputs or monitoring data, the **Inputs** and **Monitoring** tabs now say so and offer a **Request access** button, which changes to **Access pending** or **Access denied** as your request is handled. The API URL in the details sidebar appears only to users with access to the App.

<hr />

### Notes for platform administrators

* **Users sign in again after this update.** Browser sessions from before Cloud Release 93 end when the update reaches your deployment, so users need to sign in again. No action is required.

- **Starburst data stores stay visible when Starburst is off.** When you create a [Data Source](/cloud/reference/glossary#data-source) from **Admin** > **Data sources** and Starburst isn't enabled, the data store list shows the Starburst data stores as disabled, below the native ones, with a notice that on Domino Cloud tells you to contact Domino to provision a Starburst cluster. No action is required.

* **Execution support bundles include resource usage.** Support bundles now contain `resource-usage.json`, with CPU and memory usage over the run, percentile summaries, disk usage, [Hardware Tier](/cloud/reference/glossary#hardware-tier) limits, and GPU usage when the run used GPUs and metrics are enabled, plus `service-metadata.json`. No action is required.

* **Security fixes.** This release remediates known vulnerabilities in the dependencies of platform components. No action is required.

### Bug fixes

* `workerStorageMB` in responses from `POST /api/jobs/v1/jobs`, `GET /api/jobs/beta/jobs`, and `GET /api/jobs/beta/jobs/{jobId}` now reports megabytes. Previously it reported megabits, eight times the actual value, so 5 MB came back as `40`. If your code divided this value by 8, remove that conversion.

- Adding an [Organization](/cloud/reference/glossary#organization) as a collaborator on a Domino endpoint no longer breaks the endpoint's overview page.

* When an ONTAP create or delete call fails, NetApp Volumes no longer end up existing in Domino but not in ONTAP, or the reverse.

- Runs no longer fail with "Execution pod not found" while preparing when Domino hasn't yet registered a pod that exists.

* Signing in through Domino Connect no longer removes Organization memberships that come from single sign-on (SSO).

- Administrators with the `ManageOrganizations` permission can view and edit the members of any Organization through `GET /v4/organizations/{organizationUserId}` and `PUT /v4/organizations/{organizationUserId}/members` without being a member.

* The **Last Used** value in the admin Dataset snapshot deletion windows shows when a snapshot was last mounted, or `--` if it never was, instead of its creation time.

- For a failed App in a [Workspace's](/cloud/reference/glossary#workspace) [App Preview](/cloud/reference/glossary#app-preview), the **Support bundle** section has a working **Download** link.

* In the Governance [evidence](/cloud/reference/glossary#evidence) notebook, the stage sidebar and the question area scroll independently.

- Governance date fields save automatically when you leave the field.

* The **Added** column in a bundle's attachments table shows the time an attachment was added, not only the date.

- The **Status** column in the user and admin personal access token tables is renamed **Expiration date**, matching what it shows.

* A flow artifact's download button reads **Download (n) files**, or **Download file** for a single file, making clear that it downloads the whole artifact.

- Long [Launcher](/cloud/reference/glossary#launcher) descriptions in the run dialog wrap between words instead of breaking mid-word.

* Long App and bundle names in the Apps & Agents table truncate to one line and show the full name when you hover over them.

- Long values in details sidebars wrap instead of showing a horizontal scroll bar.

* The Domino endpoint details page's browser tab reads "Endpoint Details - Domino" instead of "Domino Admin", the Dataset details breadcrumb shows **Datasets** instead of `Datasets / --`, and the **Edit Environment Visibility** dialog closes from its close button or by clicking outside it.

### Breaking changes & deprecations

* Domino endpoint Viewers keep access to an endpoint's overview, versions list, and tester, but can no longer open version details, logs, or monitoring results; those requests now return an error. Previously, Viewers could also view versions and logs. The logs and **Audit Log** links are hidden from Viewers, but some links to version details and monitoring still appear and fail, and the endpoint's running status doesn't refresh automatically for Viewers. Existing collaborator roles aren't migrated. To restore this access, an endpoint Owner can give the collaborator the Editor role, which also lets them publish versions and change settings. See [Add collaborators](/cloud/platform-capabilities/features/model-deployment/security#add-collaborators).

<hr />

September 3, 2026

Cloud Release 92 adds more precise Domino endpoint sharing, expands knowledge management to Policies and NetApp Volume search, and brings point-in-time data selection and coding assistants to more compute workflows.

### Apps, development, and compute

#### Pin data to a specific snapshot at launch

When you launch a [Workspace](/cloud/reference/glossary#workspace) or [Job](/cloud/reference/glossary#job), or define a [Scheduled Job](/cloud/reference/glossary#scheduled-job), choose independently for each mounted [Dataset](/cloud/reference/glossary#dataset) or [NetApp Volume](/cloud/reference/glossary#netapp-volume) whether to mount all snapshots or pin one. A pinned [snapshot](/cloud/reference/glossary#snapshot) mounts read-only at the asset's usual path, so existing code can read point-in-time data without a path change. See [Choose a snapshot per asset at launch](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots#choose-a-snapshot-per-asset-at-launch).

#### Coding assistants in Slurm Workspaces

The `domino-slurm-environment` image now includes Claude Code, Codex, and GitHub Copilot CLI, together with Domino plugins, skills, and agents. Claude Code and Codex are also available as VS Code extensions, and assistant configuration persists between Workspace sessions. See [Coding assistants](/cloud/platform-capabilities/features/coding-assistants).

#### Recognizable NetApp Volume names in ONTAP

New NetApp Volumes use the Domino display name as their immutable ONTAP name instead of a generated `trident-{volume-id}` name, making them easier to identify from storage tools. Renaming a Volume in Domino does not rename it in ONTAP, and existing Volumes keep their current ONTAP names. See [NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes).

#### Redesigned NetApp Volume details page

The NetApp Volume details page now matches Domino's standard detail page layout: breadcrumbs, a page-level **Share** button for permissions, and a sidebar with the editable description, [Data Plane](/cloud/reference/glossary#data-plane), storage name and path, maximum snapshots, [tags](/cloud/reference/glossary#tag), and [Property](/cloud/reference/glossary#property) values. A snapshot selector next to the volume name switches between the latest data and a read-only view of any snapshot, and the tabs are now labeled **Files** and **Access**. The **Files** tab adds search, sortable and configurable columns, pagination, **Take Snapshot** and **Upload files** buttons, and bulk download and delete of selected files from an **Actions** menu. See [View and edit NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/view-and-edit-netapp-volumes).

#### Programmatic App access URLs

An [App](/cloud/reference/glossary#app) overview now shows the programmatic access URL alongside its vanity URL, so you no longer have to derive the address manually.

#### Redesigned App and Agent details page

The details page for [Apps](/cloud/reference/glossary#app) and [Agents](/cloud/platform-capabilities/features/agents) now keeps the description, status, running duration, creator, creation date, [Data Plane](/cloud/reference/glossary#data-plane), project, tags, and custom fields in a details panel that stays visible on every tab. A new **Inputs** tab shows the selected version's code details and lists its Datasets, [Data Sources](/cloud/reference/glossary#data-source), NetApp Volumes, and [External Data Volumes](/cloud/reference/glossary#external-data-volume-edv) in one table, and a new **Monitoring** tab holds the deployment configuration and runtime metrics. These replace the **Overview** and **Fields** tabs. The **Usage**, **Performance**, and **Governance** tabs are unchanged. See [Understand App usage](/cloud/platform-capabilities/features/apps/app-logs-and-versioning/understanding-app-usage).

#### Resolve a Job's configuration through the API before starting it

`POST /api/jobs/v1/projects/{projectId}/job-defaults` reports the exact hardware tier, environment and revision, Git ref, snapshots, and volumes a [Job](/cloud/reference/glossary#job) in a [Project](/cloud/reference/glossary#project) would run with. Each field you send is applied as an override, and Domino fills in the rest from the Project's defaults. Resolved fields use the same names as `POST /api/jobs/v1/jobs`, so you can pass them straight into a `startJob` request. See [Resolve Job defaults before you start a Job](/cloud/platform-capabilities/core-concepts/jobs/create-and-run-jobs#resolve-job-defaults-before-you-start-a-job).

<hr />

### Governance and knowledge management

#### Tags and properties extend to policies

[Tags and Properties](/cloud/platform-capabilities/core-concepts/knowledge-management) now cover [Policies](/cloud/reference/glossary#policy). Classify a Policy against your organization's [tag structure](/cloud/reference/glossary#tag), record typed [Property](/cloud/reference/glossary#property) values, and set Properties independently on each [Policy version](/cloud/reference/glossary#policy-version). The Policies list shows and filters by Tags, and creating a Policy from an existing one carries its Tags and Properties forward.

The [Compute Environments](/cloud/reference/glossary#compute-environment) list also shows and filters by Tags. Duplicating an Environment now copies its Properties as well as its Tags.

#### Find NetApp Volumes with global search

[NetApp Volumes](/cloud/reference/glossary#netapp-volume) now appear in [Domino search](/cloud/platform-capabilities/features/collaboration/domino-search). Search across a Volume's name, description, filesystem, owner, associated [Projects](/cloud/reference/glossary#project), Tags, and Properties, then open the Volume from the results table or command palette. Results are filtered against RemoteFS permissions so users see only Volumes they can access.

<hr />

### Identity and API access

#### Device code sign-in for the Domino CLI

Sign in from a machine that has no local browser with `domino login --device https://your-domino-url`. The [Domino CLI](/cloud/reference/cli/install-the-cli) prints a verification URL and short code that you can open on another device while the CLI waits for authorization, which is useful on remote hosts reached over SSH.

#### Set a repository's default Git reference through the public API

`PUT /api/projects/v1/projects/{projectId}/repositories/{repositoryId}/default-ref` updates the branch, tag, commit, or default-branch head that Domino checks out for a [Project's](/cloud/reference/glossary#project) main repository or one of its imported repositories; pass `_` as the repository ID to target the main repository. Previously this was only possible through an internal endpoint, which is now deprecated in favor of the public path. See [Change a repository's default reference](/cloud/platform-capabilities/core-concepts/projects/dfs-projects#change-a-repositorys-default-reference).

#### Fetch a rendered view of a repository file through the public API

`GET /api/projects/v1/projects/{projectId}/repositories/{repositoryId}/rendered-file` returns a file from a Project's main repository or one of its imported repositories as an HTML fragment, for formats Domino knows how to render such as Markdown and notebooks. Pass `_` as the repository ID for the main repository, and use `fileName` with an optional `branchName` or `commit` to pick the file; `commit` takes precedence. The HTML comes from repository content, so sanitize or sandbox it before inserting it into a page, and note that images and links in it point at an internal route outside the public contract. Previously this was only possible through an internal endpoint, which is now deprecated in favor of the public path.

<hr />

### Notes for platform administrators

* **Remote [Data Planes](/cloud/reference/glossary#data-plane) can now run on Google Kubernetes Engine (GKE).** The installer now provisions GKE Data Plane DNS, certificates, storage, and workload identity, including deployments configured without Data Plane storage. Existing Data Planes require no action; work with your Domino representative when [adding a GKE Data Plane](/cloud/admin/platform-configuration/manage-data-planes).

- **Cloud administrators can edit limits on customer-created node pools.** From **Platform Settings** > **Deployment Configuration**, update a node pool's description, maximum CPU, and maximum memory without recreating it or its [Hardware Tiers](/cloud/reference/glossary#hardware-tier). Built-in node pools remain read-only. Action is required only to add, change, or remove a limit; see [Manage node pools](/cloud/admin/infrastructure-and-compute/manage-compute-resources/node-pools#edit-a-node-pool).

* **Platform backups cover more service databases.** Data Importer backup and restore workflows now include the Taxonomy, Filetask, IDSM authorization, and RemoteFS databases. This change protects Tags, Properties, NetApp Volume metadata, snapshots, and grants during disaster recovery and requires no action. See [Disaster recovery](/cloud/admin/operations/disaster-recovery).

- **Executions started with Domino Service Account tokens keep API access after the initial token expires.** Domino now obtains an offline session token for an App or Job started from a service-account request. This change is transparent and requires no action.

### Bug fixes

* Restarting an unchanged LLM endpoint now reuses its existing [App version](/cloud/reference/glossary#app-version) instead of creating another App version and instance.

- The AI endpoint sharing dialog now lets you grant access to named users and organizations from either a Project or the global Endpoints page.

* Pausing and resuming a Scheduled Job now preserves its storage and snapshot selections.

- [HPC Job](/cloud/reference/glossary#hpc-job) tables and details now report the correct submission script, CPU percentage, average CPU, peak memory, resource requests, and array-job submission details.

* Governance [bundle](/cloud/reference/glossary#bundle) PDF reports now generate when the bundle contains upgraded Policies.

### Breaking changes & deprecations

* Forking a [Domino File System (DFS)](/cloud/reference/glossary#domino-file-system-dfs) Project is removed from the Domino Cloud UI. The Project overview page no longer offers the **Fork** and **Request Review** actions, so you can't create a fork or merge one back from the UI. The underlying [API](/cloud/reference/api) endpoints are unchanged, and existing forks continue as standalone Projects. To move a fork's changes into the original Project, [export and import Project content](/cloud/platform-capabilities/core-concepts/projects/manage-projects#export-and-import-project-content) or use the API. To duplicate a DFS Project, [copy the Project](/cloud/platform-capabilities/core-concepts/projects/manage-projects#copy-a-dfs-project). For branch-and-merge collaboration, use a [Git-based Project](/cloud/reference/glossary#git-based-project), which keeps your Git provider's pull requests, branch protection, and CI; see [Git-based Projects](/cloud/platform-capabilities/core-concepts/projects/git-based-projects). Forking is still available in self-managed Domino 6.3.

- The legacy [AI Gateway](/cloud/reference/glossary#ai-gateway-legacy) is removed. See [LLM Gateway 2.0](/cloud/platform-capabilities/features/llms/llm-gateway) for the successor and migration path.

<hr />

## 2026 July

### Cloud Release 90

Shipped July 28, 2026.

#### New features

##### Governance for Apps

[Apps](/cloud/reference/glossary#app) now carry a **Governance** tab, so an App is reviewed and approved in the same place it's built. Create a [bundle](/cloud/reference/glossary#bundle) when you create the App or later from the Governance tab, and Apps that are already gated link to their existing bundle instead of starting a new one.

Mandatory [policies](/cloud/reference/glossary#policy) attach automatically to each new App version, so a new version can't quietly ship without the review its policy requires. When you publish, the enforced-policy view shows which policies apply and what's outstanding. Reviewers can diff evidence sets between App versions to see exactly what changed since the last approval.

Previously, governance covered projects and models, and App review had to be tracked outside Domino. Apps Governance is on by default in this release, and the Data Importer carries Apps Governance records through lift-and-shift migrations.

##### Apps SDLC: build, preview, and publish Apps from a Workspace

You can now develop an App end to end without leaving a [Workspace](/cloud/reference/glossary#workspace). Start a live preview of a draft App next to your code, apply new commits to the running preview with **Sync all** instead of recreating it, and publish from the Workspace when the App is ready, either as a new App or as a new version of an existing one.

Previews work for [Projects](/cloud/reference/glossary#project) backed by the Domino File System and for imported Git repositories. The App API exposes the last-synced commit state for main, DFS, and imported repositories, so you can tell what a preview is actually running. Previews stop when the Workspace shuts down, so a forgotten preview doesn't keep consuming compute.

##### Submit HPC jobs to Slurm

High-performance computing (HPC) runs now have a dedicated **HPC** tab alongside [Jobs](/cloud/reference/glossary#job). It's the Jobs table plus 2 columns that matter for HPC: the [Data Plane](/cloud/reference/glossary#data-plane) the run landed on and the number of GPUs it used.

Submit HPC jobs to a Slurm cluster running on a Data Plane, from Git-based Projects as well as Domino File System Projects, and pin a submission to a specific commit. Submission validates 2 conditions before it accepts a job: Slurm is installed on the target Data Plane, and a nodeset on that Data Plane is provisioned for your Project. Slurm workloads run on dedicated nodes, which fixes GPU jobs that previously failed because Slurm and Kubernetes workloads shared nodes with manually assigned resources, and multiple Slurm jobs can now run on a single node.

#### Enhancements

* Tags and Properties now cover [Compute Environments](/cloud/reference/glossary#compute-environment). Tag and add properties to an Environment, browse Environments in the Tag Browser and property browser, and search them by tag, tag description, or property value. Environment owners get the new `EditEnvironmentTaxonomyTag` and `EditEnvironmentPropertyValues` permissions, so tagging follows environment ownership rather than platform admin rights. Duplicating an Environment copies its tags.

- Model Registry search is consolidated into a single bar. Free-text search on model name and MLflow tags, and taxonomy tag chips, now live in one control instead of 2 adjacent search boxes. Properties on models are set per model version rather than per model name.

* Global search shows entity counts per type, adds columns to the results table, and covers files and Datasets. Property queries such as `project.name=` return the matching Project. Grouping and a **Clear** control on the entity type filter make long result lists easier to scan.

- The Posit Package Manager integration is on by default, set through the `com.cerebro.domino.workbench.workspace.enablePpmUI` central configuration flag. The Workspace panel also surfaces packages from repositories that had no snapshots before: previously only repositories subscribed to a public source carried the date-based snapshot labels the panel needs.

* [Flows](/cloud/reference/glossary#flow) run on remote Data Planes by default, through the `com.cerebro.domino.flows.enableRemoteDataPlaneSupport` central configuration flag. Flows fixes in this release: NetApp Volume snapshots declared in a Task's data configuration now mount, input files are no longer removed by executor cleanup, artifact export succeeds at large artifact counts, and raw file uploads retry on transient idle-connection errors.

- [Extensions](/cloud/reference/glossary#extension) improvements: Domino-official Extensions sort first, install cards link to the matching configuration accordion and back, and NetApp Volume and NetApp Volume file mount points are available in manifest v1 (data volume links now carry the snapshot ID). The installer warns when an installed Extension reports `isCompatible=false`, when a version lookup fails, and when the install list is degraded. Cancelling an upgrade restarts the previous App version rather than leaving the Extension down.

* Job and scheduling fixes: an invalid cron expression is rejected before it's saved instead of deleting the scheduled job, editing a scheduled job saves the revision shown in the Environment field rather than defaulting to Active, External Data Volume checkboxes in **Start a Job** default to checked again, and job cluster launch forms accept unrestricted [Hardware Tiers](/cloud/reference/glossary#hardware-tier) for Spark, Ray, and Dask worker slots instead of silently clearing the selection.

- The global Apps list loads faster on large deployments, where a separate instance lookup for every App previously caused timeouts. Saving **Edit App configuration** with no changes no longer creates an empty version, App view counts are consistent, and the list no longer fails on older App records that are missing user information.

* Governance fixes and polish: the Policy Enforcement page has updated terms and an **Edit** button for safeguards, published policies no longer display as archived, findings from earlier policy versions stay reachable after an upgrade, resetting a bundle uses the correct policy version, approval notification links point to the right place, and the **Attach** button is disabled until a policy is selected.

- Idle timeout preserves your context. When Domino signs you out for inactivity, signing back in returns you to where you were, such as a Workspace or a Dataset, instead of the home page.

* Git fixes: the Project code view loads after Git credentials are removed, so you can select a new credential from the UI, and default credential fallback no longer picks a credential for the wrong provider. Bitbucket Cloud HTTPS credentials support the username and API key scheme that replaces App Passwords. In Workspaces, **Continue sync** stays enabled when conflicting files are in subdirectories.

- Data fixes: mounting a [Dataset](/cloud/reference/glossary#dataset) that was shared with you works, mounts resolve for Dataset selections on remote Data Plane executions, and Volume tags appear in the volumes list. The `/entity-tags` endpoint filters on the caller's permissions, tag search filters nested items in tree view, App, Dataset, and Volume tags and properties copy into Project templates, and a nightly job clears orphaned tags and properties.

* Agent performance charts get usability and functional improvements. Domino Model Monitor (DMM) notification settings accept an SMTP relay with no username or password, which unblocks internal IP-based relays that don't require authentication.

- **Tags now extend to Models.** Tag, filter, and browse Models through your organization's [tag structure](/cloud/reference/glossary#tag), which completes tag coverage across Apps, Projects, Project Templates, Datasets, NetApp Volumes, and Models. Tags on a Model are separate from any MLflow tags on the underlying model. See [Tags](/cloud/platform-capabilities/core-concepts/knowledge-management/tags).

#### Notes for platform administrators

* Secure identity propagation for Apps is enabled by default in Domino Cloud. `SecureIdentityPropagationToAppsEnabled` (enhanced) and `com.cerebro.domino.apps.extendedIdentityPropagationToAppsEnabled` (extended) both default to `true`. Previously both were off by default and had to be requested. Running Apps aren't affected until they're restarted. Action is required only for Apps that host secondary pages or additional assets: those need the root-path change described in the App security and identity documentation before their next restart.

- The API Gateway (Envoy Gateway with a gRPC external-authorization service) gained a large set of routing and authentication fixes ahead of general availability. Nucleus routes outside `/api/` and `/v4/` no longer return 404 from execution pods, the Governance, Cost, and Audit Trail APIs resolve in-cluster whether or not the gateway is on, Arrow FlightService keeps its audit trail and API-key authentication, the nginx sidecar stops overwriting an explicit `Authorization` or `X-Domino-Api-Key` header set by user code, and the external-authorization check has an explicit timeout so a slow check can't stall the gateway. Envoy Gateway is re-pinned to 1.8.2 before 1.7.3 reaches end of life. No action is required.

* SpiceDB authorization data migrations run by default. A readiness gate reads the `domino-platform/authz-data-migrations` ConfigMap and keeps authorization on the legacy path until the required migrations are applied, blocking grant writes only while a planned migration is pending. Pre-flight and post-migration checks compare expected and written relationship counts and sample permission checks. Organization membership reads move to SpiceDB as the source of truth, behind a flag that can be reverted at runtime, and Domino logs a message when the legacy and SpiceDB authorization systems disagree. No action is required.

- Keycloak client secrets are randomized per deployment. The `domino-play` and `realm-management` clients previously shipped with a secret hardcoded in the realm configuration, which anyone with that value and network access could use to mint a service-account token. A dedicated Keycloak client is also added for the workload service. No action is required.

* ECR credential handling: the `nucleus-ecr-credential-refresher` CronJob refreshes credentials for `image.registry` as well as `config.dockerRegistry.host`. Where those point at different ECR accounts, the platform-image token previously expired after roughly 12 hours and remote Data Planes started failing with `ImagePullBackOff`. On AWS GovCloud, the credential refresher init job now calls the GovCloud STS endpoint instead of the commercial one, which unblocks upgrades that use ECR as the external Docker registry. No action is required.

- Karpenter `EC2NodeClass` resolves the root device name from the Amazon Machine Image (AMI) that's actually selected. Previously the device name was set from the global `eks.use_domino_ami` flag, so a node class pinning a stock AL2023 image attached its configured 100Gi or 500Gi volume to `/dev/sda1` as an orphan while the node booted on the image's default root volume of roughly 20 GiB. No action is required.

* All SMB volume mounts failed in 6.2.3 and now work. NetApp Trident components are updated to 26.06.0, with the Trident, Trident operator, and `tridentctl` images moved to Chainguard. No action is required.

- Model endpoints work in offline and air-gapped deployments. The `harness-proxy` image can be overridden and resolves from the Data Plane rather than being pulled from Quay. No action is required.

* Provisioning Project partitions is partly automated for admins, and HPC job submission checks partition configuration before accepting a job.

- Taxonomy service reliability: the `taxonomy-postgres-migrations` job no longer fails on hardcoded credentials, `GET /api/taxonomy/v1/namespaces` no longer returns 500 during Cloud Release windows, and `idsm-audit-trail-mongodb` memory limits were reviewed. No action is required.

* New Grafana dashboards cover the API Gateway, and the Restart Nucleus admin page is easier to use. No action is required.

- Platform maintenance: Kubernetes 1.35 is validated for GKE and 1.34 for OpenShift 4.21, platform apps move to Python 3.14, and Fluent Bit images move to Chainguard. This release also includes a security hardening pass across platform components and Environment images, removes committed secrets from model development lifecycle repositories, and stops the deployer from printing sensitive values in its configuration output. No action is required.

* The workload operator can be configured not to mount `/tmp` from the execution volume. This is opt-in and the default behavior is unchanged.

#### Breaking changes & deprecations

* The `CreateApp` and `CreateEndpoint` gate actions are deprecated in favor of `Deploy`. Existing gates that reference them keep working and can still be read, but gate creation and updates that use them are rejected. Update gate YAML to `Deploy` before your next policy edit. The deprecated `failOpen` key no longer appears in gate YAML.

- The **Jira integration** item is removed from the admin panel.
- **The `project.tag.approved=` search operator is removed.** It filtered on the approved flag of the previous free-form Project tags, which the shared tag structure replaces; queries using it no longer resolve. Filter a result set by tag from the filter side panel beside your search results or by typing the tag into the search bar instead. The `project.tag=` operator is unchanged. See [Use Domino search](/cloud/platform-capabilities/features/collaboration/domino-search#filter-results-by-tag).
- **Project Tasks, Goals, and the Project Stages and Status UI are removed.** Domino no longer shows the Tasks and Goals page, the Project stage box and stage filters, the combined Stage and Status component in the Project sidebar, the Stage and Tasks columns in the Projects Portfolio, or the Admin console's Project stages entry. The read-only Project Status badge next to the Project name remains, and the [Activity feed](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#view-project-activity) still records Job, Workspace, and file-comment events. Use your organization's issue tracker for task-level milestones. The backend Goal APIs remain available through the API and CLI.
- **The Project-level Jira integration UI is removed.** Domino no longer shows the UI to link a Domino Project to a Jira ticket or to sync Domino tasks with Jira subtasks, and existing per-Project Jira links are no longer surfaced in the Project. The backend Jira APIs and the deployment-wide Jira OAuth registration at `/admin/jiraConfig` are unchanged.

### Cloud Release 89

#### New features

##### Knowledge management: tags and properties

Tags and Properties replaces Domino's earlier tagging feature with a controlled knowledge management system. Tags categorize and discover assets with values from a controlled, hierarchical list, organized as key-value pairs: Namespaces are the allowed keys, and Tags are the allowed values within each Namespace. Properties enrich an entity with custom information through typed fields: unlike Tags, a Property's values aren't drawn from a controlled list; users supply any value that matches the property's type, with validation to keep values consistent. The new Select field type is the exception, letting a property offer a fixed set of options where that fits better.

Attach Properties to Apps, Projects, Project Templates, Datasets, NetApp Volumes, and Models to capture values like data classification, review status, or internal IDs directly on the entity. On Apps and Models, values can be set globally and independently on each version, so stable information and per-version detail coexist. Project Templates carry their values into new Projects, or require the creator to supply them. Editing is scoped to the app author and project manager, so property values stay controlled, and project properties are indexed for search. See [Properties](/cloud/platform-capabilities/core-concepts/knowledge-management/properties).

Properties replace the earlier Fields concept; anywhere you saw **Fields** before, you now see **Properties**. Existing values carry over automatically, with no migration step to run.

##### Browser-based CLI login

Logging in with the Domino [CLI](/cloud/reference/cli) now uses an OAuth flow. When you log in, the CLI opens your default browser to sign in and completes authorization automatically through a local callback on the same machine. Previously you had to copy an authorization code from the browser and paste it back into the CLI; that manual step is gone. If a browser doesn't open automatically, the terminal prints a URL to open on the machine running the CLI.

#### Enhancements

* The bundle view now shows an indicator when a policy on the bundle is enforced, so reviewers can see at a glance which policies are enforced.

- Governance admins get a new admin view for creating policy enforcers, making it easier to see and manage enforcement without digging through policy YAML.

* Sequential governance workflows support bulk updates, and the Deliverables table got usability improvements to match.

- Extension installs now show a **Cancel** button, so you can stop an in-progress installation instead of waiting for it to finish.

* The [Audit Trail](/cloud/reference/glossary#audit-trail) OpenAPI schema now includes the e-signature fields that were missing, so programmatic consumers of the Audit Trail get the full record.

- Snapshot events for Domino Datasets and NetApp Volumes now carry more detail, including on "Create Snapshot" and "Mark Snapshot for Deletion" events, for better traceability.

* Runs now get an environment variable pointing to the NetApp Volume location, so scripts can reference it without hardcoding paths.

- Imported dataset mount paths can now include the project name, so identically named datasets from different projects stay distinguishable. The old paths remain available as symlinks for backward compatibility, and the behavior is enabled with a central configuration flag. This continues last release's fix to imported dataset naming.

* Notifications are less noisy: several fixes address notifications that were unclear, overly persistent, or queued incorrectly.

- Fixed bugs affecting page navigation and display: the Audit Trail page could appear completely empty instead of showing logs, refreshing the tags page in tree view sent the browser to a 404 page, and tag display was broken in table columns that have been widened.

* Fixed a failure in experiment execution when running on a remote data plane, and an issue where using a Git tag (instead of a branch) in a Git-based project could break the Jobs output.

- Fixed environment-related issues: `domino_data.data_sources` was not found in the R reticulate environment (restoring Data Source access from R through the Python SDK), a package mismatch in the Ray GPU cluster environment, and a bug where updating an app's environment could produce a version that couldn't actually be deployed.

* Assorted governance UI polish: corrected wording on the bundle-creation toast, fixed formatting in the policy description modal, and fixed several deploy-modal bugs.

#### Notes for platform administrators

* Fixed an issue that blocked upgrading `nginx-ingress-fips` to 1.13.9-r29: the upgraded nginx drops legacy TLS ciphers that AWS Classic Load Balancers depend on, which could take deployments behind those load balancers offline. The fix enables DHE, a cipher both sides support, so deployments behind Classic Load Balancers upgrade safely. No action is required.

- Air-gapped SpiceDB deployments now upgrade correctly. This is a fix to the upgrade path itself, not a configuration change; no action is required.

* Domino Standard Environment (DSE) images are upgraded to Ubuntu 24.04.&#x20;

- Fixed a remote data plane failure where the workload-operator chart could not install because of a missing `nucleus-execution-jwt` secret. No action is required.

* This release includes a broad security hardening pass across platform components and environment images, remediating known vulnerabilities in dependencies such as log4j, netty, aiohttp, and nginx ingress. No action is required.

- Keycloak 26.6.2 tightens token introspection to require the calling client in the token's audience. Domino now includes the client ID in the audience for service accounts and Personal Access Tokens, so both keep authenticating after the Keycloak upgrade. This change is transparent and no action is required.

* The Audit Trail UI used a hardcoded one-year date range even though the back end retains five years by default; the UI now reflects the configured retention period. No action is required.

- Fixed a bug where workspace `/tmp` permissions were set to 777 even when `nonRootExecutions` was disabled. No action is required, but if you rely on restrictive `/tmp` permissions for compliance, confirm the fix took effect after upgrading.

#### Breaking changes & deprecations

* CLI versions older than 6.2.3 use a retired authentication client and can no longer log in. If `domino login` fails after this release, download and install the latest [Domino CLI](/cloud/reference/cli).

<hr />

## 2026 June

### Cloud Release 88

#### New features

##### Relative revalidation schedules for Governance approvals

[Governance](/cloud/reference/glossary#governance-center) policies can require an approval to be validated again on a recurring schedule. That schedule can now be relative to the approval, not only a fixed calendar. Previously a revalidation schedule was absolute: it recurred from a fixed start date regardless of when the approval was granted. With a relative schedule, the recurrence is anchored to the first approval, so the next revalidation falls due a set interval after the approval is granted, for example one year after approval.

##### Secondary Git repositories in Flow tasks

A [Flow](/cloud/reference/glossary#flow) Task can now reference a secondary or imported Git repository in its job configuration, not just the project's main repository. Teams composing multi-repository workflows can pull code from more than one repository in a single Flow Task.

#### Enhancements

* The Flyte Domino agent now supports runtime overrides, so you can customize a task's execution parameters when you launch it without editing the underlying task definition.

- The global navigation now includes [Governance](/cloud/reference/glossary#governance-center), making policies, evidence, and bundles easier to find from anywhere in the product.

* The Job detail page now shows the default branch name for jobs started from a [Flow](/cloud/reference/glossary#flow). Previously the branch was not shown for Flow-started jobs, which made runs harder to trace back to their source.

- Environment variables now have an optional description field, so teams can record what each variable is for alongside its name and value.

#### Breaking changes & deprecations

None in this release.

### Cloud Release 86

Shipped June 10, 2026.

#### New features

##### Compute usage visibility to manage compute costs

A new Hardware Efficiency dashboard in Grafana shows requested versus actually consumed CPU, memory, and GPU across your workloads, so you can spot over-provisioned [Hardware Tiers](/cloud/reference/glossary#hardware-tier) and right-size them. GPU monitoring is expanded alongside it: new utilization recording rules feed charts for actual usage, usage by workload type, and usage by project over time. Together they answer the questions that drive compute spend: which tiers are oversize, whether your GPUs are actually busy, and which projects and workload types consume them.

The dashboards are available in Grafana by default; no setup is required.

#### Enhancements

* New columns in the [Jobs](/cloud/reference/glossary#job) dashboard: **Finished** (completion timestamp) and **Total Duration**. You can spot slow or stuck jobs from the list without opening each one.

- R jobs can now accept any command-line parameter that the R executable accepts.

* More control over email notifications when you share an [App](/cloud/reference/glossary#app). Admins can set whether sharing an App sends a notification email by default.&#x20;

- When you use an [Extension](/cloud/reference/glossary#extension), the sidebar now highlights which Extension you are in.

* Fixed a defect that allowed Apps to be created with special characters in their names that later caused routing and deployment failures. Invalid names are now rejected at creation with a clear error.

#### Notes for platform administrators

* A new admin policy can block project invitations to email addresses outside your Domino organization. When the policy is enabled, project access and sharing rejects email invites to non-Domino addresses and shows the inviter a message explaining the policy. Adding existing Domino users through the user picker is unchanged, and the default behavior without the policy is unchanged. Action required only if you want to enforce the boundary: enable the policy.&#x20;

- Access to Grafana dashboards is now enforced through Keycloak, so observability access follows the same roles as the rest of the platform. No action is required.

* `enableServiceLinks` is now disabled for components in the `domino-compute` namespace, so Kubernetes no longer injects service-discovery environment variables into workload pods. This change is transparent and does not affect workload behavior; no action is required.

- Domino Model Monitor (DMM) ingestion job health checks were added to the admin toolkit, and DMM-related checks were consolidated. No action is required.

#### Breaking changes & deprecations

None in this release.

<hr />

## 2026 May

### Authentication, identity & security

* **User information API**: New backend API to retrieve user information based on user ID.

* **Iframe permissions for microphone and clipboard**: Customer-driven addition of "microphone" and "clipboard" to the iframe `allow` src tags.

* **Grant ReadOnlySupportStaff ViewEnvironments permissions**: Fixes ReadOnlySupportStaff global role to have permissions to view environments.

* **Personal Access Tokens**: Domino now supports Personal Access Tokens (PATs), allowing users to generate their own API credentials with a custom name, description, and expiration time. Administrators can oversee and manage tokens across the platform, including the ability to revoke or delete them when needed.

### Platform & UI

* **Workspace Breadcrumb Project Link**: Workspace breadcrumbs now link back to their parent project.

* **Env Vars Table: Sort & Copy**: The environment variables table now supports sorting, and each value has a copy button.

* **Custom Job Email Template Cleanup**: Removed test content from the custom job email template.

* **NetApp Volume Mount Points**: Full support for NetApp Volume mount points in the Domino UI, including a new API component, details panel, file-open integration, and UI adjustments to accommodate the new mount point type.

* **Model Details: Full Markdown Overview**: The Model Details page now renders the full model overview markdown instead of a truncated preview.

* **Scheduled Jobs Notifications: Security controls**: Logs no longer included by default in emails; ability to restrict custom recipients via CC flag.

### Tag Browser

* **Tagging** now extends to NetApp Volumes. Tag, search, and browse NetApp Volumes through your custom tag structure. We’ll be integrating tagging with more assets in updates coming soon.

* **Tagging** extends to Domino Datasets. Tag, search, and browse Datasets through your custom tag structure.

### App Governance & SDLC

* **AI Gateway UI** is now hidden by default behind a feature flag.

### Curated configuration records management

* **New API Endpoints**: Added backend endpoints for Curated Configuration Records, enabling Cloud Admins to manage curated configuration sets for their tenants.

* **Feature Flag**: Added feature flag to show/hide the Curated Configuration Records section, allowing controlled rollout per tenant.

* **Project Behavior Page: UI Sections**: Built the Long-running Workspaces, Restrictions & Limitations, and Flexibility & Customization sections of the Project Behavior configuration page.

### Governance

* **Governance Enhancements**: Governance demands more than a one-time sign-off: approvals can now be scheduled to renew on a recurring basis, so your compliance posture stays current as models and projects evolve. Approvers are notified when renewal is due, and bundles automatically fall out of compliance if a renewal lapses.

<hr />

## 2026 April

### Knowledge management

#### Tags

[Tags](/cloud/reference/glossary#tag) now extend to Domino Projects and Project Templates. Define a tag structure that reflects how you categorize and organize knowledge in your organization, so you can find and reuse past work instead of reinventing the wheel. Tag, search, and browse Projects, Project Templates, and Apps through your custom tag structure. We’ll be integrating tagging with more assets (Datasets, NetApp Volumes, etc.) in updates coming soon.

### Platform & infrastructure

* GPU metrics panel now renders based on available GPU data rather than presence of DCGM label.

### Compute Environments & DSE

* **Admin access to cluster Web UIs for troubleshooting**: Admins can now view the Spark, Ray, and Dask cluster Web UIs for other users' workspace sessions to inspect executor logs, configs, and worker state, without being granted access to the user’s notebook/IDE. When an admin opens another user’s workspace session, the session page now opens directly on the Cluster UI tab, giving admins a working diagnostic view even though the IDE itself remains restricted. Enabled by default: no central configuration or feature-flag change required.

* **Environment select pattern**: A consistent environment select UI pattern introduced across Domino for choosing a compute environment when launching tasks.

### UI & UX improvements

* **Expanded span type support in tracing decorator**: The manual span tracing decorator now accepts the `span_type` argument, enabling full MLflow tracing argument support. Users can now manually instrument unsupported scenarios without accessing raw MLflow features directly, and traces display the correct icons in the trace viewer.

* **Improved terminology in Experiment Manager**: When agent traces are detected, the UI now displays "Agent Versions" in place of "Runs," allowing users to distinguish agent version tracking from traditional ML run tracking within the Experiment Manager.

### Extensions

This release contains updates to the Extensions framework with new mount points, Extension management UI capabilities, and general availability.

<AccordionGroup>
  <Accordion title="New mount points">
    * **Edit Extension modal**: Users can now edit an existing Extension from the Extension management page.

    * **Dataset detail page mount point**: Extensions can now embed in dataset detail pages.

    * **Model detail page mount point**: Backend support added for the model card mount point.

    * **Dataset file mount point**: Extensions can now embed in dataset file views.

    * **Model detail page mount point**: Frontend support for model detail page mount point.

    * **Admin mount point**: Frontend support for admin mount point.

    * **Improved DatasetFileContext support**: Enhanced context support for extensions embedded in dataset file views.
  </Accordion>

  <Accordion title="Additional mount points">
    * **Additional mount points at Create/Edit extension modal**: Create and Edit Extension modals now support selection of the new mount points.
  </Accordion>

  <Accordion title="Admin extensions UI">
    * **Admin extensions list page**: A new admin extensions list page with routing, table view, enable/disable controls, and authorization is now available.

    * **Extensions management menu**: A managed extension menu added inside the extension accordion for in-context control.

    * **Create Extension modal**: Users can now create an Extension directly from the Extensions management page or from an App detail page.
  </Accordion>

  <Accordion title="Backend & APIs">
    * **Core APIs**: APIs for creating, patching, and retrieving extensions are live with feature flag gating.

    * **Extensions auditing**: Audit logging added to all Extensions APIs.

    * **Admin panel mount point**: Backend support added for extensions to embed in the admin panel.

    * **Dataset mount point**: Backend support added for extensions to embed in dataset detail pages.

    * **Extensions FF enabled by default**: The Extensions feature flag is now on by default for all users.

    * **MongoDB indexes**: Performance indexes added for extensions-related MongoDB collections.
  </Accordion>

  <Accordion title="Frontend">
    * **Extension renderer**: Renderer passes and parses query params; deep-linking support added so extensions can redirect to iframeless app views when the underlying app supports deep linking. TopBar removed as a mount point; overflow pattern added for the Project sidebar.
  </Accordion>
</AccordionGroup>

### App versioning & Apps

App versioning is significantly expanded, enabling users to manage, start, stop, and edit non-latest versions.

* **PPM language update**: Updated PPM-related language in UI and documentation.

* **API access section on Dataset details page**: The API access UI on the Dataset details page has been relocated to accommodate the new Extensions card mount point.

* **Version-specific start/stop**: Users can start and stop historical as well as current App versions.

* **Edit previous App versions**: Pen icon controls added to App detail cards for previous versions.

* **App list view cleanup**: Edit option removed from the App list view row menu to avoid version ambiguity; editing is now only available from the App details page.

* **Edit App wizard update**: Deployment step updated to clarify that editing hardware tier no longer triggers a new App version.

* **App list performance**: Resolved N+1 query issues in the listApps endpoint, significantly reducing MongoDB queries per request.

* **Reproducibility for imported repos**: App versions now snapshot imported project and Git repo state to ensure reproducibility when restarting older versions.

* **App search**: Apps now appear as a dedicated tab in global search and in the global search dropdown.

### AI Governance & model lifecycle

Policy lifecycle and governance workflow improvements.

* **Policy version diff**: Governance Admins can now visualize the diff between policy versions, both in YAML and through the UI.

* **Bundle reset**: Reviewers in regulated environments can now reset a bundle at any stage moving it back to the first stage, with flexible control over which attachments are retained, while all approvals and evidence answers are cleared for resubmission. A complete reset audit trail is automatically captured in the bundle’s history view.

### AutoML

* **AutoML compute routing**: Training jobs from the AutoML Extension now run in the project where the dataset lives via the Domino Jobs API, ensuring correct experiment tracking, per-user compute isolation, and proper permission scoping.

<hr />

## 2026 March

This release expands GenAI cloud coverage, introduces AI governance policy versioning, and delivers broad improvements to agent monitoring, platform infrastructure, workspace management, and coding assistants.

### Workspace coding assistants

Domino Workspaces now include built-in support for AI coding assistants as part of the Domino Standard Environment, enabling model developers to accelerate data analysis, model training, and validation workflows directly from their workspace.

* Includes built-in support for Copilot, Claude Code, and Codex; available as pre-installed VS Code extensions and CLI tools.

* Agents are Domino-aware, with automatic access to datasets, jobs, and registered models.

* A new skills framework adds Domino-specific actions (model training, job execution, deployment, and more).

* Coding agent extensions and settings and chat history persist across workspace restarts.

### Extensions framework

Introduced admin-only APIs for creating and managing extensions. New UI for exposing extensions and creating them from an app.

### AI Governance & policy versioning

Policy versioning is now available in the Policy Builder, with new compliance visibility and bundle lifecycle tools to support governance at scale. This release continued to build out the AI Governance framework introduced in the previous release.

#### Audit Trail

* Telemetry and audit trail for policy versioning; audit events for policy versioning operations added.

#### Policy versioning

* "Reason for Change" field (renamed from "Version Description") is now required to publish.

* Create versions of existing policies directly from the Policy Builder.

* Upgrade prompts notify data science practitioners when newer policy versions are available.

#### Bundle lifecycle management

* Reset bundles and bulk-delete attachments from the bundle management interface

* Email notifications now track stage transitions

#### Compliance visibility

* Compliance status displayed directly in the policies and bundles tables

* Policy usage counts now reflect bundles across all versions

### Authentication & security

A more secure and maintainable auth layer emerges from this release, with key infrastructure changes to Vault, Keycloak, and authorization foundations.

#### Auth persistence

* Execution now knows if home persistence is enabled.

#### Infrastructure

* Vault agent injector removed from the cluster

* Keycloak service account provisioned

* AuthZ and unified auth foundations established

### Platform & infrastructure

Network and infrastructure upgrades in this release reduce log processing overhead and expand GPU workload visibility across remote data planes. Additional improvements were also made to Grafana, and Environments workflows.

#### Grafana

* Single source of truth for dashboards and alerts via Grafana Operator CRs.

#### Environments

* DSE split into AI-enabled and standard variants; Copilot VS Code extension added; VS Code marketplace switched.

#### GPU monitoring

* `kube_pod_labels` metrics now available from remote data planes for GPU workload monitoring

#### Logging

* Fluent-bit event filtering reduces `fluentd` load for more efficient log processing

### GenAI & model serving

This release broadens cloud coverage for GenAI model serving and resolves key deployment and data plane issues across all supported environments.

#### Support for multicloud deployments

* GenAI model registration and deployment now supported on Azure (AKS) and GCP

* NVIDIA environment variables and CUDA toolkit included in vLLM environment for GCP

#### Data Plane improvements

* Remote data plane support added for model APIs

* Fixed Istio resource configuration and control/data plane compatibility

#### Agent deployment

* Fixed issue preventing agent deployment from the experiment run overview page

#### Experiment runs table

* 3-dot menu column is now pinned on the Experiment Runs table

### Agent monitoring & tracing

Teams get clearer visibility into agent performance with corrected latency metrics, smarter chart bucketing, and improved tracing output in this release.

#### Chart bucketing

* Monitoring charts automatically re-bucket to 1-hour intervals on 24-hour windows

* Time axis now scales correctly across window sizes

#### Latency and metrics

* Fixed latency chart rendering issues and corrected unit display

* Fixed LLM chart failures when runs are stopped mid-session

* Fixed LLM metrics bucket edges incorrectly based on start time

#### Tracing UI

* Tool call span outputs now wrap correctly within the UI container

### Data, compute & workloads

Distributed workloads are easier to run, storage options are broader, and workspace controls are more flexible.

#### MPI

* MPI cluster support enabled by default and no longer requires manual feature flag enablement

#### Blob storage backend

* Azure Blob Storage now supported as a storage backend for Workspace File Audit Trail

#### Workspaces

* New Long-running Workspaces section added to Project behavior settings

* Administrators can configure automatic shutdown duration and notification preferences

### Deprecations

* Governance "Complete" stage removed in favor of the new compliance-based workflow

<hr />

## 2026 January

### Agentic system experimentation

Register and deploy LLMs, monitor AI systems with full trace visibility, and iterate on production configurations.

### LLM registration & deployment

Register and deploy large language models (LLMs) as optimized, hosted endpoints:

* Pull models directly from Hugging Face or your experiment runs

* Deploy with OpenAI-compatible APIs

* Make LLMs available to applications and agentic systems

### Production monitoring & tracing

Monitor agentic systems and LLM applications with full trace visibility:

* Capture every LLM call, agent interaction, and prompt chain as traces

* Works with MLflow auto-tracing integrations (LangChain, OpenAI, Pydantic AI) or standalone functions

* Attach evaluations to traces to measure quality, accuracy, or safety

* Compare performance across experiments to understand what’s working

### Production deployment & iteration

Deploy and iterate on agentic systems with infrastructure optimized for agent workloads:

* Use the same proven infrastructure as Domino Apps

* Instrumentation carries over automatically from experimentation

* Relaunch production configurations into a workspace for debugging

* Maintain clear lineage between production agents and source experiments

### Workspace audit capabilities

Extend Domino’s audit trail into interactive workspaces for complete visibility and regulatory compliance:

* Log all file operations (read, write, create, rename, delete) on NetApp Volumes and Datasets

* Investigate activity quickly with a dedicated UI for searching, filtering, and exporting events

* Customize the audit app or integrate programmatically with governance tools

* Retain events long-term in customer-managed blob storage with automatic deduplication

* Meet GxP and SOC 2 requirements with comprehensive, auditable records of in-workspace data access

### Reduce compute costs with spot instances

Cut compute costs by using discounted, interruptible AWS cloud capacity across Jobs, Workspaces, Apps, Endpoints, and Launchers:

* Select hardware tiers marked Spot tag to use spot pricing automatically

* Ideal for batch jobs, distributed training, and parallel analytics that tolerate interruptions

* Falls back to on-demand infrastructure if spot capacity is unavailable

* Interrupted workloads retry automatically on available capacity

<hr />

## 2025 December

### Governance update: Bundle history tab

Track every change to your bundles with the new Bundle History tab on the bundle overview page. See what changed, when, and by whom.

### Governance monitoring checks

Governance now responds automatically to Domino Model Monitoring alerts. When drift, data quality, or detection issues trigger, Governance creates findings, assigns severity levels, notifies owners, and logs everything for compliance - no manual intervention required.

### Certificate-based authentication

Certificates are now a supported authentication method for Oracle, DB2, and Postgres Data Sources.

### Flow integration with NetApp Volumes

* **Export Flow Outputs via UI**: Flow Artifacts of type `DATA` can now be exported to NetApp Volume snapshots via the UI, making them consumable for downstream tools like Apps.

* **Programmatic Output Registration**: You can also use the [Flyte Python SDK](/cloud/platform-capabilities/features/flows/define-flow-artifacts#programmatically-exporting-to-domino-datasets-and-netapp-volumes) to export Flow artifacts to a NetApp Volume and snapshot them automatically as part of your Flow.

### NetApp permissioning for Apps via Data Sources

SDK Access with Permissions: Domino Apps can now access NetApp Volumes through Python or R Datasource SDKs with automatic permission enforcement for secure, shared executions.

### Snowflake deprecation of single-factor authentication

To prepare for the Snowflake deprecation of [single-factor authentication](https://docs.snowflake.com/en/user-guide/security-mfa-rollout#deprecation-timeline), new Snowflake Datasources no longer accept Single-Factor Authentication methods. Existing Datasources show a warning notification to encourage updating the authentication method.

<hr />

## 2025 November

### Default branch selection for Git repos

Project owners can now configure a default Git branch in project settings. The selected branch is automatically populated in relevant workflows across the project and can be updated anytime to match evolving needs.

### Custom autoscaling settings for Apps

[Configure your apps](/cloud/platform-capabilities/features/apps/app-logs-and-versioning/autoscaling-apps) to scale exactly how you need them. Set maximum pods, CPU/memory thresholds, and scale timing when you launch. Then forget about it. Your settings persist across all sessions and are always visible when you hover over the Launch App window.

### NetApp Volumes enhancements

* **Project Template Support**: NetApp Volumes can now be included when creating or using project templates, making it easier to standardize data access across projects.

* **Snapshot Tags for Mounting**: Tag [NetApp Volume snapshots](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots) for programmatic mounting. Add optional commit messages at creation time to improve traceability and workflow automation

### Flow integration with NetApp Volumes

* **Mounting Snapshots in Flows**: Flows can now [mount NetApp Volume snapshots](/cloud/platform-capabilities/features/flows/define-flows#declare-tasks) at runtime, providing read-optimized data across execution contexts.

### Streamlined Governance policy editing

We’ve improved the way that Governance admins can edit policies using the [Policy Builder](/cloud/platform-capabilities/features/governance/define-governance-policies). Great for quick changes and versioning policies over time.

<hr />

## 2025 October

### Apps autoscaling

Domino Apps now support [horizontal autoscaling](/cloud/platform-capabilities/features/apps/app-logs-and-versioning/autoscaling-apps), making it easier to keep apps responsive under varying demand while optimizing resource costs.

* **Automatic pod scaling**: Adjusts app pods based on CPU and memory usage.

* **Default configurations**: Ships with built-in settings.

### Enhanced App discovery experience

Finding and browsing apps is now faster and more visual:

* New card-based layout with filtering, sorting, and support for app thumbnails.

* Pin your favorite apps for quick access.

* Lite Users get a dedicated home screen and improved project search.

### Domino Governance

#### Real-time autosave for evidence notebooks

Governance Evidence Notebooks now autosave in real time. This removes the need to manually click **Save** after editing, and makes sure that visibility rules and compliance checks are triggered immediately as edits are made.

#### Attach snapshots and flow artifacts

You can now attach NetApp volume snapshots and Flow artifacts directly to governance bundles, expanding traceability and auditability across data and workflow assets.

### NetApp Volumes previews

In NetApp Volumes and remote datasets, the UI now supports live previews for additional file types, including:

* **Tabular data** - `.csv`, `.xls`, `.xlsx`, `.parquet`

* **SAS formats** - `.sas7bdat`

* **R formats** - `.rds`

* **Text and metadata** - `.rtf`, `Define.XML`, `.xpt`

### Audit Trail enhanced filtering

You can now create more precise audit queries using operators like `contains`, `is`, and `is not` across all audit attributes - including users, events, projects, and datasets.

### Authenticate with a Personal Access Token

A new **Personal Access Token** panel has been added to the account settings page where you can fetch a short-lived token to use in any requests against the [Domino API](/cloud/reference/api).

### Cost Center dashboard updates

The Cost Center dashboard now shows memory costs of executions alongside the existing costs (CPU, GPU & Storage).

### Keycloak updates

* Upgraded Keycloak to 26.3.5.

<hr />

## 2025 September

### Track, evaluate, and compare generative AI experiments

Domino now makes it easier to [evaluate and improve generative AI systems](/cloud/platform-capabilities/features/agents) with built-in tools for instrumentation, structured evaluation, and comparison.

* **Logging SDK to allow for Universal Tracing**: Single-line instrumentation of GenAI across frameworks, such as LangChain, LlamaIndex, Pydantic, and OpenAI Agents SDK, makes it easy to track interactions in your GenAI System.

* **Trace Explorer**: Visualize GenAI interactions with execution paths, inputs and outputs, and latency to make debugging faster and clearer.

* **Evaluations**: Add metrics or labels to traces from the SDK or UI, enabling programmatic evaluations with your preferred framework or quick hand scoring by SMEs.

* **Trace Diffs**: Compare traces and evaluation metrics across runs to understand how different configurations perform, with a granular view into sample-level differences in your evaluation set.

### RabbitMQ updates

Upgraded RabbitMQ to 4.1.2.

### Enhanced model endpoint monitoring

Model endpoint monitoring is now more capable and accessible:

* **In Domino**: Data scientists can view endpoint performance metrics directly from endpoint version pages, without leaving the interface.

* **In Grafana**: Administrators get [advanced monitoring dashboards](/cloud/admin/operations/grafana-workload-monitoring/model-endpoint-monitoring) with deeper visibility into system metrics and performance trends.

These improvements make it easier to [monitor model health and performance](/cloud/platform-capabilities/features/model-deployment/monitor-model-endpoints), troubleshoot issues, and optimize resource usage.

### App sharing and access management updates

We’ve streamlined how you share and manage access to Apps in Domino:

* A new [Access & sharing](/cloud/platform-capabilities/features/apps/share-an-app) view adapts based on viewer settings, with additional user management options for project-specific apps and clear notices for broader access levels.

* A new **Access & sharing** button is now available on the app overview page.

* After publication, app access is managed through the **Share App** dialog.

* Users now receive an email and in-platform notifications when they are granted app access, including the app name and a direct link to the app details page.

<hr />

## 2025 August

### Governance: Sequential workflows enforce stage order

Sequential workflows now support required evidence fields to improve governance and compliance. Each stage must be completed in order, and bundles cannot move forward until key questions are answered. This ensures proper review, traceability, and audit readiness throughout the workflow.

### TLS 1.3 update

The Transport Layer Security (TLS) configuration for all Domino Cloud environments was upgraded. Our TLS cipher policy to support TLS 1.3 in addition to TLS 1.2, while disabling weaker legacy ciphers from TLS 1.2 by switching to AWS’s default policy configuration: `ELBSecurityPolicy-TLS13-1-2-Res-2021-06`.

### Governance gates for Apps and model endpoints

Policies can now define [gates](/cloud/platform-capabilities/features/governance/gating-domino-governance) that block access to resources like hardware tiers until the governed bundle has the required approvals. Governance admins configure gates in the policy YAML, and enforcement prevents publishing apps or creating model endpoints until approvals are complete.

### UX driven Governance policy configuration

Quickly create, edit, and test governance policies in a streamlined interface. Designed for fast, accurate updates to meet evolving regulatory needs.

### Self-service node pool management

Easily manage node pools in your Domino cluster. This includes viewing existing node pools, adding new ones, and deleting existing pools directly from the Cloud Admin portal.

<hr />

## 2025 July

### Domino Apps enhancements

[Apps](/cloud/platform-capabilities/features/apps) have been reimagined in Domino to include the following improvements:

* A new app publishing UI.

* The ability to create multiple apps in a single Domino Project.

* The new global apps list page and viewing running applications that use iFrames vs those without iFrames (which can use deep links with query parameters).

* The new app details page and switching between viewing different versions.

* (Soft-)Deleting an app.

* Managing user access to apps.

* App users can now deploy an app using any file type. They are no longer limited to `.sh` files only.

### Domino Governance enhancements

* **Multiple Policies per Bundle**: Multiple policies can now be applied to a single bundle, reducing duplication and simplifying reviews. Learn more about [creating](/cloud/platform-capabilities/features/governance/work-with-bundles) or [reviewing bundles](/cloud/platform-capabilities/features/governance/work-with-bundles/review-bundles).

* **Policy Archiving**: Policies can now be archived from the Policies screen. Archived policies are only visible with the Archived filter and can’t be restored or removed if still in use by active bundles. Learn more about [defining policies](/cloud/platform-capabilities/features/governance/define-governance-policies).

* **Improved PDF Bundle Report**: Improved experience in the PDF bundle report, including displaying attachments in an appendix section.

* **Per-Bundle Approval Customization**: [Different approvers](/cloud/platform-capabilities/features/governance/work-with-bundles/review-bundles) can now be assigned to individual bundles under the same policy, enabling more flexible and context-specific reviews.

* Users can now assign individuals to stages, manage multiple bundles at once, and rename terms like “Bundle” or “Deliverable” to match industry standards.

### Workload observability

Cloud admins now have comprehensive visibility into their workload health and performance. Learn more about [monitoring and observability](/cloud/admin/operations/grafana-workload-monitoring).

### Project environment variable redaction in Audit Trail

Project-level environment variables can now be redacted in the audit trail. This functionality is not enabled by default, so these environment variables are logged as plain text. Please reach out to Domino Support to update this setting for your deployment.

### Package persistence for Workspaces

Users are now able to persist packages and home directory files across workspace sessions. Package persistence saves time by retaining installed packages across workspace sessions in Python or R. When properly configured, you don’t need to reinstall your packages after starting or restarting a workspace. Learn more about [package persistence](/cloud/platform-capabilities/core-concepts/workspaces/use-package-persistence).

### Domino Volumes for NetApp ONTAP

[NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes) allow for users to instantly snapshot training data for auditability and reproducibility with no extra storage required.

### Configurable datetime precision for Data Sources

Users can now configure the resolution of datetime columns for relational Data Sources. See the [Data Source overridable configurations page](/cloud/reference/api/data_api/datasource_usecases/config_override) for more information.

<hr />

## 2025 June

### Attach Dataset snapshot files to Governance bundles

Users can now attach files from a dataset snapshot to a governance bundle within the same project. Snapshot files can be linked to an existing bundle or used to create a new one.

### Keycloak updates

* Upgraded Keycloak to 26.1.4.

<hr />

## 2025 May

### Domino Governance

* **Scripted checks**: Use centralized scripts to evaluate information on a Governed Bundle. Scripted checks can also be used to measure bias consistently in a dataset.

* **Findings**: An enhancement to Findings is now available to allow for conditional approval, allowing approvers to select findings that must be completed for the bundle to be fully approved, without obstructing productivity. This also includes the ability to set due dates for findings and receive notifications based on the lifecycle of the approval.

### Streamlined flow output registration to Domino Datasets

Flow Artifacts with type DATA can now be exported to Domino Dataset snapshots via the UI for consumption of Flow Outputs in downstream workflows such as Apps. Traceability between the Flow Artifact and Dataset snapshot is maintained and viewable.

<hr />

## 2025 April

### Cost Center dashboard

The Cost Center Dashboard lets administrators easily view infrastructure costs across their Domino deployment. It provides quick insights into resource usage and cost trends over time. Learn more about [the Cost Center](/cloud/admin/operations/finops).

### Automatic environment rebuilds

Environments can now automatically be rebuilt based on changes to the base environment image. Environments have a toggle in the Revisions tab, letting users choose whether to automatically update to the latest revision or stay on a specific revision.

### Self-service IP allowlisting

Cloud Admins are now able to configure IP allowlisting through the Domino Admin Panel. IP allowlisting is a security feature that lets you choose which IP addresses can access your Domino cluster. This improves security by allowing only trusted IPs and blocking unauthorized access. See our documentation to [configure IP allowlisting](/cloud/admin/architecture-and-deployment/deployment-configuration/ip-whitelisting).

<hr />

## 2025 March

### Custom App URLs

Domino apps provide persistent URLs that remain unchanged across redeployments and can be customized into human-readable formats. This ensures that links can be reliably shared, bookmarked, and understood by users without requiring IT involvement for redirects.

### Governance: Support for Git based Projects

We now support artifacts from Git-Based Project (GBP) bundles for Governance.

### Flexible Git reference support for Apps

Teams can now deploy apps from a specific Git reference (branch, tag, or commit SHA). The reference must contain an `app.sh` file to be successful.

### In-product guides & alerting

A new functionality is now available to enable in-product guides, alerting, and more. This functionality aims to improve adoption and enablement of the Domino Platform.

<hr />

## 2025 February

### Updates to Domino Governance

* **Governance Findings**: Create, track, and manage compliance gaps and potential risks within a governed bundle with Governance Findings.

* **Metrics Checks**: Automatically attach metrics to a governed bundle, eliminating the need for manual entry.

### Configurable idle timeout

Users are now logged out of Domino after a customizable period of inactivity. Users receive a pop-up letting them know they are about to be logged out, and they have the option of dismissing the pop-up to stay logged in. *This setting is turned off by default. Please reach out to Domino Support if you’d like to configure an inactivity period for your deployment.*

### Project templates support for "Other" Git providers

Git-based projects from "other" providers (including Azure DevOps) can now be used as project templates or copied into other projects. To copy a template repository, users must provide an uninitialized destination repository from the same provider.

### Cloud Admin global role improvements

Extended the permission set of the Cloud Admin global role to be able to create and manage Service Accounts.

<hr />

## 2025 January

### CSV download for Audit Trail

Domino’s new **Download as CSV** feature allows CloudAdmin and GovernanceAdmin users to export Audit Trail data directly from the interface. This functionality helps support both internal and external compliance requirements.

When the **Download CSV** button is clicked, the generated CSV files reflect the filters and table columns selected.

<hr />

## 2024 December

### Unified Audit Trail

Domino’s new [Unified Audit Trail](/cloud/platform-capabilities/features/governance/audit-trail) tracks user interactions and system events, allowing organizations to monitor data creation, viewing, and permission changes for better transparency and accountability.

Audit trails capture broader compliance events, such as changes to the following:

* Datasets

* Project configurations

* User roles

* Data access

### Support for Trainium instances

Domino adds support for AWS Neuron/Trainium instances for Domino Cloud.

### Keycloak updates

Upgraded Keycloak to 25.0.6.

<hr />

## 2024 November

### Deploy models to SageMaker

Domino provides [model deployment to SageMaker](/cloud/platform-capabilities/features/models/deploy-sagemaker). Leverage Domino’s flexible development experience to build your models before deploying them to production on SageMaker, while still maintaining centralized governance and tracking of all of your models. Administrators apply guardrails around cost, performance, and security, and the platform picks the resources needed to meet them.

<hr />

## 2024 October

### New Domino user interface

Domino Data Lab introduces a new UI to improve your core workflows. It has an updated look and feel, with new navigation patterns and a dashboard experience to quickly get you into recent work. For more detail, see the [New UI migration guide](https://docs.dominodatalab.com/en/6.0/user_guide/5c9e60/new-ui-migration-guide/).

### NetApp integration

Domino adds support for NetApp FSxN for Domino Cloud.

### Domino Governance

[Domino Governance](/cloud/platform-capabilities/features/governance) simplifies, organizes, and automates the process of gathering, tracking, and reviewing all assets to help enforce internal or external policies. It provides a single interface closely connected and integrated within a data scientist’s environment, ensuring smooth governance, version control, and compliance for all stakeholders involved.

### Datasets in Nexus

[Domino Datasets](/cloud/platform-capabilities/core-concepts/data/datasets) can now be created in remote data planes.

### Project templates

Create [Project templates](/cloud/platform-capabilities/core-concepts/projects/create-projects#create-a-project-from-a-template) to kickstart future projects instead of starting from scratch. Project templates enable collaboration, efficiency, and standardization with your team, organization, and company.

### App secure identity propagation

[App secure identity propagation](/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security) allows apps to identify a Domino user making a request to provide differentiated services to users. Existing apps that host additional assets or secondary pages should adapt their root path as described in the [App security](/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security) section when secure identity propagation is turned on. Running applications are not affected and do not need changes until they are restarted.

### Upgrade notes

* The new Data Plane version has advanced to a major release (now 4.1.1-3). As a result, the upgrade button does not function due to the version gap. To complete the upgrade successfully, a manual process is required. For details, refer to the [upgrade the Data Plane](/cloud/admin/platform-configuration/manage-data-planes#upgrade-a-data-plane) section.

<hr />

## 2024 September

### RabbitMQ updates

Upgraded RabbitMQ to 3.13.4.

<hr />

## 2024 July

### Removing support widget

Domino is removing the Support widget from the UI as part of improving and streamlining our support processes. There are no changes to any other support processes with Domino. You can continue to raise support tickets via the [online portal](https://support.domino.ai/support/s/) or by emailing [support@dominodatalab.com](mailto:support@dominodatalab.com).

### Run Job from Workspace

Users can run jobs directly from the workspace interface, enabling them to run jobs with the latest changes visible on their workspace screen. The job execution button becomes active once all changes have been committed to the repository. Additionally, a summary panel is provided to monitor the user’s most recent job executions.

### Changes to the Admin navbar

The old Admin navbar has been updated with a new visual appearance and improved menu organization.

* Key changes in the **components** include:

  | Previous implementation | Changes to new implementation |
  | - | - |
  | **Admin Area**. | Renamed to **Admin Panel**. |
  | Link assigned to **Admin Home** menu item. | Link assigned to app logo. |
  | **Search** was in the old implementation. | Removed from the new implementation. |
  | - | Page titles and Browser tab titles changed according to the name changes for the menu items. |

* Key changes in the **menu items** include:

  | Menu items | Previous implementation | Changes to new implementation |
  | - | - | - |
  | Projects, Workspaces, Executions | At the top-level navigation. | Part of the submenu of the top-level menu item **Reports**. |
  | Usage, Resource usage report, User activity report, Monitoring (renamed to Grafana Monitoring) | Part of the submenu of the top-level menu item **Advanced**. | Part of the submenu of the top-level menu item **Reports**. |
  | Datasets | Part of the submenu of the top-level menu item **Data**. | Part of the submenu of the top-level menu item **Reports**. |
  | Users, Organizations | Tabs of the top-level menu item **Users**. | Part of the submenu of the top-level menu item **Manage resources** |
  | Governance, Data Planes, Notifications, Manage Cost (renamed to Cost Tracking) | Were top-level menu items. | Part of the submenu of the top-level menu item **Manage Resources**. |
  | Manage Environments (renamed to Environments), Hardware tiers, Resource quotas, External deployments | Part of the submenu of the top-level menu item **Advanced**. | Part of the submenu of the top-level menu item **Manage Resources**. |
  | Data Sources, Feature Store | Part of the submenu of the top-level menu item **Data**. | Part of the submenu of the top-level menu item **Manage Resources**. |
  | Project Stage Configuration (renamed to Project stages), Jira Configuration (renamed to JIRA integration), Diagnostics toolkit | Part of the submenu of the top-level menu item **Advanced**. | Part of the submenu of the top-level menu item **Platform settings**. |
  | Username | Was a top-level menu item. | Part of the submenu of the top-level menu item **Account**. |
  | Account settings, Download CLI, Log out | Part of the top-level menu item **Username**. | Part of the top-level menu item **Account**. |
  | Feedback (opens a feedback modal on click), Exit Admin panel, App Version | Not in the old implementation. | New menu items added to the submenu of the top-level menu item **Account**. |

<hr />

## 2024 March

### Qdrant vector DB Data Source

The new [Qdrant vector database Data Source connector](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-qdrant) enables easy, secure access to vectorized content stored in Qdrant. Domino Data Sources provide enhanced security and governance capabilities while supporting cutting-edge generative AI use cases.

<hr />

## 2024 January

### AI Gateway

The [AI Gateway](/cloud/platform-capabilities/features/llms/llm-gateway) provides Domino users a safe and streamlined way to access Large Language Models (LLMs) hosted by external service providers like OpenAI or AWS Bedrock. This lets users enjoy the benefits of provider-hosted models while ensuring that they follow security best practices.

### Pinecone vector DB Data Source

The new [Pinecone vector database Data Source connector](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-pinecone) enables easy, secure access to vectorized content stored in Pinecone. Domino Data Sources provide enhanced security and governance capabilities while supporting cutting-edge generative AI use cases.


# How-to guides
Source: https://docs.domino.ai/knowledge-base/how-to-guides-overview

Step-by-step guides for apps, environments, CLI tools, and Git integration.

Step-by-step guides for apps, environments, CLI tools, and Git integration.

## Apps

* [FastAPI App with Multiple Endpoints](/knowledge-base/fastapi-app-with-multiple-endpoints)
* [Getting started with Streamlit in Domino](/knowledge-base/getting-started-with-streamlit-in-domino)
* [Use Gradio in a Jupyter workspace](/knowledge-base/use-gradio-in-a-jupyter-workspace)
* [Creating and Publishing React Apps](/knowledge-base/creating-and-publishing-react-apps)
* [Increasing the timeout for Shiny Server](/knowledge-base/increasing-the-timeout-for-shiny-server)
* [App URLs and URL parameters](/knowledge-base/app-urls-and-url-parameters)

## CLI

* [The lsof utility - 'list open files'](/knowledge-base/the-lsof-utility-list-open-files)
* [htop: Interactive process viewer, a brief overview](/knowledge-base/htop-interactive-process-viewer-a-brief-overview)

## Environments

* [Installing Archived R Packages No Longer Available on CRAN](/knowledge-base/installing-r-packages-from-archive)
* [Installing Packages And Dependencies](/knowledge-base/installing-packages-and-dependencies)
* [Configuring and Sending Emails in Domino Using Python smtplib](/knowledge-base/sending-emails-python-smtplib)

## Git Integration

* [Adding a new Git Repository using a Personal Access Token (PAT)](/knowledge-base/adding-a-new-git-repository-to-your-project-using-personal-access-token-pat)
* [How to connect and authenticate to GitLab & GitHub with SSH keys](/knowledge-base/how-to-connect-and-authenticate-to-gitlab-github-with-ssh-keys-using-the-native-bash-terminal)

## Infrastructure and Kubernetes

* [Resolve internal (on-premise) DNS Server in AWS EKS via CoreDNS](/knowledge-base/resolve-internal-on-premise-dns-server-in-aws-eks-via-coredns)

## Jobs

* [Preventing Jupyter timeout errors when running notebooks as a batch job](/knowledge-base/preventing-jupyter-timeout-errors-when-running-notebooks-as-a-batch-scheduled-job)


# Domino Knowledge Base
Source: https://docs.domino.ai/knowledge-base/index

Support articles and troubleshooting guides from the Domino Support team, covering known issues, workarounds, and diagnostic steps.

The Knowledge Base collects support articles written by the Domino Support team: known issues, workarounds, and diagnostic steps that fall outside the product documentation.

<CardGroup>
  <Card title="How-to guides" icon="book-open" href="/knowledge-base/how-to-guides-overview">
    Step-by-step guides for apps, environments, CLI tools, and Git integration.
  </Card>

  <Card title="Troubleshooting" icon="wrench" href="/knowledge-base/troubleshooting-overview">
    Fixes for common errors across environments, workspaces, models, and infrastructure.
  </Card>

  <Card title="Administration" icon="shield" href="/knowledge-base/administration-overview">
    Account and user management tasks for admins.
  </Card>

  <Card title="Support Policies" icon="headset" href="/knowledge-base/support-policies-overview">
    What's covered under Domino Support, and what isn't.
  </Card>
</CardGroup>

For everything else, use the [Domino documentation](/knowledge-base/cloud/platform-overview/domino-overview) for product guides and reference, or contact [Domino Support](https://support.domino.ai) directly.


# Domino 6.3 release notes
Source: https://docs.domino.ai/release-notes/6-3-0

Domino 6.3 release notes: on-demand Slurm HPC compute, a full App SDLC, coding assistants in Workspaces, tags and properties, governance gates, and Extensions.

Domino 6.3 brings on-demand Slurm compute for high-performance computing (HPC) workloads, a full [App](/cloud/reference/glossary#app) software development lifecycle (SDLC), coding assistants in [Workspaces](/cloud/reference/glossary#workspace), [tags](/cloud/reference/glossary#tag) and [Properties](/cloud/reference/glossary#property) for knowledge management, governance enforcement for Apps and [agents](/6.3/platform-capabilities/features/agents), and reusable [Extensions](/cloud/reference/glossary#extension).

## Highlights

### On-demand Slurm compute for HPC workloads

Run existing HPC job scripts from a Workspace while Domino provisions Slurm workers on demand and tracks [HPC Jobs](/cloud/reference/glossary#hpc-job) with other [Project](/cloud/reference/glossary#project) [Jobs](/cloud/reference/glossary#job). Domino deploys Slurm on Kubernetes with [Slinky](https://slinky.schedmd.com/), SchedMD's Slurm operator, so each [Data Plane](/cloud/reference/glossary#data-plane) runs a single persistent Slurm cluster with worker capacity that scales with demand. See [Run HPC workloads with Slurm](/6.3/platform-capabilities/features/compute-clusters-hpc/slurm).

### Full App SDLC

Take Apps from private draft to an immutable published [App version](/cloud/reference/glossary#app-version) with distinct create, publish, and deploy stages. Pinned commits make versions reproducible, restartable, and easier to roll back; see the [App lifecycle](/6.3/platform-capabilities/features/apps/apps-in-domino/app-structure-lifecycle#app-lifecycle) and [App versioning and reproducibility](/6.3/platform-capabilities/features/apps/app-versioning-and-reproducibility).

### Coding assistants in Workspaces

Start with GitHub Copilot, Claude Code, and Codex pre-installed in Domino Standard Environment (DSE) Workspaces, together with Domino skills for common platform tasks. See [Coding assistants](/6.3/platform-capabilities/features/coding-assistants) for availability by IDE and customization options.

### Knowledge management with tags and properties

Give teams a shared way to organize and discover Domino assets. SysAdmins and [Librarians](/6.3/admin/identity-and-access/manage-users/roles) can define tags and typed Properties, while users can find classified assets through search, filters, and the Tag Browser; see [Knowledge management](/6.3/platform-capabilities/core-concepts/knowledge-management).

### Governance enforcement for Apps & Agents

Turn governance requirements into deployment [gates](/cloud/reference/glossary#gate) for Apps and agents by linking them to [governance bundles](/cloud/reference/glossary#bundle). Required [policies](/cloud/reference/glossary#policy) can block deployment, while App versions carry [Evidence](/cloud/reference/glossary#evidence) forward with version diffs; see [Govern models, Apps, agents, and files](/6.3/platform-capabilities/features/governance/work-with-bundles/govern-models-apps-files).

### Extensions

Extensions bring Apps into Project, [Dataset](/cloud/reference/glossary#dataset), [Model](/6.3/platform-capabilities/features/models), file, and Admin pages. [Build your own Extensions](/6.3/platform-capabilities/features/extensions/create-manage-extensions), or install AutoML Studio, Auto Model Documentation, and Clinical Data Explorer from the [official Extensions catalog](/6.3/platform-capabilities/features/extensions/install-domino-official-extensions).

<hr />

## Flows

### Runtime configuration for flow runs

Rerun a registered [Flow](/cloud/reference/glossary#flow) with different inputs, [Compute Environments](/cloud/reference/glossary#compute-environment), [Hardware Tiers](/cloud/reference/glossary#hardware-tier), or Git references without editing its definition. Apply the same overrides from the UI, API, or CLI; see [Launch Flows](/6.3/platform-capabilities/features/flows/launch-flows#relaunch-an-existing-flow).

### Consistent paths and secondary repositories in Flow tasks

Use the same input and output paths across Flows, Workspaces, and Jobs. Flow tasks can also pin a branch, tag, or commit from an imported Git repository; see [Define Flows](/6.3/platform-capabilities/features/flows/define-flows#write-task-code).

<hr />

## Governance

### Governance bundle lifecycle and review controls

See compliance status at a glance and move completed bundles to **Completed** or **Archived**. Reviewers can reset sequential bundles, replace their primary policy, and select eligible Project collaborators as approvers; see [Work with bundles](/6.3/platform-capabilities/features/governance/work-with-bundles).

### Governance stage notifications and reassignment

Keep reviews moving with email and in-product notifications when stages start. A global **Stages** view shows status and assignees and supports bulk reassignment; see [Manage stages across bundles](/6.3/platform-capabilities/features/governance/work-with-bundles#manage-stages-across-bundles).

### Flow artifacts as governance attachments

Capture Flow outputs directly in governance bundles. Flows can attach files automatically, or users can attach them from a run's **Artifacts** tab; see [Flow artifact files](/6.3/platform-capabilities/features/governance/work-with-bundles/attachments#flow-artifact-files).

### Periodic revalidation for Governance

Keep [Approvals](/cloud/reference/glossary#approval) current with a recurring, organization-wide renewal schedule. Bundles remain compliant during the renewal window and become non-compliant if approval expires; see [Periodic revalidation](/6.3/platform-capabilities/features/governance/approvals#periodic-revalidation).

### Policy versioning for Governance

Develop changes to a published policy in a draft, then choose whether the update is mandatory and whether existing Approvals remain valid. Compare versions from the **Changes** tab; see [Create a new version of a policy](/6.3/platform-capabilities/features/governance/define-governance-policies#create-a-new-version-of-a-policy).

<hr />

## Identity and API access

### Personal Access Tokens replace user API keys

Move user-context API access to expiring personal access tokens (PATs) that administrators can audit and revoke. Starting in 6.3, PATs are the supported replacement for legacy user API keys, which remain functional but are deprecated; see [Personal access tokens](/6.3/admin/identity-and-access/manage-users/personal-access-tokens).

### Cluster-specific API documentation

Explore cluster-specific API documentation directly at `https://<your-domino-domain>/docs`. See the [API browser](/6.3/reference/api/domino-open-api#api-browser) for details.

<hr />

## Notes for platform administrators

* **Slurm support is enabled when a Data Plane is deployed.** Set `compute_clusters.slurm.enabled` to `true` when deploying the [Data Plane](/cloud/reference/glossary#data-plane); adding Slurm later requires a redeployment. Create the required Compute Environments, worker Hardware Tiers, and shared Dataset, and enable Data Plane storage and Workspaces for remote Data Planes; deployments that leave Slurm disabled require no action. See [Administer Slurm](/6.3/admin/infrastructure-and-compute/administer-slurm).

- **HPC Job submission from the UI and API is off by default.** Set [`ShortLived.EnableHPCJobs`](/6.3/admin/configuration/feature-flags#param-short-lived-enable-hpc-jobs) to `true` to add a **Run** button to the **HPC** tab. Terminal submission needs no flag; see [Administer Slurm](/6.3/admin/infrastructure-and-compute/administer-slurm).

* **Configuration records have Basic and Advanced views.** **Basic** provides validated controls for common settings, while **Advanced** retains the full editor. The new interface is off by default; set [`EnableNewCentralConfigsEditor`](/6.3/admin/configuration/feature-flags#param-enable-new-central-configs-editor) to `true` to enable it without changing existing values, or leave it off with no action required. See [Basic and Advanced views](/6.3/admin/configuration/central-configuration#basic-and-advanced-views).

- **Administrators can start another user's stopped Workspace through the API.** The `adminStartWorkspace` operation requires `ManageWorkspaces` and preserves the Workspace owner's previous mounts. Pair it with the existing stop operation to automate restarts; see [Start a user's Workspace as an administrator](/6.3/platform-capabilities/core-concepts/workspaces/manage-workspaces#start-a-users-workspace-as-an-administrator).

* **The platform Grafana instance is operator-managed.** Manage custom dashboards, alerts, contact points, and notification policies at `/grafana` as Kubernetes resources. Don't edit Domino-managed resources because upgrades overwrite them; deployments without custom Grafana content require no action. See [Manage Grafana dashboards and alerts as custom resources](/6.3/admin/operations/monitoring/grafana/manage-with-custom-resources).

- **Legacy Grafana remains available during the 6.3.x migration window.** Migrate custom content from `/grafana-classic` to `/grafana` before upgrading beyond 6.3.x; see [Access the legacy Grafana instance during migration](/6.3/admin/operations/monitoring/grafana/access_grafana#access-the-legacy-grafana-instance-during-migration) and [Migrate Grafana dashboards and alerts to the operator-managed instance in Domino 6.3](https://support.domino.ai/support/s/article/Migrate-Grafana-dashboards-and-alerts-to-the-operator-managed-instance-in-Domino-6-3-1789393086479467000) on the Support site (requires sign-in).

* **Deployment identification for product analytics.** Set the optional `customer_name` and `deployment_name` values to identify your organization and environment in product analytics and Support cases. Changes take effect after Domino services restart; see [Deployment identification](/6.3/admin/configuration/deployment-identification).

## Bug fixes

## Known issues

<AccordionGroup>
  <Accordion title="Coding assistants">
    * When working with Workspace coding assistants, if you are authenticating a coding agent for the first time, the link in the popup dialog does not work. To complete authentication, copy the URL displayed in the terminal and open it in your browser.

    * Domino skills might not work by default for coding agents other than Claude Code.

    * In some cases, the VS Code Claude Code extension panel appears blank. To fix this, refresh your browser.

    * Claude Code can default to a model tier other than the one you prefer. Use the model selector at the top of the Claude Code panel to switch tiers.
  </Accordion>

  <Accordion title="Workspaces">
    * When restarting an RStudio workspace with home directory persistence enabled, the user may see a notice stating that the previous session was abnormally terminated due to a crash. This notice is only due to how the new feature interacts with RStudio and does not result in any data loss.
  </Accordion>

  <Accordion title="Flows">
    * [Flows](/cloud/reference/glossary#flow) are supported in Domino Cloud and self-managed deployments on AWS and Azure. Google Cloud and on-premises deployments aren't supported.

    - Some Flyte features are not yet available in Flows, such as [dynamic workflows](https://docs.flyte.org/en/latest/user_guide/advanced_composition/dynamic_workflows.html) and [map\_task](https://docs.flyte.org/en/latest/user_guide/advanced_composition/map_tasks.html).
  </Accordion>

  <Accordion title="Azure Blob Storage">
    * In Azure Blob Store deployments, Projects with many files may fail to sync through the Domino CLI. To work around this issue, do not disable file locking when prompted by Domino.

    - When uploading a large file to the Azure blob store by [syncing a Workspace](/6.3/platform-capabilities/core-concepts/workspaces/sync-changes-in-a-workspace), you may encounter a Java Out of Memory error from Azure if the file/blob already exists. To work around this issue, use the [Domino CLI](/6.3/reference/cli) to upload the file to the Project.

    * Domino instances that make use of Azure Blob Storage may experience stalled Jobs within Projects with many large files.
  </Accordion>

  <Accordion title="Projects and files">
    * If you attach a Git repository to a DFS Project that points to a tagged release, the tag is not honored when building a Domino endpoint in that Project. The build log shows an error similar to the following, and the model is built using the default branch of your Git repository instead of the tagged branch:

      ```
      Jul 05 2023 14:36:27 -0500 #10 6.481 WARN [d.r.d.GitRepoUpdater] could not parse ref: v1.3.0 checking out default branch correlationId="iA2qWrYSLQ" thread="main"
      ```

      To work around this issue, use the branch name when building Domino endpoints instead of the release tag.
  </Accordion>

  <Accordion title="Sessions and authentication">
    * If idle timeout is configured on a deployment, Domino may not recognize pages opened in spawned tabs (for example, via `Ctrl+Click`) or outside of an iFrame as being active. Users may be unintentionally logged out despite activity in the spawned pages.

    - If an admin resets a user’s password, it invalidates all the user’s authentication tokens, including tokens used for long-running tasks like Jobs, Workspaces, or Apps. The user must create a new password, log back into Domino, and restart all executions. This also applies to CLI authentication; the user must re-login to their Domino CLI.
  </Accordion>

  <Accordion title="Monitoring">
    * [Model Monitoring](/cloud/reference/glossary#model-monitoring) data sources aren’t validated. If you enter an invalid bucket name and attempt to save, the entry goes through. However, you can’t see metrics for that entry, because the name points to an invalid bucket.

    - Agent monitoring dashboard retains only the current day’s data (charts have missing data and traces are gone). Historical data from previous days are lost, regardless of what time window is selected.
  </Accordion>

  <Accordion title="Platform and infrastructure">
    * During a 48-hour soak test, the `nginx-ingress-controller` pods exhibited a gradual increase in memory usage while handling heavy [Domino endpoint](/cloud/reference/glossary#domino-endpoint) workloads. The cause is under investigation.
  </Accordion>
</AccordionGroup>

## Validated frameworks

Domino 6.3.0 validates the following framework versions. Other versions might be compatible but aren't guaranteed.

| Framework | Version |
| :- | :- |
| [Kubernetes](/6.3/admin/architecture-and-deployment/kubernetes-infrastructure#compatibility-matrix) | See the compatibility matrix |
| [Apache Spark](/6.3/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/validated-spark-version) | 4.1.2 |
| [Dask](/6.3/platform-capabilities/features/compute-clusters-hpc/dask/validated-dask-version) | 2026.3.0 |
| [Ray](/6.3/platform-capabilities/features/compute-clusters-hpc/ray/validated-ray-version) | 2.55.1 |

## Upgrade notes

<Warning>
  Domino upgrades its PostgreSQL service databases from version 11 to 17 during the 6.3.0 upgrade. Before starting the upgrade, take a full filesystem-level backup of each PostgreSQL persistent volume. The migration changes data in place and can't automatically roll back after it starts. Coordinate the backup with your Domino representative and plan a longer maintenance window than usual for the database upgrade.
</Warning>

<Warning>
  NGINX ingress now runs a Federal Information Processing Standards (FIPS)-validated OpenSSL that drops the legacy static-RSA TLS ciphers. If you set an explicit `ssl-ciphers` override and front NGINX with an AWS Classic Load Balancer over TLS, update it before upgrading or the installer stops with a configuration error. Default installs and Network Load Balancer (NLB) or Application Load Balancer (ALB) deployments are unaffected. See [Deploy Domino on AWS EKS](/6.3/admin/installation/manage-deployments/aws/deploy).
</Warning>

This release supports the following versions in the [Kubernetes compatibility matrix](/6.3/admin/architecture-and-deployment/kubernetes-infrastructure#compatibility-matrix):

| Platform | Kubernetes version |
| :- | :- |
| Red Hat OpenShift | 1.34 (OpenShift 4.21) |
| Google GKE | 1.34, 1.35 |
| Azure AKS | 1.34, 1.35 |
| Amazon EKS | 1.34, 1.35 |
| Nexus data plane (non-vCluster on Amazon EKS) | 1.34, 1.35 |

## Breaking changes & deprecations

* **Legacy Domino user API keys are deprecated.** Existing keys still work in 6.3, but migrate existing integrations and build new ones with a Personal Access Token or [Domino Service Account](/6.3/admin/identity-and-access/manage-users/service-accounts); see [Domino API authentication](/6.3/reference/api/domino-api-authentication).

- **The `project.tag.approved=` search operator is removed.** Filter by tag from the search bar or filter panel instead; `project.tag=` is unchanged. See [Use Domino search](/6.3/platform-capabilities/features/collaboration/domino-search#filter-results-by-tag).

* **Project Tasks, Goals, and the Project Stages and Status UI are removed.** Use your organization's issue tracker for task-level milestones; the read-only Project Status badge, [Activity feed](/6.3/platform-capabilities/core-concepts/projects/collaborate-on-projects#view-project-activity), and backend Goal APIs remain available. Domino migrates existing Project goal comments to bundle comments during the upgrade.

- **The Project-level Jira integration UI is removed.** Project-to-ticket links and task synchronization are no longer shown, while the backend Jira APIs and deployment-wide OAuth registration remain unchanged.

* **The legacy platform Grafana instance at `/grafana-classic` is deprecated.** Migrate custom content to `/grafana` before upgrading beyond 6.3.x; see [Access Grafana](/6.3/admin/operations/monitoring/grafana/access_grafana) and [Migrate Grafana dashboards and alerts to the operator-managed instance in Domino 6.3](https://support.domino.ai/support/s/article/Migrate-Grafana-dashboards-and-alerts-to-the-operator-managed-instance-in-Domino-6-3-1789393086479467000) on the Support site (requires sign-in).

Additional component updates are available in [ddlctl and fleetcommand-agent Release Notes](/6.3/admin/installation/releases).


# Domino Cloud release notes
Source: https://docs.domino.ai/release-notes/cloud-release

What shipped in each Domino Cloud release, for practitioners and platform administrators.

This page includes release notes and updates for [Domino Cloud](/cloud/platform-overview/deployment-options#domino-cloud). Keep track of upcoming changes, new features, and improvements.

Releases are listed newest first. Scroll down for previous Cloud releases. For self-managed release notes, see [Domino 6.3](/release-notes/6-3-0).

## September 2026

September 30, 2026

Cloud Release 93 lets Governance visibility rules read numeric answers, sends in-app notifications for Governance events, adds a **Trigger** column to the [Jobs](/cloud/reference/glossary#job) table, and redesigns the Git reference picker. It also reorganizes access settings for [Domino endpoints](/cloud/reference/glossary#domino-endpoint) and narrows what endpoint Viewers can see.

### Governance

#### Numeric answers in visibility rules

[Policy](/cloud/reference/glossary#policy) visibility rules can now read answers to `numeric` questions through a `numericInputs` map, so a rule can show a question or evidence set only above a threshold, such as `numericInputs["drift-score"] > 0.75`. Previously, answers to `numeric` questions weren't available to visibility rules. The existing `atoi` and `atof` helpers still convert numeric-looking string answers from other question types. See [Visibility rules](/cloud/platform-capabilities/features/governance/define-governance-policies/policy-components#visibility-rules).

#### In-app notifications for Governance events

Governance now sends an in-app Domino notification alongside the existing email when an [approval](/cloud/reference/glossary#approval) is requested, a [finding](/cloud/reference/glossary#finding) is assigned to you, a finding review is requested, a finding reaches a final state, or a [bundle](/cloud/reference/glossary#bundle) [stage](/cloud/reference/glossary#stage) is assigned to you. The navigation bar alerts you to new notifications. See [View event notifications](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#view-event-notifications).

<hr />

### Jobs, Projects, and development

#### Trigger column in the Jobs table

The Jobs table in a [Project](/cloud/reference/glossary#project) has a new **Trigger** column that shows **Scheduled** or **Manual** for each Job. Previously, a clock icon and "Scheduled by" hover text in the **User** column marked a [Scheduled Job](/cloud/reference/glossary#scheduled-job) run; the **User** column now shows only the user's name. See [Jobs Dashboard in a Project](/cloud/platform-capabilities/core-concepts/jobs/create-and-run-jobs#jobs-dashboard-in-a-project).

#### Git tag in Job details

When a Job runs from a Git tag, the code panel in its details shows a **Tag** field alongside **Commit** and **Branch**. The field shows `--` when the Job didn't use a tag.

#### Redesigned Git reference picker

The Git reference picker, used when you create an [App](/cloud/reference/glossary#app), run a Job, submit a high-performance computing ([HPC](/cloud/reference/glossary#hpc-job)) Job, or run a [flow](/cloud/reference/glossary#flow) or override a flow node, is redesigned. Choose a reference type, then enter or search for the matching branch, commit, or tag. The options are renamed: **Git repo default branch** (formerly **Use default branch** or **Use repo default git branch**), **Other branch** (formerly **Branch**), **Commit ID**, **Tag**, **Custom Git reference** (formerly **Custom Git Ref**), and **Project default branch** (formerly **Use project default git branch**).

#### Spaces become hyphens in Project names

When you type a Project name while creating a Project or copying a [Git-based Project](/cloud/reference/glossary#git-based-project), Domino replaces spaces with a hyphen, and a run of spaces becomes one hyphen. Other invalid characters still show a validation error. See [Create Projects](/cloud/platform-capabilities/core-concepts/projects/create-projects).

<hr />

### Apps, endpoints, data, and environments

#### Endpoint access settings on one tab

The **Access & Sharing** tab in a Domino endpoint's **Settings** is renamed **Access** and now sets default access and named collaborators in one form with a single **Save** button. Default access options are **Anyone in Domino**, **Discoverable**, and **Restricted**, and collaborators get the **Viewer**, **Editor**, or **Owner** role. An endpoint must keep at least 1 Owner. The **Invocation** tab is marked **Deprecated**, but you still use it to choose **Unrestricted** or **Restricted** invocation and to add endpoint access tokens. See [Domino endpoint security](/cloud/platform-capabilities/features/model-deployment/security).&#x20;

#### Dataset file list keeps filters and pagination

The file list of a [Dataset](/cloud/reference/glossary#dataset) now keeps its filter, sort order, page, and page size when you open a file preview and go back.

#### Read-only toolbar for NetApp Volume snapshots

When you view a snapshot of a [NetApp Volume](/cloud/reference/glossary#netapp-volume), the **Files** toolbar offers only **Download selected files** and **Delete Snapshot**, because snapshots are read-only. The latest view keeps **Take Snapshot**, **Upload files**, and the **Actions** menu. See [View and edit NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/view-and-edit-netapp-volumes).

#### Environment properties in the details sidebar

On a [Compute Environment's](/cloud/reference/glossary#compute-environment) details page, [Property](/cloud/reference/glossary#property) values move from their own tab into the details sidebar, next to the description, environment base, visibility, and [tags](/cloud/reference/glossary#tag). The remaining tabs are **Overview**, **Revisions**, **Projects**, **Endpoints**, and **Dependency Tree**.

#### Request access from App and agent tabs

If you can't access an App's or [agent's](/cloud/platform-capabilities/features/agents) inputs or monitoring data, the **Inputs** and **Monitoring** tabs now say so and offer a **Request access** button, which changes to **Access pending** or **Access denied** as your request is handled. The API URL in the details sidebar appears only to users with access to the App.

<hr />

### Notes for platform administrators

* **Users sign in again after this update.** Browser sessions from before Cloud Release 93 end when the update reaches your deployment, so users need to sign in again. No action is required.

- **Starburst data stores stay visible when Starburst is off.** When you create a [Data Source](/cloud/reference/glossary#data-source) from **Admin** > **Data sources** and Starburst isn't enabled, the data store list shows the Starburst data stores as disabled, below the native ones, with a notice that on Domino Cloud tells you to contact Domino to provision a Starburst cluster. No action is required.

* **Execution support bundles include resource usage.** Support bundles now contain `resource-usage.json`, with CPU and memory usage over the run, percentile summaries, disk usage, [Hardware Tier](/cloud/reference/glossary#hardware-tier) limits, and GPU usage when the run used GPUs and metrics are enabled, plus `service-metadata.json`. No action is required.

* **Security fixes.** This release remediates known vulnerabilities in the dependencies of platform components. No action is required.

### Bug fixes

* `workerStorageMB` in responses from `POST /api/jobs/v1/jobs`, `GET /api/jobs/beta/jobs`, and `GET /api/jobs/beta/jobs/{jobId}` now reports megabytes. Previously it reported megabits, eight times the actual value, so 5 MB came back as `40`. If your code divided this value by 8, remove that conversion.

- Adding an [Organization](/cloud/reference/glossary#organization) as a collaborator on a Domino endpoint no longer breaks the endpoint's overview page.

* When an ONTAP create or delete call fails, NetApp Volumes no longer end up existing in Domino but not in ONTAP, or the reverse.

- Runs no longer fail with "Execution pod not found" while preparing when Domino hasn't yet registered a pod that exists.

* Signing in through Domino Connect no longer removes Organization memberships that come from single sign-on (SSO).

- Administrators with the `ManageOrganizations` permission can view and edit the members of any Organization through `GET /v4/organizations/{organizationUserId}` and `PUT /v4/organizations/{organizationUserId}/members` without being a member.

* The **Last Used** value in the admin Dataset snapshot deletion windows shows when a snapshot was last mounted, or `--` if it never was, instead of its creation time.

- For a failed App in a [Workspace's](/cloud/reference/glossary#workspace) [App Preview](/cloud/reference/glossary#app-preview), the **Support bundle** section has a working **Download** link.

* In the Governance [evidence](/cloud/reference/glossary#evidence) notebook, the stage sidebar and the question area scroll independently.

- Governance date fields save automatically when you leave the field.

* The **Added** column in a bundle's attachments table shows the time an attachment was added, not only the date.

- The **Status** column in the user and admin personal access token tables is renamed **Expiration date**, matching what it shows.

* A flow artifact's download button reads **Download (n) files**, or **Download file** for a single file, making clear that it downloads the whole artifact.

- Long [Launcher](/cloud/reference/glossary#launcher) descriptions in the run dialog wrap between words instead of breaking mid-word.

* Long App and bundle names in the Apps & Agents table truncate to one line and show the full name when you hover over them.

- Long values in details sidebars wrap instead of showing a horizontal scroll bar.

* The Domino endpoint details page's browser tab reads "Endpoint Details - Domino" instead of "Domino Admin", the Dataset details breadcrumb shows **Datasets** instead of `Datasets / --`, and the **Edit Environment Visibility** dialog closes from its close button or by clicking outside it.

### Breaking changes & deprecations

* Domino endpoint Viewers keep access to an endpoint's overview, versions list, and tester, but can no longer open version details, logs, or monitoring results; those requests now return an error. Previously, Viewers could also view versions and logs. The logs and **Audit Log** links are hidden from Viewers, but some links to version details and monitoring still appear and fail, and the endpoint's running status doesn't refresh automatically for Viewers. Existing collaborator roles aren't migrated. To restore this access, an endpoint Owner can give the collaborator the Editor role, which also lets them publish versions and change settings. See [Add collaborators](/cloud/platform-capabilities/features/model-deployment/security#add-collaborators).

<hr />

September 3, 2026

Cloud Release 92 adds more precise Domino endpoint sharing, expands knowledge management to Policies and NetApp Volume search, and brings point-in-time data selection and coding assistants to more compute workflows.

### Apps, development, and compute

#### Pin data to a specific snapshot at launch

When you launch a [Workspace](/cloud/reference/glossary#workspace) or [Job](/cloud/reference/glossary#job), or define a [Scheduled Job](/cloud/reference/glossary#scheduled-job), choose independently for each mounted [Dataset](/cloud/reference/glossary#dataset) or [NetApp Volume](/cloud/reference/glossary#netapp-volume) whether to mount all snapshots or pin one. A pinned [snapshot](/cloud/reference/glossary#snapshot) mounts read-only at the asset's usual path, so existing code can read point-in-time data without a path change. See [Choose a snapshot per asset at launch](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots#choose-a-snapshot-per-asset-at-launch).

#### Coding assistants in Slurm Workspaces

The `domino-slurm-environment` image now includes Claude Code, Codex, and GitHub Copilot CLI, together with Domino plugins, skills, and agents. Claude Code and Codex are also available as VS Code extensions, and assistant configuration persists between Workspace sessions. See [Coding assistants](/cloud/platform-capabilities/features/coding-assistants).

#### Recognizable NetApp Volume names in ONTAP

New NetApp Volumes use the Domino display name as their immutable ONTAP name instead of a generated `trident-{volume-id}` name, making them easier to identify from storage tools. Renaming a Volume in Domino does not rename it in ONTAP, and existing Volumes keep their current ONTAP names. See [NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes).

#### Redesigned NetApp Volume details page

The NetApp Volume details page now matches Domino's standard detail page layout: breadcrumbs, a page-level **Share** button for permissions, and a sidebar with the editable description, [Data Plane](/cloud/reference/glossary#data-plane), storage name and path, maximum snapshots, [tags](/cloud/reference/glossary#tag), and [Property](/cloud/reference/glossary#property) values. A snapshot selector next to the volume name switches between the latest data and a read-only view of any snapshot, and the tabs are now labeled **Files** and **Access**. The **Files** tab adds search, sortable and configurable columns, pagination, **Take Snapshot** and **Upload files** buttons, and bulk download and delete of selected files from an **Actions** menu. See [View and edit NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes/view-and-edit-netapp-volumes).

#### Programmatic App access URLs

An [App](/cloud/reference/glossary#app) overview now shows the programmatic access URL alongside its vanity URL, so you no longer have to derive the address manually.

#### Redesigned App and Agent details page

The details page for [Apps](/cloud/reference/glossary#app) and [Agents](/cloud/platform-capabilities/features/agents) now keeps the description, status, running duration, creator, creation date, [Data Plane](/cloud/reference/glossary#data-plane), project, tags, and custom fields in a details panel that stays visible on every tab. A new **Inputs** tab shows the selected version's code details and lists its Datasets, [Data Sources](/cloud/reference/glossary#data-source), NetApp Volumes, and [External Data Volumes](/cloud/reference/glossary#external-data-volume-edv) in one table, and a new **Monitoring** tab holds the deployment configuration and runtime metrics. These replace the **Overview** and **Fields** tabs. The **Usage**, **Performance**, and **Governance** tabs are unchanged. See [Understand App usage](/cloud/platform-capabilities/features/apps/app-logs-and-versioning/understanding-app-usage).

#### Resolve a Job's configuration through the API before starting it

`POST /api/jobs/v1/projects/{projectId}/job-defaults` reports the exact hardware tier, environment and revision, Git ref, snapshots, and volumes a [Job](/cloud/reference/glossary#job) in a [Project](/cloud/reference/glossary#project) would run with. Each field you send is applied as an override, and Domino fills in the rest from the Project's defaults. Resolved fields use the same names as `POST /api/jobs/v1/jobs`, so you can pass them straight into a `startJob` request. See [Resolve Job defaults before you start a Job](/cloud/platform-capabilities/core-concepts/jobs/create-and-run-jobs#resolve-job-defaults-before-you-start-a-job).

<hr />

### Governance and knowledge management

#### Tags and properties extend to policies

[Tags and Properties](/cloud/platform-capabilities/core-concepts/knowledge-management) now cover [Policies](/cloud/reference/glossary#policy). Classify a Policy against your organization's [tag structure](/cloud/reference/glossary#tag), record typed [Property](/cloud/reference/glossary#property) values, and set Properties independently on each [Policy version](/cloud/reference/glossary#policy-version). The Policies list shows and filters by Tags, and creating a Policy from an existing one carries its Tags and Properties forward.

The [Compute Environments](/cloud/reference/glossary#compute-environment) list also shows and filters by Tags. Duplicating an Environment now copies its Properties as well as its Tags.

#### Find NetApp Volumes with global search

[NetApp Volumes](/cloud/reference/glossary#netapp-volume) now appear in [Domino search](/cloud/platform-capabilities/features/collaboration/domino-search). Search across a Volume's name, description, filesystem, owner, associated [Projects](/cloud/reference/glossary#project), Tags, and Properties, then open the Volume from the results table or command palette. Results are filtered against RemoteFS permissions so users see only Volumes they can access.

<hr />

### Identity and API access

#### Device code sign-in for the Domino CLI

Sign in from a machine that has no local browser with `domino login --device https://your-domino-url`. The [Domino CLI](/cloud/reference/cli/install-the-cli) prints a verification URL and short code that you can open on another device while the CLI waits for authorization, which is useful on remote hosts reached over SSH.

#### Set a repository's default Git reference through the public API

`PUT /api/projects/v1/projects/{projectId}/repositories/{repositoryId}/default-ref` updates the branch, tag, commit, or default-branch head that Domino checks out for a [Project's](/cloud/reference/glossary#project) main repository or one of its imported repositories; pass `_` as the repository ID to target the main repository. Previously this was only possible through an internal endpoint, which is now deprecated in favor of the public path. See [Change a repository's default reference](/cloud/platform-capabilities/core-concepts/projects/dfs-projects#change-a-repositorys-default-reference).

#### Fetch a rendered view of a repository file through the public API

`GET /api/projects/v1/projects/{projectId}/repositories/{repositoryId}/rendered-file` returns a file from a Project's main repository or one of its imported repositories as an HTML fragment, for formats Domino knows how to render such as Markdown and notebooks. Pass `_` as the repository ID for the main repository, and use `fileName` with an optional `branchName` or `commit` to pick the file; `commit` takes precedence. The HTML comes from repository content, so sanitize or sandbox it before inserting it into a page, and note that images and links in it point at an internal route outside the public contract. Previously this was only possible through an internal endpoint, which is now deprecated in favor of the public path.

<hr />

### Notes for platform administrators

* **Remote [Data Planes](/cloud/reference/glossary#data-plane) can now run on Google Kubernetes Engine (GKE).** The installer now provisions GKE Data Plane DNS, certificates, storage, and workload identity, including deployments configured without Data Plane storage. Existing Data Planes require no action; work with your Domino representative when [adding a GKE Data Plane](/cloud/admin/platform-configuration/manage-data-planes).

- **Cloud administrators can edit limits on customer-created node pools.** From **Platform Settings** > **Deployment Configuration**, update a node pool's description, maximum CPU, and maximum memory without recreating it or its [Hardware Tiers](/cloud/reference/glossary#hardware-tier). Built-in node pools remain read-only. Action is required only to add, change, or remove a limit; see [Manage node pools](/cloud/admin/infrastructure-and-compute/manage-compute-resources/node-pools#edit-a-node-pool).

* **Platform backups cover more service databases.** Data Importer backup and restore workflows now include the Taxonomy, Filetask, IDSM authorization, and RemoteFS databases. This change protects Tags, Properties, NetApp Volume metadata, snapshots, and grants during disaster recovery and requires no action. See [Disaster recovery](/cloud/admin/operations/disaster-recovery).

- **Executions started with Domino Service Account tokens keep API access after the initial token expires.** Domino now obtains an offline session token for an App or Job started from a service-account request. This change is transparent and requires no action.

### Bug fixes

* Restarting an unchanged LLM endpoint now reuses its existing [App version](/cloud/reference/glossary#app-version) instead of creating another App version and instance.

- The AI endpoint sharing dialog now lets you grant access to named users and organizations from either a Project or the global Endpoints page.

* Pausing and resuming a Scheduled Job now preserves its storage and snapshot selections.

- [HPC Job](/cloud/reference/glossary#hpc-job) tables and details now report the correct submission script, CPU percentage, average CPU, peak memory, resource requests, and array-job submission details.

* Governance [bundle](/cloud/reference/glossary#bundle) PDF reports now generate when the bundle contains upgraded Policies.

### Breaking changes & deprecations

* Forking a [Domino File System (DFS)](/cloud/reference/glossary#domino-file-system-dfs) Project is removed from the Domino Cloud UI. The Project overview page no longer offers the **Fork** and **Request Review** actions, so you can't create a fork or merge one back from the UI. The underlying [API](/cloud/reference/api) endpoints are unchanged, and existing forks continue as standalone Projects. To move a fork's changes into the original Project, [export and import Project content](/cloud/platform-capabilities/core-concepts/projects/manage-projects#export-and-import-project-content) or use the API. To duplicate a DFS Project, [copy the Project](/cloud/platform-capabilities/core-concepts/projects/manage-projects#copy-a-dfs-project). For branch-and-merge collaboration, use a [Git-based Project](/cloud/reference/glossary#git-based-project), which keeps your Git provider's pull requests, branch protection, and CI; see [Git-based Projects](/cloud/platform-capabilities/core-concepts/projects/git-based-projects). Forking is still available in self-managed Domino 6.3.

- The legacy [AI Gateway](/cloud/reference/glossary#ai-gateway-legacy) is removed. See [LLM Gateway 2.0](/cloud/platform-capabilities/features/llms/llm-gateway) for the successor and migration path.

<hr />

## 2026 July

### Cloud Release 90

Shipped July 28, 2026.

#### New features

##### Governance for Apps

[Apps](/cloud/reference/glossary#app) now carry a **Governance** tab, so an App is reviewed and approved in the same place it's built. Create a [bundle](/cloud/reference/glossary#bundle) when you create the App or later from the Governance tab, and Apps that are already gated link to their existing bundle instead of starting a new one.

Mandatory [policies](/cloud/reference/glossary#policy) attach automatically to each new App version, so a new version can't quietly ship without the review its policy requires. When you publish, the enforced-policy view shows which policies apply and what's outstanding. Reviewers can diff evidence sets between App versions to see exactly what changed since the last approval.

Previously, governance covered projects and models, and App review had to be tracked outside Domino. Apps Governance is on by default in this release, and the Data Importer carries Apps Governance records through lift-and-shift migrations.

##### Apps SDLC: build, preview, and publish Apps from a Workspace

You can now develop an App end to end without leaving a [Workspace](/cloud/reference/glossary#workspace). Start a live preview of a draft App next to your code, apply new commits to the running preview with **Sync all** instead of recreating it, and publish from the Workspace when the App is ready, either as a new App or as a new version of an existing one.

Previews work for [Projects](/cloud/reference/glossary#project) backed by the Domino File System and for imported Git repositories. The App API exposes the last-synced commit state for main, DFS, and imported repositories, so you can tell what a preview is actually running. Previews stop when the Workspace shuts down, so a forgotten preview doesn't keep consuming compute.

##### Submit HPC jobs to Slurm

High-performance computing (HPC) runs now have a dedicated **HPC** tab alongside [Jobs](/cloud/reference/glossary#job). It's the Jobs table plus 2 columns that matter for HPC: the [Data Plane](/cloud/reference/glossary#data-plane) the run landed on and the number of GPUs it used.

Submit HPC jobs to a Slurm cluster running on a Data Plane, from Git-based Projects as well as Domino File System Projects, and pin a submission to a specific commit. Submission validates 2 conditions before it accepts a job: Slurm is installed on the target Data Plane, and a nodeset on that Data Plane is provisioned for your Project. Slurm workloads run on dedicated nodes, which fixes GPU jobs that previously failed because Slurm and Kubernetes workloads shared nodes with manually assigned resources, and multiple Slurm jobs can now run on a single node.

#### Enhancements

* Tags and Properties now cover [Compute Environments](/cloud/reference/glossary#compute-environment). Tag and add properties to an Environment, browse Environments in the Tag Browser and property browser, and search them by tag, tag description, or property value. Environment owners get the new `EditEnvironmentTaxonomyTag` and `EditEnvironmentPropertyValues` permissions, so tagging follows environment ownership rather than platform admin rights. Duplicating an Environment copies its tags.

- Model Registry search is consolidated into a single bar. Free-text search on model name and MLflow tags, and taxonomy tag chips, now live in one control instead of 2 adjacent search boxes. Properties on models are set per model version rather than per model name.

* Global search shows entity counts per type, adds columns to the results table, and covers files and Datasets. Property queries such as `project.name=` return the matching Project. Grouping and a **Clear** control on the entity type filter make long result lists easier to scan.

- The Posit Package Manager integration is on by default, set through the `com.cerebro.domino.workbench.workspace.enablePpmUI` central configuration flag. The Workspace panel also surfaces packages from repositories that had no snapshots before: previously only repositories subscribed to a public source carried the date-based snapshot labels the panel needs.

* [Flows](/cloud/reference/glossary#flow) run on remote Data Planes by default, through the `com.cerebro.domino.flows.enableRemoteDataPlaneSupport` central configuration flag. Flows fixes in this release: NetApp Volume snapshots declared in a Task's data configuration now mount, input files are no longer removed by executor cleanup, artifact export succeeds at large artifact counts, and raw file uploads retry on transient idle-connection errors.

- [Extensions](/cloud/reference/glossary#extension) improvements: Domino-official Extensions sort first, install cards link to the matching configuration accordion and back, and NetApp Volume and NetApp Volume file mount points are available in manifest v1 (data volume links now carry the snapshot ID). The installer warns when an installed Extension reports `isCompatible=false`, when a version lookup fails, and when the install list is degraded. Cancelling an upgrade restarts the previous App version rather than leaving the Extension down.

* Job and scheduling fixes: an invalid cron expression is rejected before it's saved instead of deleting the scheduled job, editing a scheduled job saves the revision shown in the Environment field rather than defaulting to Active, External Data Volume checkboxes in **Start a Job** default to checked again, and job cluster launch forms accept unrestricted [Hardware Tiers](/cloud/reference/glossary#hardware-tier) for Spark, Ray, and Dask worker slots instead of silently clearing the selection.

- The global Apps list loads faster on large deployments, where a separate instance lookup for every App previously caused timeouts. Saving **Edit App configuration** with no changes no longer creates an empty version, App view counts are consistent, and the list no longer fails on older App records that are missing user information.

* Governance fixes and polish: the Policy Enforcement page has updated terms and an **Edit** button for safeguards, published policies no longer display as archived, findings from earlier policy versions stay reachable after an upgrade, resetting a bundle uses the correct policy version, approval notification links point to the right place, and the **Attach** button is disabled until a policy is selected.

- Idle timeout preserves your context. When Domino signs you out for inactivity, signing back in returns you to where you were, such as a Workspace or a Dataset, instead of the home page.

* Git fixes: the Project code view loads after Git credentials are removed, so you can select a new credential from the UI, and default credential fallback no longer picks a credential for the wrong provider. Bitbucket Cloud HTTPS credentials support the username and API key scheme that replaces App Passwords. In Workspaces, **Continue sync** stays enabled when conflicting files are in subdirectories.

- Data fixes: mounting a [Dataset](/cloud/reference/glossary#dataset) that was shared with you works, mounts resolve for Dataset selections on remote Data Plane executions, and Volume tags appear in the volumes list. The `/entity-tags` endpoint filters on the caller's permissions, tag search filters nested items in tree view, App, Dataset, and Volume tags and properties copy into Project templates, and a nightly job clears orphaned tags and properties.

* Agent performance charts get usability and functional improvements. Domino Model Monitor (DMM) notification settings accept an SMTP relay with no username or password, which unblocks internal IP-based relays that don't require authentication.

- **Tags now extend to Models.** Tag, filter, and browse Models through your organization's [tag structure](/cloud/reference/glossary#tag), which completes tag coverage across Apps, Projects, Project Templates, Datasets, NetApp Volumes, and Models. Tags on a Model are separate from any MLflow tags on the underlying model. See [Tags](/cloud/platform-capabilities/core-concepts/knowledge-management/tags).

#### Notes for platform administrators

* Secure identity propagation for Apps is enabled by default in Domino Cloud. `SecureIdentityPropagationToAppsEnabled` (enhanced) and `com.cerebro.domino.apps.extendedIdentityPropagationToAppsEnabled` (extended) both default to `true`. Previously both were off by default and had to be requested. Running Apps aren't affected until they're restarted. Action is required only for Apps that host secondary pages or additional assets: those need the root-path change described in the App security and identity documentation before their next restart.

- The API Gateway (Envoy Gateway with a gRPC external-authorization service) gained a large set of routing and authentication fixes ahead of general availability. Nucleus routes outside `/api/` and `/v4/` no longer return 404 from execution pods, the Governance, Cost, and Audit Trail APIs resolve in-cluster whether or not the gateway is on, Arrow FlightService keeps its audit trail and API-key authentication, the nginx sidecar stops overwriting an explicit `Authorization` or `X-Domino-Api-Key` header set by user code, and the external-authorization check has an explicit timeout so a slow check can't stall the gateway. Envoy Gateway is re-pinned to 1.8.2 before 1.7.3 reaches end of life. No action is required.

* SpiceDB authorization data migrations run by default. A readiness gate reads the `domino-platform/authz-data-migrations` ConfigMap and keeps authorization on the legacy path until the required migrations are applied, blocking grant writes only while a planned migration is pending. Pre-flight and post-migration checks compare expected and written relationship counts and sample permission checks. Organization membership reads move to SpiceDB as the source of truth, behind a flag that can be reverted at runtime, and Domino logs a message when the legacy and SpiceDB authorization systems disagree. No action is required.

- Keycloak client secrets are randomized per deployment. The `domino-play` and `realm-management` clients previously shipped with a secret hardcoded in the realm configuration, which anyone with that value and network access could use to mint a service-account token. A dedicated Keycloak client is also added for the workload service. No action is required.

* ECR credential handling: the `nucleus-ecr-credential-refresher` CronJob refreshes credentials for `image.registry` as well as `config.dockerRegistry.host`. Where those point at different ECR accounts, the platform-image token previously expired after roughly 12 hours and remote Data Planes started failing with `ImagePullBackOff`. On AWS GovCloud, the credential refresher init job now calls the GovCloud STS endpoint instead of the commercial one, which unblocks upgrades that use ECR as the external Docker registry. No action is required.

- Karpenter `EC2NodeClass` resolves the root device name from the Amazon Machine Image (AMI) that's actually selected. Previously the device name was set from the global `eks.use_domino_ami` flag, so a node class pinning a stock AL2023 image attached its configured 100Gi or 500Gi volume to `/dev/sda1` as an orphan while the node booted on the image's default root volume of roughly 20 GiB. No action is required.

* All SMB volume mounts failed in 6.2.3 and now work. NetApp Trident components are updated to 26.06.0, with the Trident, Trident operator, and `tridentctl` images moved to Chainguard. No action is required.

- Model endpoints work in offline and air-gapped deployments. The `harness-proxy` image can be overridden and resolves from the Data Plane rather than being pulled from Quay. No action is required.

* Provisioning Project partitions is partly automated for admins, and HPC job submission checks partition configuration before accepting a job.

- Taxonomy service reliability: the `taxonomy-postgres-migrations` job no longer fails on hardcoded credentials, `GET /api/taxonomy/v1/namespaces` no longer returns 500 during Cloud Release windows, and `idsm-audit-trail-mongodb` memory limits were reviewed. No action is required.

* New Grafana dashboards cover the API Gateway, and the Restart Nucleus admin page is easier to use. No action is required.

- Platform maintenance: Kubernetes 1.35 is validated for GKE and 1.34 for OpenShift 4.21, platform apps move to Python 3.14, and Fluent Bit images move to Chainguard. This release also includes a security hardening pass across platform components and Environment images, removes committed secrets from model development lifecycle repositories, and stops the deployer from printing sensitive values in its configuration output. No action is required.

* The workload operator can be configured not to mount `/tmp` from the execution volume. This is opt-in and the default behavior is unchanged.

#### Breaking changes & deprecations

* The `CreateApp` and `CreateEndpoint` gate actions are deprecated in favor of `Deploy`. Existing gates that reference them keep working and can still be read, but gate creation and updates that use them are rejected. Update gate YAML to `Deploy` before your next policy edit. The deprecated `failOpen` key no longer appears in gate YAML.

- The **Jira integration** item is removed from the admin panel.
- **The `project.tag.approved=` search operator is removed.** It filtered on the approved flag of the previous free-form Project tags, which the shared tag structure replaces; queries using it no longer resolve. Filter a result set by tag from the filter side panel beside your search results or by typing the tag into the search bar instead. The `project.tag=` operator is unchanged. See [Use Domino search](/cloud/platform-capabilities/features/collaboration/domino-search#filter-results-by-tag).
- **Project Tasks, Goals, and the Project Stages and Status UI are removed.** Domino no longer shows the Tasks and Goals page, the Project stage box and stage filters, the combined Stage and Status component in the Project sidebar, the Stage and Tasks columns in the Projects Portfolio, or the Admin console's Project stages entry. The read-only Project Status badge next to the Project name remains, and the [Activity feed](/cloud/platform-capabilities/core-concepts/projects/collaborate-on-projects#view-project-activity) still records Job, Workspace, and file-comment events. Use your organization's issue tracker for task-level milestones. The backend Goal APIs remain available through the API and CLI.
- **The Project-level Jira integration UI is removed.** Domino no longer shows the UI to link a Domino Project to a Jira ticket or to sync Domino tasks with Jira subtasks, and existing per-Project Jira links are no longer surfaced in the Project. The backend Jira APIs and the deployment-wide Jira OAuth registration at `/admin/jiraConfig` are unchanged.

### Cloud Release 89

#### New features

##### Knowledge management: tags and properties

Tags and Properties replaces Domino's earlier tagging feature with a controlled knowledge management system. Tags categorize and discover assets with values from a controlled, hierarchical list, organized as key-value pairs: Namespaces are the allowed keys, and Tags are the allowed values within each Namespace. Properties enrich an entity with custom information through typed fields: unlike Tags, a Property's values aren't drawn from a controlled list; users supply any value that matches the property's type, with validation to keep values consistent. The new Select field type is the exception, letting a property offer a fixed set of options where that fits better.

Attach Properties to Apps, Projects, Project Templates, Datasets, NetApp Volumes, and Models to capture values like data classification, review status, or internal IDs directly on the entity. On Apps and Models, values can be set globally and independently on each version, so stable information and per-version detail coexist. Project Templates carry their values into new Projects, or require the creator to supply them. Editing is scoped to the app author and project manager, so property values stay controlled, and project properties are indexed for search. See [Properties](/cloud/platform-capabilities/core-concepts/knowledge-management/properties).

Properties replace the earlier Fields concept; anywhere you saw **Fields** before, you now see **Properties**. Existing values carry over automatically, with no migration step to run.

##### Browser-based CLI login

Logging in with the Domino [CLI](/cloud/reference/cli) now uses an OAuth flow. When you log in, the CLI opens your default browser to sign in and completes authorization automatically through a local callback on the same machine. Previously you had to copy an authorization code from the browser and paste it back into the CLI; that manual step is gone. If a browser doesn't open automatically, the terminal prints a URL to open on the machine running the CLI.

#### Enhancements

* The bundle view now shows an indicator when a policy on the bundle is enforced, so reviewers can see at a glance which policies are enforced.

- Governance admins get a new admin view for creating policy enforcers, making it easier to see and manage enforcement without digging through policy YAML.

* Sequential governance workflows support bulk updates, and the Deliverables table got usability improvements to match.

- Extension installs now show a **Cancel** button, so you can stop an in-progress installation instead of waiting for it to finish.

* The [Audit Trail](/cloud/reference/glossary#audit-trail) OpenAPI schema now includes the e-signature fields that were missing, so programmatic consumers of the Audit Trail get the full record.

- Snapshot events for Domino Datasets and NetApp Volumes now carry more detail, including on "Create Snapshot" and "Mark Snapshot for Deletion" events, for better traceability.

* Runs now get an environment variable pointing to the NetApp Volume location, so scripts can reference it without hardcoding paths.

- Imported dataset mount paths can now include the project name, so identically named datasets from different projects stay distinguishable. The old paths remain available as symlinks for backward compatibility, and the behavior is enabled with a central configuration flag. This continues last release's fix to imported dataset naming.

* Notifications are less noisy: several fixes address notifications that were unclear, overly persistent, or queued incorrectly.

- Fixed bugs affecting page navigation and display: the Audit Trail page could appear completely empty instead of showing logs, refreshing the tags page in tree view sent the browser to a 404 page, and tag display was broken in table columns that have been widened.

* Fixed a failure in experiment execution when running on a remote data plane, and an issue where using a Git tag (instead of a branch) in a Git-based project could break the Jobs output.

- Fixed environment-related issues: `domino_data.data_sources` was not found in the R reticulate environment (restoring Data Source access from R through the Python SDK), a package mismatch in the Ray GPU cluster environment, and a bug where updating an app's environment could produce a version that couldn't actually be deployed.

* Assorted governance UI polish: corrected wording on the bundle-creation toast, fixed formatting in the policy description modal, and fixed several deploy-modal bugs.

#### Notes for platform administrators

* Fixed an issue that blocked upgrading `nginx-ingress-fips` to 1.13.9-r29: the upgraded nginx drops legacy TLS ciphers that AWS Classic Load Balancers depend on, which could take deployments behind those load balancers offline. The fix enables DHE, a cipher both sides support, so deployments behind Classic Load Balancers upgrade safely. No action is required.

- Air-gapped SpiceDB deployments now upgrade correctly. This is a fix to the upgrade path itself, not a configuration change; no action is required.

* Domino Standard Environment (DSE) images are upgraded to Ubuntu 24.04.&#x20;

- Fixed a remote data plane failure where the workload-operator chart could not install because of a missing `nucleus-execution-jwt` secret. No action is required.

* This release includes a broad security hardening pass across platform components and environment images, remediating known vulnerabilities in dependencies such as log4j, netty, aiohttp, and nginx ingress. No action is required.

- Keycloak 26.6.2 tightens token introspection to require the calling client in the token's audience. Domino now includes the client ID in the audience for service accounts and Personal Access Tokens, so both keep authenticating after the Keycloak upgrade. This change is transparent and no action is required.

* The Audit Trail UI used a hardcoded one-year date range even though the back end retains five years by default; the UI now reflects the configured retention period. No action is required.

- Fixed a bug where workspace `/tmp` permissions were set to 777 even when `nonRootExecutions` was disabled. No action is required, but if you rely on restrictive `/tmp` permissions for compliance, confirm the fix took effect after upgrading.

#### Breaking changes & deprecations

* CLI versions older than 6.2.3 use a retired authentication client and can no longer log in. If `domino login` fails after this release, download and install the latest [Domino CLI](/cloud/reference/cli).

<hr />

## 2026 June

### Cloud Release 88

#### New features

##### Relative revalidation schedules for Governance approvals

[Governance](/cloud/reference/glossary#governance-center) policies can require an approval to be validated again on a recurring schedule. That schedule can now be relative to the approval, not only a fixed calendar. Previously a revalidation schedule was absolute: it recurred from a fixed start date regardless of when the approval was granted. With a relative schedule, the recurrence is anchored to the first approval, so the next revalidation falls due a set interval after the approval is granted, for example one year after approval.

##### Secondary Git repositories in Flow tasks

A [Flow](/cloud/reference/glossary#flow) Task can now reference a secondary or imported Git repository in its job configuration, not just the project's main repository. Teams composing multi-repository workflows can pull code from more than one repository in a single Flow Task.

#### Enhancements

* The Flyte Domino agent now supports runtime overrides, so you can customize a task's execution parameters when you launch it without editing the underlying task definition.

- The global navigation now includes [Governance](/cloud/reference/glossary#governance-center), making policies, evidence, and bundles easier to find from anywhere in the product.

* The Job detail page now shows the default branch name for jobs started from a [Flow](/cloud/reference/glossary#flow). Previously the branch was not shown for Flow-started jobs, which made runs harder to trace back to their source.

- Environment variables now have an optional description field, so teams can record what each variable is for alongside its name and value.

#### Breaking changes & deprecations

None in this release.

### Cloud Release 86

Shipped June 10, 2026.

#### New features

##### Compute usage visibility to manage compute costs

A new Hardware Efficiency dashboard in Grafana shows requested versus actually consumed CPU, memory, and GPU across your workloads, so you can spot over-provisioned [Hardware Tiers](/cloud/reference/glossary#hardware-tier) and right-size them. GPU monitoring is expanded alongside it: new utilization recording rules feed charts for actual usage, usage by workload type, and usage by project over time. Together they answer the questions that drive compute spend: which tiers are oversize, whether your GPUs are actually busy, and which projects and workload types consume them.

The dashboards are available in Grafana by default; no setup is required.

#### Enhancements

* New columns in the [Jobs](/cloud/reference/glossary#job) dashboard: **Finished** (completion timestamp) and **Total Duration**. You can spot slow or stuck jobs from the list without opening each one.

- R jobs can now accept any command-line parameter that the R executable accepts.

* More control over email notifications when you share an [App](/cloud/reference/glossary#app). Admins can set whether sharing an App sends a notification email by default.&#x20;

- When you use an [Extension](/cloud/reference/glossary#extension), the sidebar now highlights which Extension you are in.

* Fixed a defect that allowed Apps to be created with special characters in their names that later caused routing and deployment failures. Invalid names are now rejected at creation with a clear error.

#### Notes for platform administrators

* A new admin policy can block project invitations to email addresses outside your Domino organization. When the policy is enabled, project access and sharing rejects email invites to non-Domino addresses and shows the inviter a message explaining the policy. Adding existing Domino users through the user picker is unchanged, and the default behavior without the policy is unchanged. Action required only if you want to enforce the boundary: enable the policy.&#x20;

- Access to Grafana dashboards is now enforced through Keycloak, so observability access follows the same roles as the rest of the platform. No action is required.

* `enableServiceLinks` is now disabled for components in the `domino-compute` namespace, so Kubernetes no longer injects service-discovery environment variables into workload pods. This change is transparent and does not affect workload behavior; no action is required.

- Domino Model Monitor (DMM) ingestion job health checks were added to the admin toolkit, and DMM-related checks were consolidated. No action is required.

#### Breaking changes & deprecations

None in this release.

<hr />

## 2026 May

### Authentication, identity & security

* **User information API**: New backend API to retrieve user information based on user ID.

* **Iframe permissions for microphone and clipboard**: Customer-driven addition of "microphone" and "clipboard" to the iframe `allow` src tags.

* **Grant ReadOnlySupportStaff ViewEnvironments permissions**: Fixes ReadOnlySupportStaff global role to have permissions to view environments.

* **Personal Access Tokens**: Domino now supports Personal Access Tokens (PATs), allowing users to generate their own API credentials with a custom name, description, and expiration time. Administrators can oversee and manage tokens across the platform, including the ability to revoke or delete them when needed.

### Platform & UI

* **Workspace Breadcrumb Project Link**: Workspace breadcrumbs now link back to their parent project.

* **Env Vars Table: Sort & Copy**: The environment variables table now supports sorting, and each value has a copy button.

* **Custom Job Email Template Cleanup**: Removed test content from the custom job email template.

* **NetApp Volume Mount Points**: Full support for NetApp Volume mount points in the Domino UI, including a new API component, details panel, file-open integration, and UI adjustments to accommodate the new mount point type.

* **Model Details: Full Markdown Overview**: The Model Details page now renders the full model overview markdown instead of a truncated preview.

* **Scheduled Jobs Notifications: Security controls**: Logs no longer included by default in emails; ability to restrict custom recipients via CC flag.

### Tag Browser

* **Tagging** now extends to NetApp Volumes. Tag, search, and browse NetApp Volumes through your custom tag structure. We’ll be integrating tagging with more assets in updates coming soon.

* **Tagging** extends to Domino Datasets. Tag, search, and browse Datasets through your custom tag structure.

### App Governance & SDLC

* **AI Gateway UI** is now hidden by default behind a feature flag.

### Curated configuration records management

* **New API Endpoints**: Added backend endpoints for Curated Configuration Records, enabling Cloud Admins to manage curated configuration sets for their tenants.

* **Feature Flag**: Added feature flag to show/hide the Curated Configuration Records section, allowing controlled rollout per tenant.

* **Project Behavior Page: UI Sections**: Built the Long-running Workspaces, Restrictions & Limitations, and Flexibility & Customization sections of the Project Behavior configuration page.

### Governance

* **Governance Enhancements**: Governance demands more than a one-time sign-off: approvals can now be scheduled to renew on a recurring basis, so your compliance posture stays current as models and projects evolve. Approvers are notified when renewal is due, and bundles automatically fall out of compliance if a renewal lapses.

<hr />

## 2026 April

### Knowledge management

#### Tags

[Tags](/cloud/reference/glossary#tag) now extend to Domino Projects and Project Templates. Define a tag structure that reflects how you categorize and organize knowledge in your organization, so you can find and reuse past work instead of reinventing the wheel. Tag, search, and browse Projects, Project Templates, and Apps through your custom tag structure. We’ll be integrating tagging with more assets (Datasets, NetApp Volumes, etc.) in updates coming soon.

### Platform & infrastructure

* GPU metrics panel now renders based on available GPU data rather than presence of DCGM label.

### Compute Environments & DSE

* **Admin access to cluster Web UIs for troubleshooting**: Admins can now view the Spark, Ray, and Dask cluster Web UIs for other users' workspace sessions to inspect executor logs, configs, and worker state, without being granted access to the user’s notebook/IDE. When an admin opens another user’s workspace session, the session page now opens directly on the Cluster UI tab, giving admins a working diagnostic view even though the IDE itself remains restricted. Enabled by default: no central configuration or feature-flag change required.

* **Environment select pattern**: A consistent environment select UI pattern introduced across Domino for choosing a compute environment when launching tasks.

### UI & UX improvements

* **Expanded span type support in tracing decorator**: The manual span tracing decorator now accepts the `span_type` argument, enabling full MLflow tracing argument support. Users can now manually instrument unsupported scenarios without accessing raw MLflow features directly, and traces display the correct icons in the trace viewer.

* **Improved terminology in Experiment Manager**: When agent traces are detected, the UI now displays "Agent Versions" in place of "Runs," allowing users to distinguish agent version tracking from traditional ML run tracking within the Experiment Manager.

### Extensions

This release contains updates to the Extensions framework with new mount points, Extension management UI capabilities, and general availability.

<AccordionGroup>
  <Accordion title="New mount points">
    * **Edit Extension modal**: Users can now edit an existing Extension from the Extension management page.

    * **Dataset detail page mount point**: Extensions can now embed in dataset detail pages.

    * **Model detail page mount point**: Backend support added for the model card mount point.

    * **Dataset file mount point**: Extensions can now embed in dataset file views.

    * **Model detail page mount point**: Frontend support for model detail page mount point.

    * **Admin mount point**: Frontend support for admin mount point.

    * **Improved DatasetFileContext support**: Enhanced context support for extensions embedded in dataset file views.
  </Accordion>

  <Accordion title="Additional mount points">
    * **Additional mount points at Create/Edit extension modal**: Create and Edit Extension modals now support selection of the new mount points.
  </Accordion>

  <Accordion title="Admin extensions UI">
    * **Admin extensions list page**: A new admin extensions list page with routing, table view, enable/disable controls, and authorization is now available.

    * **Extensions management menu**: A managed extension menu added inside the extension accordion for in-context control.

    * **Create Extension modal**: Users can now create an Extension directly from the Extensions management page or from an App detail page.
  </Accordion>

  <Accordion title="Backend & APIs">
    * **Core APIs**: APIs for creating, patching, and retrieving extensions are live with feature flag gating.

    * **Extensions auditing**: Audit logging added to all Extensions APIs.

    * **Admin panel mount point**: Backend support added for extensions to embed in the admin panel.

    * **Dataset mount point**: Backend support added for extensions to embed in dataset detail pages.

    * **Extensions FF enabled by default**: The Extensions feature flag is now on by default for all users.

    * **MongoDB indexes**: Performance indexes added for extensions-related MongoDB collections.
  </Accordion>

  <Accordion title="Frontend">
    * **Extension renderer**: Renderer passes and parses query params; deep-linking support added so extensions can redirect to iframeless app views when the underlying app supports deep linking. TopBar removed as a mount point; overflow pattern added for the Project sidebar.
  </Accordion>
</AccordionGroup>

### App versioning & Apps

App versioning is significantly expanded, enabling users to manage, start, stop, and edit non-latest versions.

* **PPM language update**: Updated PPM-related language in UI and documentation.

* **API access section on Dataset details page**: The API access UI on the Dataset details page has been relocated to accommodate the new Extensions card mount point.

* **Version-specific start/stop**: Users can start and stop historical as well as current App versions.

* **Edit previous App versions**: Pen icon controls added to App detail cards for previous versions.

* **App list view cleanup**: Edit option removed from the App list view row menu to avoid version ambiguity; editing is now only available from the App details page.

* **Edit App wizard update**: Deployment step updated to clarify that editing hardware tier no longer triggers a new App version.

* **App list performance**: Resolved N+1 query issues in the listApps endpoint, significantly reducing MongoDB queries per request.

* **Reproducibility for imported repos**: App versions now snapshot imported project and Git repo state to ensure reproducibility when restarting older versions.

* **App search**: Apps now appear as a dedicated tab in global search and in the global search dropdown.

### AI Governance & model lifecycle

Policy lifecycle and governance workflow improvements.

* **Policy version diff**: Governance Admins can now visualize the diff between policy versions, both in YAML and through the UI.

* **Bundle reset**: Reviewers in regulated environments can now reset a bundle at any stage moving it back to the first stage, with flexible control over which attachments are retained, while all approvals and evidence answers are cleared for resubmission. A complete reset audit trail is automatically captured in the bundle’s history view.

### AutoML

* **AutoML compute routing**: Training jobs from the AutoML Extension now run in the project where the dataset lives via the Domino Jobs API, ensuring correct experiment tracking, per-user compute isolation, and proper permission scoping.

<hr />

## 2026 March

This release expands GenAI cloud coverage, introduces AI governance policy versioning, and delivers broad improvements to agent monitoring, platform infrastructure, workspace management, and coding assistants.

### Workspace coding assistants

Domino Workspaces now include built-in support for AI coding assistants as part of the Domino Standard Environment, enabling model developers to accelerate data analysis, model training, and validation workflows directly from their workspace.

* Includes built-in support for Copilot, Claude Code, and Codex; available as pre-installed VS Code extensions and CLI tools.

* Agents are Domino-aware, with automatic access to datasets, jobs, and registered models.

* A new skills framework adds Domino-specific actions (model training, job execution, deployment, and more).

* Coding agent extensions and settings and chat history persist across workspace restarts.

### Extensions framework

Introduced admin-only APIs for creating and managing extensions. New UI for exposing extensions and creating them from an app.

### AI Governance & policy versioning

Policy versioning is now available in the Policy Builder, with new compliance visibility and bundle lifecycle tools to support governance at scale. This release continued to build out the AI Governance framework introduced in the previous release.

#### Audit Trail

* Telemetry and audit trail for policy versioning; audit events for policy versioning operations added.

#### Policy versioning

* "Reason for Change" field (renamed from "Version Description") is now required to publish.

* Create versions of existing policies directly from the Policy Builder.

* Upgrade prompts notify data science practitioners when newer policy versions are available.

#### Bundle lifecycle management

* Reset bundles and bulk-delete attachments from the bundle management interface

* Email notifications now track stage transitions

#### Compliance visibility

* Compliance status displayed directly in the policies and bundles tables

* Policy usage counts now reflect bundles across all versions

### Authentication & security

A more secure and maintainable auth layer emerges from this release, with key infrastructure changes to Vault, Keycloak, and authorization foundations.

#### Auth persistence

* Execution now knows if home persistence is enabled.

#### Infrastructure

* Vault agent injector removed from the cluster

* Keycloak service account provisioned

* AuthZ and unified auth foundations established

### Platform & infrastructure

Network and infrastructure upgrades in this release reduce log processing overhead and expand GPU workload visibility across remote data planes. Additional improvements were also made to Grafana, and Environments workflows.

#### Grafana

* Single source of truth for dashboards and alerts via Grafana Operator CRs.

#### Environments

* DSE split into AI-enabled and standard variants; Copilot VS Code extension added; VS Code marketplace switched.

#### GPU monitoring

* `kube_pod_labels` metrics now available from remote data planes for GPU workload monitoring

#### Logging

* Fluent-bit event filtering reduces `fluentd` load for more efficient log processing

### GenAI & model serving

This release broadens cloud coverage for GenAI model serving and resolves key deployment and data plane issues across all supported environments.

#### Support for multicloud deployments

* GenAI model registration and deployment now supported on Azure (AKS) and GCP

* NVIDIA environment variables and CUDA toolkit included in vLLM environment for GCP

#### Data Plane improvements

* Remote data plane support added for model APIs

* Fixed Istio resource configuration and control/data plane compatibility

#### Agent deployment

* Fixed issue preventing agent deployment from the experiment run overview page

#### Experiment runs table

* 3-dot menu column is now pinned on the Experiment Runs table

### Agent monitoring & tracing

Teams get clearer visibility into agent performance with corrected latency metrics, smarter chart bucketing, and improved tracing output in this release.

#### Chart bucketing

* Monitoring charts automatically re-bucket to 1-hour intervals on 24-hour windows

* Time axis now scales correctly across window sizes

#### Latency and metrics

* Fixed latency chart rendering issues and corrected unit display

* Fixed LLM chart failures when runs are stopped mid-session

* Fixed LLM metrics bucket edges incorrectly based on start time

#### Tracing UI

* Tool call span outputs now wrap correctly within the UI container

### Data, compute & workloads

Distributed workloads are easier to run, storage options are broader, and workspace controls are more flexible.

#### MPI

* MPI cluster support enabled by default and no longer requires manual feature flag enablement

#### Blob storage backend

* Azure Blob Storage now supported as a storage backend for Workspace File Audit Trail

#### Workspaces

* New Long-running Workspaces section added to Project behavior settings

* Administrators can configure automatic shutdown duration and notification preferences

### Deprecations

* Governance "Complete" stage removed in favor of the new compliance-based workflow

<hr />

## 2026 January

### Agentic system experimentation

Register and deploy LLMs, monitor AI systems with full trace visibility, and iterate on production configurations.

### LLM registration & deployment

Register and deploy large language models (LLMs) as optimized, hosted endpoints:

* Pull models directly from Hugging Face or your experiment runs

* Deploy with OpenAI-compatible APIs

* Make LLMs available to applications and agentic systems

### Production monitoring & tracing

Monitor agentic systems and LLM applications with full trace visibility:

* Capture every LLM call, agent interaction, and prompt chain as traces

* Works with MLflow auto-tracing integrations (LangChain, OpenAI, Pydantic AI) or standalone functions

* Attach evaluations to traces to measure quality, accuracy, or safety

* Compare performance across experiments to understand what’s working

### Production deployment & iteration

Deploy and iterate on agentic systems with infrastructure optimized for agent workloads:

* Use the same proven infrastructure as Domino Apps

* Instrumentation carries over automatically from experimentation

* Relaunch production configurations into a workspace for debugging

* Maintain clear lineage between production agents and source experiments

### Workspace audit capabilities

Extend Domino’s audit trail into interactive workspaces for complete visibility and regulatory compliance:

* Log all file operations (read, write, create, rename, delete) on NetApp Volumes and Datasets

* Investigate activity quickly with a dedicated UI for searching, filtering, and exporting events

* Customize the audit app or integrate programmatically with governance tools

* Retain events long-term in customer-managed blob storage with automatic deduplication

* Meet GxP and SOC 2 requirements with comprehensive, auditable records of in-workspace data access

### Reduce compute costs with spot instances

Cut compute costs by using discounted, interruptible AWS cloud capacity across Jobs, Workspaces, Apps, Endpoints, and Launchers:

* Select hardware tiers marked Spot tag to use spot pricing automatically

* Ideal for batch jobs, distributed training, and parallel analytics that tolerate interruptions

* Falls back to on-demand infrastructure if spot capacity is unavailable

* Interrupted workloads retry automatically on available capacity

<hr />

## 2025 December

### Governance update: Bundle history tab

Track every change to your bundles with the new Bundle History tab on the bundle overview page. See what changed, when, and by whom.

### Governance monitoring checks

Governance now responds automatically to Domino Model Monitoring alerts. When drift, data quality, or detection issues trigger, Governance creates findings, assigns severity levels, notifies owners, and logs everything for compliance - no manual intervention required.

### Certificate-based authentication

Certificates are now a supported authentication method for Oracle, DB2, and Postgres Data Sources.

### Flow integration with NetApp Volumes

* **Export Flow Outputs via UI**: Flow Artifacts of type `DATA` can now be exported to NetApp Volume snapshots via the UI, making them consumable for downstream tools like Apps.

* **Programmatic Output Registration**: You can also use the [Flyte Python SDK](/cloud/platform-capabilities/features/flows/define-flow-artifacts#programmatically-exporting-to-domino-datasets-and-netapp-volumes) to export Flow artifacts to a NetApp Volume and snapshot them automatically as part of your Flow.

### NetApp permissioning for Apps via Data Sources

SDK Access with Permissions: Domino Apps can now access NetApp Volumes through Python or R Datasource SDKs with automatic permission enforcement for secure, shared executions.

### Snowflake deprecation of single-factor authentication

To prepare for the Snowflake deprecation of [single-factor authentication](https://docs.snowflake.com/en/user-guide/security-mfa-rollout#deprecation-timeline), new Snowflake Datasources no longer accept Single-Factor Authentication methods. Existing Datasources show a warning notification to encourage updating the authentication method.

<hr />

## 2025 November

### Default branch selection for Git repos

Project owners can now configure a default Git branch in project settings. The selected branch is automatically populated in relevant workflows across the project and can be updated anytime to match evolving needs.

### Custom autoscaling settings for Apps

[Configure your apps](/cloud/platform-capabilities/features/apps/app-logs-and-versioning/autoscaling-apps) to scale exactly how you need them. Set maximum pods, CPU/memory thresholds, and scale timing when you launch. Then forget about it. Your settings persist across all sessions and are always visible when you hover over the Launch App window.

### NetApp Volumes enhancements

* **Project Template Support**: NetApp Volumes can now be included when creating or using project templates, making it easier to standardize data access across projects.

* **Snapshot Tags for Mounting**: Tag [NetApp Volume snapshots](/cloud/platform-capabilities/core-concepts/data/datasets/version-data-with-snapshots) for programmatic mounting. Add optional commit messages at creation time to improve traceability and workflow automation

### Flow integration with NetApp Volumes

* **Mounting Snapshots in Flows**: Flows can now [mount NetApp Volume snapshots](/cloud/platform-capabilities/features/flows/define-flows#declare-tasks) at runtime, providing read-optimized data across execution contexts.

### Streamlined Governance policy editing

We’ve improved the way that Governance admins can edit policies using the [Policy Builder](/cloud/platform-capabilities/features/governance/define-governance-policies). Great for quick changes and versioning policies over time.

<hr />

## 2025 October

### Apps autoscaling

Domino Apps now support [horizontal autoscaling](/cloud/platform-capabilities/features/apps/app-logs-and-versioning/autoscaling-apps), making it easier to keep apps responsive under varying demand while optimizing resource costs.

* **Automatic pod scaling**: Adjusts app pods based on CPU and memory usage.

* **Default configurations**: Ships with built-in settings.

### Enhanced App discovery experience

Finding and browsing apps is now faster and more visual:

* New card-based layout with filtering, sorting, and support for app thumbnails.

* Pin your favorite apps for quick access.

* Lite Users get a dedicated home screen and improved project search.

### Domino Governance

#### Real-time autosave for evidence notebooks

Governance Evidence Notebooks now autosave in real time. This removes the need to manually click **Save** after editing, and makes sure that visibility rules and compliance checks are triggered immediately as edits are made.

#### Attach snapshots and flow artifacts

You can now attach NetApp volume snapshots and Flow artifacts directly to governance bundles, expanding traceability and auditability across data and workflow assets.

### NetApp Volumes previews

In NetApp Volumes and remote datasets, the UI now supports live previews for additional file types, including:

* **Tabular data** - `.csv`, `.xls`, `.xlsx`, `.parquet`

* **SAS formats** - `.sas7bdat`

* **R formats** - `.rds`

* **Text and metadata** - `.rtf`, `Define.XML`, `.xpt`

### Audit Trail enhanced filtering

You can now create more precise audit queries using operators like `contains`, `is`, and `is not` across all audit attributes - including users, events, projects, and datasets.

### Authenticate with a Personal Access Token

A new **Personal Access Token** panel has been added to the account settings page where you can fetch a short-lived token to use in any requests against the [Domino API](/cloud/reference/api).

### Cost Center dashboard updates

The Cost Center dashboard now shows memory costs of executions alongside the existing costs (CPU, GPU & Storage).

### Keycloak updates

* Upgraded Keycloak to 26.3.5.

<hr />

## 2025 September

### Track, evaluate, and compare generative AI experiments

Domino now makes it easier to [evaluate and improve generative AI systems](/cloud/platform-capabilities/features/agents) with built-in tools for instrumentation, structured evaluation, and comparison.

* **Logging SDK to allow for Universal Tracing**: Single-line instrumentation of GenAI across frameworks, such as LangChain, LlamaIndex, Pydantic, and OpenAI Agents SDK, makes it easy to track interactions in your GenAI System.

* **Trace Explorer**: Visualize GenAI interactions with execution paths, inputs and outputs, and latency to make debugging faster and clearer.

* **Evaluations**: Add metrics or labels to traces from the SDK or UI, enabling programmatic evaluations with your preferred framework or quick hand scoring by SMEs.

* **Trace Diffs**: Compare traces and evaluation metrics across runs to understand how different configurations perform, with a granular view into sample-level differences in your evaluation set.

### RabbitMQ updates

Upgraded RabbitMQ to 4.1.2.

### Enhanced model endpoint monitoring

Model endpoint monitoring is now more capable and accessible:

* **In Domino**: Data scientists can view endpoint performance metrics directly from endpoint version pages, without leaving the interface.

* **In Grafana**: Administrators get [advanced monitoring dashboards](/cloud/admin/operations/grafana-workload-monitoring/model-endpoint-monitoring) with deeper visibility into system metrics and performance trends.

These improvements make it easier to [monitor model health and performance](/cloud/platform-capabilities/features/model-deployment/monitor-model-endpoints), troubleshoot issues, and optimize resource usage.

### App sharing and access management updates

We’ve streamlined how you share and manage access to Apps in Domino:

* A new [Access & sharing](/cloud/platform-capabilities/features/apps/share-an-app) view adapts based on viewer settings, with additional user management options for project-specific apps and clear notices for broader access levels.

* A new **Access & sharing** button is now available on the app overview page.

* After publication, app access is managed through the **Share App** dialog.

* Users now receive an email and in-platform notifications when they are granted app access, including the app name and a direct link to the app details page.

<hr />

## 2025 August

### Governance: Sequential workflows enforce stage order

Sequential workflows now support required evidence fields to improve governance and compliance. Each stage must be completed in order, and bundles cannot move forward until key questions are answered. This ensures proper review, traceability, and audit readiness throughout the workflow.

### TLS 1.3 update

The Transport Layer Security (TLS) configuration for all Domino Cloud environments was upgraded. Our TLS cipher policy to support TLS 1.3 in addition to TLS 1.2, while disabling weaker legacy ciphers from TLS 1.2 by switching to AWS’s default policy configuration: `ELBSecurityPolicy-TLS13-1-2-Res-2021-06`.

### Governance gates for Apps and model endpoints

Policies can now define [gates](/cloud/platform-capabilities/features/governance/gating-domino-governance) that block access to resources like hardware tiers until the governed bundle has the required approvals. Governance admins configure gates in the policy YAML, and enforcement prevents publishing apps or creating model endpoints until approvals are complete.

### UX driven Governance policy configuration

Quickly create, edit, and test governance policies in a streamlined interface. Designed for fast, accurate updates to meet evolving regulatory needs.

### Self-service node pool management

Easily manage node pools in your Domino cluster. This includes viewing existing node pools, adding new ones, and deleting existing pools directly from the Cloud Admin portal.

<hr />

## 2025 July

### Domino Apps enhancements

[Apps](/cloud/platform-capabilities/features/apps) have been reimagined in Domino to include the following improvements:

* A new app publishing UI.

* The ability to create multiple apps in a single Domino Project.

* The new global apps list page and viewing running applications that use iFrames vs those without iFrames (which can use deep links with query parameters).

* The new app details page and switching between viewing different versions.

* (Soft-)Deleting an app.

* Managing user access to apps.

* App users can now deploy an app using any file type. They are no longer limited to `.sh` files only.

### Domino Governance enhancements

* **Multiple Policies per Bundle**: Multiple policies can now be applied to a single bundle, reducing duplication and simplifying reviews. Learn more about [creating](/cloud/platform-capabilities/features/governance/work-with-bundles) or [reviewing bundles](/cloud/platform-capabilities/features/governance/work-with-bundles/review-bundles).

* **Policy Archiving**: Policies can now be archived from the Policies screen. Archived policies are only visible with the Archived filter and can’t be restored or removed if still in use by active bundles. Learn more about [defining policies](/cloud/platform-capabilities/features/governance/define-governance-policies).

* **Improved PDF Bundle Report**: Improved experience in the PDF bundle report, including displaying attachments in an appendix section.

* **Per-Bundle Approval Customization**: [Different approvers](/cloud/platform-capabilities/features/governance/work-with-bundles/review-bundles) can now be assigned to individual bundles under the same policy, enabling more flexible and context-specific reviews.

* Users can now assign individuals to stages, manage multiple bundles at once, and rename terms like “Bundle” or “Deliverable” to match industry standards.

### Workload observability

Cloud admins now have comprehensive visibility into their workload health and performance. Learn more about [monitoring and observability](/cloud/admin/operations/grafana-workload-monitoring).

### Project environment variable redaction in Audit Trail

Project-level environment variables can now be redacted in the audit trail. This functionality is not enabled by default, so these environment variables are logged as plain text. Please reach out to Domino Support to update this setting for your deployment.

### Package persistence for Workspaces

Users are now able to persist packages and home directory files across workspace sessions. Package persistence saves time by retaining installed packages across workspace sessions in Python or R. When properly configured, you don’t need to reinstall your packages after starting or restarting a workspace. Learn more about [package persistence](/cloud/platform-capabilities/core-concepts/workspaces/use-package-persistence).

### Domino Volumes for NetApp ONTAP

[NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes) allow for users to instantly snapshot training data for auditability and reproducibility with no extra storage required.

### Configurable datetime precision for Data Sources

Users can now configure the resolution of datetime columns for relational Data Sources. See the [Data Source overridable configurations page](/cloud/reference/api/data_api/datasource_usecases/config_override) for more information.

<hr />

## 2025 June

### Attach Dataset snapshot files to Governance bundles

Users can now attach files from a dataset snapshot to a governance bundle within the same project. Snapshot files can be linked to an existing bundle or used to create a new one.

### Keycloak updates

* Upgraded Keycloak to 26.1.4.

<hr />

## 2025 May

### Domino Governance

* **Scripted checks**: Use centralized scripts to evaluate information on a Governed Bundle. Scripted checks can also be used to measure bias consistently in a dataset.

* **Findings**: An enhancement to Findings is now available to allow for conditional approval, allowing approvers to select findings that must be completed for the bundle to be fully approved, without obstructing productivity. This also includes the ability to set due dates for findings and receive notifications based on the lifecycle of the approval.

### Streamlined flow output registration to Domino Datasets

Flow Artifacts with type DATA can now be exported to Domino Dataset snapshots via the UI for consumption of Flow Outputs in downstream workflows such as Apps. Traceability between the Flow Artifact and Dataset snapshot is maintained and viewable.

<hr />

## 2025 April

### Cost Center dashboard

The Cost Center Dashboard lets administrators easily view infrastructure costs across their Domino deployment. It provides quick insights into resource usage and cost trends over time. Learn more about [the Cost Center](/cloud/admin/operations/finops).

### Automatic environment rebuilds

Environments can now automatically be rebuilt based on changes to the base environment image. Environments have a toggle in the Revisions tab, letting users choose whether to automatically update to the latest revision or stay on a specific revision.

### Self-service IP allowlisting

Cloud Admins are now able to configure IP allowlisting through the Domino Admin Panel. IP allowlisting is a security feature that lets you choose which IP addresses can access your Domino cluster. This improves security by allowing only trusted IPs and blocking unauthorized access. See our documentation to [configure IP allowlisting](/cloud/admin/architecture-and-deployment/deployment-configuration/ip-whitelisting).

<hr />

## 2025 March

### Custom App URLs

Domino apps provide persistent URLs that remain unchanged across redeployments and can be customized into human-readable formats. This ensures that links can be reliably shared, bookmarked, and understood by users without requiring IT involvement for redirects.

### Governance: Support for Git based Projects

We now support artifacts from Git-Based Project (GBP) bundles for Governance.

### Flexible Git reference support for Apps

Teams can now deploy apps from a specific Git reference (branch, tag, or commit SHA). The reference must contain an `app.sh` file to be successful.

### In-product guides & alerting

A new functionality is now available to enable in-product guides, alerting, and more. This functionality aims to improve adoption and enablement of the Domino Platform.

<hr />

## 2025 February

### Updates to Domino Governance

* **Governance Findings**: Create, track, and manage compliance gaps and potential risks within a governed bundle with Governance Findings.

* **Metrics Checks**: Automatically attach metrics to a governed bundle, eliminating the need for manual entry.

### Configurable idle timeout

Users are now logged out of Domino after a customizable period of inactivity. Users receive a pop-up letting them know they are about to be logged out, and they have the option of dismissing the pop-up to stay logged in. *This setting is turned off by default. Please reach out to Domino Support if you’d like to configure an inactivity period for your deployment.*

### Project templates support for "Other" Git providers

Git-based projects from "other" providers (including Azure DevOps) can now be used as project templates or copied into other projects. To copy a template repository, users must provide an uninitialized destination repository from the same provider.

### Cloud Admin global role improvements

Extended the permission set of the Cloud Admin global role to be able to create and manage Service Accounts.

<hr />

## 2025 January

### CSV download for Audit Trail

Domino’s new **Download as CSV** feature allows CloudAdmin and GovernanceAdmin users to export Audit Trail data directly from the interface. This functionality helps support both internal and external compliance requirements.

When the **Download CSV** button is clicked, the generated CSV files reflect the filters and table columns selected.

<hr />

## 2024 December

### Unified Audit Trail

Domino’s new [Unified Audit Trail](/cloud/platform-capabilities/features/governance/audit-trail) tracks user interactions and system events, allowing organizations to monitor data creation, viewing, and permission changes for better transparency and accountability.

Audit trails capture broader compliance events, such as changes to the following:

* Datasets

* Project configurations

* User roles

* Data access

### Support for Trainium instances

Domino adds support for AWS Neuron/Trainium instances for Domino Cloud.

### Keycloak updates

Upgraded Keycloak to 25.0.6.

<hr />

## 2024 November

### Deploy models to SageMaker

Domino provides [model deployment to SageMaker](/cloud/platform-capabilities/features/models/deploy-sagemaker). Leverage Domino’s flexible development experience to build your models before deploying them to production on SageMaker, while still maintaining centralized governance and tracking of all of your models. Administrators apply guardrails around cost, performance, and security, and the platform picks the resources needed to meet them.

<hr />

## 2024 October

### New Domino user interface

Domino Data Lab introduces a new UI to improve your core workflows. It has an updated look and feel, with new navigation patterns and a dashboard experience to quickly get you into recent work. For more detail, see the [New UI migration guide](https://docs.dominodatalab.com/en/6.0/user_guide/5c9e60/new-ui-migration-guide/).

### NetApp integration

Domino adds support for NetApp FSxN for Domino Cloud.

### Domino Governance

[Domino Governance](/cloud/platform-capabilities/features/governance) simplifies, organizes, and automates the process of gathering, tracking, and reviewing all assets to help enforce internal or external policies. It provides a single interface closely connected and integrated within a data scientist’s environment, ensuring smooth governance, version control, and compliance for all stakeholders involved.

### Datasets in Nexus

[Domino Datasets](/cloud/platform-capabilities/core-concepts/data/datasets) can now be created in remote data planes.

### Project templates

Create [Project templates](/cloud/platform-capabilities/core-concepts/projects/create-projects#create-a-project-from-a-template) to kickstart future projects instead of starting from scratch. Project templates enable collaboration, efficiency, and standardization with your team, organization, and company.

### App secure identity propagation

[App secure identity propagation](/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security) allows apps to identify a Domino user making a request to provide differentiated services to users. Existing apps that host additional assets or secondary pages should adapt their root path as described in the [App security](/cloud/platform-capabilities/features/apps/authentication-and-identity/app-security) section when secure identity propagation is turned on. Running applications are not affected and do not need changes until they are restarted.

### Upgrade notes

* The new Data Plane version has advanced to a major release (now 4.1.1-3). As a result, the upgrade button does not function due to the version gap. To complete the upgrade successfully, a manual process is required. For details, refer to the [upgrade the Data Plane](/cloud/admin/platform-configuration/manage-data-planes#upgrade-a-data-plane) section.

<hr />

## 2024 September

### RabbitMQ updates

Upgraded RabbitMQ to 3.13.4.

<hr />

## 2024 July

### Removing support widget

Domino is removing the Support widget from the UI as part of improving and streamlining our support processes. There are no changes to any other support processes with Domino. You can continue to raise support tickets via the [online portal](https://support.domino.ai/support/s/) or by emailing [support@dominodatalab.com](mailto:support@dominodatalab.com).

### Run Job from Workspace

Users can run jobs directly from the workspace interface, enabling them to run jobs with the latest changes visible on their workspace screen. The job execution button becomes active once all changes have been committed to the repository. Additionally, a summary panel is provided to monitor the user’s most recent job executions.

### Changes to the Admin navbar

The old Admin navbar has been updated with a new visual appearance and improved menu organization.

* Key changes in the **components** include:

  | Previous implementation | Changes to new implementation |
  | - | - |
  | **Admin Area**. | Renamed to **Admin Panel**. |
  | Link assigned to **Admin Home** menu item. | Link assigned to app logo. |
  | **Search** was in the old implementation. | Removed from the new implementation. |
  | - | Page titles and Browser tab titles changed according to the name changes for the menu items. |

* Key changes in the **menu items** include:

  | Menu items | Previous implementation | Changes to new implementation |
  | - | - | - |
  | Projects, Workspaces, Executions | At the top-level navigation. | Part of the submenu of the top-level menu item **Reports**. |
  | Usage, Resource usage report, User activity report, Monitoring (renamed to Grafana Monitoring) | Part of the submenu of the top-level menu item **Advanced**. | Part of the submenu of the top-level menu item **Reports**. |
  | Datasets | Part of the submenu of the top-level menu item **Data**. | Part of the submenu of the top-level menu item **Reports**. |
  | Users, Organizations | Tabs of the top-level menu item **Users**. | Part of the submenu of the top-level menu item **Manage resources** |
  | Governance, Data Planes, Notifications, Manage Cost (renamed to Cost Tracking) | Were top-level menu items. | Part of the submenu of the top-level menu item **Manage Resources**. |
  | Manage Environments (renamed to Environments), Hardware tiers, Resource quotas, External deployments | Part of the submenu of the top-level menu item **Advanced**. | Part of the submenu of the top-level menu item **Manage Resources**. |
  | Data Sources, Feature Store | Part of the submenu of the top-level menu item **Data**. | Part of the submenu of the top-level menu item **Manage Resources**. |
  | Project Stage Configuration (renamed to Project stages), Jira Configuration (renamed to JIRA integration), Diagnostics toolkit | Part of the submenu of the top-level menu item **Advanced**. | Part of the submenu of the top-level menu item **Platform settings**. |
  | Username | Was a top-level menu item. | Part of the submenu of the top-level menu item **Account**. |
  | Account settings, Download CLI, Log out | Part of the top-level menu item **Username**. | Part of the top-level menu item **Account**. |
  | Feedback (opens a feedback modal on click), Exit Admin panel, App Version | Not in the old implementation. | New menu items added to the submenu of the top-level menu item **Account**. |

<hr />

## 2024 March

### Qdrant vector DB Data Source

The new [Qdrant vector database Data Source connector](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-qdrant) enables easy, secure access to vectorized content stored in Qdrant. Domino Data Sources provide enhanced security and governance capabilities while supporting cutting-edge generative AI use cases.

<hr />

## 2024 January

### AI Gateway

The [AI Gateway](/cloud/platform-capabilities/features/llms/llm-gateway) provides Domino users a safe and streamlined way to access Large Language Models (LLMs) hosted by external service providers like OpenAI or AWS Bedrock. This lets users enjoy the benefits of provider-hosted models while ensuring that they follow security best practices.

### Pinecone vector DB Data Source

The new [Pinecone vector database Data Source connector](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/connect-to-pinecone) enables easy, secure access to vectorized content stored in Pinecone. Domino Data Sources provide enhanced security and governance capabilities while supporting cutting-edge generative AI use cases.


# Adding a new Git Repository to your project using Personal Access Token (PAT)
Source: https://docs.domino.ai/knowledge-base/adding-a-new-git-repository-to-your-project-using-personal-access-token-pat



#### **Version/Environment (if relevant):**

Domino Versions 5.x and later

## **What:**

Connecting a new Git repository to projects using a Personal Access Token (PAT) generated from any Git service account.

## **How:**

To link a Git repository to your Domino project using a Personal Access Token (PAT), follow these steps:

## Creating credentials

If you are adding a private repository, want to write commits to remote, or are using SSH, you will need to add Git credentials to your Domino account. Domino will use these credentials to authenticate with the service hosting your repository when you start a [run](https://docs.dominodatalab.com/en/latest/user_guide/af97b7/create-and-run-jobs/).

### Personal Access Token creation

You will need a personal access token to access a private repository via HTTPS. You will need a personal access token if the URI you want to use to interact with a repository is formatted as:

`https://<domain>/<user>/<repository>.git`

Personal Access Tokens are supported by the following Git services.

* [GitHub Personal Access Tokens](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/)
* [GitLab Personal Access Tokens](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html)

The personal access token you generate needs to have read and write access to your private repositories.

To generate a personal access token in your **GitHub**

1. Navigate to your Git account settings, then Developer Settings. Click the Personal access tokens menu, then click Generate new token.
2. Select repo as the scope. The token will be applicable for all the specified actions in your repositories.
3. Click Generate Token.

To generate a personal access token in your **GitLab**

1. From your home page, open your user settings, and then select Personal access tokens.
2. And then select + New Token.
3. Name your token, select the organization where you want to use the token, and then choose a lifespan for your token.
4. Select the scopes for this token to authorize for your specific tasks.

After generating a personal access token in your Git service, use these steps to add it to Domino:

1. In Domino, click your username in the upper right, then click **Account Settings**.
2. Scroll down to the panel labeled *Git Credentials*, then click **Add a New Credential**.
3. In the *Domain field*, enter the exact domain of the service hosting your repositories, such as github.com, **gitlab.com**, or **your-internal-gitlab-url.com**.
4. For *Authentication Credential Type*, click to select **Personal Access Token**.
5. Enter your personal access token, then click **Add Credential**.
6. You should now see your credential listed in the Git *Credentials* panel. You can also delete it from this panel if desired.

## [Add a repository to a project.](https://docs.dominodatalab.com/en/latest/user_guide/314004/import-git-repositories/)

1. Open the project you want to add a repository to, then click **Files** from the left navigation bar.
2. Click to open the **Git Repositories** tab, then click **Add a New Repository**.
3. Enter an optional directory name and the HTTPS or SSH URI of the repository you want to add. The directory name will be the directory in that `/repos` this repository clones into. It defaults to the name of the repository.
4. Use the dropdown menu to choose which branch of the repository you want Domino to check out when it clones this repository into a run or workspace. If you leave this setting at *Use default branch*, Domino will check out the branch specified as default by your Git service, typically `master`. You can also specify a different branch name, tag name, or commit ID, or supply a custom Git ref.
5. Click **Add Repository**.

**You're All Set!**

## **Notes/Information:**

### **References/Internal Records:**

[https://docs.dominodatalab.com/en/latest/user\_guide/314004/import-git-repositories/](https://docs.dominodatalab.com/en/latest/user_guide/314004/import-git-repositories/)


# Administration
Source: https://docs.domino.ai/knowledge-base/administration-overview

Account and user management tasks for admins.

Account and user management tasks for admins.

## User Management

* [Updating Your Domino Account Details](/knowledge-base/changing-domino-account-information)


# App URLs and URL parameters
Source: https://docs.domino.ai/knowledge-base/app-urls-and-url-parameters

For Domino versions < 6.1.0: This article provides an overview of the different URLs you can use to access your Web App, as well as some options if you'd like to append parameters in your URL.

**NOTE:** The [release of Domino 6.1.0](https://docs.dominodatalab.com/release_notes/6-1-0/#_domino_apps_enhancements) included a significant update and overhaul of the Domino Apps feature. This article is mainly written for Domino Apps on versions previous to Domino 6.1.0 and may not be applicable to Domino 6.1.0 and newer.

This article provides an overview of the different URLs you can use to access your Web App, as well as some options if you'd like to append parameters in your URL.

### App URLs

There are essentially 3 different URLs you can use to access an App:

1. The official app URL:

```
http://<your-domino-url>/modelproducts/<app_id>
```

This URL will not ever change for an app. Going to this URL in a browser gives you your app with a Domino banner at the top.

2. The "static" internal app URL:

```
http://<your-domino-url>/u/<user>/<project>/app
```

Like the first URL, this URL will not change if you stop and start the app, but it will change if you rename the project or change ownership, since it has the project owner and project name in the URL.

3. The current session URL:

```
http://<your-domino-url>/<user>/<project>/r/notebookSession/<session_id>
```

This URL will change every time you start and stop the app.

### Using parameters in URLs

What if you want to use a parameter in your URL? For example, what if you want your landing page to be

```
<app url>/page.html?key1=value1?
```

Currently one of the three URL options will allow URL parameters. To summarize:

1. The official app URL will not ever change for an app. Currently, you can't append a parameter to it, as it just gets ignored/thrown out.

2. The "static" internal app UR: will not change if you stop and start the app, but it will change if you rename the project or change ownership. Currently, you can't append a parameter to it, as it just gets ignored/thrown out. Going to this URL in a browser immediately redirects to the next URL, your app without the Domino banner:

3. The current session URL will change every time you start and stop the app. You can append parameters or paths to other pages and the app will parse them.

### Example App with URL parameters

As we see above, there is a URL we can use for our App where we can append URL parameters. Great! But this URL is going to change any time the App is stopped/restarted. One way to address this is to make your initial app page a landing page, which can add your parameters before going to your main app page, or, if appropriate, prompt your app user for the parameters.

Let's look at a couple of examples! For both Apps, we will use the same app.sh and app.py:

**app.sh:**

```
#!/usr/bin/env bash
python ./app.py
```

**app.py:**

```
import http.server
import socketserver
PORT = 8888
Handler = http.server.SimpleHTTPRequestHandler
httpd = socketserver.TCPServer(("", PORT), Handler)
print(f'serving at port {PORT}')
httpd.serve_forever()
```

Both Apps will also use the same main App page. This page simply greets the user by name:

**appPage.html:**

```
<html>
<head>
 <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.6.0/jquery.min.js"></script>
</head>

<body onload="greetings()">
 <h1>Welcome to the App!</h1>
 <p id="greet"></p>
 <script>
  function greetings() {
   let urlParams = new URLSearchParams(window.location.search);
   let username = urlParams.get('name');
   let timeperiod = urlParams.get('period');
   if (username != null && timeperiod != null) {
    document.getElementById("greet").innerHTML =
    "Hello " + username + "! How are you to" + timeperiod + "?";
   }
  }
 </script>

</body>
</html>
```

**Example App #1**

For our first example, we'll just pass coded variables as URL parameters.

**Index.html:**

```
<html>
<head>
 <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.6.0/jquery.min.js"></script>
</head>

<body>
 <script>
  let person = "Domino";
  const d = new Date();
  let theHour = d.getHours();
  if(4<=theHour<=20){
   var period= "day";
  }
  else{
   period = "night";
  }
  location.href="appPage.html?name="+person+"&period="+period;
 </script>

</body>
</html>
```

With this example, we can use any of the URLs to get to our main App.

The official app URL (http\://`<your-domino-url>`/modelproducts/`<app\_id>`) will automatically go to appPage.html.

The "static" internal app URL (http\://`<your-domino-url>`/u/`<user>`/`<project>`/app) will automatically redirect to the current session URL and append the parameters: http\://`<your-domino-url>`/`<user>`/`<project>`/r/notebookSession/`<session\_id>`appPage.html?name=Domino\&period=day

**Example App #2**

For our second example, we'll actually prompt the user for some input.

**Index.html:**

```
<html>

<head>
 <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.6.0/jquery.min.js"></script>
</head>

<body onload="myFunction()">
 <script>
  function myFunction() {
   const d = new Date();
   let theHour = d.getHours();
   if(4<=theHour<=20){
    var period= "day";
   }
   else{
    period = "night";
   }

   let person = prompt("Please enter your name", "");
   if (person != null) {
    location.href="appPage.html?name="+person+"&period="+period;
   }
  }
 </script>
</body>
</html>
```

The end result of this is the same (we can use any of the URLs to get to our main App). The only difference is, this App is going to stop and prompt the user for input before moving on to our appPage.html.

Once the user adds their input - which will become a URL parameter - the official app URL (http\://`<your-domino-url>`/modelproducts/`<app\_id>`) will automatically go to appPage.html.

The "static" internal app URL (http\://`<your-domino-url>`/u/`<user>`/`<project>`/app) will automatically redirect to the current session URL and append the parameters: http\://`<your-domino-url>`/`<user>`/`<project>`/r/notebookSession/`<session\_id>`appPage.html?name=Domino\&period=day

### Summary

The examples provided above are incredibly basic, but give you an idea of some of the ways you can utilise URL parameters for your Apps!

\*Note: There is a feature request to allow users to append paths and query parameters to published apps URLs without needing alternatives such as the options described above. This request is currently tracked under DOM-26880.


# Build error for nvidia GPG key
Source: https://docs.domino.ai/knowledge-base/build-error-for-nvidia-gpg-key



**Version/Environment (if relevant):**

This issue affects Ubuntu-based environments (Ubuntu 16.04, 18.04, 20.04, 22.04) that use NVIDIA CUDA repositories in Dockerfiles for custom compute environments. This is relevant to all Domino versions where users build custom environments with NVIDIA/CUDA dependencies.

**Issue:**

Build failures occur during `apt-get update` commands with GPG signature verification errors. Common symptoms include:

```
Reading package lists... Done
W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
```

**Root Cause:**

NVIDIA's GPG signing key for their CUDA repositories expired, causing package managers to reject repository updates for security reasons. The old GPG key (7fa2af80) is no longer valid and must be replaced with the updated keyring package.

**Resolution:**

Install the new NVIDIA CUDA keyring package in your Dockerfile before any `apt-get update` operations. Add the following commands early in your Dockerfile:

```
# Fix NVIDIA GPG key - place before any apt-get update commands
# Remove old repository files if they exist (optional cleanup)
RUN sudo rm /etc/apt/sources.list.d/cuda.list || echo "cuda.list not found, skipping"
RUN sudo rm /etc/apt/sources.list.d/nvidia-ml.list || echo "nvidia-ml.list not found, skipping"

# Install new NVIDIA keyring (adjust Ubuntu version as needed)
# For Ubuntu 18.04:
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb && \
    sudo apt-key del 7fa2af80 && \
    sudo dpkg -i cuda-keyring_1.0-1_all.deb

# For Ubuntu 20.04, use:
# RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb && \
#     sudo apt-key del 7fa2af80 && \
#     sudo dpkg -i cuda-keyring_1.0-1_all.deb

# For Ubuntu 22.04, use:
# RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.0-1_all.deb && \
#     sudo apt-key del 7fa2af80 && \
#     sudo dpkg -i cuda-keyring_1.0-1_all.deb
```

**Notes/Information:**

* **Ubuntu Version Identification**: Check your Ubuntu version in a workspace terminal using: `lsb_release -a`
* **URL Pattern**: Update the URL to match your Ubuntu version: `ubuntu1804` (18.04), `ubuntu2004` (20.04), `ubuntu2204` (22.04)
* **Execution Order**: These commands must run before any other `apt-get update` or package installation operations
* **GPG/PGP/SBT Fixes**: If you have other GPG-related fixes in your Dockerfile, place the NVIDIA fix before them
* **Error Handling**: The `|| echo` statements prevent build failures if repository list files don't exist
* **Alternative Approach**: For environments that don't require specific CUDA versions, consider using Domino's pre-built GPU environments from quay.io/repository/domino/

**References/Internal Records:**

For the most current NVIDIA guidance, refer to: [https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/](https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/)


# Can Not Connect to KeyServer Caused by GPG Error
Source: https://docs.domino.ai/knowledge-base/can-not-connect-to-keyserver-caused-by-gpg-error

When attempting to refresh a CRAN GPG key during a Domino compute environment build, the build may fail if the environment cannot reach the keyserver. This article explains why this happens and how ow to resolve it without requiring keyserver connectivity

**Affected Versions:** All Domino versions building Ubuntu-based compute environments

## Issue

A compute environment build fails with a keyserver connection timeout similar to the following:

```
gpg: keyserver receive failed: Connection timed out

The command 'gpg --keyserver hkp://keyserver.ubuntu.com:80 \
  --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9 \
  && gpg -a --export E298A3A825C0D65DFD57CBB651716619E084DAB9 \
  | sudo apt-key add -' returned a non-zero code: 2
```

This typically occurs in environments where outbound access to keyserver infrastructure is restricted — for example, on-premises deployments, GovCloud environments, or deployments with strict egress ACLs.

## Root Cause

The build is attempting to fetch the CRAN repository signing key from the Ubuntu keyserver (keyserver.ubuntu.com) using the HKP protocol (port 11371 or port 80). Egress rules or firewall policies in the deployment environment block this outbound connection, causing the GPG keyserver lookup to time out and the build to fail.

**Deprecated:** The gpg … | apt-key add - pattern used in older Dockerfile instructions is deprecated as of Ubuntu 20.10 and removed in Ubuntu 24.04+. Even where it still functions, it should not be used in new or updated compute environments.

## Resolution

The recommended fix is to **bypass the keyserver entirely** by fetching the CRAN signing key directly from CRAN's own HTTPS endpoint at cloud.r-project.org. This avoids any dependency on HKP keyserver access and uses the modern gpg --dearmor approach.

Add the following to your compute environment's **Dockerfile Instructions**:

```
# Step 1 — Install prerequisites (if not already present)
RUN apt-get update && apt-get install -y --no-install-recommends \
    curl \
    ca-certificates \
    gnupg \
    && rm -rf /var/lib/apt/lists/*

# Step 2 — Fetch the CRAN signing key directly from CRAN (no keyserver needed)
RUN curl -fsSL https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc \
    | gpg --dearmor -o /usr/share/keyrings/cran_ubuntu_key.gpg

# Step 3 — Verify the key fingerprint (optional but recommended)
# Expected: E298 A3A8 25C0 D65D FD57 CBB6 5171 6619 E084 DAB9  [expires: 2027-09-30]
RUN gpg --show-keys /usr/share/keyrings/cran_ubuntu_key.gpg
```

**Tip:** This approach works in any environment that allows outbound HTTPS to cloud.r-project.org — no keyserver port access required.

## Fully Air-Gapped Environments

If your environment cannot reach any external endpoint during the build, the key must be pre-staged:

1. On a machine with internet access, download the key:

   ```
   curl -fsSL https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc \
     -o marutter_pubkey.asc
   ```
2. Upload the .asc file to your Domino project files.
3. Reference the key file in your Dockerfile Instructions using the Domino project file access pattern, or embed the key as a base64-encoded string in your Dockerfile.

For detailed guidance on accessing project files during environment builds, see:  [Access Project Files During Docker Build](https://support.domino.ai/support/s/article/Access-project-files-during-docker-build-using-Dockerfile-Instructions-1718868028462)

## Legacy Proxy Workaround (Not Recommended)

An older workaround involved routing keyserver traffic through an HTTP proxy:

```
# NOT RECOMMENDED — apt-key is deprecated on Ubuntu 22.04+
RUN apt-key adv \
  --keyserver-options http-proxy=http://PROXY_URL:PORT/ \
  --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9
```

**Not Recommended:** This approach is no longer recommended because:

* apt-key is deprecated since Ubuntu 20.10 and removed in Ubuntu 24.04+
* It still requires keyserver connectivity even with a proxy
* The direct CRAN HTTPS fetch described above is simpler and more reliable


# Updating Your Domino Account Details
Source: https://docs.domino.ai/knowledge-base/changing-domino-account-information



## **Version/Environment (if relevant):**

All Domino Version

## **What:**

The user intended to modify the Domino account information to verify that the listed users are correct.

## **How:**

To update your account information in your Domino deployment, follow these steps:

Sign in to your Domino deployment.

Navigate to Account Settings.

Select Manage Login Profile.

Update the desired fields (e.g., username, email, first name, or last name).

Click Change to save your updates.

**You're all set!**

## **Notes/Information:**

### **References/Internal Records:**


# Clearing Git Cache
Source: https://docs.domino.ai/knowledge-base/clearing-git-cache



There are security measures set on each deployment. They are placed in such a way they can perform a scan on each users script. When users sync codes to linked repository (github , gitlab etc) , the script maybe tagged as\*\*"sensitive information"\*\*and will be denied to the push request.

Even after completely wiping the script and starting over with the code to avoid this error, they are still unable to push to the github, gitlab etc. This would tell us there are still remnants of the code initially scanned by the organizations security measures .

We can address this by simply removing git cache.

The easiest way to clear your Git cache is to use the *git rm* command with the --*cached* option.

We can also use :

```
> git rm -r --cached .
> git add .
> git commit -m 'git cache cleared'
> git push
```


# Creating and Publishing React Apps
Source: https://docs.domino.ai/knowledge-base/creating-and-publishing-react-apps

Walks through creating a React app in a Domino Workspace with Create React App, configuring app.sh to set the dynamic app URL, publishing it as a Domino App, and managing node_modules sync via .dominoignore.

### ***Version/Environment (if relevant):***

Applies to creating a React app in a Domino Workspace and deploying it as a Domino App. This example uses [Create React App](https://create-react-app.dev/), though an existing React app can also be uploaded directly to a project instead.

## **Resolution:**

### Create the app structure

If you're using a [Domino Standard Environment](https://docs.dominodatalab.com/en/latest/user_guide/0d73c6/domino-standard-environments/) on Domino 5.9.x, Node.js and npm may already be included. If not present in your environment, add your preferred version by following the [Node.js installation instructions](https://nodejs.org/en/download/package-manager) (using nvm or fnm is recommended for Linux).

Once Node.js and npm are available, open a Terminal in your Domino Workspace and run the following, substituting your own app name for `domino-react`:

`npx create-react-app domino-react`

![Running npx create-react-app in a Domino Workspace terminal](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr\&ids=068UY00000HroMg\&d=%2Fa%2FUY0000018nnl%2FStZVxFtcPJdPpRU2GYb.8PTMT4bbmjydIPlsbiFxAdQ\&asPdf=false)

You may be prompted to confirm installing the `create-react-app` package — confirm to proceed. Once installed, it will create your React app. On success, you'll see output similar to:

![Successful create-react-app output in terminal](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr\&ids=068UY00000HroMh\&d=%2Fa%2FUY0000018nnm%2FJ2qwecN3RISYbOrmDuAn9W1zzIWBb37Po4HIiJj7l74\&asPdf=false)

This creates the React project's files and folder structure inside your `/mnt` folder:

![React app folder structure in /mnt](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr\&ids=068UY00000HroMi\&d=%2Fa%2FUY0000018nnn%2FFn2SRy4sc8O6u85HMP7NyXtG40zmf5HSxQ6OEYmPMDg\&asPdf=false)

A large number of files will now be available to sync from your Workspace. All of these need to be synced at least once for the app to run — given the file count, this initial sync may take a while.

![Syncing React app files from Workspace](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr\&ids=068UY00000HroMj\&d=%2Fa%2FUY0000018nno%2FfEtpE8Mjjpw.0byOVMNXV.ENjBllEEMT45MV4M_8g70\&asPdf=false)

### Starting the app

Once the React app's folder/file structure has synced from your Workspace to your project, add an `app.sh` to start the app. App URLs are dynamic — they include the run ID — so the app's `homepage` field in `package.json` must be set at runtime. The sample `app.sh` below does this, assuming a React app named `domino-react`:

```
URL_BASE="https://<platform-domino-url>/$DOMINO_PROJECT_OWNER/$DOMINO_PROJECT_NAME/r/notebookSession/$DOMINO_RUN_ID"
HOME_STRING="\"homepage\": \"$URL_BASE\","
cd domino-react/

#Set hostname to the base url for the app

if grep -F '"homepage"' package.json; then
#replace homepage if exists
sudo sed -i "s|\"homepage\":.*|$HOME_STRING|" package.json
else
#add homepage if line doesn't exist
sudo sed -i "4i $HOME_STRING" package.json
fi

#install necessary packages
sudo npm install create-react-app
sudo npm i web-vitals --save-dev
sudo npm install -g serve

#Below lines run a Production Build (does not auto refresh upon changes to src)
sudo npm run build
serve -s build -l 8888
```

This script writes a `homepage` line to `package.json` if one doesn't already exist, or overwrites the existing line with the current run's app URL.

With this `app.sh` in place, publish your App. It will take a few minutes to prepare; once ready, you can view it:

![Published React app running in Domino](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr\&ids=068UY00000HroMk\&d=%2Fa%2FUY0000018nnp%2Fssle5igej16aSbvLu9GcSTFMPJgrGSgW3OTn07MPDMY\&asPdf=false)

### Managing the app's resources

Creating the app with `create-react-app` generates a large number of subfolders and files in `node_modules`. These are required for the app to run, but you may want to exclude them from syncing on future runs or Workspace startups. Add this folder to your `.dominoignore` file to do so:

![Adding node\_modules to .dominoignore](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr\&ids=068UY00000HroMl\&d=%2Fa%2FUY0000018nnq%2F7fcZxYNfL8TRtVrcFrNW_6u6jPl2W2xu.V47AGHqlhs\&asPdf=false)

### **Notes/Information:**

The `sed -i "4i ..."` fallback in `app.sh` assumes the `homepage` line should be inserted at line 4 of `package.json`. This works for the default file layout produced by Create React App, but if your `package.json` has a different structure, adjust the line number accordingly.


# Database Locking
Source: https://docs.domino.ai/knowledge-base/database-locking

Database locks occur when queries or transactions conflict. Use the database-specific fix steps or Linux tools (ps, lsof, strace, fuser) to identify and resolve them.

Version/Environment (if relevant):
Applicable to all databases used within Domino deployments.

## Issue:

A database lock occurs when a query or transaction blocks or stalls due to a lock conflict. This may appear in code execution or log messages, for example:

Raw code

`sqlite3.OperationalError: database is locked`

## Root Cause:

Common causes of database locks include:

* Concurrent data access — multiple processes try to read and write the same data simultaneously, requiring the database to serialize access
* Long-running transactions — a transaction holds locks for an extended period, blocking other processes
* Resource contention — multiple processes compete for limited disk or memory resources
* Deadlocks — two or more processes each wait for the other to release a lock, creating a cycle that never resolves
* Hardware or software failures — locks persist beyond a failure even after the failure itself resolves

## Resolution:

MySQL

Error messages:

Raw code

`Deadlock found when trying to get lock
Lock wait timeout exceeded; try restarting transaction`

Fix steps:

1. Run `SHOW PROCESSLIST` to identify the process holding the lock.

Raw code

`SHOW PROCESSLIST;`

2. Terminate the blocking process using its process ID.

Raw code

`KILL <process_id>;`

3. If an open transaction is causing the lock, roll it back.

Raw code

`ROLLBACK;`

SQL Server

Error message:

Raw code

`Transaction (Process ID) was deadlocked on lock resources with another process
and was chosen as the victim of the deadlock. Restart the transaction.`

Fix steps:

1. Run `sp_who2` to identify the blocking process.

Raw code

`EXEC sp_who2;`

2. Terminate the blocking process using its process ID.

Raw code

`KILL <process_id>;`

3. If an open transaction is causing the lock, roll it back.

Raw code

`ROLLBACK;`

SQLite

Error message:

Raw code

`sqlite3.OperationalError: database is locked`

Fix steps:

1. Commit any open transaction.

Raw code

`conn.commit()`

2. Close the connection to release the lock.

Raw code

`conn.close()`

3. If multiple connections exist, close each one before reopening. Use the `sqlite3` module in Python to manage open connections.

Raw code

`# Example: close all connections before re-querying
for conn in open_connections:
conn.close()`

MongoDB

Error messages:

Raw code

`Error: couldn't execute operation: couldn't execute command.
CommandFailed: Transaction 12 is being rolled back - lock was not available`

Fix steps:

1. If a long-running operation holds the lock, wait for it to finish.
2. If the operation is stuck, terminate it using its operation ID.

Raw code

`db.killOp(<op_id>)`

3. As a last resort, restart the MongoDB server if a bug or system issue is causing the lock.

Linux tooling

Outside the database, use these Linux tools to identify signs of locking:

* ps — lists running processes and shows which processes hold locks
* lsof — lists all open files, including locked ones, and shows which process locks each file
* strace — traces system calls and signals for a process, helping identify the root cause of a lock
* fuser — shows which processes use a specific file or file system, including those holding locks

### Notes/Information:

* Design your schema and queries to minimize contention and reduce the need for locks.
* Experiment with different isolation levels to prevent transactions from conflicting with each other.
* Address locks promptly — persistent locks degrade database performance.

### References/Internal Records:

* Public KB Articles:
  * [Database Locking (KB-000001303)](https://support.domino.ai/support/s/article/Database-Locking-1718868034992)


# Docker Layers - failed to register layer: max depth exceeded
Source: https://docs.domino.ai/knowledge-base/docker-layers-failed-to-register-layer-max-depth-exceeded

Resolve the "failed to register layer: max depth exceeded" Docker error in Domino by reducing image layers via combined RUN commands or multi-stage builds.

## **Version:**

All docker versions

## **Error:**

When attempting to launch a workspace you are presented with this error: "***failed to register layer: max depth exceeded"*** in the logs. The error is also visible on the describe of a new pod (Under Events)

```
Warning Failed 12m (x7 over 26m) kubelet Failed to pull image "100.97.188.95:5000/dominodatalab/environment:63a051030e277d167bec6bfd-35": rpc error: code = Unknown desc = failed to register layer: max depth exceeded
```

## **Cause:**

Docker images consist of stacked layers. Each instruction in a Dockerfile creates one new layer. When using the overlay2 storage driver, Docker enforces a maximum of 127 layers per image.

This error occurs when your image exceeds that limit. Note that the base image your compute environment builds on may already use many layers. Your own Dockerfile instructions stack on top of those existing layers.

## **Resolution:**

Reduce the total number of layers in your Dockerfile. The most effective approaches are combining multiple `RUN` instructions and using multi-stage builds.

### **1. Combine RUN instructions**

Each separate **`RUN`** instruction creates a new layer. Combine related commands into a single `RUN` instruction using **`&&`**. Install all packages in one step:

```
RUN apt-get update && \
    apt-get install -y package1 && \
    apt-get install -y package2 && \
    apt-get install -y package3 && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*
```

### **2. Use multi-stage builds**

If your build process requires multiple tools, use multi-stage builds. Build artifacts in one stage, then copy only what you need into the final image. This avoids carrying intermediate layers into the final image:

```
FROM builder AS build
# Build commands go here

FROM base_image
# Copy artifacts from the build stage
COPY --from=build /path/to/artifacts /app
```

### **3. Create a new compute environment**

If your layer count still exceeds 127 after combining instructions, the base image itself may account for too many layers. In that case, create a new compute environment in Domino and copy your existing Dockerfile instructions into it, confirming the total layer count stays within the 127-layer limit.

For help editing Dockerfile instructions in a Domino compute environment, see [Use Dockerfile instructions](https://docs.dominodatalab.com/en/latest/user_guide/813d89/use-dockerfile-instructions/).

## **Additional resources**

* [Dockerfile best practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) — guidance on reducing layers and optimizing image size
* [Dockerfile reference](https://docs.docker.com/engine/reference/builder/) — full reference for all Dockerfile instructions


# Domino Product Version Lifecycle and Support Policies
Source: https://docs.domino.ai/knowledge-base/domino-product-version-lifecycle-and-support-policies



The following page provides information related to the lifecycle of each Domino product release. Timelines for both Product Patching and Technical Support are covered.

## **Domino Versions: Naming Conventions and Version Types**

Domino's product versions follow a common Semantic Version ([SemVer](https://semver.org/)) numbering scheme, **Ma.Mi.Pa**, where:

* **Ma (Major version)** - Contains new features, fixes, potential platform/technology changes, and may include breaking changes or interface updates that are not backward compatible

* **Mi (Minor version)** - Contains new features, fixes, and potential platform/technology support

* **Pa (Patch version)** - Contains only security updates and critical bug fixes

* **Example:** Version 6.0.2 is on the 6.x major release line, and 6.0.2 is the latest patch for the 6.0 minor version.

## **Domino Release Schedule**

### **Domino Cloud**

Domino Cloud targets weekly updates with new features, bug fixes, and platform/technology support.

### **Domino Enterprise (Self-Hosted)**

Self-hosted releases follow this schedule:

* **Major/Minor releases** - Released roughly every 6 months with new features, fixes, and possible platform/technology support
* **Patch releases** - Released between 30-50 days of the last major/minor/patch release, and includes only bug fixes and security updates (no new features)
* **Patch availability** - Patches are only released for the current minor version. If a bug fix or security issue necessitates a patch release, only supported versions of Domino will be patched.

***

## **End of Patching and End of Support Policies**

These policies apply to all self-hosted Domino versions.

### **Policy Definitions**

**End of Patching (EOP)** After this date, we will not release any more security or bug fix patchesfor a given version.

\*\*End of Life (EOL)\*\*After this date, Domino Support will no longer accept tickets, provide proactive monitoring, or answer technical questions about this version. For support on EOL versions of Domino, customers can purchase extended support.

***

## **Support Timelines**

### **End of Patching (EOP)**

**LATEST MAJOR RELEASE:**

* Security patches are provided between 30-50 days of release for the current minor version of the latest major release.

**PREVIOUS MAJOR RELEASE:**

* Domino will only patch previous major releases as needed AND
* Patches will only be released for supported versions of Domino (see EOL policy for supported Domino versions)

***

### **End of Life (EOL)**

**ALL VERSIONS:**

* **12 months** after the next major or minor version is released

**Examples:**

* Version 5.10.x reaches EOL 12 months after 5.11 is released
* Version 5.11.x reaches EOL 12 months after 6.0 is released
* Version 6.0.x reaches EOL 12 months after 6.1 is released


# Environment build fails with "The method driver /usr/lib/apt/methods/https could not be found"
Source: https://docs.domino.ai/knowledge-base/environment-build-fails-with-the-method-driver-usr-lib-apt-methods-https-could-not-be-found

Fix a compute environment build failure caused by missing apt-transport-https by adding a RUN apt-get install command to your Dockerfile before other apt commands.

The following error in a compute environment build

"*The method driver /usr/lib/apt/methods/https could not be found.*"

indicates that the "apt-transport-https" package is not installed. To correct this in your compute environment add the following line to your docker definition in the compute image prior to any other apt commands.

```
RUN apt-get install apt-transport-https -y
```


# ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied:
Source: https://docs.domino.ai/knowledge-base/error-could-not-install-packages-due-to-an-environmenterror-errno-13-permission-denied



Often times, users customize their environments to tailor their analytic needs. Package installation on the docker file is the most common path a user takes.

But , during the package installation process users sometimes encounter an error :

```
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied:
```

followed by the package name. This usually indicates you lack the necessary permissions to continue the install.

Luckily fixing the error can be simple. Just try installing the package by adding '**--user**' option.

```
 ! pip install (package) --user
```


# VSCode Error: ENOSPC: System limit for number of file watchers reached
Source: https://docs.domino.ai/knowledge-base/error-enospc-system-limit-for-number-of-file-watchers-reached

VSCode workspaces fail with "ENOSPC: System limit for number of file watchers reached" when large directories exhaust inotify handles. Fix by excluding folders like .venv from files.watcherExclude in VSCode settings, or increase fs.inotify.max_user_watches.

### **Version:**

All supported versions of Domino using VSCode as the workspace IDE.

## **Issue:**

An error prompt appears in a VSCode workspace with the following message:

```
Error: ENOSPC: System limit for number of file watchers reached
```

This error is also visible in the workspace support bundle logs:

```
[File Watcher (node.js)] Failed to watch /home/ubuntu/.local/share/code-server/Machine/settings.json
for changes using fs.watch() (Error: ENOSPC: System limit for number of file watchers reached, watch
'/home/ubuntu/.local/share/code-server/Machine/settings.json')
```

## **Root Cause:**

VSCode uses the Linux `inotify` subsystem to watch files for changes. Each file or directory being watched consumes one inotify handle. When a workspace contains a large number of files — for example, due to a Python virtual environment (`.venv`), `node_modules`, or a large project directory — VSCode exhausts the system's available file watcher handles, triggering this error.

## **Resolution:**

There are two approaches to resolve this issue. Try **Option 1** first before increasing system limits.

**Option 1 – Exclude large directories from file watching (Recommended)**
Configure VSCode to exclude large folders from the file watcher by adding them to the `files.watcherExclude` setting. This reduces the number of handles consumed without changing system limits.

Open your VSCode settings and add the following, expanding the list to include any additional large directories in your workspace (e.g., `.venv`):

```
"files.watcherExclude": {
  "**/.git/objects/**": true,
  "**/.git/subtree-cache/**": true,
  "**/node_modules/*/**": true,
  "**/.hg/store/**": true,
  "**/.venv/**": true
}
```

This setting can be applied at two levels:

* **User Settings** — applies globally to all VSCode instances for the user.
* **Workspace Settings** — stored in the workspace and applies only when that workspace is open.

To edit settings via the UI, first enable the split JSON editor by searching for and enabling the following setting:

```
workbench.settings.useSplitJSON
```

This opens a split pane view with a read-only reference on the left and a writeable editor on the right. Make your changes in the right pane and save.

**Option 2 – Increase the system inotify file watcher limit**
If excluding directories does not resolve the issue, a Domino Admin can increase the maximum number of file watchers at the OS level. Run the following commands on the node hosting the workspace:

```
# Set the new value (maximum is 524288)
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

# Verify the new value was applied
cat /proc/sys/fs/inotify/max_user_watches
```

The maximum supported value is **524288**. Set the value appropriate to the size of the workspaces on the deployment.

### **Notes/Information:**

Option 1 is preferred as it resolves the issue without requiring system-level changes or admin access. Common high-file-count directories to exclude include `.venv`, `node_modules`, and `.git/objects`.

For more information on this error in VSCode, see the official VSCode documentation: [Visual Studio Code – Unable to watch for file changes (ENOSPC)](https://code.visualstudio.com/docs/setup/linux#_visual-studio-code-is-unable-to-watch-for-file-changes-in-this-large-workspace-error-enospc).


# Errors during model build : Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/prometheus-client/
Source: https://docs.domino.ai/knowledge-base/errors-during-model-build-failed-to-establish-a-new-connection-errno-2-name-or-service-not-known-simple-prometheus-client



#### ***Version/Environment (if relevant):***

4.x, 5.x

## **Issue:**

When deploying a Model API the build is very slow and the build logs list repeating network related errors like:

```
Jan 23 2023 18:14:08 +0100 #13 4.807 WARNING: Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f081026bd90>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/prometheus-client/
```

The model eventually deploys, but the build process is slow.

## **Root Cause:**

Model API initialization code tries to install the python package prometheus-client 0.11.0 (among others), a client for metrics & monitoring tools. If your system is air-gapped or public/internet access is tightly controlled your system might not be able to fetch that python package from [https://pypi.org](https://pypi.org), resulting in the error above.

## **Resolution:**

Install prometheus-client in your project's Compute Environment via a dockerfile instruction:

**RUN pip install prometheus-client==0.11.0**

If this isn't possible another idea is to modify pip's config files to make it not retry and "fail fast", like:

```
pip config --quiet --user set global.retries 0
pip config --quiet --user set global.timeout 1
pip config --quiet --user set global.disable-pip-version-check True
```

## **Notes/Information:**

If the above isn't possible please reach out to Domino Technical Support referencing DOM #36045 and DOM #41738


# failed to pull image 602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5
Source: https://docs.domino.ai/knowledge-base/failed-to-pull-image-602401143452-dkr-ecr-us-east-1-amazonaws-com-eks-pause-3-5



#### ***Version/Environment:***

This applies to deployments hosted in Amazon AWS.

## **Issue:**

A customer's pod was failing to transition to a running state, it's describe showed this event:

```
Warning FailedCreatePodSandBox 7s (x15 over 3m11s) kubelet Failed to create pod sandbox: rpc error: code= Unknown desc = failed to get sandbox image "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5": failed to pull image "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5": failed to pull and unpack image "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5": failed to resolve reference "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5": pulling from host 602401143452.dkr.ecr.us-east-1.amazonaws.com failed with status code [manifests 3.5]: 401 Unauthorized
```

The issue impacted their DEV environment, but not their PROD environment and in some ways didn't seem 100% reproducible. The customer found that this problem did NOT occur when a new NODE was spun-up to host the pod.

## **Root Cause:**

Container image pause is regularly used by kubernetes as a temporary placeholder for future containers during pod creation. Due to a problem in EKS there is a broken process in which a garbage collector is pruning images on nodes and removing the pause container image that is initially added as part of the node provisioning. Apparently it is no longer possible to fetch that "pause" from the default referenced location also due to a change on the EKS side. The EKS bug is partly discussed here: [https://github.com/aws/amazon-vpc-cni-k8s/issues/2030#issuecomment-1372204463](https://github.com/aws/amazon-vpc-cni-k8s/issues/2030#issuecomment-1372204463)

The support for Container Runtime Interface (CRI) for Docker (Dockershim) has been discontinued with the introduction of Kubernetes 1.24 in EKS. Official Amazon EKS Amazon Machine Images (AMIs) now use containerd as the runtime exclusively.

The issue occurs when kubelet, a key component of Kubernetes, seeks to garbage collect these "pause" containers. While Kubernetes offers credentials for obtaining pod images, containerd does not use these credentials when requesting the sandbox image. As a result, users who attempt to pull these images may experience authentication issues due to a lack of saved credentials in containerd. This drawback necessitates the re-deployment of nodes.

To address this, kubelet offers the '--pod-infra-container-image' flag which is specific to the EKS "pause" image. This flag has a two fold purpose: it guarantees that the image is not pruned by the image garbage collector and so helps prevent the occurrence of such issues. Ordinarily, kubelet should not undertake the garbage collection of the "pause" container. If this occurs, it may potentially result from various underlying causes, such as the "pause" being garbage collected due to disk pressure (evidenced by commands like ('df -h /var/lib..').

For further insights and discussions related to these challenges and solutions, you may refer to the following resources:

* Kubernetes GitHub issue discussing pause container image: [https://github.com/kubernetes/kubernetes/issues/81756](https://github.com/kubernetes/kubernetes/issues/81756)
* Rancher GitHub issue regarding Kubernetes and pause container: [https://github.com/rancher/rke2/issues/1830](https://github.com/rancher/rke2/issues/1830)
* Kubernetes GitHub issue about kubelet's behavior and pause container: [https://github.com/kubernetes/kubernetes/issues/62388](https://github.com/kubernetes/kubernetes/issues/62388)

## **Resolution:**

If you are witnessing this problem you can attempt to

* Ensure the nodes have policy AmazonEC2ContainerRegistryReadOnly, if not add it. The following can check.

  ```
  aws iam list-entities-for-policy --policy-arn "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly" --entity-filter Role --query 'PolicyRoles[].RoleName'
  ```
* You can also try the bootstrap args listed in the link above.
* Check the root volume is sufficiently sized (#df -h | column -t) , also check that AMI versions used by the nodes is recent and compatible with EKS 1.24
* Check the config.toml on these nodes looks good, especially for sandbox\_image.

```
root@ip-10-0-86-99 ~]# cd /etc/containerd/
[root@ip-10-0-86-99 containerd]# ls
config.toml
[root@ip-10-0-86-99 containerd]# cat config.toml
version = 2
root = "/var/lib/containerd"
state = "/run/containerd"

[grpc]
address = "/run/containerd/containerd.sock"

[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
discard_unpacked_layers = true

[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pause:3.5"
```

* Check kubelet entry for --pod-infra-container-image

```
root   17634 2.3 0.3 2817932 119732 ?   Ssl 23:11  0:07
/usr/bin/kubelet --config /etc/kubernetes/kubelet/kubelet-config.json
--kubeconfig /var/lib/kubelet/kubeconfig --container-runtime-endpoint
unix:///run/containerd/containerd.sock --image-credential-provider-config
/etc/eks/image-credential-provider/config.json --image-credential-provider-bin-dir
/etc/eks/image-credential-provider --node-ip=10.0.86.99 --pod-infra-container-image=602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pause:3.5 --v=2 --cloud-provider=aws --container-runtime=remote --node-labels lifecycle=OnDemand --node-labels=dominodatalab.com/node-pool=default,dominodatalab.com/domino-node=true
```

* If urgent you can cordon the broken node and cycle it out with a newer node.
* If the above does not work you can also attempt to cordon the node, restart kubelet on the effected node and uncordon the node:

```
 On the kubernetes environment, cordon the node
```

```
#kubectl cordon <node name>
```

```
Logon and run on the node itself
```

```
#sudo systemctl restart kubelet && sudo systemctl status kubelet
```

```
On the kubernetes environment, uncordon the node
```

```
#kubectl uncordon <node name>
```

* If the above does not help, you should contact AWS Support.

**Caveat:**

In Domino 5.4.x. An issue has been identified under DOM-44673, where the Image-cache-agent is pruning the pause image from EKS nodes in ECR. This has been fixed under Domino 5.5. If you identify this to be the issue, then there is the option of disabling the Image-cache-agent, doing this however will have a negative impact on performance since the agent will no longer be able to prune cached images on compute nodes.

To disable:
This command patches the image-cache-agent daemonset in the domino-platform namespace by adding a nodeSelector that will not match any nodes.

```
kubectl -n domino-platform patch daemonset image-cache-agent -p '{"spec": {"template": {"spec": {"nodeSelector": {"non-existing": "true"}}}}}'
```

To re-enable:

This command patches the image-cache-agent daemonset in the domino-platform namespace to remove the nodeSelector that was added in the previous example.

```
kubectl -n domino-platform patch daemonset image-cache-agent --type json -p=[{"op": "remove", "path": "/spec/template/spec/nodeSelector/non-existing"}]
```


# FailedMount event "efs.csi.aws.com not found in the list of registered CSI drivers"
Source: https://docs.domino.ai/knowledge-base/failedmount-event-efs-csi-aws-com-not-found-in-the-list-of-registered-csi-drivers

Troubleshoot the FailedMount "efs.csi.aws.com not found in registered CSI drivers" warning on EKS pods, including how to verify driver installation with kubectl.

## Issue

Admins may see the following warning when describing a pod:

```
Warning FailedMount 55s (x2 over 1m2s) kubelet MountVolume.MountDevice failed for volume "domino-shared-store-domino-compute" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name efs.csi.aws.com not found in the list of registered CSI drivers
```

## Root cause

This warning typically appears only in Elastic Kubernetes Service (EKS) clusters. It's usually temporary — the Container Storage Interface (CSI) driver is still exposing the file system to the node's pods.

If the `FailedMount` warnings stop and the volume mounts successfully, the issue is transient and requires no action.

## Resolution

If `FailedMount` warnings persist and the pod does not mount successfully, follow the steps below.

### Step 1: Confirm the EFS CSI driver is registered

Run the following command to check whether the `efs.csi.aws.com` driver appears in the list of registered CSI drivers:

```
kubectl get csidrivers.storage.k8s.io
```

Verify that `efs.csi.aws.com` appears in the output.

If the driver is not listed, the node may not have been provisioned correctly. Review your EKS node provisioning configuration to confirm the EFS CSI add-on was enabled during cluster or node group setup.

### Step 2: Inspect CSI driver pods

If `efs.csi.aws.com` is listed but the mount failure continues, inspect the CSI-related pods across all namespaces:

```
kubectl get po -A | grep csi
```

For any pods returned, review their descriptions and logs. Include the `kube-system` namespace in your search. Check both the CSI controller pods and the CSI node pod assigned to the same node as your failing pod.

If you need help, contact Technical Support and include your screenshots, pod descriptions, and logs.


# FastAPI App with Multiple Endpoints
Source: https://docs.domino.ai/knowledge-base/fastapi-app-with-multiple-endpoints



**Version/Environment (if relevant):** Domino Version 6.1 and Above

## **What:**

Domino's [Simple FastAPI example](https://support.domino.ai/support/s/article/Simple-FastAPI-example-as-a-Published-App-1718868040509) is a great way to get started with FastAPI in Domino, but what if you're ready to go a step further and set up a FastAPI app to take multiple GET and POST requests?

This guide assumes you've already installed FastAPI in your environment or set it up to install via *requirements.txt*.

## **How:**

## app.sh

Like many of our other App examples, this simply calls the Python script:

```
python app.py
```

## Python script

First, we need to import our packages and collect some environment variables:

```
from fastapi import FastAPI
from pydantic import BaseModel
import uvicorn
import os

runID = os.environ['DOMINO_RUN_ID']
userID = os.environ['DOMINO_PROJECT_OWNER']
projectID = os.environ['DOMINO_PROJECT_NAME']
```

Domino utilizes a reverse proxy, and App URLs have path prefixes that need to be included (e.g. `/username/app-name/app`). Therefore, we need to set a root\_path for our FastAPI app (see [https://fastapi.tiangolo.com/advanced/behind-a-proxy/#setting-the-root\_path-in-the-fastapi-app](https://fastapi.tiangolo.com/advanced/behind-a-proxy/#setting-the-root_path-in-the-fastapi-app)). With that example in mind, I was able to get to the swagger UI by modifying our code example as follows:

```
path_prefix = f"/{userID}/{projectID}/r/notebookSession/{runID}"

app = FastAPI(root_path=path_prefix)
```

We can then add routes as desired for our API:

```
class Item(BaseModel):
  name: str
  description: str
  price: float

@app.get("/")
async def root():
  return {"message": "Hello World"}

@app.get("/items/{item_id}")
async def read_item(item_id):
  return {"item_id": item_id}

@app.post("/items/")
async def create_item(item: Item):
  return item
```

Lastly, run the app:

```
if __name__ == "__main__":
  uvicorn.run("app:app",host="0.0.0.0",port=8888,reload = True)
```

Putting it all together, the complete app.py looks like:

```
from fastapi import FastAPI
from pydantic import BaseModel
import uvicorn
import os

runID = os.environ['DOMINO_RUN_ID']
userID = os.environ['DOMINO_PROJECT_OWNER']
projectID = os.environ['DOMINO_PROJECT_NAME']

path_prefix = f"/{userID}/{projectID}/r/notebookSession/{runID}"

print(f"App URL is https://<domino_url>/{path_prefix}")

app = FastAPI(root_path=path_prefix)

class Item(BaseModel):
  name: str
  description: str
  price: float

@app.get("/")
async def root():
  return {"message": "Hello World"}

@app.get("/items/{item_id}")
async def read_item(item_id):
  return {"item_id": item_id}

@app.post("/items/")
async def create_item(item: Item):
  return item

if __name__ == "__main__":
  uvicorn.run("app:app",host="0.0.0.0",port=8888,reload = True)
```

Similar to our Simple FastAPI example, hitting "View App" will bring up your FastAPI app with our "Hello World" message.

While our root page is accessible via the "official" app URL, you aren't able to append paths to this. For this, you need to use the session URL: `http://<your-domino-url>/<user>/<project>/r/notebookSession/<session_id>` (see [App URLS and URL parameters](https://support.domino.ai/support/s/article/App-URLs-and-URL-parameters-1718868036753)). To find this easily, you might add a print statement to your app.py, such as:

```
print(f"App URL is https://<domino_url>/{path_prefix}")
```

...which would print in your User Output when you start your App.

With that URL, you can utilize your other endpoints. For example:

`http://<your-domino-url>/<userID>/<projectID>/r/notebookSession/<runID>/item/foo`

You can also view the Swagger UI for your FastAPI app at:

`http://<your-domino-url>/<userID>/<projectID>/r/notebookSession/<runID>/docs`

**You're all set.**

## **Notes/Information:**

[https://support.domino.ai/support/s/global-search/fastAPI](https://support.domino.ai/support/s/global-search/fastAPI)

### **References/Internal Records:**


# Getting started with Streamlit in Domino
Source: https://docs.domino.ai/knowledge-base/getting-started-with-streamlit-in-domino

Install Streamlit in a Compute Environment, build your app in /mnt, live-test via Jupyter Server Proxy, then publish using an app.sh that binds to port 8888 on 0.0.0.0 with headless mode enabled.

### **Version/Environment (if relevant):**

All versions. Requires a Compute Environment with Streamlit installed. Domino Standard Environments are recommended.

## **How To:**

### **Step 1 — Install Streamlit**

If you are using a [Domino Standard Environment](https://docs.dominodatalab.com/en/latest/user_guide/0d73c6/domino-standard-environments/), add the Streamlit package to your Compute Environment via one of the following methods:

**Dockerfile instruction:**

```
USER root
RUN pip install -U streamlit starlette
```

**requirements.txt:**

`streamlit
starlette`
**Note:** Some older versions of streamlit and starlette have compatibility issues, make sure you are using the latest versions.

### **Step 2 — Build your Streamlit app**

Create your Streamlit app file (e.g. `streamlit_app.py`) in your project's `/mnt` directory. The example below is a working demo adapted from [Streamlit's NYC Uber Ridesharing demo](https://github.com/streamlit/demo-uber-nyc-pickups) (updated for Streamlit 1.3.8):

```
import os
import altair as alt
import numpy as np
import pandas as pd
import pydeck as pdk
import streamlit as st

st.set_page_config(layout="wide", page_title="NYC Ridesharing Demo", page_icon=":taxi:")

@st.cache_resource
def load_data():
    path = "uber-raw-data-sep14.csv.gz"
    if not os.path.isfile(path):
        path = f"https://github.com/streamlit/demo-uber-nyc-pickups/raw/main/{path}"
    data = pd.read_csv(
        path,
        nrows=100000,
        names=["date/time", "lat", "lon"],
        skiprows=1,
        usecols=[0, 1, 2],
        parse_dates=["date/time"],
    )
    return data

def map(data, lat, lon, zoom):
    st.write(pdk.Deck(
        map_style="mapbox://styles/mapbox/light-v9",
        initial_view_state={"latitude": lat, "longitude": lon, "zoom": zoom, "pitch": 50},
        layers=[pdk.Layer("HexagonLayer", data=data, get_position=["lon", "lat"],
            radius=100, elevation_scale=4, elevation_range=[0, 1000],
            pickable=True, extruded=True)],
    ))

@st.cache_data
def filterdata(df, hour_selected):
    return df[df["date/time"].dt.hour == hour_selected]

@st.cache_data
def mpoint(lat, lon):
    return (np.average(lat), np.average(lon))

@st.cache_data
def histdata(df, hr):
    filtered = df[(df["date/time"].dt.hour >= hr) & (df["date/time"].dt.hour < (hr + 1))]
    hist = np.histogram(filtered["date/time"].dt.minute, bins=60, range=(0, 60))[0]
    return pd.DataFrame({"minute": range(60), "pickups": hist})

data = load_data()
row1_1, row1_2 = st.columns((2, 3))

if not st.session_state.get("url_synced", False):
    try:
        pickup_hour = int(st.query_params["pickup_hour"][0])
        st.session_state["pickup_hour"] = pickup_hour
        st.session_state["url_synced"] = True
    except KeyError:
        pass

def update_query_params():
    st.query_params["pickup_hour"] = st.session_state["pickup_hour"]

with row1_1:
    st.title("NYC Uber Ridesharing Data")
    hour_selected = st.slider("Select hour of pickup", 0, 23,
        key="pickup_hour", on_change=update_query_params)

with row1_2:
    st.write("""
## Examining how Uber pickups vary over time in New York City
    and at its major regional airports.""")

row2_1, row2_2, row2_3, row2_4 = st.columns((2, 1, 1, 1))
la_guardia = [40.7900, -73.8700]
jfk = [40.6650, -73.7821]
newark = [40.7090, -74.1805]
zoom_level = 12
midpoint = mpoint(data["lat"], data["lon"])

with row2_1:
    st.write(f"**All NYC from {hour_selected}:00 to {(hour_selected + 1) % 24}:00**")
    map(filterdata(data, hour_selected), midpoint[0], midpoint[1], 11)
with row2_2:
    st.write("**La Guardia Airport**")
    map(filterdata(data, hour_selected), la_guardia[0], la_guardia[1], zoom_level)
with row2_3:
    st.write("**JFK Airport**")
    map(filterdata(data, hour_selected), jfk[0], jfk[1], zoom_level)
with row2_4:
    st.write("**Newark Airport**")
    map(filterdata(data, hour_selected), newark[0], newark[1], zoom_level)

chart_data = histdata(data, hour_selected)
st.write(f"**Rides per minute between {hour_selected}:00 and {(hour_selected + 1) % 24}:00**")
st.altair_chart(
    alt.Chart(chart_data).mark_area(interpolate="step-after")
    .encode(x=alt.X("minute:Q", scale=alt.Scale(nice=False)),
            y=alt.Y("pickups:Q"), tooltip=["minute", "pickups"])
    .configure_mark(opacity=0.2, color="red"),
    use_container_width=True,
)
```

### **Step 3 — Live test from a workspace**

### JupyterLab

You can live test your Streamlit app from a JupyterLab workspace using Jupyter Server Proxy. Ensure that [JupyterLab](https://jupyterlab.readthedocs.io/en/stable/getting_started/installation.html#pip) and [jupyter-server-proxy](https://jupyter-server-proxy.readthedocs.io/en/latest/install.html) are installed in your Compute Environment. Both are pre-installed in Domino Standard Environments.

1. Launch a JupyterLab workspace and open a new terminal.
2. Run your app from the `/mnt` directory:

   ```
   streamlit run streamlit_app.py --server.port 8501
   ```
3. Access the app in your browser using the following URL pattern (default Streamlit port is `8501`):

   ```
   https://<your-domino-url>/<username>/<project-name>/notebookSession/<run-id>/proxy/8501/
   ```

**Note:** The trailing `/` at the end of the URL is required. Omitting it will prevent the app from loading. The `<run-id>` can be found in the URL of the active workspace session.

### VSCode

Use the same process to start the app as above, but for live testing from a **VSCode workspace**, the URL structure differs — insert `/r/` before `notebookSession`:
ex.
`https://<your-domino-url>/<username>/<project-name>/r/notebookSession/<run-id>/proxy/8501/`

### **Step 4 — Publish the app in Domino**

Once development is complete, publish the app using an `app.sh` file. Create `app.sh` in the root of your project with the following contents:

```
#!/bin/bash
set -x
streamlit run streamlit_app.py \
    --server.port 8888 \
    --server.address 0.0.0.0 \
    --server.headless true

streamlit run streamlit_app.py
```

This file configures Streamlit to bind to the host and port Domino expects for published Apps. Once saved:

1. Navigate to **Deployment > Apps & Agents** in your project.
2. Click **Publish** and select **App**
3. Complete the form and click **Publish Domino App**.
4. Once the App is running, click **View App** to open it.

**Notes/Information:**

* See: [Live testing Streamlit in Domino on VSCode workspace](https://support.domino.ai/support/s/article/Live-testing-the-Streamlit-in-Domino-on-VSCode-workspace-1718868026350) (KB 000001036).
* For full Domino App publishing documentation, see: [Publish Apps — Domino Docs](https://docs.dominodatalab.com/en/latest/user_guide/71635d/publish-apps/).


# How to connect and authenticate to Gitlab & GitHub with ssh keys using the native bash terminal
Source: https://docs.domino.ai/knowledge-base/how-to-connect-and-authenticate-to-gitlab-github-with-ssh-keys-using-the-native-bash-terminal

Set up SSH keys for GitLab and GitHub on a Linux bash terminal, test the connection, clone a repo, and push/pull changes

Use these steps to set up a Secure Shell (SSH) key using the native Linux bash terminal, test the connection, clone a repository, and perform basic Git operations like pushing and pulling changes.

## Step 1: Install package and generate SSH key

Run the following commands to install Git and generate an SSH key:

```
sudo apt install git
ssh-keygen -t ed25519 -C "your_email@test.com"
```

Follow the prompts to save the key (typically to `~/.ssh/id_ed25519`) and optionally add a passphrase.

## Step 2: Start the SSH agent and add key to the agent

```
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519
```

## Step 3: Copy the public SSH key to GitLab

Run the following command to print your public key:

```
cat ~/.ssh/id_ed25519.pub
```

Copy the full output, then add it to GitLab:

1. Go to your GitLab profile settings.
2. Navigate to **SSH Keys**.
3. Paste your SSH key into the **Key** field.
4. Add a descriptive title in the **Title** field.
5. Click **Add key**.

## Step 4: Test the connection

```
ssh -T git@gitlab.com
```

A successful connection returns:

```
Welcome to GitLab, @your_username!
```

## Step 5: Clone a repository, commit, push, and pull

Clone the repository and create a test file:

```
git clone git@gitlab.com:username/repo.git
cd repo
echo "Hello, GitLab!" > testfile.txt
git add testfile.txt
git commit -m "Add a test file"
```

**Error → Fix:** If Git prompts you to set up a global identity, run the following commands, then retry the commit:

```
git config --global user.name "Your Name"
git config --global user.email "you@test.com"
```

### Push changes

```
git push origin main
```

### Pull changes

```
git pull origin main
```

> **Note:** Replace `main` with the appropriate branch name if it differs.

## References

* [Adding a new SSH key to your GitHub account](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account)
* [Add an SSH key to your GitLab account](https://docs.gitlab.com/ee/user/ssh.html#add-an-ssh-key-to-your-gitlab-account)


# How to prevent the permission denied error when running "pip install" in a workspace
Source: https://docs.domino.ai/knowledge-base/how-to-prevent-the-permission-denied-error-when-running-pip-install-in-a-workspace



If you are attempting to install or upgrade packages using "pip install" in a workspace, then you may receive a permission denied error.

For example:

```
 !pip install --upgrade scikit-learn

OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/scikit-learn'
```

It's possible to install/upgrade packages in a workspace session, but you need to include the --user flag to avoid the permissions error.

For example:

```
!pip install --upgrade --user scikit-learn
```

However, this installation will not persist from run to run. If you consistently need this library upgrade, you can install this in a custom environment by following the instructions [here](https://support.domino.ai/support/s/article/Installing-Packages-And-Dependencies-1723748824009).


# How to Troubleshoot Kubernetes Autoscaling (ASG) Issues in Domino
Source: https://docs.domino.ai/knowledge-base/how-to-troubleshoot-autoscaling-asg-issues

Pods fail to schedule due to autoscaling issues such as ASG misconfiguration, AZ mismatches, insufficient resources, or node label conflicts. Check execution.log, pod events, and ASG settings to diagnose.

### **Version/Environment (if relevant):**

All versions of Domino; applies to AWS-based deployments using Kubernetes Cluster Autoscaler and AutoScaling Groups (ASGs)

## **Issue:**

Executions fail to start or pods fail to schedule due to autoscaling issues. Common symptoms include pods stuck in Pending state, scale-up not triggering, or volume node affinity conflicts. Errors appear in the support bundle's execution.log or events.json, for example:

```
2023-06-26T15:06:39.204Z: pod didn't trigger scale-up:2node(s) had volume node affinity conflict,26node(s) didn't match Pod's node affinity/selector
Please check if the hardware tier'p3-Large-GPU'has a valid nodePool
```

```
"resourceKind":"Pod",
"reason":"NotTriggerScaleUp",
"description":"pod didn't trigger scale-up: 26 node(s) didn't match Pod's node affinity/selector, 2 node(s) had volume node affinity conflict",
```

## **Root Cause:**

Autoscaling failures in Domino are typically caused by one or more of the following:

* Autoscaling is not enabled in the cluster yaml
* Pod resource requests (CPU/memory) exceed what any node in the group can provide
* Mismatch between ASG configuration in AWS and the Kubernetes autoscaler deployment settings
* ASG configured across multiple Availability Zones, causing volume node affinity conflicts
* Node labels not matching the Hardware Tier's node selector or affinity rules
* ASG max node group size has been reached

## **Resolution:**

**1. Verify autoscaling is enabled in the cluster yaml:**

```
autoscaler:
 enabled: true
```

**2. Describe the unscheduled pod and inspect its Events section:**

Common event messages and their meanings:

* **Autoscaler intentionally not triggering scale-up** — pods request more CPU/memory than any node can provide:

```
Normal NotTriggerScaleUp 2m24s (x18 over 27m) cluster-autoscaler pod didn't trigger scale-up (it wouldn't fit if a new node is added): 45 Insufficient memory, 3 max node group size reached, 45 Insufficient cpu
```

* **Autoscaler not triggering when it should** — mismatch between AWS ASGs and Kubernetes settings:

```
Warning FailedScheduling <unknown> default-scheduler 0/12 nodes are available: 1 Insufficient memory, 1 node(s) didn't match node selector, 10 Insufficient cpu.
```

* **Volume node affinity conflict** — ASG spans multiple AZs that don't match storage volume AZs:

```
pod didn't trigger scale-up (it wouldn't fit if a new node is added): 1 node(s) had volume node affinity conflict
```

**3. Verify and reconcile ASG settings between AWS and Kubernetes:**

```
kubectl get configmap cluster-autoscaler-status -n <install namespace> -o yaml
kubectl edit deployment -n <platform namespace> <cluster autoscaler deployment name>
```

Ensure each ASG has only one Availability Zone, both in the AWS console and in the configmap. Storage volume AZs must match ASG instance AZs.

**4. If scale-up is intentionally not triggering due to insufficient resources, reduce pod resource requests** via Hardware Tier settings. Refer to: [https://admin.dominodatalab.com/en/4.4.1/compute/hardware-tiers.html?highlight=hardware%20tier](https://admin.dominodatalab.com/en/4.4.1/compute/hardware-tiers.html?highlight=hardware%20tier)

**5. Review min, max, and currently running nodes:**

```
kubectl describe cm -n <could be platform namespace, could be something like kube-system> cluster-autoscaler-status
```

### **Notes/Information:**

Other less common scheduling messages and their meanings:

* `2 node(s) didn't match pod affinity rules, 2 node(s) didn't match pod affinity/anti-affinity rules` — Pod labels or affinity rules don't match the available nodes. Describe the pod and check the affinity section.
* `11 node(s) didn't match Pod's node affinity` or `4 node(s) didn't match node selector` — Nodes exist but don't carry the label required by the Hardware Tier's Node Selector (e.g., dominodatalab.com/node-pool=xlarge-memory). Check pod affinity, Node Selector, and node labels.
* `3 max node group size reached` — The ASG's maximum node count in AWS has been reached. Increase the ASG maxSize to allow additional nodes to spin up.

If the Kubernetes autoscaler deployment is edited directly, also update the autoscaler section of the cluster yaml (deploy manifest) to persist changes across upgrades and maintenance.


# How to use the apt-get install command in environments
Source: https://docs.domino.ai/knowledge-base/how-to-use-the-apt-get-install-command-in-environments

Fix the apt-get install non-zero exit code error in environment builds by adding the -y flag to run in non-interactive mode.

# How to use apt-get install in environments

When building an environment, running apt-get install without the -y flag causes the following error:

```
The command '/bin/sh -c sudo apt-get install <package>' returned a non-zero code: 100
ERROR: ProgressMessage{id=null, status=null, stream=null,
error=The command '/bin/sh -c sudo apt-get install <package>' returned a non-zero code: 100, progress=null,
progressDetail=null}
```

The error message suggests a connectivity issue, but the actual cause is the missing -y confirmation switch. Without -y, apt-get waits for interactive input during the install prompt, which causes the non-zero exit code.

## Fix: add the -y flag to apt-get install

Add the -y confirmation switch to your command so apt-get runs in non-interactive mode:

```
USER root
RUN apt-get install -y <package>
```


# htop – Interactive process viewer: a brief overview
Source: https://docs.domino.ai/knowledge-base/htop-interactive-process-viewer-a-brief-overview

Understand and use htop's interface sections, color coding, process columns, and installation steps for Linux system monitoring.

htop is a terminal-based system monitoring tool that displays process statistics and resource usage in a color-coded, human-readable format. It complements the standard `top` utility by adding mouse support, direct process termination, and improved readability. Use htop to monitor running processes and their full command lines to identify the source of system load.

## **Installation**

htop is available in most major distributions' standard repositories.

On any Debian-based distribution:
`sudo apt-get install htop -y`

On any Red Hat-based distribution:
`sudo yum install htop -y`

## **Run htop**

To launch htop, type the following command and press **Enter**:
`htop`

## **The user interface explained**

The htop interface contains four sections:

* **Section 1** — Interactive bar gauges for processors and memory, which you can customize
* **Section 2** — A quick overview of running tasks, load average, and uptime
* **Section 3** — A detailed view of running processes
* **Section 4** — A list of common keyboard shortcuts, which you can also click with a mouse

## **Section 1: Processors and memory**

### **CPU utilization**

This section displays CPU and memory resources. The number of bars indicates the number of CPU cores — a 4-core server shows bars labeled 0 through 3, a 5-core server shows five bars, and so on. Each bar represents one CPU core. The percentage on the right side of each bar shows current CPU utilization.

htop uses color coding to categorize CPU utilization by process type:

* **Blue** — Low-priority processes
* **Green** — User processes
* **Red** — System/kernel processes
* **Yellow/Orange** — Hardware Interrupt Request (IRQ) time
* **Magenta** — Soft IRQ time
* **Grey** — I/O wait time
* **Cyan** — Steal time

In the example above, the bars are mostly empty, indicating the CPU is largely idle. Sustained high CPU usage — not occasional spikes — warrants investigation, as it affects overall server performance.

### **RAM and swap utilization**

This section shows current memory usage. RAM is system memory; swap is a portion of disk storage the server uses as overflow memory. In the example, only 383 MB of RAM is in use.

htop uses the following colors for memory:

* **Green** — Used memory pages
* **Blue** — Buffer pages
* **Yellow** — Cache pages

When RAM is exhausted, the system falls back to swap space. High usage of both RAM and swap degrades the server's ability to handle tasks effectively. In the example, swap shows 0K/0K, indicating no swap space is configured.

## **Section 2: Processes and load average**

This section summarizes running tasks. In the example, 33 total processes are present, with 1 currently running. The value labeled thr (18 in the example) represents the number of kernel threads.

Load average (also known as the processor queue length) shows the average system load over the past 1, 5, and 15 minutes — displayed as three values such as `0.14`, `0.07`, and `0.04`. To assess whether load is problematic, apply the formula `4 × CPU cores`. If the load average exceeds this threshold, investigate further. Sustained high CPU load is a stronger indicator of a problem than high CPU usage alone. Tools such as `sysstat` can collect and analyze these metrics over time.

This section also displays **server uptime** — the total time the server has been running since its last restart.

## **Section 3: Process information**

When you notice high resource utilization, review the process list to identify the source. Per the `htop` man page, the default columns are:

* **PID** — The process ID
* **USER** — The process owner's username, or user ID if the name can't be determined
* **PRI** — The kernel's internal priority for the process, usually the NI value plus 20; differs for real-time processes
* **NI** — The nice value, from 19 (low priority) to -20 (high priority); only root can lower this value
* **VIRT** — Total program size in memory
* **RES** — Resident set size: the text and data sections plus stack usage
* **SHR** — Size of the process's shared pages
* **S** — Process state:
  * **S** — Sleeping (idle)
  * **R** — Running
  * **D** — Disk sleep (uninterruptible)
  * **Z** — Zombie (waiting for parent to read its exit status)
  * **T** — Traced or suspended (for example, by SIGTSTP)
  * **W** — Paging
* **CPU%** — Percentage of CPU time the process currently uses
* **MEM%** — Percentage of memory the process currently uses, based on its resident memory size
* **TIME+** — Total time the process has spent in user and system time, measured in clock ticks
* **Command** — Full path of the command

## **Section 4: Keyboard shortcuts**

The shortcuts displayed at the bottom of the htop interface are self-explanatory, configurable, and mouse-clickable.

## **Further reading**

For full details on htop's configuration options, flags, and features, consult the man page:
`man htop`

The project's documentation is also available at [https://htop.dev/](https://htop.dev/).


# I cannot install the pywin32 package on Domino using pip install
Source: https://docs.domino.ai/knowledge-base/i-cannot-install-the-pywin32-package-on-domino-using-pip-install

pywin32 cannot be installed on Domino because it is a Windows-only package and Domino workspaces run on Linux. Use Linux-compatible alternatives such as python-xlib, subprocess, psutil, or DB-API libraries depending on your use case.

Version/Environment (if relevant):
All Domino deployments running Linux-based workspaces.

## Issue:

Running `pip install pywin32` fails with the following error:

Raw code

`ERROR: Could not find a version that satisfies the requirement pywin32 (from versions: none)
ERROR: No matching distribution found for pywin32
Note: you may need to restart the kernel to use updated packages.`

## Root Cause:

`pywin32` provides Python bindings exclusively for the Windows API. Domino workspaces run on Linux, where the following Windows-specific dependencies do not exist:

* Windows API — system calls, GUI manipulation, and registry operations
* COM (Component Object Model) objects and Active Directory services
* WMI (Windows Management Instrumentation) components
* DLLs (Dynamic Link Libraries) and other Windows system resources

## Resolution:

Use one of the following Linux-compatible libraries as an alternative. Install the one that matches your use case.

python-xlib — GUI and window management Interact with the Linux X Window System, including windows, events, and input devices.

Raw code

`pip install python-xlib`

subprocess — run command-line tools Run Linux command-line tools and utilities from within your Python script. `subprocess` is part of the Python standard library — no installation required.

Raw code

`import subprocess
result = subprocess.run(['ls', '-l'], capture_output=True, text=True)
print(result.stdout)`

psutil — system and process monitoring Retrieve and monitor system resources such as CPU, memory, disk usage, and network data.

Raw code

`pip install psutil`

DB-API — database connectivity Connect Python to databases using a standardized interface. Use a database-specific library for your platform:

Raw code

\`# PostgreSQL
pip install psycopg2

# MySQL

pip install mysql-connector-python

# Oracle Database

pip install cx\_Oracle\`

For more information about `pywin32`, see the [PyPI project page](https://pypi.org/project/pywin32/).

### Notes/Information:

* Domino Support focuses on Domino's core product. For environment issues, contact your local admins as a first step.

### References/Internal Records:

* Public KB Articles:
  * [I Cannot Install the pywin32 Package on Domino Using pip install (KB-000001194)](https://support.domino.ai/support/s/article/I-cannot-install-the-pywin32-package-on-Domino-using-pip-install-1718868031408)


# 415 Unsupported Media Type Error When Calling the Domino REST API
Source: https://docs.domino.ai/knowledge-base/i-keep-getting-a-415-error-unsupported-media-type-when-i-call-domino-s-rest-api-what-am-i-doing-wrong

A 415 Unsupported Media Type error from the Domino REST API is caused by a missing or incorrect Content-Type header. Add Content-Type: application/json to your request headers to resolve.

### **Version/Environment (if relevant):**

All Domino Versions

## **Issue:**

A user receives a **415 Unsupported Media Type** error when calling the Domino REST API.

## **Root Cause:**

The API request is missing the Content-Type header, or it is set to an unsupported value. The Domino REST API requires requests to specify application/json as the content type.

## **Resolution:**

Add the Content-Type header to your API request:

```
Content-Type': 'application/json'
```

Since the header should also include your API key, the full header section will look like this:

```
headers = {
'Content-Type': 'application/json',
'X-Domino-Api-Key': '<api key>'
}
```

If you continue to experience issues, use [Postman](https://www.postman.com/) to test your API calls. Postman is a free application that allows you to test different media types and headers with minimal overhead.

Additional API examples can be found here: [API Examples for Domino's 5.x REST API](https://support.domino.ai/support/s/article/API-Examples-for-Domino-s-5-x-REST-API-1718868032320).

### **Notes/Information:**


# Increasing the timeout for Shiny Server
Source: https://docs.domino.ai/knowledge-base/increasing-the-timeout-for-shiny-server



When you access a published Shiny app, the app times out after 60 seconds of inactivity by default. Use one of the two methods below to increase this timeout.

## Prerequisites

Before applying the non-Pro workaround, confirm you have:

* A published Shiny app with separate UI and server files
* Access to edit and republish the app in Domino

## Method 1: Shiny Server Pro

If your team has a Shiny Server Pro license, you can configure application timeouts directly within your Domino installation. For configuration details, see the application timeouts section in the Shiny Server Pro documentation at [https://docs.posit.co/shiny-server/#application-timeouts.](https://docs.posit.co/shiny-server/#application-timeouts.)

If you have a license and need assistance setting this up, contact Domino support.

## Method 2: WebSocket keep-alive workaround (non-Pro)

This workaround sends a ping through the WebSocket every 15 seconds to keep the connection open, preventing the default 60-second inactivity timeout.

### Step 1: Add the keep-alive script to your UI file

Add the following code block to your UI file:

```
tags$head(
    HTML(
     "
     <script>
     var socket_timeout_interval
     var n = 0
     $(document).on('shiny:connected', function(event) {
     socket_timeout_interval = setInterval(function(){
     Shiny.onInputChange('count', n++)
     }, 15000)
     });
     $(document).on('shiny:disconnected', function(event) {
     clearInterval(socket_timeout_interval)
     });
     </script>
     "
    )
)
```

### Step 2: Add the output element to your UI file

Anywhere after the script added in Step 1, add the following line to the same UI file:

```
textOutput("keepAlive")
```

### Step 3: Add the render function to your server file

In your server file, add:

```
output$keepAlive <- renderText({
   req(input$count)
   paste("keep alive ", input$count)
 })
```

### Step 4: Verify the changes

Republish your Shiny app and confirm it loads without errors. To verify the new timeout is in effect, leave the app idle for longer than 60 seconds and confirm the session remains active.

## Still timing out?

If the workaround above doesn't resolve the issue, see [Shiny app shows a gray screen after some time or times out](https://support.domino.ai/support/s/article/Shiny-App-shows-a-gray-screen-after-sometime-or-timeouts-1718868032640) .


# Installing Packages And Dependencies
Source: https://docs.domino.ai/knowledge-base/installing-packages-and-dependencies



# Installing packages and dependencies

Domino is pre-installed with many common libraries, so we recommend that you first try running your code before customizing your configuration. If you need to, there are specific ways to add your own dependencies for...

* [Python](https://tickets.dominodatalab.com/hc/en-us/articles/28352932497044-Installing-Packages-And-Dependencies#python)
* [R](https://tickets.dominodatalab.com/hc/en-us/articles/28352932497044-Installing-Packages-And-Dependencies#r)

If you have more specialized dependencies, please [let us know](https://support.domino.ai/support/s).

# Persistent installations

By default, Domino runs each of your scripts (or interactive sessions) from a fresh environment. I.e., any packages you install, will install each time. We have more advanced functionality for managing your own compute environments with software changes that stay permanently installed. Learn more about [custom environments](https://docs.dominodatalab.com/en/latest/user_guide/e46d54/customize-environments/).

# Python

## Checking what's already installed

Many common modules are installed by default. To get a list of pre-installed modules, you can include help('modules') at the start of your script or in a new IPython Notebook session. Or using the Domino CLI tool, you can run: domino run --direct "pip freeze"

## Adding your own packages

To specify your own additional dependencies, you can use '[pip](http://www.pip-installer.org/en/latest/)' with Domino. To specify module dependencies for your project, add a '[pip requirements file](https://pip.pypa.io/en/latest/user_guide/#requirements-files)' named 'requirements.txt' to the root of your project folder.

The requirements file specifies which libraries and any version requirements for them. An example:

```
pandas
lxml==3.2.3
numpy>=1.7.1
```

For a full reference on the syntax of the requirements file, read [this](https://pip.pypa.io/en/latest/user_guide/#requirements-files).

If you're using pip on your local machine, the easiest way to generate the 'requirements.txt' file is to run the following command in the root of your project folder:

```
~/domino/myProject $ pip freeze > requirements.txt
```

For performance reasons, you should prune that file so that it includes only the libraries you need for your actual analysis.

Alternatively, if you're working in a Jupyter Notebook, you can also use pip to install dependencies interactively. In a notebook cell, you can run

```
! pip install --user <package>
```

(The '!' tells the notebook to execute the cell as a shell command)

## Installing Packages Hosted from a Git Repository

**Warning:** this is an advanced topic!

Pip can install Python packages from source by cloning a public Git repository over https. For full reference, see [here](https://docs.readthedocs.io/en/stable/guides/private-python-packages.html). To specify this, you will need to add something like the following line to your 'requirements.txt' file:

```
-e git+https://git.yourproject.org/you/Project.git#egg=YourProject
```

The most common host of Git projects is GitHub. If the package you wish to install is publicly accessible, then the instructions above will work. However, if you need to install any private repositories, Domino can securely integrate with GitHub to access those private repositories.

**Never embed your GitHub credentials** directly in the requirements.txt file. Instead, integrate Domino with GitHub securely.

# R

If you're using Domino for R scripts, you may also want to check out our R Package for controlling Domino in your R IDE.

Most common packages are installed by default (you can include "installed.packages()" at the start of your script to print out a list of installed packages). If you need additional packages, you can use R's built-in package manager to install and load them: simply add "install.packages" calls for any packages your code needs to the top of your R scripts, e.g.,

```
install.packages("rpart", dependencies=TRUE, repos='http://cran.us.r-project.org')
# install other packages...

library('rpart')

# rest of your script...
```

These package installation calls must be at the top of **every** R script that Domino runs. If your packages take a long time to install, we recommend that you create a [custom compute environment](https://docs.dominodatalab.com/en/latest/user_guide/e46d54/customize-environments/) for efficiency.


# Installing Archived R Packages No Longer Available on CRAN
Source: https://docs.domino.ai/knowledge-base/installing-r-packages-from-archive

R packages removed from CRAN can be installed from the archive. Download the .tar.gz directly and use install.packages(pkgs=pkgFile, type="source", repos=NULL) to install.

### **Version/Environment (if relevant):**

All Domino versions

## **Issue:**

RStudio CRAN repository has removed a number of packages, which are now only available in the CRAN archive at [https://cran.r-project.org/src/contrib/Archive/](https://cran.r-project.org/src/contrib/Archive/). When attempting to install such a package using the standard `install.packages()` command, R returns a warning that the package is no longer available. For example:

```
> install.packages('RSQLServer')
Installing package into "/usr/local/lib/R/site-library"
(as "lib" is unspecified)
Warning in install.packages :
package "RSQLServer" is not available for this version of R

A version of this package for your version of R might be available elsewhere,
see the ideas at
https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installing-packages
```

## **Root Cause:**

The package has been removed from the active CRAN repository and is no longer returned by the standard `install.packages()` function. The package still exists in the CRAN archive but must be installed directly from the archive URL using a source-based install method.

## **Resolution:**

Install the package directly from the CRAN archive using the following code (replace the URL and filename with the specific package version needed):

```
> url <- "https://cran.r-project.org/src/contrib/Archive/RSQLServer/RSQLServer_0.3.0.tar.gz"
> pkgFile <- "RSQLServer_0.3.0.tar.gz"
> download.file(url = url, destfile = pkgFile)
trying URL 'https://cran.r-project.org/src/contrib/Archive/RSQLServer/RSQLServer_0.3.0.tar.gz'
Content type 'application/x-gzip' length 633887 bytes (619 KB)
==================================================
downloaded 619 KB

> install.packages(pkgs=pkgFile, type="source", repos=NULL)
Installing package into "/usr/local/lib/R/site-library"
(as "lib" is unspecified)
* installing *source* package "RSQLServer" ...
** package "RSQLServer" successfully unpacked and MD5 sums checked
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (RSQLServer)
> library("RSQLServer")
```

### **Notes/Information:**

The full CRAN archive is available at [https://cran.r-project.org/src/contrib/Archive/](https://cran.r-project.org/src/contrib/Archive/). Browse to the package directory to find available versions and their corresponding `.tar.gz` download URLs.


# Invalid Client Credentials When Generating A Service Account Token
Source: https://docs.domino.ai/knowledge-base/invalid-client-credential-when-generating-service-account-token



## Overview

When creating a service account in Domino, you may encounter the following error while generating a token:

```
{"message":"Request unauthorized: {\"error\":\"unauthorized_client\",\"error_description\":\"Invalid client or Invalid client credentials\"}","success":false}
```

This typically indicates a mismatch between the **client secret** stored in Kubernetes and the one configured in **Keycloak** for the `domino-service-accounts` client.

## Symptoms

* Service account creation succeeds, but token generation fails.
* Error response shows `unauthorized_client` or `Invalid client credentials`.
* Nucleus frontend logs display `OidcUnauthorizedException` related to the `domino-service-accounts` client.

```
2024-11-12T09:32:09.048739313-05:00 domino.helpers.oidc.OidcUnauthorizedException: Request unauthorized: {"error":"unauthorized_client","error_description":"Invalid client or Invalid client credentials"}
```

* Issue may appear after making changes to Keycloak configuration (e.g., adding or modifying JWT keys).

## Root Cause

The `domino-service-accounts` client credentials in Keycloak and the Kubernetes secret may become out of sync. Since **Keycloak client secrets cannot be directly edited**, and **Kubernetes secrets are auto-generated**, reconciliation is required to realign both values.

## How to Verify Credentials

1. **Fetch values from the cluster:**

```
# Get clientId
kubectl -n domino-platform get secret nucleus-domino-service-accounts-client -o jsonpath='{.data.clientId}' | base64 -d; echo

# Get clientSecret
kubectl -n domino-platform get secret nucleus-domino-service-accounts-client -o jsonpath='{.data.clientSecret}' | base64 -d; echo
```

* The `clientId` should be `domino-service-accounts`.
* Note down the `clientSecret`.

**2. Check Keycloak configuration:**

* Log in to [Keycloak](https://docs.dominodatalab.com/en/latest/admin_guide/1a7dfa/set-up-keycloak/#tr1).
* Go to: **Domino Realm > Clients > domino-service-accounts > Credentials**.
* Reveal the **Client Secret** and compare it with the one from Kubernetes.

!

If they **do not match**, reconciliation is needed.

## Reconciliation Procedure

Since neither Keycloak nor Kubernetes allows direct editing of these secrets, the recommended approach is to **re-import the `domino-service-accounts` client** into Keycloak:

**1. Export the client configuration:**

* In Keycloak UI, go to:
  **Domino Realm > Clients > domino-service-accounts > Action > Export**.

!

* Save the JSON file.

**2. Update the exported file:**

* Locate the `"secret"` property in the JSON.
* Replace its value with the `clientSecret` obtained from Kubernetes.

```
"secret": "your-cluster-secret-value"
```

!

**3. Delete the old client:**

* In Keycloak UI, navigate to **Domino Realm > Clients > domino-service-accounts > Action > Delete**.

**4. Re-import the updated client:**

* Go to **Domino Realm > Clients > Clients List > Import Client**.
* Upload the modified JSON file.

!

* Click **Save**

**5. Verify the reconciliation:**

* Navigate back to **Credentials** for the `domino-service-accounts` client.
* Confirm that the secret matches the value in Kubernetes.
* Retry generating the service account token.

!

## Important Notes

* **Avoid** **toggling** the **Client Authentication** setting in Keycloak unnecessarily. Doing so may regenerate the secret and cause mismatches.
* If new JWT keys are required (e.g., for S3 connectivity), ensure existing Domino-related client configurations are not modified.
* After reconciliation, service account token generation should work again without impacting other Keycloak configurations.

## Related Documentation

* [Domino Service Accounts Guide](https://docs.dominodatalab.com/en/cloud/admin_guide/6921e5/manage-domino-service-accounts/#_what_are_domino_service_accounts)


# "Invalid host" error when trying to connect MiniO server as a Generic S3 data source
Source: https://docs.domino.ai/knowledge-base/invalid-host-error-when-trying-to-connect-minio-server-as-a-generic-s3-data-source

After a Chrome update, Chromium-based browsers rejected  incomplete SSL certificate chain with a NET::ERR_CERT_COMMON_NAME_INVALID error. Fix involved converting the .p7b chain file to PEM using OpenSSL and re-importing it into AWS Certificate Manager

**Version/Environment (if relevant):**

Applies to all version of Domino after browser update.

## Issue:

After a Chrome update, Chromium-based browsers rejected the SSL certificate chain for the deployment, making it unreachable. The following error was displayed in the browser:

```
NET::ERR_CERT_COMMON_NAME_INVALID
```

## Root Cause:

Per AWS documentation, NET::ERR\_CERT\_COMMON\_NAME\_INVALID on an Application Load Balancer can have the following causes:

* The domain name used in the HTTPS request does not match the alternate name specified in the listener's associated AWS Certificate Manager (ACM) certificate.
* The load balancer's default DNS name is being used — public certificates cannot be issued for the \*.amazonaws.com domain.

This error occurs when the browser cannot verify a website's SSL certificate and is unable to establish a secure connection. It typically results from a misconfigured or incomplete certificate chain on the server.

Per MinIO docs:

* [https://min.io/docs/minio/linux/index.html](https://min.io/docs/minio/linux/index.html):

> While the port 9000 is used for connecting to the API, MinIO automatically redirects browser access to the MinIO Console.

* [https://min.io/docs/minio/linux/administration/minio-console.html#static-vs-dynamic-port-assignment](https://min.io/docs/minio/linux/administration/minio-console.html#static-vs-dynamic-port-assignment)

> respond to S3 API operations on the default MinIO server port :9000 and browser access on the MinIO Console port :9090

## **Resolution:**

In the **Host** field, replace the port number with **9000**:

**Step 1: Extract the private key and certificate body from the .pfx file**

If you need to obtain the private key and certificate body from a .pfx file, run the following commands:

```
openssl pkcs12 -in domino.p12 -nocerts -nodes -out privatekey.pem
openssl pkcs12 -in domino.p12 -clcerts -nokeys -out certificate.pem
```

The first command outputs the private key to privatekey.pem. The second outputs the certificate body to certificate.pem.

**Step 2: Convert the certificate chain file to PEM format**

Run the following command to convert the .p7b file to PEM format:

```
openssl pkcs7 -inform der -in DSNYCA_Chain-7-28-23.p7b -print_certs -out certificates.pem
```

This outputs the full certificate chain to certificates.pem.

**Step 3: Re-import the certificate into ACM**

1. Open the **AWS Certificate Manager (ACM)** console.
2. Locate the existing certificate and select **Reimport certificate**.
3. Paste the contents of certificates.pem into the **Certificate chain** field. Supply the certificate body and private key from Step 1 if prompted.
4. Complete the import.

## **Notes/Information:**

If changing the port to 9000 doesn't resolve the issue, please open a Support ticket and send in the following output for the datasource-proxy pod. This requires the Admin to have kubectl access:

1. Data Source Pod Status: kubectl -n `<platform-namespace>` get po --selector=app.kubernetes.io/name=datasource-proxy
2. Data Source Logs: kubectl -n `<platform-namespace>` logs --selector=app.kubernetes.io/name=datasource-proxy -c datasource-proxy
3. Data Source Proxy Pod describe: kubectl describe pod -n domino-platform `<datasource-proxy-name>`

Depending on the issue, Support may also ask for:

1. Data Source Proxy Deployment: kubectl get deployment -n domino-platform datasource-proxy -o yaml
2. Data Source Proxy Configure Map: kubectl get configmap -n domino-platform datasource-proxy -o yaml
3. Data Source Proxy Service: kubectl get service datasource-proxy -n domino-platform -o yaml

Since the certificate was originally imported into ACM manually, ACM provides the option to re-import it with an updated certificate chain.

### References/Internal Records:

* New York Dept-Sanitation Zendesk ticket #80270


# javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
Source: https://docs.domino.ai/knowledge-base/javax-net-ssl-sslexception-unsupported-or-unrecognized-ssl-message



#### **Version/Environment (if relevant):**

Not version specific

## **Issue:**

After specifying new SMTP settings for email notifications in Central Config a customer failed to receive any notifications. To find more detail we triggered a quick job and tailed the nucleus-dispatcher log. We found an exception occurring. The "caused by" provided a clue:

```
Caused by: javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
at java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(SSLSocketInputRecord.java:451)
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:175)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:110)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1418)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1324)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411)
at com.sun.mail.util.SocketFetcher.configureSSLSocket(SocketFetcher.java:626)
at com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:400)
at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:238)
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2175)
... 22 common frames omitted
```

## **Root Cause:**

`javax.net.ssl.SSLException: Unsupported or unrecognized SSL message` can occur if protocols are incorrect. For example if your specify a port that is **NOT** SSL but you chose settings that trying to connect via SSL, this can occur. It is the equivalent of trying to browse https protocol on a webserver port that is only listening for http protocol.

## **Resolution:**

You need to correct either the port or the protocol. Double-check your values for `com.cerebro.domino.email.smtp.ssl` and `com.cerebro.domino.email.channel` and ensure they're aligned with the mailserver's specifications on the port you've designated.

For example if your mailserver administrator told you to use port 25 then you probably need `com.cerebro.domino.email.smtp.ssl=false` rather than true since 25 is usually is not SSL-based.

If your mailserver requires TLS then use:
`com.cerebro.domino.email.smtp.ssl`: false
`com.cerebro.domino.email.smtp.channel`: starttls
and your port would likely be 587 (but confirm with your mailserver admin).

## **Notes/Information:**

Tail dispatcher with timestamps:
kubectl logs deployment.apps/nucleus-dispatcher -n `<your namespace>` --container=nucleus-frontend --follow --timestamps=true


# pip install throws SSL certificate error
Source: https://docs.domino.ai/knowledge-base/pip-install-throws-ssl-certificate-error

A pip install SSL certificate verification failure is typically caused by a firewall or proxy stripping SSL certificates, and can be resolved by using the --trusted-host flag either inline or persistently via pip.conf.

# pip install throws SSL certificate error

Version: Not specific to Domino version

## Issue

A `pip install` fails with:

`connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed`

## Root cause

By default, pip doesn't use system certificates. For details, see the [pip HTTPS certificates documentation](https://pip.pypa.io/en/stable/topics/https-certificates/). This error can occur when you're behind a firewall or proxy that strips the SSL certificate.

## Resolution

Use the `--trusted-host` flag to bypass certificate verification for the affected hosts.

**Option 1 — CLI flag**

Pass `--trusted-host` directly to the install command. Replace `<package>` with your package name:

`pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org <package>`

**Option 2 — pip.conf (persistent)**

Add the following to your `pip.conf` file to apply the setting to all pip commands.

The location of `pip.conf` varies by environment:

* **Linux/macOS (user-level):** `~/.config/pip/pip.conf`
* **Linux/macOS (virtual environment):** `<venv>/pip.conf`
* **Windows (user-level):** `%APPDATA%\pip\pip.ini`

```
[global]
trusted-host = pypi.python.org
               pypi.org
               files.pythonhosted.org
```

## Notes

This error can also appear in Dockerfile instructions used in Domino Compute Environments. The same `--trusted-host` workaround applies.

Add `--trusted-host` for each host your `RUN pip install` command accesses. For example:

```
USER root
RUN pip install --trusted-host <hostname> --trusted-host <cdn-hostname> <package-or-url>
```


# Pods stuck in ContainerCreating with an event mentioning errno 524
Source: https://docs.domino.ai/knowledge-base/pods-stuck-in-containercreating-with-an-event-mentioning-errno-524



#### ***Version/Environment (if relevant):***

All versions on AWS/EKS

## **Issue:**

Customers find Pods stuck in ContainerCreating and/or liveness probes fail with an event in the kubectl describe which mentions **`errno 524`**:

```
Warning FailedCreatePodSandBox 4m9s (x384 over 89m) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524: unknown
```

##

## **Root Cause:**

There is a linux kernel memory leak bug in the EKS optimized AMI based on linux kernel version 5.10.x. The linux bug can be verified by using *`cat /proc/sys/net/core/bpf_jit_limit`* and seeing what the limit currently is and by using *`cat /proc/vmallocinfo | grep bpf_jit | awk '{s+=$2} END {print s}'`*, to see if the limit has been exceeded.

Example:

```
[root@ip-10-0-35-23 /]# cat /proc/sys/net/core/bpf_jit_limit
264241152
[root@ip-10-0-35-23 /]# cat /proc/vmallocinfo | grep bpf_jit | awk '{s+=$2} END {print s}'
352337920
```

More details: [https://github.com/awslabs/amazon-eks-ami/issues/1219#issuecomment-1533797378](https://github.com/awslabs/amazon-eks-ami/issues/1219#issuecomment-1533797378)

"This issue is more likely to be encountered with kernel versions *`kernel-5.10.176-157.645.amzn2`thru*`kernel-5.10.177-158.645.amzn` where the rate of the memory leak is higher."

***

## **Resolution:**

AWS corrected the problem by patching by a new AMI [Release AMI Release v20230501 awslabs/amazon-eks-ami](https://github.com/awslabs/amazon-eks-ami/releases/tag/v20230501), released May 3rd 2023.

Temporary workaround is to run ***`sysctl net.core.bpf_jit_limit=452534528`*** on all **platform** nodes by ssh'ing directly onto them.

The permanent fix is to get your EKS to use this AMI: [Release AMI Release v20230501 awslabs/amazon-eks-ami](https://github.com/awslabs/amazon-eks-ami/releases/tag/v20230501)

## **Notes/Information:**

Note the *"temporary workaround"* is short-term only, since the kernel bug can continue leaking.


# Preventing Jupyter timeout errors when running notebooks as a batch/scheduled job
Source: https://docs.domino.ai/knowledge-base/preventing-jupyter-timeout-errors-when-running-notebooks-as-a-batch-scheduled-job



If you are running a Jupyter notebook as a batch job, and it contains a long-running cell, you may encounter the following error:

```
[NbConvertApp] ERROR | Timeout waiting for execute reply (30s). If your cell should take longer than this, you can increase the timeout with: c.ExecutePreprocessor.timeout = SECONDS in jupyter_nbconvert_config.py
```

This indicates that the cell caused Jupyter to timeout. You can avoid this by configuring a longer timeout. Here's how to do it in the pre-setup script of a [custom environment](https://docs.dominodatalab.com/en/latest/reference/environments/Environment_management.html) definition:

```
echo "c.ExecutePreprocessor.timeout = $SECONDS" >> /home/ubuntu/.ipython/profile_default/ipython_nbconvert_config.py
chown ubuntu:ubuntu /home/ubuntu/.ipython/profile_default/ipython_nbconvert_config.py
```

Substitute an appropriate value for **\$SECONDS**. Note the *chown* command - this is needed because the pre-setup script runs as the root user, so the file ownership must be changed back to the ubuntu user in order to avoid a permissions error during your run. Keep in mind this is for batch runs.

**Note:** later versions of Jupyter require you to modify a different config file *(`/home/ubuntu/.jupyter/jupyter_nbconvert_config.py`* instead of `/home/ubuntu/.ipython/profile_default/ipython_nbconvert_config.py`).

If you are running a recent Jupyter version and the above commands do not work, use this instead:

```
echo "c.ExecutePreprocessor.timeout = $SECONDS" >> /home/ubuntu/.jupyter/jupyter_nbconvert_config.py
chown ubuntu:ubuntu /home/ubuntu/.jupyter/jupyter_nbconvert_config.py
```


# Resolve internal (on-premise) DNS Server in AWS EKS via CoreDNS
Source: https://docs.domino.ai/knowledge-base/resolve-internal-on-premise-dns-server-in-aws-eks-via-coredns

Configure CoreDNS on AWS EKS and Domino environments to resolve on-premise DNS hostnames from Domino Workspaces via a VPN tunnel.

# Resolve internal (on-premise) DNS server in AWS EKS via CoreDNS

Domino users often need to access on-premise services from their Domino workspace. This article explains how to configure Amazon Web Services (AWS) Elastic Kubernetes Service (EKS) CoreDNS to forward DNS queries to an on-premise DNS server, then update a Domino environment to resolve the corresponding search domain.

This configuration assumes you have already established a VPN tunnel or equivalent network connectivity between AWS and your on-premise network.

## Prerequisites

* CoreDNS running on your EKS cluster (default since EKS v1.12). If CoreDNS isn't installed, follow the [AWS CoreDNS documentation](https://docs.aws.amazon.com/eks/latest/userguide/coredns.html) to install it.
* `kubectl` access to the EKS cluster running your Domino deployment. Update your local kubeconfig or connect directly in AWS before proceeding.

## Update the CoreDNS configmap

1. Open the CoreDNS configmap for editing:

   ```
   kubectl -n kube-system edit configmap coredns
   ```
2. Locate the `Corefile` section. It looks similar to this:

   ```
   Corefile: |
     .:53 {
       log
       errors
       health
       kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        upstream
        fallthrough in-addr.arpa ip6.arpa
       }
       prometheus :9153
       forward . /etc/resolv.conf
       cache 30
       loop
       reload
       loadbalance
     }
   ```
3. Add a new forwarding block after the closing `}` of the existing stanza. Replace `google.com` with your on-premise domain and `8.8.8.8` with the IP address of your DNS server (for example, the load balancer IP for your VPN tunnel). Keep the period (`.`) between `forward` and the IP address.

   The updated `Corefile` section should look like this:

   ```
   Corefile: |
     .:53 {
       log
       errors
       health
       kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        upstream
        fallthrough in-addr.arpa ip6.arpa
       }
       prometheus :9153
       forward . /etc/resolv.conf
       cache 30
       loop
       reload
       loadbalance
     }
     google.com:53 {
      forward . 8.8.8.8
     }
   ```
4. Save the file. The editor displays a confirmation message when the configmap saves successfully.
5. Restart the CoreDNS pods to apply the change immediately:

   ```
   kubectl get pods -n kube-system -oname |grep coredns |xargs kubectl delete -n kube-system
   ```
6. Verify that the CoreDNS pods return to a **Running** status:

   ```
   kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
   ```

   All listed pods should show `Running` in the **STATUS** column before you proceed.

## Configure the Domino environment

Add a pre-setup script to the Domino environment. This script appends the search domain to `/etc/resolv.conf` inside each workspace container.

1. In Domino, navigate to **Environments** and select the environment you want to update.
2. Click **Edit Definition**.
3. Scroll down and expand the **Advanced** section.
4. Enter the following script in the **Pre Setup Script** field, replacing `google.com` with your on-premise domain:

   ```
   cat /etc/resolv.conf >> /tmp/resolv.conf.tmp
   sed -i '2s/$/ google.com/' /tmp/resolv.conf.tmp
   cat /tmp/resolv.conf.tmp > /etc/resolv.conf
   ```
5. Save and build the environment.

## Verify DNS resolution

1. Start a new Jupyter workspace and open a terminal.
2. Run an `nslookup` for a hostname in your domain. Using the Google example:

   ```
   nslookup maps
   ```
3. Confirm that the output resolves the short hostname to a fully qualified domain name (FQDN). Expected output:

   ```
   ubuntu@run-5ea9d22e84e7430006080fdb-7bfkr:/mnt$ nslookup maps
   Server:     10.100.0.10
   Address:    10.100.0.10#53

   Non-authoritative answer:
   Name:  maps.google.com
   Address: 172.217.14.206
   Name:  maps.google.com
   Address: 2607:f8b0:400a:808::200e
   ```

   Successful output shows the short name resolving to the FQDN (for example, `maps` → `maps.google.com`), confirming that on-premise DNS resolution is working


# Scope of Support Coverage
Source: https://docs.domino.ai/knowledge-base/scope-of-support-coverage



## **Scope of Service**

The following information relates to the Support services Domino bundles with our customer/user licenses, at no additional cost, via Domino Support's Customer Portal ([https://support.domino.ai](https://support.domino.ai) and [support@dominodatalab.com](mailto:support@dominodatalab.com)).

Domino Support provides world-class Technical Support and an outstanding customer experience by ensuring timely, thorough resolution to the issues we investigate. We maintain a constant feedback loop with consumers of our services, acting quickly and frequently on their input to achieve continuous improvement.

Domino Support's area of expertise is in core Domino products. We maintain very high standards for customer satisfaction and responsiveness when investigating product-related issues on behalf of our customers. Please see the information below for details.

Domino Support provides support for:

* Usage of our core products
* Diagnosis of issues within our core products
* Configuration of core product features

Domino Support does not provide expertise related to:

* Modified installers or deployment packages
* Building custom environments and dependency resolution
* Product customizations or code development
* System or network design (for systems and networks not owned by Domino)
  * Customers on Domino Cloud may escalate issues to Domino Cloud Operations through Domino Support
* Implementation of development or security policies/practices
* Undocumented product features or functionality
* Beta releases, release candidates, or feature previews
* Training modules
* Untested platforms
* Installation, upgrades, or migration of Domino
* Disaster recovery:
  * This includes data recovery exercises and data migration

This applies to the commercial releases of Domino products, including Domino Workbench and Domino Model Monitor

### **Domino APIs: Scope of Coverage**

In addition to the information above, the following information applies to Domino Support's service coverage for our APIs (example: REST, Model API).

Domino Support offers the following assistance for APIs:

* Confirmation that Domino's proprietary methods and inputs return results as expected, based on our product documentation

Domino Support does not provide assistance for:

* Customers' scripts, code, application, customized system configuration, or issues that result from conflicts therein


# Impact of Secure Identity Propagation Feature Flag on Domino Apps
Source: https://docs.domino.ai/knowledge-base/secureidentitypropagationtoappsenabled-feature-flag-impacts-app-behavior

In Domino 6.1.1+, SecureIdentityPropagationToAppsEnabled defaults to true, breaking apps on restart if they manually construct requests_pathname_prefix. Fix by replacing the manual path with os.environ.get("DOMINO_RUN_HOST_PATH").

# Apps Break When SecureIdentityPropagationToAppsEnabled Is True

## Issue

A Domino App stops working after it is restarted on a deployment where the SecureIdentityPropagationToAppsEnabled feature flag is set to true. The app may have been functioning correctly before the restart or before the flag was enabled. The most common symptom is a routing or 404 error caused by a stale, manually constructed requests\_pathname\_prefix path that no longer matches the new routing structure.

**Typical symptoms include:**

* App loads a blank page or returns a 404/502 after restart
* Dash (Plotly) app fails to serve assets or callbacks
* The app was working before the flag was enabled or before the last restart
* The app code manually constructs the path prefix from DOMINO\_PROJECT\_OWNER, DOMINO\_PROJECT\_NAME, and DOMINO\_RUN\_ID environment variables

## Cause

When SecureIdentityPropagationToAppsEnabled is true, Domino changes the URL routing structure used to proxy requests to a running app. Apps that manually construct the requests\_pathname\_prefix from individual environment variables (DOMINO\_PROJECT\_OWNER, DOMINO\_PROJECT\_NAME, DOMINO\_RUN\_ID) produce a path that no longer matches the new routing structure, causing request routing to fail.

An app that is **already running** when the feature flag is enabled will continue to work until it is restarted. The breakage occurs on the next restart because the new routing structure takes effect at that point.

The legacy path construction pattern that causes the issue:

```
user    = os.environ.get("DOMINO_PROJECT_OWNER")
project = os.environ.get("DOMINO_PROJECT_NAME")
run_id  = os.environ.get("DOMINO_RUN_ID")

requests_pathname_prefix = f"/{user}/{project}/r/notebookSession/{run_id}/"
```

## Resolution

Replace the manually constructed path prefix with the Domino-provided DOMINO\_RUN\_HOST\_PATH environment variable. This variable is automatically set by Domino and always reflects the correct routing path for the current session, regardless of whether SecureIdentityPropagationToAppsEnabled is true or false.

### Step 1 — Locate the path prefix in your app code

Search your app's entry point for any reference to requests\_pathname\_prefix that manually constructs the path from environment variables, for example:

```
user    = os.environ.get("DOMINO_PROJECT_OWNER")
project = os.environ.get("DOMINO_PROJECT_NAME")
run_id  = os.environ.get("DOMINO_RUN_ID")
runurl  = f'/{user}/{project}/r/notebookSession/{run_id}/'

app = Dash(__name__, routes_pathname_prefix='/', requests_pathname_prefix=runurl)
```

### Step 2 — Replace with DOMINO\_RUN\_HOST\_PATH

Update the code to read the path directly from the Domino-provided environment variable:

```
runurl = os.environ.get("DOMINO_RUN_HOST_PATH")

app = Dash(__name__, routes_pathname_prefix='/', requests_pathname_prefix=runurl)
```

### Step 3 — Restart the app

Publish the updated code to your Domino project and restart the app. Verify that the app loads correctly and that callbacks and assets resolve without errors.

## Additional Notes

**Important:** If SecureIdentityPropagationToAppsEnabled is set to false and you need to allow anonymous (unauthenticated) access to Domino Apps, ensure the Central Config parameter com.cerebro.domino.launchpad.allowPublicModelProducts is also set to true. See KB article [000002243](https://support.domino.ai/support/s/article/We-are-no-longer-seeing-the-option-to-allow-anonymous-users-to-view-apps) for details.

* **Domino Cloud / 6.1.1+:** The flag defaults to true. All new and existing apps should be updated to use DOMINO\_RUN\_HOST\_PATH.
* **Domino 6.0.x (self-managed):** The flag defaults to false. No immediate action is required, but migrating to DOMINO\_RUN\_HOST\_PATH is strongly recommended to avoid breakage during future upgrades.
* **Non-Dash frameworks:** Any Python web framework (Streamlit, Flask, etc.) that accepts an explicit base path or prefix should also be updated to read from DOMINO\_RUN\_HOST\_PATH.

## Related Articles

* [000002243 — Anonymous user access to Domino Apps](https://support.domino.ai/support/s/article/We-are-no-longer-seeing-the-option-to-allow-anonymous-users-to-view-apps)
* [000001814 — Feature Flags in Domino](https://support.domino.ai/support/s/article/Feature-flags-1718868051921)
* [000002017 — Changing a specific user's feature flag](https://support.domino.ai/support/s/article/Changing-a-specific-user-s-feature-flag-1724943001691)


# Configuring and Sending Emails in Domino Using Python smtplib
Source: https://docs.domino.ai/knowledge-base/sending-emails-python-smtplib

Guide to sending emails from Domino using Python's smtplib. Covers secure SMTP setup via SMTPS or STARTTLS, sending plain-text emails, and HTML-formatted emails using MIME.

## Issue

Users need guidance on how to send emails from Domino using Python's `smtplib` library, including how to set up a secure SMTP connection, send plain-text emails, and send HTML-formatted emails using MIME.

## Root cause

Sending emails from Domino requires configuring a secure SMTP connection and using the correct Python libraries. Without proper setup of connection security, authentication, and MIME formatting, emails may fail to send or render incorrectly.

## Resolution

This article covers three topics:

* Setting up a secure SMTP connection using SMTPS or STARTTLS
* Sending plain-text emails using `smtplib`
* Sending HTML-formatted emails using the `email.mime` module

### 1. Setting up a local SMTP debug server

To test your code locally, use the Python `smtpd` binary bound to port 1025. Messages will not leave localhost and will be output to stdout as a byte stream:

```bash theme={null}
python -m smtpd -c DebuggingServer -n localhost:1025
```

### 2. Starting a secure SMTP connection

There are two ways to secure your SMTP connection. Both examples use Gmail's SMTP server and Python's `ssl` package to load the system's default trusted CA bundles via `create_default_context()`.

**Option A: SMTPS (secured from the outset)**

```python theme={null}
# Using SMTPS to secure the connection from the outset
import smtplib, ssl

port = 465  # For SSL
password = input("Input password and press enter: ")

# Create a secure SSL context, loads system's default trusted CAs
context = ssl.create_default_context()

with smtplib.SMTP_SSL("smtp.gmail.com", port, context=context) as server:
    server.login("my@gmail.com", password)
```

**Option B: STARTTLS (connection secured after initial handshake)**

```python theme={null}
# Using STARTTLS to secure the connection from a point onwards
import smtplib, ssl

smtp_server = "smtp.gmail.com"
port = 587  # For STARTTLS
sender_email = "my@gmail.com"  # Enter your address
password = input("Input password and press enter: ")

# Create a secure SSL context, loads system's default trusted CAs
context = ssl.create_default_context()

try:
    server = smtplib.SMTP(smtp_server, port)
    server.ehlo()  # Can be omitted, called implicitly by the method if required
    server.starttls(context=context)  # Secure the connection
    server.ehlo()  # Can be omitted, called implicitly by the method if required
    server.login(sender_email, password)
    # TODO: Send email here
except Exception as e:
    # Print any error messages to stdout
    print(e)
finally:
    server.quit()
```

### 3. Sending a plain-text email

Define the sender, receiver, and message body at the top of your script. Use `server.sendmail` to send the email.

**Using SMTPS:**

```python theme={null}
import smtplib, ssl

port = 465  # For SSL
smtp_server = "smtp.gmail.com"
sender_email = "my@gmail.com"  # Enter your address
receiver_email = "your@gmail.com"  # Enter receiver address
password = input("Input password and press enter: ")
message = """\
Subject: Hi there

Message body after 2 new lines."""

context = ssl.create_default_context()
with smtplib.SMTP_SSL(smtp_server, port, context=context) as server:
    server.login(sender_email, password)
    server.sendmail(sender_email, receiver_email, message)
```

**Using STARTTLS:**

```python theme={null}
import smtplib, ssl

port = 587  # For STARTTLS
smtp_server = "smtp.gmail.com"
sender_email = "my@gmail.com"  # Enter your address
receiver_email = "your@gmail.com"  # Enter receiver address
password = input("Input password and press enter: ")
message = """\
Subject: Hi there

Message body after 2 new lines."""

context = ssl.create_default_context()
with smtplib.SMTP(smtp_server, port) as server:
    server.ehlo()  # Can be omitted, called implicitly by the method if required
    server.starttls(context=context)
    server.ehlo()  # Can be omitted, called implicitly by the method if required
    server.login(sender_email, password)
    server.sendmail(sender_email, receiver_email, message)
```

### 4. Sending HTML-formatted emails using MIME

Use the `email.mime` module to send emails combining HTML and plain-text content. Since email clients render the last MIME part first, the HTML part must be added after the plain-text part.

```python theme={null}
import smtplib, ssl
from email.mime.text import MIMEText  # Contains the HTML and plain-text content
from email.mime.multipart import MIMEMultipart  # Combines the two

sender_email = "my@gmail.com"  # Enter your address
receiver_email = "your@gmail.com"  # Enter receiver address
password = input("Input password and press enter: ")

message = MIMEMultipart("alternative")
message["Subject"] = "multipart test"
message["From"] = sender_email
message["To"] = receiver_email

# Create the plain-text and HTML version of your message
text = """\
Hi,
How are you?"""
html = """\
<html>
  <body>
    <p>Hi,<br>
      How are you?<br>
    </p>
  </body>
</html>
"""

# Convert these into MIMEText objects
text1 = MIMEText(text, "plain")
html1 = MIMEText(html, "html")

# Add the objects to the MIMEMultipart object
# The email client renders the last part first, hence we add the HTML part last
message.attach(text1)
message.attach(html1)

# Create secure connection with server and send email
context = ssl.create_default_context()
with smtplib.SMTP_SSL("smtp.gmail.com", 465, context=context) as server:
    server.login(sender_email, password)
    server.sendmail(
        sender_email, receiver_email, message.as_string()
    )
```

## Notes

When configuring for a production Domino deployment (assumed to use AWS SES as the SMTP server), the following values are required:

* **SMTP host:** The server name from the SMTP settings page, accessible via the AWS SES service page.
* **SMTP username and password:** Generate these from the SMTP settings page using the **Create My SMTP Credentials** button.
* These values should be available in Domino Central Config. If not, they must be added as Central Config key-value pairs. Contact your internal Domino admin for assistance.
* The code examples in this article are provided as a reference. Some adjustments and indentation corrections may be needed to suit your specific application.


# Spark Executor Pods Unable to Find Kerberos Credentials Cache
Source: https://docs.domino.ai/knowledge-base/spark-on-domino-klist-no-credentials-cache-found-error-when-trying-to-use-kerberos-to-authenticate-into-sql-server

Kerberos klist: No credentials cache found on Spark executors. Fix: set KRB5CCNAME and KRB5_CONFIG via ENV in the Dockerfile, run kinit in Pre-Run, and concatenate all extraJavaOptions in one line.

### ***Version/Environment (if relevant):***

Applies to all versions of Domino.

## **Issue:**

When using Kerberos authentication with PySpark in a Domino Spark cluster to connect to a SQL Server, the following errors are encountered:

```
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1) (12.34.567.89 executor 0): com.microsoft.sqlserver.jdbc.SQLServerException: Integrated authentication failed. ClientConnectionId:1234d56a-78c9-1234-beda-eedc1234b567
```

The [Spark configuration](https://docs.dominodatalab.com/en/latest/user_guide/f11f6a/work-with-your-cluster/#_spark_executors_versus_workers) has been set at both the code level and on the Spark Web UI's Project Integration page. Running `kinit` in the Spark driver (workspace) succeeds, and the token is visible via `klist` in the workspace logs:

```
I have no name!@spark-123456cf7c89123b456fa789-spark-worker-2:/tmp$ klist
Ticket cache: FILE:/tmp/krb5cc_12345
Default principal: sys-data@AD.COMPANY.COM

Valid starting   Expires       Service principal
03/29/23 15:28:44  03/29/23 23:28:44  krbtgt/AD.COMPANY.COM@AD.COMPANY.COM
   renew until 04/03/23 15:28:44
```

However, `klist` fails on the executor Spark worker pods with the following error:

```
I have no name!@spark-1234567f1c23456b789faed1-spark-worker-0:/opt/bitnami/spark$ klist
klist: No credentials cache found (filename: /tmp/krb5cc_12345)
```

## **Root Cause:**

There are two root causes:

1. Setting environment variables via `spark.executorEnv.ENVNAME` does not work. Environment variables must be set using `ENV` in the cluster environment's Dockerfile.
2. When setting multiple `extraJavaOptions` parameters with one line per option, only the last parameter is applied at runtime — each new line overwrites the previous one.

## **Resolution:**

**1. Dockerfile Instructions**

Set the following environment variables in the executor cluster environment's Dockerfile:

* `ENV KRB5CCNAME=FILE:/mnt/data/workspace_name/krb5cc_12345`
* `ENV KRB5_CONFIG=/mnt/data/workspace_name/krb5.conf`

This changes the default `ccache` directory to a shared directory accessible by both the driver and executors, allowing executors to read the `ccache` generated by the driver during pre-run. Setting `KRB5CCNAME` via `spark.executorEnv.KRB5CCNAME` or any other method will not work — executors will always fall back to the default location `/tmp/krb5cc_12345`. Use `ENV` instead of `ARG` when setting `KRB5_CONFIG`.

**2. Pre-Run Script**

Run `kinit` in the Spark driver's environment Pre-Run script to generate and cache the Kerberos token on the dataset so that executors can use it for authentication.

**3. `extraJavaOptions` Parameters**

Under **Project Settings > Integrations > Apache Spark mode > Spark Configuration Options**, setting multiple `extraJavaOptions` with one line per parameter will not work, as each line overwrites the previous:

```
spark_conf.set("spark.executor.extraJavaOptions","-Djavax.security.auth.useSubjectCredsOnly=false")
spark_conf.set("spark.executor.extraJavaOptions","-Djava.security.auth.login.config=/mnt/data/workspace_name/SQLJDBCDriver2.conf")
spark_conf.set("spark.executor.extraJavaOptions","-Djava.security.krb5.conf=/mnt/data/workspace_name/krb5.conf")
spark_conf.set("spark.executor.extraJavaOptions","-Dsun.security.krb5.debug=true")
![Project_settings_-_Integrations_-_Apache_Spark_mode.png](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr&ids=068UY00000Hru0q&d=%2Fa%2FUY0000018p1k%2FT0zoitoAOVJ0hrugfPbV4jh50.msvIGPLAufFbN3Yto&asPdf=false)
```

Instead, concatenate all `spark.driver.extraJavaOptions` parameters into a single line, separated by spaces:

```
spark_conf.set("spark.executor.extraJavaOptions","-Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.auth.login.config=/mnt/data/workspace_name/SQLJDBCDriver2.conf -Djava.security.krb5.conf=/mnt/data/workspace_name/krb5.conf -Dsun.security.krb5.debug=true")
```

![Spark\_Web\_UI\_-Environment-Spark\_Properties-\_all\_params.png](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr\&ids=068UY00000Hru0s\&d=%2Fa%2FUY0000018p1m%2FT8Et6o60_ob2jCkG.RVRJ9ufONxIQFEYytdAYaz6AMM\&asPdf=false)

You can verify all parameters are correctly applied by checking **Spark Web UI > Environment > Spark Properties**.

![Spark\_Web\_UI\_-Env-Spark\_Properties-\_only\_last\_param.png](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr\&ids=068UY00000Hru0r\&d=%2Fa%2FUY0000018p1l%2FrjC1BtmO6uZkRsyiSCYp8yG8pzZs6_0.3q_rqBYq8FQ\&asPdf=false)

### **Notes/Information:**

Additional resources:

* [Apache Spark 3.0 — Kerberos ticket cache](https://spark.apache.org/docs/3.0.0-preview/security.html#using-a-ticket-cache): By setting `spark.kerberos.renewal.credentials` to `ccache`, the local Kerberos ticket cache will be used for authentication. The location can be customized via the `KRB5CCNAME` environment variable.
* [Spark on Domino — Configure Prerequisites](https://docs.dominodatalab.com/en/latest/user_guide/1962f3/configure-prerequisites/): Configure PySpark compute environments for workspaces and/or jobs connecting to your cluster.
* [Domino Kerberos Authentication](https://docs.dominodatalab.com/en/latest/user_guide/ed5f5a/kerberos-authentication/): Domino supports keytab-based Kerberos authentication at the project or user level. Once configured, runs will automatically execute `kinit` and retrieve the ticket for authentication.

### **References/Internal Records:**


# Domino Support Service Information
Source: https://docs.domino.ai/knowledge-base/support-policies-overview

What's covered under Domino Support, and what isn't.

What's covered under Domino Support, and what isn't.

## Domino Support Policies and Procedures

* [Scope of Support Coverage](/knowledge-base/scope-of-support-coverage)
* [Domino Product Version Lifecycle and Support Policies](/knowledge-base/domino-product-version-lifecycle-and-support-policies)


# The lsof utility - 'list open files'
Source: https://docs.domino.ai/knowledge-base/the-lsof-utility-list-open-files

Troubleshoot open files, network ports, and process locks on Linux using lsof basic and advanced command examples.

**Version/Environment (if relevant):** Applicable to any Linux-based system including Domino deployments.

## Issue:

Administrators and engineers need to identify processes holding file locks, investigate socket activity, or troubleshoot process ID (PID) conflicts on a Linux-based system.

## Root Cause:

This is a reference article for the `lsof` utility.

## Resolution:

### What is lsof?

`lsof` lists open files on Linux-based systems. It displays information about file descriptors and the processes that hold them open, including network connections and listening ports. Use `lsof` to identify processes holding file locks, investigate socket activity, and troubleshoot process ID (PID) conflicts.

> ⚠️ **Prerequisite:** `lsof` may require privilege elevation to access files belonging to other users. Prepend `sudo` to any command when running as a non-root user.

**Basic usage**

Use the following commands to troubleshoot open files on a Linux system:

Find all open files for a user:

```
lsof -u <username>
```

Find all open files associated with a particular process using a PID:

```
lsof -p <PID>
```

List and locate all open socket files:

```
lsof +D / -a -d sock
```

Locate all open files listening on a specific port:

```
lsof -i :<port number>
```

Identify all open files linked to an application by program name:

```
lsof -c <program name>
```

**Advanced usage**

Use these commands for more targeted investigation across processes, file descriptors, and time intervals:

Display the process name, PID, and file name for all open files at a set time interval:

```
lsof -n -r <time interval> +L -F pn -F f
```

Locate all open files for a specific process at a set time interval and sort by PID:

```
lsof -p <PID> -n -r <time interval> +L -s
```

Find all open files and the command line that opened them:

```
lsof -c <command name> -a -c <command name> +L1
```

Find all open files linked to a specific file descriptor:

```
lsof -a -d <file descriptor>
```

### Notes/Information:

* `lsof` can also help identify processes blocking a filesystem unmount. Running `lsof <filesystem_name>` will display all open files on the named filesystem.
* `lsof +L1` can be used to find unlinked open files on local disks — files that have been deleted but are still held open by a process, which can cause unexpected disk usage.


# Troubleshooting
Source: https://docs.domino.ai/knowledge-base/troubleshooting-overview

Fixes for common errors across environments, workspaces, models, and infrastructure.

Fixes for common errors across environments, workspaces, models, and infrastructure.

## Administration

* [javax.net.ssl.SSLException: Unsupported or unrecognized SSL message](/knowledge-base/javax-net-ssl-sslexception-unsupported-or-unrecognized-ssl-message)

## Apps

* [Impact of Secure Identity Propagation Feature Flag on Domino Apps](/knowledge-base/secureidentitypropagationtoappsenabled-feature-flag-impacts-app-behavior)

## Authentication

* [Unexpected error when authenticating with identity provider (duplicate key constraint)](/knowledge-base/unexpected-error-when-authenticating-with-identity-provider-duplicate-key-value-violates-unique-constraint-constraint-40)
* [Invalid Client Credentials When Generating A Service Account Token](/knowledge-base/invalid-client-credential-when-generating-service-account-token)

## Data Sources & Datasets

* [Database Locking](/knowledge-base/database-locking)
* ["Invalid host" error connecting MiniO server as a Generic S3 data source](/knowledge-base/invalid-host-error-when-trying-to-connect-minio-server-as-a-generic-s3-data-source)

## Environments

* [Preventing the permission denied error when running pip install](/knowledge-base/how-to-prevent-the-permission-denied-error-when-running-pip-install-in-a-workspace)
* [Build error for nvidia GPG key](/knowledge-base/build-error-for-nvidia-gpg-key)
* [Docker Layers: failed to register layer, max depth exceeded](/knowledge-base/docker-layers-failed-to-register-layer-max-depth-exceeded)
* [pip install throws SSL certificate error](/knowledge-base/pip-install-throws-ssl-certificate-error)
* [Unable to install R package: package not available for your version of R](/knowledge-base/unable-to-install-r-package-error-package-is-not-available-for-your-version-of-r)
* [Yarn Package invalid key](/knowledge-base/yarn-package-invalid-key)
* [Could not install packages: EnvironmentError Errno 13 Permission denied](/knowledge-base/error-could-not-install-packages-due-to-an-environmenterror-errno-13-permission-denied)
* [Cannot install the pywin32 package using pip install](/knowledge-base/i-cannot-install-the-pywin32-package-on-domino-using-pip-install)
* [Environment build fails: method driver /usr/lib/apt/methods/https not found](/knowledge-base/environment-build-fails-with-the-method-driver-usr-lib-apt-methods-https-could-not-be-found)
* [Can Not Connect to KeyServer Caused by GPG Error](/knowledge-base/can-not-connect-to-keyserver-caused-by-gpg-error)
* [How to use the apt-get install command in environments](/knowledge-base/how-to-use-the-apt-get-install-command-in-environments)

## Git Integration

* [Clearing Git Cache](/knowledge-base/clearing-git-cache)

## Infrastructure and Kubernetes

* [FailedMount event: efs.csi.aws.com not found in registered CSI drivers](/knowledge-base/failedmount-event-efs-csi-aws-com-not-found-in-the-list-of-registered-csi-drivers)
* [Pods stuck in ContainerCreating with an event mentioning errno 524](/knowledge-base/pods-stuck-in-containercreating-with-an-event-mentioning-errno-524)
* [Spark Executor Pods Unable to Find Kerberos Credentials Cache](/knowledge-base/spark-on-domino-klist-no-credentials-cache-found-error-when-trying-to-use-kerberos-to-authenticate-into-sql-server)
* [vm\_memory\_high\_watermark error in RabbitMQ service logs](/knowledge-base/vm-memory-high-watermark-error-in-rabbitmq-service-logs)
* [How to Troubleshoot Kubernetes Autoscaling (ASG) Issues](/knowledge-base/how-to-troubleshoot-autoscaling-asg-issues)
* [Failed to pull EKS pause image from ECR](/knowledge-base/failed-to-pull-image-602401143452-dkr-ecr-us-east-1-amazonaws-com-eks-pause-3-5)

## Models

* [Unable to deploy Model: Errno 13 Permission denied /.cache](/knowledge-base/unable-to-deploy-model-errno-13-permission-denied-cache)
* [uwsgi error loading shared libraries: libpcre.so.1 not found](/knowledge-base/uwsgi-error-while-loading-shared-libraries-libpcre-so-1-cannot-open-shared-object-file-no-such-file-or-directory)
* [Model build errors: failed to establish connection to prometheus-client](/knowledge-base/errors-during-model-build-failed-to-establish-a-new-connection-errno-2-name-or-service-not-known-simple-prometheus-client)

## REST API

* [415 Unsupported Media Type Error When Calling the Domino REST API](/knowledge-base/i-keep-getting-a-415-error-unsupported-media-type-when-i-call-domino-s-rest-api-what-am-i-doing-wrong)

## Workspaces

* [VSCode Error: ENOSPC, system limit for file watchers reached](/knowledge-base/error-enospc-system-limit-for-number-of-file-watchers-reached)
* [VSCode Failed to get latest version: getaddrinfo ENOTFOUND](/knowledge-base/vscode-failed-to-get-latest-version-error-getaddrinfo-enotfound-api-github-com)
* [Why did my execution fail with "No space left on device"?](/knowledge-base/why-did-my-execution-fail-with-the-error-no-space-left-on-device)


# Unable to deploy Model - [Errno 13] Permission denied: '/.cache'
Source: https://docs.domino.ai/knowledge-base/unable-to-deploy-model-errno-13-permission-denied-cache



## Issue

When deploying a model as a Model API, the build completes successfully but the model fails to start, with the following error:

```
Jan 05 2023 14:21:05 -0500 There was a problem when trying to write in your cache folder (/.cache/huggingface/hub). You should set the environment variable TRANSFORMERS_CACHE to a writable directory.
Jan 05 2023 14:21:05 -0500 PermissionError: [Errno 13] Permission denied: '/.cache'
```

The model's Python code uses the Transformers package. By default, Transformers sets its cache directory to `/.cache`, which may not be writable in the Model API environment.

## Solution

Override the default cache path by setting `TRANSFORMERS_CACHE` to a writable directory.

**Python:**

```
import os
os.environ['TRANSFORMERS_CACHE'] = '/path/to/writable/cache/'
```

**Bash:**

```
export TRANSFORMERS_CACHE=/path/to/writable/cache/
```

Replace `/path/to/writable/cache/` with a directory your model environment can write to, such as a path under your project or home directory.

## Set HF\_HOME for broader cache coverage

The `TRANSFORMERS_CACHE` variable controls only the Transformers cache. The Hugging Face (HF) `HF_HOME` variable controls a broader set of caches, including the datasets cache. Set `HF_HOME` to cover both:

```
export HF_HOME=/path/to/writable/cache/
```

To also protect against cases where the environment variable isn't set, create a symlink from the default cache location to your writable directory:

```
ln -s /path/to/writable/cache/ ~/.cache/huggingface
```

### How the default cache directory is determined

The Transformers documentation describes [cache setup](https://huggingface.co/docs/transformers/installation?highlight=transformers_cache#cache-setup) and the order of priority for environment variables:

> Pretrained models are downloaded and locally cached at: `~/.cache/huggingface/transformers/`. This is the default directory given by the shell environment variable `TRANSFORMERS_CACHE`. On Windows, the default directory is given by `C:\Users\username\.cache\huggingface\transformers`. You can change the shell environment variables shown below — in order of priority — to specify a different cache directory:
>
> 1. Shell environment variable (default): `TRANSFORMERS_CACHE`
> 2. Shell environment variable: `HF_HOME` + `transformers/`
> 3. Shell environment variable: `XDG_CACHE_HOME` + `/huggingface/transformers`

`HF_HOME` defaults to `$XDG_CACHE_HOME/huggingface`. If `XDG_CACHE_HOME` is not set, it defaults to `$HOME/.cache`, making the usual default `$HOME/.cache/huggingface`.


# Unable to install R package | Error : Package is not available for your version of R
Source: https://docs.domino.ai/knowledge-base/unable-to-install-r-package-error-package-is-not-available-for-your-version-of-r

Fix the "package is not available for this version of R" CRAN error by specifying an alternate mirror inline or updating your default mirror in .Rprofile or rsession.conf.

## **Issue**

When building an environment or installing a package inside a workspace, R raises an error similar to the following:

```
Warning in install.packages :
  package 'renv' is not available for this version of R

Warning in install.packages :
  unable to access index for repository https://cran.us.r-project.org/bin/...:
  cannot open URL 'https://cran.us.r-project.org/bin/.../PACKAGES'
```

The package isn't available from the default Comprehensive R Archive Network (CRAN) mirror ([https://cran.us.r-project.org/](https://cran.us.r-project.org/)) configured for this environment. The same package may be available from a different CRAN mirror.

## **Quick fix: specify a different CRAN mirror inline**

Pass a different repo argument directly to install.packages():

`install.packages("renv", repo = "https://cran.r-project.org/")`

This overrides the default mirror for this one installation call without changing any configuration files.

## **Permanent fix: update the default CRAN mirror**

To avoid specifying the `repo` argument every time, update the default CRAN mirror using one of the options below.

### **Option 1: edit `rsession.conf` in your compute environment**

Add the r-cran-repos setting to your rsession.conf file and point it to the correct CRAN mirror. For example:

`r-cran-repos=https://cran.r-project.org/`

Locate rsession.conf in your compute environment's R session configuration directory.

### **Option 2: edit .Rprofile or Rprofile.site**

Set the default CRAN mirror in your .Rprofile or Rprofile.site file. Find both files under your R home directory. Add the following:

`options(repos = c(CRAN = "https://cran.r-project.org/"))`

* .Rprofile — applies settings for your user session only

* Rprofile.site — applies settings for all users on the system

After saving either file, restart your R session for the changes to take effect.


# Unexpected error when authenticating with identity provider: duplicate key value violates unique constraint "constraint_40"
Source: https://docs.domino.ai/knowledge-base/unexpected-error-when-authenticating-with-identity-provider-duplicate-key-value-violates-unique-constraint-constraint-40

Resolve the Keycloak "duplicate key value violates unique constraint constraint_40" SSO login error by removing and re-linking the affected user's Identity Provider record.

**Applies to:** All versions through 6.x with Single Sign-On (SSO) (such as Okta, WebSSO, etc.)

## **Issue:**

An individual user attempts to log in and sees "Unexpected error when authenticating with Identity provider":

Clearing the browser cache and cookies or switching to a different browser does not resolve the error. Search for `exception` in the Keycloak logs using the following command:

```
kubectl logs -n domino-platform keycloakpodname --timestamps | grep -i exception
```

The logs return an entry similar to this:

```
2023-04-25T13:06:37.247217108Z ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (default task-3858) ERROR: duplicate key value violates unique constraint "constraint_40"
2023-04-25T13:06:37.247219975Z Detail: Key (identity_provider, user_id)=(domino-credentials, 39a47782-8c3e-46d1-9091-a7105ff795ea) already exists.
```

## **Root Cause:**

Domino requests that Keycloak verify authentication against the SSO provider. Based on the SSO response, Keycloak attempts to create a new user record for the combination of **Provider Username** and email address.

Keycloak creates a new record when it detects a change in the casing of the username, email address, or another attribute. For example, if the SSO Security Assertion Markup Language (SAML) response returns an email address with an uppercase letter, Keycloak treats it as a new record.

The duplicate key violation occurs because the username or **Provider User ID** already exists in the database.

## **Resolution:**

1. In the Keycloak Admin Console, open the individual user record.
2. On the **Identity Provider Links** tab, select **Remove**.
3. Have the user re-attempt login through the Domino UI.

### **Notes**

To understand why Keycloak is treating the user as a new record, review the SAML requests and responses exchanged between Keycloak and the SSO provider.


# Use Gradio in a Jupyter workspace
Source: https://docs.domino.ai/knowledge-base/use-gradio-in-a-jupyter-workspace



#### **Version/Environment:**

This applies to all versions of Domino.

## **Issue:**

[Gradio's "Hello, World" demo](https://pypi.org/project/gradio/) fails to render in a Jupyter workspace and/or a browser window.

## **Root Cause:**

There is a bug in the Gradio package which strips the trailing slash from URLs being loaded in-line, i.e. in the notebook.

## **Resolution:**

Download the attached example Jupyter notebook to get started using Gradio in Domino. This runs on the current Domino Standard Environment (DSE) and DSE with GPU images. The app renders both in the Jupyter workspace and in a browser tab. In this line, replace the URL demo2.dominodatalab.com with your deployment's URL (without https\://):

* host = os.environ.get('GRADIO\_PROXY\_HOST') or "demo2.dominodatalab.com"

Please note that you'll need access to jupyter-server-proxy >=3.2.2 and access to the Pypi repositories to download the Gradio packages.

* To check the version in a Jupyter Terminal, run pip list | grep -i jupyter-server-proxy

Please note "show\_api=False" is not supported in newer version of Gradio 6.x, you will need to remove it from launch,

```
# Launch gradio app in the Domino workspace (behind proxy)
        interface.launch(root_path=application_path, server_name=server, server_port=port, quiet=True, inline=False, share=False)
```

### References/Internal Records:

* PTE-666


# uwsgi: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory
Source: https://docs.domino.ai/knowledge-base/uwsgi-error-while-loading-shared-libraries-libpcre-so-1-cannot-open-shared-object-file-no-such-file-or-directory

Fix the uWSGI libpcre.so.1 missing shared library error in Domino model deployments using ldd diagnostics and symlink or LD_LIBRARY_PATH remediation steps.

# uWSGI error: cannot open shared object file libpcre.so.1

## Error

`uwsgi: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory`

## Cause

This error occurs in model builds when uWSGI can't find one of its dynamically linked libraries. The model builds successfully, but fails to start. Instance logs show the uWSGI error.

## Solution

Use the `ldd` (list dynamic dependencies) command to identify the missing library and its expected location, then create a symlink or update the library path to resolve it.

### Step 1: Reproduce the error in a workspace

`$ uwsgi
uwsgi: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory`

### Step 2: Locate the uWSGI binary

`$ which uwsgi
/usr/local/anaconda/bin/uwsgi`

### Step 3: Identify the missing library

Run `ldd` against the uWSGI binary to see all dynamically linked libraries:

`$ ldd /usr/local/anaconda/bin/uwsgi
linux-vdso.so.1 (0x00007ffe9d5ec000)
libpthread.so.0 (0x00007f95e83ad000)
libm.so.6 (0x00007f95e800f000)
libdl.so.2 (0x00007f95e7e0b000)
libz.so.1 (0x00007f95e7bee000)
libpcre.so.1 => not found
libuuid.so.1 (0x00007f95e79e7000)
libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f95e777d000)
libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f95e7305000)
libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f95e6f44000)
libutil.so.1 (0x00007f95e6d41000)
libcrypt.so.1 (0x00007f95e6b09000)
libc.so.6 (0x00007f95e6718000)
/lib64/ld-linux-x86-64.so.2 (0x00007f95e85cc000)
libicuuc.so.60 => /usr/lib/x86_64-linux-gnu/libicuuc.so.60 (0x00007f95e6361000)
liblzma.so.5 (0x00007f95e613b000)
libicudata.so.60 => /usr/lib/x86_64-linux-gnu/libicudata.so.60 (0x00007f95e4592000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f95e4209000)
libgcc_s.so.1 (0x00007f95e3ff1000)`

Two things to note in this output:

* `libpcre.so.1` shows `not found`
* Most resolved libraries point to `/usr/lib/x86_64-linux-gnu/`, which is where uWSGI searches for its dependencies

Confirm the search path by checking the `LD_LIBRARY_PATH` environment variable:

`$ echo $LD_LIBRARY_PATH
/usr/lib/x86_64-linux-gnu/:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/opt/oracle/instantclient_12_1:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/opt/oracle/instantclient_12_1:`

### Step 4: Resolve the missing library

Choose one of the following options:

1. **Create a symlink from an existing versioned library in `/usr/lib/x86_64-linux-gnu/`.** If `libpcre.so.3` exists there, run:

   `ln -s /usr/lib/x86_64-linux-gnu/libpcre.so.3 /usr/lib/x86_64-linux-gnu/libpcre.so.1`
2. **Create a symlink from a library found elsewhere on the system.** Use `find` to locate available copies:

   `$ find / -name libpcre.so.1
   /usr/local/anaconda/pkgs/pcre-8.43-he6710b0_0/lib/libpcre.so.1
   /usr/local/anaconda/pkgs/pcre-8.44-he6710b0_0/lib/libpcre.so.1
   /usr/local/anaconda/lib/libpcre.so.1
   /libpcre.so`

   Then link one of those paths into the expected directory:

   `ln -s /usr/local/anaconda/lib/libpcre.so.1 /usr/lib/x86_64-linux-gnu/libpcre.so.1`
3. **Update `LD_LIBRARY_PATH`** to include the directory containing `libpcre.so.1`:

   `export LD_LIBRARY_PATH=/usr/local/anaconda/lib/:$LD_LIBRARY_PATH`

### Step 5: Verify the fix

1. Re-run `ldd` and confirm `libpcre.so.1` now resolves to a path instead of showing `not found`:

   `$ ldd /usr/local/anaconda/bin/uwsgi
   linux-vdso.so.1 (0x00007ffffc181000)
   libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f30d6d35000)
   libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f30d6997000)
   libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f30d6793000)
   libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f30d6576000)
   libpcre.so.1 => /lib/x86_64-linux-gnu/libpcre.so.1 (0x00007f30d6304000)
   libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f30d60fd000)
   libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f30d5e93000)
   libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f30d5a1b000)
   libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f30d565a000)
   libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f30d5457000)
   libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f30d521f000)
   libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f30d4e2e000)
   /lib64/ld-linux-x86-64.so.2 (0x00007f30d6f54000)
   libicuuc.so.60 => /usr/lib/x86_64-linux-gnu/libicuuc.so.60 (0x00007f30d4a77000)
   liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f30d4851000)
   libicudata.so.60 => /usr/lib/x86_64-linux-gnu/libicudata.so.60 (0x00007f30d2ca8000)
   libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f30d291f000)
   libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f30d2707000)`

   `libpcre.so.1` now resolves to `/lib/x86_64-linux-gnu/libpcre.so.1`.
2. Start uWSGI and confirm it runs without the shared library error:

   `$ uwsgi
   Starting uWSGI 2.0.19.1 (64bit) on [Fri Oct 9 22:49:20 2020] ***
   compiled with version: 4.8.5 on 09 October 2020 19:29:50
   os: Linux-4.4.0-1075-aws #85-Ubuntu SMP Thu Jan 17 17:15:12 UTC 2019
   nodename: domino-run-5f80e2df4cedfd0008f5d7d4
   machine: x86_64
   clock source: unix
   pcre jit disabled
   detected number of CPU cores: 4
   current working directory: /lib/x86_64-linux-gnu
   detected binary path: /usr/local/anaconda/bin/uwsgi
   WARNING: you are running uWSGI without its master process manager ***
   your memory page size is 4096 bytes
   detected max file descriptor number: 4096
   lock engine: pthread robust mutexes
   thunder lock: disabled (you can enable it with --thunder-lock)
   The s/-socket option is missing and stdin is not a socket.`

   uWSGI starts successfully without the shared library error.

If this solution doesn't resolve your issue, please contact Domino Support.


# vm_memory_high_watermark" error in RabbitMQ service logs
Source: https://docs.domino.ai/knowledge-base/vm-memory-high-watermark-error-in-rabbitmq-service-logs

Resolve RabbitMQ vm_memory_high_watermark memory alarms in Domino on Kubernetes by restarting the affected pod or permanently raising the absolute memory threshold in the ConfigMap.

## **Issue:**

Examining the logs for the RabbitMQ StatefulSet/rabbitmq-ha-XXX shows the following error:

```
[info] <0.449.0> vm_memory_high_watermark set. Memory used:1858117632 allowed:1717986918
[warning] <0.447.0> memory resource limit alarm set on node 'rabbit@rabbitmq-ha-311-0.rabbitmq-ha-discovery.example.svc.cluster.local'.
[warning] <0.447.0>
[warning] <0.447.0> **********************************************************
[warning] <0.447.0> *** Publishers will be blocked until this alarm clears ***
[warning] <0.447.0> **********************************************************
[warning] <0.447.0>
[warning] <0.447.0> memory resource limit alarm cleared on node 'rabbit@rabbitmq-ha-311-0.rabbitmq-ha-discovery.example.svc.cluster.local'
[warning] <0.447.0> memory resource limit alarm cleared across the cluster
[warning] <0.447.0> memory resource limit alarm set on node 'rabbit@rabbitmq-ha-311-0.rabbitmq-ha-discovery.example.svc.cluster.local'.
```

This condition typically surfaces in the Domino frontend as an error similar to:

> "Workspace could not be started. Data plane Local is not in a healthy state: DataPlaneStatus(Disconnected, Not responding, Some(1700470196563))"

## **Root Cause:**

When RabbitMQ reaches the vm\_memory\_high\_watermark threshold, it sets a memory resource limit alarm on the affected node and stops accepting new connections. The data-plane-agent service detects its connections to RabbitMQ are refused and marks itself as unhealthy, which causes the frontend error above.

To confirm this in the logs, retrieve the RabbitMQ container logs using one of the following methods.

**Recent versions of Domino** — stream logs directly from the Kubernetes container:

```
kubectl logs -n <your-domino-platform-namespace> rabbitmq-ha-311-0 --follow
```

> **Note:** The pod name rabbitmq-ha-311-0 is illustrative. Replace it with the name of the affected pod in your environment.

**Older versions of Domino** — exec into the container and read the log file:

```
kubectl exec -it -n <your-domino-platform-namespace> rabbitmq-ha-311-0 -c rabbitmq-ha -- /bin/bash
cat /var/log/rabbitmq/*log
```

Look for vm\_memory\_high\_watermark set and memory resource limit alarm set entries to confirm the root cause.

## **Resolution:**

**Temporary fix**

Restart the affected RabbitMQ pod to clear the alarm and restore operations. Use the following command, replacing the pod name with the affected pod in your environment:

```
kubectl delete pod -n <your-domino-platform-namespace> rabbitmq-ha-311-0
```

The pod restarts automatically. However, memory usage typically increases over time and the alarm recurs without a permanent configuration change.

**Permanent fix**

Increase the vm\_memory\_high\_watermark setting in the rabbitmq-ha ConfigMap to set an absolute memory limit appropriate for your container's resource limits.

The default value is vm\_memory\_high\_watermark.relative = 0.4, which allocates 40% of available memory. This default protects against out-of-memory (OOM) situations. Where container memory limits allow, replacing the relative value with an explicit absolute value provides more predictable behavior. For example, on a container with a 6 GB memory limit, 3500MB is an acceptable threshold.

**Step 1: Edit the ConfigMap.**

```
kubectl edit cm -n <your-domino-platform-namespace> rabbitmq-ha
```

**Step 2: Replace the relative setting with an absolute value.**

```
## Memory Threshold
# vm_memory_high_watermark.relative = 0.4
vm_memory_high_watermark.absolute = 3500MB
```

**Step 3: Restart the RabbitMQ pods to apply the change.** Run the following command to perform a rolling restart of the StatefulSet:

```
kubectl rollout restart statefulset -n <your-domino-platform-namespace> rabbitmq-ha
```

**Step 4: Verify the new watermark is active.** After the pods restart, check the logs for confirmation that the updated value loaded:

```
kubectl logs -n <your-domino-platform-namespace> <rabbitmq-pod-name> | grep vm_memory_high_watermark
```

Confirm the output references the absolute value you set (for example, 3500MB) rather than the previous relative value.


# VSCode Failed to get latest version {"error":"getaddrinfo ENOTFOUND api.github.com"}
Source: https://docs.domino.ai/knowledge-base/vscode-failed-to-get-latest-version-error-getaddrinfo-enotfound-api-github-com

Explains VSCode "getaddrinfo ENOTFOUND api.github.com" and unknown IP log errors in air-gapped Domino environments, with settings to reduce network calls.

# VSCode error: "Failed to get latest version — getaddrinfo ENOTFOUND api.github.com"

When you run VSCode (code-server) in a closed or air-gapped environment, you may see error messages in your logs. These errors are benign — the Domino Standard Environment (DSE) doesn't rely on internet resources to function.

## Failed to get latest version

The error `Failed to get latest version {"error":"getaddrinfo ENOTFOUND api.github.com"}` occurs because code-server attempts to update itself on startup. You can see the relevant code in [code-server/update.ts](https://github.com/cdr/code-server/blob/e07ae004b73823483e9f7015548dc7d38334558a/src/node/update.ts#L64).

Apply the following network-related settings to suppress these update attempts.

### Apply the settings

1. Open `$DOMINO_WORKING_DIR/.vscode/User/settings.json` in your editor.
2. Add or merge the following configuration into the file:

   ```
   {
     "update.mode": "none",
     "extensions.autoUpdate": false,
     "workbench.enableExperiments": false,
     "extensions.autoCheckUpdates": false,
     "json.schemaDownload.enable": false,
     "update.showReleaseNotes": false,
     "npm.fetchOnlinePackageInfo": false,
     "workbench.settings.enableNaturalLanguageSearch": false
   }
   ```
3. Save the file and restart VSCode.

> **Note:** This workaround has not been verified in a fully air-gapped environment. Results may vary.

## Unknown IP addresses in logs

The IP addresses `142.250.179.176` and `216.58.214.16` appear in the logs because code-server tries to update the extensions that ship with the DSE. This behavior originates from the `downloadInstallableExtension` function in code-server.


# Why did my execution fail with "No space left on device"?
Source: https://docs.domino.ai/knowledge-base/why-did-my-execution-fail-with-the-error-no-space-left-on-device

Resolve 'No space left on device' execution failures in Domino by freeing disk space, moving files to Datasets, or increasing the allocated disk size (v4.4+).

**Applies to:** All Domino versions. Disk size configuration in project settings is available from Domino v4.4 and later. The default disk size is 10 GB for versions before v4.4 and 15 GB for v4.4 and later.

## **Issue**

An execution fails with the following error in the logs:

```
java.lang.IllegalStateException: Blob transfer (download) failed because 24 operations failed:
Download blob -> java.io.IOException: No space left on device
```

The execution has exceeded the available disk space on the compute instance.

## **Root Cause**

Each execution runs inside a compute instance with a finite amount of disk space. Total disk usage combines project files, external Git repository contents, the compute environment image, and any data generated during the run. When combined usage exceeds the available disk size, the execution fails.

Crash or dump files left behind by integrated development environments (IDEs) or libraries can also consume large amounts of disk space without being immediately visible.

## **Resolution**

### **For users**

1. **Switch to a different hardware tier.** This moves the execution to a different compute instance with available disk space. It's the quickest fix if you don't have admin privileges.
2. **Reduce project data.** Remove unnecessary files from the project. Git repository contents count toward disk usage.
3. **Move large files to Datasets.** Move large project files to a Domino Dataset rather than keeping them in the project directory.
4. **Check for crash or dump files.** Jupyter and other IDEs can crash and leave large files in the workspace that aren't immediately visible.
5. **Increase the disk size (v4.4 and later).** In your project, go to **Settings → Hardware & Environment → Workspace and Jobs Volume Size** and increase the allocated disk size. Changes apply to new workspaces only and don't affect currently running sessions.

### **For admins**

1. Place the affected executor into **Maintenance Mode** to take it out of service.
2. In the Domino Dispatcher UI, locate the executor and verify that no executions are currently running on it. Then terminate the executor to release the storage. Because executors are ephemeral, terminating one is a safe and fast resolution.

> **Note:** Garbage collection may also recover disk space on the executor without requiring termination, but termination is the fastest option. If this issue recurs, consider increasing the default disk size using the Central Config setting: com.cerebro.domino.computegrid.kubernetes.volume.volumesSizeInGB

### **Additional information**

**Domino v4.0 and later:** Each execution gets its own dedicated volume and is no longer affected by shared resource contention from other executions. Users on v4.4 and later can adjust their individual disk size in project settings.

**Executor maintenance mode:** Domino polls executors for disk status and automatically places them into Maintenance Mode when a partition has less than 10% free space. A common scenario is that the disk sits just above this threshold and an execution fills the remaining space. When this happens, new executions on that executor can fail repeatedly until an admin intervenes.

**Domino v3.6 and earlier:** On startup, Domino copies the project files and the compute environment Docker image to a partition on the executor. If this data exceeds the available disk space, the execution fails on startup. Long-lived executors can also accumulate cached project data and environment images over time, gradually filling the disk.


# Yarn Package invalid key
Source: https://docs.domino.ai/knowledge-base/yarn-package-invalid-key



### **Version/Environment (if relevant):**

Not Domino version specific. Applies to any Domino compute environment using the yarn package.

## **Issue:**

A Domino environment build fails with a GPG signature error related to the yarn package, similar to the following:

```
GPG error: https://dl.yarnpkg.com/debian stable InRelease: The following signatures
were invalid: EXPKEYSIG 23E7166788B63E1E Yarn Packaging <yarn@dan.cx>
```

## **Root Cause:**

The GPG key used to sign the yarn package repository has expired or become invalid. The environment's Dockerfile is using a stale key that can no longer be verified, causing the build to fail.

## **Resolution:**

Add one of the following to the top of the environment's Dockerfile to refresh the yarn package key. Try each option in order if the previous one does not work, as the correct approach may vary depending on your specific environment version.

**Option 1 — Standard key refresh (try this first):**

```
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
```

**Option 2 — Alternative using gpg dearmor:**

```
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg|gpg --dearmor|sudo tee /usr/share/keyrings/yarnkey.gpg>/dev/null
```

**Option 3 — Recommended if apt-key is deprecated in your environment:**

```
ENV YARNKEY=yarn-keyring.gpg
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo gpg --dearmour -o /usr/share/keyrings/$YARNKEY && \
 echo "deb [signed-by=/usr/share/keyrings/$YARNKEY] https://dl.yarnpkg.com/debian stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
```

### **Notes/Information:**

* These are best-effort suggestions that have been verified to work in a standard Domino environment. Results may vary depending on your specific environment configuration.
* `apt-key` is becoming deprecated in newer Linux distributions. If Options 1 and 2 do not work, use Option 3 which uses the modern `gpg --dearmour` approach with a signed-by keyring.
* If you are seeing a similar error for an Nvidia GPG key rather than yarn, refer to the related article: **Build error for Nvidia GPG key**.
* While Domino Support will assist where possible, environment build issues are primarily the responsibility of local admins. If these steps do not resolve the issue, the best first step is to engage your local environment administrators.