Model deployment administration

Once practitioners deploy a model, platform administration decides who receives Model Monitoring alerts, who can open Model Monitoring at all, how much compute Domino endpoints get, and which external accounts models can be deployed to.

Model Monitoring notifications

Users can send notifications to recipients about thresholds that are breached during scheduled checks. Domino sends those notifications by email, so the email server must be configured before any of them go out.

Set the email server configuration records

In the admin portal, go to Platform settings > Configuration records, then set the keys specified in Notifications for monitoring.

Restart services

Click here in the message at the top of the page to restart services.

Once the server is configured, users choose where notifications go:

Notification channels covers the channels available.
Set up notifications covers a specific Domino endpoint.
Set up notifications for a Model Monitor covers a Model Monitor.

Model Monitoring access control

Access to Domino Model Monitor is granted by membership of the dmm-users group in Keycloak. Use the Keycloak UI for a handful of users, and the access control script for a list of them.

Keycloak UI
Access control script

Open the admin UI at https://<deployment-url>/auth/admin and log in to Keycloak.
Select the DominoRealm realm from the dropdown:
Go to the Users tab, find the user (you may need to click View all users), and click edit.
Go to the Groups tab. To grant access, click Available Groups > dmm-users, then click Join. To revoke access, click Group Membership > dmm-users, then click Leave.
Repeat for each user.

If there is a high volume of users, adjust the Keycloak AccessToken settings in the UI so the script runs uninterrupted:

Log in to Keycloak.
Go to Realm Settings in the Master realm.
Click the Tokens tab.
Increase the Access Token Lifespan to something sufficiently large.
Save the changes, then run the script.

Undo the changes after the script finishes.

To use the Domino Model Monitor Access Control Script, set environment variables based on your deployment:
- Domino host URL (DOMINO_HOST)
- Keycloak admin username (KEYCLOAK_ADMIN_USERNAME)
- Keycloak admin password (KEYCLOAK_ADMIN_PASSWORD)
- The Keycloak realm (KEYCLOAK_REALM) defaults to DominoRealm and can be overridden if needed.
Replace the placeholders and run these commands in the terminal:
export DOMINO_HOST=https://placeholder.domino.tech export KEYCLOAK_ADMIN_USERNAME=placeholder_username export KEYCLOAK_ADMIN_PASSWORD=placeholder_password
Make sure that the trailing ”/” is not included in the DOMINO_HOST variable.
Run the script with a list of usernames and the command to execute. The command is either GRANT or REMOVE.
./keycloak_dmm_access_control.sh '("foo", "bar")' GRANT
Check the spelling of every username. If a username is misspelled or does not exist, the script silently skips that user and continues with the rest.

Domino endpoint Hardware Tiers

Domino endpoints often have different requirements from Workspaces and Jobs, so Domino keeps a separate class of hardware tier for them. Domino endpoint tiers and regular hardware tiers are not interchangeable: Domino endpoints can only use Domino endpoint tiers, and Domino endpoint tiers cannot be used outside Domino endpoints. To create one:

From the admin home page, go to Advanced > Hardware Tiers.
Click New to create a hardware tier, or click Edit to modify an existing one or set a default.
Select the hardware tier values. See Manage a hardware tier for what each field controls.
Select Is Domino endpoint Tier.
Optionally, make this tier the default for all Domino endpoints.

Scale Domino endpoints

Scaling the hardware attached to a Domino endpoint is one lever. The other is the degree of parallelism, which Domino sets deployment-wide for all Python Domino endpoints.

Only synchronous models support the degree of parallelism setting.

Go to Admin > Platform settings > Configuration records.
Set com.cerebro.domino.modelmanager.uWsgi.workerCount to a value greater than its default of 1 to increase the uWSGI worker count. See the uWSGI documentation for what the worker count changes.
Domino shows the message Changes here do not take effect until services are restarted. Click here to restart services. Click here to restart the services.

External deployment targets

An external deployment target is a connection to an external account that models can be deployed to from Domino. Typically, a target is one account and region.

Amazon SageMaker is the only type of external deployment target currently supported. Deploying models to SageMaker is only supported when the Domino control plane runs on an AWS EKS cluster.

To create a target, go to External deployment targets under Resources in the admin panel, click Create deployment target, and complete these fields:

The External deployment targets page in the admin panel

Field	What it sets
Name	The name of the target. Use something a data scientist can choose confidently at deployment time, such as `Production West`.
Account ID	The AWS account ID to use with this target.
Use IRSA role credentials	Whether to assume the execution role in the target AWS account using the Domino external deployments IAM role.
Credentials file contents	The contents of an AWS credentials file holding credentials for a user that can assume the execution role in the target account. See the AWS credentials file format. Ignored when IRSA role credentials are enabled.
Execution role name	The name of the execution role to assume when accessing AWS. This role must already exist in the target account.
ECR repository name	The ECR repository to store deployment images in. It is created if it does not exist in the target account.
Artifacts S3 bucket name	The S3 bucket to store deployment artifacts in. It is created if it does not exist in the target account.
Region	The AWS region code that deployments using this target create resources in, for example `us-east-1` or `eu-west-2`.
Resource configurations	The compute resources available in this target, which practitioners choose from when they deploy. Confirm the selected instance types are available in the selected region and that your AWS account has quota for them.
Permissions	Whether the target is available to everyone or only to specific users and organizations.

When Domino runs on an AWS EKS cluster, a default deployment target for the Domino AWS account is created automatically, so users can deploy external endpoints inside the AWS account that runs the Domino platform. This can be disabled when Domino is deployed, and admins can change or remove the default target afterwards.

To configure an AWS account for use as a Domino deployment target, use the Terraform module in the dominodatalab/terraform-external-sagemaker repository.

Architecture & deployment

Domino installation

Identity & access

Infrastructure & compute

Platform configuration

Data administration

Operations, monitoring & cost

Security & compliance

Troubleshooting

Model deployment administration

Model Monitoring notifications

Model Monitoring access control

Domino endpoint Hardware Tiers

Scale Domino endpoints

External deployment targets

​Model Monitoring notifications

​Model Monitoring access control

​Domino endpoint Hardware Tiers

​Scale Domino endpoints

​External deployment targets

​Related

Model Monitoring notifications

Model Monitoring access control

Domino endpoint Hardware Tiers

Scale Domino endpoints

External deployment targets

Related