Skip to main content
Data is at the core of every data science project. Data varies across different use cases and lives in many places within an organization. Domino provides the flexibility to access all of your data, whatever and wherever it may be. Data overview Each of these data access and storage mechanisms comes with pros and cons, suited to different use cases. The topics in this section explain how to connect your data to Domino and use it in your projects.

​
Access all your data, anywhere

Domino provides a single interface for accessing all of your data, wherever it lives. Domino Data Source Connectors provide easy setup for connections to popular data services, or you can connect Domino directly to any data service using the same code you use in your local environment. Domino Datasets enable you to upload, store and manage data within the Domino system. You can also mount external storage volumes to Domino and use your Git repositories as project file stores. The following table compares Domino’s data access and storage methods:
NetApp VolumesDomino DatasetsData Source connectorsExternal Data VolumesProject artifacts

Description

Native integration with NetApp ONTAP enterprise storage, letting users mount NetApp volumes directly into Domino workloads.

Read/Write managed folders shareable within Domino. Can be versioned with snapshots for reproducibility.

Managed data connectors that can connect to SQL and file-type stores. Or, connect directly to any data service using a library.

Managed external file store mounts.

A special, version-controlled folder that works similarly to Git LFS.

Location

External NetApp ONTAP filesystems connected via a Domino data plane and Kubernetes storage class.

Network File System (NFS) storage or Amazon Elastic File System.

Any external data service.

NFS or Server Message Block (SMB) storage.

Domino File System (DFS).

Intended use cases

High-performance shared storage for teams using Domino with large, persistent datasets or needing enterprise backup/recovery.

  • Training sets that can’t easily be shared/controlled outside of Domino.

  • Derived data that’s an input to a downstream processing step in a pipeline.

Accessing data in existing external stores such as databases.

Exposing existing IT data storage interfaces within Domino.

Storing outputs (such as charts, serialized model files, output CSVs, and so on).

Intended data sizes

Multi-terabyte scale and beyond, leveraging ONTAP’s scalability and snapshot efficiency.

Up to ~1TB per Dataset and hundreds of TB across Datasets.

Constrained only by what you query and pull into the machines executing your code.

Constrained by your existing network storage.

Up to ~10GB.

Advantages

Fast, cost-efficient snapshots; high availability; reduced replication; strong data protection and retention options.

Supports much larger data than project artifacts, with snapshots for reproducibility.

Connectors provide an easy and secure way to connect to external data without drivers or configuration. Direct connections use the same code you would use outside of Domino, with the flexibility to access files or data however you want.

Use the same code you would use outside of Domino.

Simple to use and share.

Limitations

Requires external NetApp infrastructure and admin setup; not version-controlled natively within Domino. Has limits on number of snapshots per volume and volumes per filesystem.

Snapshots must be managed to minimize storage costs.

Code for accessing connector-supported data sources is not portable outside Domino. Domino does not automatically track snapshots for reproducibility.

Will not automatically track snapshots for reproducibility.

Not performant at scale of data size or many thousands of files.

Query methods

Accessed via mounted file paths inside Domino workloads. Also accessible via Domino Datasource SDK.

Path to the mounted Dataset.

Path to the mounted volume and files.

Path to the project’s files.

Supported executions

Workspaces, Jobs, Launchers, and Apps within Domino that mount the NetApp volume, or any other execution by querying via the Domino Datasource SDK.

  • Workspaces

  • Jobs

  • Launchers

  • Apps

  • Workspaces

  • Jobs

  • Launchers

  • Apps

  • Domino endpoints

  • Workspaces

  • Jobs

  • Launchers

  • Apps

  • On-demand Spark clusters

All

Access control

Role based.

Role based.

  • Domino data sources: Based on Domino user/group or a service account.

  • Direct access: Based on your data service’s access control (IAM token propagation is supported for some AWS data services).

Based on Domino user/organization as configured by an admin.

Per-project collaborator permissions.

​
Data reproducibility

When your project requires reproducible data, you can create Dataset snapshots that are immutable and versioned. You can also use the Data API to create training sets based on any of your data in any location. If your external data sources or network filesystems support dataframes, snapshots, or versioning, you can use those features in your code in Domino.

​
Share your data securely

Data in Domino is reusable across multiple projects, with full control over who can access it. You can share data with your collaborators and decide who has read-only access or the ability to edit the data. See Data sharing and security to learn how to share your data with collaborators and keep it secure.