Flow files
Workflow basics
Flows are defined inside a Python file. In this document, we call the fileworkflow.py, but the name can be anything.
workflow.py
We’re using the
@workflow decorator to declare a function as a flow. The method name, training_workflow, is implicitly used as the name of the flow.Inputs & outputs
Your flow can be parameterized so that you can re-run it with new inputs without modifying the code of your flow. Inputs can be either structured data or files.Files
Files
Declare file inputs and outputs using the generic
FlyteFile type.To enable file rendering in the UI, this type must either:- Be combined with a
TypeVarthat defines the file extension type, for example,FlyteFile[TypeVar("csv")]andFlyteFile[TypeVar("pdf")]. - Be a Flow Artifact like
Artifact(name="My Data", type=DATA).File(name="data.csv")
Python base types
Python base types
Use base Python types for passing structured data through flows:
str, bool, int, float, list, dict, datetimeData science types
Data science types
A number of common data science types are supported as well:
np.ndarray, pandas.DataFrame, pyspark.DataFrame, torch.Tensor / torch.nn.Module, sklearn.base.BaseEstimator, tf.keras.ModelExplicit, strongly typed inputs and outputs ensure that tasks pass data to each other correctly. For instance, a data processing task may emit a Pandas dataframe as output, while a training task accepts a Pandas dataframe as input. Flows analyze workflow submissions to prevent user error before execution, providing actionable feedback for incorrectly specified parameters that don’t satisfy task contracts.
n) to our @workflow function and returning one of our results (the "results" output in sim):
workflow.py
Flow tasks
Declare tasks
Tasks are the core building blocks within a flow and are isolated within their own container during an execution. A task maps to a single Domino Job.Flow Task Jobs vs standalone Jobs
Flow Task Jobs vs standalone Jobs
While all tasks trigger a unique Domino Job, there are some differences between jobs launched by a flow and standalone jobs. More specifically, for jobs launched by a flow:
-
Only Domino Dataset snapshots and NetApp Volume snapshots can be mounted, unlike standalone Domino Jobs which support snapshot mounts and read-write mounts. For data to be used in a flow, it must be part of a versioned Dataset snapshot (
version 0of a dataset, that is, the read-write directory is NOT considered a snapshot), or part of a NetApp Volume snapshot. -
Only one snapshot of a Dataset or NetApp Volume may be mounted at a time. When leveraging the
use_latestflag, the latest snapshot is mounted. -
Snapshots are
read-onlyand cannot be modified during a job. Any processed data that needs to be persisted should be defined and written as a task output and therefore written to the Flow blob storage. -
Snapshots are mounted to a standard location that doesn’t include a snapshot ID in the path (the same location used for the
latestversion of a dataset or NetApp Volume in a workspace). For DFS projects, the path is:-
/domino/datasets/local/{name}for local dataset snapshots. -
/domino/datasets/{name}for shared dataset snapshots. -
/domino/netapp-volumes/{name}for NetApp Volume snapshots.
-
/mnt/data/{name}for local dataset snapshots. -
/mnt/imported/data/{name}for shared dataset snapshots. -
/mnt/netapp-volumes/{name}for NetApp Volume snapshots.
-
- Snapshots of the project code and artifacts (results) are not taken at the end of the job. Any results that need to be persisted should be defined and written as a task output and therefore written to the Flow blob storage.
-
There are two additional directories:
/workflow/inputsand/workflow/outputs. These are where the task inputs/outputs of the flow are stored. A job status is considered failed if the expected task outputs are not produced by the end of execution. See the Write task code section for more details on how to write your code accordingly. - Stopping a job orchestrated by a Flow in the Jobs UI stops the entire flow, including other jobs that are running as part of it.
- Completed jobs cannot be re-run through the Domino Jobs UI. They must be relaunched by re-running the task from the Flows UI.
- Additional job metadata is captured and displayed in the Job Details to reference the flow and task that launched it.
-
Use the base
DominoJobConfigandDominoJobTaskclasses. These provide more flexibility and direct control of the exact parameters to use. -
Use the
run_domino_job_taskhelper method. This offers a more user-friendly abstraction that enables definition and execution of a task in the same line of code.
Pass data
It’s common for one task to depend on another task; that is, one task accepts an input that is produced by another task as an output. This ensures that the dependent task does not start execution until outputs from the other task are produced first. To create dependent tasks, you can use either the base classes or helper methods to define them. In the example below, note how the second task uses the output from the first task by callingdata_prep_results["processed_data"].
Return outputs
You can set the output of the flow by returning it in the method. Note that defining an overall Flow output is not required and does not elevate this particular output in the UI. Please see Define Flow Artifacts for elevating important task outputs.Write task code
Domino code execution aims for consistency across all execution modalities. To support this, flows can leverage the same data mount paths used in a workspace or standalone job. As an input, declared inputs that reside under/mnt are populated for reading. Data written back to data paths can optionally be used to create a new snapshot of the data at the end of the flow run.
However, for explicit inputs and outputs, flows additionally mounts data for input and output in two special directories under /workflow. Files can be read from /workflow/inputs and written to /workflow/outputs. In addition to files, flows also allows for the passing of structured data such as dataframes and lists. These objects also show up in the /workflow directories, but require special handling to read them as structured data in Python.
Read inputs
For each input that is defined for a task, a file is created and accessible within a Job within/workflow/inputs/ in a file whose name exactly matches that of the named input.
For file type inputs, the file can be read directly at the given path.
str, bool, int, list, dict, etc.), the blob contents contain the input value. Example usage:
Write outputs
Outputs defined for a task must be written into/workflow/outputs/ to a file that exactly matches the name of the output.
Flow jobs do not make commits or dataset snapshots. Results should always be written as outputs. Snapshots are taken at the end of flow execution if configured.
Domino Flyte plugin
Domino provides a set of Flyte-compatible classes, as well as utility functions for defining tasks using these classes, in a way that integrates nicely with Domino Jobs. Notably, theflytekitplugins.domino library provides groups of utilities in the helpers, task, and artifact submodules.
Class constructors
The following example defines a task using the base classes.ImportedGitRepoOverrides to DominoJobConfig (Git-based projects only):
DominoJobConfig()
The DominoJobConfig defines the configuration for the Domino Job that is triggered by the task.
DominoJobTask()
Calling the Domino Job task with the relevant inputs (data_prep_job(data_path=data_path)) runs the Domino Job and returns the results as a Promise, which can be used as an input to downstream tasks.
EnvironmentRevisionSpec()
ComputeClusterProperties()
Helpers
run_domino_job_task()
Helper methods reduce the amount of code necessary to invoke a task. Instead of separately defining a DominoJobConfig and passing it to a DominoJobTask in the examples above, use run_domino_job_task to define the task config, job, inputs and outputs all using a single function call.
To pin imported Git repos for a task, use the base
DominoJobConfig and DominoJobTask classes with ImportedGitRepoOverrides. The run_domino_job_task helper also accepts imported_git_repo_overrides.