Definition file and class imports
Flows are defined inside a Python file. In this document, we will call the fileworkflow.py, but the name can be anything.
We recommend that you import a few relevant classes after you create the file. This will be important later for defining flows.
Define the flow interface
You’ll need to define the interface for your flow after you’ve created a definition file and imported relevant classes. Defining the interface entails creating a Python method with appropriate decorators and strongly typed inputs/outputs.-
Flow decorator: The
@workflowdecorator marks the method as a flow definition. -
Flow name: The name of the method (
training_workflow) is also the name that will be given to the flow when it is registered. -
Launch inputs: The parameters to the method are the initial inputs of the flow. They are later passed in as inputs to individual tasks. In the example above, there is one launch input called
data_pathof typestr. -
Flow outputs: The return type to the method are the final outputs of the flow. Outputs from the final tasks are typically returned as the results to the flow. It is recommended to use a
NamedTupleso that a name will be assigned to each output. This keeps the results more organized and easier to consume in downstream tasks. In the example above, there is one output calledmodelof typeFlyteFile[TypeVar("pt")]. Theptvalue in this case represents the file extension of the output (i.e., the extension of a PyTorch model).
-
Python primitives:
str,bool,int,float -
Python non-primitives:
list,dict,datetime -
Data science types:
np.ndarray,pandas.DataFrame,pyspark.DataFrame,torch.Tensor / torch.nn.Module,sklearn.base.BaseEstimator,tf.keras.Model -
Generic
FlyteFiletype: To enable file rendering in the UI, this type must either:-
Be combined with a
TypeVarthat defines the file extension type, for example,FlyteFile[TypeVar("csv")]andFlyteFile[TypeVar("pdf")]. -
Be a Flow Artifact like
Artifact(name="My Data", type=DATA).File(name="data.csv")
-
Be combined with a
Define tasks
Tasks are the core building blocks within a flow and are isolated within their own container during an execution. A task maps to a single Domino Job.Flow-generated vs standalone Domino Jobs
While all tasks trigger a unique Domino Job, there are some differences between jobs launched by a flow and standalone jobs. More specifically, for jobs launched by a flow:-
Only Domino Dataset snapshots and NetApp Volume snapshots can be mounted, unlike standalone Domino Jobs which support snapshot mounts and read-write mounts. For data to be used in a flow, it must be part of a versioned Dataset snapshot (
version 0of a dataset, i.e. the read-write directory is NOT considered a snapshot), or part of a NetApp Volume snapshot. -
Only one snapshot of a Dataset or NetApp Volume may be mounted at a time. When leveraging the
use_latestflag, the latest snapshot will be mounted. -
Snapshots are
read-onlyand cannot be modified during a job. Any processed data that needs to be persisted should be defined and written as a task output and therefore written to the Flow blob storage. -
Snapshots are mounted to a standard location that doesn’t include a snapshot ID in the path (the same location used for the
latestversion of a dataset or NetApp Volume in a workspace). For DFS projects, the path is:/domino/datasets/local/{name}for local dataset snapshots./domino/datasets/{name}for shared dataset snapshots./domino/netapp-volumes/{name}for NetApp Volume snapshots. For Git-based projects, the path is:/mnt/data/{name}for local dataset snapshots./mnt/imported/data/{name}for shared dataset snapshots./mnt/netapp-volumes/{name}for NetApp Volume snapshots. - Snapshots of the project code and artifacts (results) are not taken at the end of the job. Any results that need to be persisted should be defined and written as a task output and therefore written to the Flow blob storage.
-
There are two additional directories:
/workflow/inputsand/workflow/outputs. These are where the task inputs/outputs of the flow will be stored. A job status is considered failed if the expected task outputs are not produced by the end of execution. See the Writing task code section for more details on how to write your code accordingly. - Stopping a job orchestrated by a Flow in the Jobs UI will stop the entire flow, including other jobs that are running as part of it.
- Completed jobs cannot be re-run through the Domino Jobs UI. They must be relaunched by re-running the task from the Flows UI.
- Additional job metadata is captured and displayed in the Job Details to reference the flow and task that launched it.
Add a task to the flow
Tasks for a flow can be defined in one of the following ways:-
Use the base
DominoJobConfigandDominoJobTaskclasses. These provide more flexibility and direct control of the exact parameters to use. -
Use the
run_domino_job_taskhelper method. This offers a more user-friendly abstraction that enables definition and execution of a task in the same line of code.
Use base classes
The following example defines a task using the base classes:-
The
DominoJobConfigdefines the configuration for the Domino Job that will be triggered by the task. The only required parameter is theCommandconfigured in the example above. The full list of available parameters include:Parameter (* Required) Type Description and Example Title
String
The title that will be given to the Domino Job. Ignored when calculating the config’s cache version.
Example:
My training jobCommand (*)
String
The command that will be executed in the Domino Job.
Example:
python train.pyCommitId
String
For projects hosted in the Domino File System, this refers to the commit ID of the code. For Git-based projects, this refers to the commit ID of Artifacts.
Example:
953f66f3153b71658dMainRepoGitRef
GitRef
Reference a specific branch, commit, or tag (for Git-based projects only). See the API guide for more details.
Example:
GitRef(Type=“commitId”, Value=“2f1cb9bf696921f0858”)ResolvedMainRepoGitRef
String
The resolved commit hash to use. In most cases, if a specific commit is needed, it is best to use the
Valueargument toGitRef. Specifying the resolved commit explicitly can be used to track a floating reference, but resolve to an older commit.
Example:
2f1cb9bf696921f0858UseLatestGitRef
Boolean =
FalseWhen
True, floatinggitreferences specified inMainRepoGitRefandImportedGitRepoOverrideswill be resolved to the latest commit when the Flow is run or relaunched. WhenFalse, floatinggitreferences will resolve the reference on the first run, which will be reused when the Flow is relaunched.
Example:
TrueHardwareTierId
String
The ID of the Domino Hardware Tier. Note that this is different than the name of the hardware tier. Ignored when calculating the config’s cache version.
Example:
small-k8sEnvironmentId
String
The ID of the Domino Environment. Note that this is different than the name or revisionId of the Environment.
Example:
6646530dcbd87f1a3dec0050EnvironmentRevisionSpec
EnvironmentRevisionSpecification
The revision of the specified Domino Environment.
Example:
EnvironmentRevisionSpecification(
EnvironmentRevisionType=EnvironmentRevisionType.SomeRevision,
EnvironmentRevisionId=“6659daf5fc8de”
)ComputeClusterProperties
Optional[ClusterProperties]
Compute cluster properties for the job.
Example:
ClusterProperties(
ClusterType=ComputeClusterType.Dask,
ComputeEnvironmentId=“computeEnvId”,
WorkerHardwareTierId=“workerHardwareTierId”,
WorkerCount=3,
WorkerStorageGiB=100.0,
MaxWorkerCount=10,
ComputeEnvironmentRevisionSpec=EnvironmentRevisionSpecification(
EnvironmentRevisionType.SomeRevision,
“revisionId”
),
MasterHardwareTierId=“masterHardwareTierId”,
ExtraConfigs={“foo”: “bar”},
)VolumeSizeGiB
Float
The amount of disk space (in GiB) to allocate. Ignored when calculating the config’s cache version.
Example:
10.0DatasetSnapshots
List[DatasetSnapshot]
List of the dataset snapshots to include in the job. Note that
version 0of a dataset cannot be used, since it is mutable. You must take a snapshot first before using a dataset in a flow.
Example:
[DatasetSnapshot(Id=“6615af2820a4”, Version=1)]NetAppVolumeSnapshots
List[NetAppVolumeSnapshot]
List of the NetApp Volume snapshots to include in the job. Note that
version 0of a NetAppVolume cannot be used, since it is mutable. You must take a snapshot first before using a NetApp Volume in a flow.
Example:
[NetAppVolumeSnapshot(Id=“5d1204c2-24c8”, Version=1)]ExternalVolumeMountIds
List[String]
List of the external data volume mounts (referenced by ID) to include in the Job.
Example:
[“9625af24kida4dc035aa881b7”]ImportedGitRepoOverrides
List[ImportedGitRepoOverride]
List of
ImportedGitRepoOverride, which are used to override how importedgitrepositories are resolved on Flow relaunch.
Example:
[ImportedGitRepoOverride(Name="org.utils", GitRef=GitRef(Type="branch", Value="dev"))]cache_ignore_vars
Optional[Tuple[str, …]] = None
Optional tuple of names of arguments to the
DominoJobConfigto ignore when calculating the config’s cache version.
Example:
(“EnvironmentId”,) -
The
DominoJobTaskdefines the actual task itself. Each of the available parameters can be described as follows:Parameter (* Required) Type Description and Example name (*)
String
The name that will be given to the task.
Example:
My training taskdomino_job_config (*)
DominoJobConfig
The job configuration, as defined above.
Example:
DominoJobConfig(Command=“python prep-data.py”)inputs
Dict[String, Type]
Inputs that are required by the task. See above for different input types that are supported. Inputs may be specified from the workflow OR may be outputs from other tasks.*
Example:
{‘data_path’: str}outputs
Dict[String, Type]
Outputs that will be produced by the task. See above for different output types that are supported.
Example:
{‘processed_data’: FlyteFile[TypeVar(“csv”)]}use_latest
Boolean
If set to
True, this will use the latest project defaults for parameters that were not explicitly provided, like the compute environment version and hardware tier. For better reproducibility, it is recommended to set this toFalseand explicitly define the necessary parameters.
Example:
Falsecache
Union[bool, Cache]
Indicates if caching should be enabled. Can be a
boolor a FlyteCacheobject. Defaults tofalse. Settingcache=trueis a simple way to enable caching for a task, but using aCacheobject may be more flexible depending on your use case. See the caching documentation for a full explanation of how caching works.
Example:
Truecache_version
String
When
cache=true, the cache version to use. Changes to the task signature will automatically trigger a cache miss, but you can always manually update this field as well to force a cache miss. You should also manually bump this version if the function body/business logic has changed, but the signature hasn’t.
Example:
“1.0”cache_ignore_input_vars
Tuple[str, …]
When
cache=true, variable names ofinputsto theDominoJobTaskthat should not be included when calculating the hash used for caching. If not provided, all input variables will be included when calculating the hash.
Example:
(batch_size,)retries
Integer
Number of times to retry this task during a workflow execution. This can be used to help automatically mitigate intermittent failures.
Example:
0timeout
Union[timedelta, int]
The maximum amount of time for which one execution of this task should be executed. The execution will be terminated if the runtime exceeds the given timeout.
Example:
timedelta(hours=3) -
Calling the Domino Job task with the relevant inputs (
data_prep_job(data_path=data_path)) will run the Domino Job and return the results as aPromise, which can be used as an input to downstream tasks.
Use helper methods
Helper methods reduce the amount of code necessary to invoke a task. Instead of separately defining aDominoJobConfig and passing it to a DominoJobTask in the examples above, use run_domino_job_task to define the task contract and execute it immediately:
| Parameter | Type | Description and Example |
|---|---|---|
flyte_task_name | String | The title that will be given to the task. |
job_title | String | The title that will be given to the Domino Job. |
use_project_defaults_for_omitted | Boolean | If set to |
dfs_repo_commit_id | String | For projects hosted in the Domino File System, this refers to the commit ID of the code. For Git-based projects, this refers to the commit ID of the artifacts. |
main_git_repo_ref | GitRef | Reference a specific branch, commit, or tag (for Git-based projects only). See the API guide for more details. |
environment_name | String | Name of the Environment to use in the Domino Job. |
environment_id | String | ID of the Environment to use in the Domino Job. This is recommended over using |
environment_revision_id | String | A specific revisionId of the environment to use. |
hardware_tier_name | String | Name of the hardware tier to use in the Domino Job. |
hardware_tier_id | String | ID of the hardware tier to use in the Domino Job. This is recommended over using |
inputs | List[Input] | Inputs that are required by the task. See above for different input types that are supported. |
output_specs | List[Output] | Outputs that will be produced by the task. See above for different output types that are supported. |
volume_size_gib | Integer | The amount of disk space (in GiB) to allocate. |
dataset_snapshots | List[DatasetSnapshot] | List of the dataset snapshots to include in the Job. Note that |
netapp_volume_snapshots | List[NetAppVolumeSnapshot] | List of the NetApp Volume snapshots to include in the Job. Note that |
external_data_volume_ids | List[str] | List of the external data volume mounts (referenced by ID) to include in the Job. |
cache | Union[bool, Cache] | Indicates if caching should be enabled. Can be a |
cache_version | String | When |
cache_ignore_input_vars | Tuple[str, …] | When |
retries | Integer | Number of times to retry this task during a workflow execution. |
timeout | Union[timedelta, int] | The maximum amount of time for which one execution of this task should be executed. The execution will be terminated if the runtime exceeds the given timeout. |
Add a dependent task to the flow
It’s common for one task to depend on another task; that is, one task accepts an input that is produced by another task as an output. This ensures that the dependent task will not start execution until outputs from the other task are produced first. To create dependent tasks, you can use either the base classes or helper methods to define them. In the example below, note how the second task uses the output from the first task by callingdata_prep_results[“processed_data]”.
Use base classes
Use helper methods
Return the final output
You can set the output of the flow by returning it in the method. Note that defining an overall Flow output is not required and does not elevate this particular output in the UI. Please see Define Flow Artifacts for elevating important task outputs.Write task code
Writing code for jobs that were generated by flows is slightly different than writing code for a standalone Domino Job. Flow-generated jobs have inputs that come in from the task and additional logic needs to be added to read those inputs. Once results are produced, they also need to be explicitly written as an output to the assigned output location.Read inputs
For each input that is defined for a task, a unique blob is created and accessible within a Job at/workflow/inputs/<NAME OF INPUT>.
For file type inputs, the blob is the actual file that was inputted to the task. Example usage:
str, bool, int, list, dict, etc.), the blob contents contain the input value. Example usage:
-
To set up inputs manually, create each
/workflow/inputs/<NAME OF INPUT>file that the task code expects. For example, if your task code expects a CSV input file namedmy_dataand you have a file in your current working directory calledtest_data.csvthat you want to use as that input, then run the terminal commandcp test_data.csv /workflow/inputs/my_data. For primitive inputs like strings and integers, write the data as a string to the file/workflow/inputs/<NAME OF INPUT>. -
Alternatively, the Domino Python helper library has functions that facilitate setting up input data. These functions allow you to use inputs or outputs from past Flyte executions.
Here is an example that shows setting up local input data for a task that expects five inputs:
-
my_csv_one, using the output of a previous Flow execution. -
my_csv_two, using the input of a previous Flow execution. -
my_csv_three, using the local filetest_data.csv. -
my_str, using the string"my string input data". -
my_int, using the integer42.
-
Write outputs
Outputs defined for a task must be written to/workflow/outputs/<NAME OF OUTPUT>. For example:
Best Practice - Dynamic Pathing
If you want your task code to seamlessly run across workspaces/jobs and flows then it is recommended that you use theDOMINO_IS_WORKFLOW_JOB default environment variable. This variable is true when your code is running as part of a flow and false when it is running in a workspace or standalone job. You can therefore reference this variable and set pathing for code inputs/outputs depending on where it is running.