> ## Documentation Index
> Fetch the complete documentation index at: https://docs.domino.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Track external data

You can use the [Domino Reproducibility Engine](/cloud/platform-capabilities/core-concepts/reproducibility) to track external data in Domino for reproducibility and auditability by persisting the data in Domino when your code runs. If your code connects to an external data source (e.g., a database, an API, a network drive, or a [Domino Data Source](/cloud/platform-capabilities/core-concepts/data/data-source-connectors/use-data-sources)), Domino does not know about the data coming from those sources.

## Materialize in-memory data as files for the Reproducibility Engine

To track external data for reproducibility reasons, you can materialize the in-memory data as a file so that the Domino Reproducibility Engine can track it. You can materialize it using any of the following options:

* [Domino Dataset](/cloud/platform-capabilities/core-concepts/data/datasets)

* A file like a CSV or Parquet in the Domino File System (DFS)

* [Domino TrainingSet](/cloud/reference/api/data_api/trainingsets_usecases)

<Tip>
  For best performance, keep your results set under 10GB if you are persisting files to the Domino file system.
</Tip>

### Use Datasets to track external data

Persist a data frame as a [Domino Dataset](/cloud/platform-capabilities/core-concepts/data/datasets) using the following code:

<Tabs>
  <Tab title="Python">
    To materialize a dataframe as a Dataset in Python, run the following:

    ```python theme={null}
    # Save data to the "quick-start" dataset
    csv_file_path = "/domino/datasets/local/quick-start/data.csv"
    parquet_file_path = "/domino/datasets/local/quick-start/data.parquet"

    # Write the DataFrame to a CSV file
    df.to_csv(csv_file_path, encoding='utf-8', index=False)

    # Write the DataFrame to a binary parquet format
    df.to_parquet(parquet_file_path)
    ```
  </Tab>

  <Tab title="R">
    To materialize a table as a Dataset in R, use the following code:

    ```r theme={null}
    library(arrow, warn.conflicts=FALSE)

    # Save data to the "quick-start" dataset
    csv_file_path <- "/domino/datasets/local/quick-start/data.csv"
    parquet_file_path <- "/domino/datasets/local/quick-start/data.parquet"

    # Write the table to a CSV file
    write.csv(table, file=csv_file_path, fileEncoding="UTF-8")

    # Write the table to a binary parquet format
    write_parquet(table, parquet_file_path)
    ```
  </Tab>
</Tabs>

### Store files in the DFS to track external data

Use the following code to persist a data frame as a CSV or binary file in the [Domino File System](/cloud/platform-capabilities/core-concepts/projects/manage-dfs-projects).

<Tabs>
  <Tab title="Python">
    To materialize a data frame as a file in your working directory, run the following Python code in a notebook code cell, then [sync the file changes](/cloud/platform-capabilities/core-concepts/workspaces/sync-changes-in-a-workspace) to persist the file in the backing repository.

    ```python theme={null}
    ## Save data to the project's default working directory `mnt` as an artifact add an additional sync step
    csv_file_path = "/mnt/data.csv"
    parquet_file_path = "/mnt/data.parquet"

    # Write the DataFrame to a CSV file
    df.to_csv(csv_file_path, encoding='utf-8', index=False)

    # Write the DataFrame to a binary parquet format
    df.to_parquet(parquet_file_path)
    ```
  </Tab>

  <Tab title="R">
    To materialize a table as a file in the root of the working `/mnt/` directory, run the following R code in RStudio console, then [sync the file changes](/cloud/platform-capabilities/core-concepts/workspaces/sync-changes-in-a-workspace) to persist the file in the backing repository:

    ```r theme={null}
    library(arrow, warn.conflicts=FALSE)

    # Save data to the project's default working directory `mnt`
    csv_file_path <- "/mnt/data.csv"
    parquet_file_path <- "/mnt/data.parquet"

    # Write the table to a CSV file
    write.csv(table, file=csv_file_path, fileEncoding="UTF-8")

    # Write the table to a binary parquet format
    write_parquet(table, parquet_file_path)
    ```
  </Tab>
</Tabs>

### Use TrainingSets to track external data

To save a data frame as a TrainingSet, run the following Python code.

```python theme={null}
from domino.training_sets import TrainingSetClient

training_set_version = TrainingSetClient.create_training_set_version(
    training_set_name="my-training-set",
    df=df
)

# To retrieve the data back as Data Frame
# tsv_by_num = TrainingSetClient.get_training_set_version(
#     training_set_name="my-training-set",
#     number=1,
# )
# training_df = tsv_by_num.load_training_pandas()
```

## Data Source audit logs

For external data accessed using a [Domino Data Source connector](/cloud/platform-capabilities/core-concepts/data/data-source-connectors), you can also use the [Data Source audit log](/cloud/admin/operations/audit-logs/monitor-data-source-logs) to track Data Source activity.

Data Source logs provide a way to track user activity, which can be helpful for reproducibility and lineage. However, Domino cannot reproduce the state of your Data Source at the time of execution.

## Next steps

This section explains how to ensure workflow reproducibility in Domino:

[Reproducibility use cases](/cloud/platform-capabilities/core-concepts/reproducibility/reproducibility-use-cases)\
Learn how to reproduce the results of a Job, Workspace, Model, App, or Launcher.

[Selectively revert past materials](/cloud/platform-capabilities/core-concepts/reproducibility/revert-materials)\
Selectively restore a part of a Project, such as the package library version, while keeping your latest code and data.

[File syncing and persistence](/cloud/platform-capabilities/core-concepts/reproducibility/file-syncing)\
Domino tracks project files automatically, storing previous versions in the blob store.

[Remove a file from the DRE: Permanent deletion](/cloud/platform-capabilities/core-concepts/reproducibility/permanent-deletion)\
Purge a file completely and permanently from the blob store.

[Tips for reproducibility in Domino](/cloud/platform-capabilities/core-concepts/reproducibility/reproducibility-tips)\
Tips for maximizing the power of the Domino Reproducibility Engine.