{"id":3886,"date":"2022-05-25T13:04:35","date_gmt":"2022-05-25T11:04:35","guid":{"rendered":"https:\/\/dev.littlebigcode.fr\/how-dvc-manages-data-sets-training-ml-models-git\/"},"modified":"2022-07-04T23:27:39","modified_gmt":"2022-07-04T21:27:39","slug":"how-dvc-manages-data-sets-training-ml-models-git","status":"publish","type":"post","link":"https:\/\/dev.littlebigcode.fr\/en\/how-dvc-manages-data-sets-training-ml-models-git\/","title":{"rendered":"MLOps : How DVC smartly manages your data sets for training your machine learning models on top of Git"},"content":{"rendered":"

[et_pb_section fb_built=”1″ admin_label=”section” _builder_version=”4.16″ da_disable_devices=”off|off|off” global_colors_info=”{}” da_is_popup=”off” da_exit_intent=”off” da_has_close=”on” da_alt_close=”off” da_dark_close=”off” da_not_modal=”on” da_is_singular=”off” da_with_loader=”off” da_has_shadow=”on”][et_pb_row admin_label=”row” _builder_version=”4.16″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” custom_padding=”3px||3px||true|false” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.16″ custom_padding=”|||” global_colors_info=”{}” custom_padding__hover=”|||”][et_pb_text admin_label=”Text” _builder_version=”4.17.4″ text_font=”Average Sans||||||||” text_text_color=”#242B57″ link_font=”Average Sans||||||||” link_text_color=”#1CACE4″ ul_font=”Average Sans||||||||” ul_text_color=”#242B57″ ol_text_color=”#242B57″ quote_font=”Average Sans||||||||” quote_text_color=”#242B57″ header_text_color=”#1CACE4″ header_2_text_color=”#1CACE4″ header_3_text_color=”#1CACE4″ header_4_font=”Average Sans||||||||” header_4_text_color=”#1CACE4″ header_5_font=”Century Gothic Bold||||||||” header_5_text_color=”#1CACE4″ header_6_font=”Century Gothic Bold||||||||” header_6_text_color=”#1CACE4″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” custom_padding=”8px|||||” inline_fonts=”Century Gothic Bold,Century Gothic,Average Sans” global_colors_info=”{}”]<\/p>\n

This article belongs to a series of articles about MLOps tools and practices for data and model experiment tracking. In the first part, we explained why data and model experiment tracking was important, and how tools like DVC and Mlflow could solve this challenge. Today, we\u2019ll see how Data Version Control (DVC) smartly manages your data sets for training your machine learning models on top of Git.<\/p>\n

By Samson ZHANG<\/a>, Data Scientist at LittleBigCode<\/p>\n

What are we talking about? DVC is a MLOps tool that works on top of Git repositories and has a <\/span>similar command line interface and workflow to <\/span><\/span>Git. It is designed to tackle the challenge of data sets traceability and reproducibility<\/span> when training data-driven models.<\/p>\n

 <\/p>\n

Why do we need DVC ?<\/h1>\n

 <\/p>\n

All data-driven models require data to be trained. Managing and creating the data sets used for training data-driven models requires a lot of time and space. Depending on the project, there can be up to thousands of versions of the data set to train the models. This can quickly become muddled due to multiple users altering and updating the data which can greatly jeopardize the traceability and reproducilibity of experiments.<\/p>\n

In your data scientist career, you probably experienced data versions tracking issues when exploring and cleaning your data set, just like me.<\/p>\n

For<\/span> instance, I often worked on computer vision problems with thousands of images\/annotation files. Counting the raw noisy data, the cleaned data and the preprocessed data, there are already 3 different versions to keep. And that is still without keeping track of some processing steps results<\/span>!<\/p>\n

Without DVC, a possible approach would be zipping files and storing hashes (file content checksum), and locations in Git commits. The data set would be fully duplicated for each version. It would be complicated to update and to keep track of. Just imagine the work you would have to do each and every time you have new data to add or wrong labels to correct!<\/p>\n

This iterative process on the data set can be applied <\/span>to many data science projects and it is not scalable without proper tools.<\/p>\n

DVC has been created to exactly handle this iterative process in an efficient way.<\/p>\n

\n\t\t\t\t\t
As its name suggests, DVC is a data-versioning tool. It is first designed to track sequential updates of a data set. It is not designed to handle complex tasks such as concurrent labeling and data set cleansing (file removal and modification) in a multiple-users setting. A third-party specialized tool would be more appropriate for those tasks.<\/div><\/div><\/p>\n

Why use DVC for data version management instead of other tools such as Git or Mlflow ?<\/h2>\n

Mlflow is not designed to track a lot of large files (for instance, thousands of images) as it does not optimize storage for file duplication. Tracking datasets version with Mlflow would be inefficient. Mlflow itself does not guarantee the reproducibility of a data set used during an experiment run, unless you save the whole data set during each run, which is not scalable.<\/p>\n

Git is unsuited for large files versioning in general<\/span> (especially for datasets). Furthermore, saving your data set with your source code can be a huge security breach as anybody that works on the code can access potentially sensitive<\/span> data (even worse for public git repo).<\/p>\n

\n\t\t\t\t\t
Git-LFS exists as a git extension for large file storage but it has not been designed with data science in mind. DVC: About Git-LFS<\/a>. DVC can work with any cloud storage (or even a simple SSH server) and does not require a dedicated LFS server unlike Git-LFS.<\/div><\/div><\/p>\n

Those are the main reasons that motivate the use of an additional type of tool for data versioning such as DVC for improving your ML <\/span>experiments tracking experience. DVC complements Mlflow and Git in order to provide a complete<\/span> ML tracking experience.<\/p>\n

Technically, DVC is a file-versioning tool that can work with any type of data<\/span> (image, text, video) as it saves files. But the latter does not mean that it is adapted to version complex data types for ML purposes such as large video files because DVC simply tracks file versions with hashes (content checksum). For instance, a few seconds modification to an 1h-long video file (several GBs) results in 2 full 1h-long video files stored which implies a lot of duplication.<\/p>\n

How does it work ?<\/span><\/h2>\n

You can version datasets in your Git repository by only storing small *.dvc metafiles (text) tracked by Git commits (cf. figure 1). It has an optimized versioning capability like git by only storing the minimal quantity of information to describe <\/span>the data across all the data set versions in a repository. The same file appearing in multiple data set versions is stored only once<\/strong>.<\/p>\n

\"\"

Figure 1. How DVC works. Source: DVC.org<\/a><\/p><\/div>\n

 <\/p>\n

Project structure<\/span><\/h2>\n

A DVC repository is a Git repository that tracks DVC files. <\/span>Setting up a DVC repository and do data versioning <\/span>is easy.<\/p>\n

Let\u2019s take a look at the composition of a DVC repository :<\/p>\n