ITK/Git/Develop/Data

From KitwarePublic
< ITK‎ | Git‎ | Develop
Revision as of 16:19, 15 June 2011 by Brad.king (talk | contribs)
Jump to navigationJump to search

This page documents how to add test data while developing ITK. See our table of contents for more information.

Workflow

Our workflow for adding data integrates with our standard Git development process. Start by creating a topic. Return here when you reach the "edit files" step.

These instructions follow a typical use case of adding a new test with a baseline image.

Add Data

Copy the data file into your local source tree.

$ mkdir -p Modules/.../test/Baseline
$ cp ~/MyTest.png Modules/.../test/Baseline/MyTest.png

Add Test

Edit the test CMakeLists.txt file and reference the data file in an itk_add_test call. Specify the file inside DATA{...} using a path relative to the test directory:

$ edit Modules/.../test/CMakeLists.txt
itk_add_test(NAME MyTest COMMAND ... --compare DATA{Baseline/MyTest.png} ...)

ExternalData.cmake

Run CMake

Run cmake on the build tree:

$ cd ../ITK-build
$ cmake .
(Or just run "make" to do a full configuration and build.)
$ cd ../ITK

During configuration CMake will display a message such as:

Linked Modules/.../test/Baseline/MyTest.png.md5 to ExternalData MD5/...

This means that CMake converted the file into a data object referenced by a "content link".

ExternalData.cmake

Commit

Continue to create the topic and edit other files as necessary. Add the content link and commit it along with the other changes:

$ git add Modules/.../test/Baseline/MyTest.png.md5
$ git add Modules/.../test/CMakeLists.txt
$ git commit

git help add
git help commit

The local pre-commit hook will display a message such as:

Content link Modules/.../test/Baseline/MyTest.png.md5 -> .ExternalData/MD5/...

This means that the pre-commit hook recognized that the content link references a new data object and prepared it for upload.

pre-commit

The hook stores new objects in a local .ExternalData directory at the top of the source tree. It also adds them to the local Git repository in a location disjoint from the project history.

Push

Follow the instructions to share the topic. When you push it to Gerrit for review using

$ git gerrit-push

git-gerrit-push

part of the output will be of the form

*       ...:refs/data/commits/...      [new branch]
*       HEAD:refs/for/master/my-topic  [new branch]
Pushed refs/data and removed local copy:
  MD5/...

This means that the git-gerrit-push script pushed the topic and uploaded the data it references.

Discussion

An ITK test data file is not stored in the main source tree under version control. Instead the source tree contains a "content link" that refers to a data object by a hash of its content. At build time the the ExternalData.cmake module fetches data needed by enabled tests. This allows arbitrarily large data to be added and removed without bloating the version control history.

The above workflow allows developers to add a new data file almost as if committing it to the source tree. The following subsections discuss details of the workflow implementation.

ExternalData

While CMake runs the ExternalData module evaluates DATA{} references. ITK sets the ExternalData_LINK_CONTENT option to MD5 to enable automatic conversion of raw data files into content links. When the module detects a real data file in the source tree it performs the following transformation as specified in the module documentation:

  • Compute the MD5 hash of the file
  • Store the ${hash} in a file with the original name plus .md5
  • Rename the original file to .ExternalData_MD5_${hash}

The real data now sit in a file that we tell Git to ignore.

pre-commit

While committing a new or modified content link the pre-commit hook moves the real data object from the .ExternalData_MD5_${hash} file left by the ExternalData module to a local object repository stored in a .ExternalData directory at the top of the source tree.

The hook also uses Git plumbing commands to store the data object as a blob in the local Git repository. The blob is not referenced by the new commit but instead by refs/data/MD5/${hash}. This keeps the blob alive in the local repository but does not add it to the project history.

git gerrit-push

The "git gerrit-push" command is actually an alias for the Utilities/Git/git-gerrit-push script. In addition to pushing the topic branch to Gerrit the script also detects content links added or modified by the commits in the topic. It reads the data object hashes from the content links and looks for matching refs/data/ entries in the local Git repository.

The script pushes the matching data objects to Gerrit inside a temporary commit object disjoint from the rest of history. A robot runs every few minutes to fetch the objects from Gerrit and upload them to a location that we tell ExternalData to search at build time.