Upload Binary Data¶

Motivation¶

Since every local Git repository contains a copy of the entire project history, it is important to avoid adding large binary files directly to the repository. Large binary files added and removed throughout a project’s history will cause the repository to become bloated, take up too much disk space, require excessive time and bandwidth to download, etc.

A solution to this problem which has been adopted by this project is to store binary files, such as images, in a separate location outside the Git repository, then download the files at build time with CMake.

A “content link” file contains an identifying SHA512 hash. The content link is stored in the Git repository at the path where the file would exist, but with a “.sha512” extension appended to the file name. CMake will find these content link files at build time, download them from a list of server resources, and create symlinks or copies of the original files at the corresponding location in the build tree.

Prerequisites¶

The data.kitware.com server is an ITK community resource where any community member can upload binary data files. There are two methods available to upload data files:

The Girder web interface.
The girder-cli command line executable that comes with the girder-client Python package.

Before uploading data, please visit data.kitware.com and register for an account.

Once files have been uploaded to your account, they will be publicly available and accessible since data is content addressed. At release time, the release manager will upload and archive repository data references in the ITK collection and other redundant storage locations.

Upload Via the Web Interface¶

Log in welcome page — After logging in, you will be presented with the welcome page. Click on the **personal data space** link.¶

Personal data space — Next, select the **Public** folder of your personal data space.¶

Public folder — Click the **green upload button**.¶

The Upload files dialog — Click the **Browse or drop files** to select the files to upload.¶

The Upload files dialog with files selected — Click **Start Upload** to upload the file to the server.¶

Next, proceed to Download the Content Link.

Upload Via Python Script¶

A Python script to upload files from the command line, girder-cli, is available with the girder-client python package. To install it:

python -m pip install girder-client

To upload files with the girder-cli script, we need to obtain an API key and a parent folder id from the web interface.

My account link — After logging in, select **My account** from the user drop down.¶

API key tab — Next, select the **API keys** tab.¶

Create new key — Create a new API key if one is not available.¶

My Folders link — Next, select **My Folders** from the user drop down.¶

Public folder information — Click the i button for information about the folder.¶

Public folder information modal — The **Unique ID** can be copied into the command line.¶

Use both the API key and the folder ID when calling girder-cli. For example,

girder-cli \
  --api-key 12345ALongSetOfCharactersAndNumbers \
  --api-url https://data.kitware.com/api/v1 \
  upload \
  58becaee8d777f0aefede556 \
  /tmp/cthead1.png

Next, proceed to Download the Content Link.

Download the Content Link¶

File has been uploaded — Click on the file that has been uploaded.¶

Item has been clicked — Click on the i button for further information.¶

File information — Finally, click on the **Download key file** icon to download the key file.¶

Move the content link file to the source tree at the location where the actual file is desired in the build tree. Stage the new file to your commit:

git add -- path/to/file.sha512