Batchmake slicer: Difference between revisions

From KitwarePublic
Jump to navigationJump to search
 
(10 intermediate revisions by one other user not shown)
Line 1: Line 1:
===BatchMake in Slicer===
===BatchMake in Slicer===
BatchMake has been integrated in Slicer since version 3, then highly improved in [http://wiki.slicer.org/slicerWiki/index.php/Documentation-3.4 version 3.4]. The [http://www.slicer.org/slicerWiki/index.php/Documentation-3.4#Batch_processing Batch Processing] category can be found in the [http://www.slicer.org/slicerWiki/index.php/Documentation-3.4#List_of_Modules Slicer modules list].<br>
BatchMake has been integrated in Slicer since version 3, and highly improved in [http://wiki.slicer.org/slicerWiki/index.php/Documentation-3.4 version 3.4]. The [http://www.slicer.org/slicerWiki/index.php/Documentation-3.4#Batch_processing Batch Processing] category can be found in the [http://www.slicer.org/slicerWiki/index.php/Documentation-3.4#List_of_Modules Slicer modules list].<br>
There are 4 modules into the Batch Processing subsection: [http://wiki.slicer.org/slicerWiki/index.php/Modules:GaussianBlurBatch-Documentation-3.4 Gaussian Blur BatchMake], [http://wiki.slicer.org/slicerWiki/index.php/Modules:RegisterImagesBatch-Documentation-3.4 Register Images BatchMake], [http://wiki.slicer.org/slicerWiki/index.php/Modules:EMSegmentBatch-Documentation-3.4 EMSegment BatchMake] and [http://wiki.slicer.org/slicerWiki/index.php/Modules:ResampleVolumeBatch-Documentation-3.4 Resample Scalar Volume BatchMake]. Each Batch module internally calla a Slicer module, respectively [http://wiki.slicer.org/slicerWiki/index.php/Modules:GaussianBlur-Documentation-3.4 Gaussian Blur], [http://wiki.slicer.org/slicerWiki/index.php/Modules:RegisterImages-Documentation-3.4 Register Images], [http://wiki.slicer.org/slicerWiki/index.php/Modules:EMSegment-Simple EMSegment Simple] and [http://wiki.slicer.org/slicerWiki/index.php/Modules:ResampleVolume-Documentation-3.4 Resample Scalar Volume].<br>
There are 4 modules into the Batch Processing subsection: [http://wiki.slicer.org/slicerWiki/index.php/Modules:GaussianBlurBatch-Documentation-3.4 Gaussian Blur BatchMake], [http://wiki.slicer.org/slicerWiki/index.php/Modules:RegisterImagesBatch-Documentation-3.4 Register Images BatchMake], [http://wiki.slicer.org/slicerWiki/index.php/Modules:EMSegmentBatch-Documentation-3.4 EMSegment BatchMake] and [http://wiki.slicer.org/slicerWiki/index.php/Modules:ResampleVolumeBatch-Documentation-3.4 Resample Scalar Volume BatchMake]. Each BatchMake module internally call a Slicer module, respectively [http://wiki.slicer.org/slicerWiki/index.php/Modules:GaussianBlur-Documentation-3.4 Gaussian Blur], [http://wiki.slicer.org/slicerWiki/index.php/Modules:RegisterImages-Documentation-3.4 Register Images], [http://wiki.slicer.org/slicerWiki/index.php/Modules:EMSegment-Simple EMSegment Simple] and [http://wiki.slicer.org/slicerWiki/index.php/Modules:ResampleVolume-Documentation-3.4 Resample Scalar Volume].<br>
<br>
<br>
Batch processing allows you to automatically run on multiple machines multiple modules multiple times with multiple parameters on multiple inputs. <br>
Batch processing allows to automatically run on multiple machines multiple modules multiple times with multiple parameters on multiple inputs. <br>
Since [http://wiki.slicer.org/slicerWiki/index.php/Documentation-3.4 Slicer 3.4], you can easily transform any Slicer modules to support batch processing. Only minor changes in the module XML file is needed, no code has to be written specifically for a Batch module.<br>
Since [http://wiki.slicer.org/slicerWiki/index.php/Documentation-3.4 Slicer 3.4], any Slicer modules can easily be transformed to support batch processing. Only minor changes in the module XML file is needed, no code has to be written specifically for a Batch module.<br>
In this tutorial we will introduce you with BatchMake in Slicer and guide you step by step to batch process your data. We will then explain how you can create your own BatchMake module based on a Slicer module.<br>
In this tutorial we will present BatchMake in Slicer and give a step by step tutorial to batch process data. We will then explain how to create a BatchMake module based on a Slicer module.<br>


===How to run a BatchMake module locally?===
===How to run a BatchMake module locally?===
Line 13: Line 13:
|}
|}
</center>
</center>
In this section, we will see how to run on a local machine a Gaussian Blur filter. For a single input, multiple output will be generated, one for each sigma value.
In this section, we will see how to run on a local machine a Gaussian Blur filter. For a single input image, the module will generate multiple outputs, one for each sigma value.
*Launch Slicer
*Launch Slicer
*Open the "Gaussian Blur BatchMake" module located in "Batch Processing"
*Open the "Gaussian Blur BatchMake" module located in <i>Batch Processing</i>.
*Set <b>Sigma Minimum</b>, <b>Sigma Maximum</b> and <b>Sigma Step</b>, it changes the number of times the Gaussian Blur module will be called with a different sigma. If Sigma minimum, maximum and step are respectively set to 1, 10 and 1; Gaussian Blur will be run 10 times, with a sigma varying from 1 to 10.
*Set <b>Sigma Minimum</b>, <b>Sigma Maximum</b> and <b>Sigma Step</b>. They change the number of times the Gaussian Blur module is called with a different sigma. If Sigma minimum, maximum and step are respectively set to 1, 10 and 1; Gaussian Blur will be run 10 times, with a sigma varying from 1 to 10.
*Then select a directory for <b>Data directory</b>. The directory shall contains the input images to blur.  
*Then select a directory for <b>Data directory</b>. The directory shall contains the input images to blur.  
*Define a mask (using wildcards) to select the images to blur based on the filename and/or extension. Each image fitting the mask requirements will be processed separately.
*Define a wildcards mask to select the images to blur in <b>Data directory</b> based on the filename and/or extension. Each image that fits the mask requirements is processed separately.
*Check <b>Recursive</b> if you want to recursively apply the "Data file" mask in the "Data directory".
*Check <b>Recursive</b> if you want to apply the "Data file" mask in the subdirectories of <b>Data directory</b>.
* Select a directory for <b>Output directory</b>. The ( nb_input * (sigma_max - sigma_min) / sigma_step ) result images will be saved in that directory.
*Select a directory for <b>Output directory</b>. The ( <i>nb_input_images * (sigma_max - sigma_min) / sigma_step</i> ) result images will be saved in that directory.
*Make sure the checkbox <b>Run using Grid/Condor</b> is unchecked as we want to run the module locally
*Make sure the checkbox <b>Run using Grid/Condor</b> is unchecked as the module must be run locally.
*Click <b>Apply</b> to launch the processing.
*Click <b>Apply</b> to start the processing.
Note: The inputs of the Gaussian Blur filter are the files found in the input directory. If Sigma minimum, Sigma maximum and Sigma step are respectively set to 1, 10 and 1, then 10 files will be created for each input file. For example, if 3 files in the input directory  
Note: The inputs of the Gaussian Blur filter are the files found in the input directory. If Sigma minimum, Sigma maximum and Sigma step are respectively set to 1, 10 and 1, then 10 files will be created for each input file. For 3 files in the input directory  
  imageX.mha, imageY.mha and imageZ.mha  
  imageX.mha, imageY.mha and imageZ.mha  
match the <b>Data file mask</b>, (for example all three matching *.mha); then the output directory will contain after the module is executed :  
matching the <b>Data file mask</b>, *.mha, after execution the output directory will contain :  
  imageX-1.mha, imageX-2.mha ... imageX-10.mha,
  imageX-1.mha, imageX-2.mha ... imageX-10.mha,
  imageY-1.mha, imageY-2.mha ... imageY-10.mha,
  imageY-1.mha, imageY-2.mha ... imageY-10.mha,
Line 38: Line 38:
===How to run Register Images on a Condor grid===
===How to run Register Images on a Condor grid===
In this section, we will see how to register multiple moving images with a single fixed image. The computation will be done on several machines using Condor.  
In this section, we will see how to register multiple moving images with a single fixed image. The computation will be done on several machines using Condor.  
Condor must be already installed on your machine(s) and work properly. You must have created and configured a condor user account on your machine, the condor scripts generated by BatchMake will be executed by the condor user ( +Owner = condor ). For the example, we used the following file/folder hierarchy:
[[http://www.cs.wisc.edu/condor Condor] must be already installed and work properly on your machine(s). A "condor" user account must be created on your machine(s). The condor scripts generated by BatchMake will be executed by the condor user ( <i>+Owner = condor</i> is added to every scripts ). For the tutorial, we used the following file/folder hierarchy:
  /...
  /...
   /project
   /project
Line 52: Line 52:
     /RegisterImagesScripts
     /RegisterImagesScripts
   /applications
   /applications
    /RegisterImages -> /home/user/Slicer3-build/lib/Slicer3/Plugins/RegisterImages
    /bmGridSend ->/home/user/Slicer3-lib/BatchMake-build/bin/bmGridSend
    /bmGridStore ->/home/user/Slicer3-lib/BatchMake-build/bin/bmGridStore
    /bmSliceExtractor ->/home/user/Slicer3-lib/BatchMake-build/bin/bmSliceExtractor
    /...


====Module configuration====
====Module configuration====
*Start Slicer ( as a regular user, there is no need to be a condor user ).
*Start Slicer as a normal user. There is no need to be a condor user.
*Open the [http://www.slicer.org/slicerWiki/index.php/Modules:RegisterImagesBatch-Documentation-3.4 Register Images BatchMake] module located in <b>Batch Processing</b>
*Open the [http://www.slicer.org/slicerWiki/index.php/Modules:RegisterImagesBatch-Documentation-3.4 Register Images BatchMake] module located in <b>Batch Processing</b>
*[http://www.writers.ph/ writer jobs]
<center>
<center>
{|
{| style="width:100%"
|[[Image:RegistrationBatchMake-Selection.png|thumb|400px|Selection of the Register Images Batch module]]
|[[Image:RegistrationBatchMake-Selection.png|thumb|center|400px|Selection of the Register Images Batch module]]
|[[Image:RegisterImagesBatchMakeModule-RigidRegistration.png|thumb|400px|Register Images Batch GUI]]
|[[Image:RegisterImagesBatchMakeModule-RigidRegistration.png|thumb|center|400px|Register Images Batch GUI]]
|}
|}
</center>
</center>
*In the IO section, select an input <b>Fixed Image</b> file. All the moving image will be registered with it.  
*In the <b>IO section</b>, select the input <b>Fixed Image</b> file. All the moving images will be registered with it.  
*Then select a directory for <b>Moving Image Directory</b>. The moving images can be in its subdirectories.  
*Then select a directory for <b>Moving Image Directory</b>. The moving images can be in subdirectories.
*Select a mask to filter the moving images from the <b>Moving Image Directory</b>. If the moving images are in sub-directories, the mask shall be a relative path. The sub-directories can be wildcarded as well as the file names( i.e. "Patient*/Image*.mha")
*Select a mask to filter the moving images in the <b>Moving Image Directory</b>. If the moving images are in subdirectories, the mask must be a relative path. The subdirectories can be wildcarded as well as the file names: "<i>Patient*/Image*.mha</i>".
*Select an output directory for <b>Resample Image</b> where the output images will be saved.
*Select an output directory for <b>Resample Image</b> where the output images will be saved.
*A mask describes how to generate the filename of the output images based on the input parameters. The XML variable names can be used by enclosing their name into a dollar sign and braces, i.e. "Registered-${rigidMaxIterations}-${fixedImage}".  
*A mask describes how to generate the filenames of the output images based on the input parameters values. The XML variables names must be enclosed into a dollar sign and braces, i.e. "Registered-${rigidMaxIterations}-${movingImage}", the output files will have the names of Registered-10-Normal002-T2-resampled.mha, Registered-60-Normal002-T2-resampled.mha...
*Fill the parameters of the next module sections: Registration, Advanced Registration, Registration Testing... .
*Choose the parameters of the following module sections: Registration, Advanced Registration, Registration Testing... [[http://www.slicer.org/slicerWiki/index.php/Modules:RegisterImages-Documentation-3.4#Parameter_Choices]] can help you to choose the best values.
*The <b>Advanced Rigid</b>, <b>Affine</b> and <b>BSpline Registration</b> parameters use a min/max/step mechanism. This allow you to automatically vary the module parameters. By defining a min, max and step inputs for a single parameter (<b>Max Iterations</b>), the Register Image module will be run (max-min)/step times for each inputs with a different <b>Rigid</b> (or <b>Affine</b> or <b>BSpline</b>) <b>Max Iterations</b> value. Please note that ${rigidMaxIterations} must be in the <b>Resample Image</b> mask in order to generate unique output file names at each step. We will see in a next section, how the module parameters can be modified in the XML file to automatically support an iteration over the parameter.
*The <b>Advanced Rigid</b>, <b>Affine</b> and <b>BSpline Registration</b> parameters use a min/max/step mechanism. This allow you to automatically vary the module parameters. By defining a min, max and step inputs for a single parameter (<b>Max Iterations</b>), the Register Image module is run (max-min)/step times for each input with a different <b>Rigid</b> (or <b>Affine</b> or <b>BSpline</b>) <b>Max Iterations</b> value. Please note that ${rigidMaxIterations} must be in the <b>Resample Image</b> mask in order to generate unique output file names at each step. Look at [[#How_to_vary_parameters_.3F How to vary parameters ?]] to know how to modify the module parameters in the XML file for the support of parameter variation.


==== Condor configuration ====
==== Condor configuration ====
Once all the module parameters are set up, the <b>BatchMake</b> section must be configured.
Once all the module parameters are set up, the <b>BatchMake</b> section must be configured.
* Check the <b>condor</b> box for the running mode.
* Check the <b>condor</b> box for the running mode.
* Select a directory for <b>Grid Input directory</b>. If the option <b>Transfer file</b> is set to NONE (recommended), the selected grid input directory must exist and be accessible by all the condor machines. The directory can be local or on the network. The <b>Grid Input directory</b> is the root directory for all the required inputs of the module. For the Register Images module, it's the root directory of the <b>Fixed Image</b> AND the <b>Moving Images</b>. Here, <b>Grid Input directory</b> has the value of <i>/.../project/RegisterImagesInput</i>.
* Select a directory for <b>Grid Input directory</b>. If <b>Transfer file</b> is different from <b>ALL</b> and <b>INPUT_FILES</b>(recommended), the selected grid input directory must exist and be accessible by all the condor machines. The directory can be local or on the network. The <b>Grid Input directory</b> is the root directory for all the input files of the module. For the Register Images module, it's the directory the <b>Fixed Image</b> AND the <b>Moving Images</b> have in common: "<i>/.../project/RegisterImagesInput</i>".
* Select an output directory for <b>Grid Output directory</b>. It is the root directory for all the generated files by the module. It must have the write rights for all the condor machines. In our example, <b>Grid Output directory</b> has the same value than <b>Resample Image Directory</b>, <i>/.../project/RegisterImagesOutput</i>.
* Select an output directory for <b>Grid Output directory</b>. It is the directory of all the generated files by the module. It must have the writing rights for all the condor machines. In our example, <b>Grid Output directory</b> has the same value than <b>Resample Image Directory</b>: "<i>/.../project/RegisterImagesOutput</i>".
* <b>Grid Executable directory</b> is the directory where all the applications are. It must be the same for every condor machines. Applications can be symbolic links (recommended). The value given in our example is "<i>/.../applications</i>".
* <b>Grid Working directory</b> is a directory where all the files generated by condor (bmcondor.??.bmc.tmp, bmcondor.bmc.tmp.dagman.out, bmGrid.out.txt...)  are stored. These files are useful to debug.
* <b>Transfer Files</b> shall be set to <i>None</i>. However, condor experts, can use a different value.
* Click on <b>Apply</b> to generate the condor script and run condor.


===How to vary the module parameters ?===
===How to vary parameters ?===
As we have seen in the previous section <b>Module Configuration</b>, it is possible to vary the parameters values. The default varying parameters in Register Images BatchMake module are the <b>Rigid</b>, <b>Affine</b> and <b>BSpline Max Iterations</b> parameters, but you can change RegisterImagesBatchMakeModule.xml to have more varying parameters. RegisterImagesBatchMakeModule.xml can be found in <i>Slicer3/Applications/CLI/BatchMakeApplications/RegisterImagesBatchMakeModule</i>". The variation mechanism works by adding a min, a max and a step suffix to a scalar parameter:
{|
! Static parameter
! Varying parameter
|- valign="top"
|
<pre><nowiki>
<integer>
  <name>rigidMaxIterations</name>
  <description>Maximum number of rigid optimization iterations</description>
  <label>Rigid Max Iterations</label>
  <longflag>rigidMaxIterations</longflag>
  <default>100</default>
</integer>
</nowiki></pre>
|
<pre><nowiki>
<integer>
  <name>rigidMaxIterations_Min</name>
  <description>Maximum number of rigid optimization iterations</description>
  <label>Rigid Max Iterations (min)</label>
  <longflag>rigidMaxIterations_Min</longflag>
  <default>100</default>
</integer>
<integer>
  <name>rigidMaxIterations_Max</name>
  <description>Maximum number of rigid optimization iterations</description>
  <label>Rigid Max Iterations (max)</label>
  <longflag>rigidMaxIterations_Max</longflag>
  <default>100</default>
</integer>
<integer>
  <name>rigidMaxIterations_Step</name>
  <description>Maximum number of rigid optimization iterations</description>
  <label>Rigid Max Iterations (step)</label>
  <longflag>rigidMaxIterations_Step</longflag>
  <default>50</default>
</integer>
</nowiki></pre>
|}


===How to write your own Batch Processing module ?===
===How to write your own Batch Processing module ?===
The BatchMake modules running on Condor don't support the <image> tags in the module XML files. Slicer pass <image> arguments to the modules the memory address where the image is loaded and not the filename of the image.  as <im they must be changed into <file> or <directory>+<string>. Share
Writing a Batch Processing module is fast and easy.
* Create a new directory in <i>Slicer3/Applications/CLI/BatchMakeApplications</i> by suffixing the module name by BatchMakeModule.
* Copy-Paste a <i>CMakeLists.txt</i> file from another Batch module into your directory. Replace any occurrence of the copied module name with the name of the new module.
* Copy-Paste your module XML file into your created directory and suffix its name with "BatchMakeModule".
* Replace any <image> tags in the XML file by <file> (or <directory>+<string>). The BatchMake modules running on Condor don't support the <image> tags in the module XML files. With the <image> tag, Slicer pass to the modules the memory address where the image is loaded. It wouldn't work when running the module on different machines with Condor.
{|
! Parameter in module
! Parameter in BatchMake module
|- valign="top"
|
<pre><nowiki>
<image>
<name>fixedImage</name>
<label>Fixed Image</label>
<channel>input</channel>
<index>0</index>
<description>Image that defines the space into which
the moving image is registered</description>
</image>
</nowiki></pre>
|
<pre><nowiki>
<file>
<name>fixedImage</name>
<label>Fixed Image</label>
<channel>input</channel>
<index>0</index>
<description>Image that defines the space into which
the moving image is registered</description>
</file>
</nowiki></pre>
|}
To run a module multiple times on different image inputs, the directory/mask mechanism must be used. Suffix the transform the <image> tag into a <directory> and a <string> tag. For each tag, suffix the name of the parameter by <b>_Directory</b> and <b>_Mask</b> respectively:
{|
! Parameter in module
! Parameter in BatchMake module for multiple inputs
|- valign="top"
|
<pre><nowiki>
<image>
<name>movingImage</name>
<label>Moving Image</label>
<channel>input</channel>
<index>1</index>
<description>The transform goes from the fixed image's space
into the moving image's space</description>
</image>
</nowiki></pre>
|
<pre><nowiki>
<directory>
<name>movingImage_Directory</name>
<label>Moving Image Directory</label>
<channel>input</channel>
<index>1</index>
<description>The transform goes from the fixed image's space
into the moving image's space</description>
</directory>
<string>
<name>movingImage_Mask</name>
<label>Moving Image Mask</label>
<longflag>movingImage_Mask</longflag>
<description>The transform goes from the fixed image's space
into the moving image's space</description>
</string>
</nowiki></pre>
|}
* See the previous section [[#How_to_vary_parameters_.3F | How to vary parameters ?]] to use the min/max/step mechanism.
* Add the following BatchMake section into the XML file
<parameters advanced="true">
    <label>BatchMake</label>
    <description>BatchMake specific parameters</description>
    <string-enumeration>
      <name>RunningMode_Grid</name>
      <longflag>runningMode_Grid</longflag>
      <description>Running Mode: Where the module shall be executed? On your local machine or on a grid?</description>
      <label>Running Mode</label>
      <default>local</default>
      <element>local</element>
      <element>condor</element>
    </string-enumeration>
    <directory>
      <name>InputDirectory_Grid</name>
      <label>Grid input directory</label>
      <longflag>inputDirectory_Grid</longflag>
      <default></default>
      <description>Directory where the input data are. Necessary if dealing with relative input paths.</description>
    </directory>
    <directory>
      <name>OutputDirectory_Grid</name>
      <label>Grid output directory</label>
      <longflag>outputDirectory_Grid</longflag>
      <default></default>
      <description>Directory where the output data are. Necessary if dealing with relative output paths.</description>
    </directory>
    <directory>
      <name>ExecutableDirectory_Grid</name>
      <label>Grid executable directory</label>
      <longflag>executableDirectory_Grid</longflag>
      <default></default>
      <description>Directory where the executables are.</description>
    </directory>
    <directory>
      <name>WorkingDirectory_Grid</name>
      <label>Grid working directory</label>
      <longflag>workingDirectory_Grid</longflag>
      <default></default>
      <description>Directory where the condor scripts are written.</description>
    </directory>
    <string-enumeration>
      <name>TransferFile_Grid</name>
      <longflag>transferFile_Grid</longflag>
      <description>Type of files that shall be transfered by the grid</description>
      <label>Transfer File</label>
      <default>NONE</default>
      <element>NONE</element>
      <element>EXECUTABLE</element>
      <element>INPUT_FILES</element>
      <element>OUTPUT_FILES</element>
      <element>ALL</element>
    </string-enumeration>
  </parameters>

Latest revision as of 08:39, 22 September 2010

BatchMake in Slicer

BatchMake has been integrated in Slicer since version 3, and highly improved in version 3.4. The Batch Processing category can be found in the Slicer modules list.
There are 4 modules into the Batch Processing subsection: Gaussian Blur BatchMake, Register Images BatchMake, EMSegment BatchMake and Resample Scalar Volume BatchMake. Each BatchMake module internally call a Slicer module, respectively Gaussian Blur, Register Images, EMSegment Simple and Resample Scalar Volume.

Batch processing allows to automatically run on multiple machines multiple modules multiple times with multiple parameters on multiple inputs.
Since Slicer 3.4, any Slicer modules can easily be transformed to support batch processing. Only minor changes in the module XML file is needed, no code has to be written specifically for a Batch module.
In this tutorial we will present BatchMake in Slicer and give a step by step tutorial to batch process data. We will then explain how to create a BatchMake module based on a Slicer module.

How to run a BatchMake module locally?

Gaussian Blur BatchMake GUI

In this section, we will see how to run on a local machine a Gaussian Blur filter. For a single input image, the module will generate multiple outputs, one for each sigma value.

  • Launch Slicer
  • Open the "Gaussian Blur BatchMake" module located in Batch Processing.
  • Set Sigma Minimum, Sigma Maximum and Sigma Step. They change the number of times the Gaussian Blur module is called with a different sigma. If Sigma minimum, maximum and step are respectively set to 1, 10 and 1; Gaussian Blur will be run 10 times, with a sigma varying from 1 to 10.
  • Then select a directory for Data directory. The directory shall contains the input images to blur.
  • Define a wildcards mask to select the images to blur in Data directory based on the filename and/or extension. Each image that fits the mask requirements is processed separately.
  • Check Recursive if you want to apply the "Data file" mask in the subdirectories of Data directory.
  • Select a directory for Output directory. The ( nb_input_images * (sigma_max - sigma_min) / sigma_step ) result images will be saved in that directory.
  • Make sure the checkbox Run using Grid/Condor is unchecked as the module must be run locally.
  • Click Apply to start the processing.

Note: The inputs of the Gaussian Blur filter are the files found in the input directory. If Sigma minimum, Sigma maximum and Sigma step are respectively set to 1, 10 and 1, then 10 files will be created for each input file. For 3 files in the input directory

imageX.mha, imageY.mha and imageZ.mha 

matching the Data file mask, *.mha, after execution the output directory will contain :

imageX-1.mha, imageX-2.mha ... imageX-10.mha,
imageY-1.mha, imageY-2.mha ... imageY-10.mha,
imageZ-1.mha, imageZ-2.mha ... imageZ-10.mha
Input images
Output images

How to run Register Images on a Condor grid

In this section, we will see how to register multiple moving images with a single fixed image. The computation will be done on several machines using Condor. [Condor must be already installed and work properly on your machine(s). A "condor" user account must be created on your machine(s). The condor scripts generated by BatchMake will be executed by the condor user ( +Owner = condor is added to every scripts ). For the tutorial, we used the following file/folder hierarchy:

/...
  /project
    /RegisterImagesInput
      /Fixed
        /Normal001-T2-resampled.mha
      /Moving
        /Normal002-T2-resampled.mha
        /Normal003-T2-resampled.mha
        /...
        /Normal100-T2-resampled.mha
    /RegisterImagesOutput
    /RegisterImagesScripts
  /applications
    /RegisterImages -> /home/user/Slicer3-build/lib/Slicer3/Plugins/RegisterImages
    /bmGridSend ->/home/user/Slicer3-lib/BatchMake-build/bin/bmGridSend
    /bmGridStore ->/home/user/Slicer3-lib/BatchMake-build/bin/bmGridStore
    /bmSliceExtractor ->/home/user/Slicer3-lib/BatchMake-build/bin/bmSliceExtractor
    /...

Module configuration

Selection of the Register Images Batch module
Register Images Batch GUI
  • In the IO section, select the input Fixed Image file. All the moving images will be registered with it.
  • Then select a directory for Moving Image Directory. The moving images can be in subdirectories.
  • Select a mask to filter the moving images in the Moving Image Directory. If the moving images are in subdirectories, the mask must be a relative path. The subdirectories can be wildcarded as well as the file names: "Patient*/Image*.mha".
  • Select an output directory for Resample Image where the output images will be saved.
  • A mask describes how to generate the filenames of the output images based on the input parameters values. The XML variables names must be enclosed into a dollar sign and braces, i.e. "Registered-${rigidMaxIterations}-${movingImage}", the output files will have the names of Registered-10-Normal002-T2-resampled.mha, Registered-60-Normal002-T2-resampled.mha...
  • Choose the parameters of the following module sections: Registration, Advanced Registration, Registration Testing... [[1]] can help you to choose the best values.
  • The Advanced Rigid, Affine and BSpline Registration parameters use a min/max/step mechanism. This allow you to automatically vary the module parameters. By defining a min, max and step inputs for a single parameter (Max Iterations), the Register Image module is run (max-min)/step times for each input with a different Rigid (or Affine or BSpline) Max Iterations value. Please note that ${rigidMaxIterations} must be in the Resample Image mask in order to generate unique output file names at each step. Look at #How_to_vary_parameters_.3F How to vary parameters ? to know how to modify the module parameters in the XML file for the support of parameter variation.

Condor configuration

Once all the module parameters are set up, the BatchMake section must be configured.

  • Check the condor box for the running mode.
  • Select a directory for Grid Input directory. If Transfer file is different from ALL and INPUT_FILES(recommended), the selected grid input directory must exist and be accessible by all the condor machines. The directory can be local or on the network. The Grid Input directory is the root directory for all the input files of the module. For the Register Images module, it's the directory the Fixed Image AND the Moving Images have in common: "/.../project/RegisterImagesInput".
  • Select an output directory for Grid Output directory. It is the directory of all the generated files by the module. It must have the writing rights for all the condor machines. In our example, Grid Output directory has the same value than Resample Image Directory: "/.../project/RegisterImagesOutput".
  • Grid Executable directory is the directory where all the applications are. It must be the same for every condor machines. Applications can be symbolic links (recommended). The value given in our example is "/.../applications".
  • Grid Working directory is a directory where all the files generated by condor (bmcondor.??.bmc.tmp, bmcondor.bmc.tmp.dagman.out, bmGrid.out.txt...) are stored. These files are useful to debug.
  • Transfer Files shall be set to None. However, condor experts, can use a different value.
  • Click on Apply to generate the condor script and run condor.

How to vary parameters ?

As we have seen in the previous section Module Configuration, it is possible to vary the parameters values. The default varying parameters in Register Images BatchMake module are the Rigid, Affine and BSpline Max Iterations parameters, but you can change RegisterImagesBatchMakeModule.xml to have more varying parameters. RegisterImagesBatchMakeModule.xml can be found in Slicer3/Applications/CLI/BatchMakeApplications/RegisterImagesBatchMakeModule". The variation mechanism works by adding a min, a max and a step suffix to a scalar parameter:

Static parameter Varying parameter
 <integer>
  <name>rigidMaxIterations</name>
  <description>Maximum number of rigid optimization iterations</description>
  <label>Rigid Max Iterations</label>
  <longflag>rigidMaxIterations</longflag>
  <default>100</default>
 </integer>
 <integer>
  <name>rigidMaxIterations_Min</name>
  <description>Maximum number of rigid optimization iterations</description>
  <label>Rigid Max Iterations (min)</label>
  <longflag>rigidMaxIterations_Min</longflag>
  <default>100</default>
 </integer>
 <integer>
  <name>rigidMaxIterations_Max</name>
  <description>Maximum number of rigid optimization iterations</description>
  <label>Rigid Max Iterations (max)</label>
  <longflag>rigidMaxIterations_Max</longflag>
  <default>100</default>
 </integer>
 <integer>
  <name>rigidMaxIterations_Step</name>
  <description>Maximum number of rigid optimization iterations</description>
  <label>Rigid Max Iterations (step)</label>
  <longflag>rigidMaxIterations_Step</longflag>
  <default>50</default>
 </integer>

How to write your own Batch Processing module ?

Writing a Batch Processing module is fast and easy.

  • Create a new directory in Slicer3/Applications/CLI/BatchMakeApplications by suffixing the module name by BatchMakeModule.
  • Copy-Paste a CMakeLists.txt file from another Batch module into your directory. Replace any occurrence of the copied module name with the name of the new module.
  • Copy-Paste your module XML file into your created directory and suffix its name with "BatchMakeModule".
  • Replace any <image> tags in the XML file by <file> (or <directory>+<string>). The BatchMake modules running on Condor don't support the <image> tags in the module XML files. With the <image> tag, Slicer pass to the modules the memory address where the image is loaded. It wouldn't work when running the module on different machines with Condor.
Parameter in module Parameter in BatchMake module
<image>
 <name>fixedImage</name>
 <label>Fixed Image</label>
 <channel>input</channel>
 <index>0</index>
 <description>Image that defines the space into which 
the moving image is registered</description>
</image>
<file>
 <name>fixedImage</name>
 <label>Fixed Image</label>
 <channel>input</channel>
 <index>0</index>
 <description>Image that defines the space into which 
the moving image is registered</description>
</file>

To run a module multiple times on different image inputs, the directory/mask mechanism must be used. Suffix the transform the <image> tag into a <directory> and a <string> tag. For each tag, suffix the name of the parameter by _Directory and _Mask respectively:

Parameter in module Parameter in BatchMake module for multiple inputs
<image>
 <name>movingImage</name>
 <label>Moving Image</label>
 <channel>input</channel>
 <index>1</index>
 <description>The transform goes from the fixed image's space 
into the moving image's space</description>
</image>
<directory>
 <name>movingImage_Directory</name>
 <label>Moving Image Directory</label>
 <channel>input</channel>
 <index>1</index>
 <description>The transform goes from the fixed image's space 
into the moving image's space</description>
</directory>
<string>
 <name>movingImage_Mask</name>
 <label>Moving Image Mask</label>
 <longflag>movingImage_Mask</longflag>
 <description>The transform goes from the fixed image's space 
into the moving image's space</description>
</string>
  • See the previous section How to vary parameters ? to use the min/max/step mechanism.
  • Add the following BatchMake section into the XML file
<parameters advanced="true">
   <label>BatchMake</label>
   <description>BatchMake specific parameters</description>
   <string-enumeration>
     <name>RunningMode_Grid</name>
     <longflag>runningMode_Grid</longflag>
     <description>Running Mode: Where the module shall be executed? On your local machine or on a grid?</description>
     <label>Running Mode</label>
     <default>local</default>
     <element>local</element>
     <element>condor</element>
   </string-enumeration>
   <directory>
     <name>InputDirectory_Grid</name>
     <label>Grid input directory</label>
     <longflag>inputDirectory_Grid</longflag>
     <default></default>
     <description>Directory where the input data are. Necessary if dealing with relative input paths.</description>
   </directory>
   <directory>
     <name>OutputDirectory_Grid</name>
     <label>Grid output directory</label>
     <longflag>outputDirectory_Grid</longflag>
     <default></default>
     <description>Directory where the output data are. Necessary if dealing with relative output paths.</description>
   </directory>
   <directory>
     <name>ExecutableDirectory_Grid</name>
     <label>Grid executable directory</label>
     <longflag>executableDirectory_Grid</longflag>
     <default></default>
     <description>Directory where the executables are.</description>
   </directory>
   <directory>
     <name>WorkingDirectory_Grid</name>
     <label>Grid working directory</label>
     <longflag>workingDirectory_Grid</longflag>
     <default></default>
     <description>Directory where the condor scripts are written.</description>
   </directory>
   <string-enumeration>
     <name>TransferFile_Grid</name>
     <longflag>transferFile_Grid</longflag>
     <description>Type of files that shall be transfered by the grid</description>
     <label>Transfer File</label>
     <default>NONE</default>
     <element>NONE</element>
     <element>EXECUTABLE</element>
     <element>INPUT_FILES</element>
     <element>OUTPUT_FILES</element>
     <element>ALL</element>
   </string-enumeration>
 </parameters>