View source for ITK/Procedure for Adding a Test (itk 3.x)

__TOC__

Appropriate testing is the most important aspects of writing software.

= How to Add a Test to ITK=

'''NOTE:''' ITKv4 has a new simpler procedure for adding tests. ITKv3's procedure is described [[ITKv3_Procedure_for_Adding_a_Test|here]].

The general procedure involves the following steps

* Write the test
* Add the Test to the ''CMakeLists.txt'' file
* Run the Test locally
* Submit an Experimental build
* Commit the Test

== Write the test ==

=== Naming the test file ===

Unit tests should be named with the name of the class they are testing and the word Test at the end.

For example:

* class itkBMPImageIO will have a test called itkBMPImageIOTest.cxx
* class itkIndex will have a test called itkIndexTest.cxx

Sometimes it may be necessary to have multiple tests, in which case a number will be added after the "Test" string of the name.

For example:

* itkBMPImageIOTest.cxx
* itkBMPImageIOTest2.cxx
* itkBMPImageIOTest3.cxx

Note that we skip the "1" entry. It is implicitly the first test.

=== Writing the content ===

The test file must contain a function that has the '''EXACT''' same
name as the file (without the extension).

For example, the file itkBMPImageIOTest.cxx must contain a function

     int itkBMPImageIOTest( int argc, char * argv [] )

The function must have a return value, and must be one of the
following two options:

* EXIT_FAILURE
* EXIT_SUCCESS

'''NOTE:''' Input for tests should reside in ''Testing/Data/Input'' or
''Examples/Data''. Tests should not depend on the output of another
test. All tests are not necessarily run in the same session. So each
test should be able to run with a clean Testing/Temporary directory.

== Test Drivers ==
ITK uses test drivers to manage the very large numbers of unit test. A
test driver aggregates many tests into a single executable by
registering all of them as functions.

Prior to ITKv4, test drivers resided in the source tree and it was the
responsibility of the developer to maintain these files. ITKv4 uses
the CMake test driver generator. The generated test driver is created
by CMake in the binary tree of the test directory.

As a general guideline, there is one Test driver per major
directory. The test driver is named after the directory that it is
testing.

For example, the classes in the directory

    Insight/Code/Algorithms

are tested by files in the directory

    Insight/Testing/Code/Algorithms

and will use a test driver file called

    AlgorithmsCxxTests.cxx

== How to Add a Test ==
'''NOTE: This procedure has changed since ITKv3.'''

=== Add the Test to the List in the CMakelists.txt file ===
Tests are added by listing them in the CMakeLists.txt file of the Testing/Code/KIT directory. Each Testing CMakeLists.txt file follows a convention.

For example, the CMakeLists.txt file in Testing/Code/BasicFilters has this structure:

 set(KIT BasicFilters)
 set(KIT_LIBS ITKIO ITKNumeris ITKStatistics)
Add new tests source files to this list. 
 set(KitTests
   itkAbsImageFilterAndAdaptorTest
   itkAbsoluteValueDifferenceImageFilterTest
 ...
   itkZeroCrossingImageFilterTest
 )

The macro
 CreateTestDriver(${KIT} "${KIT_LIBS}" "${KitTests}" TestDriver)
generates the C++ code for this test driver and place it in the binary directory for the tests (in this example, ITK_BINARY_DIR/Testing/Code/BasicFilters/BasicFiltersCxxTests.cxx)

=== ADD_TEST ===

==== Minimum Case ====

Add a line like the following
  ADD_TEST(itkBMPImageIOTest5
     ${TestDriver} 
     itkBMPImageIOTest2 
     )
where

* itkBMPImageIOTest5 is a symbolic name for the tests. This is the name that ctest will use, and the one that will appear in a Dashboard
* ${TestDriver} is the CMake variable containing the executable for that directory.
* itkBMPImageIOTest2 is the name of the test itself, and it must match the test filename and the name of the function inside the test file.

==== Passing arguments ====

Some tests may require command line arguments. In that case, these arguments can be added after the name of the tests function. For example

  ADD_TEST(itkBMPImageIOTest5
     ${TestDriver} 
     itkBMPImageIOTest2 
     ${ITK_DATA_ROOT}/Input/image_color.bmp
     ${ITK_TEST_OUTPUT_DIR}/image_color.bmp)
     )

will pass two BMP images as arguments to the test.

==== Adding regression Testing ====

Some test may produce images as output, in which case we should add regression testing instructions that make possible to compare the test output against a baseline image.

A typical case will look like:

 ADD_TEST(itkBMPImageIOTest5
     ${TestDriver} 
     --compare ${ITK_DATA_ROOT}/Baseline/IO/image_color.bmp
               ${ITK_TEST_OUTPUT_DIR}/image_color.bmp
     itkBMPImageIOTest2 
     ${ITK_DATA_ROOT}/Input/image_color.bmp
     ${ITK_TEST_OUTPUT_DIR}/image_color.bmp)
     )

Note that the "--compare" string goes just after the name of the executable, and it is followed by the filename of the baseline image and the filename of the test output.

Since different platforms may produce slightly different, but still acceptable, results; the regression testing system allows users to define a tolerance for the comparison. Three different tolerances are available

* --compareIntensityTolerance
* --compareRadiusTolerance
* --compareNumberOfPixelsTolerance


'''compareIntensityTolerance''' sets the intensity difference after which two pixels are considered to be different. For example, if this tolerance is set to 5, then a pixel with value 123 is considered to be the same as pixels with values in the range [118,128].

'''compareNumberOfPixelsTolerance''' defines a Manhattan-type neighborhood around a pixel. When comparing pixel A from one image to pixel B in another image, all pixels in the neighborhood of pixel B are compared against pixel A, The most similar one is selected and then their intendity difference is tested against '''compareIntensityTolerance'''. If the difference is larger, then the pixel "FAILS" the comparison.

'''compareNumberOfPixelsTolerance''' defines how many pixels can be accepted to fail, and still consider the two images to be the same. If more than '''compareNumberOfPixelsTolerance''' failed the comparison, then the test will '''FAIL''' and a difference image between the test output and the baseline will be produced and posted in the Dashboard.

WARNING: Tolerances must be used sparingly. Before adding a tolerance, you must exhaust the options for making sure that there is not a bug in the test, or a bug in the class, that is being revealed by the test failure in that platform. Once you demonstrate that a tolerance is needed, add just enough of a tolerance to make the test pass. Do not over-relax the test, because then it will lose its capacity for detecting real failures in the future, and it will end up providing a false sense of security.

A test with regression verification and tolerances will look like


 ADD_TEST(itkBMPImageIOTest5
     ${IO_TESTS} 
     --compare ${ITK_DATA_ROOT}/Baseline/IO/image_color.bmp
               ${ITK_TEST_OUTPUT_DIR}/image_color.bmp
     --compareIntensityTolerance 5
     --compareRadiusTolerance 1
     --compareNumberOfPixelsTolerance 25
     itkBMPImageIOTest2 
     ${ITK_DATA_ROOT}/Input/image_color.bmp
     ${ITK_TEST_OUTPUT_DIR}/image_color.bmp)
     )

This test will consider two pixels to be different only if their intensity values differ by more than '''5''' units. It will compare a pixels with the 3x3 neighborhood of the other pixel (in 2D), and it will tolerate 25 pixel failures. Pixel failures above that value it will result in the test failing, and a difference image being posted to the Dashboard.


== Run the Test locally ==

Rerun CMake on your project in order to include the new test in the project configuration of your native build system.

Run the test by calling

   ctest -V -R TestName

Where "TestName" is a regular expression that identifies the test or tests that you want to run.


== Submit an Experimental build ==

You can do this by calling

    ctest -D Experimental

or

    make Experimental

or in Windows, Visual Studio, by selecting the Experimental project and invoking "Build" on int.