ITK/Release 4/Modularization/ Add tests

From KitwarePublic
Jump to navigationJump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

The safe assumption behind testing is that "if it is not tested, it is broken".

Non-regression tests

The same way it used to be, except the newer syntax in "add_test" command:

<syntaxhighlight lang="cmake">

add_executable(itkBinaryThresholdFilterTest itkBinaryThresholdFilterTest.cxx)
target_link_libraries(itkBinaryThresholdFilterTest ${ITK-BinaryThreshold_LIBRARIES})
     
# adopt the modern CMake syntax :[NAME, COMMAND]
itk_add_test(NAME itkBinaryThresholdFilterTest  
        COMMAND itkBinaryThresholdFilterTest input.png output.png)

</syntaxhighlight>


Regression tests

Approach A

Using the test driver macro The CMake macro "CreateTestDriver" defined in

 Module/Core/TestKernal/CreateTestDriver.cmake

is designed for grouped regression tests in ITK.

All the tests in one module share one test driver.

<syntaxhighlight lang="cmake">

 set(ITKFooFilterTests
    itkFooFilterTest1.cxx
    itkFooFilterTest2.cxx)
 CreateTestDriver(ITKFooFilter  "${ITK_FooFilter-Test_LIBRARIES}"  "${ITK-FooFilterTest}")
 itk_add_test(NAME  itkFooFilterTest1
          COMMAND  ITKFooFilterTestDriver  itkFooFilterTest1)

 itk_add_test(NAME  itkFooFilterTest2
          COMMAND  ITKFooFilterTestDriver  itkFooFilterTest2)


</syntaxhighlight>


You can find Approach A used in most of the ITK modules. for example in

   ITK/Modules/IO/BMP/test/CMakeLists.txt

Approach B

Stand-alone regression tests

There is also a test driver executable("itkTestdriver") designed for stand-alone regression tests. For instance, all the tests in Examples are stand-alone programs each has its own main function call, in which case, the approach A is not appropriate.


<syntaxhighlight lang="cmake">

  add_exectuable(itkFooTest itkFooTest.cxx)
  target_line_libraries( itkFooTest ${ITK_LIBRARIES})
  itk_add_test(NAME  itkFooTest
           COMMAND  itkTestDriver
           --compare  outputBaseline.png output.png
                      $<TARGET_FILE:itkFooFilterTest> input.png output.png)

</syntaxhighlight>


The new syntax

    $<TARGET_FILE: xxx> 

in the add_test command triggers CMake to locate the executable automatically. No need to specify the path for the executable here.

You can find Approach B used in Examples.

For example in

 ITK/Examples/Filtering/test/CMakeLists.txt


Add testing data to ITK

  Please refer to: | Add Data

Writing tests

Unit tests

Writing a unit test helps ensuring that the class being exercised

  • works as expected,
  • works invariably along time,
  • works invariably across different platforms.

It also helps in detecting potential issues, and checking how a given class is affected by changes introduced in the code.

A direct consequence of writing and executing a test is that the code coverage is improved. When writing a unit test you should consider the following principles:

  • Make sure that all methods of the class being tested are exercised.
  • Make sure that if the class is templated, a number of different instances with different arguments are created so that the class is thoroughly tested.
  • Make sure your test returns an EXIT_SUCCESS value when succeeding and an EXIT_FAILURE value when failing.
    • When a test checks multiple features that individually may return either an EXIT_SUCCESS or an EXIT_FAILURE code, make sure that the whole test is run, and return the code in the end. This will help in locating more issues in the same run.

<source lang="cpp"> // appropriate include files

int main( int, char *[] ) {

 int testPassStatus = EXIT_SUCCESS;
 //
 // ...
 //
 if( itk::Math::FloatDifferenceULP( floatRepresentationfx1.asFloat, floatRepresentationfx2.asFloat ) != -1 )
   {
   std::cout << "Unexpected float distance." << std::endl;
   testPassStatus = EXIT_FAILURE;
   }
 if( itk::Math::FloatAlmostEqual( floatRepresentationfx1.asFloat, floatRepresentationfx2.asFloat ) )
   {
   std::cout << "floatRepresentationfx1 is almost equal to floatRepresentationfx2\n" << std::endl;
   }
 else
   {
   std::cout << "floatRepresentationfx1 is NOT almost equal to floatRepresentationfx2\n" << std::endl;
   testPassStatus = EXIT_FAILURE;
   }
 return testPassStatus;

} </source>

instead of

<source lang="cpp"> // appropriate include files

int main( int, char *[] ) {

 int testPassStatus = EXIT_SUCCESS;
 //
 // ...
 //
 if( itk::Math::FloatDifferenceULP( floatRepresentationfx1.asFloat, floatRepresentationfx2.asFloat ) != -1 )
   {
   std::cout << "Unexpected float distance." << std::endl;
   return EXIT_FAILURE;
   }
 if( itk::Math::FloatAlmostEqual( floatRepresentationfx1.asFloat, floatRepresentationfx2.asFloat ) )
   {
   std::cout << "floatRepresentationfx1 is almost equal to floatRepresentationfx2\n" << std::endl;
   }
 else
   {
   std::cout << "floatRepresentationfx1 is NOT almost equal to floatRepresentationfx2\n" << std::endl;
   return EXIT_FAILURE;
   }
 return testPassStatus;

} </source>

  • When an exception can be thrown by a class, write a condition (in a try/catch block) in the test that is able to reproduce the exception condition.
  • When appropriate, exercise a given method a number of times (with different arguments) so that the method is thoroughly tested.
  • When required, print messages to the standard output (or error output) so that any errors happened can be checked by a human reader.
  • Whenever possible, test against a baseline (ground truth): if an object is not tested against a baseline, we will not know whether it is working as expected, even if its code coverage is 100%.
  • Use the testing macros defined in itkTestingMacros.h. Particularly, make sure that the macro EXERCISE_BASIC_OBJECT_METHODS is called right after the creation of the object whose class is being tested.