Difference between revisions of "ITK/Release 4/UnitTesting"

From KitwarePublic
< ITK‎ | Release 4
Jump to navigationJump to search
Line 25: Line 25:
 
==== First tests ====
 
==== First tests ====
  
Next, we will create an edit itkGaussianImageFilterTests.cxx:
+
Next, we will create an edit itkRecursiveGaussianImageFilterUnitTests.cxx:
  
 
<pre>
 
<pre>
 
#include "itkTestHarness.h"
 
#include "itkTestHarness.h"
  
TEST(GaussianImageFilter,Recursive)
+
TEST(RecursiveGaussianImageFilter,Basics)
 
{
 
{
 
   ASSERT_TRUE ( true );
 
   ASSERT_TRUE ( true );
 
}
 
}
  
TEST(GaussianImageFilter,Discrete)
 
{
 
  ASSERT_TRUE ( true );
 
}
 
 
</pre>
 
</pre>
  
Line 44: Line 40:
 
  #include "itkTestHarness.h"
 
  #include "itkTestHarness.h"
  
We define two tests:
+
We define a test:
  TEST(GaussianImageFilter,Recursive)
+
  TEST(RecursiveGaussianImageFilter,Basics)
TEST(GaussianImageFilter,Discrete)
 
  
The '''TEST''' macro is provided by Google Test.  The first argument is the TestGroup or TestSuite name.  The second argument is the name of the test to run.  In our case, we have added two tests called GaussianImageFilter.Recursive and GaussianImageFilter.Discrete.
+
The '''TEST''' macro is provided by Google Test.  The first argument is the TestGroup or TestSuite name.  The second argument is the name of the test to run.  In our case, we have added a test called RecursiveGaussianImageFilter.Basics
  
Looking at the body of each test we see:
+
Looking at the body of the test we see:
  
 
  ASSERT_TRUE ( true );
 
  ASSERT_TRUE ( true );
Line 58: Line 53:
 
<pre>
 
<pre>
 
Test project /Users/blezek/Source/ITK-macosx
 
Test project /Users/blezek/Source/ITK-macosx
     Start 3: GaussianImageFilter.Discrete
+
     Start 3: RecursiveGaussianImageFilter.Basics
3/4 Test #3: GaussianImageFilter.Discrete .....  Passed    0.01 sec
+
3/4 Test #3: RecursiveGaussianImageFilter.Basics .......  Passed    0.01 sec
    Start 4: GaussianImageFilter.Recursive
 
4/4 Test #4: GaussianImageFilter.Recursive ....  Passed    0.01 sec
 
 
</pre>
 
</pre>
  
Line 67: Line 60:
  
 
<pre>
 
<pre>
3: Test command: /Users/blezek/Source/ITK-macosx/bin/itkBasicFiltersUnitTests --gtest_filter=GaussianImageFilter.Discrete /Users/blezek/Source/ITK/Testing/Data /Users/blezek/Source/ITK-macosx/Testing/Temporary
+
3: Test command: /Users/blezek/Source/ITK-macosx/bin/itkBasicFiltersUnitTests --gtest_filter=RecursiveGaussianImageFilter.Basics /Users/blezek/Source/ITK/Testing/Data /Users/blezek/Source/ITK-macosx/Testing/Temporary
 
3: Test timeout computed to be: 1500
 
3: Test timeout computed to be: 1500
3: Note: Google Test filter = GaussianImageFilter.Discrete
+
3: Note: Google Test filter = RecursiveGaussianImageFilter.Basics
 
3: [==========] Running 1 test from 1 test case.
 
3: [==========] Running 1 test from 1 test case.
 
3: [----------] Global test environment set-up.
 
3: [----------] Global test environment set-up.
 
3: [----------] 1 test from GaussianImageFilter
 
3: [----------] 1 test from GaussianImageFilter
3: [ RUN      ] GaussianImageFilter.Discrete
+
3: [ RUN      ] RecursiveGaussianImageFilter.Basics
3: [      OK ] GaussianImageFilter.Discrete (0 ms)
+
3: [      OK ] RecursiveGaussianImageFilter.Basics (0 ms)
3: [----------] 1 test from GaussianImageFilter (1 ms total)
+
3: [----------] 1 test from RecursiveGaussianImageFilter (1 ms total)
 
3:  
 
3:  
 
3: [----------] Global test environment tear-down
 
3: [----------] Global test environment tear-down
 
3: [==========] 1 test from 1 test case ran. (1 ms total)
 
3: [==========] 1 test from 1 test case ran. (1 ms total)
 
3: [  PASSED  ] 1 test.
 
3: [  PASSED  ] 1 test.
3/4 Test #3: GaussianImageFilter.Discrete .....  Passed    0.01 sec
+
3/4 Test #3: RecursiveGaussianImageFilter.Basics .....  Passed    0.01 sec
 
</pre>
 
</pre>
  
 
The extra output is generated by Google test.
 
The extra output is generated by Google test.
 +
 
====Test failure====
 
====Test failure====
  

Revision as of 17:34, 16 August 2010

ITK v4 Unit Testing Framework

Many of the existing ITK 3.20 tests were written for the purpose of testing methods and increasing code coverage. These tests accomplish this task exceedingly well, as code coverage for ITK 3.20 testing exceeds 70% of the entire toolkit. Some of the tests evaluate outputs to validate correctness of filter results. In ITK v4, the Google Testing framework will be added to assist developers in writing sound unit tests.

Unit and regression testing concepts

Formal unit and regression testing helps developers of ITK assure correct behavior of the toolkit. Characteristics of a good unit/regression test framework are:

  • Tests clearly test one discrete unit of functionality
  • Tests verify output(s) of code against a regression standard
  • Tests are simple to read and write by developers

The Google Test framework is a well designed and thoroughly documented unit test framework. Google Test (GTest) is a natural fit with the ctest/CDash framework. GTest creates and manages individual tests, ctest executes GTest recording the results and posts to CDash. GTest is also well integrated with CMake.

Unit testing proposal

An integrated Google Test framework has been proposed for ITK v4. This combines Google Testing with ITK specific utilities to aid the developer in writing solid unit tests. A draft implementation has been completed and is available on Github in a topic branch (

Unit testing tutorial

The new unit testing framework leverages GTest and some utility functions specific for ITK. This tutorial will demonstrate many features of the new ITK testing framework (ITKTestHarness). This example show the stages of writing an ITK test, progressing from simple concepts to more complex.

Setup

In this tutorial, we are going to write a test suite or test group for recursive gaussian smoothing available in ITK (itkRecursiveGaussianImageFilter). The first step in the process is to create a new file in the appropriate directory. Our TestGroup will be GaussianImageFilter, so we create the file Testing/Unit/BasicFilters/itkGaussianImageFilterTests.cxx. To let CMake know about our test we edit the CMakeLists.txt file in Testing/Unit/BasicFilters to include our new file:

set(BasicFiltersTestSource 
 itkRecursiveGaussianImageFilterUnitTests.cxx )

First tests

Next, we will create an edit itkRecursiveGaussianImageFilterUnitTests.cxx:

#include "itkTestHarness.h"

TEST(RecursiveGaussianImageFilter,Basics)
{
  ASSERT_TRUE ( true );
}

The first line includes the new ITK test harness.

#include "itkTestHarness.h"

We define a test:

TEST(RecursiveGaussianImageFilter,Basics)

The TEST macro is provided by Google Test. The first argument is the TestGroup or TestSuite name. The second argument is the name of the test to run. In our case, we have added a test called RecursiveGaussianImageFilter.Basics

Looking at the body of the test we see:

ASSERT_TRUE ( true );

ASSERT_TRUE is another Google Test macro. It's function is to ensure the body of the macro evaluates to true. If the body is true, the test continues, if the body evaluates to false, the test stops and prints out an error message. If we compile and run ctest, we see that the tests pass:

Test project /Users/blezek/Source/ITK-macosx
    Start 3: RecursiveGaussianImageFilter.Basics
3/4 Test #3: RecursiveGaussianImageFilter.Basics .......   Passed    0.01 sec

Under the hood, ctest is running Google Tests:

3: Test command: /Users/blezek/Source/ITK-macosx/bin/itkBasicFiltersUnitTests --gtest_filter=RecursiveGaussianImageFilter.Basics /Users/blezek/Source/ITK/Testing/Data /Users/blezek/Source/ITK-macosx/Testing/Temporary
3: Test timeout computed to be: 1500
3: Note: Google Test filter = RecursiveGaussianImageFilter.Basics
3: [==========] Running 1 test from 1 test case.
3: [----------] Global test environment set-up.
3: [----------] 1 test from GaussianImageFilter
3: [ RUN      ] RecursiveGaussianImageFilter.Basics
3: [       OK ] RecursiveGaussianImageFilter.Basics (0 ms)
3: [----------] 1 test from RecursiveGaussianImageFilter (1 ms total)
3: 
3: [----------] Global test environment tear-down
3: [==========] 1 test from 1 test case ran. (1 ms total)
3: [  PASSED  ] 1 test.
3/4 Test #3: RecursiveGaussianImageFilter.Basics .....   Passed    0.01 sec

The extra output is generated by Google test.

Test failure

Now, let's see what happens if we fail a test. Modify the body of GaussianImageFilter.Recursive to be:

TEST(GaussianImageFilter,Recursive)
{
  EXPECT_TRUE ( false ) << "Continue anyway";
  ASSERT_TRUE ( false ) << "Stop running the test";
  ASSERT_TRUE ( false ) << "Never gets here";
}

This introduces the EXPECT_TRUE macro. EXPECT_TRUE evaluates it's body, if it evaluates to false, the test continues, but a failure is recorded for the test. The macro allows a text string to be piped in using the '<<' syntax to record the purpose of the test. In the code above, the test will fail on the EXPECT_TRUE, but continue to the first ASSERT_TRUE. This macro halts the test, so the second ASSERT_TRUE is never run. ASSERT's are typically used when a failure is fatal, while EXPECT's are used when the test failed, but it is safe to run the rest of the test.

When we run ctest we see:

4: [==========] Running 1 test from 1 test case.
4: [----------] Global test environment set-up.
4: [----------] 1 test from GaussianImageFilter
4: [ RUN      ] GaussianImageFilter.Recursive
4: /Users/blezek/Source/ITK/Testing/Unit/BasicFilters/itkGaussianImageFilterTests.cxx:25: Failure
4: Value of: false
4:   Actual: false
4: Expected: true
4: Continue anyway
4: /Users/blezek/Source/ITK/Testing/Unit/BasicFilters/itkGaussianImageFilterTests.cxx:26: Failure
4: Value of: false
4:   Actual: false
4: Expected: true
4: Stop running the test
4: [  FAILED  ] GaussianImageFilter.Recursive (0 ms)
4: [----------] 1 test from GaussianImageFilter (0 ms total)
4: 
4: [----------] Global test environment tear-down
4: [==========] 1 test from 1 test case ran. (0 ms total)
4: [  PASSED  ] 0 tests.
4: [  FAILED  ] 1 test, listed below:
4: [  FAILED  ] GaussianImageFilter.Recursive
4: 
4:  1 FAILED TEST
4/4 Test #4: GaussianImageFilter.Recursive ....***Failed    0.01 sec

In the output above, we can see the 'Continue anyway' message and the line the failure came from (line 25) as well as the 'Stop running the test' message. This helps the developer quickly understand why the test was failing.

Image test

We now edit the test to include the proper header files, and load an image. This code snippit introduces helper functions defined in the ITKTestHarness.

typedef itk::Image<float,3> FloatImage;
TEST(GaussianImageFilter,Recursive)
{
  typedef itk::RecursiveGaussianImageFilter<FloatImage, FloatImage>  GaussianFilterType;
  GaussianFilterType::Pointer filter = GaussianFilterType::New();
  FloatImage::Pointer image = LoadImage<FloatImage> ( dataFinder.GetFile ( "Input/HeadMRVolumeWithDirection.nhdr" ) );
  filter->SetInput ( image );
  ASSERT_EQ ( "", ImageSHA1Hash<FloatImage> ( filter->GetOutput() ) ) << "Failed to match the hash";
}

The first helper is dataFinder. dataFinder is a small helper class to assist in determining file names. The GetFile() method returns a filename in the ITK_DATA_ROOT directory. In this case, we would like to find Input/HeadMRVolumeWithDirection.nrdr. The next helper is LoadImage(). LoadImage is templated over the image datatype, and simply loads an image from the filename passed as an argument. These two helper functions avoid much repetitious code when writing tests.

The third helper function is ImageSHA1Hash(). This helper function computes a [SHA1 hash] of the image, and it's origin, spacing and directions. In this way, we can be sure that a filter produces exactly the same output each time it is run. If we compile and run this code, we see it fails because the hash does not match.

4: [ RUN      ] GaussianImageFilter.Recursive
4: /Users/blezek/Source/ITK/Testing/Unit/BasicFilters/itkGaussianImageFilterTests.cxx:33: Failure
4: Value of: ImageSHA1Hash<FloatImage> ( filter->GetOutput() )
4:   Actual: "93081d5322d29724e8ff49874f6b0d925adb5c1c"
4: Expected: ""
4: Failed to match the hash

If we copy the SHA1 of the image into the test, we have now documented the exact output this filter should produce each time it is run.

ASSERT_EQ ( "93081d5322d29724e8ff49874f6b0d925adb5c1c", ImageSHA1Hash<FloatImage> ( filter->GetOutput() ) ) << "Failed to match the hash";

However, the RecursiveGaussianFilter may produce different results on other platform. This will be examined in the next section.

Image comparison

Different platforms have differing levels of numeric precision. Because of this, a SHA1 hash value will not be valid on all platforms unless the filter is designed to produce exactly the same output independent of numeric precision (morphology filters for instance). The testing harness provides several helper macros that operate exactly like the Google Test macros. These helpers operate on images and require a few extra bits of information.

  EXPECT_IMAGE_NEAR(FloatImage, filter->GetOutput(), "float", 1.0);
  filter->SetSigma( 2.0 );
  filter->SetZeroOrder();
  EXPECT_IMAGE_NEAR(FloatImage, filter->GetOutput(), "sigma_2.0", 1.0);

The macros are:

#define EXPECT_IMAGE_EQ(ImageType, image, name)
#define EXPECT_IMAGE_NEAR(ImageType, image, name, tolerance)
#define ASSERT_IMAGE_EQ(ImageType, image, name)
#define ASSERT_IMAGE_NEAR(ImageType, image, name, tolerance)

The first argument is the type of the image and is used to instantiate the correct reading and comparison classes. The image argument is the actual image produced during this run of the test. The name specifies which image this is. Generally speaking, a test may execute a filter several times producing different images each time. The name is used to construct the baseline image file on disk. The tolerance argument for the "_NEAR" functions provide a threshold for the image difference from baseline. Running the test at this stage produces this output.

4: [ RUN      ] GaussianImageFilter.Recursive
4: /Users/blezek/Source/ITK/Testing/Unit/BasicFilters/itkGaussianImageFilterTests.cxx:33: Failure
4: Value of: imageCompare.Compare ( image, "float", 1.0 )
4:   Actual: false
4: Expected: true
4: Baseline does not exist, wrote /Users/blezek/Source/ITK-macosx/Testing/Temporary/Newbaseline/GaussianImageFilter_Recursive_float.nrrd
4: cp /Users/blezek/Source/ITK-macosx/Testing/Temporary/Newbaseline/GaussianImageFilter_Recursive_float.nrrd 
/Users/blezek/Source/ITK/Testing/Data//Baseline/GaussianImageFilter_Recursive_float.nrrd
4: Value of: imageCompare.Compare ( filter->GetOutput(), "sigma_2.0", 1.0 )
4:   Actual: false
4: Expected: true
4: Baseline does not exist, wrote /Users/blezek/Source/ITK-macosx/Testing/Temporary/Newbaseline/GaussianImageFilter_Recursive_sigma_2.0.nrrd
4: cp /Users/blezek/Source/ITK-macosx/Testing/Temporary/Newbaseline/GaussianImageFilter_Recursive_sigma_2.0.nrrd /Users/blezek/Source/ITK/Testing/Data//Baseline/GaussianImageFilter_Recursive_sigma_2.0.nrrd
4: [  FAILED  ] GaussianImageFilter.Recursive (207 ms)

On the first run, the test failed because no baseline existed. EXPECT_IMAGE_NEAR writes the image to disk in the temporary directory. Since we used the EXPECT version of the macro, the test continued and wrote the second baseline image. The naming convention for the files in TestGroup_Test_name.png; this example wrote GaussianImageFilter_Recursive_float.nrrd and GaussianImageFilter_Recursive_sigma_2.0.nrrd. If we take a look at the image we see (slice 16): GaussianImageFilter Recursive float.png

The image is fairly small because the input image is very small. After we examine these images, we can copy them to the Baseline image directory in ITK_DATA_ROOT and make them available for other to use. The macro helpfully supplies the proper command line:

cp /Users/blezek/Source/ITK-macosx/Testing/Temporary/Newbaseline/GaussianImageFilter_Recursive_float.nrrd 
/Users/blezek/Source/ITK/Testing/Data//Baseline/GaussianImageFilter_Recursive_float.nrrd
cp /Users/blezek/Source/ITK-macosx/Testing/Temporary/Newbaseline/GaussianImageFilter_Recursive_sigma_2.0.nrrd /Users/blezek/Source/ITK/Testing/Data//Baseline/GaussianImageFilter_Recursive_sigma_2.0.nrrd

Now when we run the testing, the test passes.