Proposals:Increasing ITK Code Coverage

From KitwarePublic
Revision as of 17:04, 30 December 2008 by Ibanez (talk | contribs) (→‎Motivation)
Jump to navigationJump to search

Motivation

ITK currently (Dec 20th 2008) has a 80.5% code coverage.

http://www.cdash.org/CDash/viewCoverage.php?buildid=240378

Sloccount report on number of lines of code in the Insight/Code directory returns:

 158,928 lines of code

This means that about 31,000 lines of code are not tested.

We could significantly increase the code coverage of the toolkit, and in the process reduce the number of hidden bugs, by asking volunteers to adopt particular classes and write additional tests for increasing their code coverage. This could be done at the image of the "Adopt a Bug" program.

Infrastructure

It has been pointed out that the current testing infrastructure of ITK impose a high threshold of effort on contributors of new tests.

There are some existing unit test harnesses that might decrease the effort and provide additional functionality. A unit testing package for ITK must meet the following requirements:

  1. It must have an itk-compatible license.
  2. We must be able to distribute it with itk.
  3. It must support all itk platforms.
  4. It must fit within the itk test harness facility. Recall that we try to minimize the number of executables by combining large numbers of tests into FooTests.cxx files.
  5. It must be compatible with cmake/ctest/cdash. For example, a test must be able to "return EXIT_SUCCESS" and "return EXIT_FAILURE".
  6. It must not add complexity to an already complex testing process.
  7. It must be compatible with itk's strict attention to compile warnings and dynamic memory anlysis. In other words, it must not produce warnings or purify defects.
  8. It should have a minimal source footprint.

Suggestions for improving the testing system to make easier for contributors to introduce new tests include

Boost Test

Suggested by Steve Robbins


How it could work

--------------------- itkImageRegionTest.cxx ---------------------------------

#define BOOST_AUTO_TEST_MAIN
#include <boost/test/auto_unit_test.hpp>

#include "itkImageRegion.h"


template< unsigned int VImageDimension >
struct Fixture
{
    typedef itk::ImageRegion<VImageDimension>            RegionType;
    typedef typename RegionType::IndexType               IndexType;
    typedef typename RegionType::SizeType                SizeType;

    RegionType mRegion;
};

struct Fixture1 : public Fixture<1>
{
    Fixture1( int start0,
	      int size0 )
    {
	IndexType start = {{ start0 }};
	SizeType  size  = {{ size0 }};
	mRegion = RegionType( start, size );
    }
};

struct Fixture2 : public Fixture<2>
{
    Fixture2( int start0, int start1,
	      int size0,  int size1 )
    {
	IndexType start = {{ start0, start1 }};
	SizeType  size  = {{ size0,  size1 }};
	mRegion = RegionType( start, size );
    }
};

struct Fixture3 : public Fixture<3>
{
    Fixture3( int start0, int start1, int start2,
	      int size0,  int size1,  int size2 )
    {
	IndexType start = {{ start0, start1, start2 }};
	SizeType  size  = {{ size0,  size1,  size2 }};
	mRegion = RegionType( start, size );
    }
};


BOOST_AUTO_TEST_CASE( testSlice )
{
    Fixture3 volume( 12, 12, 12, 10, 20, 30 );
    Fixture2 slice0( 12, 12, 20, 30 );
    Fixture2 slice1( 12, 12, 10, 30 );
    Fixture2 slice2( 12, 12, 10, 20 );

    BOOST_CHECK_EQUAL( slice0.mRegion, volume.mRegion.Slice( 0 ) );
    BOOST_CHECK_EQUAL( slice1.mRegion, volume.mRegion.Slice( 1 ) );
    BOOST_CHECK_EQUAL( slice2.mRegion, volume.mRegion.Slice( 2 ) );
}
    
BOOST_AUTO_TEST_CASE( testSliceOutOfBounds )
{
    Fixture3 volume( 12, 12, 12, 10, 20, 30 );

    BOOST_CHECK_THROW( volume.mRegion.Slice( -1 ), std::exception );
    BOOST_CHECK_THROW( volume.mRegion.Slice( 3 ), std::exception );
}

BOOST_AUTO_TEST_CASE( testVolumeIsInside )
{
    Fixture3 volumeA( 12, 12, 12, 10, 20, 30 );
    Fixture3 volumeB( 14, 14, 14,  5, 10, 15 );

    BOOST_CHECK(   volumeA.mRegion.IsInside( volumeB.mRegion ) );
    BOOST_CHECK( ! volumeB.mRegion.IsInside( volumeA.mRegion ) );
}

--------------------- itkImageRegionTest.cxx ---------------------------------

Google Test

The Google Test framework is very similar to the Boost test harness. GTest is essentially a set of macros that assist the developer in writing concise tests. The framework does not make use of exceptions, nor templates and is supported on all major platforms and some minor ones, i.e., Cygwin, Windows CE, and Symbian. The code is available under the BSD license. The framework supports many of the features already in place through ctest, e.g., run every N'th test, run all matching tests, etc. The source code and includes are ~600K and trivially compiles using CMake without configuration or modification.

# Build Google Testing
set ( GTestSource
  Utilities/gtest-1.2.1
  Utilities/gtest-1.2.1/src/gtest.cc
  Utilities/gtest-1.2.1/src/gtest-death-test.cc
  Utilities/gtest-1.2.1/src/gtest-filepath.cc
  Utilities/gtest-1.2.1/src/gtest-port.cc
  Utilities/gtest-1.2.1/src/gtest-test-part.cc
  Utilities/gtest-1.2.1/src/gtest-typed-test.cc
)
include_directories ( ${MI3CLib_SOURCE_DIR}/Testing/Utilities/gtest-1.2.1/include )
add_library(gtest ${BUILD_SHARED_LIBS} ${GTestSource})

Test Driver

The test driver is a very simple main function:

#include <gtest/gtest.h>

int main(int argc, char* argv[])
{
  testing::InitGoogleTest ( &argc, argv );
  return RUN_ALL_TESTS();
}

Types of Test

There are two basic types of tests, simple tests using the TEST macro, and test fixtures using the TEST_F macro. Many different macros are available to when executing tests, ranging from string comparisons, expected exceptions, floating point comparisons, etc. The basic framework is well documented with advanced guidance for those who dig deeper. Below is example code from an internal project that demonstrates how to write a test. Text fixtures run as methods in a sub-class of the fixture and have access to all public and protected ivars of the fixture. All test macros function as stream operators with any text directed into them appearing in the output. NB: in this example an MD5 hash is used to verify correct output rather than comparison to a known good image.

TEST(IO, LoadCT) {
  mi3c::ImageLoader loader;
  mi3c::Image::Pointer image = loader.setFilename ( dataFinder.getFile ( "CT.hdr" ) ).execute();
  ASSERT_EQ ( "c1d43aaa5b991431a9daa1dc4b55dbb1", image->getMD5() ) << " failed to load the expected image data";
}


class ImageDataTest : public testing::Test {
public:
  ImageDataTest () {
    image = NULL;
    floatImage = NULL;
  }
  virtual void SetUp() {
    mi3c::ImageLoader loader;
    try {
      image = loader.setFilename ( dataFinder.getFile ( "MRA.hdr" ) ).execute();
      mi3c::ConvertDataType convert ( mi3c::mi3cFLOAT );
      floatImage = convert.execute ( image );
    } catch ( itk::ImageFileReaderException e ) {
      FAIL(); // Couldn't load, so fail this test before we go any further with bad data.
    }
  }
  virtual void TearDown() {
    image = NULL;
    floatImage = NULL;
  }

  mi3c::Image::Pointer image;
  mi3c::Image::Pointer floatImage;
};

TEST_F(ImageDataTest, DiscreteGaussianFilter) {
  mi3c::DiscreteGaussianFilter filter;
  mi3c::Image::Pointer o = filter.execute ( image );
  EXPECT_EQ ( "6adeb490bda64b47e9c1bd6c547e570e", o->getMD5() ) << " Filtered with a gaussian";
  EXPECT_EQ ( "300c7ee796d1b3c2b49a7649789bbf55", filter.execute ( floatImage )->getMD5() ) << " Filtered with a gaussian";
}

TEST_F(ImageDataTest, MeanFilter) {
  mi3c::MeanFilter filter;
  filter.setRadius ( 1 );
  mi3c::Image::Pointer o = filter.execute ( image );
  EXPECT_EQ ( "8b7235e1f8497b0a7fb84eb5c94af00b", o->getMD5() ) << " Mean filtered";
  EXPECT_EQ ( "069a6670309db5c03a79af11a9c6e526", filter.execute ( floatImage )->getMD5() ) << " Mean filtered";
}

Running the tests

Test status is reported (in color) when running the tests and final status is reported as the exit status, much like current ITK testing.

[blezek@mi3bld04 MI3CLib-linux86-gcc]$ bin/NoOp  /mi3c/projects/Source/MI3CTestData
[==========] Running 9 tests from 3 test cases.
[----------] Global test environment set-up.
[----------] 4 tests from ImageDataTest
[ RUN      ] ImageDataTest.MD5
[       OK ] ImageDataTest.MD5
[ RUN      ] ImageDataTest.Threshold
[       OK ] ImageDataTest.Threshold
[ RUN      ] ImageDataTest.DiscreteGaussianFilter
[       OK ] ImageDataTest.DiscreteGaussianFilter
[ RUN      ] ImageDataTest.MeanFilter
[       OK ] ImageDataTest.MeanFilter
[----------] 2 tests from IO
[ RUN      ] IO.LoadCT
[       OK ] IO.LoadCT
[ RUN      ] IO.LoadInvalidFile
[       OK ] IO.LoadInvalidFile
[----------] 3 tests from Image
[ RUN      ] Image.InstantiateImage
[       OK ] Image.InstantiateImage
[ RUN      ] Image.InstantiateImage2
[       OK ] Image.InstantiateImage2
[ RUN      ] Image.TestHash
[       OK ] Image.TestHash
[----------] Global test environment tear-down
[==========] 9 tests from 3 test cases ran.
[  PASSED  ] 9 tests.
[blezek@mi3bld04 MI3CLib-linux86-gcc]$ echo $?
0

CMake Integration

With a slightly clever CMake macro, and a regular expression or two, Google tests are trivially integrated into CMake projects. Here, all TEST and TEST_F macros found in the source code are added as tests to the project. Each test is run as:

NoOp --gtest_filter=TestGroup.TestName

where TestGroup is the first argument to the TEST macro, and TestName is the second.

# C++ tests
set ( mi3cTestSource
  Source/NoOp.cxx
  Source/ImageTests.cxx
  Source/IOTests.cxx
  Source/FilterTests.cxx
  )

add_executable(NoOp ${mi3cTestSource})

macro(ADD_GOOGLE_TESTS executable)
  foreach ( source ${ARGN} )
    file(READ "${source}" contents)
    string(REGEX MATCHALL "TEST_?F?\\(([A-Za-z_0-9 ,]+)\\)" found_tests ${contents})
    foreach(hit ${found_tests})
      string(REGEX REPLACE ".*\\(([A-Za-z_0-9]+)[, ]*([A-Za-z_0-9]+)\\).*" "\\1.\\2" test_name ${hit})
      add_test(${test_name} ${executable} --gtest_filter=${test_name} ${MI3CTestingDir})
    endforeach(hit)
  endforeach()
endmacro()


# Add all tests found in the source code, calling the executable to run them
add_google_tests ( ${EXECUTABLE_OUTPUT_PATH}/NoOp ${mi3cTestSource})

UnitTestCpp


Suggested by Mathieu Malaterre

This package is distributed under an MIT License: