[Insight-developers] Test framework
Bill Lorensen
bill.lorensen at gmail.com
Tue Dec 30 08:41:14 EST 2008
I have looked a bit more at both proposed test frameworks. I'm not
sure how helpful they will be for itk testing.
For example, the google testing has, as Dan pointed out, several ctest
like features. Well, we don't need ctest features in another test
framework since we already have ctest.
Also, google test can run many tests from one executable. We also do
that. Because of the way we do cdash reporting, I think we would still
run the tests one at a time.
I think we would still have to run the new tests from our current test
executables to take advantage of ctest control and cdash reporting.
This would still permit regression testing and thread control.
Finally, for many of our classes, writing a test is very similar to
using the class in an application. This is a good exercise for the
developer and illustrates how the filter can be used. I think the
google tests appear a bit more cryptic.
Perhaps we should look at the two proposed frameworks and provide some
of their ideas into our testing framework.
Bill
On Tue, Dec 30, 2008 at 5:33 AM, Gaëtan Lehmann
<gaetan.lehmann at jouy.inra.fr> wrote:
>
> Le 29 déc. 08 à 22:12, Blezek, Daniel J. a écrit :
>
>> Yep, it is cool. We've been using it extensively and I really like it.
>> On reflection, I'm not sure it will be a drop in replacement for the
>> test routines currently in use, without some messing around with the
>> internals. The framework manages test discovery and execution and
>> leaves it intentionally opaque. However, a good strategy might be to
>> leave the existing tests in place and start using a new test framework,
>> i.e., make itkCommonTests3 use the new framework.
>>
>
> Hi,
>
> I don't think we should fully switch to a new test framework, even if it's
> limited to the new tests.
> The current framework has some good points that no other test framework
> would have currently, like the expected/produced images comparison in cdash
> for example.
>
> My understanding is that Bill was looking for a unit test framework to
> complete the current test suite with unit tests, wich is nice IMO, not to
> fully replace the current one. Right?
>
> I actually have some tests which would benefit of such framework - like
> http://voxel.jouy.inra.fr/darcs/contrib-itk/binaryAttributeMorphology/attribute_values_test.cxx,
> one of the worth I've ever wrote (works in Debug mode only and, according to
> the result on the IJ, doesn't pass on other computers than mine).
>
> Gaëtan
>
>
>> I documented our experiences with Google Test on the wiki as you
>> suggested:
>> http://www.itk.org/Wiki/Proposals:Increasing_ITK_Code_Coverage#Google_Te
>> st
>>
>> Cheers,
>> -dan
>>
>> -----Original Message-----
>> From: insight-developers-bounces at itk.org
>> [mailto:insight-developers-bounces at itk.org] On Behalf Of Bill Lorensen
>> Sent: Monday, December 29, 2008 12:17 PM
>> To: Daniel Blezek
>> Cc: insight-developers at itk.org
>> Subject: Re: [Insight-developers] Test framework
>>
>> Dan,
>>
>> Sounds cool. Can you add some info to:
>>
>> http://www.itk.org/Wiki/Proposals:Increasing_ITK_Code_Coverage
>>
>> Bill
>>
>> On Mon, Dec 29, 2008 at 1:14 PM, Daniel Blezek <daniel.blezek at gmail.com>
>> wrote:
>>>
>>> Hi Bill,
>>>
>>> I would add the Google Test framework to the mix. We've been using
>>> it extensively in our lab. It meets 2, 3, 4, 5, 6, 7. Not sure about
>>
>>> platform support across _all_ ITK's platform, but I've never had a
>>> compile warning from it nor any memory leaks. Given that they support
>>
>>> several embedded platforms, I'd bet it compiles cleanly on all our
>>> platforms. It's a very minimalist library, mostly implemented through
>>
>>> macros and supports many of the nice ctest features, i.e. run every
>>> N'th test, run all matching tests, etc.
>>>
>>> http://code.google.com/p/googletest/
>>>
>>> I found a CMake macro that parses the source code and automatically
>>> adds tests making it trivial to add new tests:
>>>
>>> macro(ADD_GOOGLE_TESTS executable)
>>> foreach ( source ${ARGN} )
>>> file(READ "${source}" contents)
>>> string(REGEX MATCHALL "TEST_?F?\\(([A-Za-z_0-9 ,]+)\\)"
>>> found_tests
>>> ${contents})
>>> foreach(hit ${found_tests})
>>> string(REGEX REPLACE ".*\\(([A-Za-z_0-9]+)[,
>>
>> ]*([A-Za-z_0-9]+)\\).*"
>>>
>>> "\\1.\\2" test_name ${hit})
>>> add_test(${test_name} ${executable} --gtest_filter=${test_name}
>>> ${MI3CTestingDir})
>>> endforeach(hit)
>>> endforeach()
>>> endmacro()
>>>
>>>
>>> # Add all tests found in the source code, calling the executable to
>>> run them add_google_tests ( ${EXECUTABLE_OUTPUT_PATH}/NoOp
>>> ${mi3cTestSource})
>>>
>>> The main looks like this:
>>>
>>> #include <gtest/gtest.h>
>>>
>>> int main(int argc, char* argv[])
>>> {
>>> testing::InitGoogleTest ( &argc, argv );
>>> return RUN_ALL_TESTS();
>>> }
>>>
>>> and a test looks like this (we use MD5 hashes to test for pass/fail):
>>>
>>> #include <iostream>
>>>
>>> #include <mi3cImage.h>
>>> #include <mi3cImageLoader.h>
>>> #include <gtest/gtest.h>
>>> #include <NoOp.h>
>>>
>>> TEST(IO, LoadCT) {
>>> mi3c::ImageLoader loader;
>>> mi3c::Image::Pointer image = loader.setFilename ( dataFinder.getFile
>>
>>> ( "CT.hdr" ) ).execute();
>>> ASSERT_EQ ( "c1d43aaa5b991431a9daa1dc4b55dbb1", image->getMD5() ); }
>>>
>>>
>>> Fixtures are also supported to remove redundant code.
>>>
>>> Cheers,
>>> -dan
>>>
>>> On Mon, Dec 29, 2008 at 10:35 AM, Bill Lorensen
>>> <bill.lorensen at gmail.com>
>>> wrote:
>>>>
>>>> I agree that we should look at a unit test facility.
>>>>
>>>> We should start a list of requirements. For example,
>>>>
>>>> 1) Must support all itk platforms.
>>>> 2) We must be able to distribute it with itk.
>>>> 3) It must fit within our test harness facility. Recall that we try
>>>> to minimize the number of executables by combining large numbers of
>>>> tests into FooTests.cxx files.
>>>> 4) It must be compatible with cmake/ctest/cdash. For example, a test
>>>> must be able to "return EXIT_SUCCESS" and "return EXIT_FAILURE".
>>>> 5) It should not add complexity to an already complex testing
>>
>> process.
>>>>
>>>> 6) It must be compatible with itk's strict attention to compile
>>>> warnings and dynamic memory anlysis. In other words, it must not
>>>> produce warnings or purify defects.
>>>> 7) It should have a minimal source footprint.
>>>>
>>>> These are just a few requirements off the top of my head. I'll add
>>>> them to the new page when I get a chance,
>>>>
>>>> Bill
>>>>
>>>> On Mon, Dec 29, 2008 at 10:47 AM, Luis Ibanez
>>>> <luis.ibanez at kitware.com>
>>>> wrote:
>>>>>
>>>>> Hi Mathieu,
>>>>>
>>>>> Minor correction:
>>>>>
>>>>> The text of the UnitTestCpp license:
>>>>>
>>>>> https://unittest-cpp.svn.sourceforge.net/svnroot/unittest-cpp/UnitT
>>>>> est++/COPYING
>>>>>
>>>>> doesn't correspond to a BSD license:
>>>>> http://www.opensource.org/licenses/bsd-license.php
>>>>>
>>>>> but to an MIT license:
>>>>> http://www.opensource.org/licenses/mit-license.php
>>>>>
>>>>> which is a technicality after all, since both licenses could be
>>
>> combined
>>>>>
>>>>> without any conflict. :-)
>>>>>
>>>>>
>>>>> -----
>>>>>
>>>>>
>>>>> Could you tell us more about your experience with UnitTestCpp ?
>>>>>
>>>>> or if you prefer, could you add your comments to the Wiki page:
>>>>>
>>>>> http://www.itk.org/Wiki/Proposals:Increasing_ITK_Code_Coverage#Unit
>>>>> TestCpp
>>>>>
>>>>>
>>>>> The size of the package looks fine. It could fit neatly in
>>>>> Insight/Utilities...
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>> Luis
>>>>>
>>>>>
>>>>> --------------------------
>>>>> Mathieu Malaterre wrote:
>>>>>>
>>>>>> On Mon, Dec 29, 2008 at 5:29 AM, Steve M. Robbins
>>>>>> <steve at sumost.ca>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> On Wed, Dec 10, 2008 at 03:51:56PM -0500, Bill Lorensen wrote:
>>>>>>>
>>>>>>>
>>>>>>>> Given the success of "Adopt a Bug", maybe we should initiate a
>>>>>>>> "Cover a Class".
>>>>>>>
>>>>>>> Now that I have some holiday time, I'd like to raise the issue of
>>
>>>>>>> unit testing frameworks.
>>>>>>>
>>>>>>> The few test codes that I've looked at are all written out
>>>>>>> long-hand, without benefit of a modern unit testing framework
>>>>>>> such as CppUnit or Boost.Test.
>>>>>>
>>>>>>
>>>>>> ... or UnitTestCpp :)
>>>>>>
>>>>>> http://unittest-cpp.sourceforge.net/
>>>>>>
>>>>>> Which comes with a BSD like license ;)
>>>>>>
>>>>>>
>>>>>>
>>>>>> https://unittest-cpp.svn.sourceforge.net/svnroot/unittest-cpp/Unit
>>>>>> Test++/COPYING
>>>>>>
>>>>>> I wrote a cmakelists.txt file for it:
>>>>>>
>>>>>> http://gdcm.svn.sf.net/viewvc/gdcm/Sandbox/UnitTest%2B%2B/
>>>>>>
>>>>>> src files are very lightweight "du -sk src" return 712
>>>>>>
>>>>>>
>>>>>> 2cts
>>>>>
>>>> _______________________________________________
>>>> Insight-developers mailing list
>>>> Insight-developers at itk.org
>>>> http://www.itk.org/mailman/listinfo/insight-developers
>>>
>>>
>> _______________________________________________
>> Insight-developers mailing list
>> Insight-developers at itk.org
>> http://www.itk.org/mailman/listinfo/insight-developers
>> _______________________________________________
>> Insight-developers mailing list
>> Insight-developers at itk.org
>> http://www.itk.org/mailman/listinfo/insight-developers
>
> --
> Gaëtan Lehmann
> Biologie du Développement et de la Reproduction
> INRA de Jouy-en-Josas (France)
> tel: +33 1 34 65 29 66 fax: 01 34 65 29 09
> http://voxel.jouy.inra.fr http://www.mandriva.org
> http://www.itk.org http://www.clavier-dvorak.org
>
>
More information about the Insight-developers
mailing list