[Insight-users] Re: [CMake] ctest multithreaded

[Luis Ibanez]

Hi Gaetan,

I agree that it is an interesting feature to be able to run several tests
simultaneously.

It is true that *the large majority* of ITK tests do not depend on the
output from
previous tests. There are however some exceptions, for example, the hole
filling
filters apply their computation on the output of the binary threshold image
filter test.

You may get away with the lack of dependencies if you run a non-clean build,
with the drawback that if test A depends on the output of test B, and
yesterday
test B was broken, then Today's test A will appear as broken, even if the
code
for both A and B is fine today.

Any effort that reduces the computation time of the tests is worth of
consideration,
because that also increases the chances that we will actually run the full
testing
before committing code to the repository ( I say this with the guilt of
having
committed code on Monday without running the full testing...)

The dependencies are probably very few, the challenge is to find them   :-/
and then to make sure that they are never introduced again.  I would guess
that most of the dependencies can be solved by adding the output file  to
the
Data/Input or the Data/Baseline directory, assuming that the file is not too
big.


     Luis



----------------------------------------------------------------------------------------------------------
On 7/25/07, Gaëtan Lehmann <gaetan.lehmann at jouy.inra.fr> wrote:
>
> Selon Mathieu Malaterre <mathieu.malaterre at gmail.com>:
>
> > Hi Alex "hanfei",
> >
> >   I believe there is nothing in the ADD_TEST to declare dependencies.
> > So chance is that you would run a test (depending on output from
> > another) *before* the dependant test is run.
> >
> >   Is this correct ?
>
> But that's also the case when using the -I option, or any option which
> allow to
> restrict the set of tests.
>
> The ITK tests for example don't seem to have a dependency on other tests.
> And the itkTestDriver program can now be used to easily run a test program
> and
> test the output images in a single ADD_TEST() command, so there is no
> dependencies on other tests like with ImageCompare.
>
> That's surely not possible for all the projects, but being able to run the
> tests
> in parallel as an option (-j ?) would definitively be a nice feature.
>
> >
> > -Mathieu
> >
> > On 7/24/07, Dr. Alex. GOUAILLARD <hanfei at caltech.edu> wrote:
> > > hi all,
> > >
> > > Annoyed by the time needed by the ITK test suite to run, I made a
> little
> > > test trying to use ctest in a multithreaded fashion (thanks to the -I
> > > option). Attached is a little test that is yet neither automatic or
> > > complete, but that already give nice results on windows. It could be
> > > directly extended to deal with the memcheck command, which would be
> > > great too (as it is even slower).
> > >
> > > compile it, copy it in your build directory, and it should work fine.
> > >
> > > It first fake running ctest on the first test, for you to read the
> total
> > > number of tests. It is then prompting you for that number, and launch
> 8
> > > threads (you can change that number in the code) each running
> > > MaxNumberOfTest / 8 tests. It prints the logs in files named
> > > ctest.log.<ThreadNumber>
> > >
> > > pros:
> > > -> On my 8 cores (2 4-cores) machine the gain is indeed a factor 8.
> 1012
> > > ITK tests in 5 mn (yum!)
> > >
> > > cons:
> > > -> cmake is trying to write on some temp files, and all but the first
> > > fail to do that.
> > > -> I did not address merging and writing the resulting xml files for
> > > dashboard submission. I need to figure how to redirect each individual
> > > ctest xml output first, then parsing and merging should not really be
> an
> > > issue.
> > >
> > > questions:
> > > -> how to use ITK framework to make it plateform independent?
> > > -> how to get the number of cores automatically?
> > > -> how to be compliant with the existing 1: writing temp files?
> > > -> how to be compliant with the existant 2: how to redirect the xml
> output?
> > >
> > > alex "hanfei"
> > >
> > > _______________________________________________
> > > CMake mailing list
> > > CMake at cmake.org
> > > http://www.cmake.org/mailman/listinfo/cmake
> > >
> > >
> >
> >
> > --
> > Mathieu
> > _______________________________________________
> > Insight-users mailing list
> > Insight-users at itk.org
> > http://www.itk.org/mailman/listinfo/insight-users
> >
>
>
> _______________________________________________
> CMake mailing list
> CMake at cmake.org
> http://www.cmake.org/mailman/listinfo/cmake
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://public.kitware.com/pipermail/insight-users/attachments/20070725/6379374b/attachment.htm