[Insight-developers] Seeking advice about long running tests

[Bill Lorensen]

I think it is OK to drop the iterations in the Debug tests. We are still
exercising the code. If we need alternate baselines, we will still catch
exceptions. This will also allow more valgrind tests to run to completion.



On Fri, Aug 2, 2013 at 8:30 AM, Bradley Lowekamp <blowekamp at mail.nih.gov>wrote:

> Bill,
>
> I think the compiled tests should be the same in release and debug. Need
> to be able to debug the release tests!
>
> However, I think most of the not so fast FastMarching tests should be made
> shorter.
>
> Simular, to what you are saying, but at the CMake level [1] with the
> CONFIGURATIONS option specified to Release|RelMinSize|RelDebInfo....
>
> [1] http://www.cmake.org/cmake/help/v2.8.11/cmake.html#command:add_test
>
> Brad
>
> On Aug 2, 2013, at 7:58 AM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>
> Dave C,
>
> I agree that these test run way too long in Debug. Perhaps we could
> conditionally modify some parameters if the tests are build Debug. We might
> need additional baselines and the results may look funky, but at least we
> still exercise the code.
>
> #ifndef NDEBUG
> // Set params for debug runs
> #else
> // Set params for normal runs
> #endif
>
> I'll try it out on itkExponentialImageRegistrationTest
>
>
> On Fri, Aug 2, 2013 at 6:17 AM, David Cole <dlrdave at aol.com> wrote:
>
>>  Hi there.
>>
>> First of all: kudos to all of you on dashboard progress, and fixing
>> issues with Windows Debug builds. I have now submitted my first ever
>> entirely green dashboard row using the Microsoft compiler in a Debug build
>> for ITK!
>>
>> However...... 😊
>>
>> I recently started submitting ITK dashboards from “mug.neocisinc” and
>> observe the following today:
>>
>> - 20 longest running ITK tests (today):
>> itkFastMarchingImageFilterTest_wm_multipleSeeds_StrictTopo 2h 37m 54s 70ms
>> itkFastMarchingImageFilterTest_wm_multipleSeeds_NoHandlesTopo 2h 37m 28s
>> 440ms
>> itkFastMarchingImageFilterTest_wm_multipleSeeds_NoTopo 2h 27m 34s 760ms
>> itkBSplineExponentialImageRegistrationTest 22m 9s 470ms
>> itkExponentialImageRegistrationTest 19m 55s 840ms
>> itkFastMarchingImageFilterTest_torus_multipleSeeds_NoHandlesTopo 13m 37s
>> 950ms
>> itkFastMarchingImageFilterTest_torus_multipleSeeds_StrictTopo 13m 34s
>> 700ms
>> itkSimpleImageRegistrationTest 12m 16s 690ms
>> itkSimpleImageRegistrationTest2 11m 32s 560ms
>> itkFastMarchingImageFilterTest_torus_multipleSeeds_NoTopo 11m 4s 70ms
>> itkTimeVaryingVelocityFieldImageRegistrationTest 8m 12s 240ms
>> SpatialObjectToImage2 7m 4s 710ms
>> ImageRegistration15Test 5m 6s 410ms
>> ImageRegistration14Test 3m 56s
>> itkParameterizationQuadEdgeMeshFilterTestb0_c3_s0 3m 47s 950ms
>> itkParameterizationQuadEdgeMeshFilterTestb1_c3_s0 3m 47s 150ms
>> DeformableRegistration13Test1 3m 3s 480ms
>> itkTimeVaryingBSplineVelocityFieldImageRegistrationTest 2m 54s 310ms
>> itkSyNImageRegistrationTest 2m 38s 180ms
>> DeformableRegistration12Test1 2m 27s 350ms
>>
>>
>> (from this Debug build using the Microsoft Visual Studio 2012 compiler:
>> http://open.cdash.org/buildSummary.php?buildid=2985530 )
>>
>> Does anybody in this community have any advice about how I can get the
>> Debug dashboard test suite to run in a more reasonable amount of time?
>> (Other than simply excluding these 20 tests or so by name?)
>>
>> It seems completely ridiculous to me that I have to set a test timeout
>> value of greater than 10 minutes (let alone 2.5 HOURS) for an individual
>> test just for the 10 longest running ITK tests here...... especially when
>> the ENTIRE TEST SUITE finishes in less than 6 minutes for the Release build.
>>
>> Can these tests be pared down to use smaller data, or can they be marked
>> as requiring seriously long timeout values using the test TIMEOUT property,
>> or can they be optionally added by people willing to test such long running
>> tests, or perhaps optionally avoided by people like me who are reluctant to
>> run such long running tests .... ?? There’s got to be something we can do.
>>
>>
>> Thanks,
>> David C.
>>
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Kitware offers ITK Training Courses, for more information visit:
>> http://kitware.com/products/protraining.php
>>
>> Please keep messages on-topic and check the ITK FAQ at:
>> http://www.itk.org/Wiki/ITK_FAQ
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.itk.org/mailman/listinfo/insight-developers
>>
>>
>
>
> --
> Unpaid intern in BillsBasement at noware dot com
>  _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Kitware offers ITK Training Courses, for more information visit:
> http://kitware.com/products/protraining.php
>
> Please keep messages on-topic and check the ITK FAQ at:
> http://www.itk.org/Wiki/ITK_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.itk.org/mailman/listinfo/insight-developers
>
>
>


-- 
Unpaid intern in BillsBasement at noware dot com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.itk.org/pipermail/insight-developers/attachments/20130802/f9d52655/attachment.htm>