[Insight-developers] Very suspicious behavior.
Luis Ibanez
luis.ibanez at kitware.com
Fri Feb 25 09:45:02 EST 2011
Hi Hans,
This rings a bell,...
It reminds me of a bug that Andriy identified a while ago,
and that we probably never got to squash.
Andriy: does Hans' description matches what you were
observing in registration with multiple threads ?
Luis
--------------------------------------------------
On Thu, Feb 24, 2011 at 9:29 PM, Johnson, Hans J <hans-johnson at uiowa.edu> wrote:
> Hello ITK Experts:
> I have a really nasty problem that is almost certainly some sort algorithmic
> anomaly that occurs during the creation of multiple threads under the right
> data conditions. The failure is a lot more than just numerical precision,
> it is about 15 degrees and several mm off.
> Mark Scully has been chasing this down for several days now, and we have
> been able to narrow down the environments for which this occurs.
> The problem attempting to be solved is a straight forward
> Versor3DTransformOptimizer, MattesMutualInformation, LiniearInterpolator ITK
> registration process between two T1 weighted images.
> ---To cause failure run the process with exactly 4 threads (it also fails
> with 7, 14,15 threads, but not with 1,2,3,5,6,8,9,10,11,12,13, or 16
> threads).
> ---We have replicated the problem on Window 32bi, Linux 32/64, and Mac 64
> bit OS. Additionally, on mac we've tested in both Debug and Release mode,
> and we've built it against both ITKv3 and ITKv4. The results are consistent
> in that they consistently only fail when the thread count is one of
> {4,7,14,15} , and passing with any of the other listed successful number of
> threads (see listing at end of message).
> ---Changing the metric from MattesMutualInformation to MeanSquareError
> allows this to pass with any number of threads.
> --ITK is built with OptimizedRegistration in ITKv3 (In ITKv4
> OptimizedRegistration is the default).
> Create a bash script with the following contents and run to see the
> behavior. The first instance of the test will pass, the second will fail.
> \/\//\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
> #!/bin/sh
> svn checkout --username slicerbot --password
> slicer https://www.nitrc.org/svn/brains/StandAloneApps/StandAloneBRAINSFit StandAloneBRAINSFit
> mkdir StandAloneBRAINSFit-build
> cd StandAloneBRAINSFit-build
> cmake ../StandAloneBRAINSFit
> make
> cd BRAINSFit-build
> echo export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=1
> export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=1
> ctest -R RigidRotGeom
> echo export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=4
> export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=4
> ctest -R RigidRotGeom
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> Any hints on a strategy to help identify and correct the failure would be
> GREATLY appreciated.
> Thanks,
> Hans
>
> [hjohnson at hjhomebuildbox BRAINSFit-build]$ for i in 1 2 3 4 5 6 7 8 9 10 11
> 12 13 14 15 16;do echo "=============== Running with ${i} Threads =========
> " ; export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=${i} ; ctest -R
> BRAINSFitTest_RigidRotGeomNoMasks$ ; done
> =============== Running with 1 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 184.79 sec
> 100% tests passed, 0 tests failed out of 1
> Total Test time (real) = 184.81 sec
> =============== Running with 2 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 140.10 sec
> 100% tests passed, 0 tests failed out of 1
> Total Test time (real) = 140.13 sec
> =============== Running with 3 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 54.98 sec
> 100% tests passed, 0 tests failed out of 1
> Total Test time (real) = 55.01 sec
> =============== Running with 4 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed 22.80 sec
> 0% tests passed, 1 tests failed out of 1
> Total Test time (real) = 22.83 sec
> The following tests FAILED:
> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
> Errors while running CTest
> =============== Running with 5 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 35.23 sec
> 100% tests passed, 0 tests failed out of 1
> Total Test time (real) = 35.29 sec
> =============== Running with 6 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 29.61 sec
> 100% tests passed, 0 tests failed out of 1
> Total Test time (real) = 29.63 sec
> =============== Running with 7 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed 55.38 sec
> 0% tests passed, 1 tests failed out of 1
> Total Test time (real) = 55.41 sec
> The following tests FAILED:
> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
> Errors while running CTest
> =============== Running with 8 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 17.25 sec
> 100% tests passed, 0 tests failed out of 1
> Total Test time (real) = 17.26 sec
> =============== Running with 9 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 27.49 sec
> 100% tests passed, 0 tests failed out of 1
> Total Test time (real) = 27.51 sec
> =============== Running with 10 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 36.89 sec
> 100% tests passed, 0 tests failed out of 1
> Total Test time (real) = 36.92 sec
> =============== Running with 11 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 19.71 sec
> 100% tests passed, 0 tests failed out of 1
> Total Test time (real) = 19.72 sec
> =============== Running with 12 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 17.97 sec
> 100% tests passed, 0 tests failed out of 1
> Total Test time (real) = 17.98 sec
> =============== Running with 13 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 34.96 sec
> 100% tests passed, 0 tests failed out of 1
> Total Test time (real) = 34.98 sec
> =============== Running with 14 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed 18.18 sec
> 0% tests passed, 1 tests failed out of 1
> Total Test time (real) = 18.20 sec
> The following tests FAILED:
> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
> Errors while running CTest
> =============== Running with 15 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed 17.45 sec
> 0% tests passed, 1 tests failed out of 1
> Total Test time (real) = 17.48 sec
> The following tests FAILED:
> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
> Errors while running CTest
> =============== Running with 16 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 23.14 sec
> 100% tests passed, 0 tests failed out of 1
> Total Test time (real) = 23.15 sec
>
>
> ________________________________
> Notice: This UI Health Care e-mail (including attachments) is covered by the
> Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential
> and may be legally privileged. If you are not the intended recipient, you
> are hereby notified that any retention, dissemination, distribution, or
> copying of this communication is strictly prohibited. Please reply to the
> sender that you have received the message in error, then delete it. Thank
> you.
> ________________________________
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Kitware offers ITK Training Courses, for more information visit:
> http://kitware.com/products/protraining.html
>
> Please keep messages on-topic and check the ITK FAQ at:
> http://www.itk.org/Wiki/ITK_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.itk.org/mailman/listinfo/insight-developers
>
>
More information about the Insight-developers
mailing list