[Insight-developers] Very suspicious behavior.
Bill Lorensen
bill.lorensen at gmail.com
Fri Feb 25 14:54:11 EST 2011
Should you get exactly the same answer regardless of the number of threads?
On Fri, Feb 25, 2011 at 10:57 AM, Daniel Blezek <Blezek.Daniel at mayo.edu> wrote:
> Hi Hans,
>
> I can verify your test fails for different numbers of threads.
>
> This machine has 8 cores. Fails at 7, 9, 12.
>
> Will continue to look into this for you.
> -dan
>
>
> mi3c-pong:BRAINSFit-build blezek$ for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14
> 15 16;do echo "=============== Running with ${i} Threads ========= " ;
> export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=${i} ; ctest -R
> BRAINSFitTest_RigidRotGeomNoMasks$ ; done
> =============== Running with 1 Threads =========
> 100% tests passed, 0 tests failed out of 1
> =============== Running with 2 Threads =========
> 100% tests passed, 0 tests failed out of 1
> =============== Running with 3 Threads =========
> 100% tests passed, 0 tests failed out of 1
> =============== Running with 4 Threads =========
> 100% tests passed, 0 tests failed out of 1
> =============== Running with 5 Threads =========
> 100% tests passed, 0 tests failed out of 1
> =============== Running with 6 Threads =========
> 100% tests passed, 0 tests failed out of 1
> =============== Running with 7 Threads =========
> 0% tests passed, 1 tests failed out of 1
> =============== Running with 8 Threads =========
> 100% tests passed, 0 tests failed out of 1
> =============== Running with 9 Threads =========
> 0% tests passed, 1 tests failed out of 1
>
> The following tests FAILED:
> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
> Errors while running CTest
> =============== Running with 10 Threads =========
> 100% tests passed, 0 tests failed out of 1
> =============== Running with 11 Threads =========
> 100% tests passed, 0 tests failed out of 1
> =============== Running with 12 Threads =========
> 0% tests passed, 1 tests failed out of 1
>
> The following tests FAILED:
> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
> Errors while running CTest
> =============== Running with 13 Threads =========
> 100% tests passed, 0 tests failed out of 1
> =============== Running with 14 Threads =========
> 100% tests passed, 0 tests failed out of 1
> =============== Running with 15 Threads =========
> 100% tests passed, 0 tests failed out of 1
> =============== Running with 16 Threads =========
> 100% tests passed, 0 tests failed out of 1
>
>
>
> On 2/24/11 8:29 PM, "Johnson, Hans J" <hans-johnson at uiowa.edu> wrote:
>
> Hello ITK Experts:
>
> I have a really nasty problem that is almost certainly some sort algorithmic
> anomaly that occurs during the creation of multiple threads under the right
> data conditions. The failure is a lot more than just numerical precision,
> it is about 15 degrees and several mm off.
>
> Mark Scully has been chasing this down for several days now, and we have
> been able to narrow down the environments for which this occurs.
>
> The problem attempting to be solved is a straight forward
> Versor3DTransformOptimizer, MattesMutualInformation, LiniearInterpolator ITK
> registration process between two T1 weighted images.
>
> ---To cause failure run the process with exactly 4 threads (it also fails
> with 7, 14,15 threads, but not with 1,2,3,5,6,8,9,10,11,12,13, or 16
> threads).
>
> ---We have replicated the problem on Window 32bi, Linux 32/64, and Mac 64
> bit OS. Additionally, on mac we've tested in both Debug and Release mode,
> and we've built it against both ITKv3 and ITKv4. The results are consistent
> in that they consistently only fail when the thread count is one of
> {4,7,14,15} , and passing with any of the other listed successful number of
> threads (see listing at end of message).
>
> ---Changing the metric from MattesMutualInformation to MeanSquareError
> allows this to pass with any number of threads.
>
> --ITK is built with OptimizedRegistration in ITKv3 (In ITKv4
> OptimizedRegistration is the default).
>
> Create a bash script with the following contents and run to see the
> behavior. The first instance of the test will pass, the second will fail.
>
> \/\//\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
> #!/bin/sh
> svn checkout --username slicerbot --password slicer
> https://www.nitrc.org/svn/brains/StandAloneApps/StandAloneBRAINSFit
> StandAloneBRAINSFit
> mkdir StandAloneBRAINSFit-build
> cd StandAloneBRAINSFit-build
> cmake ../StandAloneBRAINSFit
> make
> cd BRAINSFit-build
>
> echo export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=1
> export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=1
> ctest -R RigidRotGeom
>
> echo export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=4
> export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=4
> ctest -R RigidRotGeom
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
>
> Any hints on a strategy to help identify and correct the failure would be
> GREATLY appreciated.
>
> Thanks,
> Hans
>
>
> [hjohnson at hjhomebuildbox BRAINSFit-build]$ for i in 1 2 3 4 5 6 7 8 9 10 11
> 12 13 14 15 16;do echo "=============== Running with ${i} Threads =========
> " ; export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=${i} ; ctest -R
> BRAINSFitTest_RigidRotGeomNoMasks$ ; done
> =============== Running with 1 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 184.79 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 184.81 sec
> =============== Running with 2 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 140.10 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 140.13 sec
> =============== Running with 3 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 54.98 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 55.01 sec
> =============== Running with 4 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed 22.80 sec
>
> 0% tests passed, 1 tests failed out of 1
>
> Total Test time (real) = 22.83 sec
>
> The following tests FAILED:
> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
> Errors while running CTest
> =============== Running with 5 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 35.23 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 35.29 sec
> =============== Running with 6 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 29.61 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 29.63 sec
> =============== Running with 7 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed 55.38 sec
>
> 0% tests passed, 1 tests failed out of 1
>
> Total Test time (real) = 55.41 sec
>
> The following tests FAILED:
> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
> Errors while running CTest
> =============== Running with 8 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 17.25 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 17.26 sec
> =============== Running with 9 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 27.49 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 27.51 sec
> =============== Running with 10 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 36.89 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 36.92 sec
> =============== Running with 11 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 19.71 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 19.72 sec
> =============== Running with 12 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 17.97 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 17.98 sec
> =============== Running with 13 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 34.96 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 34.98 sec
> =============== Running with 14 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed 18.18 sec
>
> 0% tests passed, 1 tests failed out of 1
>
> Total Test time (real) = 18.20 sec
>
> The following tests FAILED:
> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
> Errors while running CTest
> =============== Running with 15 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed 17.45 sec
>
> 0% tests passed, 1 tests failed out of 1
>
> Total Test time (real) = 17.48 sec
>
> The following tests FAILED:
> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
> Errors while running CTest
> =============== Running with 16 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 23.14 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 23.15 sec
>
>
>
> ________________________________
> Notice: This UI Health Care e-mail (including attachments) is covered by the
> Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential
> and may be legally privileged. If you are not the intended recipient, you
> are hereby notified that any retention, dissemination, distribution, or
> copying of this communication is strictly prohibited. Please reply to the
> sender that you have received the message in error, then delete it. Thank
> you.
> ________________________________
> ________________________________
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Kitware offers ITK Training Courses, for more information visit:
> http://kitware.com/products/protraining.html
>
> Please keep messages on-topic and check the ITK FAQ at:
> http://www.itk.org/Wiki/ITK_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.itk.org/mailman/listinfo/insight-developers
>
> --
> Daniel Blezek, PhD
> Medical Imaging Informatics Innovation Center
>
> P 127 or (77) 8 8886
> T 507 538 8886
> E blezek.daniel at mayo.edu
>
> Mayo Clinic
> 200 First St. S.W.
> Harwick SL-44
> Rochester, MN 55905
> mayoclinic.org
> "It is more complicated than you think." -- RFC 1925
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Kitware offers ITK Training Courses, for more information visit:
> http://kitware.com/products/protraining.html
>
> Please keep messages on-topic and check the ITK FAQ at:
> http://www.itk.org/Wiki/ITK_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.itk.org/mailman/listinfo/insight-developers
>
>
More information about the Insight-developers
mailing list