[Insight-developers] Very suspicious behavior.
Johnson, Hans J
hans-johnson at uiowa.edu
Fri Feb 25 16:15:47 EST 2011
For passing cases all results are quite close (numerically good enough). The failing cases are drastically wrong.
Sent from my iPhonerecision
On Feb 25, 2011, at 1:54 PM, "Bill Lorensen" <bill.lorensen at gmail.com> wrote:
> Should you get exactly the same answer regardless of the number of threads?
>
> On Fri, Feb 25, 2011 at 10:57 AM, Daniel Blezek <Blezek.Daniel at mayo.edu> wrote:
>> Hi Hans,
>>
>> I can verify your test fails for different numbers of threads.
>>
>> This machine has 8 cores. Fails at 7, 9, 12.
>>
>> Will continue to look into this for you.
>> -dan
>>
>>
>> mi3c-pong:BRAINSFit-build blezek$ for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14
>> 15 16;do echo "=============== Running with ${i} Threads ========= " ;
>> export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=${i} ; ctest -R
>> BRAINSFitTest_RigidRotGeomNoMasks$ ; done
>> =============== Running with 1 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 2 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 3 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 4 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 5 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 6 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 7 Threads =========
>> 0% tests passed, 1 tests failed out of 1
>> =============== Running with 8 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 9 Threads =========
>> 0% tests passed, 1 tests failed out of 1
>>
>> The following tests FAILED:
>> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
>> Errors while running CTest
>> =============== Running with 10 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 11 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 12 Threads =========
>> 0% tests passed, 1 tests failed out of 1
>>
>> The following tests FAILED:
>> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
>> Errors while running CTest
>> =============== Running with 13 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 14 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 15 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 16 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>>
>>
>>
>> On 2/24/11 8:29 PM, "Johnson, Hans J" <hans-johnson at uiowa.edu> wrote:
>>
>> Hello ITK Experts:
>>
>> I have a really nasty problem that is almost certainly some sort algorithmic
>> anomaly that occurs during the creation of multiple threads under the right
>> data conditions. The failure is a lot more than just numerical precision,
>> it is about 15 degrees and several mm off.
>>
>> Mark Scully has been chasing this down for several days now, and we have
>> been able to narrow down the environments for which this occurs.
>>
>> The problem attempting to be solved is a straight forward
>> Versor3DTransformOptimizer, MattesMutualInformation, LiniearInterpolator ITK
>> registration process between two T1 weighted images.
>>
>> ---To cause failure run the process with exactly 4 threads (it also fails
>> with 7, 14,15 threads, but not with 1,2,3,5,6,8,9,10,11,12,13, or 16
>> threads).
>>
>> ---We have replicated the problem on Window 32bi, Linux 32/64, and Mac 64
>> bit OS. Additionally, on mac we've tested in both Debug and Release mode,
>> and we've built it against both ITKv3 and ITKv4. The results are consistent
>> in that they consistently only fail when the thread count is one of
>> {4,7,14,15} , and passing with any of the other listed successful number of
>> threads (see listing at end of message).
>>
>> ---Changing the metric from MattesMutualInformation to MeanSquareError
>> allows this to pass with any number of threads.
>>
>> --ITK is built with OptimizedRegistration in ITKv3 (In ITKv4
>> OptimizedRegistration is the default).
>>
>> Create a bash script with the following contents and run to see the
>> behavior. The first instance of the test will pass, the second will fail.
>>
>> \/\//\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
>> #!/bin/sh
>> svn checkout --username slicerbot --password slicer
>> https://www.nitrc.org/svn/brains/StandAloneApps/StandAloneBRAINSFit
>> StandAloneBRAINSFit
>> mkdir StandAloneBRAINSFit-build
>> cd StandAloneBRAINSFit-build
>> cmake ../StandAloneBRAINSFit
>> make
>> cd BRAINSFit-build
>>
>> echo export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=1
>> export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=1
>> ctest -R RigidRotGeom
>>
>> echo export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=4
>> export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=4
>> ctest -R RigidRotGeom
>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>
>>
>> Any hints on a strategy to help identify and correct the failure would be
>> GREATLY appreciated.
>>
>> Thanks,
>> Hans
>>
>>
>> [hjohnson at hjhomebuildbox BRAINSFit-build]$ for i in 1 2 3 4 5 6 7 8 9 10 11
>> 12 13 14 15 16;do echo "=============== Running with ${i} Threads =========
>> " ; export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=${i} ; ctest -R
>> BRAINSFitTest_RigidRotGeomNoMasks$ ; done
>> =============== Running with 1 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 184.79 sec
>>
>> 100% tests passed, 0 tests failed out of 1
>>
>> Total Test time (real) = 184.81 sec
>> =============== Running with 2 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 140.10 sec
>>
>> 100% tests passed, 0 tests failed out of 1
>>
>> Total Test time (real) = 140.13 sec
>> =============== Running with 3 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 54.98 sec
>>
>> 100% tests passed, 0 tests failed out of 1
>>
>> Total Test time (real) = 55.01 sec
>> =============== Running with 4 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed 22.80 sec
>>
>> 0% tests passed, 1 tests failed out of 1
>>
>> Total Test time (real) = 22.83 sec
>>
>> The following tests FAILED:
>> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
>> Errors while running CTest
>> =============== Running with 5 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 35.23 sec
>>
>> 100% tests passed, 0 tests failed out of 1
>>
>> Total Test time (real) = 35.29 sec
>> =============== Running with 6 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 29.61 sec
>>
>> 100% tests passed, 0 tests failed out of 1
>>
>> Total Test time (real) = 29.63 sec
>> =============== Running with 7 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed 55.38 sec
>>
>> 0% tests passed, 1 tests failed out of 1
>>
>> Total Test time (real) = 55.41 sec
>>
>> The following tests FAILED:
>> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
>> Errors while running CTest
>> =============== Running with 8 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 17.25 sec
>>
>> 100% tests passed, 0 tests failed out of 1
>>
>> Total Test time (real) = 17.26 sec
>> =============== Running with 9 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 27.49 sec
>>
>> 100% tests passed, 0 tests failed out of 1
>>
>> Total Test time (real) = 27.51 sec
>> =============== Running with 10 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 36.89 sec
>>
>> 100% tests passed, 0 tests failed out of 1
>>
>> Total Test time (real) = 36.92 sec
>> =============== Running with 11 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 19.71 sec
>>
>> 100% tests passed, 0 tests failed out of 1
>>
>> Total Test time (real) = 19.72 sec
>> =============== Running with 12 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 17.97 sec
>>
>> 100% tests passed, 0 tests failed out of 1
>>
>> Total Test time (real) = 17.98 sec
>> =============== Running with 13 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 34.96 sec
>>
>> 100% tests passed, 0 tests failed out of 1
>>
>> Total Test time (real) = 34.98 sec
>> =============== Running with 14 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed 18.18 sec
>>
>> 0% tests passed, 1 tests failed out of 1
>>
>> Total Test time (real) = 18.20 sec
>>
>> The following tests FAILED:
>> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
>> Errors while running CTest
>> =============== Running with 15 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed 17.45 sec
>>
>> 0% tests passed, 1 tests failed out of 1
>>
>> Total Test time (real) = 17.48 sec
>>
>> The following tests FAILED:
>> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
>> Errors while running CTest
>> =============== Running with 16 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 23.14 sec
>>
>> 100% tests passed, 0 tests failed out of 1
>>
>> Total Test time (real) = 23.15 sec
>>
>>
>>
>> ________________________________
>> Notice: This UI Health Care e-mail (including attachments) is covered by the
>> Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential
>> and may be legally privileged. If you are not the intended recipient, you
>> are hereby notified that any retention, dissemination, distribution, or
>> copying of this communication is strictly prohibited. Please reply to the
>> sender that you have received the message in error, then delete it. Thank
>> you.
>> ________________________________
>> ________________________________
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Kitware offers ITK Training Courses, for more information visit:
>> http://kitware.com/products/protraining.html
>>
>> Please keep messages on-topic and check the ITK FAQ at:
>> http://www.itk.org/Wiki/ITK_FAQ
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.itk.org/mailman/listinfo/insight-developers
>>
>> --
>> Daniel Blezek, PhD
>> Medical Imaging Informatics Innovation Center
>>
>> P 127 or (77) 8 8886
>> T 507 538 8886
>> E blezek.daniel at mayo.edu
>>
>> Mayo Clinic
>> 200 First St. S.W.
>> Harwick SL-44
>> Rochester, MN 55905
>> mayoclinic.org
>> "It is more complicated than you think." -- RFC 1925
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Kitware offers ITK Training Courses, for more information visit:
>> http://kitware.com/products/protraining.html
>>
>> Please keep messages on-topic and check the ITK FAQ at:
>> http://www.itk.org/Wiki/ITK_FAQ
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.itk.org/mailman/listinfo/insight-developers
>>
>>
________________________________
Notice: This UI Health Care e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential and may be legally privileged. If you are not the intended recipient, you are hereby notified that any retention, dissemination, distribution, or copying of this communication is strictly prohibited. Please reply to the sender that you have received the message in error, then delete it. Thank you.
________________________________
More information about the Insight-developers
mailing list