[Insight-developers] Very suspicious behavior.

Daniel Blezek Blezek.Daniel at mayo.edu
Fri Feb 25 15:03:06 EST 2011 

If the Mattes is seeded with the same random number generator, it should be
deterministic, up to floating point round off.

I would have assumed that the answer would be stable independent of the
number of threads.  If it is not, it is a bug.

Oddly, the performance does not increase with the number of threads.  this
is very strange, and could be the source of the problems?

-dan

On 2/25/11 1:54 PM, "Bill Lorensen" <bill.lorensen at gmail.com> wrote:

> Should you get exactly the same answer regardless of the number of threads?
> 
> On Fri, Feb 25, 2011 at 10:57 AM, Daniel Blezek <Blezek.Daniel at mayo.edu>
> wrote:
>> Hi Hans,
>> 
>>   I can verify your test fails for different numbers of threads.
>> 
>> This machine has 8 cores.  Fails at 7, 9, 12.
>> 
>> Will continue to look into this for you.
>> -dan
>> 
>> 
>> mi3c-pong:BRAINSFit-build blezek$ for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14
>> 15 16;do echo "=============== Running with ${i} Threads ========= " ;
>> export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=${i} ; ctest -R
>> BRAINSFitTest_RigidRotGeomNoMasks$ ; done
>> =============== Running with 1 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 2 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 3 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 4 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 5 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 6 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 7 Threads =========
>> 0% tests passed, 1 tests failed out of 1
>> =============== Running with 8 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 9 Threads =========
>> 0% tests passed, 1 tests failed out of 1
>> 
>> The following tests FAILED:
>>      28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
>> Errors while running CTest
>> =============== Running with 10 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 11 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 12 Threads =========
>> 0% tests passed, 1 tests failed out of 1
>> 
>> The following tests FAILED:
>>      28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
>> Errors while running CTest
>> =============== Running with 13 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 14 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 15 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> =============== Running with 16 Threads =========
>> 100% tests passed, 0 tests failed out of 1
>> 
>> 
>> 
>> On 2/24/11 8:29 PM, "Johnson, Hans J" <hans-johnson at uiowa.edu> wrote:
>> 
>> Hello ITK Experts:
>> 
>> I have a really nasty problem that is almost certainly some sort algorithmic
>> anomaly that occurs during the creation of multiple threads under the right
>> data conditions.  The failure is a lot more than just numerical precision,
>> it is about 15 degrees and several mm off.
>> 
>> Mark Scully has been chasing this down for several days now, and we have
>> been able to narrow down the environments for which this occurs.
>> 
>> The problem attempting to be solved is a straight forward
>> Versor3DTransformOptimizer, MattesMutualInformation, LiniearInterpolator ITK
>> registration process between two T1 weighted images.
>> 
>> ---To cause failure run the process with exactly 4 threads (it also fails
>> with 7, 14,15 threads, but not with 1,2,3,5,6,8,9,10,11,12,13, or 16
>> threads).
>> 
>> ---We have replicated the problem on Window 32bi, Linux 32/64, and Mac 64
>> bit OS.  Additionally, on mac we've tested in both Debug and Release mode,
>> and we've built it against both ITKv3 and ITKv4.  The results are consistent
>> in that they consistently only fail when the thread count is one of
>> {4,7,14,15} , and passing with any of the other listed successful number of
>> threads (see listing at end of message).
>> 
>> ---Changing the metric from MattesMutualInformation to MeanSquareError
>> allows this to pass with any number of threads.
>> 
>> --ITK is built with OptimizedRegistration in ITKv3 (In ITKv4
>> OptimizedRegistration is the default).
>> 
>> Create a bash script with the following contents and run to see the
>> behavior.  The first instance of the test will pass, the second will fail.
>> 
>> \/\//\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
>> \/\/\/\/\/
>> #!/bin/sh
>> svn checkout --username slicerbot --password slicer
>> https://www.nitrc.org/svn/brains/StandAloneApps/StandAloneBRAINSFit
>> StandAloneBRAINSFit
>> mkdir StandAloneBRAINSFit-build
>> cd StandAloneBRAINSFit-build
>> cmake ../StandAloneBRAINSFit
>> make
>> cd BRAINSFit-build
>> 
>> echo export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=1
>> export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=1
>> ctest -R RigidRotGeom
>> 
>> echo export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=4
>> export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=4
>> ctest -R RigidRotGeom
>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> 
>> 
>> Any hints on a strategy to help identify and correct the failure would be
>> GREATLY appreciated.
>> 
>> Thanks,
>> Hans
>> 
>> 
>> [hjohnson at hjhomebuildbox BRAINSFit-build]$ for i in 1 2 3 4 5 6 7 8 9 10 11
>> 12 13 14 15 16;do echo "=============== Running with ${i} Threads =========
>> " ; export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=${i} ; ctest -R
>> BRAINSFitTest_RigidRotGeomNoMasks$ ; done
>> =============== Running with 1 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>>     Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed  184.79 sec
>> 
>> 100% tests passed, 0 tests failed out of 1
>> 
>> Total Test time (real) = 184.81 sec
>> =============== Running with 2 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>>     Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed  140.10 sec
>> 
>> 100% tests passed, 0 tests failed out of 1
>> 
>> Total Test time (real) = 140.13 sec
>> =============== Running with 3 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>>     Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   54.98 sec
>> 
>> 100% tests passed, 0 tests failed out of 1
>> 
>> Total Test time (real) =  55.01 sec
>> =============== Running with 4 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>>     Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed   22.80 sec
>> 
>> 0% tests passed, 1 tests failed out of 1
>> 
>> Total Test time (real) =  22.83 sec
>> 
>> The following tests FAILED:
>> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
>> Errors while running CTest
>> =============== Running with 5 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>>     Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   35.23 sec
>> 
>> 100% tests passed, 0 tests failed out of 1
>> 
>> Total Test time (real) =  35.29 sec
>> =============== Running with 6 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>>     Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   29.61 sec
>> 
>> 100% tests passed, 0 tests failed out of 1
>> 
>> Total Test time (real) =  29.63 sec
>> =============== Running with 7 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>>     Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed   55.38 sec
>> 
>> 0% tests passed, 1 tests failed out of 1
>> 
>> Total Test time (real) =  55.41 sec
>> 
>> The following tests FAILED:
>> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
>> Errors while running CTest
>> =============== Running with 8 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>>     Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   17.25 sec
>> 
>> 100% tests passed, 0 tests failed out of 1
>> 
>> Total Test time (real) =  17.26 sec
>> =============== Running with 9 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>>     Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   27.49 sec
>> 
>> 100% tests passed, 0 tests failed out of 1
>> 
>> Total Test time (real) =  27.51 sec
>> =============== Running with 10 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>>     Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   36.89 sec
>> 
>> 100% tests passed, 0 tests failed out of 1
>> 
>> Total Test time (real) =  36.92 sec
>> =============== Running with 11 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>>     Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   19.71 sec
>> 
>> 100% tests passed, 0 tests failed out of 1
>> 
>> Total Test time (real) =  19.72 sec
>> =============== Running with 12 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>>     Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   17.97 sec
>> 
>> 100% tests passed, 0 tests failed out of 1
>> 
>> Total Test time (real) =  17.98 sec
>> =============== Running with 13 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>>     Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   34.96 sec
>> 
>> 100% tests passed, 0 tests failed out of 1
>> 
>> Total Test time (real) =  34.98 sec
>> =============== Running with 14 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>>     Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed   18.18 sec
>> 
>> 0% tests passed, 1 tests failed out of 1
>> 
>> Total Test time (real) =  18.20 sec
>> 
>> The following tests FAILED:
>> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
>> Errors while running CTest
>> =============== Running with 15 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>>     Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed   17.45 sec
>> 
>> 0% tests passed, 1 tests failed out of 1
>> 
>> Total Test time (real) =  17.48 sec
>> 
>> The following tests FAILED:
>> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
>> Errors while running CTest
>> =============== Running with 16 Threads =========
>> Test project
>> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
>>     Start 28: BRAINSFitTest_RigidRotGeomNoMasks
>> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   23.14 sec
>> 
>> 100% tests passed, 0 tests failed out of 1
>> 
>> Total Test time (real) =  23.15 sec
>> 
>> 
>> 
>> ________________________________
>> Notice: This UI Health Care e-mail (including attachments) is covered by the
>> Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential
>> and may be legally privileged.  If you are not the intended recipient, you
>> are hereby notified that any retention, dissemination, distribution, or
>> copying of this communication is strictly prohibited.  Please reply to the
>> sender that you have received the message in error, then delete it.  Thank
>> you.
>> ________________________________
>> ________________________________
>> _______________________________________________
>> Powered by www.kitware.com
>> 
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>> 
>> Kitware offers ITK Training Courses, for more information visit:
>> http://kitware.com/products/protraining.html
>> 
>> Please keep messages on-topic and check the ITK FAQ at:
>> http://www.itk.org/Wiki/ITK_FAQ
>> 
>> Follow this link to subscribe/unsubscribe:
>> http://www.itk.org/mailman/listinfo/insight-developers
>> 
>> --
>> Daniel Blezek, PhD
>> Medical Imaging Informatics Innovation Center
>> 
>> P 127 or (77) 8 8886
>> T 507 538 8886
>> E blezek.daniel at mayo.edu
>> 
>> Mayo Clinic
>> 200 First St. S.W.
>> Harwick SL-44
>> Rochester, MN 55905
>> mayoclinic.org
>> "It is more complicated than you think." -- RFC 1925
>> 
>> _______________________________________________
>> Powered by www.kitware.com
>> 
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>> 
>> Kitware offers ITK Training Courses, for more information visit:
>> http://kitware.com/products/protraining.html
>> 
>> Please keep messages on-topic and check the ITK FAQ at:
>> http://www.itk.org/Wiki/ITK_FAQ
>> 
>> Follow this link to subscribe/unsubscribe:
>> http://www.itk.org/mailman/listinfo/insight-developers
>> 
>> 

-- 
Daniel Blezek, PhD
Medical Imaging Informatics Innovation Center

P 127 or (77) 8 8886
T 507 538 8886
E blezek.daniel at mayo.edu

Mayo Clinic
200 First St. S.W.
Harwick SL-44
Rochester, MN 55905
mayoclinic.org
"It is more complicated than you think." -- RFC 1925

More information about the Insight-developers mailing list