[Insight-developers] Very suspicious behavior.

Johnson, Hans J hans-johnson at uiowa.edu
Thu Feb 24 21:29:49 EST 2011 

Hello ITK Experts:

I have a really nasty problem that is almost certainly some sort algorithmic anomaly that occurs during the creation of multiple threads under the right data conditions.  The failure is a lot more than just numerical precision, it is about 15 degrees and several mm off.

Mark Scully has been chasing this down for several days now, and we have been able to narrow down the environments for which this occurs.

The problem attempting to be solved is a straight forward Versor3DTransformOptimizer, MattesMutualInformation, LiniearInterpolator ITK registration process between two T1 weighted images.

---To cause failure run the process with exactly 4 threads (it also fails with 7, 14,15 threads, but not with 1,2,3,5,6,8,9,10,11,12,13, or 16 threads).

---We have replicated the problem on Window 32bi, Linux 32/64, and Mac 64 bit OS.  Additionally, on mac we've tested in both Debug and Release mode, and we've built it against both ITKv3 and ITKv4.  The results are consistent in that they consistently only fail when the thread count is one of {4,7,14,15} , and passing with any of the other listed successful number of threads (see listing at end of message).

---Changing the metric from MattesMutualInformation to MeanSquareError allows this to pass with any number of threads.

--ITK is built with OptimizedRegistration in ITKv3 (In ITKv4 OptimizedRegistration is the default).

Create a bash script with the following contents and run to see the behavior.  The first instance of the test will pass, the second will fail.

\/\//\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
#!/bin/sh
svn checkout --username slicerbot --password slicer https://www.nitrc.org/svn/brains/StandAloneApps/StandAloneBRAINSFit StandAloneBRAINSFit
mkdir StandAloneBRAINSFit-build
cd StandAloneBRAINSFit-build
cmake ../StandAloneBRAINSFit
make
cd BRAINSFit-build

echo export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=1
export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=1
ctest -R RigidRotGeom

echo export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=4
export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=4
ctest -R RigidRotGeom
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


Any hints on a strategy to help identify and correct the failure would be GREATLY appreciated.

Thanks,
Hans


[hjohnson at hjhomebuildbox BRAINSFit-build]$ for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16;do echo "=============== Running with ${i} Threads ========= " ; export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=${i} ; ctest -R BRAINSFitTest_RigidRotGeomNoMasks$ ; done
=============== Running with 1 Threads =========
Test project /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
    Start 28: BRAINSFitTest_RigidRotGeomNoMasks
1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed  184.79 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 184.81 sec
=============== Running with 2 Threads =========
Test project /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
    Start 28: BRAINSFitTest_RigidRotGeomNoMasks
1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed  140.10 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 140.13 sec
=============== Running with 3 Threads =========
Test project /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
    Start 28: BRAINSFitTest_RigidRotGeomNoMasks
1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   54.98 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =  55.01 sec
=============== Running with 4 Threads =========
Test project /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
    Start 28: BRAINSFitTest_RigidRotGeomNoMasks
1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed   22.80 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =  22.83 sec

The following tests FAILED:
28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
Errors while running CTest
=============== Running with 5 Threads =========
Test project /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
    Start 28: BRAINSFitTest_RigidRotGeomNoMasks
1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   35.23 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =  35.29 sec
=============== Running with 6 Threads =========
Test project /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
    Start 28: BRAINSFitTest_RigidRotGeomNoMasks
1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   29.61 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =  29.63 sec
=============== Running with 7 Threads =========
Test project /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
    Start 28: BRAINSFitTest_RigidRotGeomNoMasks
1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed   55.38 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =  55.41 sec

The following tests FAILED:
28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
Errors while running CTest
=============== Running with 8 Threads =========
Test project /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
    Start 28: BRAINSFitTest_RigidRotGeomNoMasks
1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   17.25 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =  17.26 sec
=============== Running with 9 Threads =========
Test project /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
    Start 28: BRAINSFitTest_RigidRotGeomNoMasks
1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   27.49 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =  27.51 sec
=============== Running with 10 Threads =========
Test project /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
    Start 28: BRAINSFitTest_RigidRotGeomNoMasks
1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   36.89 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =  36.92 sec
=============== Running with 11 Threads =========
Test project /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
    Start 28: BRAINSFitTest_RigidRotGeomNoMasks
1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   19.71 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =  19.72 sec
=============== Running with 12 Threads =========
Test project /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
    Start 28: BRAINSFitTest_RigidRotGeomNoMasks
1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   17.97 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =  17.98 sec
=============== Running with 13 Threads =========
Test project /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
    Start 28: BRAINSFitTest_RigidRotGeomNoMasks
1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   34.96 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =  34.98 sec
=============== Running with 14 Threads =========
Test project /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
    Start 28: BRAINSFitTest_RigidRotGeomNoMasks
1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed   18.18 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =  18.20 sec

The following tests FAILED:
28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
Errors while running CTest
=============== Running with 15 Threads =========
Test project /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
    Start 28: BRAINSFitTest_RigidRotGeomNoMasks
1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed   17.45 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =  17.48 sec

The following tests FAILED:
28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
Errors while running CTest
=============== Running with 16 Threads =========
Test project /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
    Start 28: BRAINSFitTest_RigidRotGeomNoMasks
1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...   Passed   23.14 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =  23.15 sec



________________________________
Notice: This UI Health Care e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential and may be legally privileged.  If you are not the intended recipient, you are hereby notified that any retention, dissemination, distribution, or copying of this communication is strictly prohibited.  Please reply to the sender that you have received the message in error, then delete it.  Thank you.
________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.itk.org/mailman/private/insight-developers/attachments/20110225/910630ad/attachment.htm>

More information about the Insight-developers mailing list