<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hello,<div><br></div><div>I have a follow-up to my previous question. I've determined that I am using multiple threads, but not the number that I specify using SetNumberOfThreads(N). It looks like if there's more than one, it will use all available threads. Is there something else that is overriding SetNumberOfThreads(N) to determine the number used?</div><div><br></div><div>Also, regardless of the number of threads, the registration is taking the same amount of clock time. Please let me know if you have any thoughts.</div><div><br></div><div><br></div><div>Thanks,</div><div>Sara</div><div><br></div><div><br><div><div>On Sep 27, 2011, at 10:41 AM, Sara Rolfe wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hello,<div><br></div><div>I'm attempting to run a multithreaded registration. I am using a computing cluster with 13 nodes managed by Condor. Each node us a dual quad-core with 32GB RAM, running the 64-bit version of RHEL. </div><div><br></div><div>I recompiled ITK with the CMAKE flags: </div><div><span class="Apple-style-span" style="color: rgb(84, 0, 0); ">ITK_USE_REVIEW</span></div><div><span class="Apple-style-span" style="color: rgb(84, 0, 0); "></span><span class="Apple-style-span" style="color: rgb(84, 0, 0); ">ITK_USE_OPTIMIZED_REGISTRATION_METHODS</span></div><div><span class="Apple-style-span" style="color: rgb(84, 0, 0); "><br></span></div><div><font class="Apple-style-span" color="#540000"><div style="color: rgb(0, 0, 0); "><font class="Apple-style-span" color="#540000"><font class="Apple-style-span" color="#000000">and added the line:</font></font></div><div style="color: rgb(0, 0, 0); "><span class="Apple-style-span" style="color: rgb(84, 0, 0); ">registration->SetNumberOfThreads( 8 );</span></div><div style="color: rgb(0, 0, 0); "><span class="Apple-style-span" style="color: rgb(84, 0, 0); "><br></span></div><div style="color: rgb(0, 0, 0); "><span class="Apple-style-span" style="color: rgb(84, 0, 0); "><div style="color: rgb(0, 0, 0); ">to my registration code. I've run the registration with the number of threads set to 1, 4 and 8. However, I'm getting no improvement in the run time. I would any advice on debugging this. I suspect that I am not actually getting the additional threads. When I look at the job details it looks like only one CPU was requested.</div><div style="color: rgb(0, 0, 0); "><br></div><div style="color: rgb(0, 0, 0); ">Below are the results from "condor_history -long pid" for the three jobs (1, 4, and 8 threads):<br><br>2076.0 1 Thread <span class="Apple-tab-span" style="white-space: pre; "> </span>Run Time: 0+05:28:28<br>LocalUserCpu<span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>0<br>LocalSysCpu<span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>0<br>RemoteUserCpu<span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>23858<br>RemoteSysCpu<span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>10<br>RequestCpus<span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>1<br><br>2075.0 4 threads <span class="Apple-tab-span" style="white-space: pre; "> </span>Run Time: 0+05:27:41<br>LocalUserCpu <span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>0<br>LocalSysCpu<span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>0<br>RemoteUserCpu<span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>23680<br>RemoteSysCpu<span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>9<br>RequestCpus<span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>1<br><br>2077.0 8 Threads <span class="Apple-tab-span" style="white-space: pre; "> </span>Run Time: 0+05:30:14<br>LocalUserCpu<span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>0<br>LocalSysCpu<span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>0<br>RemoteUserCpu<span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>24025<br>RemoteSysCpu<span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>10<br>RequestCpus<span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>1<br><br>I also reran the job requesting 8 threads so I could look at the allocation. The job was allocated one slot on one node. I then Iogged onto that node and used the "top" command and the results are below. Here it looks to me like multiple CPUs are being used. I'd appreciate any thoughts on interpreting this. <br><br>top - 09:53:43 up 14 days, 20:45, 1 user, load average: 2.31, 0.84, 0.31<br>Tasks: <span class="Apple-tab-span" style="white-space: pre; "> </span>233 total, 3 running, 230 sleeping, 0 stopped, 0 zombie<br>Cpu0 : <span class="Apple-tab-span" style="white-space: pre; "> </span>0.0%us, 0.0%sy, 40.2%ni, 59.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st<br>Cpu1 : <span class="Apple-tab-span" style="white-space: pre; "> </span>0.0%us, 0.0%sy, 85.7%ni, 14.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st<br>Cpu2 : <span class="Apple-tab-span" style="white-space: pre; "> </span>0.0%us, 0.3%sy, 49.2%ni, 50.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st<br>Cpu3 : <span class="Apple-tab-span" style="white-space: pre; "> </span>0.0%us, 0.0%sy, 84.9%ni, 15.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st<br>Cpu4 : <span class="Apple-tab-span" style="white-space: pre; "> </span>0.0%us, 0.3%sy, 45.0%ni, 54.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st<br>Cpu5 : <span class="Apple-tab-span" style="white-space: pre; "> </span>0.0%us, 0.0%sy, 84.4%ni, 15.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st<br>Cpu6 : <span class="Apple-tab-span" style="white-space: pre; "> </span>0.0%us, 0.3%sy, 39.9%ni, 59.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st<br>Cpu7 : <span class="Apple-tab-span" style="white-space: pre; "> </span>0.0%us, 0.3%sy, 84.4%ni, 15.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st<br>Cpu8 : <span class="Apple-tab-span" style="white-space: pre; "> </span>0.0%us, 0.3%sy, 95.7%ni, 4.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st<br>Cpu9 : <span class="Apple-tab-span" style="white-space: pre; "> </span>0.0%us, 0.0%sy, 93.0%ni, 7.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st<br>Cpu10 : 0.0%us, 0.0%sy, 85.7%ni, 14.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st<br>Cpu11 : 0.0%us, 0.3%sy, 95.0%ni, 4.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st<br>Cpu12 : 0.0%us, 0.0%sy, 94.7%ni, 5.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st<br>Cpu13 : 0.0%us, 0.0%sy, 99.3%ni, 0.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st<br>Cpu14 : 0.0%us, 0.0%sy, 92.0%ni, 8.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st<br>Cpu15 : 0.0%us, 0.0%sy, 96.7%ni, 3.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st<br>Mem: 32948672k total, 3668612k used, 29280060k free, 359324k buffers<br>Swap: 31262480k total, 0k used, 31262480k free, 1014788k cached<br><br>PID <span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>USER PR NI VIRT <span class="Apple-tab-span" style="white-space: pre; "> </span>RES <span class="Apple-tab-span" style="white-space: pre; "> </span>SHR <span class="Apple-tab-span" style="white-space: pre; "> </span>S <span class="Apple-tab-span" style="white-space: pre; "> </span>%CPU %MEM TIME+ <span class="Apple-tab-span" style="white-space: pre; "> </span>COMMAND<br>3333 <span class="Apple-tab-span" style="white-space: pre; "> </span>smrolfe 35 10 3004m <span class="Apple-tab-span" style="white-space: pre; "> </span>2.0g <span class="Apple-tab-span" style="white-space: pre; "> </span>3700 <span class="Apple-tab-span" style="white-space: pre; "> </span>R <span class="Apple-tab-span" style="white-space: pre; "> </span>1267.0 <span class="Apple-tab-span" style="white-space: pre; "> </span> 6.4 <span class="Apple-tab-span" style="white-space: pre; "> </span>5:40.47 <span class="Apple-tab-span" style="white-space: pre; "> </span>condor_exec.exe<br>3406 <span class="Apple-tab-span" style="white-space: pre; "> </span>smrolfe 15 0 12880 <span class="Apple-tab-span" style="white-space: pre; "> </span>1216 <span class="Apple-tab-span" style="white-space: pre; "> </span>832 <span class="Apple-tab-span" style="white-space: pre; "> </span>R <span class="Apple-tab-span" style="white-space: pre; "> </span> 0.3 <span class="Apple-tab-span" style="white-space: pre; "> </span> 0.0 <span class="Apple-tab-span" style="white-space: pre; "> </span>0:00.22 <span class="Apple-tab-span" style="white-space: pre; "> </span>top<br>1 root <span class="Apple-tab-span" style="white-space: pre; "> </span> 15 0 10368 <span class="Apple-tab-span" style="white-space: pre; "> </span>632 <span class="Apple-tab-span" style="white-space: pre; "> </span>536 <span class="Apple-tab-span" style="white-space: pre; "> </span>S <span class="Apple-tab-span" style="white-space: pre; "> </span> 0.0 <span class="Apple-tab-span" style="white-space: pre; "> </span> 0.0 <span class="Apple-tab-span" style="white-space: pre; "> </span>0:01.79 <span class="Apple-tab-span" style="white-space: pre; "> </span>init<br>2 root <span class="Apple-tab-span" style="white-space: pre; "> </span> RT -5 0 <span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>0 <span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>0 <span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>S <span class="Apple-tab-span" style="white-space: pre; "> </span> 0.0 <span class="Apple-tab-span" style="white-space: pre; "> </span> 0.0 <span class="Apple-tab-span" style="white-space: pre; "> </span>0:00.04 <span class="Apple-tab-span" style="white-space: pre; "> </span>migration/0<br>3 root <span class="Apple-tab-span" style="white-space: pre; "> </span> <span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span> 34 19 0 <span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>0 <span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>0 <span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>S <span class="Apple-tab-span" style="white-space: pre; "> </span> 0.0 <span class="Apple-tab-span" style="white-space: pre; "> </span> 0.0 0:00.00 <span class="Apple-tab-span" style="white-space: pre; "> </span>ksoftirqd/0<br>4 root <span class="Apple-tab-span" style="white-space: pre; "> </span> <span class="Apple-tab-span" style="white-space: pre; "> </span> RT -5 0 <span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>0 <span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>0 <span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>S <span class="Apple-tab-span" style="white-space: pre; "> </span> 0.0 <span class="Apple-tab-span" style="white-space: pre; "> </span> 0.0 0:00.00 <span class="Apple-tab-span" style="white-space: pre; "> </span>watchdog/0<br>5 root <span class="Apple-tab-span" style="white-space: pre; "> </span> <span class="Apple-tab-span" style="white-space: pre; "> </span> RT -5 0 <span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>0 <span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>0 <span class="Apple-tab-span" style="white-space: pre; "> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>S <span class="Apple-tab-span" style="white-space: pre; "> </span> 0.0 <span class="Apple-tab-span" style="white-space: pre; "> </span> 0.0 0:00.03 <span class="Apple-tab-span" style="white-space: pre; "> </span>migration/1<br></div></span></div></font></div></div>_____________________________________<br>Powered by <a href="http://www.kitware.com">www.kitware.com</a><br><br>Visit other Kitware open-source projects at<br><a href="http://www.kitware.com/opensource/opensource.html">http://www.kitware.com/opensource/opensource.html</a><br><br>Kitware offers ITK Training Courses, for more information visit:<br>http://www.kitware.com/products/protraining.html<br><br>Please keep messages on-topic and check the ITK FAQ at:<br>http://www.itk.org/Wiki/ITK_FAQ<br><br>Follow this link to subscribe/unsubscribe:<br>http://www.itk.org/mailman/listinfo/insight-users<br></blockquote></div><br></div></body></html>