[Insight-users] Lack of memory for segmentation; SetNumberOfThreads

[Miller, James V (GE, Research)]

Robert,

Streaming and threading are separate concepts that leverage the same
underlying architecture.

In threading, a filter will divide an image into num_processors sections, 
and process all the section is parallel.  The MedianImageFilter and the 
CurvatureFlowImageFilter support threading.

In streaming, the problem is divided in N sections, and each filter in 
the pipeline is called N times, once for each section.  Within each 
filter execution, the section may be further divided for threading.
Not all filters support streaming.  For instance, any filter that requires
global information can effectively stream.  The MedianImageFilter supports
streaming but the CurvatureFlowImageFilter does not.

These concepts are explained further in the doxygen documentation.

http://www.itk.org/Doxygen16/html/ThreadingPage.html

http://www.itk.org/Doxygen16/html/StreamingPage.html


To use streaming, you need to place a StreamingImageFilter in the pipeline.
If some of the filters you are using do not support streaming, you want to 
place the StreamingImageFilter downstream from the filters that do support
streaming and upstream from the filters that do not support streaming.

Jim


-----Original Message-----
From: Atwood, Robert C [mailto:r.atwood at imperial.ac.uk]
Sent: Tuesday, January 24, 2006 11:26 AM
To: Miller, James V (GE, Research); insight-users at itk.org
Subject: Re: [Insight-users] Lack of memory for segmentation;
SetNumberOfThreads


Hi,
I have finally got round to trying this. However, the implication I get
from the suggestion is that the entire pipleline will be run N times if
one of the filters is set to stream in N chunks.
I have a pipline like (minimized code snippet, Volmedian is
itkMedianImageFilter  and Curvflowdiff is itkCurvatureFlowImageFilter)

itk::SimpleFilterWatcher volmwatcher(Volmedian,"VolMedian");
itk::SimpleFilterWatcher pfwatcher(Curvflow,"PostFilter");

Volmedian->SetNumberOfThreads(nchunks);

Volmedian->SetInput(ExistingImage);
Curvflow->SetInput(Volmedian->GetOutput());
Writer->SetInput(Curvflow->GetOutput());

Writer->Update(); /* in a try/catch block */ 


The output of the SimpleFilterWatcher indicates that the number of
threads for the volume median is nchunks (8) as requested, but the
number of threads for the post-filter (Curvflow) is 2 (it seems to use
the number of processors, it goes to 4 on a similar machine with
'hyperthreading' enabled)


Is this what should happen? Did I misunderstand? From the watcher
otuput, it seems that the entire volume median filter completes
executing before the post-filter starts, or is this a misunderstanding
of the output again? 

Thanks in advance for clarifying these matters.

Robert

(sections of the output .. There is a lot of output ... )

-------- End MedianImageFilter "VolMedian" MedianImageFilter (0x8452538)

  .....

  Thread Count: 8
  Global Maximum Number Of Threads: 0
  Radius: [1, 1, 1]

-------- Start CurvatureFlowImageFilter "PostFilter"
CurvatureFlowImageFilter (0x8459780)
 
.....

-------- End CurvatureFlowImageFilter "PostFilter"
CurvatureFlowImageFilter (0x8459780)

....
   Debug: Off
    Observers: 
      none
    Thread Count: 2
    Global Maximum Number Of Threads: 0
.....



_________________Original message quote regarding running out of memory
___________

Miller, James V (Research) 
Fri Nov 18 09:16:50 EST 2005
 
There are a number of strategies for managing memory.
....
2) StreamingImageFilter.  You can place a StreamingImageFilter at the
end of
your pipeline (or anywhere within the pipeline) and tell the
StreamingImageFilter
to divide the problem into N sections.  The pipeline will then execute N
times, 
....