[Insight-users] Evolution : overall pipeline memory print

[Bill Lorensen]

Luis,

I just checked in a test, Testing/Code/Common/itkObjectFactoryTest2,cxx that
exercises the factory load from a library code in itkObjectFactoryBase.cxx.
The test provides a factory-based ImportImageContainer that does some simple
memory allocation reporting. It might stimulate some further ideas on how to
report memory usage.

We'll see if the object loading is portable across ITK systems. I already
had to exclude the test from the Borland build because of problems. I'll
leave it out of the Borland build until I can track down the problem.

Bill

On Nov 9, 2007 11:58 AM, Luis Ibanez <luis.ibanez at kitware.com> wrote:

>
> Hi Julien,
>
>
> We will discuss this topic further today during the ITK tcon.
>
>
> I agree with Karthik that it is not a trivial problem to solve,
> but also agree with you that it is a problem worth looking at.
>
>
> Images are only going to get bigger...
> and they will do so a lot faster than computer memory is
> going to get cheaper.   :-/
>
>
> We have also challenging situations when dealing with microscopy
> data from sources such as electron microscopy and confocal
> microscopy...
>
>
>
> One way of simulating the memory behavior of the pipeline
> is to overload/intercept the image method Allocate(), and
> have it report its memory request to a central log, instead
> of actually allocating the memory, then having the pixel
> container simulate that it actually have allocated the
> memory and return a constant (or a random) value for every
> pixel request. It may be done also by intercepting the
> method "AllocateElements(size)" in the itkImportImageContainer,
> which is where the memory allocation is ultimately done.
> (Note that even the itkVectorImage delegates memory allocation
>  to its internal ImportImageContainer).
>
>
>
> By running your pipeline in such mode, it will be possible
> to collect the *actual* memory allocation requests, instead
> of the *estimated* ones from every filter, that as Karthik
> pointed out, will be very hard to do in a reliably way for
> the entire toolkit.
>
>
> In that mode, then you could experiment with different
> image partition strategies, for example the ones in the
> StreamingImageFilter.  These strategies are managed by
> the RegionSplitter classes.  You could by trial an error
> identify what size of image pieces would make possible
> to run the pipeline in the amount of memory that is
> actually available in the hardware.
>
>
> The drawback of this method is that it still will use the
> CPU time of actually computing the output of the filters,
> instead of just making a pass to estimate the memory
> footprint of the toolkit....
>
>
> You may have other ideas on how to implement other
> memory estimation strategies.
>
>
>
> We certainly agree that if such functionality is developed
> it will be much better to add it to ITK than to have it
> in your local copy of the toolkit.
>
>
>
>    Regards,
>
>
>        Luis
>
>
>
> -------------------------
>  Julien Michel wrote:
> > Karthik Krishnan a écrit :
> >
> >> Julien:
> >>
> >> I can understand your issue with being bogged down by memory. However,
> >> I don't completely agree with your proposed solution. Here are my
> >> disagreements.
> >
> >
> > Karthik,
> >
> > Thanks a lot for your response. I agree with you on the fact that the
> > GetMemoryPrint() method will not apply to all DataObjects. Regarding the
> >  step where to call the GetPipelineMemoryPrint() method, I was indeed
> > planning to do this after a
> > itk::ProcessObject::GenerateOutputInformation() and a
> > GenerateInputRequestedRegion() (otherwise it as no meaning at all).
> > I was hoping that minipipeline would be processed seamlessly by the
> > recursive call, but maybe I get it wrong.
> >
> >  > In that case, we do not know about the run-time pixel sizes as
> >  > illustrated in point 1. It would work for for images where the
> >  > pixel-size information is known at build time.
> >
> > Well in our toolkit we recommend the use of VectorImage, from which we
> > can retrieve the pixel number of components afther an
> > UpdateOutputInformation(), at run time (but before the
> > UpdateOutputData()). This is not a problem for us because we always have
> > the same number of components for all pixels (our images beeing acquired
> > by remote sensors). Anyway I agree that this does not fit the generic
> case.
> >
> > We need to adress this memory problem. The best would be to know the
> > peak memory usage, but I think we could do well with the overall
> > pipeline memory usage, even if it is only an estimate that takes into
> > account only those DataObjects for which we can compute the memory
> > print. But put this way it sounds more like a workaround and I
> > understand that ITK would not like to have a workaround stuck into its
> > very base classes. We would welcome any other suggestion on idea to
> > solve this problem :o)
> >
> > Thanks a lot for your reply, and for ITK itself,
> >
> > Best regards,
> >
> > Julien MICHEL
> > _______________________________________________
> > Insight-users mailing list
> > Insight-users at itk.org
> > http://www.itk.org/mailman/listinfo/insight-users
> >
>
> _______________________________________________
> Insight-users mailing list
> Insight-users at itk.org
> http://www.itk.org/mailman/listinfo/insight-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://public.kitware.com/pipermail/insight-users/attachments/20071110/8d4fbcd6/attachment.html