[Insight-users] Failed to allocate memory

[Luis Ibanez]

Hi Sajendra,

Sorry that I missed your description of the memory
usage on a 64bits machine....    :-/

You are probably right in that there may be other pieces of the
pipeline that are taking the remaining memory and ramping up the
total use to 4.9Gb.

---

Matt Turek here just suggested something that will help you to find
where the memory is going.

We have recently added to the ITK CVS repository a couple class called

    MemoryProbe
    MemoryProbesCollectorBase

They work very similar to the TimeProbe and TimeProbeCollector.


This class will measure the memory allocations of calls made in between
its invocations to Start() and Stop() methods.


It usage is as follows:

    #include "itkMemoryProbesCollectorBase.h"

    itk::MemoryProbesCollectorBase memorymeter;

    filter1->SetInput( reader->GetOutput() );
    filter2->SetInput( filter1->GetOutput() );
    filter3->SetInput( filter2->GetOutput() );


    memorymeter.Start("Reader");
    reader->Update();
    memorymeter.Stop("Reader");

    memorymeter.Start("Filter1");
    filter1->Update();
    memorymeter.Stop("Filter1");


    memorymeter.Start("Filter2");
    filter2->Update();
    memorymeter.Stop("Filter2");

    memorymeter.Start("Filter3");
    filter3->Update();
    memorymeter.Stop("Filter3");

    memorymeter.Report( std::cout );



Note that it is important to call Update() on the
filter in the order in which the pipeline progresses.
Otherwise you will see all the memory allocations
required by a filter *and its predecessors*.


BTW: In order to use this code you should use ITK 3.6,
      and in CMake, turn ON the option  ITK_USE_REVIEW.


Please run the test above and let us know what you find.


     Thanks


        Luis


-------------------
Luis Ibanez wrote:
> 
> Hi Sajendra,
> 
> I'm confused with your last email.
> 
> In your previous email you describe that you expected to find 2.5Gb
> of memory allocation.
> 
> The amount of 2.5Gb is consistent with what we think the Demons filter
> will require in this case.
> 
> This amount of memory (2.5Gb) cannot be allocated in a standard
> Windows 32 system. The limit of memory for a single process in
> a Window 32 system is 2Gb.
> 
> 
> Could you please clarify what you mean by:
> 
>  1) Expected memory usage
>  2) Observer memory usage
> 
> 
> and what number of GigaBytes are you finding for both items ?
> 
> 
>   Thanks
> 
> 
>      Luis
> 
> 
> 
> --------------------------------
> Nithiananthan, Sajendra wrote:
> 
>> Luis,
>>
>> So the memory usage I observed was much larger than would be expected 
>> (see
>> below).
>> As I mentioned, I was importing the images into ITK, so the increased 
>> memory
>> usage might be a problem further up the pipeline.
>>
>> -Sajendra N.
>>
>>
>>> -----Original Message-----
>>> From: Luis Ibanez [mailto:luis.ibanez at kitware.com]
>>>
>>> Hi Sajendra,
>>>
>>> Yes,
>>> 2.5 Gb sounds about right for the amount of RAM memory that
>>> you would need for running this filter.
>>>
>>> Note however that a process in Windows 32bits system cannot
>>> allocate more than 2Gb of RAM.
>>>    Regards,
>>>
>>>        Luis
>>> -------------------------------
>>> Nithiananthan, Sajendra wrote:
>>>
>>>> Luis,
>>>>
>>>> The pixel type is "float", 4 bytes/pixel. So the image size 
>>>
>>>
>>> is approximately
>>>
>>>> 100 million pixels, or 400MB.
>>>> I chose to represent the deformation field with a vector of 
>>>
>>>
>>> floats, so they
>>>
>>>> are three times as big, or about 1.2GB.
>>>> Specifically:
>>>>    const unsigned int DIMENSION = 3;
>>>>    typedef itk::Image<float, DIMENSION>    ImageType;
>>>>    //representation of the deformation field:
>>>>    typedef itk::Vector<float, DIMENSION>   VectorPixelType;
>>>>    typedef itk::Image<VectorPixelType, DIMENSION>  
>>>
>>>
>>> DeformationFieldType;
>>>
>>>> So to represent the fixed image, moving image, deformation 
>>>
>>>
>>> field, and output
>>>
>>>> image, I would expect usage about 2.5GB?
>>>>
>>>> -Sajendra
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Luis Ibanez [mailto:luis.ibanez at kitware.com]
>>>>> Subject: Re: [Insight-users] Failed to allocate memory
>>>>>
>>>>> Hi Sajendra,
>>>>>
>>>>> What was the image size (in pixels) ?
>>>>>
>>>>> From a 400Mb image, if we assume that you are using "signed short"
>>>>> as pixel type, then we are probably talking about:
>>>>>
>>>>>                      200 Million pixels
>>>>>
>>>>> Demons will internally construct a deformation field with a 
>>>
>>>
>>> 3D vector
>>>
>>>>> of doubles per pixel. That will be:
>>>>>
>>>>>              2e8 x 3 x 8 bytes = 4.9 Gigabytes.
>>>>>
>>>>> To this you will have to add the 400Mb of the output image.
>>>>>
>>>>>
>>>>> Therefore your observed memory actually looks small for 
>>>
>>>
>>> this problem.
>>>
>>>>>
>>>>>   Regards,
>>>>>
>>>>>
>>>>>
>>>>>      Luis
>>>>>
>>>>>
>>>>>
>>>>> -------------------------------
>>>>> Nithiananthan, Sajendra wrote:
>>>>>
>>>>>
>>>>>> I was in a situation similar to the OP, Zhihong, trying to 
>>>>>
>>>>>
>>>>> use ITK based
>>>>>
>>>>>
>>>>>> Demons registration on larger images (~400MB). It failed 
>>>>>
>>>>>
>>>>> spectacularly  on
>>>>>
>>>>>
>>>>>> 32 bit Windows. Compiling and running on a 64 bit machine 
>>>
>>>
>>> solved the
>>>
>>>>>> problem.
>>>>>> The peak memory was about 4.5 GB, which seems high (no 
>>>>>
>>>>>
>>>>> histogram matching or
>>>>>
>>>>>
>>>>>> casting used). I had imported the images into ITK from VTK 
>>>>>
>>>>>
>>>>> so perhaps that
>>>>>
>>>>>
>>>>>> explains some of the increased memory usage. Either way, it 
>>>>>
>>>>>
>>>>> did run well and
>>>>>
>>>>>
>>>>>> gave me the results I would have expected (compared the 
>>>>>
>>>>>
>>>>> output for a smaller
>>>>>
>>>>>
>>>>>> image to that from a 32bit machine). I will hopefully 
>>>>>
>>>>>
>>>>> confirm later that
>>>>>
>>>>>
>>>>>> nothing strange is happening with memory usage.
>>>>>>
>>>>>> Thank you for your input
>>>>>> -Sajendra
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>
>> This e-mail may contain confidential and/or privileged information for 
>> the sole use of the intended recipient. Any review or distribution by 
>> anyone other than the person for whom it was originally intended is 
>> strictly prohibited. If you have received this e-mail in error, please 
>> contact the sender and delete all copies. Opinions, conclusions or 
>> other information contained in this e-mail may not be that of the 
>> organization.
>>
>> _______________________________________________
>> Insight-users mailing list
>> Insight-users at itk.org
>> http://www.itk.org/mailman/listinfo/insight-users
>>
>