[Insight-users] itk.PyBuffer memory leak

[Rick Giuly]

Your test code is working for me and I think I can use it to make a work 
around. I'll be reporting more results in a few days.

My testing indicates that you can use numpy.ones() to create the numpy 
array, and the conversion (to itk) will happen without error if array 
dimensions are reasonably small - like (100, 100, 10). Although, if you 
convert many arrays (from numpy.ones()), eventually memory runs out.

If the array is larger, like the dimension you were using for testing, 
the behavior was somewhat different.


Thanks

-Rick

Gaëtan Lehmann wrote:
> 
> My test also work on the stable version of wrapitk (with a few changes 
> in command manipulation).
> 
> I tried with the test you provided, and this time I get an error:
> 
>    RuntimeError: Contiguous array couldn't be created from input python 
> object
> 
> This error is sent from PyBuffer when it can't convert the input python 
> object to the expected type. Maybe the array returned by numpy.ones() is 
> not a contigus array?
> 
> Gaëtan
> 
> 
> Le 22 mai 09 à 15:56, Gaëtan Lehmann a écrit :
> 
>>
>> Hi Rick,
>>
>> Yes, I looked at it in wrapitk *unstable* - it may not be the one 
>> you're using.
>> I can't reproduce the problem you have. Here is the test I've run:
>>
>> #!/usr/bin/env python
>>
>> import itk, sys, gc
>> itk.auto_progress()
>>
>> # image size is 1 GB
>> # IT = itk.Image.UC3
>> # img = IT.New(Regions=1000)
>>
>> # image size is 1 GB
>> IT = itk.Image.F3
>> img = IT.New(Regions=[1000, 1000, 250])
>>
>> img.Allocate()
>> img.FillBuffer(0)
>>
>> print img
>>
>> # exercise buffer conversion from itk to numpy
>> for i in range(1000):
>>  buf = itk.PyBuffer[IT].GetArrayFromImage(img)
>> del buf
>>
>> # exercise buffer conversion from numpy to itk
>>
>> count = 0
>>
>> def callback():
>>  global count
>>  sys.stderr.write(str(count)+"\t")
>>  if count%10 == 0:
>>    sys.stderr.write("\n")
>>  sys.stderr.flush()
>>  count += 1
>> com = itk.PyCommand.New()
>> com.SetCommandCallable( callback )
>>
>> buf = itk.PyBuffer[IT].GetArrayFromImage(img)
>>
>> for i in range(1000):
>>  bi = itk.PyBuffer[IT].GetImageFromArray(buf)
>>  bi.AddObserver( itk.DeleteEvent(), com )
>> del bi
>>
>> # force garbage collection
>> gc.collect()
>>
>> sys.stderr.write(str(count)+"\n")
>>
>> sys.exit(abs(1000-count))
>>
>>
>>
>>
>> I'm now looking at wrapitk stable. I'll let you know what I'll found.
>>
>> Gaëtan
>>
>>
>> Le 22 mai 09 à 06:16, Rick Giuly a écrit :
>>
>>> Hi Gaëtan,
>>>
>>> Have you had a chance to look at this?
>>>
>>> thanks
>>>
>>> -rick
>>>
>>> Gaëtan Lehmann wrote:
>>>> Le 13 mai 09 à 10:35, Rick Giuly a écrit :
>>>>>
>>>>> I've now tested invoking garbage collection with gc.collect() at 
>>>>> every iteration and that didn't change the problem. So, my best 
>>>>> guess would be that the image is not deleted when the python object 
>>>>> is destroyed. Maybe a bug?
>>>> Maybe - can you file a bug on ITK's bug tracker and assign it to me?
>>>> I'll try to find a bit of time to investigate this in the next days.
>>>>>
>>>>>
>>>>>
>>>>> Gaëtan Lehmann wrote:
>>>>>> Hi Rick,
>>>>>> GetImageFromArray() returns a smart pointer to an image, so the 
>>>>>> image should be deallocated once the python object is destroyed.
>>>>>> Perhaps the problem is there: the garbage collection my not run 
>>>>>> fast enough.
>>>>>> The right way to go may be to implement PyBuffer as a filter which 
>>>>>> reuse the output image, as all the filters in itk.
>>>>>> For the Delete() method: you shouldn't use it in python (and I 
>>>>>> can't see any case for use it but the internal memory management 
>>>>>> in c++). In the latest version of wrapitk (the one hosted at 
>>>>>> googlecode), Delete() and the other methods related to smart 
>>>>>> pointers are hidden.
>>>>>> Regards,
>>>>>> Gaëtan
>>>>>> Le 11 mai 09 à 05:43, Rick Giuly a écrit :
>>>>>>>
>>>>>>> Hello All,
>>>>>>>
>>>>>>> It seems that converter.GetImageFromArray(inputNumpyVolume) 
>>>>>>> allocates memory in some way and never releases it. When I tried 
>>>>>>> using Delete() the program actually crashed silently after one 
>>>>>>> iteration.
>>>>>>>
>>>>>>> Test code is below. (I'm running this on ubuntu, and the itk 
>>>>>>> package is from Paul Novo's site.)
>>>>>>>
>>>>>>> Is there some way to release memory?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> import itk
>>>>>>> import numpy
>>>>>>>
>>>>>>> for i in range(10000):
>>>>>>>
>>>>>>> print i
>>>>>>>
>>>>>>> ImageType = itk.Image[itk.F, 3]
>>>>>>> converter = itk.PyBuffer[ImageType]
>>>>>>>
>>>>>>> inputNumpyVolume = numpy.ones((100, 100, 200))
>>>>>>> inputVolume = converter.GetImageFromArray(inputNumpyVolume)
>>>>>>> #inputVolume.Delete()
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ----------
>>>>>>> Thanks,
>>>>>>> --Rick
>>>>>>> _____________________________________
>>>>>>> Powered by www.kitware.com
>>>>>>>
>>>>>>> Visit other Kitware open-source projects at
>>>>>>> http://www.kitware.com/opensource/opensource.html
>>>>>>>
>>>>>>> Please keep messages on-topic and check the ITK FAQ at: 
>>>>>>> http://www.itk.org/Wiki/ITK_FAQ
>>>>>>>
>>>>>>> Follow this link to subscribe/unsubscribe:
>>>>>>> http://www.itk.org/mailman/listinfo/insight-users
>>>>>
>>>
>>
>> -- 
>> Gaëtan Lehmann
>> Biologie du Développement et de la Reproduction
>> INRA de Jouy-en-Josas (France)
>> tel: +33 1 34 65 29 66    fax: 01 34 65 29 09
>> http://voxel.jouy.inra.fr  http://www.mandriva.org
>> http://www.itk.org  http://www.clavier-dvorak.org
>>
>