[ITK-users] why can the itkStatisticsImageFilter not stream?

Fri Apr 17 14:13:27 EDT 2015

Hello,

Glad to hear it worked for you.

I have just fixed the progress reporting [1].

The StreamingImageFilter overloads the UpdateData method [2], by passing the standard ImageToImage/ImageSource methods such and GenerateData, ThreadedGenerateData etc. It's actually not really designed to be derived from, and it used mostly to drive the streaming of a pipeline. It can be seen as slicing up the output image in to pieces, and requesting those pieces from the pipeline, the assembling them into the output image.

The resulting behavior of the ImageSinc, with the StreamingProcessObject goes something like this:

BeforeStreamedGenerateData()

For each requested streamed chunk:
      For each thread process part of streamed chunk:
          ThreadedStreamedGenerateData()

AfterStreamedGenerateData()

So the input image is getting sliced up twice the first for streaming and the second for multi-threading.

The StreamingImageFitler could be rewritten onto this framework to perhaps provide a generalized base class.

For the StreamingStatisticsImage filter, the statistics are accumulated for each thread multiple times ( for each streamed chunk ), then the totals from the threads are reduced to the results.

These classes certainly need more documentation before further integration into ITK.

Brad

[1] https://github.com/blowekamp/itkStreamingSinc/commit/d8415cf137d69a0e81b6dd46145d8cb7df2825c3
[2] https://github.com/InsightSoftwareConsortium/ITK/blob/8f7c404aff99f5ae3dfedce6e480701f0304864c/Modules/Core/Common/include/itkStreamingImageFilter.hxx#L118-Lundefined

On Apr 17, 2015, at 12:12 PM, Grothausmann, Roman Dr. <grothausmann.roman at mh-hannover.de> wrote:

> Dear Brad, dear Richard,
> 
> 
> Many thanks for Your replies. So, if I understand You correctly, the problem is that the current framework (with StreamingImageFilter) will run AfterThreadedGenerateData for each chunk and not just when the whole image has been processed. So one would need to combine StatisticsImageFilter and StreamingImageFilter into one filter, i.e. basically that the part of AfterThreadedGenerateData is executed by StreamingImageFilter.
> If I understand the code of itkStreamingSinc correctly, this is what it does? itkStreamingSinc provides a base class to combine multi-threaded filters with a StreamingImageFilter to make them streamable. This comes with the trade-off that the resulting filter will not provide an output, which is no problem if it is not needed. So, e.g. itkStreamingStatisticsImageFilter cannot be connected to itkPipelineMonitorImageFilter as it would need an output of itkStreamingStatisticsImageFilter?
> 
> Many thanks Brad for implementing itkStreamingStatisticsImageFilter. I just tested it and it works perfectly (https://github.com/romangrothausmann/ITK-CLIs/commit/38bcad35bf004feaf2189bdaaffcf398fbca1f60).
> Though, I do not understand the changes You made in: https://github.com/blowekamp/itkStreamingSinc/commit/b4ae9b61dc08388fbf1b3bfd9e5cf570dd40a0c6
> Is the accumulation that still resides in AfterStreamedGenerateData not used at all? (https://github.com/blowekamp/itkStreamingSinc/commit/dae948bea0966a2c3924351bf9c15bfcef318ac6#diff-ad64d497b8f9aadd5da97c5e1c793dc9R210)
> Just as a side note: The progress reported by itkStreamingStatisticsImageFilter jumps to 100% for each chunk.
> 
> All in all Your itkStreamingSinc and the implemented streaming-filters are really a great contribution to ITK and I fully recommend incorporating it into ITK.
> 
> Many thanks again
> Roman
> 
> 
> On 17/04/15 15:47, Bradley Lowekamp wrote:
>> Hi,
>> 
>> I was able to easy update a the StatisticsImageFilter to use my streaming
>> base class for reduction algorithms [1]. It runs basically the same expect
>> that you can set the number of stream divisions[2], there is the potential
>> for setting the way the input image is split via the RegionSplitter in a base
>> class [3].
>> 
>> You should be able to just clone the repository into you ITK/Modules/External
>> directory, then enable it in the CMake configuration to use.
>> 
>> Please let me know if it works for you.
>> 
>> This should help :) Brad >
>> [1] https://github.com/blowekamp/itkStreamingSinc/blob/master/include/itkStreamingStatisticsImageFilter.h
>> [2] https://github.com/blowekamp/itkStreamingSinc/blob/master/test/itkStreamingStatisticsImageFilterTest2.cxx#L61
>> [3] https://github.com/blowekamp/itkStreamingSinc/blob/master/include/itkImageSinc.h#L99
>> 
>> 
>> 
>> On Apr 17, 2015, at 7:14 AM, Bradley Lowekamp <blowekamp at mail.nih.gov>
>> wrote:
>> 
>>> Hello fellow mailing list member,
>>> 
>>> Currently in ITK there are two base classes for streaming images in ITK.
>>> The
> ImageToImageFilter provide the framework of streaming one input region to one
> output region. The other is the StreamingImageFilter which provides the
> framework to stream multiple image regions and merge them together into a single
> output image. This StatisticsImageFilter needs a framework which consumes
> multiple input image regions and reduces the data produced. I happen to have
> implemented such a framework here [1]. It currently has the
> LabelStatisticsImageFilter implemented in a streaming fashion.
>>> 
>>> Let me give it a try and see how quickly I can get the
>>> StatisticsImageFilter
> in this framework. Help writing tests would be appreciated.
>>> 
>>> Thanks, Brad
>>> 
>>> [1] https://github.com/blowekamp/itkStreamingSinc
>>> 
>>> On Apr 17, 2015, at 5:08 AM, Dr. Roman Grothausmann <grothausmann.roman at mh-hannover.de> wrote:
>>> 
>>>> Dear mailing list members,
>>>> 
>>>> 
>>>> I'm wondering why the itkStatisticsImageFilter cannot stream?
>>>> Looking into itkStatisticsImageFilter.hxx it requests the whole input in GenerateInputRequestedRegion and EnlargeOutputRequestedRegion but in ThreadedGenerateData only iterates over the outputRegionForThread.
>>>> Where in the code is the whole input image needed as one single block?
>>>> 
>>>> Thanks for any hints
>>>> Roman
>>>> 
>>>> --
>>>> Dr. Roman Grothausmann
>>>> 
>>>> Tomographie und Digitale Bildverarbeitung
>>>> Tomography and Digital Image Analysis
>>>> 
>>>> Institut für Funktionelle und Angewandte Anatomie, OE 4120
>>>> Medizinische Hochschule Hannover
>>>> Carl-Neuberg-Str. 1
>>>> D-30625 Hannover
>>>> 
>>>> Tel. +49 511 532-2900
>>>> _____________________________________
>>>> Powered by www.kitware.com
>>>> 
>>>> Visit other Kitware open-source projects at
>>>> http://www.kitware.com/opensource/opensource.html
>>>> 
>>>> Kitware offers ITK Training Courses, for more information visit:
>>>> http://www.kitware.com/products/protraining.php
>>>> 
>>>> Please keep messages on-topic and check the ITK FAQ at:
>>>> http://www.itk.org/Wiki/ITK_FAQ
>>>> 
>>>> Follow this link to subscribe/unsubscribe:
>>>> http://public.kitware.com/mailman/listinfo/insight-users
>>> 
>>> _____________________________________
>>> Powered by www.kitware.com
>>> 
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>> 
>>> Kitware offers ITK Training Courses, for more information visit:
>>> http://www.kitware.com/products/protraining.php
>>> 
>>> Please keep messages on-topic and check the ITK FAQ at:
>>> http://www.itk.org/Wiki/ITK_FAQ
>>> 
>>> Follow this link to subscribe/unsubscribe:
>>> http://public.kitware.com/mailman/listinfo/insight-users
>> 
> 
> -- 
> Dr. Roman Grothausmann
> 
> Tomographie und Digitale Bildverarbeitung
> Tomography and Digital Image Analysis
> 
> Institut für Funktionelle und Angewandte Anatomie, OE 4120
> Medizinische Hochschule Hannover
> Carl-Neuberg-Str. 1
> D-30625 Hannover
> 
> Tel. +49 511 532-2900