[Insight-developers] read stdin, write stdout

Bill Lorensen bill.lorensen at gmail.com
Tue Oct 1 07:44:41 EDT 2013


We did a memory read/write imageio for Slicer4. The code is here:
http://viewvc.slicer.org/viewvc.cgi/Slicer4/trunk/Libs/MRML/IDImageIO/
 A similar approach might be used to read/write pipes.

Bill


On Tue, Oct 1, 2013 at 1:43 AM, David Froger <david.froger at inria.fr> wrote:
> Dear all,
>
> We are looking for a proper way to integrate in ITK reading/writing
> in standard input/output HDF5 images.
>
> We write a "proove concept" that works, but involves a lot of code
> duplication that should be avoided. The rest of this long mail is divided
> into these sections:
>     - Motivations
>     - HDF5 and unix pipe
>     - Building ITK with recent HDF5 version
>     - itkHDF5unixpipeImageIO module
>     - ImageUnixPipeReader module
>     - Using the new modules
>     - Questions
>     - References
>
> We can create topic branch as described in the Git workflow [2] if need.
>
> Thanks for reading!
>
> Best regards,
> David Froger
>
>
> Motivations
> -----------
>
> We are rewriting command line tools based on the INRimage [1] image
> format/library so that they are now based ITK.
>
> A particularity of these command line tools is that they can be piped with unix
> pipe. For example, the 'so' command subtracts the pixel values of two images,
> while the 'ical' command computes min/max/average pixel values of a image. To
> test if two images are equals, we pipe the commands as follow:
>     so image1.inr image2.inr | ical
>
> While we are aware of the ITK pipeline and that we can use it to pipe ITK
> sources/mappers/filters into one processus (as suggested in [3] for example),
> we want to keep the possibility of using unix pipe in order to keep a backward
> compatibilty because we use numerous bash and perl scripts.
>
> HDF5 and unix pipe
> ------------------
>
> The new file image operations library [4], introduced  in HDF5 at the 1.8.9 version
> on May 2012 makes easy to read/write HDF5 images on standard input/output.
>
> An example of writting an HDF5 image to standard output is:
>
>     #include <stdio.h>
>     #include "hdf5.h"
>     #include "H5Cpp.h"
>     #include "H5LTpublic.h"
>
>     // C array -> HDF5 image file -> buffer -> stdout
>
>     #define NDATA 5
>
>     int main()
>     {
>         // Create C array.
>         int data[NDATA] = {2, 4, 8, 16, 32};
>
>         // Open HDF5 image file.
>         hbool_t backing_store = 0;
>         H5::FileAccPropList fapl;
>         fapl.setCore((size_t)1,backing_store);
>         H5::H5File file("nosuch.h5",H5F_ACC_TRUNC,H5::FileCreatPropList::DEFAULT,fapl);
>
>         // Write C array in HDF5 image file.
>         hsize_t dims[1]= {NDATA};
>         H5::DataSpace space(1,dims);
>         H5::DataSet dset = file.createDataSet("pow2",H5::PredType::NATIVE_INT,space);
>         dset.write(data, H5::PredType::NATIVE_INT);
>
>         // Get HDF5 image file buffer.
>         file.flush(H5F_SCOPE_GLOBAL);
>         hid_t fileid = file.getId();
>         size_t size = H5Fget_file_image(fileid, NULL, 0);
>         char* buffer = new char[size];
>         H5Fget_file_image(fileid, buffer, size);
>
>         // Write buffer to standard output.
>         fwrite(buffer,1,size,stdout);
>     }
>
> Compile and run:
>
>   g++ -o write_h5_stdout write_h5_stdout.cxx -lhdf5_hl -lhdf5 -lhdf5_cpp
>   ./write_h5_stdout > f.hdf5
>
> A example of reading this f.hdf5 file from the standard input is:
>
>     #include <stdio.h>
>     #include <stdlib.h>
>     #include "hdf5.h"
>     #include "H5Cpp.h"
>     #include "H5LTpublic.h"
>
>     #define NDATA 5
>     #define READ_SIZE 512
>     #define CHUNCK_SIZE 8192
>
>     // stdin -> buffer -> HDF5 image file -> C array
>
>     main()
>     {
>         // size of buffer is buffer_size=CHUNCK_SIZE*nchunck, size of h5data is
>         // h5data_size <= buffer_size
>         if (CHUNCK_SIZE < READ_SIZE) {
>            fprintf(stderr, "ERROR: CHUNCK_SIZE < READ_SIZE\n");
>            exit(EXIT_FAILURE);
>         }
>
>         // Read stdin into buffer.
>         size_t nchunck = 1, n;
>         size_t buffer_size = CHUNCK_SIZE*nchunck;
>         char* buffer = (char*) malloc(buffer_size);
>         size_t h5data_size = 0;
>         while (1) {
>             // Make sure buffer is big enough to receive READ_SIZE new bytes.
>             if (h5data_size+READ_SIZE > buffer_size) {
>                 nchunck += 1;
>                 buffer_size = nchunck*CHUNCK_SIZE;
>                 buffer = (char*) realloc(buffer,buffer_size);
>             }
>             // copy from stdin to buffer.
>             n = fread(buffer+h5data_size,1,READ_SIZE,stdin);
>             h5data_size += n;
>
>             if (n <= 0) break;
>         }
>
>         // Create HDF5 image file from buffer.
>         unsigned flags = H5LT_FILE_IMAGE_DONT_COPY | H5LT_FILE_IMAGE_DONT_RELEASE;
>         hid_t fileid = H5LTopen_file_image(buffer, h5data_size, flags);
>         H5::H5File *h5file;
>         h5file = new H5::H5File;
>         h5file->setId(fileid);
>
>         // Read C array in HDF5 file image.
>         int data[NDATA];
>         H5::DataSet dset = h5file->openDataSet("pow2");
>         dset.read(data,dset.getDataType());
>
>         for (int i=0 ; i<NDATA ; i++) {
>             printf("%d\n",data[i]);
>         }
>
>         free(buffer);
>
>         return 0;
>     }
>
> Compile and run:
>
>   g++ -o read_h5_stdin read_h5_stdin.cxx -lhdf5_hl -lhdf5 -lhdf5_cpp
>   cat f.hdf5 | ./read_h5_stdin
>   ./write_h5_stdout | ./read_h5_stdin
>
> Building ITK with recent HDF5 version
> -------------------------------------
>
> ITK must be compiled and and linked with a HDF5 > 1.8.9 (the current ITK 4.4.0
> embeds HDF5==1.8.7).
>
> itkHDF5unixpipeImageIO module
> -----------------------------
>
> In order to read/write HDF5 in standard input/output, we have created a
> itkHDF5unixpipeImageIO module. To this purpose, we have copied the directory
> Modules/IO/HDF5 into Modules/External/HDF5unixpipe (a bad practice that should
> be fixed as it duplicates a lot of code).
>
> src/itkHDF5unixpipeImageIO.cxx is the same as src/itkHDF5ImageIO.cxx, except
> that:
>
> - CanWriteFile and CanReadFile test on file extensions .h5stdout and .h5stdin,
>   CanReadFile no more test on file existance.
>
> - An helper function ReadStandardInput is added:
>
>     void
>     HDF5unixpipeImageIO
>     ::ReadStandardInput()
>     {
>         // Do not call the function twice.
>         if (this->m_StandardInputReaden) return;
>
>         /* Size of m_H5FileImageBuffer is H5FileImageBufferSize = chunckSize*nChunck.
>          * Size of actual data is m_H5FileImageDataSize <= H5FileImageBufferSize.
>          */
>
>         size_t readSize = 512;
>         size_t chunckSize = 8192;
>         size_t nChunck = 1, nBytesReaden;
>         size_t H5FileImageBufferSize = chunckSize*nChunck;
>
>         this->m_H5FileImageBuffer = (char*) malloc(H5FileImageBufferSize);
>         this->m_H5FileImageDataSize = 0.;
>
>         while (1) {
>             /* make sure m_H5FileImageBuffer is big enough to receive readSize new bytes */
>             if (this->m_H5FileImageDataSize+readSize > H5FileImageBufferSize) {
>                 nChunck += 1;
>                 H5FileImageBufferSize = nChunck*chunckSize;
>                 this->m_H5FileImageBuffer = (char*) realloc(this->m_H5FileImageBuffer,H5FileImageBufferSize);
>             }
>
>             /* copy from stdin to m_H5FileImageBuffer */
>             nBytesReaden  = fread(this->m_H5FileImageBuffer+this->m_H5FileImageDataSize,1,readSize,stdin);
>             this->m_H5FileImageDataSize += nBytesReaden;
>
>             if (nBytesReaden <= 0) break;
>         }
>
>         this->m_StandardInputReaden = true;
>     }
>
> In ReadImageInformation():
>
>     this->m_H5File = new H5::H5File(this->GetFileName(),
>                                     H5F_ACC_RDONLY);
>
> is replaced with:
>
>     // Read stdin into m_H5FileImageBuffer.
>     this->ReadStandardInput();
>
>     // Create HDF5 image file from buffer.
>     unsigned flags = H5LT_FILE_IMAGE_DONT_COPY | H5LT_FILE_IMAGE_DONT_RELEASE;
>     hid_t fileid = H5LTopen_file_image(this->m_H5FileImageBuffer, this->m_H5FileImageDataSize, flags);
>     this->m_H5File = new H5::H5File;
>     this->m_H5File->setId(fileid);
>
> in WriteImageInformation(void):
>
>     this->m_H5File = new H5::H5File(this->GetFileName(),
>                                     H5F_ACC_TRUNC);
>
> is replaced with:
>
>     hbool_t backing_store = 0;
>     H5::FileAccPropList fapl;
>     fapl.setCore((size_t)1,backing_store);
>     this->m_H5File = new H5::H5File("nosuch.h5",H5F_ACC_TRUNC,
>         H5::FileCreatPropList::DEFAULT,fapl);
>
> In Write(const void *buffer), at this end of the try block, we add:
>
>     // Get HDF5 image file buffer.
>     this->m_H5File->flush(H5F_SCOPE_GLOBAL);
>     hid_t fileid = this->m_H5File->getId();
>     size_t imageFileBufferSize = H5Fget_file_image(fileid, NULL, 0);
>     char* imageFileBuffer = (char*) malloc(imageFileBufferSize);
>     H5Fget_file_image(fileid, imageFileBuffer, imageFileBufferSize);
>
>     // Write imageFileBuffer to standard output.
>     fwrite(imageFileBuffer,1,imageFileBufferSize,stdout);
>
>
> ImageUnixPipeReader module
> --------------------------
>
> In the directory ./IO/ImageBase/include, we copied itkImageFileReader.h
> in itkImageUnixPipeReader.h and itkImageFileReader.hxx  in
> itkImageUnixPipeReader.hxx (again, a bad practice that should
> be fixed as it duplicates a lot of code).
>
> The only change is to remove the use of the TestFileExistanceAndReadability
> function.
>
> Note that itkHDF5unixpipe can be used directly with ImageFileWritter.
>
> Using the new modules
> ---------------------
>
> The new modules to read/write in standard input/output can now be used as the
> ones that read in files, but by specifying "virtual" file names with extension
> ".h5stdin" and ".h5stdout", so that ImageUnixPipeReader and ImageFileWritter
> know they can make use of itkHDF5unixpipe, if we want to add support of other
> image formats.
>
> Missing features
> ----------------
>
> A module ImageUnixPipeWriter should be added. ImageUnixPipeWriter and
> ImageUnixPipeReader, should not test on file extension (there is no file name),
> but require an ImageIO to be explicity passed.
>
> Questions
> ---------
>
> 1. Is there any chance that this functionnalty could be integrated in ITK?
> (if not, I suppose we should put everything in Modules/external).
>
> 2. In either case, how to avoid code duplication? Can we use class inheritance?
>
> References
> ----------
>
> [1] (in french) http://inrimage.gforge.inria.fr/WWW/index.html
> [2] http://www.itk.org/Wiki/ITK/Git/Develop
> [3] http://www.itk.org/pipermail/insight-users/2004-September/010274.html
> [4] http://www.hdfgroup.org/HDF5/doc/Advanced/FileImageOperations/HDF5FileImageOperations.pdf
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Kitware offers ITK Training Courses, for more information visit:
> http://kitware.com/products/protraining.php
>
> Please keep messages on-topic and check the ITK FAQ at:
> http://www.itk.org/Wiki/ITK_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.itk.org/mailman/listinfo/insight-developers



-- 
Unpaid intern in BillsBasement at noware dot com


More information about the Insight-developers mailing list