[Insight-users] [ITK Community] Working with big dataset.

[Bradley Lowekamp]

Hello,

For IO streaming to work the ImageIO you are using needs to support it. File formats which are supported for stream reading include MetaIO (mha,mhd), VTK, MRC, SCFIO, NIFTI, HDF5 and the SeriesReader. So RAW files are not supported for streaming. Further more RAW, is very much not recommended because it's missing the meta-data.

You can use the mhd/mha file format for full streaming for reading and writing. Additionally you can use you same raw file too! What you need to do is write a mhd header file, which is well described here[1].  The mhd file will contain a description of the raw file, what you now have hard coded. If you specify the output file with an "mhd", extension an mhd and a raw file will be created. Doing it this way you will not need to specify the ImageIO for the reader or writer, and the ITK ImageIOFactory will choose the correct one.

Hope that helps,
Brad

[1] http://www.itk.org/Wiki/ITK/MetaIO/Documentation

On Mar 13, 2014, at 7:12 AM, Yansen <d9999 at poczta.onet.pl> wrote:

> Hello. 
> 
> I try to write program that can handle huge files. My data are saved in raw
> file. My first step into ITK was try to write program that can basically
> copy data from one file to another. Basically I success - if data files are
> small and can fit into memory ("small.raw"). But the reason I wish to use
> ITK is not small files but large ones (i typically got 100GB input file and
> my computer got only 8 GB memory). Anyone can drive me to some complete
> example how to do this? Example below fail with 12BG file. I read many
> samples and try witch regions and StreamingImageFilter without success. 
> 
> Best regard 
> Yansen 
> 
> 
> #include "itk\itkImageFileReader.h"
> #include "itk\itkImageFileWriter.h"
> #include "itk\itkImage.h"
> #include "itk\itkRawImageIO.h"
> #include "itk\itkPNGImageIO.h"
> #include "itk\itkRegionOfInterestImageFilter.h"
> #include "itk\itkStreamingImageFilter.h"
> 
> int main( int argc, char ** argv )
> {
> //input file
> typedef unsigned char InputDataType;
> typedef itk::Image< InputDataType, 3 >    InputImageType;
> 
> itk::RawImageIO< InputDataType, 3 >::Pointer inputRAW = itk::RawImageIO<
> InputDataType, 3 >::New();
> itk::ImageFileReader< itk::Image<InputDataType, 3 >>::Pointer reader =
> itk::ImageFileReader< itk::Image<InputDataType, 3 >>::New();
> 
> //inputRAW->SetFileName("e:\\vol_and_raws\\small.raw");
> inputRAW->SetFileName("e:\\vol_and_raws\\big.raw");
> inputRAW->SetFileTypeToBinary();
> inputRAW->SetHeaderSize(0); 
> inputRAW->SetFileDimensionality(3);
> 
> //inputRAW->SetDimensions( 0, 571 );  
> //inputRAW->SetDimensions( 1, 571 );
> //inputRAW->SetDimensions( 2, 426 );
> 
> inputRAW->SetDimensions( 0, 2283 ); 
> inputRAW->SetDimensions( 1, 2284 ); 
> inputRAW->SetDimensions( 2, 2304 );
> 
> inputRAW->SetPixelType(itk::ImageIOBase::SCALAR);
> inputRAW->SetNumberOfComponents(1);
> 
> reader->SetFileName(inputRAW->GetFileName());
> reader->SetImageIO(inputRAW);
> reader->UpdateOutputInformation();
> 
> //output file
> typedef unsigned char OutputDataType;
> typedef itk::Image< OutputDataType, 3 >    OutputImageType;
> 
> itk::RawImageIO< OutputDataType, 3 >::Pointer outputRAW = itk::RawImageIO<
> OutputDataType, 3 >::New();
> itk::ImageFileWriter< itk::Image<OutputDataType, 3 >>::Pointer writer =
> itk::ImageFileWriter< itk::Image<OutputDataType, 3 >>::New();
> 
> outputRAW->SetFileName("e:\\vol_and_raws\\big_wr.vol");
> outputRAW->SetFileTypeToBinary();
> outputRAW->SetHeaderSize(0); 
> outputRAW->SetFileDimensionality(3);
> 
> //outputRAW->SetDimensions( 0, 571 );  
> //outputRAW->SetDimensions( 1, 571 );
> //outputRAW->SetDimensions( 2, 426 );
> 
> outputRAW->SetDimensions( 0, 2283 );  
> outputRAW->SetDimensions( 1, 2284 );  
> outputRAW->SetDimensions( 2, 2304 );
> 
> outputRAW->SetPixelType(itk::ImageIOBase::SCALAR);
> outputRAW->SetNumberOfComponents(1);
> 
> writer->SetFileName(outputRAW->GetFileName());
> writer->SetImageIO(outputRAW);
> writer->UpdateOutputInformation();
> 
> 
> 
> writer->SetInput(reader->GetOutput());
> writer->SetNumberOfStreamDivisions( 300 );
> 
> system("PAUSE");
> 
> try
>   {
>    writer->Update();
>   }
>  catch( itk::ExceptionObject & err )
>    {
>    std::cerr << "ExceptionObject caught !" << std::endl;
>    std::cerr << err << std::endl;
>    return EXIT_FAILURE;
>    }
> 
> 	
>  system("PAUSE");
>  return EXIT_SUCCESS;
> }
> 
> 
> 
> --
> View this message in context: http://itk-users.7.n7.nabble.com/Working-with-big-dataset-tp33564.html
> Sent from the ITK - Users mailing list archive at Nabble.com.
> _____________________________________
> Powered by www.kitware.com
> 
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
> 
> Kitware offers ITK Training Courses, for more information visit:
> http://www.kitware.com/products/protraining.php
> 
> Please keep messages on-topic and check the ITK FAQ at:
> http://www.itk.org/Wiki/ITK_FAQ
> 
> Follow this link to subscribe/unsubscribe:
> http://www.itk.org/mailman/listinfo/insight-users
> _______________________________________________
> Community mailing list
> Community at itk.org
> http://public.kitware.com/cgi-bin/mailman/listinfo/community