[Insight-users] Image index limits
Robert.Atwood at diamond.ac.uk
Robert.Atwood at diamond.ac.uk
Wed Jan 13 06:57:23 EST 2010
Hi, All:
I believe I have pinpointed the problem here. It is due to the use of
::gzread to read the file whether or not the file is compressed, and to
do this in a single call. The ::gzread function is limited to the size
of "unsigned int" and thus cannot read more than 4gb in a single call.
Unfortunately this appears to apply even when the file is not
compressed, i.e. the fallback behaviour is not exactly identical to the
normal file reading operation.
I am not sure what the preferred way to solve this would be; separate
compressed and uncompressed thus not relying on gzread for uncompressed
file; test the requested size and divide the reading into chunks if
necessary; test the return value and read more chunks if necessary until
the whole file is read?
itkAnalyzeImageIO.cxx:
705 //NOTE: gzFile operations act just like FILE * operations when
the files
706 // are not in gzip fromat.
This is not quite true! The data type of the requested amount of data is
limited to "unsigned int" in gzread.
707 // This greatly simplifies the following code,
I believe this simplification will have to be sacrificed to allow files
> 4Gb in this format.
See below, the code, its output, and some extracts from the zlib web
page.
I hope this helps.
Robert
itkAnalyzeImageIO.cxx: (Modified)
745 // applied for every image in the file.
746 long int byteOffset = static_cast<long
int>(fabs(m_Hdr.dime.vox_offset));
747 long int retval; //ROBERT
748 if (byteOffset > 0)
749 {
750 ::gzseek(file_p, byteOffset, SEEK_SET);
751 }
752
753 // read image in
754 std::cout << "this->GetImageSizeInBytes: " <<
this->GetImageSizeInBytes() << std::endl; //ROBERT
755
756 //ORIGINAL ::gzread( file_p, p, static_cast< unsigned >(
this->GetImageSizeInBytes() ) );
757 retval = ::gzread( file_p, p, static_cast< SizeType >(
this->GetImageSizeInBytes() ) ); //ROBERT
758
759 std::cout << "cast SizeType .. this->GetImageSizeInBytes: " <<
static_cast < SizeType > (this->GetImageSizeInBytes()) << std::endl;
//ROBERT
760 std::cout << "return value of gzread: " << retval << std::endl;
//ROBERT
761 gzclose( file_p );
762 SwapBytesIfNecessary( buffer, numberOfPixels );
763 }
764
765
Output of JustReadWrite:
NumberOfPixels = 1254240000
ImageFileName: test_cvs.img
this->GetImageSizeInBytes: 5016960000
cast SizeType .. this->GetImageSizeInBytes: 5016960000
return value of gzread: 721992704
http://www.zlib.net/zlib_faq.html#faq32
> Can zlib work with greater than 4 GB of data?
> Yes. inflate() and deflate() will process any amount of data
correctly. *** Each call of inflate() or deflate() is limited to input
and output chunks of the
> maximum value that can be stored in the compiler's "unsigned int"
type*** , but there is no limit to the number of chunks.
(my emphasis)
http://www.zlib.net/manual.html#Gzip :
ZEXTERN int ZEXPORT gzread OF((gzFile file, voidp buf, unsigned len
));
-----------------------------------------------------------^^^^^^^^^^^^^
Reads the given number of uncompressed bytes from the compressed file.
If the input file was not in gzip format, gzread copies the given number
of bytes into the buffer. gzread returns the number of uncompressed
bytes actually read (0 for end of file, -1 for error).
itkImageIOBase.h:
252 /** Type for representing size of bytes, and or positions along a
file */
253 typedef std::streamoff SizeType;
254
255 /** Type for representing size of bytes, and or positions along a
memory buffer */
256 typedef size_t BufferSizeType;
257
-----Original Message-----
From: insight-users-bounces at itk.org
[mailto:insight-users-bounces at itk.org] On Behalf Of
Robert.Atwood at diamond.ac.uk
Sent: 12 January 2010 17:56
To: bill.lorensen at gmail.com; adzyubak at gmail.com
Cc: insight-users at itk.org; luis.ibanez at kitware.com
Subject: Re: [Insight-users] Image index limits
Hi,
I updated my copy of ITK at about 15:30 today (Jan 12 2010) and
performed some tests.
Note: I get the following warning from Alex's program:
WARNING: In
/dls_test/i12/robert/sources/ITK_cvs/Insight/Code/IO/itkAnalyzeImageIO.c
xx, line 1307
AnalyzeImageIO (0x8d25790): ERROR: Analyze 7.5 File Format Only Allows
RPI, PIR, and RIP Orientation
However it does not stop the execution of the program.
Results:
1. Write cubes as .mhd, then run JustReadWrite using .mhd input and .mhd
output: correct
2. Write cubes as .mhd, then run JustReadWrite using .mhd input and .hdr
output: correct
3. Write cubes as .hdr, then run JustReadWrite using .hdr input and .hdr
output: INCORRECT
4. Write cubes as .hdr, then run JustReadWrite using .hdr input and .mhd
output: INCORRECT
Conclusion:
The problem lies in the reading of the .hdr format and is still present
in the current CVS version. Writing of .hdr files seems unaffected.
I suspect the problem lies here:
1
/*======================================================================
===
2
3 Program: Insight Segmentation & Registration Toolkit
4 Module: $RCSfile: itkAnalyzeImageIO.cxx,v $
5 Language: C++
6 Date: $Date: 2009-12-23 20:15:06 $
7 Version: $Revision: 1.102 $
....
681 void AnalyzeImageIO::Read(void* buffer)
682 {
683 unsigned int dim;
684 const unsigned int dimensions = this->GetNumberOfDimensions();
685 unsigned int numberOfPixels = 1;
//////<<<<<<<<<<<<<<<<<<<<<<<<<<<< UNSIGNED INT ????
686 for(dim=0; dim< dimensions; dim++ )
687 {
688 numberOfPixels *= m_Dimensions[ dim ];
689 }
I hope this helps,
Robert
-----Original Message-----
From: Bill Lorensen [mailto:bill.lorensen at gmail.com]
Sent: 12 January 2010 17:25
To: Oleksandr Dzyubak
Cc: Atwood, Robert (DLSLtd,RAL,DIA); insight-users at itk.org;
luis.ibanez at kitware.com
Subject: Re: [Insight-users] Image index limits
We have been making passes through the io's to add the proper types for
offsets. etc.
Try the cvs version and see if your test works with .mdh.
Bill
On Tue, Jan 12, 2010 at 11:42 AM, Oleksandr Dzyubak <adzyubak at gmail.com>
wrote:
>
> Hi Robert,
>
> Thanks for your help in studying this problem.
>
> Following your advise, I tested both
>
> ImageFill_Cube test.mhd
> and
> ImageFill_Cube test.hdr
>
> and in both images I see that
> the first cube stays where it should but the second one disappeared!
> The only reason for such different behavior on your and my machines I
> can think of is that on my server I am using ITK 3.16 released on
> September 15, 2009 vs your ITK CVS Nov 4, 2009.
>
> Does that mean that IOFactory was updated somehow between these two
> releases?
> Well, if it was, I have to give the CVS version a try.
>
> But anyway, according to your findings, this issue was not fixed in
> ImageFileReader for all formats though (or, in particular, for
> Analyze75 format), right?
>
> Unfortunately all my input images are in the Analyze75 format, thus
> any conversion would still require the Analyze75 support in
ImageFileReader.
>
> I see that we are getting close.
>
> Regards,
>
> Alex
>
> On Tue, Jan 12, 2010 at 6:06 AM, <Robert.Atwood at diamond.ac.uk> wrote:
>>
>> Hi Kevin:
>>
>> Which image format did you use for your tests? I observed (see my
>> other
>> post) that problem using .hdr format (as described by Oleksander's
>> original post) but not with .mhd format. I believe I observed a
>> similar problem before with .tif stack but not with a series of .tif
>> (or .mhd)
>>
>> Do you observe the problem if you use .hdr format?
>>
>> Thanks
>>
>> Robert
>>
>>
>> -----Original Message-----
>> From: insight-users-bounces at itk.org
>> [mailto:insight-users-bounces at itk.org] On Behalf Of Kevin H. Hobbs
>> Sent: 11 January 2010 15:01
>> To: adzyubak at gmail.com
>> Cc: insight-users at itk.org; Luis Ibanez
>> Subject: Re: [Insight-users] Image index limits
>>
>> On 01/07/2010 05:28 PM, Oleksandr Dzyubak wrote:
>> >
>> > ****** End JustReadWrite.cxx ***
>> >
>>
>> When I run this pair of programs I get two cubes in the output of the
>> second program.
>>
>> I am using ITK from CVS on a 64 bit Linux workstation with 12
>> gigabytes of RAM (bubbles.hooperlab on the dashboard).
>>
>> I tried running ImageCompare on the input and output images but the
>> machine started swapping and I cancelled the comparison.
>>
>>
>> --
>> This e-mail and any attachments may contain confidential, copyright
>> and or privileged material, and are for the use of the intended
>> addressee only. If you are not the intended addressee or an
>> authorised recipient of the addressee please notify us of receipt by
>> returning the e-mail and do not use, copy, retain, distribute or
>> disclose the information in or attached to the e-mail.
>> Any opinions expressed within this e-mail are those of the individual
>> and not necessarily of Diamond Light Source Ltd.
>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
>> attachments are free from viruses and we cannot accept liability for
>> any damage which you may sustain as a result of software viruses
>> which may be transmitted in or with the message.
>> Diamond Light Source Limited (company no. 4375679). Registered in
>> England and Wales with its registered office at Diamond House,
>> Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE,
>> United Kingdom
>>
>>
>>
>>
>
>
> _____________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Kitware offers ITK Training Courses, for more information visit:
> http://www.kitware.com/products/protraining.html
>
> Please keep messages on-topic and check the ITK FAQ at:
> http://www.itk.org/Wiki/ITK_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.itk.org/mailman/listinfo/insight-users
>
>
--
This e-mail and any attachments may contain confidential, copyright and
or privileged material, and are for the use of the intended addressee
only. If you are not the intended addressee or an authorised recipient
of the addressee please notify us of receipt by returning the e-mail and
do not use, copy, retain, distribute or disclose the information in or
attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual
and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any
attachments are free from viruses and we cannot accept liability for any
damage which you may sustain as a result of software viruses which may
be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in
England and Wales with its registered office at Diamond House, Harwell
Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United
Kingdom
_____________________________________
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Kitware offers ITK Training Courses, for more information visit:
http://www.kitware.com/products/protraining.html
Please keep messages on-topic and check the ITK FAQ at:
http://www.itk.org/Wiki/ITK_FAQ
Follow this link to subscribe/unsubscribe:
http://www.itk.org/mailman/listinfo/insight-users
--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
More information about the Insight-users
mailing list