[ITK Community] Windows TIFF size limit

[Gib Bogle]

Sorry about the drip-feeding of updates...

I found that replacing stat by _stat64 in the Open function enables the read to continue.  My program successfully compresses the big tiff.  The resulting file is not viewable by Irfanview, but I am able to uncompress it again.  It's possible that Irfanview cannot handle tiff files of this size.  I still need to figure out a way to check that all the files contain the correct data.

The stat() function does not appear anywhere else in itkTIFFImageIO.cxx, so checking for big tiff is needed only in Open().

Gib
________________________________
From: Community [community-bounces at itk.org] on behalf of Gib Bogle [g.bogle at auckland.ac.nz]
Sent: Friday, 7 March 2014 5:09 p.m.
To: Matt McCormick
Cc: community at itk.org
Subject: Re: [ITK Community] Windows TIFF size limit

The current failure when trying to read the big tiff file has me puzzled.

int TIFFReaderInternal::Open(const char *filename)
{
    printf("TIFFReaderInternal::Open\n");
  this->Clean();
  printf("did Clean!\n");
  printf("filename: %s\n",filename);
  struct stat fs;
  if ( stat(filename, &fs) )
    {
        printf("stat(filename,%fs) is not 0\n");
        return 0;
    }
...

(my printf statements, obviously.)  The call to stat() is returning a non-zero value, and the open fails here.  If I comment out the block, the read is successful, and my test of compressing the image again seems to be OK, but again the file generated cannot be opened (for the same reason, I presume).  My problem is I do not know about stat.  It looks as if the wrong info is written to the stat structure when the file is created.  I'm guessing that something funny might happen to st_size when the file is very big.

In fact Wikipedia says:


The C POSIX library<http://en.wikipedia.org/wiki/C_POSIX_library> header <sys/stat.h>, found on POSIX<http://en.wikipedia.org/wiki/POSIX> and other Unix-like<http://en.wikipedia.org/wiki/Unix-like> operating systems, declares the stat() functions, as well as related function called fstat() and lstat(). The functions take a struct stat buffer argument, which is used to return the file attributes. On success, the functions return zero, and on error, −1 is returned and errno<http://en.wikipedia.org/wiki/Errno> is set appropriately.

The stat() and lstat() functions take a filename<http://en.wikipedia.org/wiki/Filename> argument. If the file name is a symbolic link<http://en.wikipedia.org/wiki/Symbolic_link>, stat() returns attributes of the eventual target of the link, while lstat() returns attributes of the link itself. The fstat() function takes a file descriptor<http://en.wikipedia.org/wiki/File_descriptor> argument instead, and returns attributes of the file that it identifies.

The family of functions was extended to implement large file support<http://en.wikipedia.org/wiki/Large_file_support>. Functions named stat64(), lstat64() and fstat64() return attributes in a struct stat64 structure, which represents file sizes with a 64-bit type, allowing the functions to work on files 2 GiB and larger. When the _FILE_OFFSET_BITS macro<http://en.wikipedia.org/wiki/C_macro> is defined to 64, these 64-bit functions are available under the original names.

So it looks as if stat64() should be used for the big files, not stat().  I guess this requires detection of a big file, and probably implies changes in a few places.

Gib
________________________________
From: Matt McCormick [matt.mccormick at kitware.com]
Sent: Friday, 7 March 2014 4:44 p.m.
To: Gib Bogle
Cc: Brian Helba; community at itk.org
Subject: Re: [ITK Community] Windows TIFF size limit




On Thu, Mar 6, 2014 at 10:25 PM, Gib Bogle <g.bogle at auckland.ac.nz<mailto:g.bogle at auckland.ac.nz>> wrote:
Thanks.  I have tracked down the reason for the crash reading this compressed tiff in Windows.  The compression is DEFLATE (AKA ZIP).  The function TIFFReaderInternal::CanRead() in itkTIFFImageIO.cxx checks for COMPRESSION_NONE, COMPRESSION_PACKBITS and COMPRESSION_LZW, although COMPRESSION_DEFLATE is covered elsewhere in the code.  I added this line:
|| this->m_compression == COMPRESSION_DEFLATE
and now the read proceeds successfully, as far as I can tell, and my program creates an uncompressed 4.5GB file.   This is progress!

Well done!  Any patches [1] are greatly appreciated :-)


Unfortunately the file that is created does not identify itself as a valid TIFF.
Irfanview says: Decode error!  Invalid or unsupported TIF file.
When I try to read it with ITK, the message is:
File: ...itkImageFileReader.hxx
Line: 143
Description: Could not create IO object for file preeq_u.tif
Tried to create one of the following:
JPEGImageIO
...
TIFFImageIO
...
You probably failed to set a file suffix, or set the suffix to an unsupported type.

Well, the suffix is obviously correct, so there is another problem with the file.

The saga continues...

Keep up the good fight :-),
Matt


 [1]  http://insightsoftwareconsortium.github.io/ITKBarCamp-doc/CommunitySoftwareProcess/SubmitAPatchToGerrit/index.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/community/attachments/20140307/6eca9990/attachment-0002.html>