[Insight-users] Image offset is giving bad pointer for large datasets (7Gb)
Michael Jackson
mike.jackson at bluequartz.net
Fri Jul 10 12:46:49 EDT 2009
<rant>
I am not a C++ expert by any stretch but with my limited 4 years of
experience I have learned one thing. "Long" is the "root of all
evil" ;-)
IMNSHO, ITK (and VTK by extension) should absolutely BAN the use of
"long" in their projects. Period. Too many avoidable bugs come up from
its use. If you want a 32 bit integer use int or the standard ansi int
type, if you want a 64 bit integer then use "long long int" or the
standard ansi 64 bit integer type.
"long int" is just a mess waiting to happen. Maybe a rule should be
put into the KWStyle project to look for and flag the use of 'long'?
</rant>
A few years back I spent the better part of 2 days tracking down an
issue with VTK that turned out to be the same root problem. The use of
"long" when "long long" was really needed. Every time I see something
declared 'long' I then have to rethink the ramifications of using
those classes or variables as I flip-flop between 32 and 64 bit
compiles regularly.
Just my thoughts on this friday. Take 'em or leave 'em.
I vote for Luis Solution below. I _seems_ like it would solve the
problem ..
_________________________________________________________
Mike Jackson mike.jackson at bluequartz.net
On Jul 10, 2009, at 12:30 PM, Luis Ibanez wrote:
> Hi Kana,
>
> Thanks a lot for looking into this and sharing your findings.
>
> I just confirmed that in Windows 64bits, the "long" type is
> only 4 bytes.
>
> Here is the program I used:
>
> #include <iostream>
> #include "itkOffset.h"
> #include "itkNumericTraits.h"
>
> int main()
> {
> unsigned long tt;
> std::cout << "size = " << sizeof(tt) << std::endl;
> tt = -1;
> std::cout << "tt = " << tt << std::endl;
>
> typedef itk::Offset<3> OffsetType;
> typedef OffsetType::OffsetValueType OffsetValueType;
>
> OffsetValueType offsetValue;
>
> std::cout << "sizeof(offsetValue) = " << sizeof( offsetValue ) <<
> std::endl;
>
> offsetValue = itk::NumericTraits< OffsetValueType >::max();
>
> std::cout << "OffsetValueType max() = " << offsetValue << std::endl;
>
> return EXIT_SUCCESS;
> }
>
>
> with this CMakeLists.txt file
>
>
> CMAKE_MINIMUM_REQUIRED(VERSION 2.4)
> IF(COMMAND CMAKE_POLICY)
> CMAKE_POLICY(SET CMP0003 NEW)
> ENDIF(COMMAND CMAKE_POLICY)
>
>
> PROJECT(64bitsTest)
>
> FIND_PACKAGE(ITK REQUIRED)
> INCLUDE(${ITK_USE_FILE})
>
> ADD_EXECUTABLE(typesTest typesTest.cxx )
>
> TARGET_LINK_LIBRARIES(typesTest ITKCommon)
>
>
>
> ---------
>
>
> So, it seems that we have to introduce a new ITK type,
> that will be the longest int available in the platform.
>
>
> We currently have the following declarations in the file:
>
> Insight/Code/Common/itkIntTypes.h
>
>
> /** Convenient and more descriptive integer types. */
> typedef char ITK_INT8;
> typedef int ITK_INT32;
>
> #ifndef _WIN32
> typedef long long ITK_INT64;
> #endif
>
> #ifdef _WIN32
> typedef long ITK_INT64;
> #endif
>
> typedef unsigned char ITK_UINT8;
> typedef unsigned short ITK_UINT16;
> typedef unsigned ITK_UINT32;
>
> #ifndef _WIN32
> typedef unsigned long long ITK_UINT64;
> #endif
>
> #ifdef _WIN32
> typedef unsigned long ITK_UINT64;
> #endif
>
> typedef int ITK_INTPTR;
> typedef unsigned ITK_UINTPTR;
>
> #ifdef __cplusplus
> }
> #endif
>
>
>
> One option that comes to mind is that we
> should simply use
>
>
> size_t
> or std::size_t
>
> Which is the type expected by the "mem"
> methods, {memcpy,memdup...}, and the
> allocation "new" method.
>
> I have verified that "size_t" will have 8 bits
> on Windows 64.
>
> std::cout << "sizeof( size_t ) = " << sizeof( size_t ) << std::endl;
>
>
>
> Unless there are any objections, I'll suggest
> that we replace the type of all integer variables
> related to image offsets and image size, with
> the type "size_t".
>
>
> BTW: note that there is also "size_type"
>
> It seems that "size_type" will be the type used by STL
> containers.
>
>
> Any comments ?
>
>
> Thanks
>
>
> Luis
>
>
> ------------------------------------------------------------------------------------------------------------
> On Thu, Jul 9, 2009 at 2:53 AM, Arunachalam Kana <Kana.Arunachalam at fh-wels.at
> > wrote:
> Hi Luis,
>
> Thank you for your response. I ran the test program and the given the
> results are given below along with detailed
> System information and what option i used to compile itk.
>
> System Information
> ------------------
> Time of this report: 7/8/2009, 17:16:29
> Machine name: CT-DELL
> Operating System: Windows XP Professional x64 Edition (5.2, Build
> 3790) Service Pack 2 (3790.srv03_sp2_gdr.090319-1204)
> Language: English (Regional Setting: German)
> System Manufacturer: Dell Inc.
> System Model: Precision WorkStation T7400
> BIOS: Default System BIOS
> Processor: Intel(R) Pentium(R) III Xeon-Prozessor (8 CPUs),
> ~3.2GHz
> Memory: 65534MB RAM
> Page File: 717MB used, 65267MB available
> Windows Dir: C:\WINDOWS
> DirectX Version: DirectX 9.0c (4.09.0000.0904)
> DX Setup Parameters: Not found
> DxDiag Version: 5.03.3790.3959 32bit Unicode
>
>
> Itk compilation
> ---------------------
> Itk compiled using Microsoft visual studio 2009 x64 (option).
>
> Microsoft visual studio 2008
> ---------------------
> Configuration manager details:
> Active Solution Configuration: Debug
> Active Solution Platform: x64
>
> Test run details:
> 1. Program:
> unsigned long tt;
> std::cout << "size = " << sizeof(tt) << std::endl;
> tt = -1;
> std::cout << "tt = " << tt << std::endl;
> output: size = 4; tt = 4294967295
>
> 2. Program:
> unsigned long long tt;
> std::cout << "size = " << sizeof(tt) << std::endl;
> tt = -1;
> std::cout << "tt = " << tt << std::endl;
> output: size = 8; tt = 18446744073709551615
>
> I changed the configuration details:
> --------------------------------------
> Active Solution Configuration: Debug
> Active Solution Platform: Win32
>
> Test run details:
> 1. Program:
> unsigned long tt;
> std::cout << "size = " << sizeof(tt) << std::endl;
> tt = -1;
> std::cout << "tt = " << tt << std::endl;
> output: size = 4; tt = 4294967295
>
> 2. Program:
> unsigned long long tt;
> std::cout << "size = " << sizeof(tt) << std::endl;
> tt = -1;
> std::cout << "tt = " << tt << std::endl;
> output: size = 8; tt = 18446744073709551615
>
> For both Win32 and x64 i get the same result.
> Unsigned long is 4 byte and unsigned long long is 8 byte.
>
> I was a little confused after the result, so i searched for data type
> ranges in msdn. Link below:
>
> http://msdn.microsoft.com/en-us/library/s3f49ktz.aspx
>
> From the information given in above link, i understand that for 32 bit
> and 64 bit compiler unsigned long is 4 byte and unsigned long long
> is 8
> byte.
>
> I checked in the wiki for 64 bit machines (link below)
>
> http://en.wikipedia.org/wiki/64-bit
>
> In the link above, under topic "scientific data model": It is given
> that
> Microsoft win64 uses the LLP data model. From the above links, I
> understand the following:
>
> 1. A 64bit machine with Microsoft Windows x64 Operating system
> has
> 4 byte for long and 8 byte for long long.
> 2. Microsoft visual studio 2009 has long as 4 byte and long
> long as
> 8 bytes for both win32 and x64 compiler option.
>
> What's next?
>
> Thank you,
> Regards,
> Kana
>
> -----Original Message-----
> From: Luis Ibanez [mailto:luis.ibanez at kitware.com]
> Sent: 08 July 2009 16:10
> To: Arunachalam Kana
> Cc: insight-users at itk.org
> Subject: Re: [Insight-users] Image offset is giving bad pointer for
> large datasets (7Gb)
>
>
>
> Hi Arunachalam,
>
>
> Thanks for your detailed post.
>
>
> Let's discuss first ITK by itself,
>
>
> (before we go into the integration with VTK and VTKEdge)
>
>
>
> As you pointed out, the type
>
> OffsetValueType
>
> is currently defined as "long" in ITK.
>
>
>
> However, your assumptiong that "long" types in 64 bits
> platforms, are storing values up to 2^32 is incorrect.
>
>
> An unsigned long type in a 64bits platforms will have
> 8 bytes, and therefore can store values up to 2^64.
> which is:
>
> 18446744073709551615 L
>
>
> You can verify this with the following code:
>
>
> #include <iostream>
> int main()
> {
> unsigned long tt;
> std::cout << "size = " << sizeof(tt) << std::endl;
> tt = -1;
> std::cout << "tt = " << tt << std::endl;
> }
>
>
> The output of this program, in a 64bits machine will be:
>
>
> size = 8
> tt = 18446744073709551615
>
>
> The same code, when compiled in a 32-bits machine
> will produce as output:
>
>
> size = 4
> tt = 4294967295
>
>
> Therefore, if your OffsetValueType is getting saturated,
> it is likely that you are *not* compiling for 64 bits.
>
>
> Are you sure that when you configured ITK with CMake,
> you selected the "64 bits" version of your Visual Studio
> compiler.
>
>
> A simple way to double-check is to compile the code
> above and run it.
>
>
>
> Please do so, and let us konw what you find.
>
>
> Thanks
>
>
>
> Luis
>
>
>
> ----------------------------------
> Arunachalam Kana wrote:
> > Hi,
> >
> >
> >
> > Machine details: Windows XP, visual studio 2008, 64 bit platform,
> 64Gb Ram
> >
> > Image size: 1442 x 1566 x 1657 ( approx 7Gb )
> >
> > Other libraries: Along with ITK, VTK and VTK Edge is also used. VTK
> Edge
> > is used for changing vtk image to itk image and vice versa.
> >
> >
> >
> > Problem:
> >
> > In ITK the OffsetValueType is long. Range of long datatype:
> > -2,147,483,648 to 2,147,483,647
> >
> > But the image m_OffsetTable values of type OffsetValueType are : 1;
> > 1442; 2258172; 3,741,791,004
> >
> > In the above, it is clearly seen that the image offset value exceeds
> the
> > OffsetValueType range.
> >
> >
> >
> > Correction attempts:
> >
> > 1. I changed the OffsetValueType to unsigned long to increase
> the
> > range. Unsigned long range : 0 to 4,294,967,295. But
> >
> > this gives problems on conversion back to vtk. The vtkdataarray
> created
> > has a variable maxID which is of VtkIDType.
> >
> > VtkIDType is of long long. The maxID contains the value =
> -553,176,292.
> >
> >
> >
> > Correlation between 3,741,791,004 and -553,176,292.
> >
> > Decimal Hexa
> >
> > 3,741,791,003 DF07331B
> >
> > -553,176,292 FFFFFFFF DF07331B
> >
> >
> >
> > So, the program crashes due to bad pointer when it used vtkdata
> array
> > details to draw histogram.
> >
> > I think the probelm here is assigning of unsigned long to long
> long. I
>
> > don't want to change the
> >
> > Vtk data type to unsigned long, so in next attempt i change the itk
> > OffsetValueType to long long.
> >
> >
> >
> > 2. I changed the itk OffsetValueType to long long, but this
> leads
> > to change of lot of several other
> >
> > variable type in different file. Specially some of the basic image
> > container variables like TElementIdentifier, TElement have to be
> changed.
> >
> > For now I only can give these 2 variable, but i am afraid more
> variable
> > data types have to be changed in the future.
> >
> >
> >
> > I would be glad to have some help to solve this problem.
> >
> >
> >
> > Regards,
> >
> > Kana
> >
> >
> >
> ------------------------------------------------------------------------
> >
> > _____________________________________
> > Powered by www.kitware.com
> >
> > Visit other Kitware open-source projects at
> > http://www.kitware.com/opensource/opensource.html
> >
> > Please keep messages on-topic and check the ITK FAQ at:
> http://www.itk.org/Wiki/ITK_FAQ
> >
> > Follow this link to subscribe/unsubscribe:
> > http://www.itk.org/mailman/listinfo/insight-users
>
> _____________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ITK FAQ at: http://www.itk.org/Wiki/ITK_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.itk.org/mailman/listinfo/insight-users
More information about the Insight-users
mailing list