[Insight-users] Changed behaviour using linux kernel 2.6.7 vs.
2.4.21
Atwood, Robert C
r.atwood at imperial.ac.uk
Tue Aug 10 10:40:13 EDT 2004
Umm. There's a couple of problems with this testing; atoi() does not
return a number larger than 2 Gb and is silent about returning numbers
not equal to the input string. The new[] does not return a null pointer
(0) if it fails to allocate, it seems to return random garbage , this
may be implementation dependent (not usable in cross-platform code),
bad_alloc should be detected using a try/catch block as Luis mentions.
(See Stroustrup, Sec. 6.2.6.2 , but you know more about try/catch than I
do!)
Also , the memory available for a single array is not the same as the
memory which may be allocated in several arrays, so this test may not be
the most useful test for this situaition. Unless you tell me otherwise,
I would not expect any single array allocated within ITK to be
especially larger than the array containing all the voxels? ie. about
the size of the raw data, 980 Mb in this case.
I don't see how there would be a mismatch between the Kernel and the
libraries, since I compiled the kernel using the libraries present on
the system. I am not sure how this relates, since I believe the kernel
code uses c rather than c++ . It is more likely that I did not select
some obscure kernel options that are necessary, or selected some options
that make the memory allocation behave in a way ITK does not expect
(there are such a variety of new incomprehensable options in 2.6).
I tried tests along these lines, using eg. the program below, without
finding any difference in the behaviour of these simle programs between
kernels. Not just for this purpose, I want to know the practical limits
of the memory use for other programs too. (I read that the address space
is 4Gb but you don't seem to actually be able to get that close to it in
practice, I cannot find the correct 'effing' manual to read having tried
the gcc manual, glibc manual, Stroustrup, man pages and google)
One single allocation seems to only return a maximum of about 2.1 Gb
(seems to be 2^31 minus some stack space) , multiple allocations seem
to be able to return up to about 3.1 Gb, also depending upon the stack
size and other space used by the program. There is no trouble in running
multiple instaces of the testmem, each having about 3 Gb. These limits
are about the same using c instead of c++ and using malloc() and seem to
be somewhat larger than the memory use reported by 'top' during
execution of my ITK based program.
Also it seems to make a difference whether the memory is used or not due
to the 'optimistic' allocation strategy accordign to glibc library
manual, the memory is not guaranteed to be available just because malloc
returns successfully ....
// c++ program to fill up the memory
#include <iostream>
#include <new>
#include <cstdlib>
int i=0;
void out_of_store(){
std::cout << "Failed on allocation # " << i << " " << std::endl;
throw std::bad_alloc();
}
int main(int argc, char *argv[]) {
char ** mem;
int j;
unsigned int num;
unsigned int nchunk;
int flag = 1;
/* to see if the memory gets exhausted when writing */
std::set_new_handler(out_of_store);
num = (unsigned int) ( atoll(argv[1])) ;
nchunk = (unsigned int) (atoll(argv[2]));
std::cout << "Num = " << num << std::endl;
/* see if the number is really the one input */
if ( (long long) num != atoll(argv[1]) ){
std::cout << "Range overflow! " << num << " " << atoll(argv[1])
<<
std::endl;
return(1);
}
try{
mem = new char * [nchunk];
}catch(...){
std::cout << " ***** Catch: Exception caught! " << std :: endl;
flag = 0;
}
if (flag){
for(i=0;i<nchunk;i++){
try{
mem[i] = new char[ num ];
}
catch(...)
{
std::cout << " ***** Catch: Exception caught! " << std ::
endl;
flag = 0;
}
if (i%1000 == 0){
#ifdef FILL
for (j=0;j<num;j++){
mem[i][j]='a';
}
#endif /* FILL */
std::cout << " ... " << i/1000 << std::endl;
}
if (flag == 0 ) exit(0);
}
}
/* test the memory if it was successfully allocated */
if( flag )
{
std::cout << "Don't worry, be happy" << std::endl;
}
else
{
std::cout << "You are asking too much from life" << std::endl;
}
return 0;
}
/***********************************************************************
/
/********** c program to gobble up the memory in chunks
****************/
/***********************************************************************
/
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
int main(int argc, char * argv[]){
char * * a;
unsigned int i,j;
unsigned int num=0;
unsigned int n=0;
unsigned int nalloc=1;
num = (unsigned int) ( atoll(argv[1])) ;
n = 0;
printf("%s num = %u\n",argv[1],num);
a = (char **) malloc(1 * sizeof(char *));
while (a != NULL){
printf("nalloc = %u\n",nalloc);
a[n] = (char *) malloc(num * sizeof(char));
if ( a[n] == NULL ) break;
#ifdef FILL
for (j=0;j<num;j++){
a[n][j] = 'a';
}
#endif
nalloc++;
n++;
a = (char **) realloc(a,nalloc * sizeof(char * ));
}
printf("final nalloc = %u\n",nalloc);
printf("arrays %lu\n",nalloc * num * sizeof(char));
printf("pointer %lu\n",nalloc * sizeof(char *));
printf("size: %lu\n",nalloc * num*sizeof(char) + nalloc *
sizeof(char *));
}
-----Original Message-----
From: Luis Ibanez [mailto:luis.ibanez at kitware.com]
Sent: 09 August 2004 23:47
To: Atwood, Robert C
Cc: Insight-users at itk.org
Subject: Re: [Insight-users] Changed behaviour using linux kernel 2.6.7
vs. 2.4.21
Hi Robert,
It is very easy to check whether this is an
ITK problem or a Kernel problem, or a compiler problem.
Chances are that this is a mismatch between your new Kernel
and the C++ libraries of your compiler.
Simply compile the following program
#include <iostream>
int main(int argc, char *argv[])
{
char * mem = new char[ atoi( argv[1] ) ];
if( mem )
{
std::cout << "Don't worry, be happy" << std::endl;
delete [] mem;
}
else
{
std::cout << "You are asking too much from life" << std::endl;
}
return 0;
}
run it by passing larger and larger numbers as arguments
until you find the allocation limitation of our new kernel
+ C++ libraries.
It should in fact throw and exception when it is not able
to allocate enough memory, so you probably want to put a try/catch block
around that "new" statement.
Please let us know how far you can get in memory allocation with this
program.
Thanks
Luis
PS:
Murphy's Laws state clearly that you should *never*
update your Linux Kernel unless you really need it.
-----------------------
Atwood, Robert C wrote:
> Dear itk/ linux-kernel users/developers:
>
> I tried upgrading the linux kernel on the machine, but then my ITK
> based application would throw an exception (unable to allocate memory
> for
> image) when updating a ShrinkImageFilter , shrink factor 16, original
> image 800x800x512 float -- this works with the 2.4.21 kernel and with
> the new kernel it does not work, either with the exact executable or a
> recompiled executable. I have tried recompiling , even recompiling itk
> (though I suppose that the kernel version should't affect the object
> code ? ) I selected Bigmem = 64 in the 2.6.7, but the 2.4.21 is the
> RedHat version 2.4.21-15.Elsmp, I am unsure what options it is
compiled
> with. However, it seems to recognize the 6gb installed memory.
>
>
> Unfortunately I have no idea whether this is an ITK problem or
> actually a linux kernel problem , but I thought people might like to
> know; if any guru has advice I could try some suggested tests to see
> what the problem is -- within reason, as the time spent rebooting
> repeatedly adds up.
>
> For now, I have just put the old kernel back. I have no urgent reason
> to upgrade.
>
> Hope this information is of some use,
>
> Robert
>
>
> _______________________________________________
> Insight-users mailing list
> Insight-users at itk.org
> http://www.itk.org/mailman/listinfo/insight-users
>
More information about the Insight-users
mailing list