Proposals:Updating Nrrd library 2009

Nrrd Library

This page describes the procedure for synchronizing the NrrdIO parts of Teem, with the one in Insight/Utilities/NrrdIO

Note that patches may have to be moved in both directions.

Background

• Nrrd started as one of the libraries in Teem <http://teem.sf.net>
• NrrdIO is a subset of Teem containing only the IO functionalities (basically, parts of the air, biff, and nrrd libraries).
• The only part of Teem that's in ITK is in NrrdIO. Also, while Teem has a weakened LGPL license, NrrdIO is licensed identically to ITK.
• Creating NrrdIO from Teem is done by a GNUMakefile (NrrdIO/pre-GNUmakefile) and a perl script (NrrdIO/unteem.pl)
• The back-propagation that should happen is from ITK's NrrdIO back to Teem, because the idea is that NrrdIO should be periodically regenerated from Teem, and ITK's NrrdIO should be an up-to-date copy of the NrrdIO automatically regenerated from Teem
• You should anticipate that the process for updating and synchronizing the libraries will take several days.

Update Life Cycle

The "SF_" prefix refers to the SVN servers at SourceForge

Recipe for generating a fresh NrrdIO

• Checkout NrrdIO from SourceForge (this will only be used as a source of the files listed in YYY)
• Checkout Teem from SourceForge
• Create a new empty directory( that will contain the new generated NrrdIO)
• Copy files (YYY) from the NrrdIO SourceForge checkout into the newly created directory
• Run the 9 step script process (detailed in XXX)

Files that must be taken from SourceForge NrrdIO (and not from Teem)

The official source of these files is the NrrdIO project in SourceForge.

First, get the scripts

• pre-GNUMakefile
• unteem.pl

add to them the list of files can be seen in the pre-GNUMakefile under the variables "NEED_NOT" and "TEEM_HDRS"

• NEED_NOT
  • encodingBzip2.c
  • formatEPS.c
  • formatPNG.c
  • formatPNM.c
  • formatText.c
  • formatVTK.c

• TEEM_HDRS
  • teem32bit.h
  • teemDio.h
  • teemEndian.h
  • teemPng.h
  • teemQnanhibit.h

and add the Licensing Header

• preamble.c

and the testing files

• sample-GNUmakefile
• sampleIO.c

Update Procedure

Perform the following two tasks:

From ITK to Teem

For all the changes made to

             Insight/Utilities/NrrdIO

After the last Nrrd batch commit (from G Kindlmann):

1. Understand what the change was
2. Apply the same change (or something equivalent) to the corresponding file in Teem (not to the NrrdIO SVN checkout)
3. Verify the Teem Dashboard, to make sure that it is green.
4. Regenerate NrrdIO from Teem

Clarification

• For the .c files this should be easy; these are copied from a Teem checkout, but some non-NrrdIO-related sections are excised by unteem.pl
• For the headers some processing is needed, since they are mangled a bit.
  • For NrrdIO.h; this is actually a concatenation of air/air.h, biff/biff.h, nrrd/nrrdDefines.h, nrrd/nrrdEnums.h, nrrd/nrrdMacros.h, and nrrd/nrrd.h (these are defined in the NEED_PUB_HDRS variable of NrrdIO/pre-GNUmakefile)

Checking out Teem and NrrdIO from sourceforge

You can check-out Teem by using the SVN command:

  svn co http://teem.svn.sourceforge.net/svnroot/teem/teem/trunk teem

You will later set the TEEM_SRC_ROOT environment variable to wherever this check out is. The important synchronization that has to happen is between Sourceforge's Teem and ITK's NrrdIO.

Use the following SVN command to checkout NrrdIO:

  svn co http://teem.svn.sourceforge.net/svnroot/teem/NrrdIO/trunk NrrdIO

This is Sourceforge's NrrdIO, which has fallen out of sync with ITK's NrrdIO. After the fixes that have accumulated in ITK's NrrdIO have been back-propagated to Sourceforge's Teem, Gordon (or others) can do whatever is needed to get them back into NrrdIO.

Clarification

The way that Sourceforge names things makes this a little confusing. The sourceforge "project name" Teem includes multiple directories, or multiple separate SVN repositories. One directory, called Teem, this is the main source tree, where all the NrrdIO sources are copied from. There is also a NrrdIO directory.

Code Generation

Most of the source files in Insight/Utilities/NrrdIO directory are created automatically from the Teem directory; the rest are more like "stub" files to be place-holders for functionality that is in Teem but has no purpose in the stand-alone Nrrd read/write library that ITK needs. This is described in the comments in pre-GNUmakefile, reproduced here. To make a NrrdIO for ITK, which includes a header for name-mangling all the symbols, the steps are ("make" needs to be a GNU make) as follows. This assumes csh; environment variable commands will be different for bash.

These commands are to be executed in Insight/Utilities/NrrdIO. The "make -f pre-GNUmakefile" command requires that you have set the TEEM_SRC_ROOT environment variable to the directory of a Teem SVN checkout, no trailing "/" required; this directory should include the "src" and "include" directories, as well as files like "CMakeLists.txt" and "README.txt".

1. rm -f itk_NrrdIO_mangle.h
2. touch itk_NrrdIO_mangle.h
3. make -f pre-GNUmakefile clean
4. setenv ITK_NRRDIO
5. make -f pre-GNUmakefile
6. unsetenv ITK_NRRDIO
7. make -f sample-GNUmakefile
8. perl mangle.pl itk >! itk_NrrdIO_mangle.h
9. make -f sample-GNUmakefile clean

What pre-GNUmakefile does

The "pre-GNUmakefile" makefile

• Copies files from a Teem checkout
• Modifies them slightly (through the action of unteem.pl). Specifically, it excises lines of code that are delimited by "/* ---- BEGIN non-NrrdIO */" and "/* ---- END non-NrrdIO */"; this is code that Teem needs but NrrdIO does not.

What sample-GNUmakefile does

Then "sample-GNUmakefile"

• Builds a NrrdIO library
• Builds a small test executable

What mangle.pl does

The script "mangle.pl"

• Looks at all the symbols in the library in order to generate the name-mangling header file required for ITK.

Extra Information (Not part of the process)

If we weren't making NrrdIO for ITK, the name mangling wouldn't be required, and the process would just be:

• make -f pre-GNUmakefile clean
• make -f pre-GNUmakefile
• make -f sample-GNUmakefile
• make -f sample-GNUmakefile clean

where we run make with "sample-GNUmakefile" just to make sure that everything compiles ok.

From Nrrd to ITK

1. Make a working copy of NrrdIO by running the pre-GNUmakefile to build a NrrdIO from Teem
2. Gradually or all at once, depending on our bravery, we copy files from this NrrdIO to Insight/Utilities/NrrdIO
3. Run an Experimental ITK build and see what breaks.
   1. Problems are likely related to updates to Teem itself (including the components of NrrdIO) since 2005
   2. but might also be related to errors in how the updates to ITK's NrrdIO were back-propagated to Teem.

Progress

Table

The table below lists all the files in NrrdIO which are generated from a Teem checkout via NrrdIO/pre-GNUmakefile; these are marked with "*", and the originating directory in Teem is also indicated (the air, biff, and nrrd directories inside teem/src). Changes in these files in ITK's NrrdIO since GLK's last commits in 2005 should be back-propagated to the corresponding files in Teem, so that when NrrdIO is next regenerated from Teem, none of the improvements to ITK's NrrdIO are lost. Handling updates to these files is the hard/interesting part of this project.

The files not marked with a "*" are not generated from Teem via NrrdIO/pre-GNUmakefile; they exist only within NrrdIO. GLK doesn't see why there would have been any changes to these in files in ITK's NrrdIO, because it wouldn't effect how ITK uses Nrrd, but if there are changes they should also be reviewed and possibly back-propagated to Sourceforge's NrrdIO.

File	Check ITK for Bug Fix	Move from NrrdIO to ITK
000-README.txt	Not present in current Sourceforge Teem.	Keep?
* air/754.c	Change of ITK rev 1.8 present in NrrrdIO (l. 132); NrrdIO version updated (l. 358: mant0 >> 19). Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/accessors.c	ITK version 4 years old; NrrdIO version updated (static functions). Keep NrrdIO version.	Moved to ITK. 1/27/10
* air/array.c	ITK version 4 years old; NrrdIO version updated (AIR_CALLOC, check on a->data, ...). Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/arraysNrrd.c	ITK version 4 years old; NrrdIO version updated (AIR_CALLOC, check on a->data, ...). Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/axis.c	Casts from size_t to unsigned int, as in rev 1.13, no longer needed: everything is size_t now. Keep NrrdIO version.	Moved to ITK. 1/27/10
* biff/biffbiff.c	Casts from size_t to int, as in revs 1.8, 1.9, 1.10, no longer needed. Code, e.g biffAddErr() , superceded. Keep NrrdIO version.	Moved to ITK. Committed biff.h as well. 1/27/10
* biff/biffmsg.c	Does not exist in ITK version. Keep NrrdIO version.	Added and committed on 1/27/10.
CMakeLists.txt		Updated to account for new source files and reconcile flags. 1/27/10
* nrrd/comment.c	ITK version 4 years old; not much difference. Keep NrrdIO version.	Moved to ITK. 1/28/10
* nrrd/defaultsNrrd.c	ITK version 4 years old; not much difference. Keep NrrdIO version.	Moved to ITK. 1/27/10
* air/dio.c	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/encodingAscii.c	Casts from size_t to unsigned int, as in rev 1.6, no longer needed. Code, e.g sprintf()/biffAdd(), superceded. Keep NrrdIO version.	Moved to ITK. 1/28/10
* nrrd/encodingBzip2.c	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 1/28/10
* nrrd/encoding.c	ITK version 4 years old; NrrdIO version updated(static consts, biffAddf). Keep NrrdIO version.	Moved to ITK. 1/28/10
* nrrd/encodingGzip.c	Casts from size_t to unsigned int, as in rev 1.8, no longer needed. Code, e.g. _nrrdGzRead(),superceded. Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/encodingHex.c	Casts as in rev 1.8 in ITK now handled by AIR_CAST() in NrrdIO, + other NrrdIO version updates. Keep NrrdIO version.	Moved to ITK. 1/28/10
* nrrd/encodingRaw.c	ITK version 4 years old; NrrdIO version updated (temporary bug in gcc 4.2). Keep NrrdIO version (for now).	Moved to ITK. 1/28/10
* air/endianAir.c	ITK version 4 years old; NrrdIO version updated (const correctness). Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/endianNrrd.c	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 1/28/10
* air/enum.c	ITK version 4 years old; NrrdIO version updated (const correctness). Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/enumsNrrd.c	ITK version 4 years old; NrrdIO version updated (const correctness). Keep NrrdIO version.	Non-NrrdIO taken out. Moved to ITK. 1/27/10
* nrrd/format.c	ITK version 4 years old; NrrdIO version updated (const correctness, biffAddf). Keep NrrdIO version.	Moved to ITK. 1/28/10
* nrrd/formatEPS.c	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 1/28/10
* nrrd/formatNRRD.c	ITK version 4 years old; NrrdIO version updated (const correctness, biffAddf). Keep NrrdIO version.	Moved to ITK. 1/28/10
* nrrd/formatPNG.c	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 2/08/10
* nrrd/formatPNM.c	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 2/08/10
* nrrd/formatText.c	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 2/08/10
* nrrd/formatVTK.c	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 2/19/10 along with randMT.c.
* nrrd/gzio.c	Changes in ITK also found in NrrdIO, + latter version updated (const correctness, biffAddf). Keep NrrdIO version.	Moved to ITK. 1/28/10
* nrrd/keyvalue.c	ITK version 4 years old; NrrdIO version updated (parameters added _nrrdWriteEscaped(), _nrrdKeyValueWrite()). Keep NrrdIO version.	Moved to ITK. 1/28/10
mangle.pl	Not present in current Sourceforge Teem.	Keep?
* nrrd/methodsNrrd.c	Most changes in ITK last 4 years superceded by NrrdIO version (e.g. biffAddf() ). Rollback in rev 1.10 mysterious. Keep NrrdIO version (for now).	Moved to ITK. 1/27/10
* air/miscAir.c	ITK version 4 years old; minor changes (float to double, ...) in NrrdIO version. Keep NrrdIO version.	Non-NrrdIO taken out. Moved to ITK. 1/27/10
* air/mop.c	ITK version 4 years old; minor change (airMopAdd now returns int) in NrrdIO version. Keep NrrdIO version.	Moved to ITK. 1/27/10
* NrrdIO.h	Latest ITK changes, plus recent changes (#ifdef __APPLE__ l 696) seem to appear in NrrdIO version. Keep NrrdIO version.	Use NrrdIO version, and modify it to account for #include "itk_NrrdIO_mangle.h" and TEEM_API as in ITK. Updated in ITK. 1/28/10
NrrdIO_Srcs.txt	Not found in NrrdIO repository.	Not applicable.
* air/parseAir.c	ITK version 4 years old; recent changes (const correctness, AIR_CAST...) in NrrdIO version. Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/parseNrrd.c	ITK casts to unsigned int, rev 1.11, superceded in NrrdIO by dealing with size_t + other changes. Rollback in rev 1.10 mysterious. Keep NrrdIO version (for now).	Moved to ITK. 1/27/10
preamble.c	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 2/08/10
pre-GNUmakefile	ITK version 4 years old; minor changes (addprefix l 83, ...) in NrrdIO version. Keep NrrdIO version, but ...	... ITK currently uses cvs not svn.
* air/privateAir.h	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/privateNrrd.h	Rollback in rev 1.12 mysterious. Keep NrrdIO version (for now).	ITK version should use #include "itk_zlib.h", as currently in ITK. Moved to ITK. 1/28/10
qnanhibit.c	Not present in current Sourceforge Teem.	Keep?
* nrrd/read.c	Rollback in rev 1.9 mysterious. Keep NrrdIO version (for now). Add logic for separators in 1.11.	Moved to ITK. 1/27/10
* nrrd/reorder.c	Cast in rev. 1.10 missing (via AIR_CAST), look at pragma (array bound warning) in rev 1.11.	Moved to ITK. 1/27/10
sample-GNUmakefile	Not present in current Sourceforge Teem.	Keep?
sampleIO.c	Not present in current Sourceforge Teem.	Keep?
* air/sane.c	ITK version 4 years old; recent changes in NrrdIO version (airInsane_AIR_NAN). Keep NrrdIO version.	Causes failing tests in cross-compiled Mac.
* nrrd/simple.c	ITK casts to int, rev 1.13, superceded in NrrdIO by dealing with size_t + other changes (const). Keep NrrdIO version.	Moved to ITK. 1/27/10
* air/string.c	ITK casts either superceded in NrrdIO by dealing with size_t or handled by AIR_CAST. Keep NrrdIO version.	Moved to ITK. 1/27/10
* nrrd/subset.c	Rollback in rev 1.7 mysterious. Recent changes in NrrdIO version. Start with NrrdIO version (for now), but look at pragma (array bound warning) in rev 1.8.	Moved to ITK. 1/28/10
* include/teem32bit.h	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 2/08/10
* include/teemDio.h	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 2/08/10
* include/teemEndian.h	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 2/08/10
* include/teemPng.h	ITK version 4 years old; NrrdIO has "vtk_png_mangle.h". Keep NrrdIO version.	Moved to ITK. 2/08/10
* include/teemQnanhibit.h	ITK version 4 years old; no difference except copyright. Keep NrrdIO version.	Moved to ITK. 2/08/10
unteem.pl	Not present in current Sourceforge Teem.	Keep?
* nrrd/write.c	Reconciliation of NrrdIO & ITK versions req'd. !!airStrlen vs (int)airStrlen; rollback in rev 1.10 mysterious; airSinglePrintf(NULL, buff, "%.17lg", val[dd])-type fixes as in ITK rev 1.12.	Moved to ITK. 1/27/10

Reference Material

NrrdIO / Teem Dashboard

• http://my.cdash.org/index.php?project=Teem