The purpose of this chapter is to provide you with an overview of the Insight Toolkit system. We recommend that you read this chapter to gain an appreciation for the breadth and area of application of ITK.
The Insight Toolkit consists of several subsystems. A brief description of these subsystems follows. Later sections in this chapter—and in some cases additional chapters—cover these concepts in more detail.
This section describes some of the core concepts and implementation features found in ITK.
Generic programming is a method of organizing libraries consisting of generic—or reusable—software components [?]. The idea is to make software that is capable of “plugging together” in an efficient, adaptable manner. The essential ideas of generic programming are containers to hold data, iterators to access the data, and generic algorithms that use containers and iterators to create efficient, fundamental algorithms such as sorting. Generic programming is implemented in C++ with the template programming mechanism and the use of the STL Standard Template Library [?].
C++ templating is a programming technique allowing users to write software in terms of one or more unknown types T. To create executable code, the user of the software must specify all types T (known as template instantiation) and successfully process the code with the compiler. The T may be a native type such as float or int, or T may be a user-defined type (e.g., a class). At compile-time, the compiler makes sure that the templated types are compatible with the instantiated code and that the types are supported by the necessary methods and operators.
ITK uses the techniques of generic programming in its implementation. The advantage of this approach is that an almost unlimited variety of data types are supported simply by defining the appropriate template types. For example, in ITK it is possible to create images consisting of almost any type of pixel. In addition, the type resolution is performed at compile time, so the compiler can optimize the code to deliver maximal performance. The disadvantage of generic programming is that the analysis performed at compile time increases the time to build an application. Also, the increased complexity may produce difficult to decipher error messages due to even the simplest syntax errors. For those unfamiliar with templated code and generic programming, we recommend the two books cited above.
In ITK, classes are defined by a maximum of two files: a header file (.h) and an implementation file (.cxx) if defining a non-templated class, and a .hxx file if defining a templated class. The header files contain class declarations and formatted comments that are used by the Doxygen documentation system to automatically produce HTML manual pages.
In addition to class headers, there are a few other important header files.
Most classes in ITK are instantiated through an object factory mechanism. That is, rather than using the standard C++ class constructor and destructor, instances of an ITK class are created with the static class New() method. In fact, the constructor and destructor are protected: so it is generally not possible to construct an ITK instance on the stack. (Note: this behavior pertains to classes that are derived from itk::LightObject. In some cases the need for speed or reduced memory footprint dictates that a class is not derived from LightObject. In this case instances may be created on the stack. An example of such a class is the itk::EventObject.)
The object factory enables users to control run-time instantiation of classes by registering one or more factories with itk::ObjectFactoryBase. These registered factories support the method CreateInstance(classname) which takes as input the name of a class to create. The factory can choose to create the class based on a number of factors including the computer system configuration and environment variables. For example, a particular application may wish to deploy its own class implemented using specialized image processing hardware (i.e., to realize a performance gain). By using the object factory mechanism, it is possible to replace the creation of a particular ITK filter at run-time with such a custom class. (Of course, the class must provide the exact same API as the one it is replacing.). For this, the user compiles his class (using the same compiler, build options, etc.) and inserts the object code into a shared library or DLL. The library is then placed in a directory referred to by the ITK_AUTOLOAD_PATH environment variable. On instantiation, the object factory will locate the library, determine that it can create a class of a particular name with the factory, and use the factory to create the instance. (Note: if the CreateInstance() method cannot find a factory that can create the named class, then the instantiation of the class falls back to the usual constructor.)
In practice, object factories are used mainly (and generally transparently) by the ITK input/output (IO) classes. For most users the greatest impact is on the use of the New() method to create a class. Generally the New() method is declared and implemented via the macro itkNewMacro() found in Modules/Core/Common/include/itkMacro.h.
By their nature, object-oriented systems represent and operate on data through a variety of object types, or classes. When a particular class is instantiated, memory allocation occurs so that the instance can store data attribute values and method pointers (i.e., the vtable). This object may then be referenced by other classes or data structures during normal operation of the program. Typically, during program execution, all references to the instance may disappear at which point the instance must be deleted to recover memory resources. Knowing when to delete an instance, however, is difficult. Deleting the instance too soon results in program crashes; deleting it too late causes memory leaks (or excessive memory consumption). This process of allocating and releasing memory is known as memory management.
In ITK, memory management is implemented through reference counting. This compares to another popular approach—garbage collection—used by many systems, including Java. In reference counting, a count of the number of references to each instance is kept. When the reference goes to zero, the object destroys itself. In garbage collection, a background process sweeps the system identifying instances no longer referenced in the system and deletes them. The problem with garbage collection is that the actual point in time at which memory is deleted is variable. This is unacceptable when an object size may be gigantic (think of a large 3D volume gigabytes in size). Reference counting deletes memory immediately (once all references to an object disappear).
Reference counting is implemented through a Register()/Delete() member function interface. All instances of an ITK object have a Register() method invoked on them by any other object that references them. The Register() method increments the instances’ reference count. When the reference to the instance disappears, a Delete() method is invoked on the instance that decrements the reference count—this is equivalent to an UnRegister() method. When the reference count returns to zero, the instance is destroyed.
This protocol is greatly simplified by using a helper class called a itk::SmartPointer. The smart pointer acts like a regular pointer (e.g. supports operators -> and ⋆) but automagically performs a Register() when referring to an instance, and an UnRegister() when it no longer points to the instance. Unlike most other instances in ITK, SmartPointers can be allocated on the program stack, and are automatically deleted when the scope that the SmartPointer was created in is closed. As a result, you should rarely if ever call Register() or Delete() in ITK. For example:
In this example, reference counted objects are created (with the New() method) with a reference count of one. Assignment to the SmartPointer interp does not change the reference count. At the end of scope, interp is destroyed, the reference count of the actual interpolator object (referred to by interp) is decremented, and if it reaches zero, then the interpolator is also destroyed.
Note that in ITK SmartPointers are always used to refer to instances of classes derived from itk::LightObject. Method invocations and function calls often return “real” pointers to instances, but they are immediately assigned to a SmartPointer. Raw pointers are used for non-LightObject classes when the need for speed and/or memory demands a smaller, faster class. Raw pointers are preferred for multi-threaded sections of code.
In general, ITK uses exception handling to manage errors during program execution. Exception handling is a standard part of the C++ language and generally takes the form as illustrated below:
A particular class may throw an exception as demonstrated below (this code snippet is taken from itk::ByteSwapper:
Note that itk::ByteSwapperError is a subclass of itk::ExceptionObject. In fact, all ITK exceptions derive from ExceptionObject. In this example a special constructor and C++ preprocessor variables __FILE__ and __LINE__ are used to instantiate the exception object and provide additional information to the user. You can choose to catch a particular exception and hence a specific ITK error, or you can trap any ITK exception by catching ExceptionObject.
Event handling in ITK is implemented using the Subject/Observer design pattern [?] (sometimes referred to as the Command/Observer design pattern). In this approach, objects indicate that they are watching for a particular event—invoked by a particular instance—by registering with the instance that they are watching. For example, filters in ITK periodically invoke the itk::ProgressEvent. Objects that have registered their interest in this event are notified when the event occurs. The notification occurs via an invocation of a command (i.e., function callback, method invocation, etc.) that is specified during the registration process. (Note that events in ITK are subclasses of EventObject; look in itkEventObject.h to determine which events are available.)
To recap using an example: various objects in ITK will invoke specific events as they execute (from ProcessObject):
To watch for such an event, registration is required that associates a command (e.g., callback function) with the event: Object::AddObserver() method:
When the event occurs, all registered observers are notified via invocation of the associated Command::Execute() method. Note that several subclasses of Command are available supporting const and non-const member functions as well as C-style functions. (Look in Modules/Core/Common/include/itkCommand.h to find pre-defined subclasses of Command. If nothing suitable is found, derivation is another possibility.)
Multi-threading is handled in ITK through a high-level design abstraction. This approach provides portable multi-threading and hides the complexity of differing thread implementations on the many systems supported by ITK. For example, the class itk::PlatformMultiThreader provides support for multi-threaded execution by directly using platform-specific primitives such as pthread_create. itk::TBBMultiThreader uses Intel’s Thread Building Blocks cross-platform library, which can do dynamic workload balancing across multiple processes. This means that outputRegionForThread might have different sizes which change over time, depending on overall processor load. All multi-threader implementations derive from itk::MultiThreaderBase.
Multi-threading is typically employed by an algorithm during its execution phase. For example, in the class itk::ImageSource (a superclass for most image processing filters) the GenerateData() method uses the following methods:
In this example each thread invokes DynamicThreadedGenerateData method of the derived filter. The ParallelizeImageRegion method takes care to divide the image into different regions that do not overlap for write operations. ImageSource’s GenerateData() passes this pointer to ParallelizeImageRegion, which allows ParallelizeImageRegion to update the filter’s progress after each region has been processed.
If a filter has some serial part in the middle, in addition to initialization done in BeforeThreadedGenerateData() and finalization done in AfterThreadedGenerateData(), it can parallelize more than one method in its own version of GenerateData(), such as done by itk::CannyEdgeDetectionImageFilter:
{
this->UpdateProgress(0.0f);
Superclass::AllocateOutputs();
// Small serial section
this->UpdateProgress(0.01f);
ProgressTransformer progress1( 0.01f, 0.45f, this );
// Calculate 2nd order directional derivative
this->GetMultiThreader()->template ParallelizeImageRegion<TOutputImage::ImageDimension>(
this->GetOutput()->GetRequestedRegion(),
[this](const OutputImageRegionType & outputRegionForThread)
{ this->ThreadedCompute2ndDerivative(outputRegionForThread); },
progress1.GetProcessObject());
ProgressTransformer progress2( 0.45f, 0.9f, this );
// Calculate the gradient of the second derivative
this->GetMultiThreader()->template ParallelizeImageRegion<TOutputImage::ImageDimension>(
this->GetOutput()->GetRequestedRegion(),
[this](const OutputImageRegionType & outputRegionForThread)
{ this->ThreadedCompute2ndDerivativePos(outputRegionForThread); },
progress2.GetProcessObject());
// More processing
this->UpdateProgress(1.0f);
}
When invoking ParallelizeImageRegion multiple times from GenerateData(), either nullptr or a itk::ProgressTransformer object should be passed instead of this, otherwise progress will go from 0% to 100% more than once. And this will at least confuse any other class watching the filter’s progress events, even if it does not cause a crash. So the filter’s author should estimate how long each part of GenerateData() takes, and construct and pass ProgressTransformer objects as in the example above.
With ITK version 5.0, the Multi-Threading mechanism has been refactored. What was previously itk::MultiThreader, is now a hierarchy of classes. itk::PlatformMultiThreader is a slightly cleaned-up version of the old class - MultipleMethodExecute and SpawnThread methods have been deprecated. But much of its content has been moved to itk::MultiThreaderBase. And classes should use the multi-threaders via MultiThreaderBase interface, to allow the end user the flexibility to select the multi-threader at run time. This also allows the filter to benefit from future improvements in threading such as addition of a new multi-threader implementation.
The backwards compatible ThreadedGenerateData(Region, ThreadId) method signature has been kept, for use in filters that must know their thread number. To use this signature, a filter must invoke this->DynamicMultiThreadingOff(); before Update(); is called by the filter’s user or downstream filter in the pipeline. The best place for invoking this->DynamicMultiThreadingOff(); is the filter’s constructor.
In image filters and other descendants of ProcessObject, method SetNumberOfWorkUnits controls the level of parallelism. Load balancing is possible when NumberOfWorkUnits is greater than the number of threads. In most places where developer would like to restrict number of threads, work units should be changed instead. itk::MultiThreaderBase’s MaximumNumberOfThreads should not generally be changed, except when testing performance and scalability, profiling and sometimes debugging code.
The general philosophy in ITK regarding thread safety is that accessing different instances of a class (and its methods) is a thread-safe operation. Invoking methods on the same instance in different threads is to be avoided.
ITK uses the VNL numerics library to provide resources for numerical programming combining the ease of use of packages like Mathematica and Matlab with the speed of C and the elegance of C++. It provides a C++ interface to the high-quality Fortran routines made available in the public domain by numerical analysis researchers. ITK extends the functionality of VNL by including interface classes between VNL and ITK proper.
The VNL numerics library includes classes for:
Most VNL routines are implemented as wrappers around the high-quality Fortran routines that have been developed by the numerical analysis community over the last forty years and placed in the public domain. The central repository for these programs is the “netlib” server.4 The National Institute of Standards and Technology (NIST) provides an excellent search interface to this repository in its Guide to Available Mathematical Software (GAMS),5 both as a decision tree and a text search.
ITK also provides additional numerics functionality. A suite of optimizers, that use VNL under the hood and integrate with the registration framework are available. A large collection of statistics functions—not available from VNL—are also provided in the Insight/Numerics/Statistics directory. In addition, a complete finite element (FEM) package is available, primarily to support the deformable registration in ITK.
There are two principle types of data represented in ITK: images and meshes. This functionality is implemented in the classes itk::Image and itk::Mesh, both of which are subclasses of itk::DataObject. In ITK, data objects are classes that are meant to be passed around the system and may participate in data flow pipelines (see Section 3.5 on page 65 for more information).
itk::Image represents an n-dimensional, regular sampling of data. The sampling direction is parallel to direction matrix axes, and the origin of the sampling, inter-pixel spacing, and the number of samples in each direction (i.e., image dimension) can be specified. The sample, or pixel, type in ITK is arbitrary—a template parameter TPixel specifies the type upon template instantiation. (The dimensionality of the image must also be specified when the image class is instantiated.) The key is that the pixel type must support certain operations (for example, addition or difference) if the code is to compile in all cases (for example, to be processed by a particular filter that uses these operations). In practice, most applications will use a C++ primitive type (e.g., int, float) or a pre-defined pixel type and will rarely create a new type of pixel class.
One of the important ITK concepts regarding images is that rectangular, continuous pieces of the image are known as regions. Regions are used to specify which part of an image to process, for example in multi-threading, or which part to hold in memory. In ITK there are three common types of regions:
The itk::Mesh class represents an n-dimensional, unstructured grid. The topology of the mesh is represented by a set of cells defined by a type and connectivity list; the connectivity list in turn refers to points. The geometry of the mesh is defined by the n-dimensional points in combination with associated cell interpolation functions. Mesh is designed as an adaptive representational structure that changes depending on the operations performed on it. At a minimum, points and cells are required in order to represent a mesh; but it is possible to add additional topological information. For example, links from the points to the cells that use each point can be added; this provides implicit neighborhood information assuming the implied topology is the desired one. It is also possible to specify boundary cells explicitly, to indicate different connectivity from the implied neighborhood relationships, or to store information on the boundaries of cells.
The mesh is defined in terms of three template parameters: 1) a pixel type associated with the points, cells, and cell boundaries; 2) the dimension of the points (which in turn limits the maximum dimension of the cells); and 3) a “mesh traits” template parameter that specifies the types of the containers and identifiers used to access the points, cells, and/or boundaries. By using the mesh traits carefully, it is possible to create meshes better suited for editing, or those better suited for “read-only” operations, allowing a trade-off between representation flexibility, memory, and speed.
Mesh is a subclass of itk::PointSet. The PointSet class can be used to represent point clouds or randomly distributed landmarks, etc. The PointSet class has no associated topology.
While data objects (e.g., images and meshes) are used to represent data, process objects are classes that operate on data objects and may produce new data objects. Process objects are classed as sources, filter objects, or mappers. Sources (such as readers) produce data, filter objects take in data and process it to produce new data, and mappers accept data for output either to a file or some other system. Sometimes the term filter is used broadly to refer to all three types.
The data processing pipeline ties together data objects (e.g., images and meshes) and process objects. The pipeline supports an automatic updating mechanism that causes a filter to execute if and only if its input or its internal state changes. Further, the data pipeline supports streaming, the ability to automatically break data into smaller pieces, process the pieces one by one, and reassemble the processed data into a final result.
Typically data objects and process objects are connected together using the SetInput() and GetOutput() methods as follows:
itk::RandomImageSource<FloatImage2DType>::Pointer random;
random = itk::RandomImageSource<FloatImage2DType>::New();
random->SetMin(0.0);
random->SetMax(1.0);
itk::ShrinkImageFilter<FloatImage2DType,FloatImage2DType>::Pointer shrink;
shrink = itk::ShrinkImageFilter<FloatImage2DType,FloatImage2DType>::New();
shrink->SetInput(random->GetOutput());
shrink->SetShrinkFactors(2);
itk::ImageFileWriter<FloatImage2DType>::Pointer writer;
writer = itk::ImageFileWriter<FloatImage2DType>::New();
writer->SetInput (shrink->GetOutput());
writer->SetFileName( "test.raw" );
writer->Update();
In this example the source object itk::RandomImageSource is connected to the itk::ShrinkImageFilter, and the shrink filter is connected to the mapper itk::ImageFileWriter. When the Update() method is invoked on the writer, the data processing pipeline causes each of these filters to execute in order, culminating in writing the final data to a file on disk.
The ITK spatial object framework supports the philosophy that the task of image segmentation and registration is actually the task of object processing. The image is but one medium for representing objects of interest, and much processing and data analysis can and should occur at the object level and not based on the medium used to represent the object.
ITK spatial objects provide a common interface for accessing the physical location and geometric properties of and the relationship between objects in a scene that is independent of the form used to represent those objects. That is, the internal representation maintained by a spatial object may be a list of points internal to an object, the surface mesh of the object, a continuous or parametric representation of the object’s internal points or surfaces, and so forth.
The capabilities provided by the spatial objects framework supports their use in object segmentation, registration, surface/volume rendering, and other display and analysis functions. The spatial object framework extends the concept of a “scene graph” that is common to computer rendering packages so as to support these new functions. With the spatial objects framework you can:
Currently implemented types of spatial objects include: Blob, Ellipse, Group, Image, Line, Surface, and Tube. The itk::Scene object is used to hold a list of spatial objects that may in turn have children. Each spatial object can be assigned a color property. Each spatial object type has its own capabilities. For example, the itk::TubeSpatialObject indicates the point where it is connected with its parent tube.
There are a limited number of spatial objects in ITK, but their number is growing and their potential is huge. Using the nominal spatial object capabilities, methods such as marching cubes or mutual information registration can be applied to objects regardless of their internal representation. By having a common API, the same method can be used to register a parametric representation of a heart with an individual’s CT data or to register two segmentations of a liver.
While the core of ITK is implemented in C++, Python bindings can be automatically generated and ITK programs can be created using Python. The wrapping process in ITK is capable of handling generic programming (i.e., extensive use of C++ templates). Systems like VTK, which use their own wrapping facility, are non-templated and customized to the coding methodology found in the system, like object ownership conventions. Even systems like SWIG that are designed for general wrapper generation have difficulty with ITK code because general C++ is difficult to parse. As a result, the ITK wrapper generator uses a combination of tools to produce language bindings.
To learn more about the wrapping process, please see the section on module wrapping, Section 9.5. The wrapping process is orchestrated by a number of CMake macros found in the Wrapping directory. The result of the wrapping process is a set of shared libraries (.so in Linux or .dlls on Windows) that can be used by interpreted languages.
There is almost a direct translation from C++, with the differences being the particular syntactical requirements of each language. For example, to dilate an image using a custom structuring element using the Python wrapping:
The same code in C++ would appear as follows:
const char ⋆ outputImage = argv[2];
const unsigned int radiusValue = atoi( argv[3] );
using PixelType = unsigned char;
constexpr unsigned int Dimension = 2;
using ImageType = itk::Image< PixelType, Dimension >;
using ReaderType = itk::ImageFileReader< ImageType >;
ReaderType::Pointer reader = ReaderType::New();
reader->SetFileName( inputImage );
using StructuringElementType = itk::FlatStructuringElement< Dimension >;
StructuringElementType::RadiusType radius;
radius.Fill( radiusValue );
StructuringElementType structuringElement =
StructuringElementType::Ball( radius );
using BinaryDilateImageFilterType = itk::BinaryDilateImageFilter< ImageType,
ImageType, StructuringElementType >;
BinaryDilateImageFilterType::Pointer dilateFilter =
BinaryDilateImageFilterType::New();
dilateFilter->SetInput( reader->GetOutput() );
dilateFilter->SetKernel( structuringElement );
This example demonstrates an important difference between C++ and a wrapped language such as Python. Templated classes must be instantiated prior to wrapping. That is, the template parameters must be specified as part of the wrapping process. In the example above, the ImageFileReader[ImageType] indicates that this class, implementing an image source, has been instantiated using an input and output image type of two-dimensional unsigned char values (i.e., UC). To see the types available for a given filter, use the .GetTypes() method.
Typically just a few common types are selected for the wrapping process to avoid an explosion of types and hence, library size. To add a new type, re-run the wrapping process to produce new libraries. Some high-level options for these types, such as common pixels types and image dimensions, are specified during CMake configuration. The types of specific classes that should be instantiated, based on these basic options, are defined by the ⋆.wrap files in the wrapping directory of a module.
Conversion of common, basic wrapped ITK classes to native Python types is supported. For example, conversion between the itk::Index and Python list or tuple is possible:
The advantage of interpreted languages is that they do not require the lengthy compile/link cycle of a compiled language like C++. Moreover, they typically come with a suite of packages that provide useful functionalities. For example, the Python ecosystem provides a variety of powerful tools for creating sophisticated user interfaces. In the future it is likely that more applications and tests will be implemented in the various interpreted languages supported by ITK. Other languages like Java, Ruby, Tcl could also be wrapped in the future.
Binary python packages are available in PyPI and can be installed in Python distributions downloaded from Python.org, from system package managers like apt or homebrew, or from distributions like Anaconda.
To install the ITK Python package, run:
Binary python packages are built nightly from the Git master branch, and they can be installed by running:
In order to access the Python interface of ITK, make sure to compile with the CMake ITK_WRAP_PYTHON option. In addition, choose which pixel types and dimensions to build into the wrapped interface. Supported pixel types are represented in the CMake configuration as variables named ITK_WRAP_<pixel type>. Supported image dimensions are enumerated in the semicolon-delimited list ITK_WRAP_DIMS, the default value of which is 2;3 indicating support for 2- and 3-dimensional images. The Release CMake build configuration is recommended.
After configuration, check to make sure that the values of the following variables are set correctly:
particularly if there are multiple Python installations on the system.
Python wrappers can be accessed from the build tree without installing the library. An environment to access the itk Python module can be configured using the Python virtualenv tool, which provides an isolated working copy of Python without interfering with Python installed at the system level. Once the virtualenv package is installed on your system, create the virtual environment within the directory ITK was built in. Copy the WrapITK.pth file to the lib/python2.7/site-packages on Unix and Lib/site-packages on Windows, of the virtualenv. For example,
On Windows, it is also necessary to add the ITK build directory containing the .dll files to your PATH environmental variable if ITK is built with the CMake option BUILD_SHARED_LIBS enabled. For example, the directory containing .dll files for an ITK build at C:\ITK-build when built with Visual Studio in the Release configuration is C:\ITK-build\bin\Release.