VTK/Image Rendering Classes

The display of slices of 3D images in VTK is currently much more difficult and much less flexible than it should be. A typical pipeline for displaying an oblique reformat of an image will consist of vtkImageReslice, vtkImageMapToColors, and vtkImageActor, and most VTK novices (in fact even most experts) will have great difficulty sorting through the many settings of these filters to achieve the desired result.

The primary goal of this project is to provide a 3D image mapper that will take care of all the details so that VTK users can display reformats with ease. In order for this to be done, the vtkImageActor must be replaced with a new image prop class that has SetMapper() and SetProperty() methods like those of vtkActor and vtkVolume.

Proposal

Oblique view with dark blue background.

I propose a new prop class called vtkImage that will be the image-display equivalent of vtkActor and vtkVolume. This class will have a property class and at least two mapper classes.

• vtkImage
  • vtkImageActor (subclass)
• vtkImageProperty
• vtkImageMapper3D
  • vtkImageResliceMapper (subclass)
  • vtkImageSliceMapper (subclass)

In addition to the new classes, the vtkInteractorStyleImage will be modified so that it has a "3D mode" for interacting with 3D images.

vtkImage

Sagittal view with dark blue background.

Unlike vtkImageActor, the vtkImage class will have a very simple interface. It will inherit the vtkProp3D methods for setting the data position and orientation with respect to world coordinates, but its only non-inherited methods will be SetMapper() and SetProperty().

The existing vtkLODProp3D class will be modified so that it can make an LOD from a vtkImageProperty and vtkImageMapper3D. This will allow vtkImage to be part of an LOD, which was impossible with vtkImageActor. In addition, the VTK pickers will be modified to use vtkImage in place of vtkImageActor (note that vtkImageActor will still be supported, since it will be a subclass).

By using alpha-blending (translucency), different images can be blended together at render time. They will be blended in the order in which they were added to the renderer. Potentially, each image could be assigned a layer number.

vtkImageProperty

Axial perspective view with dark blue background.

The property will control the image display parameters.

• SetInterpolationType(int type)
• SetColorWindow(double w)
• SetColorLevel(double l)
• SetLookupTable(vtkScalarsToColors *table)
• UseLookupTableScalarRangeOn() - default Off
• SetOpacity(double opacity)
• SetAmbient(double c) - default 1.0
• SetDiffuse(double c) - default 0.0

Interpolation types will be Nearest, Linear, and Cubic.

The lookup table is optional. If no lookup table is given, then the window/level will still be applied: single-component data will be displayed as grayscale, and multi-component data will be displayed as color. If a lookup table is provided, the VectorMode of the lookup table can be used to control how multi-component data will be displayed, i.e. by component, by magnitude, or as colors.

Use ColorWindow=255.0/ColorLevel=127.5 with no lookup table to pass RGB or 8-bit grayscale data directly to the screen.

vtkImageMapper3D

This is the base class for 3D image mappers.

• SetInput()
• SetInputConnection()
• BorderOn() - show the whole image, instead of cutting the border pixels in half
• SliceAtFocalPointOn() - automatically choose the slice at the camera focal point
• SliceFacesCameraOn() - automatically face the slice towards the camera
• vtkPlane *GetSlicePlane() - get the plane that defines the slice (in world coords)
• UpdateInformation() - update the SlicePlane (do this before using it)

vtkImageResliceMapper

A mapper for oblique reformats. Usually this will be used with SliceFacesCameraOn() as part of a 2D viewer, but this class can be used equally well to display a slice plane as part of a 3D scene.

The interface consists of these methods:

• SetSlicePlane(vtkPlane *plane) - set the slice that defines the plane (in world coords)
• SliceAtFocalPointOn() - automatically set slice position to the focal point
• SliceFacesCameraOn() - automatically set slice orientation to face the camera

Internally, this mapper uses vtkImageReslice to reslice the image, and creates a 2D texture as large as the viewport, where all texels outside the image bounds are transparent. This means that the class has very low GPU memory overhead and does not require any newer OpenGL features. Although it is not as fast as a pure GPU implementation would be, it has the advantage of being able to deal with very large images at full resolution and quality.

vtkImageSliceMapper

A mapper for x, y, or z slices, i.e. no obliques. It is intended to replace vtkImageActor, and once this mapper is finished, vtkImageActor will use it internally.

• SetSliceNumber(int slice)
• GetSliceNumberMin() - call UpdateInformation() to update this value
• GetSliceNumberMax() - call UpdateInformation() to update this value
• SetOrientationToX()
• SetOrientationToY()
• SetOrientationToZ()
• SliceAtFocalPointOn() - automatically set slice number to get slice closest to camera focal point
• SliceFacesCameraOn() - automatically set slice orientation so that it faces the camera
• CroppingOn() - default is Off
• SetCroppingRegion(int extent[6])

When given axially-oriented images or 2D images that are 8-bit greyscale or color, this mapper will load the image directly into a texture and it provides the highest efficiency image display available in VTK. It also provides hardware-accelerated bicubic interpolation on systems that support it (which means any GPU released since around 2003).

vtkInteractorStyleImage

This interactor style will be modified so that it can be used for 3D image reslicing. The following methods will be added:

• SetInteractionModeToImage2D()
• SetInteractionModeToImage3D()

When the mode is set to 3D, the following bindings will be present:

• Shift-LeftButton - rotate the camera, i.e. do oblique slicing

In both the 2D and 3D modes, the following new bindings will be present:

• Shift-RightButton - move the focal point in and out, i.e. scroll through slices
• X - sagittal view
• Y - coronal view
• Z - axial view

These keys will change the position and the view-up of the camera. Exactly what view orientations will create the desired ax/cor/sag views will depend on the coordinate system used for the image data. Because of this, the direction cosines for the X, Y, and Z orientations can be set manually with the following methods:

• SetXRightVector(double vector[3])
• SetXUpVector(double vector[3])
• ditto for Y and Z

It is your responsibility to set these view vectors correctly according to the coordinate system that you are using for your images (LPS, RAS, left-is-left, left-is-right, etcetera). The default settings are for RAS, left-is-left. For more information on image coordinate systems, see Orientation.

You can also set the LeftVector and UpVector directly:

• SetImageOrientation(double horizontal[3], double vertical[3]);

A note on window/level: vtkInteractorStyleImage will automatically find the property object of the image that is being displayed and modify it. In the old version of vtkInteractorStyle image, the user had to add observers for window/level. This will no longer be necessary.

Wish List

Shader programs for compositing

Anyone who uses Photoshop or The Gimp will be familiar with the concept of "layers" and the myriad ways that layers can be composited. It would be very nice if custom fragment shaders could be used to do the same thing with images in VTK. Some of the existing infrastructure for the VTK painters could probably be used for this.

Likelihood: so-so.

12-bit display

In the medical field, 10-bit and 12-bit greyscale displays are sometimes used. To support these displays, there could be an option to scale the intensity to [0,4095] instead of the usual [0,255]. One problem is that the way to achieve 12-bit display will vary from vendor to vendor.

Likelihood: high if someone is willing to test.

Projections

It is very common to use "thick slice" averaging to clean up noisy images, or to use MIP slabs when viewing blood vessels. In both of the mappers described above, it would be easy to use vtkImageProjection to achieve this.

Likelihood: sure thing.

N-Up views

Likewise, it would be easy to take multiple slices of the input image, and reformat them to an NxM grid on a single texture. This would be tricky for picking and getting pixel values, since the portion of the viewport that would typically contain just one image would contain a grid of images instead.

Likelihood: high.

An image mapper for geometry

It sounds kind of silly, but it could be useful. For example, a FEM could be sliced and displayed as an image. Or a 3D surface contour could be sliced and displayed as an image overlay. The main idea would be to take advantage of the image compositing: there would be no need for the user to make sure that the cutter was set up right and that all the correct depth offset were applied. Instead, the user could just pop the geometry into an image mapper, and the overlay would be perfect every time.

Likelihood: so-so.

Efficiency

The image mappers are designed to achieve the highest quality result, and although they will probably be fast enough to suit almost anyone, they will definitely slow down if displayed in a very large window or if several images are being composited. Hence, it would be nice to have some built-in LOD behaviour similar to the volume mappers. The most obvious speed-up would be to have vtkImageReslice sample the image at a lower resolution, and then have the texture re-interpolate to full resolution. A full-resolution and quarter-resolution texture would always be allocated on the GPU, but only the desired texture would be updated and rendered according to the desired LOD. The interactive rendering would be four times as fast as the high-quality rendering.

Likelihood: so-so.

GPU acceleration

Efficiency could also be improved by using 3D texture maps, but I'm not sure if this would be worthwhile. From a quality perspective, 16-bit textures would have to be used for medical images. Special fragment programs would be needed for cubic or other high-order interpolation. Lots of GPU memory would be required, particularly since the only time that such high speed would be needed is if multiple images are being composited. Also, it would be tricky to get it to work correctly with pipeline streaming. The mapper would have to be smart enough to know what parts of the 3D texture would be needed, so that the correct UpdateExtent could be set on the pipeline and the correct SubTexture loaded onto the GPU. Unless that was done, the whole texture would have to be uploaded on each execution.

Overall, the current CPU-based implementation is already very fast. A GPU-based image mapper would be unlikely to improve the user experience significantly.

Likelihood: low.

Using DrawPixels instead of Texture

The reslice mapper could have an option to render via glDrawPixels. The advantage would be higher performance on non-accelerated system (i.e. Mesa) and minimal use of GPU memory on accelerated systems. The disadvantage in this mode is that the image would have to be re-uploaded to the GPU on every render. This would be particularly bad for performance when interacting with blended images, since both images would have to be uploaded on each render, instead of just the image that had changed.

Likelihood: so-so.

Color Spaces other than RGB(A)

Already vtkScalarsToColors has a VectorMode setting to say what it does with multi-component input. It would be nice to add a vector mode called "INDEPENDENT" where each component is mapped through a separate color table, and then the results are summed and clamped. There would be a method where you could add multiple child color tables to the main color table, each of which would map a different component.

Another possibility is to just write a vtkScalarsToColors subclass that does -eg- YBR to RGB conversion.

Likelihood: so-so.

Checkerboarding

For comparing images, it is very nice to have a "checkerboard" option that will use a checkerboard stencil when compositing the images.

Likelihood: sure thing.