VTK/Image Rendering Classes

The display of slices of 3D images in VTK is currently much more difficult and much less flexible than it should be. A typical pipeline for displaying an oblique reformat of an image will consist of vtkImageReslice, vtkImageMapToColors, and vtkImageActor, and most VTK novices (in fact even most experts) will have great difficulty sorting through the many settings of these filters to achieve the desired result.

The primary goal of this project is to provide a 3D image mapper that will take care of all the details so that VTK users can display reformats with ease. In order for this to be done, the vtkImageActor must be replaced with a new image prop class that has SetMapper() and SetProperty() methods like those of vtkActor and vtkVolume.

These classes were merged into VTK on March 15, 2011.

Proposal

Oblique view with dark blue background.

I propose a new prop class called vtkImageSlice that will be the image-display equivalent of vtkActor and vtkVolume. This class will have a property class and at least two mapper classes.

• vtkImageSlice
  • vtkImageActor (subclass)
• vtkImageProperty
• vtkImageMapper3D
  • vtkImageResliceMapper (subclass)
  • vtkImageSliceMapper (subclass)

In addition to the new classes, the vtkInteractorStyleImage will be modified so that it has a "3D mode" for interacting with 3D images.

vtkImageSlice

Sagittal view with dark blue background.

Unlike vtkImageActor, the vtkImageSlice class will have a very simple interface. It will inherit all the vtkProp3D methods, but its only new methods will be SetMapper() and SetProperty().

The existing vtkLODProp3D class will be modified so that it can make an LOD from a vtkImageProperty and vtkImageMapper3D. This will allow vtkImageSlice to be part of an LOD, which was impossible with vtkImageActor. In addition, the VTK pickers will be modified to use vtkImageSlice in place of vtkImageActor (note that vtkImageActor will still be supported, since it will be a subclass).

By using alpha-blending (translucency), different images can be blended together at render time. They will be blended in the order in which they were added to the renderer. Potentially, each image could be assigned a layer number.

vtkImageProperty

Axial perspective view with dark blue background.

The property will control the image display parameters.

• SetInterpolationType(int type)
• SetColorWindow(double w)
• SetColorLevel(double l)
• SetLookupTable(vtkScalarsToColors *table)
• UseLookupTableScalarRangeOn() - default Off
• SetOpacity(double opacity)
• SetAmbient(double c) - default 1.0
• SetDiffuse(double c) - default 0.0

Interpolation types will be Nearest, Linear, and Cubic.

The lookup table is optional. If no lookup table is given, then the window/level will still be applied: single-component data will be displayed as grayscale, and multi-component data will be displayed as color. If a lookup table is provided, the VectorMode of the lookup table can be used to control how multi-component data will be displayed, i.e. by component, by magnitude, or as colors.

Use ColorWindow=255.0/ColorLevel=127.5 with no lookup table to pass RGB or 8-bit grayscale data directly to the screen.

vtkImageMapper3D

This is the base class for 3D image mappers.

• SetInput()
• SetInputConnection()
• BorderOn() - show the whole image, instead of cutting the border pixels in half
• SliceAtFocalPointOn() - automatically choose the slice at the camera focal point
• SliceFacesCameraOn() - automatically face the slice towards the camera
• vtkPlane *GetSlicePlane() - get the plane that defines the slice (in world coords)
• UpdateInformation() - update the SlicePlane (do this before using the SlicePlane)

vtkImageResliceMapper

A mapper for oblique reformats. Usually this will be used with SliceFacesCameraOn() as part of a 2D viewer, but this class can be used equally well to display a slice plane as part of a 3D scene.

The interface consists of these methods:

• SetSlicePlane(vtkPlane *plane) - set the slice that defines the plane (in world coords)
• SliceAtFocalPointOn() - automatically set slice position to the focal point
• SliceFacesCameraOn() - automatically set slice orientation to face the camera

Internally, this mapper uses vtkImageReslice to reslice the image, and creates a 2D texture as large as the viewport, where all texels outside the image bounds are transparent. This means that the class has very low GPU memory overhead and does not require any newer OpenGL features. Although it is not as fast as a pure GPU implementation would be, it has the advantage of being able to deal with very large images at full resolution and quality.

vtkImageSliceMapper

A mapper for x, y, or z slices, i.e. no obliques. It is intended to replace vtkImageActor, and once this mapper is finished, vtkImageActor will use it internally.

• SetSliceNumber(int slice)
• GetSliceNumberMin() - calls UpdateInformation() internally
• GetSliceNumberMax() - calls UpdateInformation() internally
• SetOrientationToX()
• SetOrientationToY()
• SetOrientationToZ()
• SliceAtFocalPointOn() - automatically set slice number to get slice closest to camera focal point
• SliceFacesCameraOn() - automatically set slice orientation so that it faces the camera
• CroppingOn() - default is Off
• SetCroppingRegion(int extent[6])

When given axially-oriented images or 2D images that are 8-bit greyscale or color, this mapper will load the image directly into a texture. It also provides hardware-accelerated bicubic interpolation on systems that support it (which means any GPU released since around 2003).

vtkInteractorStyleImage

This interactor style will be modified so that it can be used for 3D image reslicing. The following methods will be added:

• SetInteractionModeToImage2D()
• SetInteractionModeToImage3D()

When the mode is set to 3D, the following bindings will be present:

• Shift-LeftButton - rotate the camera, i.e. do oblique slicing

In both the 2D and 3D modes, the following new bindings will be present:

• Shift-RightButton - move the focal point in and out, i.e. scroll through slices
• X - sagittal view
• Y - coronal view
• Z - axial view

These keys will change the position and the view-up of the camera. Exactly what view orientations will create the desired ax/cor/sag views will depend on the coordinate system used for the image data. Because of this, the direction cosines for the X, Y, and Z orientations can be set manually with the following methods:

• SetXRightVector(double vector[3])
• SetXUpVector(double vector[3])
• ditto for Y and Z

It is your responsibility to set these view vectors correctly according to the coordinate system that you are using for your images (LPS, RAS, left-is-left, left-is-right, etcetera). The default settings are for RAS, left-is-left. For more information on image coordinate systems, see Orientation.

You can also set the LeftVector and UpVector directly:

• SetImageOrientation(double horizontal[3], double vertical[3]);

A note on window/level: vtkInteractorStyleImage will automatically find the property object of the image that is being displayed and modify it. In the old version of vtkInteractorStyle image, the user had to add observers for window/level. This will no longer be necessary.

Known Bugs

Z-buffer coincidence

When doing overlays, if the images do not have exactly the same bounds, then some pixels of the overlayed image might not draw as a result of the limited precision of the vertex pipeline and of the depth buffer. This is a general issue with any 3D rendering, and is why vtkMapper has ResolveCoincidentGeometry options.

There are some nice methods that can be used to avoid z-buffer issues with image rendering, but they aren't trivial to implement.

1. Use exactly the same geometry for all images that lie in the same slice plane, i.e. always use a quad that fills the whole viewport. This is what I tried first, by using alpha=0 for all texels that were out-of-bounds for the images. This worked perfectly for avoiding z-buffer issues, but introduced some alpha-related artifacts at the image border.
2. Do the rendering in two passes, where in the first pass all images are drawn with testing but not setting the Z-buffer, and in the second pass all images are drawn with setting the Z-buffer. This would require modifications to vtkRenderer.

The final option is to just use the vtkMapper strategy, instead of having a strategy that is specific to image rendering. The new classes already support the PolygonOffset strategy, which is useful for resolving geometry against images but is not very useful for resolving images against each other.

Assemblies

The SliceAtFocalPoint and SliceFacesCamera options require some internal use of the Prop3D matrix. If the Prop3D is part of an assembly, then its matrix is only part of the true data-to-world transformation matrix (except for a certain period of time during the rendering process when its matrix has been Poked). The fix is non-trivial. I'm certain that some other VTK classes are similarly buggy with respect to clipping planes and assemblies.

I recommend that vtkImageSlice not be used in an assembly until this issue is fixed.

Wish List

Shader programs for compositing

Anyone who uses Photoshop or The Gimp will be familiar with the concept of "layers" and the myriad ways that layers can be composited. It would be very nice if custom fragment shaders could be used to do the same thing with images in VTK. Some of the existing infrastructure for the VTK painters could probably be used for this.

Likelihood: so-so.

Layer numbers

The order in which images are rendered (and hence composited) depends on the order in which their props they are added to the renderer. It is tedious for the application to have to remove/re-add all image props whenever the layers have to be re-ordered. If the prop (or the mapper) had a "LayerNumber" ivar then re-ordering would be easier. For this to work, image-specific code would have to be added to the renderer. The default LayerNumber would be -1, which would cause a fall-back to rendering in the order in which the props were added to the renderer.

Likelihood: high

Render as underlay/overlay

It might be nice to have an option to render images as an underlay to avoid Z-buffer fighting with geometry that is rendered in the same scene. An underlay can be done by rendering the images first (before any geometry) and by not setting the depth buffer while rendering the images. An overlay would be done by rendering the images last and without doing any depth checks.

Likelihood: low

12-bit display

In the medical field, 10-bit and 12-bit greyscale displays are sometimes used. To support these displays, there could be an option to scale the intensity to [0,4095] instead of the usual [0,255]. One problem is that the way to achieve 12-bit display will vary from vendor to vendor.

Likelihood: high if someone is willing to test.

Projections

It is very common to use "thick slice" averaging to clean up noisy images, or to use MIP slabs when viewing blood vessels. In both of the mappers described above, it would be easy to use vtkImageProjection to achieve this.

Likelihood: sure thing.

N-Up views

Likewise, it would be easy to take multiple slices of the input image, and reformat them to an NxM grid on a single texture. This would be tricky for picking and getting pixel values, since the portion of the viewport that would typically contain just one image would contain a grid of images instead.

Likelihood: high.

An image mapper for geometry

It sounds kind of silly, but it could be useful. For example, a FEM could be sliced and displayed as an image. Or a 3D surface contour could be sliced and displayed as an image overlay. The main idea would be to take advantage of the image compositing: there would be no need for the user to make sure that the cutter was set up right and that all the correct depth offset were applied. Instead, the user could just pop the geometry into an image mapper, and the overlay would be perfect every time.

Likelihood: so-so.

Efficiency

The image mappers are designed to achieve the highest quality result, and although they will probably be fast enough to suit almost anyone, they will definitely slow down if displayed in a very large window or if several images are being composited. Hence, it would be nice to have some built-in LOD behaviour similar to the volume mappers. The most obvious speed-up would be to have vtkImageReslice sample the image at a lower resolution, and then have the texture re-interpolate to full resolution. A full-resolution and quarter-resolution texture would always be allocated on the GPU, but only the desired texture would be updated and rendered according to the desired LOD. The interactive rendering would be four times as fast as the high-quality rendering.

Likelihood: so-so.

GPU acceleration

Efficiency could also be improved by using 3D texture maps, but I'm not sure if this would be worthwhile. From a quality perspective, 16-bit textures or floating-point textures would have to be used for medical images. Special fragment programs would be needed for cubic or other high-order interpolation. Lots of GPU memory would be required, particularly since the only time that such high speed would be needed is if many images are being viewed simultaneously. Also, it would be tricky to get it to work correctly with pipeline streaming. The mapper would have to be smart enough to know what parts of the 3D texture would be needed, so that the correct UpdateExtent could be set on the pipeline and the correct SubTexture loaded onto the GPU.

Overall, the current CPU-based implementation is already very fast. A GPU-based image mapper would be unlikely to improve the user experience significantly.

Likelihood: low.

Using DrawPixels instead of Texture

The reslice mapper could have an option to render via glDrawPixels. The advantage would be higher performance on non-accelerated system (i.e. Mesa) and minimal use of GPU memory on accelerated systems. The disadvantage in this mode is that the image would have to be re-uploaded to the GPU on every render. This would be particularly bad for performance when interacting with blended images, since both images would have to be uploaded on each render, instead of just the image that had changed.

Likelihood: so-so.

Color Spaces other than RGB(A)

Already vtkScalarsToColors has a VectorMode setting to say what it does with multi-component input. It would be nice to add a vector mode called "INDEPENDENT" where each component is mapped through a separate color table, and then the results are summed and clamped. There would be a method where you could add multiple child color tables to the main color table, each of which would map a different component.

Another possibility is to just write a vtkScalarsToColors subclass that does -eg- YBR to RGB conversion. The default conversion should follow the CCIR 601 video standard, but options should be provided for other common YBR/YUV definitions as well.

It would also be good to support a BGRA color space, since that is what most hardware uses natively. Any textures that are not BGRA are converted by the OpenGL driver before they are uploaded to the GPU, so it is kind of silly that VTK converts the images to RGBA just to have the driver immediately re-convert them to BGRA.

Likelihood: so-so.

Checkerboarding

For comparing images, it is very nice to have a "checkerboard" option that will use a checkerboard stencil when compositing the images.

Likelihood: sure thing.

Smart Scaling

The possibility to visualize the line scanner images (e.g. 1 pixel height and 1024 pixel width). For this purpose it would be nice to have direction-dependent scalling. Also the interaction part must take this scaling factors into account accordingly.

Could the author of this request please explain in more detail?

The current implementation of vtkImageViewer2 will not show an 1x1024 image at all. To visualize images like this you need to create a copy with greater dimension (e.g. 128x1024) and repeatedly memcopy the same image line in it. This approach breaks the interaction part, because you need to implement custom pixel picking and so on... I wonder if it would be possible to use direction-dependent scaling, so you don't need to use a copy of the image line and you can work with original data. This approach could be also applied to images like 10x4096, so you can more effectively use the display area and show the whole image at the same time.

Likelihood: ???.

Border pixel clipping

In the current implementation of vtkImageViewer2 the image border pixels are clipped to the half. It would be nice to have an option to show whole border pixels.

Likelihood: done.

The BorderOn() option in vtkImageMapper3D eliminates the border pixel clipping. But for the vtkImageSliceMapper this option only works properly on OpenGL drivers that have GL_CLAMP_TO_EDGE. Some very old GPUs only support the older GL_CLAMP mode which will blend the border with the "border color", which is black by default.