ParaView/Line Integral Convolution
The line integral convolution(LIC) vector field visualization technique convolves noise with a vector field producing streaks along vector field tangents. Originally the technique was developed for use with 2D image based data but has since been extended to use on arbitrary surfaces and volumes. ParaView supports LIC on arbitrary surfaces via the Surface LIC plugin. ParaView's surface LIC implementation has been designed to work with composite data in parallel and includes a number of customizations that facilitate interactive data exploration. Some examples of the surface LIC in action are shown in Figure 1. This document applies to ParaView 4.1 , and VTK 6.1 and later releases.
ParaView Surface LIC Plugin
VTK's surface LIC painter may be used in ParaView by loading the Surface LIC Plugin. Open the Tools menu select Manage Plugins and Load the plugin on both client and server. Once the plugin is loaded the surface LIC is enabled by selecting it from the list of Representations. The various user run-time tunable parameters are exposed in the GUI in the display Properties panel. Click on the show Advanced button to see all of the parameters. The following table and figure have a basic description of the parameters with links to more information.
|Select Input Vectors||This is used to select the vector field.|
|Number Of Steps||Number of integration steps.|
|Step Size||Step size given in the original vector field's units.|
|Normalize Vectors||Enable integrator vector field normalization.|
|Two pass image LIC||Enable the two pass algorithm.|
|Color Mode||Select the shader that is used to combine rendered lit and scalar colored surface geometry with the gray scale LIC. (Blend, Multiply)|
|LIC Intensity||Sets the intensity for LIC pattern when using Blend shader.|
|Map Mode Bias||Sets the additive term used to brighten or darken the final colors when using the Multiply shader.|
|Enhance Contrast||Enable contrast enhancement shaders (Off, LIC Only, LIC and Color, Color Only)|
|Low LIC Contrast Enhancement Factor||Adjusts the minimum intensity value realized in the output of the image LIC contrast enhancement.|
|High LIC Contrast Enhancement Factor||Adjusts the maximum intensity value realized in the output of the image LIC contrast enhancement.|
|Low Color Contrast Enhancement Factor||Adjusts the minimum lightness value realized in the output of the surface LIC painter color contrast enhancement.|
|High Color Contrast Enhancement Factor||Adjusts the maximum lightness value realized in the output of the surface LIC painter color contrast enhancement.|
|AntiAlias||Sets the number of times the image LIC antialiasing stage is applied. 0 disables the antialiasing stage.|
|Mask On Surface||Use the magnitude of the surface projected vectors for the masking test|
|Mask Threshold||Vector magnitude above which the fragments are masked. If the vector field has an out-of-surface component one may disable Mask On Surface so that masked fragments match the LUT.|
|Mask Intensity||Fraction of Mask Color to blend with lit and scalar colored surface geometry.|
|Mask Color||RGB tuple to use for masked fragments.|
|Noise Type||Select the noise distribution or type. Can be uniform, Gaussian, or Perlin.|
|Noise Texture Size||Set the dimensions of the square noise texture. Large values may negatively impact performance.|
|Noise Grain Size||Set the number of pixels that each noise value covers.|
|Min Noise Value||Set the minimum value in the generated noise.|
|Max Noise Value||Set the maximum value in the generated noise.|
|Number Of Noise Levels||Set the number of discrete values in the generated noise.|
|Impulse Noise Probability||Set the probability that any given pixel in the generated texture will be touched. When 1 all pixels are written to. Values less than 1 produce impulse noise.|
|Impulse Noise Background Value||Set the gray scale intensity to be used when a pixel is not touched when generating impulse noise.|
|Noise Generator Seed||A seed value for the random number generator.|
|Composite Strategy||Select the compositing strategy used for parallel operations (INPLACE, DISJOINT, BALANCED, AUTO).|
|Use LIC For LOD||When enabled LIC is computed on LOD during interaction. This can negatively impact performance.|
ParaView's implements surface LIC in OpenGL using a framebuffer ping-pong technique from early GPGPU computing. The required OpenGL extensions are listed in the following table. These requirements are satisfied for most OpenGL 2.1 and newer implementations. The surface LIC painter may be used without graphics hardware by using the llvmpipe OSMesa state tracker found in Mesa3D OpenGL v9.2 and newer. See ParaView and Mesa3D for information on configuring ParaView for use with OSMesa. Support for a given system may be checked using either ParaView's or VTK's regression tests. ParaView's LIC tests can be executed by issuing the command ctest -R SurfaceLIC -L PARAVIEW --verbose while VTK's LIC tests can be executed by issuing the command ctest -R LIC --verbose from their respective build directories. The output of these tests will indicate whether or not the system supports the required extensions.
|GL_ARB_vertex_buffer_object, GL_ARB_pixel_buffer_object, GL_ARB_depth_buffer_float, GL_ARB_multitexture, GL_EXT_texture3D, GL_ARB_texture_non_power_of_two, GL_ARB_texture_float, GL_ARB_depth_texture, GL_ARB_draw_buffers, GL_EXT_framebuffer_object, GL_EXT_framebuffer_blit, GL_ARB_shading_language_100, GL_ARB_shader_objects, GL_ARB_vertex_shader, GL_ARB_fragment_shader|
The surface LIC algorithm projects vectors defined on an arbitrary surface onto the surface and then from physical space into screen space where an image LIC is computed. After the image space LIC computation scalar colors may be combined with the resulting LIC using a variety of specialized shaders. When running in parallel guard pixels are needed for the image LIC computation and the transition from from world space to screen space necessitates a compositing step to make the screen space vector field consistent where there are overlapping screen space areas across processes. A schematic of the algorithm is presented in figure 3a. Optional processing stages are shaded gray, cached textures are represented by red parallelograms, and green double arrows indicate inter-process communication that occurs only during parallel operation. On the right half of the figure a break-out diagram detailing the processing stages used in our image LIC algorithm are shown. Our image LIC algorithm is implemented in two passes. In the first pass, fragments are masked based on user provided criteria then a basic LIC is computed. In the second pass, the result of the first pass is run through contrast and edge enhancement stages before being used as "noise" texture in the second LIC pass. The second pass substantially improves the visual quality of the streaks. Finally optional anti-aliasing and contrast enhancement stages are applied.
For a given vector dataset the streaking patterns produce by the LIC can vary widely based on the choice of noise dataset, screen resolution, camera, and view parameters. The properties of the noise dataset play an important role in determining the characteristics of streaking patterns produced and can be varied easily giving one a means for controlling streaking characteristics in the resulting LIC. ParaView's noise texture generator defines the following 9 run-time tunable degrees of freedom, which together can be used to modify streaking pattern, dynamic range, and contrast, in the resulting LIC. For example, by varying the impulse probability along with the choice of background intensity control over number of light or dark pixels in the LIC image is attained. Varying the noise grain size along with impulse probability and impulse background back color give one greater control over the width of the streaks produced. With the ability to selecting a noise distribution, its minimum, maximum, and number of noise levels, control over the contrast is gained. Figure 4 shows an example of using the noise texture generator to control the streaking in the resulting LIC.
Scalar color shaders
Prior to computing the LIC in image space lighting calculations and mapped scalar colors are applied to surface geometry and stored in texture. Once the LIC has been computed and itself stored in a texture, the two textures are combined to produce the final lit scalar colored rendering. ParaView implements two algorithms for blending the rendering of lit colored geometry with the LIC, a multiplicative or mapping approach and an additive or blending approach. In addition an algorithm for coloring masked fragments is provided.
Mapping colors onto the LIC
The mapping approach is described by the following equation:
where the indices identify a specific fragment, is the final RGB color, is LIC gray scale intensity, is the scalar RGB color, and is a biasing parameter, typically 0, that may be used for fine tuning. When , the typical case, colors are transferred directly to the final image where the LIC is 1, and a linearly scaled transfer of scalar colors where LIC gray scale color is less than one down to 0, where the final color is black. The bias parameter may be set to small positive or negative values between -1 and 1 to increase or decrease LIC values uniformly resulting in brighter or darker images. When final fragment colors, , are clamped such that .
With the mapping approach the distribution of intensity values in the LIC directly affect the accuracy and intensity with which scalar colors and lighting effects are transferred in the final rendering. In the final result that individual RGB channel values will be less then or equal to the maximum grayscale value in the image LIC. The greater the number of pixels close to 1 in the image LIC, the more accurately the scalar coloring and lighting are transferred into the final image. However, this must be balanced with a sufficient number of highly contrasting pixels where the value is closer to 0 in order to accurately represent the LIC pattern. Put succinctly with this multiplicative shader it's critical that the image LIC has high contrast and dynamic range with a good mix of light and dark values if it is to be effectively mapped onto scalar colors. However, the convolution process inherently reduces both contrast and dynamic range in the image LIC. To correct this we've introduced contrast enhancement stages in three places in the pipeline. See the section describing #Contrast enhancement stages below.
Blending colors with the LIC
The blending approach for combining scalar colors, lighting effects, and the image LIC is described by the following equation:
where the indices identify a specific fragment, is final RGB color, is LIC gray scale value, is the scalar RGB color, and is a constant ranging from 0 to 1, with a default of 0.8. Decreasing to obtain brighter colors diminishes the intensity of the LIC, and vise versa. In some cases when colors are bright the LIC is difficult to see and attaining the best results will require sacrificing slightly on both fronts. The blending approach benefits from an image LIC with high contrast and dynamic range as this tends to make patterns in the image LIC more easily visible after blending. Note that despite the fact that it inherently decreases visibility of features in scalar coloring and image LIC, the blending approach is especially useful with curved surfaces and pronounced lighting effects and also when scalar color map is very intense.
Masking invalid data
Masking is a technique that provides a specialized shader for handling the blending of fragments that have a vector magnitude below a user provided threshold. The masking test may be applied either to the original vectors or the surface projected vectors. For fragments with magnitude below the user supplied threshold, ParaView's implementations can render:
- unmodified noise data
- lit scalar colored geometry directly
- blend the rendering of lit scalar colored geometry with a mask color
Fragments are masked according to the following equation:
where the indices identify a specific fragment, is final RGB color, is the RGB mask color, is the scalar RGB color, and is the mask color intensity. This allows one control over the masking process so that masked fragments may be: highlighted (by setting a unique mask color and mask intensity > 0), made invisible with and without passing the un-convolved noise texture (by setting mask intensity 0), or made to show the scalar color at a similar level of intensity as the LIC (mask intensity > 0).
Two pass image LIC
ParaView implements a two pass LIC computation. Enabling this feature activates an algorithm sub-pipeline that applies a number image processing techniques which greatly improve the visibility of streaking patterns in the result. In the first pass a traditional LIC is computed. Image processing filters consisting of optional contrast enhancement(CE) filter and a Laplace edge-enhance(EE) filter are applied to the output of the first LIC pass strengthening streaking patterns. In the second pass LIC is recomputed using the output of the image processing filters in the place of noise. Integration in the second pass is made using of the number of integration steps to account for the fact that the filtered first pass output is relatively smooth. Stages in the two pass algorithm are shown in Figure 3 and the intermediate results of each are shown in figure 7.
The convolution process tends to decrease both contrast and dynamic range, narrowing and concentrating the distribution of intensity values in the resulting LIC around a mid tone. This narrowing and concentration leads to an overall dark and dull LIC making the combination with scalar colors and light fragments difficult. Using Gaussian noise during LIC computation often produces relatively smooth and pixelation free streaking in the LIC but tends to worsen this narrowing effect since the input intensities are already highly concentrated about a mid tone. In order to counteract this optional contrast enhancement (CE) stages have been added. The new stages increase the dynamic range and contrast, improve the streaking patterns that emerge during the convolution process, and facilitate combination of scalar colors. Three CE stages have been added, one after each LIC stage and one after the combination of scalar colors and LIC.
Image LIC CE stages
The LIC CE stages are implemented by histogram stretching of the gray scale colors in the LIC'ed image as follows:
where, the indices identify a specific fragment, is the fragment's gray scale color, is the gray scale color value to map to 0, is the gray scale color value to map to 1. When the contrast enhancement stage is applied on the input of the high-pass filter stage, and are always set to the minimum and maximum gray scale color of all fragments. In the final contrast enhancement stage and take on minimum and maximum gray scale colors by default but may be individually adjusted by the following set of equations:
where, , are all of the gray scale fragments in the LIC image and and are adjustment factors that take on values between 0 and 1. When and are 0 minimum and maximum are gray scale values are used. This is the default. Adjusting and above zero controls the saturation of normalization. This is useful, for example, if the brightness of pixels near the border dominate because these are convolved less because we can't integrate outside of the dataset.
The two LIC CE stages are controlled together, and only the final CE stage make use of the saturation tuning. In other words when two pass LIC is enabled the first LIC CE stage map the input distribution onto the range 0 to 1.
Occasionally the output of the use of the image LIC CE stages result in jagged or pixelated patterns in the LIC. Pixelation depends on a number of factors such as the vector data, properties of the noise texture, number of integration steps taken, and the min and max CE factors used. Pixelation can be reduced by enabling the optional anti-aliasing (AA) stage. The AA stage, when enabled, is applied to the input of the final LIC CE stage. By applying on the input of the final CE stage pixelation is reduced while increasing dynamic range and contrast. This helps to ensure bright scalar colors in the final image.
Painter CE stage
After the combination of lighting computations and mapped scalar colors with the LIC an option color contrast enhancement (CCE) stage may be applied. The CCE stage is implemented using histogram stretching on the fragments lightness in the HSL color space.
where, the indices identify a specific fragment, is the fragment's lightness in HSL space, is the lightness to map to 0, is the lightness to map to 1. and take on minimum and maximum lightness over all fragments by default but may be individually adjusted by the following set of equations:
where, are fragment lightness values and and are the adjustment factors that take on values between 0 and 1. When and are 0 minimum and maximum are lightness values are used. This is the default. Adjusting and above zero provides fine-tuning control over the saturation.
Normalizing vectors during integration is a trick that can be used to simplify integrator configuration and give the resulting LIC a uniformly smooth look. By using normalized vector field values the convolution occurs over the same integrated arclength for all pixels in the image. This gives the result a smooth and uniform look and makes it possible to provide reasonable default values for step size and number of steps to the integrator independent of the input vector field. The resulting visualization accurately shows the tangent field however variations in the relative strength of the vector field is lost and can alter the relationships of flow features making weak features prominent and strong features less so. In the context of developing a general purpose tool for interactive data exploration it's important to provide both options and let the user select the option that best fits there needs. For example figure 10 shows an example where normalization results in the emphasis of very weak features giving one a false impression of the flow.
Optimizations for interactivity
Given the complexity of the surface LIC pipeline, the computational expense of computing the LIC itself, and the large number of shader and noise generator parameters available to the user it is essential to render quickly and efficiently in order to deliver interactive rendering performance. For this reason the output of each shader stage is cached in a texture and user modifiable parameters have been grouped according to the shader stage that they affect. As a user interacts with the visualization cached results are leveraged so that only the stage affected by the interaction and the stages downstream from it are re-executed. There are large differences in the computational cost of the various stages. On large datasets the vector projection and image LIC stage split the majority of the rendering time. Other stages are orders of magnitude faster thus when either of these more expensive stages can be skipped it improves the rendering performance dramatically. Figure 1 shows the pipeline stages that are cached in order to improve interactive rendering performance in red.
ParaView's data parallel pipeline allows us to handle datasets larger than can fit on a single compute node and provides a means for achieving faster rendering time on very large datatsets. However, it's important to keep in mind that in ParaView load balancing considerations are made based on the data reported available by the reader and that filters may move data among processes, and add or subtract data process by process. In addition the distribution of rendering work depends on camera and view parameters. For example, the camera may be set such that a process's data may be off screen, or an upstream filter may have removed all data on a number of processes. Both of these examples create situations of load imbalance during parallel rendering.
The surface LIC algorithm is unique compared to other parallel rendering algorithms in two ways. First, in the data parallel setting the integration step may require access to off process vector data to produce consistent LIC at process boundaries. This is dealt with by adding a gaurd pixel halo to each screen space pixel extent over which the LIC will be computed. Second, after vectors have been locally projected into screen space and a depth test applied to ensure local correctness, the screen space vector field must then be composited in order to ensure a globally correct vector field where there is screen space overlap across multiple processes. The generation of guard pixels and vector field compositing are done together and occur within ParaView's normal image compositing pass.
Vector field compositing stage
Given the relatively high computational cost of computing the surface LIC it's important to have a good parallel distribution of rendering work so much so that it can be beneficial to redistribute screen space data when it can result in a more balanced work distribution. The vector field compositing stage provides an opportunity to load balance the LIC computation. However, working with in ParaView's normal compositing pass makes load balancing challenging. For instance, ParaView expects screen space data distribution to remain fixed which places some restrictions on what load balancing schemes will be practical since any reorganization of screen space data by moving data from the initial decomposition to a more favorable one requires moving it back after the computation has been made adding some communication overhead. When attempting to load balance rendering computations in this environment the goal of achieving equal distribution of work must be balanced by the compositing and communication costs.
ParaView provides 3 vector field compositing strategies for composite data, in-place, in-place disjoint, and balanced. To reduce communication overhead prior to compositing, screen space bounds of the input data are minimized using the cached depth buffer values. Each compositing strategy implies a specific target domain decomposition and once the target domain decomposition is determined each of it's extents are minimized and guard pixels are added.
The in-place strategy composites the vector field onto the minimized screen space extents of the input dataset without load balancing. This strategy is optimal when there is no off process screen space overlap across nodes in the input composite dataset, as in the case of computing LIC on a slice. In cases with little or no screen space overlap across processes only guard pixels are composited. However, the strategy can be efficient when there is substantial off-process screen space overlap across composite data nodes because for overlapping pixels LIC is redundantly computed on each process that overlaps. For example the in-place decomposition is shown in the center of figure 9. In this case it's relatively inefficient because of the large number of overlapping data across processes.
In-place disjoint compositing
The in-place disjoint strategy adds load balancing to the in-place strategy. The target domain decomposition to which vector field will be composited and LIC computed on is constructed by making the minimized input screen space domain decomposition disjoint with respect to itself. Data is left in-place which reduces the compositing costs and LIC is computed once for each pixel. This strategy can be more efficient than the in-place strategy when there is a high degree of off-process screen space overlap across composite dataset nodes. However, the disjointification process results in an increased number of sub-extents which tends to increase the number of ghost cells this can be a serious issue if the number of guard pixels approaches the number of valid pixels. When using the in-place disjoint strategy once the LIC is computed it must be mapped back onto the original domain decomposition in order to work with ParaView's image compositing pass. This "scatter" stage doesn't require compositing because by definition the source extents are unique. In the worst case the additional cost of the scatter stage makes the communication costs of the in-place disjoint strategy equal to those of the in-place strategy so when there are many overlapping off-process pixels the disjoint strategy can still be quite a bit more efficient.
The balanced strategy partitions the global minimized input dataset screen space bounds into equally sized tiles with one tile assigned to each process. The screen space vector field data is composited onto the new target domain decomposition where LIC is computed. Like the in-place disjoint strategy the balanced strategy assigns pixels to processes uniquely resulting in the LIC being computed only once for each process, and a scatter stage is required to move data back onto the ParaView's image compositing domain decomposition. One advantage of the balanced strategy over the in-place disjoint strategy is that because there is only one target pixel extent per ParaView process the ratio of guard pixels to valid pixels tends to be much smaller as the number of processes increases. However, the compositing costs can be relatively higher because data is not left in-place, and in situations where valid pixels don't fill the screen space extents some processes may be left with no work.
In the automatic compositing strategy an estimate of the compositing cost is made and a heuristic is used to select either the in-place or in-place disjoint strategy. Thus the benefits of both of these strategies are leveraged while some of the downsides are avoided on a case by case basis with user intervention. This is the recommended strategy and is used by default.