Gerrit provides a powerful code review tool to the VTK and ITK open source communities. For each submitted topic, Gerrit builds and tests the topic on three typical configurations: Ubuntu, Mac and Windows. The goal of these builds is to catch compile errors and test failures introduced by the topic, before the topic is merged into VTK or ITK.
Recurring test failures on the Gerrit builds can easily mask new defects introduced by a topic. Recently, the number of recurring failures for VTK was:
- Ubuntu - 28
- Mac - 4
- Windows - 3
This experiment seeks to reduce VTK Gerrit recurring test failures to 0 and keep them at 0.
The basic methodology (from Wikipedia) consists of the following five steps:
- Define process goals that are consistent with customer demands and VTK's strategy.
- Measure key aspects of the current process and collect relevant data.
- Analyze the data to verify cause-and-effect relationships. Determine what the relationships are, and attempt to ensure that all factors have been considered.
- Improve or optimize the process.
- Control to ensure that any deviations from target are corrected before they result in defects. Set up pilot runs to establish software quality, move on to production, set up control mechanisms and continuously monitor the process.
Keep the number of VTK Gerrit recurring test failures(defects) to 0. When the defects are above 0, developers find it difficult to see if their changes have increased the number of defects.
As of January 26, 2013, there were 34 defects on the three VTK Gerrit builds:
- Ubuntu - 26
- Mac - 4
- Windows - 4
Initial analysis revealed:
- Python tests accounted for 15 of the defects. Note that only the Ubuntu system has python enabled.
- One of the tests, TestSplitViewportStereoHorizontal, was failing on all three systems. It also was failing on many other nightly builds.
- One of the python tests, TestTemplates, was failing on every VTK build that had python enabled, including the Gerrit Ubuntu build.
- Seven tests were timing out on the Ubuntu system.
- The remaining tests required further analysis and individual attention.
- Tests time out on Ubuntu - 7 tests
- Python syntax errors - 5 tests
- Python tests misconfiguration- 3 tests
- Suppress recurring Mac failures - 3 tests
- Failed to locate materials for shaders - 3 tests
- NumPy tests failing - 2 tests
- Failed image regression test - 1 test
- http://review.source.kitware.com/#/c/9545/ Adjust error threshold for TestDensifyPolyData]
- Bad clipping planes - 1 test
- TestRenderView fails on some X platforms - 1 test
- CamBlur failing with X Error of failed request: BadMatch XGetImage - 1 test
- Volume mapper failed - 1 test
- The test failed to check if the volume mapper supported the required extensions.
- TestTemplates failed - 1 test
- The python-wrapped vtkVector type wasn't able to link to its python-wrapped vtkTuple superclass because vtkTuple was in a different module. This patch adds import/export declarations to allow module boundaries to be crossed.
- The trick of automatically discovering what template instantiations are used does not work across module boundaries, however, so the needed instantiations of the vtkTuple template have been hard-coded for now.
- TestTilingCxx and TestInteractorStyleTreeMapHover fail only on one Windows platform - 2 tests
- Suppressed the test on the offending gerrit cdash@home system
set(CTEST_TEST_ARGS EXCLUDE TestTilingCxx|TestInteractorStyleTreeMapHover)
- TestSplitViewportStereoHorizontal fails on all three gerrit builds - 1 test
Some linux platforms failed because their graphics cards don't do antialiasing. This commit turns of antialiasing on ALL platforms provides a new non-antialiased baseline, and increases the error threshold. All windows machines failed because SetStereoRender was called before the StereoType was set. This commit should fix that, because it calls SetStereoType before SetStereoRender. All OS X machines still fail because the ViewAngle is, for some unknown reason, not interpreted properly for this test on that platform.
As of February 3, the 6 defects remain:
- Ubuntu - 2 defects
- Mac - 1 defect
- Windows - 3 defects
As of February 8, the 3 defects remain:
- Ubuntu - 1 defect
- Mac - 1 defect
- Windows - 1 defect
As of February 10, the 3 defects remain:
- Ubuntu - 0 defects
- Mac - 1 defect
- Windows - 0 defects
Once reduced to 0, developer diligence is needed to keep the defects to 0. The burden is on the Gerrit reviewers.