ITK Release 4/Performance Experiments/Reducing CTest Output: Difference between revisions
(Created page with "Testing is critical to high quality software and the ITK developers are expected to produce unit tests for each class. However, some tests produce large amounts of output which m...") |
No edit summary |
||
| Line 2: | Line 2: | ||
This experiment looks at the size of output produced by ITKv4 and looks for ways to reduce a test's output. | This experiment looks at the size of output produced by ITKv4 and looks for ways to reduce a test's output. | ||
==Approach== | ==Approach== | ||
| Line 14: | Line 11: | ||
*'''''I'''mprove'' or optimize the process. | *'''''I'''mprove'' or optimize the process. | ||
*'''''C'''ontrol'' to ensure that any deviations from target are corrected before they result in defects. Set up pilot runs to establish software quality, move on to production, set up control mechanisms and continuously monitor the process. | *'''''C'''ontrol'' to ensure that any deviations from target are corrected before they result in defects. Set up pilot runs to establish software quality, move on to production, set up control mechanisms and continuously monitor the process. | ||
==Define== | |||
Reduce the total test output of ITKv4 without affecting code coverage or value of the tests. | |||
==Measure== | |||
As of October 1, 2011, there were 2202 ITKv4 tests producing 10.6 meg of test output for a single platform. 2 tests produced 18% of the output and 65 of the 2202 tests produced 60% of the output. This data was gathered from a cdash file provided by Dave Cole of Kitware. | |||
==Analyze== | |||
Analysis of test out sizes revealed the following types of tests: | |||
# Tests that produce valuable output, even if they pass: | |||
## itkSystemInformation - echoes the output of several CMake files pruced during the build process, e.g. CMakeCache.txt | |||
## itkNumericsTests - provides information about numeric limits and capabilities for a given platform. | |||
# Tests that produce reasonable output (< 1k characters) | |||
## Over 1100 of the 2202 tests produce < 1k characters | |||
# Tests that produce reasonable output but > 1k characters | |||
# Tests that are producing erroneous output | |||
# Tests that are producing useless output | |||
==Improve== | |||
==Control== | |||
Revision as of 17:57, 4 October 2011
Testing is critical to high quality software and the ITK developers are expected to produce unit tests for each class. However, some tests produce large amounts of output which may be useful to the developer, but places a burden on the cdash database. Furthermore, some hypothesize that test output size may affect the performance of cdash.
This experiment looks at the size of output produced by ITKv4 and looks for ways to reduce a test's output.
Approach
This experiment uses the DMAIC methodology of the Six Sigma management process to "Define", "Measure", "Analyze", "Improve" and "Control" test output in ITKv4. The basic methodology (from Wikipedia) consists of the following five steps:
- Define process goals that are consistent with customer demands and ITKv4's strategy.
- Measure key aspects of the current process and collect relevant data.
- Analyze the data to verify cause-and-effect relationships. Determine what the relationships are, and attempt to ensure that all factors have been considered.
- Improve or optimize the process.
- Control to ensure that any deviations from target are corrected before they result in defects. Set up pilot runs to establish software quality, move on to production, set up control mechanisms and continuously monitor the process.
Define
Reduce the total test output of ITKv4 without affecting code coverage or value of the tests.
Measure
As of October 1, 2011, there were 2202 ITKv4 tests producing 10.6 meg of test output for a single platform. 2 tests produced 18% of the output and 65 of the 2202 tests produced 60% of the output. This data was gathered from a cdash file provided by Dave Cole of Kitware.
Analyze
Analysis of test out sizes revealed the following types of tests:
- Tests that produce valuable output, even if they pass:
- itkSystemInformation - echoes the output of several CMake files pruced during the build process, e.g. CMakeCache.txt
- itkNumericsTests - provides information about numeric limits and capabilities for a given platform.
- Tests that produce reasonable output (< 1k characters)
- Over 1100 of the 2202 tests produce < 1k characters
- Tests that produce reasonable output but > 1k characters
- Tests that are producing erroneous output
- Tests that are producing useless output
