|
|
| (10 intermediate revisions by 5 users not shown) |
| Line 1: |
Line 1: |
| ==Coverage With C++==
| | {{CMake/Template/Moved}} |
|
| |
|
| Currently coverage is only supported on gcc compiler. To perform coverage test, make sure that your code is built
| | This page has moved [https://gitlab.kitware.com/cmake/community/wikis/doc/ctest/Coverage here]. |
| with debug symbols, without optimization, and with special flags. These flags are:
| |
| | |
| -fprofile-arcs -ftest-coverage
| |
| | |
| Also make sure to pass these flags to C compiler, CXX compiler, and the linker. For example:
| |
| | |
| CXXFLAGS="-g -O0 -Wall -W -Wshadow -Wunused-variable \
| |
| -Wunused-parameter -Wunused-function -Wunused -Wno-system-headers \
| |
| -Wno-deprecated -Woverloaded-virtual -Wwrite-strings -fprofile-arcs -ftest-coverage"
| |
| CFLAGS="-g -O0 -Wall -W -fprofile-arcs -ftest-coverage"
| |
| LDFLAGS="-fprofile-arcs -ftest-coverage"
| |
| | |
| ===Shared libs===
| |
| | |
| You might have issues with shared libs, as reported here:
| |
| * https://twiki.cern.ch/twiki/bin/view/SPI/HowToTestCoverageReports
| |
| | |
| (I was not able to use that trick, on a debian/oldstable and gcc 3.3.5, ref [http://groups.google.com/group/gnu.gcc.help/browse_thread/thread/7c71804863f408ea here]).
| |
| | |
| Other refs found on the net:
| |
| * http://lists.gforge.inria.fr/pipermail/visp-commits/2006-November/000273.html
| |
| * http://gcc.gnu.org/ml/gcc-help/2005-04/msg00019.html
| |
| | |
| Conclusion:
| |
| You should use gcc 3.4 and up, since gcc-3.3.5 and before have this issue.
| |
| | |
| ==C/C++ Coverage with Bullseye==
| |
| | |
| As of June 28th 2007 CVS CMake, there is support for the cross platform coverage tool Bullseye in CMake.
| |
| Bullseye is a cross platform commercial tool for coverage analysis. Information about
| |
| Bullseye can be found here: http://www.bullseye.com/. You will need a license for Bullseye to run
| |
| it.
| |
| | |
| Since Bullseye is a branch coverage tool, the output does not exacly match what Dart is expecting.
| |
| So, for now it makes up a fake number for lines of code covered. The number is 100 * the number of
| |
| functions in the file. (assume 100 lines per function). The percent coverage is a mix of branches
| |
| covered and functions covered.
| |
| | |
| To use Bullseye with CTest, it is easiest to use the new format ctest scripts.
| |
| You need to make sure the Bullseye installation bin directory is in your PATH
| |
| first. This can be done by setting the PATH environment variable in the ctest script.
| |
| Bullseye has a tool cov01 that can turn on/off the coverage tool. You run that to enable/disable
| |
| the coverage. It should be disabled for the configure part of a project so that coverage
| |
| is not done on the try/compile stuff. The environment variable COVFILE points to the coverage database file used
| |
| by Bullseye to store the coverage information. This needs to be set in the ctest script.
| |
| | |
| Here is an example ctest -S script that runs a CMake coverage dashboard with bullseye:
| |
| | |
| CMAKE_MINIMUM_REQUIRED (VERSION 2.2)
| |
|
| |
| # put bullseye path first
| |
| SET(ENV{PATH} "c:/Program\ Files/BullseyeCoverage/bin/:${PATH}")
| |
| | |
| # set variables for script
| |
| SET (CTEST_SITE "discworld.kitware")
| |
| SET (CTEST_BUILD_NAME "Windows")
| |
| SET (CTEST_NOTES_FILES
| |
| "${CTEST_SCRIPT_DIRECTORY}/${CTEST_SCRIPT_NAME}")
| |
| SET (CTEST_DASHBOARD_ROOT "c:/Hoffman/My Builds/")
| |
| SET (CTEST_SOURCE_DIRECTORY "${CTEST_DASHBOARD_ROOT}/CMake")
| |
| SET (CTEST_BINARY_DIRECTORY "${CTEST_DASHBOARD_ROOT}/CMake-coverage")
| |
| # Set the output file for bullsey
| |
| SET (ENV{COVFILE} "${CTEST_BINARY_DIRECTORY}/CMake.cov")
| |
| SET (CTEST_UPDATE_COMMAND "cvs")
| |
| SET (CTEST_CMAKE_GENERATOR "NMake Makefiles")
| |
| SET (CTEST_PROJECT_NAME "CMake")
| |
| | |
| # clear the binary directory and create an initial cache
| |
| CTEST_EMPTY_BINARY_DIRECTORY (${CTEST_BINARY_DIRECTORY})
| |
| # this is the initial cache to use for the binary tree, be careful to escape
| |
| # any quotes inside of this string if you use it
| |
| FILE(WRITE "${CTEST_BINARY_DIRECTORY}/CMakeCache.txt" "
| |
| CTEST_TEST_CTEST:BOOL=1
| |
| ")
| |
| | |
| # Find cov01, the bullseye tool to turn on/off coverage
| |
| FIND_PROGRAM(COV01 cov01)
| |
| IF(NOT COV01)
| |
| MESSAGE(FATAL_ERROR "Could not find cov01")
| |
| ENDIF(NOT COV01)
| |
| # turn off coverage for configure step
| |
| SET(RES 1)
| |
| EXECUTE_PROCESS(COMMAND ${COV01} -0 RESULT_VARIABLE RES)
| |
| IF(RES)
| |
| MESSAGE(FATAL_ERROR "could not run cov01 -0")
| |
| ENDIF(RES)
| |
| | |
| # do the dashboard steps
| |
| CTEST_START (NightlyCoverage)
| |
| CTEST_UPDATE (SOURCE "${CTEST_SOURCE_DIRECTORY}")
| |
| CTEST_CONFIGURE (BUILD "${CTEST_BINARY_DIRECTORY}")
| |
| # turn on coverage for build and test
| |
| SET(RES 1)
| |
| EXECUTE_PROCESS(COMMAND ${COV01} -1 RESULT_VARIABLE RES)
| |
| IF(RES)
| |
| MESSAGE(FATAL_ERROR "could not run cov01 -1")
| |
| ENDIF(RES)
| |
| CTEST_BUILD (BUILD "${CTEST_BINARY_DIRECTORY}")
| |
| CTEST_TEST (BUILD "${CTEST_BINARY_DIRECTORY}")
| |
| CTEST_COVERAGE(BUILD "${CTEST_BINARY_DIRECTORY}")
| |
| CTEST_SUBMIT ()
| |
| | |
| ==Coverage With Python==
| |
| | |
| ===Simple case ===
| |
| | |
| $ cat hello.py
| |
| <pre>
| |
| """
| |
| Comment #1
| |
| """
| |
| def untested():
| |
| print "untested"
| |
| | |
| """
| |
| Comment #2
| |
| """
| |
| def tested():
| |
| print "tested"
| |
| | |
| """
| |
| Comment #3
| |
| """
| |
| if __name__ == '__main__':
| |
| tested()
| |
| </pre>
| |
| | |
| $ python /usr/lib/python2.3/trace.py -c --coverdir=. --ignore-dir /usr/lib/python2.3 hello.py
| |
| | |
| will produce a file
| |
| | |
| hello.cover
| |
| | |
| $ cat hello.cover
| |
| <pre>
| |
| """
| |
| Comment #1
| |
| 1: """
| |
| 1: def untested():
| |
| print "untested"
| |
| | |
| """
| |
| Comment #2
| |
| """
| |
| 1: def tested():
| |
| 1: print "tested"
| |
| | |
| """
| |
| Comment #3
| |
| """
| |
| 1: if __name__ == '__main__':
| |
| 1: tested()
| |
| </pre>
| |
| | |
| ===Complex case ===
| |
| | |
| Using python code from VTK:
| |
| | |
| VTK/Wrapping/Python/vtk/test/Testing.py
| |
| | |
| $ python /usr/lib/python2.3/trace.py -c --coverdir=. --ignore-dir /usr/lib/python2.3 Testing.py
| |
| | |
| will produce:
| |
| | |
| ls *.cover
| |
| Testing.cover vtk.__init__.cover vtk.filtering.cover vtk.graphics.cover vtk.imaging.cover vtk.io.cover vtk.rendering.cover vtk.util.__init__.cover vtk.util.vtkMethodParser.cover vtk.widgets.cover
| |
| vtk.__helper.cover vtk.common.cover vtk.genericfiltering.cover vtk.hybrid.cover vtk.infovis.cover vtk.parallel.cover vtk.test.BlackBox.cover vtk.util.vtkConstants.cover vtk.volumerendering.cover
| |
| | |
| Example:
| |
| * [[Media:Testing.cover.txt|Testing.cover]]
| |
| * [[Media:Vtk.util.vtkMethodParser.cover.txt|vtk.util.vtkMethodParser.cover]]
| |
| | |
| ===Python Trace.py Coverage In CTest===
| |
| | |
| The coverage was implemented, but there are two issues.
| |
| | |
| '''Minor issue #1''': Trace.py does not differentiate between lines that are not covered and the lines that cannot possibly be covered. This means files with lots of comments will have pore coverage
| |
| | |
| '''Minor issue #2''': Trace.py does not work with the compiled python files (*.pyc).
| |
| | |
| '''Major issue''': Trace.py does not encode the location of the source file. This means that CTest has to guess where the source file is. Unfortunately the guessing can be either pore or time consuming.
| |
| | |
| '''Possible solution''': Make wrapper for Trace.py and include it with CTest. This wrapper could solve the lack of source file location.
| |
