CTest/Coverage

Coverage With C++

Currently coverage is only supported on gcc compiler. To perform coverage test, make sure that your code is build with debug symbols, without optimization, and with special flags. These flags are:

-fprofile-arcs -ftest-coverage

Also make sure to pass these flags to C compiler, CXX compiler, and the linker. For example:

CXXFLAGS="-g -O0 -Wall -W -Wshadow -Wunused-variable \
 -Wunused-parameter -Wunused-function -Wunused -Wno-system-headers \
 -Wno-deprecated -Woverloaded-virtual -Wwrite-strings -fprofile-arcs -ftest-coverage"
CFLAGS="-g -O0 -Wall -W -fprofile-arcs -ftest-coverage"
LDFLAGS="-fprofile-arcs -ftest-coverage"

Shared libs

You might have issues with shared libs, as reported here:

• https://twiki.cern.ch/twiki/bin/view/SPI/HowToTestCoverageReports

(I was not able to use that trick, on a debian/oldstable and gcc 3.3.5, ref here).

Other refs found on the net:

• http://lists.gforge.inria.fr/pipermail/visp-commits/2006-November/000273.html
• http://gcc.gnu.org/ml/gcc-help/2005-04/msg00019.html

Conclusion: You should use gcc 3.4 and up, since gcc-3.3.5 and before have this issue.

Coverage With Python

Simple case

$ cat hello.py

"""
Comment #1
"""
def untested():
  print "untested"

"""
Comment #2
"""
def tested():
  print "tested"

"""
Comment #3
"""
if __name__ == '__main__':
  tested()

$ python /usr/lib/python2.3/trace.py -c --coverdir=. --ignore-dir /usr/lib/python2.3 hello.py

will produce a file

hello.cover

$ cat hello.cover

       """
       Comment #1
    1: """
    1: def untested():
         print "untested"

       """
       Comment #2
       """
    1: def tested():
    1:   print "tested"

       """
       Comment #3
       """
    1: if __name__ == '__main__':
    1:   tested()

Complex case

Using python code from VTK:

VTK/Wrapping/Python/vtk/test/Testing.py

$ python /usr/lib/python2.3/trace.py -c --coverdir=. --ignore-dir /usr/lib/python2.3 Testing.py

will produce:

ls *.cover
Testing.cover       vtk.__init__.cover  vtk.filtering.cover         vtk.graphics.cover  vtk.imaging.cover  vtk.io.cover        vtk.rendering.cover      vtk.util.__init__.cover      vtk.util.vtkMethodParser.cover  vtk.widgets.cover
vtk.__helper.cover  vtk.common.cover    vtk.genericfiltering.cover  vtk.hybrid.cover    vtk.infovis.cover  vtk.parallel.cover  vtk.test.BlackBox.cover  vtk.util.vtkConstants.cover  vtk.volumerendering.cover

Example:

• Testing.cover
• vtk.util.vtkMethodParser.cover

Python Trace.py Coverage In CTest

The coverage was implemented, but there are two issues.

Minor issue #1: Trace.py does not differentiate between lines that are not covered and the lines that cannot possibly be covered. This means files with lots of comments will have pore coverage

Minor issue #2: Trace.py does not work with the compiled python files (*.pyc).

Major issue: Trace.py does not encode the location of the source file. This means that CTest has to guess where the source file is. Unfortunately the guessing can be either pore or time consuming.

Possible solution: Make wrapper for Trace.py and include it with CTest. This wrapper could solve the lack of source file location.