CMake/Testing With CTest

From KitwarePublic
< CMake
Revision as of 11:48, 20 August 2004 by Andy (talk | contribs) (Initial writeup about CTest)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigationJump to search

Simple Testing

CMake has support for adding tests in the project. To do that, insert the following command in the CMakeLists.txt file:

ENABLE_TESTING()

From that point on, you can add tests in the project using ADD_TEST command:

ADD_TEST(SimpleTest ${EXECUTABLE_OUTPUT_PATH}/SimpleTest Hello)

After building the project, you should be able to do:

1. Unix

make test

2. On GUI development environments such as Visual Studio rebuild the target RUN_TESTS

For more information about ENABLE_TESTING and ADD_TEST, Look at CMake Documentation or run:

cmake --help
cmake --help-full
cmake --help-command-list

or

cmake --help-command ENABLE_TESTING
cmake --help-command ADD_TEST

Dashboards

Testing Dashboards are web pages that display overview of the project testing. The testing clients configure and build the project, as well as run some number of tests. The results of these operations are then submitted to the central server, which prepares the overview pages. Examples of Testing Dashboards are: VTK, ITK, and ParaView.

To enable submission to Dart Testing Dashboard, include the following in CMakeLists.txt:

INCLUDE(Dart)

By default the settings will be to submit to Kitware's Public Dashboard. In order to submit to some other dashboard, create file "DartConfig.cmake" in the toplevel source directory and set the dashboard preferences.

Example of this file is:

# Dashboard is opened for submissions for a 24 hour period starting at
# the specified NIGHLY_START_TIME. Time is specified in 24 hour format.
SET (NIGHTLY_START_TIME "23:00:00 EDT")

# Dart server to submit results (used by client)
IF(DROP_METHOD MATCHES http)
  SET (DROP_SITE "public.kitware.com")
  SET (DROP_LOCATION "/cgi-bin/HTTPUploadDartFile.cgi")
ELSE(DROP_METHOD MATCHES http)
  SET (DROP_SITE "public.kitware.com")
  SET (DROP_LOCATION "/incoming")
  SET (DROP_SITE_USER "ftpuser")
  SET (DROP_SITE_PASSWORD "public")
ENDIF(DROP_METHOD MATCHES http)

SET (TRIGGER_SITE 
       "http://${DROP_SITE}/cgi-bin/Submit-vtk-TestingResults.pl")

This will submit testing results to the VTK dashboard.

CTest - Client for Dart Dashboard Server

CTest is a testing program distributed as a part of CMake. It can perform simple task of running a set of tests but it can also generate and submit Dart compatible Dashboard results. The good thing about CTest is that if you use CMake, you already have CTest. Detailed description of CTest option can be seen by running:

ctest --help

or

ctest --help-full

To convert existing Dart Client run from the project, find lines like:

 cd ProjectNightlyBuildDirectory
 tclsh /location/of/Dart/Source/Client/DashboardManager.tcl DartConfiguration.tcl \
	Nightly Start Update Configure Build Test Submit

and convert them to CTest style:

cd ProjectNightlyBuildDirectory
ctest -D Nightly

Dashboard can be also generated in stages. This way partial testing results can be submitted and seen before long operations are completed:

cd ProjectNightlyBuildDirectory
ctest -D NightlyStart
ctest -D NightlyUpdate
ctest -D NightlyConfigure
ctest -D NightlyBuild
ctest -D NightlySubmit
ctest -D NightlyTest
ctest -D NightlyCoverage
ctest -D NightlySubmit
ctest -D NightlyMemCheck
ctest -D NightlySubmit

Finally CTest extends functionality of Dart by adding:

  1. HTTP submission support
  2. Customization of the testing by providing:
    • Custom build error/warning regular expressions
    • Ability to suppress some tests from being tested or memory checked and ability to run subset of tests
    • Ability to run commands before and after tests are run
  3. Ability to run whole testing process described in a single script

For an example of how CTest can run the whole testing process described in a single script, look at how CMake dashboards are created with the ctest -S script.

More information about CTest can be found in Mastering CMake.