Difference between revisions of "CMake/Testing With CTest"

From KitwarePublic
Jump to navigationJump to search
(Initial writeup about CTest)
 
(Cleanup, and add intro/conclusion)
Line 1: Line 1:
 +
==Introduction==
 +
 +
CTest is a testing tool distributed as a part of CMake. It can be used to
 +
automate updating (using CVS for example), configuring, building, testing,
 +
performing memory checking, performing coverage, and submitting results to
 +
[http://public.kitware.com/Dart Dart] dashboard system. This tutorial will
 +
introduce the testing with CTest.
 +
 
==Simple Testing==
 
==Simple Testing==
  
CMake has support for adding tests in the project. To do that, insert the following command in the CMakeLists.txt file:
+
The easiest way to create CTest input files is using CMake. It has support for
 +
adding tests in the project. To do that, insert the following command in the
 +
CMakeLists.txt file:
  
 
  ENABLE_TESTING()
 
  ENABLE_TESTING()
Line 9: Line 19:
 
  ADD_TEST(SimpleTest ${EXECUTABLE_OUTPUT_PATH}/SimpleTest Hello)
 
  ADD_TEST(SimpleTest ${EXECUTABLE_OUTPUT_PATH}/SimpleTest Hello)
  
After building the project, you should be able to do:
+
After building the project, you should be able to run CTest on the project.
  
1. Unix
+
1. When using Makefile generators, such as Unix Makefiles, Borland Makefiles,
 +
or NMake Makefiles, the CTest can simply be run by running:
  
 
  make test
 
  make test
  
2. On GUI development environments such as Visual Studio rebuild the target RUN_TESTS
+
2. On GUI development environments such as Visual Studio rebuilding the target
 +
RUN_TESTS will run tests on the project.
  
For more information about ENABLE_TESTING and ADD_TEST, Look at [http://www.cmake.org/HTML/Documentation.html CMake Documentation] or run:
+
For more information about ENABLE_TESTING and ADD_TEST, Look at
 +
[http://www.cmake.org/HTML/Documentation.html CMake Documentation] or run:
  
 
  cmake --help
 
  cmake --help
Line 30: Line 43:
 
==Dashboards==
 
==Dashboards==
  
Testing Dashboards are web pages that display overview of the project testing. The testing clients configure and build the
+
Testing Dashboards are web pages that display overview of the project testing.
project, as well as run some number of tests. The results of these operations are then submitted to the central server, which
+
The testing clients update, configure, and build the project, as well as run
prepares the overview pages. Examples of Testing Dashboards are:
+
some number of tests. The results of these operations are then submitted to the
[http://www.vtk.org/Testing/Dashboard/MostRecentResults-Nightly/Dashboard.html VTK], [http://www.itk.org/Testing/Dashboard/MostRecentResults-Nightly/Dashboard.html ITK], and [http://www.paraview.org/Testing/Dashboard/MostRecentResults-Nightly/Dashboard.html ParaView].
+
central server, which prepares the overview pages. Examples of Testing
 +
Dashboards are:
 +
[http://www.vtk.org/Testing/Dashboard/MostRecentResults-Nightly/Dashboard.html
 +
VTK],
 +
[http://www.itk.org/Testing/Dashboard/MostRecentResults-Nightly/Dashboard.html
 +
ITK], and
 +
[http://www.paraview.org/Testing/Dashboard/MostRecentResults-Nightly/Dashboard.html
 +
ParaView].
  
To enable submission to Dart Testing Dashboard, include the following in CMakeLists.txt:
+
There are three types of dashboard submissions. Experimental submission is
 +
submission of the current state of the project. It can be performed at any time
 +
and will appear on the dashboard on the next roll-up. Nightly submission is
 +
similar to experimental, except that the extra update will be performed using
 +
the last nighty time. This way all nightly dashboard submissions correspond to
 +
the state of the project at the same point in time. Finally Continuous
 +
submissions are similar to Experimental submissions, except that an update is
 +
performed and if any files were modified, the full dashboard testing is
 +
performed.
 +
 
 +
===Dashboard Preparation Using CMake===
 +
 
 +
To enable submission to Dart Testing Dashboard, include the following in
 +
CMakeLists.txt:
  
 
  INCLUDE(Dart)
 
  INCLUDE(Dart)
  
By default the settings will be to submit to [http://www.kitware.com Kitware's] [http://public.kitware.com/Public/Dashboard/MostRecentResults-Nightly/Dashboard.html Public Dashboard]. In order to submit
+
By default the settings will be to submit to [http://www.kitware.com Kitware's]
 +
[http://public.kitware.com/Public/Dashboard/MostRecentResults-Nightly/Dashboard.html
 +
Public Dashboard]. In order to submit
 
to some other dashboard, create file "DartConfig.cmake" in the toplevel source directory and set the dashboard preferences.
 
to some other dashboard, create file "DartConfig.cmake" in the toplevel source directory and set the dashboard preferences.
  
Line 64: Line 99:
 
</pre>
 
</pre>
  
This will submit testing results to the [http://www.vtk.org/Testing/Dashboard/MostRecentResults-Nightly/Dashboard.html VTK dashboard].
+
This will submit testing results to the
 +
[http://www.vtk.org/Testing/Dashboard/MostRecentResults-Nightly/Dashboard.html
 +
VTK dashboard].
  
 
===CTest - Client for Dart Dashboard Server===
 
===CTest - Client for Dart Dashboard Server===
  
CTest is a testing program distributed as a part of CMake. It can perform simple task of running a set of tests but it can also generate
+
CTest started as a simple tool to run some number of tests in the project, but
and submit Dart compatible Dashboard results. The good thing about CTest is that if you use CMake, you already have CTest. Detailed description
+
has evolved in the full ''Dart'' compatible client. It can perform simple task
of CTest option can be seen by running:
+
of running a set of tests but it can also generate and submit Dart compatible
 +
Dashboard results. The good thing about CTest is that it is self sustained. All
 +
you need to do testing is CTest. Also, if you use CMake, you already have
 +
CTest. Detailed description of CTest option can be seen by running:
  
 
  ctest --help
 
  ctest --help
Line 77: Line 117:
  
 
  ctest --help-full
 
  ctest --help-full
 +
 +
A simple way to submit Experimental dashboard is:
 +
 +
ctest -D Experimental
 +
 +
This will configure project, build project and check for any warnings or
 +
errors, run tests if any tests are specified, run coverage if any coverage
 +
files are specified, and submit to the specified ''Dart'' server.
  
 
To convert existing Dart Client run from the project, find lines like:
 
To convert existing Dart Client run from the project, find lines like:
Line 114: Line 162:
 
# Ability to run whole testing process described in a single script
 
# Ability to run whole testing process described in a single script
  
For an example of how CTest can run the whole testing process described in a single script, look at how CMake dashboards are created with the [http://www.cmake.org/HTML/TestingSetup.html ctest -S script].
+
For an example of how CTest can run the whole testing process described in a
 +
single script, look at how CMake dashboards are created with the
 +
[http://www.cmake.org/HTML/TestingSetup.html CTest -S script].
 +
 
 +
More information about CTest can be found in
 +
[http://www.kitware.com/products/cmakebook.html Mastering CMake].
 +
 
 +
==Conclusion==
  
More information about CTest can be found in [http://www.kitware.com/products/cmakebook.html Mastering CMake].
+
Performing tests on the project is a great software development practice and can result in significant improvement on the quality of the project. CTest provides a simple and reliably way of performing nightly, continuous, and experimental tests.

Revision as of 11:45, 16 October 2004

Introduction

CTest is a testing tool distributed as a part of CMake. It can be used to automate updating (using CVS for example), configuring, building, testing, performing memory checking, performing coverage, and submitting results to Dart dashboard system. This tutorial will introduce the testing with CTest.

Simple Testing

The easiest way to create CTest input files is using CMake. It has support for adding tests in the project. To do that, insert the following command in the CMakeLists.txt file:

ENABLE_TESTING()

From that point on, you can add tests in the project using ADD_TEST command:

ADD_TEST(SimpleTest ${EXECUTABLE_OUTPUT_PATH}/SimpleTest Hello)

After building the project, you should be able to run CTest on the project.

1. When using Makefile generators, such as Unix Makefiles, Borland Makefiles, or NMake Makefiles, the CTest can simply be run by running:

make test

2. On GUI development environments such as Visual Studio rebuilding the target RUN_TESTS will run tests on the project.

For more information about ENABLE_TESTING and ADD_TEST, Look at CMake Documentation or run:

cmake --help
cmake --help-full
cmake --help-command-list

or

cmake --help-command ENABLE_TESTING
cmake --help-command ADD_TEST

Dashboards

Testing Dashboards are web pages that display overview of the project testing. The testing clients update, configure, and build the project, as well as run some number of tests. The results of these operations are then submitted to the central server, which prepares the overview pages. Examples of Testing Dashboards are: [http://www.vtk.org/Testing/Dashboard/MostRecentResults-Nightly/Dashboard.html VTK], [http://www.itk.org/Testing/Dashboard/MostRecentResults-Nightly/Dashboard.html ITK], and [http://www.paraview.org/Testing/Dashboard/MostRecentResults-Nightly/Dashboard.html ParaView].

There are three types of dashboard submissions. Experimental submission is submission of the current state of the project. It can be performed at any time and will appear on the dashboard on the next roll-up. Nightly submission is similar to experimental, except that the extra update will be performed using the last nighty time. This way all nightly dashboard submissions correspond to the state of the project at the same point in time. Finally Continuous submissions are similar to Experimental submissions, except that an update is performed and if any files were modified, the full dashboard testing is performed.

Dashboard Preparation Using CMake

To enable submission to Dart Testing Dashboard, include the following in CMakeLists.txt:

INCLUDE(Dart)

By default the settings will be to submit to Kitware's [http://public.kitware.com/Public/Dashboard/MostRecentResults-Nightly/Dashboard.html Public Dashboard]. In order to submit to some other dashboard, create file "DartConfig.cmake" in the toplevel source directory and set the dashboard preferences.

Example of this file is:

# Dashboard is opened for submissions for a 24 hour period starting at
# the specified NIGHLY_START_TIME. Time is specified in 24 hour format.
SET (NIGHTLY_START_TIME "23:00:00 EDT")

# Dart server to submit results (used by client)
IF(DROP_METHOD MATCHES http)
  SET (DROP_SITE "public.kitware.com")
  SET (DROP_LOCATION "/cgi-bin/HTTPUploadDartFile.cgi")
ELSE(DROP_METHOD MATCHES http)
  SET (DROP_SITE "public.kitware.com")
  SET (DROP_LOCATION "/incoming")
  SET (DROP_SITE_USER "ftpuser")
  SET (DROP_SITE_PASSWORD "public")
ENDIF(DROP_METHOD MATCHES http)

SET (TRIGGER_SITE 
       "http://${DROP_SITE}/cgi-bin/Submit-vtk-TestingResults.pl")

This will submit testing results to the [http://www.vtk.org/Testing/Dashboard/MostRecentResults-Nightly/Dashboard.html VTK dashboard].

CTest - Client for Dart Dashboard Server

CTest started as a simple tool to run some number of tests in the project, but has evolved in the full Dart compatible client. It can perform simple task of running a set of tests but it can also generate and submit Dart compatible Dashboard results. The good thing about CTest is that it is self sustained. All you need to do testing is CTest. Also, if you use CMake, you already have CTest. Detailed description of CTest option can be seen by running:

ctest --help

or

ctest --help-full

A simple way to submit Experimental dashboard is:

ctest -D Experimental

This will configure project, build project and check for any warnings or errors, run tests if any tests are specified, run coverage if any coverage files are specified, and submit to the specified Dart server.

To convert existing Dart Client run from the project, find lines like:

 cd ProjectNightlyBuildDirectory
 tclsh /location/of/Dart/Source/Client/DashboardManager.tcl DartConfiguration.tcl \
	Nightly Start Update Configure Build Test Submit

and convert them to CTest style:

cd ProjectNightlyBuildDirectory
ctest -D Nightly

Dashboard can be also generated in stages. This way partial testing results can be submitted and seen before long operations are completed:

cd ProjectNightlyBuildDirectory
ctest -D NightlyStart
ctest -D NightlyUpdate
ctest -D NightlyConfigure
ctest -D NightlyBuild
ctest -D NightlySubmit
ctest -D NightlyTest
ctest -D NightlyCoverage
ctest -D NightlySubmit
ctest -D NightlyMemCheck
ctest -D NightlySubmit

Finally CTest extends functionality of Dart by adding:

  1. HTTP submission support
  2. Customization of the testing by providing:
    • Custom build error/warning regular expressions
    • Ability to suppress some tests from being tested or memory checked and ability to run subset of tests
    • Ability to run commands before and after tests are run
  3. Ability to run whole testing process described in a single script

For an example of how CTest can run the whole testing process described in a single script, look at how CMake dashboards are created with the CTest -S script.

More information about CTest can be found in Mastering CMake.

Conclusion

Performing tests on the project is a great software development practice and can result in significant improvement on the quality of the project. CTest provides a simple and reliably way of performing nightly, continuous, and experimental tests.