CMake Generating Testing Files

Introduction

CTest is a powerful testing client compatible with Dart Dashboard system. It is distributed as a part of CMake but can be used with or without CMake. This tutorial explains some background behind the testing with CTest and the process of generating CTest testing files.

CTest Overview

CTest currently uses the old Dart input configuration file. New input format is being developed, but the old format will be valid in the future. For legacy purposes, the input files are called DartConfiguration.tcl, and DartTestfile.txt. DartConfiguration.tcl describes the system on which tests are performed, while DartTestfile.txt provides the list of tests.

CTest reads DartConfiguration.tcl and depending on the options provided it performs certain stage of testing.

Description Of The System

Even though the extension of DartConfiguration.tcl file is .tcl, it is not TCL file. It is simply list of key/value pairs. Line starting with # are considered comments and are ignored. This file has to appear at the toplevel directory of the project's build directory.

Example DartConfiguration.tcl file is:

SourceDirectory: /path/to/project/source/directory
BuildDirectory: /path/to/project/build/directory
CVSUpdateOptions: -d -A -P
Site: my-system-name
BuildName: build-name
DropSite: public.kitware.com
DropLocation: /cgi-bin/HTTPUploadDartFile.cgi 
DropMethod: http
TriggerSite: http://public.kitware.com/cgi-bin/Submit-Random-TestingResults.pl
NightlyStartTime: 21:00:00 EDT
ConfigureCommand: "/path/to/source/directory/Project/configure"
MakeCommand: /usr/bin/make -i
CVSCommand: /usr/bin/cvs
CoverageCommand: /usr/bin/gcov
TimeOut: 1500

Most entries are pretty self explanatory. Site refers to the system name. Example Site would be andoria.kitwarein.com. BuildName is a breif description of the system and the project. Example would be FLTK-Linux-g++334 which would be for project FLTK build on Linux system running g++ version 3.3.4.

NightlyStartTime is the time when Nightly dashboard starts. This is useful when doing Nightly dashboard. The Nightly dashboard will update the source directory to the state at given time.

TimeOut is a maximum time the test will be allowed to run. Once test time expires, the test will be interrupted and the status will be Timeout.

DropSite, DropLocation, DropMethod, and TriggerSite describe the location where test results are submitted. The location specified in the example is the Public Dashboard provided as a service by Kitware.

Description of Tests

For legacy purposes and for compatibility with Dart, the test description file is called DartTestfile.txt. It is parsed by full CMake parser, but the CMake commands that are available are SUBDIRS and ADD_TEST. Example DartTestfile.txt is:

ADD_TEST(testProcess-1 "/my/project/testProcess" "1")
ADD_TEST(testProcess-2 "/my/project/testProcess" "2")
ADD_TEST(testProcess-3 "/my/project/testProcess" "3")
ADD_TEST(testProcess-4 "/my/project/testProcess" "4")
ADD_TEST(testProcess-5 "/my/project/testProcess" "5")
ADD_TEST(testProcess-6 "/my/project/testProcess" "6")

Each of the ADD_TEST adds the test for CTest. The first argument is the unique name of test. It should be unique, so that it can be differentiated among all results. The second argument is the executable run as a part of the test. This is followed by arbitrary number of arguments for the executable. Executable can be build as a part of the project, or system executable can be used. For example, to test a Python feature:

ADD_TEST(testTkInter "/usr/bin/python" "-c" "import Tkinter")

It is a good practice to specify full path to the executable name, since that way CTest will always pick the desired executable. For example, let say the test executable name is python but there is a python installed on the system, the CTest may pick either the python executable built by our build process or use the installed one.

Group Tests in Subdirectories

Project tests are usually organized into some structure based on the part of the project they test. For example, if the project contains some common code, IO code, processing code, and GUI code, there may be group of tests for each subsection. Common way of organizing this would be in subdirectories. For example, let say the project has the following directory structure:

/Project
        /Common
        /IO
        /Processing
        /GUI

It is fairly easy to reproduce this structure for testing:

/Project
        /Testing
                /Common
                /IO
                /Processing
                /GUI

CTest will not automatically search all subdirectories and find all DartTestfile.txt. The project needs to tell it where the tests are. This can be achieved using SUBDIRS command in DartTestfile.txt. An example use of SUBDIRS command is:

SUBDIRS(Testing/Common)
SUBDIRS(Testing/IO)
SUBDIRS(Testing/Processing Testing/GUI)

The arguments to the SUBDIRS command are subdirectories containing the tests. If this file is placed in the top of our project, CTest will search for files DartTestfile.txt inside directories /.../Project/Testing/Common, /.../Project/Testing/IO, and so on to /.../Project/Testing/GUI.

Generating Testing Files

Testing files DartConfiguration.tcl and all DartTestfile.txt' must be generated in the same tree as the CTest is run. In most cases this is the build tree of the project. Since the testing files contain build and system specific information, they should be generated as a part of testing process and not permanently part of source code.

Using CMake

CMake has a automated way of generating CTest input files. There are three pieces to performing testing:

 ENABLE_TESTING()

Enables testing for the project. This command will gather all tests and produce DartTestfile.txt files with apropriate ADD_TEST and SUBDIRS commands.

 INCLUDE(Dart)

Including Dart CMake module will generate default DartConfiguration.tcl file, which will contain information about updating, configuring, and building the project.

 ADD_TEST(test1 ${EXECUTABLE_OUTPUT_PATH}/test1)

Describes test that will be put in DartTestfile.txt. The syntax of ADD_TEST on CMake level is the same as once it is written in DartTestfile.txt.

More details about how to generate test files with CMake is described in Testing With CTest tutorial.

Without Using CMake

CTest is distributed together with CMake, but it can be used independently. The DartConfiguration.tcl file can be easily generated using autoconf or any other tool, or even written by hand. Whatever the method of generating that file, the author has to make sure all the necessary information is written in it. Same goes for DartTestfile.txt files.

Conclusion

Though we consider CMake to be a great build configuration tool, CTest can be used without CMake. Because of its simplicity, it can be added to any project without changing the rest of the project development. Using a small number of configuration files, CTest can easily automate most if not all of the commonly used testing procedures.

---

CMake: [Welcome | Site Map]