ITK/Testing

| Testing plays a vital role in the ITK development process. We use Dart Dashboard Server to display test results.

Is ITK tested regularly?

ITK is tested throughout the day on machines distributed around the world. You can view the results at:

http://www.itk.org/Testing/Dashboard/MostRecentResults-Nightly/Dashboard.html

or

http://www.cdash.org/CDash/index.php?project=Insight

ITK uses Dart and CDash to perform builds, run tests and generate dashboards. You can find more information about Dart at http://www.itk.org/Dart/. You can find more information about CDash at http://www.cdash.org.

What is the testing schedule?

Dart opens a daily dashboard at 9:00PM Eastern Daylight Time and keeps that dashbooard open for 24 hours. Build/Test results submitted as "nightlies" will appear on the currently opened dashboard. Each test site can select a convenient time to run tests.

Fresh dashboards are generated every hour, on the hour. They can also be generated on demand using the Rollup button on the dashboard. This button should be used sparingly since dashboard rollups can be time consuming.

In addition to nightly testing, several sites run Continuous build/tests. These sites monitor cvs repository activity and begin a build/test sequence if there are updates to the repository. If a compile error is detected, the Dashboard server sends email to the "offending" party.

What kind of testing is performed?

ITK has three types of tests:

1. Unit Tests. Also called white box tests. These tests reside in the Testing/Code/XXX directories. There are meant to exercise a class's methods and verify that the code compiles, links and produces expected results. These tests do not need to produce results that are useful to a user. If a test returns a 0 status, it passes. A test fails if it returns a non-zero status.
2. Blackbox Tests. These tests are automatically (or semi-automatically) generated. For example, each Code directory has a test called itkXXXHeaderTest. This test is automatically created and has a "#include" for each file in the directory.
3. Regression Tests. These tests generate results and compare them with a known result. The purpose of regression testing is to detect changes in the output of the system.

What is a continuous build and how can I contribute one?

Continuous builds run throughout the day and report their output in the Continuous Builds portion of the dashboard. The most recent build is at the top of the list. The builds run periodically and check the status of the repository. If a file is added or changed since the last build, the working repository of the continuous build is updated, the code compiled and tests executed. Optional e-mail norifications can be sent if there are errors, warmings or test failures.

You can set up a continuous build. This entails writing some small shell scripts and setting up a scheduler (Scheduler for Windows, cron for Linux/Unix). You will also need to install tcl, checkout Dart and configure your build for testing.

For Windows 2000, here are three batch files that can be modified for your own environment.

AllowInsightContinuous.bat
BuildInsightContinuous.bat
PreventInsightContinuous.bat

The batch files use a lock to prevent/allow the continuous

rem ************* AllowInsightContinuous.bat
rem Allow the Continuous to run
rem
c:
cd c:\lorensen\InsightContinuous\Insight-VC6
if EXIST continuous.lock (
  del continuous.lock
)
cd ..
rem ************* BuildInsightContinuous.bat
rem DO a C++ Build Continuously
rem
c:
rem Setup proxy it behind a firewall
set HTTP_PROXY=webcache
set HTTP_PROXY_PORT=8080
rem
rem Setup the path
rem
set PATH=c:/lorensen/InsightContinuous/Insight-VC6/bin/Release;%PATH%
rem
rem Tell cvs where .cvspass resides
set HOME=c:/lorensen
cd c:\lorensen\InsightContinuous\Insight-VC6
if EXIST continuous.lock (
  exit /B
) else (
  date /T > continuous.lock
  time /T >> continuous.lock
  cd c:\lorensen\InsightContinuous\Insight-VC6
 tclsh83 c:/lorensen/Dart/Source/Client/DashboardManager.tcl DartConfiguration.tcl Continuous Start Update Configure Build Test Submit
 cd c:\lorensen\InsightContinuous\Insight-VC6
 del continuous.lock
 cd ..
)
rem ***************** PreventInsightContinuous.bat
rem Prevent the Continuous
rem
c:
cd c:\lorensen\InsightContinuous\Insight-VC6
if EXIST continuous.lock (
  exit /B
) else (
  echo "Prevent continuous for nightly" > continuous.lock
)
cd ..

Now you need to establish a schedule for the three batch jobs. Use the "Scheduled Tasks" on the "Control Panel" to access the Schedule Task Wizard. The following schedule runs the continuous every 10 minutes from 6:05AM until 12:00AM. The Allow and Prevent scripts turn the build ON/OFF at the appropriate time. Schedule each batch file as follows:

AllowInsightContinuous.bat - Daily, Start 6:00AM
BuildInsightContinuous.bat - Daily, Start 6:05AM, Advanced: Repeat Task every 10 minutes, duration 17 hours and 55 minutes
PreventContinuous.bat - Daily, Start 12:00AM

Supporting ITK development process can be as easy as contributing testing results from your platform.

• Submit testing results using CTest

---

ITK: [Welcome | Site Map]