12.5 Adding a Test Suite

Automake has very flexible support for automated test-suites within a project distribution, which are discussed more fully in the Automake manual. I have added a simple shell script based testing facility to Sic using this support -- this kind of testing mechanism is perfectly adequate for command line projects. The tests themselves simply feed prescribed input to the uninstalled sic interpreter and compare the actual output with what is expected.

Here is one of the test scripts:

## -*- sh -*- ## incomplete.test -- Test incomplete command handling # Common definitions if test -z "$srcdir"; then srcdir=echo "$0" | sed s,[^/]*$,,' test "$srcdir" = "$0" && srcdir=. test -z "$srcdir" && srcdir=. test "${VERBOSE+set}" != set && VERBOSE=1 fi . $srcdir/defs # this is the test script cat <<\EOF > in.sic echo "1 2 3" EOF # this is the output we should expect to see cat <<\EOF >ok 1 2 3 EOF cat <<\EOF >errok EOF # Run the test saving stderr to a file, and showing stdout # if VERBOSE == 1 $RUNSIC in.sic 2> err | tee -i out >&2 # Test against expected output if ${CMP} -s out ok; then : else echo "ok:" >&2 cat ok >&2 exit 1 fi # Munge error output to remove leading directories, `lt-' or # trailing `.exe' sed -e "s,^[^:]*[lt-]*sic[.ex]*:,sic:," err >sederr && mv sederr err # Show stderr if doesnt match expected output if VERBOSE == 1 if "$CMP" -s err errok; then : else echo "err:" >&2 cat err >&2 echo "errok:" >&2 cat errok >&2 exit 1 fi

The tricky part of this script is the first part which discovers the location of (and loads) `$srcdir/defs'. It is a little convoluted because it needs to work if the user has compiled the project in a separate build tree -- in which case the `defs' file is in a separate source tree and not in the actual directory in which the test is executed.

The `defs' file allows me to factor out the common definitions from each of the test files so that it can be maintained once in a single file that is read by all of the tests:

#! /bin/sh # Make sure srcdir is an absolute path. Supply the variable # if it does not exist. We want to be able to run the tests # stand-alone!! # srcdir=${srcdir-.} if test ! -d $srcdir ; then echo "defs: installation error" 1>&2 exit 1 fi # IF the source directory is a Unix or a DOS root directory, ... # case "$srcdir" in /* | [A-Za-z]:\\*) ;; *) srcdir=`\cd $srcdir && pwd` ;; esac case "$top_builddir" in /* | [A-Za-z]:\\*) ;; *) top_builddir=`\cd ${top_builddir-..} && pwd` ;; esac progname=`echo "$0" | sed 's,^.*/,,'` testname=`echo "$progname" | sed 's,-.*$,,'` testsubdir=${testsubdir-testSubDir} SIC_MODULE_PATH=$top_builddir/modules export SIC_MODULE_PATH # User can set VERBOSE to prevent output redirection case x$VERBOSE in xNO | xno | x0 | x) exec > /dev/null 2>&1 ;; esac rm -rf $testsubdir > /dev/null 2>&1 mkdir $testsubdir cd $testsubdir \ || { echo "Cannot make or change into $testsubdir"; exit 1; } echo "=== Running test $progname" CMP="${CMP-cmp}" RUNSIC="${top_builddir}/src/sic"

Having written a few more test scripts, and made sure that they are working by running them from the command line, all that remains is to write a suitable `Makefile.am' so that automake can run the test suite automatically.

## Makefile.am -- Process this file with automake to produce Makefile.in EXTRA_DIST = defs $(TESTS) MAINTAINERCLEANFILES = Makefile.in testsubdir = testSubDir TESTS_ENVIRONMENT = top_builddir=$(top_builddir) TESTS = \ empty-eval.test \ empty-eval-2.test \ empty-eval-3.test \ incomplete.test \ multicmd.test distclean-local: -rm -rf $(testsubdir)

I have used the `testsubdir' macro to run the tests in their own subdirectory so that the directory containing the actual test scripts is not polluted with lots of fallout files generated by running the tests. For completeness I have used a hook target(26) to remove this subdirectory when the user types:

$ make distclean ... rm -rf testSubDir ...

Adding more tests is accomplished by creating a new test script and adding it to the list in noinst_SCRIPTS. Remembering to add the new `tests' subdirectory to `configure.in' and the top-level `Makefile.am', and reconfiguring the project to propagate the changes into the various generated files, I can run the whole test suite from the top directory with:

$ make check

It is often useful run tests in isolation, either when developing new tests, or to examine more closely why a test has failed unexpectedly. Having set this test suite up as I did, individual tests can be executed with:

$ VERBOSE=1 make check TESTS=incomplete.test make check-TESTS make[1]: Entering directory /tmp/sic/tests === Running test incomplete.test 1 2 3 PASS: incomplete.test ================== All 1 tests passed ================== make[1]: Leaving directory /tmp/sic/tests $ ls testSubDir/ err errok in.sic ok out

The `testSubDir' subdirectory now contains the expected and actual output from that particular test for both `stdout' and `stderr', and the input file which generated the actual output. Had the test failed, I would be able to look at these files to decide whether there is a bug in the program or simply a bug in the test script. Being able to examine individual tests like this is invaluable, especially when the test suite becomes very large -- because you will, naturally, add tests every time you add features to a project or find and fix a bug.

Another alternative to the pure shell based test mechanism I have presented here is the Autotest facility by Fran@,cois Pinard, as used in Autoconf after release 2.13.

Later in 20. A Complex GNU Autotools Project, the Sic project will be revisited to take advantage of some of the more advanced features of GNU Autotools. But first these advanced features will be discussed in the next several chapters -- starting, in the next chapter, with a discussion of how GNU Autotools can help you to make a tarred distribution of your own projects.