Next: Rebuilding, Previous: Dist, Up: Top
Automake supports two forms of test suites.
If the variable TESTS is defined, its value is taken to be a
list of programs or scripts to run in order to do the testing.
Programs needing data files should look for them in srcdir
(which is both an environment variable and a make variable) so they
work when building in a separate directory (see Build Directories), and in
particular for the distcheck rule (see Dist).
The number of failures will be printed at the end of the run. If a given test program exits with a status of 77, then its result is ignored in the final count. This feature allows non-portable tests to be ignored in environments where they don't make sense.
The variable TESTS_ENVIRONMENT can be used to set environment
variables for the test run; the environment variable srcdir is
set in the rule. If all your test programs are scripts, you can also
set TESTS_ENVIRONMENT to an invocation of the shell (e.g.
‘$(SHELL) -x’ can be useful for debugging the tests), or any other
interpreter. For instance the following setup is used by the Automake
package to run four tests in Perl.
TESTS_ENVIRONMENT = $(PERL) -Mstrict -I $(top_srcdir)/lib -w
TESTS = Condition.pl DisjConditions.pl Version.pl Wrap.pl
You may define the variable XFAIL_TESTS to a list of tests
(usually a subset of TESTS) that are expected to fail. This will
reverse the result of those tests.
Automake ensures that each file listed in TESTS is built before
any tests are run; you can list both source and derived programs (or
scripts) in TESTS; the generated rule will look both in
srcdir and .. For instance, you might want to run a C
program as a test. To do this you would list its name in TESTS
and also in check_PROGRAMS, and then specify it as you would
any other program.
Programs listed in check_PROGRAMS (and check_LIBRARIES,
check_LTLIBRARIES...) are only built during make check,
not during make all. You should list there any program needed
by your tests that does not need to be built by make all. Note
that check_PROGRAMS are not automatically added to
TESTS because check_PROGRAMS usually lists programs used
by the tests, not the tests themselves. Of course you can set
TESTS = $(check_PROGRAMS) if all your programs are test cases.
If dejagnu appears in
AUTOMAKE_OPTIONS, then a dejagnu-based test suite is
assumed. The variable DEJATOOL is a list of names that are
passed, one at a time, as the --tool argument to
runtest invocations; it defaults to the name of the package.
The variable RUNTESTDEFAULTFLAGS holds the --tool and
--srcdir flags that are passed to dejagnu by default; this can be
overridden if necessary.
The variables EXPECT and RUNTEST can
also be overridden to provide project-specific values. For instance,
you will need to do this if you are testing a compiler toolchain,
because the default values do not take into account host and target
names.
The contents of the variable RUNTESTFLAGS are passed to the
runtest invocation. This is considered a “user variable”
(see User Variables). If you need to set runtest flags in
Makefile.am, you can use AM_RUNTESTFLAGS instead.
Automake will generate rules to create a local site.exp file,
defining various variables detected by configure. This file
is automatically read by DejaGnu. It is OK for the user of a package
to edit this file in order to tune the test suite. However this is
not the place where the test suite author should define new variables:
this should be done elsewhere in the real test suite code.
Especially, site.exp should not be distributed.
For more information regarding DejaGnu test suites, see Top.
In either case, the testing is done via ‘make check’.
The installcheck target is available to the user as a way to
run any tests after the package has been installed. You can add tests
to this by writing an installcheck-local rule.