BSD.REGRESS.MK(5)

NAME
bsd.regress.mk - regression test master Makefile fragment

SYNOPSIS
.include <bsd.regress.mk>

DESCRIPTION
bsd.regress.mk holds the standard routines used by the source regression tests. Some variables and targets are for its internal use only. The rest are documented here.

Since this file also includes bsd.prog.mk, all of the usual OpenBSD Makefile variables may be used to build the regression test programs.

TARGETS

depend
    Build any dependencies required to carry out the current set of regression tests.

regress
    Executes all of the regression targets defined in the Makefile. If one of the tests fails, the line "FAILED" is printed to the standard output. By default, execution continues with the next test and, after running all the tests, make regress exits with an exit status of 0 even if some of the tests failed.

run-regress-*
    Runs an individual regression test. If the exit status of the program indicates an error or timeout, then a failure is logged, otherwise the test is marked as a success.

VARIABLES

REGRESS_FAIL_EARLY
    If this variable is set to anything but “no”, the regress target will abort as soon as a test fails. This variable is intended to be set at runtime in the environment or in /etc/mk.conf, but not in the test's Makefile itself.

REGRESS_LOG
    Points to the fully-qualified path of a file to which regression results are appended. Defaults to /dev/null.

REGRESS_MAXTIME
    Maximum limit of CPU seconds to spend on the regression test. Exceeding this time will result in a failure being logged.

REGRESS_ROOT_TARGETS
    Targets for which root access is required to run the test. The SUDO variable is invoked for these targets. See also SUDO.

REGRESS_SKIP_SLOW
    If this variable is not empty, skip over all the regression tests which have been marked as being 'slow' using the REGRESS_SLOW_TARGETS variable.

REGRESS_SLOW_TARGETS
    Targets which are defined as 'slow'. All of these tests can be skipped by setting the REGRESS_SKIP_SLOW variable.

REGRESS_TARGETS
    Targets which are invoked to run the set of regression tests for this Makefile. Defaults to run-regress-${PROG}.

SUDO
    Location of a command used to switch to root for certain test targets which require it. See doas(1).

GUIDELINES
If an individual test passes, make testname should return with an exit status of 0. If it fails, it should return with a non-zero exit status.

If a test cannot be executed because a package is not installed or some environment variable is not set, make testname should print "SKIPPED" to stdout and exit with status 0. To skip everything, implement the regress target with a command that prints "SKIPPED".

Tests should not fail because an intended feature has not been implemented yet. To avoid such false failures, a test should show the reason, print "DISABLED" to stdout and exit with status 0.

Tests must be able to run with an obj directory. In case the test is not linked to the build or the tester forgot to run make obj before, this directory or symlink may not exist. Then the test should run anyway.

Test are executed with make regress, but running make all or make should have the same effect.

Tests should use the binaries installed and the kernel running on the local system. They may use environment variables to test alternative binaries or remote kernels running on other machines. In some cases a test may need binaries or libraries or object files to be present in /usr/obj/ that exist only after make build was run in /usr/src/. The test must not assume that they have already been built, but should run make in the appropriate source directory as a dependency. The /usr/src/ tree can be found with a relative path or with the BSDSRCDIR variable.

SEE ALSO
bsd.port.mk(5)

HISTORY
The regression system originally came from NetBSD, with many tests added by OpenBSD since. The current Makefile framework was written by Artur Grabowski and Marc Espie for OpenBSD 3.1.

BUGS AND LIMITATIONS
The build system is unable to distinguish between timeouts due to REGRESS_MAXTIME being exceeded, or a genuine failure occurring.