@hackage shelltestrunner0.8

Run a command-line program through "shell tests" defined in one or more test files. Each test specifies command-line arguments, some input, and expected output, stderr output and/or exit status. We use test-framework's test runner, so can run tests in parallel. shelltestrunner was inspired by the tests in John Wiegley's ledger project.

Usage:

shelltest [FLAG] [TESTFILES|TESTDIRS]


  -? --help[=FORMAT]           Show usage information (optional format)
  -V --version                 Show version information
  -v --verbose                 Higher verbosity
  -q --quiet                   Lower verbosity
  -d --debug                   show debug messages
     --debug-parse             show parsing debug messages and stop
     --execdir                 run tests in same directory as test file
     --extension=EXT           extension of test files when dirs specified (default=.test)
  -i --implicit=none|exit|all  provide implicit tests (default=exit)
  -w --with=EXECUTABLE         alternate executable, replaces the first word of test commands
     =OTHER FLAGS              any other flags are passed to test runner

A test file contains one or more shell tests, which look like this:

 # optional comment lines
 a one-line shell command to be tested
 <<<
 stdin lines
 >>> [/regexp to match in stdout/]
 [or expected stdout lines
 >>>2 [/regexp to match in stderr/]
 [or expected stderr lines]
 >>>= expected exit status or /regexp/

The command line is required; all other fields are optional.
The expected stdout (>>>) and expected stderr (>>>2) fields can have either
a regular expression match pattern, in which case the test passes if the
output is matched, or 0 or more data lines, in which case the output
must match these exactly. The expected exit status (>>>=) field can have
either a numeric exit code or a /regexp/. A ! preceding a /regexp/ or exit
code negates the match. The regular expression syntax is that of the
pcre-light library with the dotall flag.

By default there is an implicit test for exit status=0, but no implicit test
for stdout or stderr.  You can change this with -i/--implicit-tests.

The command runs in your current directory unless you use --execdir.
You can use --with/-w to replace the first word of command lines
(everything up to the first space) with something else, eg to test a
different version of your program. To prevent this, start the command line
with a space.

Any unrecognised options will be passed through to test-framework's runner.
You may be able to get a big speedup by running tests in parallel: try -j8.