9 B<tesh> [I<options>]... I<testsuite>
13 Tesh is the testing shell, a specialized shell for running tests. It
14 provides the specified input to the tested commands, and check that
15 they produce the expected output and return the expected value.
20 --cd some/directory : ask tesh to switch the working directory before
22 --setenv var=value : set a specific environment variable
23 --cfg arg : add parameter --cfg=arg to each command line
24 --log arg : add parameter --log=arg to each command line
25 --ignore-jenkins : ignore all cruft generated on SimGrid
26 continuous integration servers
29 =head1 TEST SUITE FILE SYTAX
31 A test suite is composed of one or several I<command blocks> separated
32 by empty lines, each of them being composed of a command to run, its
33 input text and the expected output.
35 The first char of each line specifies the type of line according to
36 the following list. The second char of each line is ignored.
38 `$' command to run in foreground
39 `&' command to run in background
41 `<' input to pass to the command
42 `>' output expected from the command
44 `!' metacommand, which can be one of:
45 `timeout' <integer>|no
46 `expect signal' <signal name>[|<signal name>]*
47 `expect return' <integer>
48 `output' <ignore|display>
49 `output sort' [integer]
53 `p' an informative message to print
56 If the expected output do not match the produced output, or if the
57 command did not end as expected, Tesh provides an error message (see
58 the OUTPUT section below) and stops.
60 =head2 Command blocks examples
62 In a given command block, you can declare the command, its input and
63 its expected output in the order that you see fit.
77 You can group several commands together, provided that they don't have
83 =head2 Enforcing the command return code
85 By default, Tesh enforces that the tested command returns 0. If not,
86 it fails with an appropriate message and returns I<code+40> itself.
88 You specify that a given command block is expected to return another
91 # This command MUST return 42
95 The I<expect return> construct applies only to the next command block.
97 =head2 Commands that are expected to raise signals
99 By default, Tesh detects when the command is killed by a signal (such
100 as SEGV on segfaults). This is usually unexpected and unfortunate. But
101 if not, you can specify that a given command block is expected to fail
102 with a signal as follows:
104 # This command MUST raise a segfault
105 ! expect signal SIGSEGV
106 $ ./some_failing_code
108 The I<expect signal> construct applies only to the next command block.
112 By default, no command is allowed to run more than 10 seconds. You can
113 change this value as follows:
115 # Allow some more time to the command
117 $ ./some_longer_command
119 You can also disable the timeout completely by passing "no" as a value:
121 # This command will never timeout
123 $ ./some_very_long_but_safe_command
125 =head2 Setting environment variables
127 You can modify the environment of the tested commands as follows:
132 You can also set an envirmnent variable from the command line:
134 tesh --setenv bindir=/opt/bin/
136 And then use it within the tesh file:
138 $ ${bindir}/myprogram
140 Tesh also supports perl default value for undefined variables:
142 $ ${bindir:=/usr/bin}/myprogram
144 =head2 Not enforcing the expected output
146 By default, the commands output is matched against the one expected,
147 and an error is raised on discrepancy. Metacommands to change this:
153 The output is completely discarded.
157 The output is displayed, but no error is issued if it differs from the
162 The output and the expected output are sorted before comparison (see next section).
166 =head2 Sorting output
168 If the order of the command output changes between runs, you want to
169 sort it before enforcing that it is exactly what you expect. In
170 SimGrid for example, this happens when parallel execution is
171 activated: User processes are run in parallel at each timestamp, and
172 the output is not reproducible anymore. Until you sort the lines.
174 You can sort the command output as follows:
177 $ ./some_multithreaded_command
179 Sorting lines this ways often makes the tesh output very intricate,
180 complicating the error analysis: the process logical order is defeated
181 by the lexicographical sort.
183 The solution is to prefix each line of your output with temporal
184 information so that lines can be grouped by timestamps. The
185 lexicographical sort then only applies to lines that occurred at the
186 same timestamp. Here is a SimGrid example:
188 # Sort only lines depending on the first 19 chars
190 $ ./some_simgrid_simulator --log=root.fmt:[%10.6r]%e(%i:%a@%h)%e%m%n
192 This approach may seem surprising at the first glance but it does its job:
196 =item Every timestamps remain separated, as it should;
198 =item In each timestamp, the output order of processes become
199 reproducible: that's the lexicographical order of their name;
201 =item For each process, the order of its execution is preserved: its
202 messages within a given timestamp are not reordered.
206 That way, tesh can do its job (no false positive, no false negative)
207 despite the unpredictable order of executions of processes within a
208 timestamp, and reported errors remain easy to analyze (execution of a
209 given process preserved).
211 This example is very SimGrid oriented, but the feature could even be
212 usable by others, who knows?
214 =head2 Ignoring some output
216 Some outputted lines can be ignored by setting the ignore command followed
217 by a regular expression:
219 ! ignore .*0x[0-9A-F]+\.
220 $ printf 'word\nMemory address: 0x42AA42.\nanotherword\n'
225 =head2 Colored and formatted text
227 Tesh removes ANSI/VT100 control sequences from outputted text to make easier the writing of tests.
229 $ printf "I \033[0;31mlove\033[0m tesh\n"
234 =head1 BUILTIN COMMANDS
236 =head2 mkfile: creating a file
238 This command creates a file of the name provided as argument, and adds
239 the content it gets as input.
245 It is not possible to use the cat command, as one would expect,
246 because stream redirections are currently not implemented in Tesh.
248 =head1 BUGS, LIMITATIONS AND POSSIBLE IMPROVEMENTS
250 The main limitation is the lack of stream redirections in the commands
251 (">", "<" and "|" shell constructs and friends). The B<mkfile> builtin
252 command makes this situation bearable.
254 It would be nice if we could replace the tesh file completely with
255 command line flags when the output is not to be verified.