fhtagn – a tiny CLI programs tester written in AWK
June 2023
fhtagn is a tiny CLI tool for literate testing for command-line programs.
What does it mean literate testing?
Let’s say you created some program command
. You want to create a set of end-to-end tests for it.
With fhtagn it’s as simple as creating a file tests.tush
with the following content:
$ command --that --should --execute correctly
| expected stdout output
$ command --that --will --cause error
@ expected stderr output
? expected-exit-code
And running it:
In case of success the tool will output nothing (in Unix tradition of being silent when all is OK).
In case if the actual output of a command line doesn’t match the expected output or exit code – the tool will show a diff
of the expected and the actual results.
- Each test file can have multiple such tests (like the example above have two, as you can see).
- fhtagn will only process lines starting
$
,|
,@
and?
. So you can have any other content there, that doesn’t start these symbols, for example description for each test. Alternatively, you can even make test files a markdown and place the tests into code blocks for readability. - Command lines can be multiline.
I use fhtagn as a testing tool for my projects:
I wrote this tool in AWK. Surprised? Check my article Fascination with AWK.
tush rewrite
In fact this project is my re-implementation of darius/tush, so all credit for the idea goes to the original author Darius Bacon.
But my rewrite is simpler (single tiny AWK script) and faster, because:
- it uses
/dev/shm
where available instead of/tmp
- it compares the expected result with the actual in the code and only calls
diff
to show the difference if they don’t match - it doesn’t create a sandbox folder for each test
- it doesn’t use
mktemp
but rather generates random name in the code
Design principles
Below I want to elaborate a bit on design principles I’ve used to achieve the best speed. Basically there are two of them:
- minimize running external processes
- minimize I/O operations
Let’s show on examples.
AWK script, not shell
By using /usr/bin/awk
shebang instead of /bin/sh
with subsequent awk invocation we make it one shell process call less.
Do more in one shell invocation
Here we do a single shell invocation and get two values from it. It’s faster than doing two separate shell invocations.
Creating one less file by using stdin
Here we do
echo actual_data | diff expected_file -
instead of
diff expected_file actual_file
Because this allows to skip creating the actual_file
.
Also note how we combine there the deletion (rm
) of a temp file in the same call.
Batching cleanup in a single call
This change optimizes the temporary files removal. Instead of calling rm
for each test we collect all temp files into ToDel
variable and do the actual removal inside the END
block.
By the way, this issue was identified by generating an arbitrary large synthetic test file and was fixed in this issue.
Results
On makesure project running the test suite (excluding the tests in 200_update.tush
that does network calls) ./makesure tested_awks
:
Before (tush*) | After (fhtagn) |
---|---|
36.1 sec | 24.2 sec |
The speedup is 33%!
*) For tush I was using the adolfopa/tush fork.
Tricky issue trying to test fhtagn with fhtagn
We test fhtagn with fhtagn! Yes, we eat our own dog food.
In fact, since fhtagn is syntactically fully compatible with tush, for the reference we firstly run the test suite with tush, and then, as an additional check, we run it with fhtagn.
Testing fhtag with fhtagn revealed very interesting bug. The test constantly stopped with an error
Testing with fhtagn (./fhtagn.awk)...
error reading file: /dev/shm/fhtagn.893568705.out
The .err
and .out
files are created for each $
line to capture stdout and stderr outputs of the executed command line. For some reason, the files were missing!
Initially I thought that for some mysterious reason, when running fhtagn tests with fhtagn the mentioned files were not created. I spent tons of time debugging this hypothesis. It appears, this was not the case.
Long story short, the files were created, but were then deleted. Let me explain.
When running fhtagn tests with fhtagh what happens is
fhtagn.awk
(runner) starts fhtagn.awk
(program under test).
Now, both fhtagn processes generate temporary .err
and .out
files with random names.
I used rand()
and srand()
AWK functions to generate a random name for a file.
It appears, that AWK’s
rand()
always generates the same random sequence unless you set a seed withsrand()
srand()
by default sets the seed number that equals to the current timestamp (with 1 sec precision).
At this point you may have already guessed the culprit.
Since the “runner” fhtagn starts the “program under test” fhtagn in under some milliseconds after own start, the seed appears to be equal for both fhtagn processes, so they generate equal “random” file names!
I solved this problem by introducing the additional source of randomness in the form of the PID of a shell process ($$
).
The problem was solved.
Overall the dogfooding practice proved to be really useful to stress-test the application.
from Hacker News https://ift.tt/tDQZBck
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.