Test suite for xvi
==================

In this directory (or the one above) you can "make check" to run
a series of tests on ../src/xvi.

You can also run a single test by saying:
	tests/hanoi
and examine the exit status, "$?", to know whether it succeeded or failed.
0 means success, non-zero means failure, with two special exit codes:
126 means that vi dumped core, while 127 means the test was not runnable.


Internals
---------

The test mechanism uses "expect" coupled with one of two implementations
of a terminal emulator:
 - "tkterm" by Don Libes,(*) based on the Tk terminal widget, and
 - "virterm" by Adrian Mariano, which uses an 80x24 TCL array instead and
   does not need Tk or the X Window System.
Both have been significantly bug-fixed and updated for use here.


Runtime environment variants
----------------------------

If you are running under X, it will use "tkterm", which shows what it is doing
on-screen, but if there is no X server, it uses virterm, which uses a
80x24 array to represent the screen.

To force use of the non-graphical test harness under X, say:
	DISPLAY= make check
or
	TKTERM=virterm make check

You can also run it using tkterm without an X display by saying:
	sudo apt-get install xserver-xorg-video-dummy
	sudo make nox
This only works as root.

To run other vi clones against this test suite, you can use
	VI=nvi make check
or
	VI="vim -c redraw" make check

The DISPLAY, TKTERM and VI tricks also work when running individual tests
(see above).

This stuff seems to work with tcl 8.5, but one test, nul-in-command, needs
tcl 8.6. On Debian, the runes to install the required software is:

    apt-get install expect tclsh tk

and you can omit tk if you'll only be using virterm.


Individual tests
----------------

Each test is an executable file in the directory "tests", and if they need
data files, those live in the directory "data".

The syntax of a test file is fairly horrible because they are written in
the fairly horrible language TCL, with its bizarre string quoting rules
and lame lists represented as strings.

Some test files have names which being with four digits, a hyphen and a name;
the number is their issue number on https://github.com/martinwguy/xvi/issues


Writing new tests
-----------------

New tests should just have names, possibly the same as the name of the
corresponding issue that is/was in ../issues/ while the issue was open.

When an issue is resolved, the file in issues/ is deleted and a test file
created, copying the juiciest parts of the issue description into comments
in the test file, which also serves as a regression test for that issue.

The test file should exit with status:
0	if the test succeeded
1-120	if the test failed (the number identifies the micro-test that failed)
125	if the test is not runnable for some reason (see "git bisect run")

Two more special exit codes are used, generated by scripts/term:
123 if xvi dumped core and
124 if xvi died for some other reason.

A minimal test file is:

	#!/bin/sh
	# The next line is executed by /bin/sh, but not tcl \
	exec tclsh "$0" ${1+"$@"}

	source scripts/term
	start_vi

	stop_vi
	exit 0

and the individual tests go between start_vi and stop_vi, and usually look
something like:

	test 51 "aabc\rdef[esc]"	2 2 [list "abc"]

test	calls the function of that name in scripts/term;
51	is the micro-test number from 1 to 120 and is the script's exit status
	if it fails;
"aabc[esc]"
	is a string of characters to send to xvi as if the user had typed them;
2	is the line the cursor should be on, starting from line 1;
2	is the column the cursor should be in, starting from column 0;
[list "abc" "def" "~"]
	is what the first lines of the screen should be, without any trailing
	spaces, when those characters have been processed by vi.

For reliable testing, you need to ensure that the results of any two
subsequent tests are different, otherwise the second test may succeed
as soon as it sees the screen resulting from the previous one,
but before xvi has processed any of the new input characters.

There is also a similar function

	ctest 48 ":se"	3 ":se"

for tests regarding the contents of the command line. Here, the "3" says
which column of the command line the cursor should be in.

Some, more complicated, micro-tests use the underlying "term_expect" routine
directly, maybe with other functions from "scripts/term". To understand those,
see Don Libes' paper "Automation and Testing of Character-Graphic Programs"
and his book "Exploring Expect", chapter 19, p.448 onwards.


	Martin Guy <martinwguy@gmail.com>, December 2016 - April 2017.
