This is an 'autotest' framework for distributed play testing of Go-playing
computer programs. You can run clients on as many computers as you want,
to test many combinations in parallel, then display results of the pairings
in an organized way.

It should be easily adaptable for testing of performance of other stochastic
programs against a reference "opponent", just replace twogtp invocations
in the scripts with your own programs.


BASIC USAGE
-----------

Create a networked (NFS or FUSE-sshfs) directory, copy the contents of this
directory there, and customize the rc file - it is heavily commented and
should be self-guiding. You will need the twogtp tool from gogui package.

Then on each computer, change to this directory and run:

	nice -n 18 ./autotest-client `hostname`-1

where 1 should be 2, 3, 4, ... for further instances of autotest you would
run on the same machine (no two instaces with the same id must be run).
The `nice -n 18` prefix will make sure the client is being run with low
priority, in case someone wants to use the computer for some real interaction.

Leave cooking, stir time by time.  You can start and stop clients as much
as you wish on the fly, or edit the rc file to change pairings without any
need for restarting clients.


Time by time, you will of course want to view the results of pairings.
First, you will want to run:

	./autotest-gather

This will collect the results all in one place. Then, you can repeatedly
examine the results using:

	./autotest-show

That will show results from all pairings. You can also use e.g.

	./autotest-show 9-*

to show only results of 9x9 pairings. To refresh the shown results, you
will need to re-run autotest-gather; if you will always want to just see
fresh results, combine the two commands:

	./autotest-gather; ./autotest-show

Note that the S.D. column is standard deviation of the shown winrate;
to get a 95%-confidence interval of the winrate, roughly double the S.D.
value. E.g. WINRATE 30% S.D. 5% means that with 95% probability, the real
winrate is 30% +-10%. The meaning of the first 'S' column is:

	x	errors reported by twogtp
	?	unknown (not in rc) active pairing (OK if temporary)
	/	unknown inactive pairing
	!	known (in rc) inactive pairing (last refresh > 2 hours ago)
	.	known active pairing (normal)

To remove results of old pairings, use:

	./autotest-prune pairing-name

pairing-name can be either simply name of the pairing, or a mask matching
many pairings (up to '*' for removal of all pairing results).

To show status of all known clients, use:

	./autotest-clients

(To remove a gone client, just remove its subdirectory in the c/ directory.)
The WFAIL column shows number of worker failures recorded in the log; examine
c/<clientid>/log for details, you can remove the log to reset the WFAIL
number. Usually, worker failures will result from interrupting a client.


DISCONNECTED USAGE
------------------

If your clients don't share a single filesystem, you can still use autotest
just fine, with two exceptions:

* You will need to manually synchronize the 'rc' file between all your machines

* You will need to manually copy over c/*/ directory tree from all your systems
back to the central server (to the appropriate c/*/ directories) every time
before running ./autotest-gather.

* You will manually need to remove old pairing results on the clients.


IMPLEMENTATION
--------------

Autotest keeps all the state in plain files within the filesystem.

./autotest-client will basically simply keep spawning `autotest-worker`
which will read the rc file, pick a random pairing and use gogui-twogtp
to play one game, then store the result in a dat file (and leave the sgf
file archived for future inspection).

The client workers will create a directory structure like:

	c/drahokam-1/beacon	# when was the client last active
	c/drahokam-1/log	# various events from client's life
	c/drahokam-1/<pair_id>/game.dat
	c/drahokam-1/<pair_id>/game-<N>.sgf

...where drahokam-1 is the ./autotest-client parameter and <pair_id>
is basically straightforwardly mangled version of `pairing` parameters.


./autotest-gather will simply go through directories of all clients and rain
down all pairing results to a single place:

	r/<pair_id>.dat
	r/<pair_id>.beacon  # when was the pairing last active
	r/<pair_id>.error  # number of games reporting error

Then, ./autotest-show will call gogui-twogtp -analyze on the selected
produced dat files.


Copyright (c) Petr Baudis <pasky@ucw.cz>
Licenced under MIT licence (close to public domain).
