Regression Testing
==================

The suite of benchmarks can be automatically regression-tested against
a set of "expected" results. This file gives some instructions for how
to add or modify a tested case.

Running the cases
-----------------

The first step is just to actually run the benchmarks. There is a
hierarchy of Makefiles from the top-level down. At the lowest level,
of an individual benchmark, the Makefile should run the case,
resulting in a `params.log` file whose first line is a description of
the columns contained within.

Generating expected results
---------------------------

Once a benchmark has been run, it should be validated by hand to
ensure it is suitable for recording as a reference result. The
`generate_expected_out.py` can be run to record `u_rms` and `nu_top`
fields from the last timestep in `params.log` as the reference result,
for later use in the test suite.

Running the test suite
----------------------

When run through *pytest* itself, the `test_all.py` script will ensure
that every `params.log` script in the list of selected test cases
matches its corresponding expected results.

Automatic running through CI
----------------------------

This entire process is automatically captured in a GitHub Actions
workflow, which will use the latest Firedrake Docker container, run
all the tests, and then ensure their results match those which have
been checked in.

**Important**: Because some tests may be expensive to run, they can be
disabled in two places:

1. To stop the results from being generated for a given case, it can
   be removed from the Makefile hierarchy. e.g. only the 2D cases may
   be actually run, even though Makefiles exist for all benchmarks.

2. Once a test isn't expected to be run through CI, it should also be
   deselected through pytest's `-k` flag in the workflow definition
   file.
