Compatibility for AdaCore's legacy testsuites
=============================================

Although all the default behaviors in ``e3.testsuite`` presented in this
documentation should be fine for most new projects, it is not realistic to
require existing big testsuites to migrate to them. A lot of testsuites at
AdaCore use similar formalisms (atomic testcases, dedicated test directories,
...), but different formats: no ``test.yaml`` file, custom files for test
execution control, etc.

These testsuites contain a huge number of testcases, and thus it is a better
investment of time to introduce compatible settings in testsuite scripts rather
than reformat all testcases. This section presents compatibility helpers for
legacy AdaCore testsuites.



Test finder
-----------

The ``e3.testsuite.testcase_finder.AdaCoreLegacyTestFinder`` class can act as a
drop-in test finder for legacy AdaCore testsuites: all directories whose name
matches a TN (Ticket Number), i.e. matching the
``[0-9A-Z]{2}[0-9]{2}-[A-Z0-9]{3}`` regular expression, are considered as
containing a testcase. Legacy AdaCore testsuites have only one driver, so this
test finder always use the same driver. For instance:

.. code-block:: python

   @property
   def test_finders(self):
       # This will create a testcase for all directories whose name matches a
       # TN, using the MyDriver test driver.
       return [AdaCoreLegacyTestFinder(MyDriver)]


Test control
------------

AdaCore legacy testsuites rely on a custom file format to lead testcase
execution control: ``test.opt`` files.

Similarly to the :ref:`YAML-based control descriptions <api_control_yaml>`,
this format provides a declarative formalism to describe settings depending on
the environment, and more precisely on a set of *discriminants*: simple case
insensitive names for environment specificities. For instance: ``linux`` on a
Linux system, ``windows`` on a Windows one, ``x86`` on Intel 32 bits
architecture, ``vxworks`` when targetting a VxWorks is involved, etc.

A parser for such files is included in ``e3.testsuite`` (see the
``optfileparser`` module), and most importantly, a ``TestControlCreator``
subclass binds it to the rest of the testsuite framework:
``AdaCoreLegacyTestControlCreator``, from the ``e3.testsuite.control`` module.
Its constructor requires the list of discriminants used to selectively evaluate
``test.opt`` directives.

This file format not only controls test execution with its ``DEAD``, ``XFAIL``
and ``SKIP`` commands: it also allows to control the name of the script file to
run (``CMD`` command), the name of the output baseline file (``OUT``), the time
limit for the script (``RLIMIT``), etc. For this reason,
``AdaCoreLegacyTestControlCreator`` works best with the AdaCore legacy test
driver: see the next section.


Test driver
-----------

All legacy AdaCore testsuites use actual/expected test output comparisons to
determine if a test passes, so the reference test driver for them derives from
``DiffTestDriver``: ``e3.testsuite.driver.adacore.AdaCoreLegacyTestDriver``.
This driver is coupled with a custom test execution control mechanism:
``test.opt`` files (see the previous section), and thus overrides the
``test_control_creator`` property accordingly.

This driver has two requirements for ``Testsuite`` subclasses using it:

* Put a process environment (string dictionary) for subprocesses in
  ``self.env.test_environ``. By default they can just put a copy of the
  testsuite's own environment: ``dict(os.environ)``.

* Put the list of discriminants (list of strings) in ``self.env.discs``.
  For the latter, starting from the result of the
  ``e3.env.AbstractEnv.discriminants`` property can help, as it computes
  standard discriminants based on the current host/build/target platforms.
  Testsuites can then add more discriminants as needed.

For instance, imagine a testsuite that wants standard dircriminants plus the
``valgrind`` discriminant if the ``--valgrind`` command-line option is passed
to the testsuite:

.. code-block:: python

   class MyTestsuite(Testsuite):
       def add_options(self, parser):
           parser.add_argument("--valgrind", action="store_true",
                               help="Run tests under Valgrind")

       def set_up(self):
           super(MyTestsuite, self).set_up()
           self.env.test_environ = dict(os.environ)
           self.env.discs = self.env.discriminants
           if self.env.options.valgrind:
               self.env.discs.append("valgrind")

There is little point describing precisely the convoluted behavior for this
driver, so we will stick here to a summary, with a few pointers to go further:

* All testcases must provide a script to run. Depending on testsuite defaults
  (``AdaCoreLegacyTestControlCreator.default_script`` property) and the content
  of each ``test.opt`` testcase file, this script can be a Windows batch script
  (``*.cmd``), a Bourne-compatible shell script (``*.sh``) or a Python script
  (``*.py``).

* It is the output of this script that is compared against the output baseline.
  To hide environment-specific differences, output refiners turn backslashes
  into forward slashes, remove ``.exe`` extensions and also remove occurences
  of the working directory.

* On Unix systems, this driver has a very crude conversion of Windows batch
  script to Bourne-compatible scripts: text substitution remove some ``.exe``
  extensions, replaces ``%VAR%`` environment variable references with ``$VAR``,
  etc. See ``AdaCoreLegacyTestDriver.get_script_command_line``. Note that
  subclasses can override this method to automatically generate a test script.

Curious readers are invited to read the sources to know the details: doing so
is necessary anyway to override specific behaviors so that this driver fits the
precise need of some testsuite. Hopefully, this documentation and inline
comments have made this process easier.
