
..  _RefTests:

Test suite
****************************************************************************

The Dragonfly library contains tests to verify its functioning and
assure its quality.  These tests come in two distinct types:

 * Tests based on `unittest <http://docs.python.org/library/unittest.html>`_.
 * Tests based on `doctest <http://docs.python.org/library/doctest.html>`_;
   these also serve as documentation by providing usage examples.

See the links below for tests of both types.

Tests in doctests format:

.. toctree::
   :maxdepth: 2

   test_grammar_elements_basic_doctest
   test_grammar_elements_compound_doctest
   test_grammar_list_doctest
   test_recobs_doctest
   test_action_base_doctest
   test_word_formatting_v10_doctest
   test_word_formatting_v11_doctest


Tests based on the unittest framework reside in the :mod:`dragonfly.test`
package.

.. toctree::
   :maxdepth: 2


Running the test suite
============================================================================

This section contains instructions on how to run dragonfly's test suite with
the SR engine backends.

Most tests were written with an English model andvocabulary in mind, so they
will not pass if English words are not in the SR engine's vocabulary. The
only exception to this is the text-input ("text") backend.


Using DNS
----------------------------------------------------------------------------

Follow the following steps to run the test suite for the DNS backend:

 #. Start DNS.  (And ensure that NatLink is also automatically started.)
 #. Extract the Dragonfly source code in a directory ``<dir>``.
 #. Run the follow commands to install test requirements:

    .. code:: shell

        cd <dir>
        pip install -r test_requirements.txt

 #. Run the tests with the following command:

    .. code:: shell

        python setup.py test --test-suite=natlink


Different tests need to be run for different DNS versions. The natlink test
suite will attempt to run the correct tests for the version of DNS you are
using.

These tests may fail with certain DNS versions, models and/or vocabularies.
It is possible for the test suite to fail even though everything still
works in command modules loaded by ``natlinkmain``. This is a natlink issue.

It is not feasible to test dragonfly with all DNS configurations.
Please get in touch if the tests fail for you **and** specific dragonfly
functionality doesn't work in natlink command modules (i.e. ``.py`` files in
the ``MacroSystem`` folder).


Using WSR
----------------------------------------------------------------------------

Follow the following steps to run the test suite for the WSR backend:

 #. Start WSR.
 #. Wake WSR up, so that it is *not* in sleeping state, and then turn the
    microphone *off*.  (It is important to wake the microphone up first,
    because otherwise it'll be off and sleeping at the same time.  This
    causes all recognitions to fail.  Weird, huh?)
 #. Extract the Dragonfly source code in a directory ``<dir>``.
 #. Run the follow commands to install test requirements:

    .. code:: shell

        cd <dir>
        pip install -r test_requirements.txt

 #. Run the tests with the following command:

    .. code:: shell

        python setup.py test --test-suite=sapi5


You can also run the test suite for the in process engine class. It has no
GUI for the moment. Run the tests with the following command:

    .. code:: shell

        python setup.py test --test-suite=sapi5inproc


Some SAPI5 engine tests will fail intermittently, particularly with the
shared process engine class (``"sapi5"`` / ``"sapi5shared"``). There is not
much we can do about this.


Using the Kaldi engine
----------------------------------------------------------------------------

Follow the following steps to run the test suite for the Kaldi backend:

 #. Extract the Dragonfly source code in a directory ``<dir>``.
 #. Follow the set up and install instructions for the Kaldi engine
    :ref:`on this page <RefKaldiEngine>`.
 #. Run the following commands to install the Kaldi engine and test
    dependencies:

    .. code:: shell

        cd <dir>
        pip install -e '.[kaldi]'
        pip install -r test_requirements.txt

 #. Run the tests with the following command:

    .. code:: shell

        python setup.py test --test-suite=kaldi


Using the CMU Pocket Sphinx engine
----------------------------------------------------------------------------

Follow the following steps to run the test suite for the Sphinx backend:

 #. Extract the Dragonfly source code in a directory ``<dir>``.
 #. Run the following commands to install the Sphinx engine and test
    dependencies:

    .. code:: shell

        cd <dir>
        pip install -e '.[sphinx]'
        pip install -r test_requirements.txt


 #. Run the tests with the following command:

    .. code:: shell

        python setup.py test --test-suite=sphinx


Using the text-input engine
----------------------------------------------------------------------------

Follow the following steps to run the test suite for the "text" backend:

 #. Extract the Dragonfly source code in a directory ``<dir>``.
 #. Run the follow commands to install test requirements:

    .. code:: shell

        cd <dir>
        pip install -r test_requirements.txt

 #. Run the tests with the following commands:

    .. code:: shell

        cd <dir>
        python setup.py test


This is the default test suite for dragonfly and has no additional
dependencies. The ``--test-suite`` argument does not need to be specified
in this case.
