.. _ostestr:

ostestr
=======

.. warning::
   ``ostestr`` command is deprecated. Use `stestr`_ command instead like
   following.

   0. Install `stestr`_ (This step is already done if you're using ostestr.)
   1. You can use ``stestr run ...`` instead of ``ostestr ...``
   2. You can use ``stestr list ...`` instead of `ostestr --list ...``

   For more sub commands and options, please refer to `stestr help` or the
   `stestr`_ document.

.. _stestr: https://stestr.readthedocs.io/

The ostestr command provides a wrapper around the testr command included in
the testrepository package. It's designed to build on the functionality
included in testr and workaround several UI bugs in the short term. By default
it also has output that is much more useful for OpenStack's test suites which
are lengthy in both runtime and number of tests. Please note that the CLI
semantics are still a work in progress as the project is quite young, so
default behavior might change in future version.

Summary
-------
::

    ostestr [-b|--blacklist-file <blacklist_file>] [-r|--regex REGEX]
            [-w|--whitelist-file <whitelist_file>]
            [-p|--pretty] [--no-pretty] [-s|--subunit] [-l|--list]
            [-n|--no-discover <test_id>] [--slowest] [--no-slowest]
            [--pdb <test_id>] [--parallel] [--serial]
            [-c|--concurrency <workers>] [--until-failure] [--print-exclude]

Options
-------

--blacklist-file BLACKLIST_FILE, -b BLACKLIST_FILE
                      Path to a blacklist file, this file contains a
                      separate regex exclude on each newline
--whitelist-file WHITELIST_FILE, -w WHITELIST_FILE
                      Path to a whitelist file, this file contains a
                      separate regex on each newline
--regex REGEX, -r REGEX
                      A normal testr selection regex.

--black-regex BLACK_REGEX, -B BLACK_REGEX
                      Test rejection regex. If the test cases durring a
                      search opration matches, it will be removed from the
                      final test list.
--pretty, -p
                      Print pretty output from subunit-trace. This is
                      mutually exclusive with --subunit
--no-pretty
                      Disable the pretty output with subunit-trace
--subunit, -s
                      output the raw subunit v2 from the test run this is
                      mutually exclusive with --pretty
--list, -l
                      List all the tests which will be run.
--no-discover TEST_ID, -n TEST_ID
                      Takes in a single test to bypasses test discover and
                      just execute the test specified
--slowest
                      After the test run print the slowest tests
--no-slowest
                      After the test run don't print the slowest tests
--pdb TEST_ID
                      Run a single test that has pdb traces added
--parallel
                      Run tests in parallel (this is the default)
--serial
                      Run tests serially
--concurrency WORKERS, -c WORKERS
                      The number of workers to use when running in parallel.
                      By default this is the number of cpus
--until-failure
                      Run the tests in a loop until a failure is
                      encountered. Running with subunit or prettyoutput
                      enable will force the loop to run testsserially
--print-exclude
                      If an exclude file is used this option will prints the
                      comment from the same line and all skipped tests
                      before the test run

Running Tests
-------------

os-testr is primarily for running tests at it's basic level you just invoke
ostestr to run a test suite for a project. (assuming it's setup to run tests
using testr already) For example::

    $ ostestr

This will run tests in parallel (with the number of workers matching the number
of CPUs) and with subunit-trace output. If you need to run tests in serial you
can use the serial option::

    $ ostestr --serial

Or if you need to adjust the concurrency but still run in parallel you can use
-c/--concurrency::

    $ ostestr --concurrency 2

If you only want to run an individual test module or more specific (a single
class, or test) and parallel execution doesn't matter, you can use the
-n/--no-discover to skip test discovery and just directly calls subunit.run on
the tests under the covers. Bypassing discovery is desirable when running a
small subset of tests in a larger test suite because the discovery time can
often far exceed the total run time of the tests.

For example::

    $ ostestr --no-discover test.test_thing.TestThing.test_thing_method

Additionally, if you need to run a single test module, class, or single test
with pdb enabled you can use --pdb to directly call testtools.run under the
covers which works with pdb. For example::

    $ ostestr --pdb tests.test_thing.TestThing.test_thing_method


Test Selection
--------------

ostestr intially designed to build on top of the test selection in testr.
testr only exposed a regex option to select tests. This functionality is
exposed via the --regex option. For example::

    $ ostestr --regex 'magic\.regex'

This will do a straight passthrough of the provided regex to testr.
When ostestr is asked to do more complex test selection than a sinlge regex,
it will ask testr for a full list of tests than passing the filtered test list
back to testr.
ostestr allows you do to do simple test exclusion via apssing rejection/black regex::

    $ ostestr --black-regex 'slow_tests|bad_tests'

ostestr also allow you to combine these argumants::

    $ ostestr --regex ui\.interface --black-regex 'slow_tests|bad_tests'

Here first we selected all tests which  matches to 'ui\.interface',
than we are dropping all test which matches
'slow_tests|bad_tests' from the final list.

ostestr also allows you to specify a blacklist file to define a set
of regexes to exclude. You can specify a blacklist file with the
--blacklist_file/-b option, for example::

    $ ostestr --blacklist_file $path_to_file

The format for the file is line separated regex, with '#' used to signify the
start of a comment on a line. For example::

    # Blacklist File
    ^regex1 # Excludes these tests
    .*regex2 # exclude those tests

The regex used in the blacklist File or passed as argument, will be used
to drop tests from the initial selection list.
Will generate a list which will exclude both any tests
matching '^regex1' and '.*regex2'. If a blacklist file is used in conjunction
with the --regex option the regex specified with --regex will be used for the intial
test selection. Also it's worth noting that the
regex test selection options can not be used in conjunction with the
--no-discover or --pdb options described in the previous section. This is
because the regex selection requires using testr under the covers to actually
do the filtering, and those 2 options do not use testr.

The dual of the blacklist file is the whitelist file which altering the initial
test selection regex, by joining the white list elements by '|'.
You can specify the path to the file with --whitelist_file/-w, for example::

    $ ostestr --whitelist_file $path_to_file

The format for the file is more or less identical to the blacklist file::

    # Whitelist File
    ^regex1 # Include these tests
    .*regex2 # include those tests

However, instead of excluding the matches it will include them.

It's also worth noting that you can use the test list option to dry run any
selection arguments you are using. You just need to use --list/-l with your
selection options to do this, for example::

    $ ostestr --regex 'regex3.*' --blacklist_file blacklist.txt --list

This will list all the tests which will be run by ostestr using that combination
of arguments.

Please not that all of this selection functionality will be expanded on in the
future and a default grammar for selecting multiple tests will be chosen in a
future release. However as of right now all current arguments (which have
guarantees on always remaining in place) are still required to perform any
selection logic while this functionality is still under development.


Output Options
--------------

By default ostestr will use subunit-trace as the output filter on the test
run. It will also print the slowest tests from the run after the run is
concluded. You can disable the printing the slowest tests with the --no-slowest
flag, for example::

    $ ostestr --no-slowest

If you'd like to disable the subunit-trace output you can do this using
--no-pretty::

    $ ostestr --no-pretty

ostestr also provides the option to just output the raw subunit stream on
STDOUT with --subunit/-s. Note if you want to use this you also have to
specify --no-pretty as the subunit-trace output and the raw subunit output
are mutually exclusive. For example, to get raw subunit output the arguments
would be::

    $ ostestr --no-pretty --subunit

An additional option on top of the blacklist file is --print-exclude option.
When this option is specified when using a blacklist file before the tests are
run ostestr will print all the tests it will be excluding from the blacklist
file. If a line in the blacklist file has a comment that will be printed before
listing the tests which will be excluded by that line's regex. If no comment is
present on a line the regex from that line will be used instead. For example,
if you were using the example blacklist file from the previous section the
output before the regular test run output would be::

    $ ostestr -b blacklist-file blacklist.txt --print-exclude
    Excludes these tests
    regex1_match
    regex1_exclude

    exclude those tests
    regex2_match
    regex2_exclude

    ...

Notes for running with tox
--------------------------

If you use `tox`_ for running your tests and call ostestr as the test command
it's recommended that you set a posargs following ostestr on the commands
stanza. For example::

    [testenv]
    commands = ostestr {posargs}

.. _tox: https://tox.readthedocs.org/en/latest/

this will enable end users to pass args to configure the output, use the
selection logic, or any other options directly from the tox cli. This will let
tox take care of the venv management and the environment separation but enable
direct access to all of the ostestr options to easily customize your test run.
For example, assuming the above posargs usage you would be to do::

    $ tox -epy34 -- --regex ^regex1

or to skip discovery::

    $ tox -epy34 -- -n test.test_thing.TestThing.test_thing_method
