.. _execute-tests:

Execute Tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``tmt run`` command is used to execute tests. By default all
steps for all discovered test plans are executed:

.. code-block:: shell

    $ tmt run
    /var/tmp/tmt/run-581

    /plans/basic
        discover
            how: fmf
            directory: /home/psss/git/tmt
            filter: tier: 0,1,2
            summary: 15 tests selected
        provision
            how: local
            distro: Fedora release 32 (Thirty Two)
            summary: 1 guest provisioned
        prepare
            how: ansible
            playbook: ansible/packages.yml
            how: install
            summary: Install required packages
            package: beakerlib
            summary: 2 preparations applied
        execute
            how: tmt
            summary: 15 tests executed
        report
            how: display
            summary: 15 tests passed
        finish
            summary: 0 tasks completed

Even if there are no :tmt:story:`/spec/plans` defined it is still
possible to execute tests and custom scripts. See the default
:tmt:story:`/stories/cli/run/default/plan` story for details.

Dry run mode is enabled with the ``--dry`` option:

.. code-block:: shell

    tmt run --dry

Each test run creates a workdir where relevant data such as tests
code from the discover step or test results from the execute step
are stored. If you don't need to investigate test logs and other
artifacts generated by the run you can remove the workdir after
the execution is finished:

.. code-block:: shell

    tmt run --remove
    tmt run --rm
    tmt run -r


Select Plans
------------------------------------------------------------------

Choose which plans should be executed:

.. code-block:: shell

    $ tmt run plan --name basic
    /var/tmp/tmt/run-083

    /plans/basic
        discover
            how: fmf
            url: https://github.com/teemtee/tmt
            ref: devel
            filter: tier: 0,1
            tests: 2 tests selected
        provision
        prepare
        execute
            how: tmt
            result: 2 tests passed, 0 tests failed
        report
        finish
        cleanup


Select Tests
------------------------------------------------------------------

Run only a subset of available tests across all plans:

.. code-block:: shell

    $ tmt run test --filter tier:1
    /plans/basic
        discover
            how: fmf
            url: https://github.com/teemtee/tmt
            ref: devel
            filter: tier: 0,1
            tests: 1 test selected
        ...

    /plans/helps
        discover
            how: shell
            directory: /home/psss/git/tmt
            tests: 0 tests selected
        ...

    /plans/smoke
        discover
            how: shell
            tests: 0 tests selected
        ...

To run only tests defined in the current working directory:

.. code-block:: shell

    $ tmt run test --name .

Select Steps
------------------------------------------------------------------

The test execution is divided into the following seven steps:
``discover``, ``provision``, ``prepare``, ``execute``, ``report``,
``finish`` and ``cleanup``. See the :ref:`specification` for more
details about individual steps.

It is possible to execute only selected steps. For example in
order to see which tests would be executed without actually
running them choose the ``discover`` step:

.. code-block:: shell

    $ tmt run discover
    /var/tmp/tmt/run-085

    /plans/basic
        discover
            how: fmf
            url: https://github.com/teemtee/tmt
            ref: devel
            filter: tier: 0,1
            tests: 2 tests selected

    /plans/helps
        discover
            how: shell
            directory: /home/psss/git/tmt
            tests: 4 tests selected

Use ``--verbose`` and ``--debug`` to enable more detailed output
such as list of individual tests or showing the progress of the
test environment provisioning:

.. code-block:: shell

    $ tmt run discover --verbose
    /var/tmp/tmt/run-767

    /plans/basic
        discover
            how: fmf
            url: https://github.com/teemtee/tmt
            ref: devel
            filter: tier: 0,1
            tests: 2 tests selected
                /one/tests/docs
                /one/tests/ls

    /plans/helps
        discover
            how: shell
            directory: /home/psss/git/tmt
            tests: 4 tests selected
                /help/main
                /help/test
                /help/plan
                /help/smoke

You can also choose multiple steps to be executed:

.. code-block:: shell

    tmt run discover provision prepare

Arguments for particular step can be specified after the step
name, options for all steps should go to the ``run`` command:

.. code-block:: shell

    # debug output for provision only
    tmt run discover provision --debug

    # debug output for all steps
    tmt run --debug discover provision

In order to execute all test steps while providing arguments to
some of them it is possible to use the ``--all`` option:

.. code-block:: shell

    tmt run --all provision --how=local


Execute Progress
------------------------------------------------------------------

When run in terminal ``execute`` step prints name of the executed test
unless ``--debug`` or ``--verbose`` options are used:

.. code-block:: shell

    $ tmt run

        execute
            how: tmt
            progress: /tests/core/enabled [5/16]


In the verbose mode the duration and result after the test is executed
are printed:

.. code-block:: shell

    $ tmt run --all execute -v

        execute
            how: tmt
                00:00:03 pass /tests/core/adjust [1/16]

More verbose mode prints summary of the test being executed first:

.. code-block:: shell

    $ tmt run --all execute -vv

        execute
            how: tmt
            exit-first: False
                test: Verify test/plan adjustments based on context
                    00:00:04 pass /tests/core/adjust [1/16]

The most verbose mode prints also the test output:

.. code-block:: shell

    $ tmt run --all execute -vvv

        execute
            how: tmt
            order: 50
            exit-first: False
                test: Verify test/plan adjustments based on context
                    out:
                    out: ::::::::::::::::::::::::::::::::::::::::::::
                    out: ::   Setup
                    out: ::::::::::::::::::::::::::::::::::::::::::::
                    out:
                    out: :: [ 17:03:06 ] :: [  BEGIN   ] :: Create...
                        ....
                    00:00:04 pass /tests/core/adjust [1/16]


Check Report
------------------------------------------------------------------

When a particular step is ``done``, it won't be executed
repeatedly unless ``--force`` is used:

.. code-block:: shell

    $ tmt run -l report --verbose
    /plans/features/core
        report
            status: done
            summary: 10 tests passed

If you need additional information about your already ``done``
run use ``--force`` together with the ``--verbose`` option:

.. code-block:: shell

    $ tmt run -l report -v --force
    /plans/features/core
        report
            how: display
                pass /tests/core/adjust
                pass /tests/core/docs
                pass /tests/core/dry
                pass /tests/core/env
                pass /tests/core/error
                pass /tests/core/force
                pass /tests/core/ls
                pass /tests/core/path
                pass /tests/core/smoke
                pass /tests/unit
            summary: 10 tests passed

In order to investigate test logs raise verbosity even more:

.. code-block:: shell

    $ tmt run -l report -vv --force
    /plans/features/core
        report
            how: display
                pass /tests/core/adjust
                    output.txt: /var/tmp/tmt/run-759/plans/features/core/execute/data/tests/core/adjust/output.txt
                    journal.txt: /var/tmp/tmt/run-759/plans/features/core/execute/data/tests/core/adjust/journal.txt
                pass /tests/core/docs
                    output.txt: /var/tmp/tmt/run-759/plans/features/core/execute/data/tests/core/docs/output.txt
                    journal.txt: /var/tmp/tmt/run-759/plans/features/core/execute/data/tests/core/docs/journal.txt
                pass /tests/core/dry
                    output.txt: /var/tmp/tmt/run-759/plans/features/core/execute/data/tests/core/dry/output.txt
                    journal.txt: /var/tmp/tmt/run-759/plans/features/core/execute/data/tests/core/dry/journal.txt
                ...
            summary: 10 tests passed

Use level 3 verbosity ``-vvv`` to show the complete test output.
For more comfortable review, generate an ``html`` report and open
it in your favorite web browser:

.. code-block:: shell

    $ tmt run --last report --how html --open --force
    $ tmt run -l report -h html -of


Provision Options
------------------------------------------------------------------

By default, tests are executed under a virtual machine so that
your laptop is not affected by unexpected changes. The following
commands are equivalent:

.. code-block:: shell

    tmt run
    tmt run -a provision -h virtual
    tmt run --all provision --how=virtual

You can also use an alternative virtual machine implementation
using the ``testcloud`` provisioner:

.. code-block:: shell

    tmt run --all provision --how=virtual.testcloud

If you already have a box ready for testing with ``ssh`` enabled,
use the ``connect`` method:

.. code-block:: shell

    tmt run --all provision --how=connect --guest=name-or-ip --user=login --password=secret --become
    tmt run --all provision --how=connect --guest=name-or-ip --key=private-key-path

The ``container`` method allows to execute tests in a container
using ``podman``:

.. code-block:: shell

    tmt run --all provision --how=container --image=fedora:latest

If you are confident that tests are safe you can execute them
directly on your ``local`` host:

.. code-block:: shell

    tmt --feeling-safe run --all provision --how=local

In order to reboot a provisioned guest use the ``reboot`` command.
By default a soft reboot is performed which should prevent data
loss, use ``--hard`` to force a hard reboot:

.. code-block:: shell

    tmt run --last reboot
    tmt run --last reboot --hard


Debug Tests
------------------------------------------------------------------

Sometimes the environment preparation can take a long time. Thus,
especially for debugging tests, it usually makes sense to run the
``provision`` and ``prepare`` step only once, then ``execute``
tests as many times as necessary to debug the test code and
finally clean up when debugging is done:

.. code-block:: shell

    tmt run --id <ID> --until execute    # prepare, run test once

    tmt run -i <ID> execute -f           # run test again
    tmt run -i <ID> execute -f           # run it again
    tmt run -i <ID> execute -f           # and again

    tmt run -i <ID> report finish cleanup

Instead of always specifying the whole run id you can also use
``--last`` or ``-l`` as an abbreviation for the last run id:

.. code-block:: shell

    tmt run --last execute --force
    tmt run -l execute -f

The ``--force`` option instructs ``tmt`` to run given step even if
it has been already completed before. Use ``discover --force`` to
synchronize test code changes to the run workdir:

.. code-block:: shell

    tmt run -l discover -f execute -f

In order to interactively debug tests use the ``--interactive``
option which disables output capturing so that you can see what
exactly is happening during test execution. This also allows to
inspect particular place of the code by inserting a ``bash`` in
the shell code or ``import pdb; pdb.set_trace()`` for python:

.. code-block:: shell

    tmt run --all execute --how tmt --interactive


Aliases
------------------------------------------------------------------

It might be useful to set up a set of shell aliases for the tmt
command lines which you often use. For a quick reservation of
a machine or a container for quick experimenting:

.. code-block:: shell

    alias reserve='tmt run login --step execute execute cleanup provision --how container --image fedora'

Reserving a testing box then can be as short as this:

.. code-block:: shell

    reserve
    reserve -h virtual
    reserve -i fedora:32
    reserve --how virtual
    reserve --image fedora:32

For interactive debugging of tests the following three aliases can
come in handy:

.. code-block:: shell

    alias start='tmt run --verbose --until report execute --how tmt --interactive test --name . provision --how virtual --image fedora'
    alias retest='tmt run --last test --name . discover -f execute -f --how tmt --interactive'
    alias stop='tmt run --last report --verbose cleanup'

The test debugging session then can look like this:

.. code-block:: shell

    start
    retest
    retest
    retest login
    ...
    stop

First you ``start`` the session in order to provision a testing
environment, then you ``retest`` your test code changes as many
times as you need to finalize the test implementation, and finally
``stop`` is used to clean up the testing environment.


Guest Login
------------------------------------------------------------------

Use the ``login`` command to get an interactive shell on the
provisioned guest. This can be useful for example for additional
manual preparation of the guest before testing or checking test
logs to investigate a test failure:

.. code-block:: shell

    tmt run login --step prepare
    tmt run login --step execute

It's possible to log in at the start or end of a step or select
the desired step phase using order:

.. code-block:: shell

    tmt run login --step prepare:start
    tmt run login --step prepare:50
    tmt run login --step prepare:end

Interactive shell session can be also enabled conditionally when
specific test result occurs:

.. code-block:: shell

    tmt run login --when fail
    tmt run login --when fail --when error

You can also enable only the ``provision`` step to easily get a
clean and safe environment for experimenting. Use the ``cleanup``
step to remove provisioned guest:

.. code-block:: shell

    tmt run provision login
    tmt run --last cleanup

Clean up the box right after your are done with experimenting by
combining the above-mentioned commands on a single line:

.. code-block:: shell

    tmt run provision login cleanup

Login can be used to run an arbitrary script on a provisioned
guest. This can be handy if you want to run arbitrary scripts between
steps for example. This is currently used in the Testing Farm's
`tmt reproducer`__:

.. code-block:: shell

    tmt run --last login < script.sh

__ https://docs.testing-farm.io/Testing%20Farm/0.1/test-results.html#_tmt_reproducer

Have you heard already that using command abbreviation is possible
as well? It might save you some typing:

.. code-block:: shell

    tmt run pro log fin

See the :tmt:story:`/stories/cli/run/login` user stories for more
details and examples.


Rerun Failed Tests
------------------------------------------------------------------

Executing failed tests again after fixing them is now possible
with `tmt run --last --all --again tests --failed-only`.

This is only possible when you have the run directory available
and `--id` argument provided (or use `--last`) as it needs the data from
execute step to select only failed test cases. After new execute step,
tmt will again merge the results from the previous run with the new ones
to keep all the data for full report.

.. code-block:: shell

    $ tmt run
    # Some tests fail, some pass

    $ tmt run --last --again discover tests --failed-only
    # Discover tests to rerun

    $ tmt run --all --last --again tests --failed-only
    # Run all failed tests again

.. versionadded:: 1.34
