==============
pytest-pyvista
==============
.. image:: https://img.shields.io/pypi/v/pytest-pyvista.svg?color=orange&label=pypi&logo=python&logoColor=white
    :target: https://pypi.org/project/pytest-pyvista
    :alt: PyPI version

.. image:: https://img.shields.io/conda/vn/conda-forge/pytest-pyvista?color=orange&label=conda-forge&logo=conda-forge&logoColor=white
    :target: https://anaconda.org/conda-forge/pytest-pyvista
    :alt: conda-forge version

.. image:: https://img.shields.io/pypi/pyversions/pytest-pyvista.svg?color=orange&logo=python&label=python&logoColor=white
    :target: https://pypi.org/project/pytest-pyvista
    :alt: Python versions

.. image:: https://github.com/pyvista/pytest-pyvista/actions/workflows/ci_cd.yml/badge.svg
    :target: https://github.com/pyvista/pytest-pyvista/actions/workflows/ci_cd.yml
    :alt: GitHub Actions: Unit Testing and Deployment

Plugin to test PyVista plot outputs.

----

This `pytest`_ plugin was generated with `Cookiecutter`_ along with
`@hackebrot`_'s `cookiecutter-pytest-plugin`_ template.


Features
--------
This plugin facilitates the comparison of the images produced by `PyVista`. It
generates a cache of images from the tests, using the `PyVista` plotting
function in its first execution. Then, further executions will compare its
results against this cache, so if there are any changes in the code that break
the image generation, the comparison against the cache will notice it. Note
that there is an error tolerance in the comparison, so minor differences won't
fail.


Requirements
------------
You must have a Python version >= 3.10, as well as PyVista installed
in your environment.

``pyvista>=0.37.0`` and ``vtk>=9.2.2`` are required.

Installation
------------
You can install ``pytest-pyvista`` via `pip`_ from `PyPI`_

.. code-block:: bash 

    pip install pytest-pyvista

Alternatively, you can also install via `conda`_ or `mamba`_ from `conda-forge`_

.. code-block:: bash

    mamba install -c conda-forge pytest-pyvista

Usage
-----
The plugin has two main use cases:

#. Evaluate images generated by calling ``pl.show()`` in unit tests.
#. Evaluate images generated by the
   `Sphinx PyVista Plot Directive <https://docs.pyvista.org/extras/plot_directive.html>`_
   when building documentation.

Specifying multiple cache images
================================
The cache directory is typically flat with no sub-directories. However,
it is possible to specify multiple cache images for a single test by
including a sub-directory with the same name as the test, and including
multiple "valid" cache images in the sub-directory. For example, a
single cached image:

.. code-block:: bash

    cache/my_image.jpg

Can be replaced with multiple versions of the image:

.. code-block:: bash

    cache/my_image/0.jpg
    cache/my_image/1.jpg

.. note::

   - The sub-directory name should match the name of the test.
   - The image names in sub-directories can be arbitrary, e.g. ``0.jpg`` or
     ``foo.jpg``.
   - Nested sub-directories are also supported, and their names can also be arbitrary.
   - Use the ``--generate_subdirs`` flag to automatically generate test images in a
     sub-directory format.

When there are multiple images, the test will initially compare the test image
to the first cached image. If that comparison fails, the test image is then
compared to all other cached images for that test. The test is successful if one
of the comparisons is successful, though a warning is still issued if it initially
failed.

Both use cases (i.e. unit tests and documentation tests) support specifying multiple
cache images.

Unit tests
----------
Once installed, you only need to use the command ``pl.show()`` in your test. The
plugin will automatically manage the cache generation if it does not exist, and
the image comparison itself. Make sure you enable ``pv.OFF_SCREEN`` when loading
PyVista, so the ``pl.show()`` doesn't pop up any window while testing.  By default,
the ``verify_image_cache`` fixture should be used for each test for image comparison:

.. code-block:: python

    import pyvista as pv

    pv.OFF_SCREEN = True


    def test_succeeds(verify_image_cache):
        pl = pyvista.Plotter()
        pl.add_mesh(pyvista.Sphere(), show_edges=True)
        pl.show()


If most tests utilize this functionality, possibly restricted to a module,
a wrapped version could be used

.. code-block:: python

    @pytest.fixture(autouse=True)
    def wrapped_verify_image_cache(verify_image_cache):
        return verify_image_cache


If you need to use any flag inside the tests, you can modify the
``verify_image_cache`` object in the test:

.. code-block:: python

    import pyvista as pv

    pv.OFF_SCREEN = True


    def test_succeeds(verify_image_cache):
        verify_image_cache.windows_skip_image_cache = True
        pl = pyvista.Plotter()
        pl.add_mesh(pyvista.Sphere(), show_edges=True)
        pl.show()


Documentation image tests
-------------------------
Unlike the unit tests, which use the ``verify_image_cache`` fixture to evaluate test
images during each test's teardown, the documentation tests do not use a fixture.
Instead, the documentation tests are invoked with the ``--doc_mode`` flag, and requires
specifying:

#. A target directory which contains all images to be tested. The directory is specified
   using the ``--doc_images_dir`` flag.
#. A cache directory containing all reference images to compare with. The directory is
   specified using the ``--image_cache_dir`` flag.

Since all images must be available for testing, the documentation tests are typically
executed `after` building documentation with Sphinx or some other build process. To test
build images against images in a cache directory use:

.. code-block:: bash

    pytest --doc_mode --doc_images_dir images --image_cache_dir cache

where ``images`` is the target directory of images to test, and ``cache`` is the cache
directory.

When executed, the test will first pre-process the build images. The images are:

#. Collected from the ``images`` directory (including images in nested directories).
#. Resized to a maximum of 400x400 pixels.
#. Renamed so that each file's parent directories are included in the name.
#. Saved as images inside a temporary directory. The directory is flat with no
   sub-directories.

   .. note::
      These temporary images may be saved using the ``--generated_image_dir`` flag.

Next, the pre-processed images are compared to the cached images.
The tests have three main modes of failure:

#.  An image is in the cache but is missing from the build.
#.  An image is in the build but is missing from the cache.
#.  The error threshold when comparing two images is exceeded.

.. note::
   Use the ``--failed_image_dir`` flag to save copies of the images for
   failed tests.

Global flags
------------
These are the flags you can use when calling ``pytest`` in the command line.

Common flags
============
These flags may be used with regular unit testing or documentation testing with
``--doc_mode`` enabled.

* ``--image_cache_dir <DIR>`` sets the image cache directory, relative to `pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>`.
  This will override any configuration, see below.

* ``--generated_image_dir <DIR>`` dumps all generated test images into the provided
  directory, relative to `pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>`.
  This will override any configuration, see below.

* ``--failed_image_dir <DIR>`` dumps copies of cached and generated test images when
  there is a warning or error raised. This directory is useful for reviewing test
  failures. It is relative to `pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>`.
  This will override any configuration, see below.

* ``--generate_subdirs`` saves generated test images in separate sub-directories
  instead of saving them directly to the ``generated_image_dir``. Without this option,
  generated images are saved as ``generated_image_dir/<test_name>.png``; with this
  option enabled, they are instead saved as
  ``<generated_image_dir>/<test_name>/<image_name>.png``, where the image name has the format
  ``<os-version>_<machine>_<gpu-vendor>_<python-version>_<pyvista-version>_<vtk-version>_<using-ci>``.
  This can be useful for providing context about how an image was generated. See the
  ``Customizing test cases`` section for customizing the info.

* Use ``--image_format`` to save test images in either ``png`` or ``jpg`` format.
  ``png`` files are saved by default. Use ``jpg`` to reduce the image file size.
  This will override any configuration, see below.

* Use ``--max_image_size`` to specify a single integer that limits the maximum width or
  height (whichever is larger) of generated images to this value or smaller. By default,
  images have dimensions ``(1024, 768)`` as determined by the ``Plotter.window_size`` or
  ``pyvista.global_theme.window_size``. If enabled, images are resized while preserving
  the aspect ratio. Images are only resized to be smaller, and are not enlarged.

Unit testing flags
==================
These flags are specific to the unit tests. They cannot be used with
``--doc_mode`` enabled.

* ``--reset_image_cache`` creates a new image for each test in
  ``tests/plotting/test_plotting.py`` and is not recommended except for
  testing or for potentially a major or minor release. 

* You can use ``--ignore_image_cache`` if you want to
  temporarily ignore regression testing, e.g. on a particular CI action.

* ``--add_missing_images`` adds any missing images from the test run to the cache.

* ``--reset_only_failed`` reset the image cache of the failed tests only.

* Use ``--allow_unused_generated`` to prevent an error from being raised when a
  test image is generated but not used. A test image is considered "used" if it has a
  corresponding cached image to compare against, or is used to reset or update the
  cache (e.g. if using ``--add_missing_images``). Otherwise, an error is raised by
  default.

* ``--disallow_unused_cache`` report test failure if there are any images in the cache
  which are not compared to any generated images.

* Use ``--allow_useless_fixture`` to prevent test failure when the ``verify_image_cache``
  fixture is used but no images are generated. If no images are generated (i.e. there are
  no calls made to ``Plotter.show()`` or ``mesh.plot()``), then these tests will fail
  by default. Set this CLI flag to allow this globally, or use the test-specific flag
  by the same name below to configure this on a per-test basis.

Documentation testing flags
===========================
These flags are specific to documentation tests. They cannot be used with regular unit
tests.

* ``--doc_mode`` is a required flag for testing documentation images. It configures
  pytest to only collect tests relevant for the image testing.

* ``--doc_images_dir <DIR>`` sets the target directory of images to be tested.

  .. note::

     With Sphinx, build images are typically saved to ``doc/_build/html/_images``.

* Use ``--include_vtksz`` to include tests for interactive plots which are generated by
  PyVista's plot-directive as ``vtksz`` files. When enabled, each ``vtksz`` file
  inside the ``doc_images_dir`` is first converted to HTML, loaded in a web browser,
  and rendered with the same window size as its corresponding static image.
  A screenshot is captured, then compared with the cached image like normal.

  .. note::

      This option requires installing additional packages, as well as installing the
      chromium browser with ``playwright``:

      .. code-block:: bash

          pip install pytest-pyvista[vtksz]
          playwright install chromium

  .. note::

    Enabling this option may increase overall test time considerably due to the additional
    file processing required.

* Use ``--max_vtksz_file_size`` to include tests to limit the size of ``vtksz`` files.
  All ``vtksz`` files inside the ``doc_images_dir`` are collected and compared to the
  specified value. The value should be specified in megabytes (MB). ``20`` is a good
  initial value to set. The value can be customized on a per-test (i.e. per-file) basis,
  see ``Customizing test cases`` for details. Setting this value overrides any configuration,
  see below.

  .. note::

     This option is completely independent from the ``--include_vtksz`` option. File
     sizes may be tested without any additional installation.

Customizing test cases
----------------------
Both the regular unit tests and documentation tests allow for some level of customization.

Customizing unit tests
======================
These are attributes of ``verify_image_cache``. You can set them as ``True`` if needed
in the beginning of your test function.

* ``high_variance_test``: If necessary, the threshold for determining if a test
  will pass or not is incremented to another predetermined threshold. This is
  currently done due to the use of an unstable version of VTK, in stable
  versions this shouldn't be necessary.

* ``windows_skip_image_cache``: For test where the plotting in Windows is different
  from MacOS/Linux.

* ``macos_skip_image_cache``: For test where the plotting in MacOS is different
  from Windows/Linux.

* ``skip``: If you have a test that plots a figure, but you don't want to compare
  its output against the cache, you can skip it with this flag.

* ``allow_useless_fixture``: Set this flag to ``True`` to prevent test failure when the
  ``verify_image_cache`` fixture is used but no images are generated. The value of this
  flag takes precedence over the global flag by the same name (see above).

* ``env_info``: Dataclass for controlling the environment info used to name the generated
  test image(s) when the ``--generate_dirs`` option is used. The info can be test-specific
  or can be modified globally by wrapping the ``verify_image_cache`` fixture, e.g.:

  .. code-block:: python

    @pytest.fixture(autouse=True)
    def wrapped_verify_image_cache(verify_image_cache):
        # Customize the environment info (NOTE: Default values are shown)
        info = verify_image_cache.env_info
        info.prefix: str = ""  # Add a custom prefix
        info.os: bool = True  # Show/hide the os version (e.g. ubuntu, macOS, Windows)
        info.machine: bool = True  # Show/hide the machine info (e.g. arm64)
        info.gpu: bool = True  # Show/hide the gpu vendor (e.g. NVIDIA)
        info.python: bool = True  # Show/hide the python version
        info.pyvista: bool = True  # Show/hide the pyvista version
        info.vtk: bool = True  # Show/hide the vtk version
        info.ci: bool = True  # Show/hide if generated in CI
        info.suffix: str = ""  # Add a custom suffix

        # Alternatively, set a custom string
        verify_image_cache.env_info = 'my_custom_string'

        return verify_image_cache

Customizing documentation tests
===============================
Similar to how the unit tests may be customized using the ``verify_image_cache`` fixture,
the documentation tests can be customized using a ``doc_verify_image_cache`` object.
Instead of a fixture, a pytest hook function is used.

In your ``conftest.py`` file, define a hook function named ``pytest_pyvista_doc_mode_hook``
with ``doc_verify_image_cache`` and ``request`` as arguments. The ``doc_verify_image_cache``
object can then be modified directly on a per-test (i.e. per-image) basis.

For example, a test comparing the build image ``images/foo.png`` to the cached image
``cache/foo.png`` will have the test name ``foo``, and can be modified as shown below.
Currently, only the ``env_info`` can be customized. Refer to the test-specific flags
for unit tests above for more details about customizing the ``env_info`` string.

.. code-block:: python

    def pytest_pyvista_doc_mode_hook(doc_verify_image_cache, request):
        if doc_verify_image_cache.test_name == 'foo':
            doc_verify_image_cache.env_info = 'my_custom_string'
        return doc_verify_image_cache

.. note::
    Customizing the ``env_info`` will affect the generated image's filename, and is only
    relevant if the ``--generate_subdirs`` option is enabled.

Since the regular ``pytest`` ``request`` fixture is also exposed by the hook, users
can further modify the test properties based based on node, markers, or other fixtures.

If the ``--max_vtksz_file_size`` option is used, these tests may similarly be customized
with a ``pytest_pyvista_max_vtksz_file_size_hook``. For example, to set the max
allowed file size to ``50`` for the ``foo.vtksz`` file:

.. code-block:: python

    def pytest_pyvista_max_vtksz_file_size_hook(test_case, request):
        if test_case.test_name == 'foo':
            test_case.max_vtksz_file_size = 50
        return test_case

Configuration
-------------
If using ``pyproject.toml`` or any other 
`pytest configuration <https://docs.pytest.org/en/latest/reference/customize.html>`_
section, consider configuring your test directory location to
avoid passing command line arguments when calling ``pytest``, for example in
``pyproject.toml``:

.. code-block:: toml

    [tool.pytest.ini_options]
    image_cache_dir = "tests/plotting/image_cache"

Additionally, to configure the directory that will contain the generated test images:

.. code-block:: toml

    [tool.pytest.ini_options]
    generated_image_dir = "generated_images"

Similarly, configure the directory that will contain any failed test images:

.. code-block:: toml

    [tool.pytest.ini_options]
    failed_image_dir = "failed_images"

Configure directories for when ``--doc_mode`` is used:

.. code-block:: toml

    [tool.pytest.ini_options]
    doc_failed_image_dir = "failed_test_images"
    doc_generated_image_dir = "generated_test_images"
    doc_image_cache_dir = "tests/doc/doc_image_cache"
    doc_images_dir = "doc/_build/html/_images"

Note that these directories are relative to `pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>`.

Include additional vtksz file tests.

.. code-block:: toml

    [tool.pytest.ini_options]
    include_vtksz = true
    max_vtksz_file_size = 20

Configure the image format to be ``jpg`` for both unit tests and when using ``--doc_mode``.
Limit the image dimensions to 400 pixels as well.

.. code-block:: toml

    [tool.pytest.ini_options]
    image_format = "jpg"
    max_image_size = 400

Or, set them to use different image formats and sizes:

.. code-block:: toml

    [tool.pytest.ini_options]
    image_format = "png"
    max_image_size = 400
    doc_image_format = "jpg"
    doc_max_image_size = 600

Enable the generation of test images inside of sub-directories for both unit tests and when using ``--doc_mode``.

.. code-block:: toml

    [tool.pytest.ini_options]
    generate_subdirs = true

Or, set them to different values:

.. code-block:: toml

    [tool.pytest.ini_options]
    generate_subdirs = true
    doc_generate_subdirs = false

Contributing
------------
Contributions are always welcome. Tests can be run with `tox`_, please ensure
the coverage at least stays the same before you submit a pull request.

License
-------
Distributed under the terms of the `MIT`_ license, ``pytest-pyvista`` is free
and open source software.


Issues
------
If you encounter any problems, please `file an issue`_ along with a detailed
description.

.. _`Cookiecutter`: https://github.com/audreyr/cookiecutter
.. _`@hackebrot`: https://github.com/hackebrot
.. _`MIT`: http://opensource.org/licenses/MIT
.. _`BSD-3`: http://opensource.org/licenses/BSD-3-Clause
.. _`GNU GPL v3.0`: http://www.gnu.org/licenses/gpl-3.0.txt
.. _`Apache Software License 2.0`: http://www.apache.org/licenses/LICENSE-2.0
.. _`cookiecutter-pytest-plugin`: https://github.com/pytest-dev/cookiecutter-pytest-plugin
.. _`file an issue`: https://github.com/pyvista/pytest-pyvista/issues
.. _`pytest`: https://github.com/pytest-dev/pytest
.. _`tox`: https://tox.readthedocs.io/en/latest/
.. _`pip`: https://pypi.org/project/pip/
.. _`PyPI`: https://pypi.org/project
.. _`conda`: https://github.com/conda/conda
.. _`mamba`: https://github.com/mamba-org/mamba
.. _`conda-forge`: https://anaconda.org/conda-forge/pytest-pyvista
