# Finesse Benchmarking

The file `example_benchmarks.py`  in `tests/testutils` (so that this is importable by the test modules) defines a series of functions. These are autogenerated from the example rst files in `docs/source/examples` by the script in `scripts/examples_to_script.py`. The `test_benchmarks.py` file defines pytest tests that can be executed with the [pytest-benchmark](https://pytest-benchmark.readthedocs.io/en/latest/usage.html) plugin.

## Regenerate benchmark file

If the examples get updated, the benchmarks would need to be updated as well. Run

```bash
python scripts/examples_to_script.py
```

to regenerate the file. If the names of the rst files (and thus the names of the functions) have changed, the imports in `tests/benchmarks/test_benchmarks.py` would need to be updated.

## Local usage

Since the benchmarks are disabled in the pytest configuration (meaning all the tests with a benchmark fixture only get run once, not multiple times), you can simply run the tests normally:

```bash
pytest tests
```

To run only the benchmarks:

```bash
pytest --benchmark-enable --benchmark-only --benchmark-autosave tests
```

Which will display a table with benchmarking results in the standard output of pytest. A json report is saved in the `.benchmarks` folder, in a subfolder depending on your operating system and python version. The filename of the json file will reference the current commit.

```bash
pytest-benchmark list
```

Will list all the benchmarks in the `.benchmarks` folder and

```bash
pytest-benchmark compare file_1 file_2
```

will produce a table in stdout with comparison between the benchmarks.

## finesse3-benchmark repository

The benchmarks are ran during the test phase of the pipelines. It pushes the resulting json files to another repository: [finesse3-benchmark](https://gitlab.com/ifosim/finesse/finesse3-benchmark). In the future this repo will also contain some tools for easy comparisons, plots etc.
