Benchmark
=========

We benchmarked several Python audio reading libraries
against each other.
The procedure follows the `python_audio_loading_benchmark project`_.


Procedure
---------

The benchmark loads 160 single channel audio files
and measures the time until the audio is converted
to a :class:`numpy.array`.

Audio files
^^^^^^^^^^^

All files have a sampling rate of 44100 Hz,
and contain white noise.
They are generated by sox.
They differ in length between 1 second and 151 seconds,
in 10 seconds steps, including 10 files per length.
Leading to an overall of 160 files per audio format.
Starting from the WAV files,
FLAC, OGG, MP3, and MP4 files were generated using ffmpeg_.

Python packages
^^^^^^^^^^^^^^^

The following Python packages are benchmarked against each other:

* aubio_ 0.4.9
* audioread_ 2.1.9
* :mod:`audiofile` 0.3.1
* librosa_ 0.8.0
* scipy_ 1.5.3
* soundfile_ 0.10.3.post1
* sox_ 1.4.1

scipy_ and librosa_ are only tested for reading files,
whereas sox_ is only tested for accessing metadata information.
audioread_ can use three different libraries under the hood:
ffmpeg_, gstreamer_, mad_.
All three are benchmarked,
but results are only reported for the best one.

Reading files
^^^^^^^^^^^^^

The benchmark loads the audio files
and measures the time until the audio is converted
to a :class:`numpy.array`.

Accessing metadata
^^^^^^^^^^^^^^^^^^

For benchmark accessing metadata information,
the following was requested for every file:

* channels
* duration
* samples
* sampling rate

Running the benchmark
^^^^^^^^^^^^^^^^^^^^^

The benchmark was executed on the following machine:

* CPU: Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz x 12
* RAM: '15.37 GiB'
* Hard drive: Samsung SSD 860
* Linux: Ubuntu 18.04.5
* Python: 3.6.9

To rerun the benchmark yourself,
clone the repository
and execute:

.. code-block:: bash

    $ cd docs/benchmark/
    $ bash generate_audiofiles.sh
    $ # Install dependencies for building wheels
    $ sudo apt-get install -y libcairo2-dev libmad0-dev libgirepository1.0-dev python3-gst-1.0
    $ # Create and activate Python virtual environment, e.g.
    $ # virtualenv --no-download --python=python3 ${HOME}/.envs/audiofile-benchmark
    $ # source ${HOME}/.envs/audiofile-benchmark/bin/activate
    $ pip install -r requirements.txt.lock
    $ bash run.sh


WAV, FLAC, OGG
--------------

Reading files
^^^^^^^^^^^^^

audioread_ has been removed from the results
as it was the slowest library.

.. image:: ./benchmark/results/benchmark_wav-flac-ogg_read.png

Results for :mod:`audiofile`, soundfile_ and librosa_ are identical here
as all of them use soundfile_ under the hood to read the data.

Accessing metadata
^^^^^^^^^^^^^^^^^^

sox_ and audioread_ have been removed from the results
as they were at least one magnitude slower.
scipy_ is only meant for reading WAV files.

.. image:: ./benchmark/results/benchmark_wav-flac-ogg_info.png


MP3, MP4
--------

Reading files
^^^^^^^^^^^^^

aubio_, soundfile_, and sox_ do not support
reading MP3 and MP4 files.

.. image:: ./benchmark/results/benchmark_mp3-mp4_read.png

Accessing metadata
^^^^^^^^^^^^^^^^^^

aubio_, librosa_, soundfile_, and sox_ do not support
accessing MP3 and MP4 metadata.

.. image:: ./benchmark/results/benchmark_mp3-mp4_info.png

For MP3 and MP4 files,
:mod:`audiofile` is not very fast in accessing the metadata.
The main focus was not speed,
but consistent results for number of samples and duration.
This can only be achieved by first converting the file to WAV
as the duration will depend on the used decoder.


.. _aubio: https://github.com/aubio/aubio/
.. _audioread: https://github.com/beetbox/audioread/
.. _ffmpeg: https://ffmpeg.org/
.. _gstreamer: https://gstreamer.freedesktop.org/
.. _librosa: https://github.com/librosa/librosa/
.. _mad: https://sourceforge.net/projects/mad/
.. _scipy: https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.wavfile.read.html
.. _soundfile: https://github.com/bastibe/SoundFile/
.. _sox: https://github.com/rabitt/pysox/
.. _python_audio_loading_benchmark project: https://github.com/faroit/python_audio_loading_benchmark
