.. _jobs-automation:

Automating Runs
===============

Profiles can be generated either manually on your own (either by individual profilers or using the
``perun collect`` and ``perun postprocess`` commands), or you can use Perun's ``runner``
infrastructure to partially automate the generation process. Perun is capable either to run the
jobs through the stored configuration which is meant for a regular project profiling (either in
local or shared configuration c.f. :doc:`config`) or through a single job specifications meant for
irregular or specific profiling jobs.

Each profile generated by specified batch jobs will be stored in ``.perun/jobs`` directory with the
following name of the template::

   command-collector-workload-Y-m-d-H-M-S.perf

Where ``command`` corresponds to the name of the application (or script), for which we collected
the data using ``collector`` on ``workload`` at given specified date. You can change the template
for profile name generation by setting :ckey:`format.output_profile_template`. New profiles are
annotated with the :preg:`origin` set to the current ``HEAD`` of the wrapped repository. `origin`
serves as a check during registering profiles in the indexes of minor versions. Profile with
`origin` different from the target minor version will not be assigned, as it would violate the
correctness of the performance history of the project. If you want to automatically register the
newly generated profile into the corresponding minor version index, then set
:ckey:`profiles.register_after_run` key to a true value.

.. image:: /../figs/perun-jobs-flow.*
    :align: center
    :width: 100%

The figure above show the overview of the jobs flow in Perun. The runner module is initialized form
user interfaces and from local (or shared) configurations and internally generates the matrix of
jobs which are run in the sequence. Each job is then finished with storing the generated profile in
the internal storage.

.. note::

    In order to obtain fine result, it is advised to run the benchmark several times (at least
    three times) and either do the average over all runs or discard the first runs. This is
    because, initial benchmarks usually have skewed times.

.. note::

    If you do not want to miss profiling, e.g. after each push, commit, etc., check out `git
    hooks`_. ``git hooks`` allows you to run custom scripts on certain git event triggers.

.. _git hooks: https://git-scm.com/book/gr/v2/Customizing-Git-Git-Hooks

Runner CLI
----------

:doc:`cli` contains group of two commands for managing the jobs---``perun run job`` for running one
specified batch of jobs (usually corresponding to irregular measuring or profilings) and ``perun
run matrix`` for running the pre-configured matrix in Yaml_ format specifing the batch job (see
:ref:`jobs-matrix` for full specification). Running the jobs by ``perun run matrix`` corresponds to
regular measuring and profiling, e.g. during end of release cycles, before push to origin/upstream
or even after each commit.

.. click:: perun.cli_groups.run_cli:job
   :prog: perun run job

.. click:: perun.cli_groups.run_cli:matrix
   :prog: perun run matrix

.. _Yaml: http://yaml.org/

.. _jobs-overview:

Overview of Jobs
----------------

Usually during the profiling of application, we first collect the data by the means of profiler (or
profiling data collector or whatever terminology we are using) and we can further augment the
collected data by ordered list of postprocessing phases (e.g. for filtering out unwanted data, normalizing
or scaling the amounts, etc.). As results we generate one profile for each application
configuration and each profiling job. Thus, we can consider one profiling jobs as collection of
profiling data from application of one certain configuration using one collector and ordered set of
postprocessors.

One configuration of application can be partitioned into three parts (two being optional):

   1. The actual **command** that is being profiled, i.e. either the binary or wrapper script that
      is executed as one command from the terminal and ends with success or failure. An example of
      command can be e.g. the ``perun`` itself, ``ls`` or ``./my_binary``.

   2. Set of **arguments** for command (`optional`), i.e. set of parameters or arguments, that are
      supplied to the profiled command. The intuition behind arguments is to allow setting
      various optimization levels or profile different configurations of one program. An example of
      argument (or parameter) can be e.g. ``log``, ``-al`` or ``-O2 -v``.

   3. Input **workloads** (`optional`), i.e. different inputs for profiled command. While workloads
      can be considered as arguments, separating them allows more finer specification of jobs, e.g.
      when we want to profile our program on workloads with different sizes under different
      configurations (since degradations usually manifest under bigger workloads). An example of
      workload can be e.g. ``HEAD`` or ``/dir/subdir`` or ``<< "Hello world"``.

So from the user specification, commands, arguments and workloads can be combined using cartesian
product which yields the list of full application configurations. Then for each such configuration
(like e.g.  ``perun log HEAD``, ``ls -al /dir/subdir`` or ``./my_binary -O2 -v << "Hello world"``)
we run specified collectors and finally the list of postprocessors. This process is automatic
either using the ``perun run job`` or ``perun run matrix``, which differ in the way how the user
specification is obtained.

Each collector (resp. postprocessor) runs in up to three phases (with `pre` and `post` phases being
optional). First the function ``before()`` is executed (if implemented by given collector or
postprocessor), where the collector (resp. postprocessor) can execute additional preparation before
the actual collection (resp. postprocessing) of the data, like e.g. compiling custom binaries. Then
the actual ``collect()`` (resp. ``postprocess()``) is executed, which runs the given job with
specified collection (resp. postprocessing) unit and generatesj profile (potentially in raw or
intermediate format).  Finally the ``after()`` phase is run, which can further postprocess the
generated profile (after the success of collection), e.g. by required filtering of data or by
transforming raw profiles to :ref:`profile-format`. See (:doc:`collectors` and
:doc:`postprocessors` for more detailed description of units). During these phases ``kwargs`` are
passed through and share the specification, or can be used for passing additional information to
following phases. The resulting ``kwargs`` has to contain the ``profile`` key, which contains the
profile w.r.t. :ref:`profile-spec`.

The overall process can described by the following pseudocode::

   for (cmd, argument, workload) in jobs:
      for collector in collectors:
         collector.before(cmd, argument, workload)
         collector.collect(cmd, argument, workload)
         profile = collector.after()
         for postprocessor in postprocessors:
            postprocessor.before(profile)
            postprocessor.postprocess(profile)
            profile = postprocessor.after(profile)

Note that each phase should return the following tripple: (``status code``, ``status message``,
``kwargs``). The ``status code`` is used for checking the success of the called phases and in case
of error prints the ``status message``.

Before this overall process, one can run a custom set of commands by stating the key
:ckey:`execute.pre_run` key. This is mostly meant for compiling of new version or preparing other
necessary requirements before actual collection.

.. image:: /../figs/lifetime-of-profile.*
   :width: 70%
   :align: center

For specification and details about collectors, postprocessors and internal storage of Perun refer
to :doc:`collectors`, :doc:`postprocessors` and :doc:`internals`.

.. _jobs-matrix:

Job Matrix Format
-----------------

In order to maximize the automation of running jobs you can specify in Perun config the
specification of commands, arguments, workloads, collectors and postprocessors (and their internal
configurations) as specified in the :ref:`jobs-overview`. `Job matrixes` are meant for a regular
profiling jobs and should reduce the profiling to a single ``perun run matrix`` command. Both the
config and the specification of job matrix is based on Yaml_ format.

Full example of one job matrix is as follows:

    .. code-block:: yaml

           cmds:
              - perun

           args:
              - log
              - log --short

           workloads:
              - HEAD
              - HEAD~1

           collectors:
              - name: time

           postprocessors:
              - name: regressogram
              - name: regression_analysis
                params:
                 - method: full
                 - steps: 10


Given matrix will create four jobs (``perun log HEAD``, ``perun log HEAD~1``, ``perun log --short
HEAD`` and ``perun log --short HEAD~1``) which will be issued for runs. Each job will be collected
by :ref:`collectors-time` and then postprocessed first by :ref:`postprocessors-regressogram` and then
by :ref:`postprocessors-regression-analysis` with specification ``{'method': 'full', 'steps':
10}``.

Run the following to configure the job matrix of the current project::

    perun config --edit

This will open the local configuration in editor specified by :ckey:`general.editor` and lets you
specify configuration for your application and set of collectors and postprocessors. Unless the
source configuration file was not modified, it should contain a helper comments. The following keys
can be set in the configuration:

.. matrixunit:: cmds

   List of names of commands which will be profiled by set of collectors. The commands should
   preferably not contain any parameters or workloads, since they can be set by different
   configuration resulting into finer specification of configuration.

   .. code-block:: yaml

           cmds:
              - perun
              - ls
              - ./myclientbinary
              - ./myserverbinary

.. matrixunit:: args

   List of arguments (or parameters) which are supplied to profiled commands. It is advised to
   differentiate between arguments/parameters and workloads. While their semantics may seem close,
   separation of this concern results into more verbose performance history

   .. code-block:: yaml

           args:
              - log
              - log --short
              - -al
              - -q -O2

.. matrixunit:: workloads

   List of workloads which are supplied to profiled commands. Workloads represents program inputs
   and supplied files.

   .. code-block:: yaml

           workloads:
              - HEAD
              - HEAD~1
              - /usr/share
              - << "Hello world!"

   From version 15.1. you can use the workload generators instead. See
   :ref:`jobs-workload-generators` for more information about supported workload generators and
   :ckey:`generators.workload` for more information how to specify the workload generators in the
   configuration files.

.. matrixunit:: collectors

   List of collectors used to collect data for the given configuration of application represented
   by commands, arguments and workloads. Each collector is specified by its `name` and additional
   `params` which corresponds to the dictionary of (key, value) parameters. Note that the same
   collector can be specified more than once (for cases, when one needs different collector
   configurations). For list of supported collectors refer to :ref:`collectors-list`.

   .. code-block:: yaml

           collectors:
              - name: memory
                params:
                    - sampling: 1
              - name: time

.. matrixunit:: postprocessors

   List of postprocessors which are used after the successful collection of the profiling data.
   Each postprocessor is specified by its `name` and additional `params` which corresponds to the
   dictionary of (key, value) parameters. Note that the same postprocessor can be specified more
   than just once. For list of supported postprocessors refer to :ref:`postprocessors-list`.

   .. code-block:: yaml

           postprocessors:
              - name: regressogram
              - name: regression_analysis
                params:
                 - method: full
                 - steps: 10

.. _jobs-workload-generators:

List of Supported Workload Generators
-------------------------------------

.. automodule:: perun.workload

Generic settings
^^^^^^^^^^^^^^^^

.. automodule:: perun.workload.generator

Singleton Generator
^^^^^^^^^^^^^^^^^^^

.. automodule:: perun.workload.singleton_generator

Integer Generator
^^^^^^^^^^^^^^^^^

.. automodule:: perun.workload.integer_generator

String Generator
^^^^^^^^^^^^^^^^

.. automodule:: perun.workload.string_generator

Text File Generator
^^^^^^^^^^^^^^^^^^^

.. automodule:: perun.workload.textfile_generator
