Introduction
============

SciExp² (aka *SciExp square* or simply *SciExp2*) stands for *Scientific Experiment Exploration*, which provides a framework for easing the workflow of creating, executing and evaluating experiments.

The driving idea of SciExp² is that of quick and effortless *design-space exploration*. It is divided into the following main pieces:

* **Launchgen**: Aids in defining experiments as a permutation of different parameters in the design space. It creates the necessary files to run these experiments (configuration files, scripts, etc.), which you define as templates that get substituted with the specific parameter values of each experiment.

* **Launcher**: Takes the files of `~sciexp2.launchgen` and runs these experiments on different execution platforms like regular local scripts or cluster jobs. It takes care of tracking their correct execution, and allows selecting which experiments to run (e.g., those with specific parameter values, or those that were not successfully run yet).

* **Data**: Aids in the process of collecting and analyzing the results of the experiments. Results are automatically collected into a data structure that maintains the relationship between each result and the parameters of the experiment that produced it. With this you can effortlessly perform complex tasks such as inspecting the results or calculating statistics of experiments sub-set, based on their parameter values.


.. _quick_example:

Quick example
-------------

As a quick example, here's how to generate scripts to run an application, run these scripts, and evaluate their results. First, we'll start by generating the per-experiment scripts in the ``experiments`` directory. Each experiment script will execute ``my-program`` with different values of the ``--size`` argument, and we can reference these per-experiment parameters with ``@parameter_name@``::

  #!/usr/bin/env python
  # -*- python -*-

  from sciexp2.launchgen.env import *

  l = Launchgen(out="experiments")

  # copy program into experiments directory
  l.pack("/path/to/my-program", "bin/my-program")

  # create one experiment for each value of size
  l.params(size=[1, 2, 4, 8])

  # generate per-experiment scripts ("scripts/@size@.sh") with the specified command (CMD)
  l.launchgen("shell", "scripts/@size@.sh",
              CMD="bin/my-program --size=@size@ --out=results/@size@.csv")


The resulting ``experiments`` directory now contains all the files we need::

  experiments
  |- jobs.jd    # auto-generated by l.launchgen()
  |- bin
  |  `- my-program
  `- scripts
     |- 1.sh
     |- 2.sh
     |- 4.sh
     `- 8.sh


To execute all the experiments, we can simply use the auto-generated ``jobs.jd`` script. It will take care of running each of the per-experiment scripts, controlling if they are finishing correctly::

  ./experiments/jobs.jd submit


After successfully executing all the scripts, the ``experiments`` directory will also contain the output files we gave on the per-experiment command (``--out=results/@size@.csv``)::

  experiments
  |- bin
  |  `- my-program
  |- scripts
  |  |- 1.sh
  |  |- 2.sh
  |  |- 4.sh
  |  `- 8.sh
  `- results
     |- 1.csv
     |- 2.csv
     |- 4.csv
     `- 8.csv


Now let's assume that ``my-program`` runs the same operation multiple times, and the output file is in CSV format with one line for each run::

  run,time
  0,3.2
  1,2.9
  ...


We therefore have one result file for each ``size`` parameter value, and each of these results files contains multiple runs of its experiment. A typical step would now be to collect all these results and calculate, for each ``size`` configuration, the mean execution time across all runs::

  #!/usr/bin/env python
  # -*- python -*-

  from sciexp2.data.env import *

  # auto-extract all results; the result is a 1-dimensional array with two
  # metadata variables: size and run
  d = extract_txt('experiments/results/@size@.csv',
                  fields_to_vars=["run"])

  # turn the data into a 2-dimensional array, with experiment size as first
  # dimension and run number as second
  d = d.reshape(["size"], ["run"])

  # get the mean of all runs, so we get a 1-dimensional array with one mean
  # per size
  d = d.mean(axis="run")

  # print CSV-like mean of each size
  print("size, time")
  for foo in d.dims["size"]:
      print("%4d," % size, d[size])


The result could be something like::

  size, time
     1, 3.05
     2, 3.39
     4, 4.61
     8, 6.37
