# Papersweep
> Combining papermill and wandb sweeps for frictionless experiments with notebooks.


I often find myself coding a machine learning experiment in a Jupyter Notebook, using [wandb](https://www.wandb.com/) to visualize and track the results of the runs. When the experiment is drafted, I always have questions such as: How will the performance be affected by the parameter `a`? What if I change the number of items of the dataset, or change the dataset completely?

[wandb sweeps](https://docs.wandb.com/sweeps) are a great tool to solve these questions. However, sweeping requires that you define a specific training function for the sweep, which I find redundant, specially when the code for training is already in the Jupyter Notebook. Furthermore, if I make some changes in the original notebook, I have to be sure that I change the sweep function too.

This library just provides a single command, `papersweep`, which uses the library [papermill](https://github.com/nteract/papermill) to execute a notebook as the function of a wandb sweep. The only thing that has to be changed in the notebook is the way the config parameters are declared in `wandb.config`.

As an example, if `a` is a parameter in your notebook declared as:

```python
wandb.config.a = 3
```

Just changing that line to:

```python
wandb.config.a = ifnone(wandb.config.get('a'), 3)
```

will use the default value `3` in case the notebook is executed as a standalone run (i.e, without a sweep), and in case the notebook is executed as a sweep function, it will use the value injected from the sweep configuration. This provides a frictionless way of using your Jupyter Notebooks both for single runs and sweep functions.

## Install

`pip install papersweep`

## How to use

`--help` provides command help.

```
$ 
```

`input_nb` contains a path to the notebook with the experiment you want to use as function of the sweep.

`sweep_config` is a path to a YAML file with the configuration of the sweep. An example is given in `examples/sweep_config.yaml`. More information aboyt sweep configurations in the [official docs](https://docs.wandb.com/sweeps/configuration).

`pm_params` is a YAML file with extra configuration for the notebook execution aside from the sweep parameters. Those parameters will be injected in the notebook by `papermill`, so they need to be placed in one cell tagged as `parameters` (See the documentation of `papermill` to see how to tag a cell in a Jupyter Notebook).

`sweep_id` allows to reuse an already existing sweep instead of creating a new one.

## Example with time series classification

The notebook `_example_tsai.ipynb` trains a time series classifier using deep learning with the library `tsai`(https://github.com/timeseriesAI/tsai). The dataset (`dsid`) and the deep learning architecture (`arch`) are part of the `wandb.config` configuration parameters. 

The file `examples/sweep_config.yaml` gives grid-like experiment in which multiple datasets and architectures are tried, looking for the combination that achieves better accuracy

Running the following commandin a terminal:

`papersweep --input_nb ./_example_tsai.ipynb --sweep_config ./examples/sweep_config.yaml ---entity vrodriguezf --project papersweep`

will run the notebook `./example_tsai.ipynb` once for every iteration of the sweep, and log the results [in a dashboard](https://wandb.ai/vrodriguezf/papersweep/sweeps/qh09r37b?workspace=user-vrodriguezf) that you can interact and play with.
