Metadata-Version: 2.1
Name: nbx
Version: 0.2.2
Summary: This module enables you to quickly convert your jupyter notebook into a bundle of files that can be run on BCS'*OpenMind*
Home-page: https://github.com/mirkoklukas/nbx
Author: Mirko Klukas
Author-email: mirko.klukas@gmail.com
License: Apache Software License 2.0
Keywords: openmind; mit
Platform: UNKNOWN
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Requires-Python: >=3.5
Description-Content-Type: text/markdown

<!--

#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: index.ipynb
# command to build the docs after a change: nbdev_build_docs

-->

# NBX: Notebook Experiments for OpenMind

> This module enables you to quickly convert your jupyter notebook into a bundle of files that can be run on BCS'*OpenMind*. 


# Getting started

## Prerequisites
#### Install the package:

- `pip install nbx`

#### Get a singulartiy image
You'll need an image that has the package installed (there are ways around that, but I am keeping it simple at the moment). Here's how you can build an image:

```
module load openmind/singularity/3.2.0
export SINGULARITY_CACHEDIR="/om2/user/{your_user_name}/.singularity"
singularity build pytorch.simg docker://mklukas/pytorch
```

#### Environment variables
For the modules to work you have to set the environment variables `om`, `omx`, `omsimg`, and `omid`:  

- **om**: your login to *OpenMind*. 
    - You need to enable logging into *OpenMind* using public key authentication. That means the command `ssh $om` should log you in whithout asking for a password. (googling for "ssh public key authentication" will provide you with a recipe like [this](https://kb.iu.edu/d/aews))
- **omx**: path to the folder where *nbx* bundles are stored. This path will automatically be added to your python path. Any modules that are not part of your bundle's `src/` folder or are included in your singularity container should go here.
- **omsimg**: path to the folder containing your singularity images
- **omid**: your Open Mind username 

Mac users can adapt and copy the following lines to their `.bash_profile` file

```
export om={your_user_name}@openmind7.mit.edu
export omid={your_user_name}
export omx=/om2/user/{your_user_name}/nbx-experiments
export omsimg=/om2/user/{your_user_name}/simg
```



## Usage

- Put your python scripts that you wrote for this experiment in a folder called `src`. The folder will be copied to the bundle so the scripts are available on *OpenMind* as well.
- **#nbx**: Each cell that contains a `#nbx` tag in its **first** line will be considered part of the experiment.
- **#xarg**: Putting `#xarg` above a variable declaration makes this variable *explicit*, it will become an argument of the experiment function. Any iterable to the right of the variable declaration, **separated by a semicolon**, will be considered the domain that will be swept during the parameter sweep.
- Each nbx-experiment has to declare the variables `task_id` and `results_dir`. The *task id* will be set by the *wrapper* script and enumerates the configurations of the parameter space. The latter variable will also be set by the *wrapper* script, it will be replaced by the folder automatically created for a specific parameter configuration. 

# Example

## Experiment

In every experiment we need to indicate which cells are part of it (using the `#nbx` flag), and need to specify these two arguments:
- `task_id`
- `results_dir`
<div class="codecell" markdown="1">
<div class="input_area" markdown="1">

```python
%matplotlib inline
%load_ext autoreload
%autoreload 2
```

</div>

</div>
<div class="codecell" markdown="1">
<div class="input_area" markdown="1">

```python
#nbx
#xarg
task_id = 0
#xarg
results_dir = "."
```

</div>

</div>

This cell will be part of the experiment
<div class="codecell" markdown="1">
<div class="input_area" markdown="1">

```python
#nbx

#xarg
x=0; range(5)

#xarg
y=0; [0,1,2,4]

z=0;

# ...
```

</div>

</div>

This cell will also be part of the experiment. The output will be written to a log file in the `io` folder that will automatically be created. 
<div class="codecell" markdown="1">
<div class="input_area" markdown="1">

```python
#nbx
print("my results:", x, y, z)
```

</div>
<div class="output_area" markdown="1">

    my results: 0 0 0


</div>

</div>

Note how we used the variable `results_dir`. It will will be replaced by `"results/task_id/"`; a corresponding folder will automatically be created. It is really just a hook so we can manipulate it behind the scenes.
<div class="codecell" markdown="1">
<div class="input_area" markdown="1">

```python
#nbx
with open(f"{results_dir}/your_file.txt", "w") as f:
    f.write("I will be written to: example_nbx_bundle/results/task_id/your_file.txt")
    f.write(f"\n{task_id}")
```

</div>

</div>

## Creating and running an NBX bundle

To run the experiment on OM we have to create a bundle that we can interact with...
<div class="codecell" markdown="1">
<div class="input_area" markdown="1">

```python
from nbx.om import NbxBundle

bundle = NbxBundle(nbname="index.ipynb", # the name of the notebook to use as exp
          name="example_bundle",         # name of the bundle
          linting=False,                 # enable basic linting
          time=[0,20],                   # comp time [hours, minutes]
          ntasks=4,                      # requested comp nodes
          step=50,                       # parallel jobs (compare bundle/run.sh)
          max_arr=10,                    # maximum number of queued jobs on OM is 1000
          mail_user="me@somewhere.com",  # notification email
          simg="pytorch.simg")           # singulrity img on OM in $omsimg
```

</div>
<div class="output_area" markdown="1">


    ** nbx bundle created **
    Path:
        example_bundle_nbx

    Source nb:
        index.ipynb

    Parameters (#configs 20):
        * x = range(5)
        * y = [0,1,2,4]
          task_id = 0
          results_dir = "."

    Instructions:
        Copy to remote, run the bash script, and pull the results
        - `bundle.push()` or `scp -r example_bundle_nbx $om:$omx`
        - `bundle.run()` or `ssh $om sbatch -D $omx/example_bundle_nbx $omx/example_bundle_nbx/run.sh`
        - `bundle.pull_results()` or `scp -r $om:$omx/example_bundle_nbx/results ./results`


</div>

</div>
<div class="codecell" markdown="1">
<div class="input_area" markdown="1">

```python
!ls example_bundle_nbx/
```

</div>
<div class="output_area" markdown="1">

    __init__.py   experiment.py job.sh        wrapper.py
    [34m__pycache__[m[m   [34mio[m[m            run.sh


</div>

</div>
<div class="codecell" markdown="1">
<div class="input_area" markdown="1">

```python
from example_bundle_nbx.experiment import sweep_params as sweep

print(len(sweep))
print(sweep[0])
print(sweep[1])
print(sweep[4])
```

</div>
<div class="output_area" markdown="1">

    12
    {'x': 0, 'y': 0}
    {'x': 0, 'y': 1}
    {'x': 1, 'y': 0}


</div>

</div>
<div class="codecell" markdown="1">
<div class="input_area" markdown="1">

```python
bundle.push()
```

</div>

</div>
<div class="codecell" markdown="1">
<div class="input_area" markdown="1">

```python
bundle.run()
```

</div>

</div>
<div class="codecell" markdown="1">
<div class="input_area" markdown="1">

```python
bundle.status()
```

</div>

</div>
<div class="codecell" markdown="1">
<div class="input_area" markdown="1">

```python
bundle.pull_results()
```

</div>

</div>

The results are now in the local folder:
<div class="codecell" markdown="1">
<div class="input_area" markdown="1">

```python
!ls example_bundle_nbx
```

</div>

</div>


