Metadata-Version: 2.1
Name: simple-gpu-scheduler
Version: 0.1.2
Summary: A simple scheduler for running commands on multiple GPUs.
Home-page: https://github.com/ExpectationMax/simple_gpu_scheduler
Author: Max Horn
Author-email: maexlich@gmail.com
License: UNKNOWN
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.6
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Requires-Python: >3.6
Description-Content-Type: text/markdown

simple_gpu_scheduler
--------------------

A simple scheduler to run your commands on individual GPUs.  Following the
[KISS principle](https://en.wikipedia.org/wiki/KISS_principle), this script
simply accepts commands via `stdin` and executes them on a specific GPU by
setting the `CUDA_VISIBLE_DEVICES` variable.

The commands read are executed using the login shell, thus redirections `>`
pipes `|` and all other kinds of bash magic can be used.

Installation
------------

The package can simply be installed from [pypi](https://pypi.org/)
```bash
$ pip install simple_gpu_scheduler
```

Example
-------

To show how this generally works, we will create jobs that simply outputs
a job id and the value of `CUDA_VISIBLE_DEVICES`:

```bash
for i in {0..10}; do echo "echo job_id=$i device=\$CUDA_VISIBLE_DEVICES && sleep 3"; done | simple_gpu_scheduler --gpus 0,1,2
```

which results in the following output:

```
Processing `command echo job_id=0 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 2
Processing `command echo job_id=1 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 1
Processing `command echo job_id=2 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 0
job_id=0 device=2
job_id=1 device=1
job_id=2 device=0
--- 3 seconds no output ---
Processing command `echo job_id=3 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 2
Processing command `echo job_id=4 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 1
Processing command `echo job_id=5 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 0
job_id=3 device=2
job_id=4 device=1
job_id=5 device=0
--- 3 seconds no output ---
Processing command `echo job_id=6 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 2
Processing command `echo job_id=7 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 1
Processing command `echo job_id=8 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 0
job_id=6 device=2
job_id=7 device=1
job_id=8 device=0
--- 3 seconds no output ---
Processing command `echo job_id=9 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 2
Processing command `echo job_id=10 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 0
job_id=9 device=2
job_id=10 device=0
```

This is equivalent to creating a file `commands.txt` with the following content:

```bash
echo job_id=0 device=$CUDA_VISIBLE_DEVICES && sleep 3
echo job_id=1 device=$CUDA_VISIBLE_DEVICES && sleep 3
echo job_id=2 device=$CUDA_VISIBLE_DEVICES && sleep 3
echo job_id=3 device=$CUDA_VISIBLE_DEVICES && sleep 3
echo job_id=4 device=$CUDA_VISIBLE_DEVICES && sleep 3
echo job_id=5 device=$CUDA_VISIBLE_DEVICES && sleep 3
echo job_id=6 device=$CUDA_VISIBLE_DEVICES && sleep 3
echo job_id=7 device=$CUDA_VISIBLE_DEVICES && sleep 3
echo job_id=8 device=$CUDA_VISIBLE_DEVICES && sleep 3
echo job_id=9 device=$CUDA_VISIBLE_DEVICES && sleep 3
echo job_id=10 device=$CUDA_VISIBLE_DEVICES && sleep 3
```

and running
```bash
simple_gpu_scheduler --gpus 0,1,2 < commands.txt
```

Simple scheduler for jobs
-------------------------

Combined with some basic command line tools, one can set up a very basic
scheduler which waits for new jobs to be "submitted" and executes them in order
of submission.

Setup and start scheduler in background or in a separate permanent session
(using for example `tmux`):
```bash
touch gpu.queue
tail -f -n 0 gpu.queue | simple_gpu_scheduler --gpus 0,1,2
```
the command `tail -f -n 0` follows the end of the gpu.queue file. Thus if there
was anything written into `gpu.queue` prior to the execution of the command it
will not be passed to `simple_gpu_scheduler`.

Then submitting commands boils down to appending text to the `gpu.queue` file:

```bash
echo "my_command_with | and stuff > logfile" >> gpu.queue
```
TODO
----

 - Multi line jobs (evtl. we would then need a submission script after all)
 - Stop, but let commands finish when receiving a defined signal


