Metadata-Version: 2.0
Name: qbatch
Version: 1.0
Summary: Execute shell command lines in parallel on SGE/PBS clusters
Home-page: https://github.com/pipitone/qbatch
Author: Jon Pipitone, Gabriel A. Devenyi
Author-email: jon@pipitone.ca, gdevenyi@gmail.com
License: Unlicense
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: License :: Public Domain
Classifier: Natural Language :: English
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Clustering
Classifier: Topic :: System :: Distributed Computing
Classifier: Topic :: Utilities

qbatch
======

Execute shell command lines in parallel on SGE/PBS clusters

|Travis CI build status|

qbatch is a tool for executing commands in parallel across a compute
cluster. It takes as input a list of *commands* (shell command lines or
executable scripts) in a file or piped to qbatch. The list of commands
are divided into arbitrarily sized *chunks* which are submitted as jobs
to the cluster either as individual submissions or an array. Each job
runs the commands in its chunk in parallel. Commands can also be run
locally on systems with no cluster capability.

Installation
------------

.. code:: sh

    $ pip install qbatch

Dependencies
------------

qbatch requires python (2 or 3) and `GNU
Parallel <https://gnu.org/s/parallel>`__. For Torque/PBS and gridengine
clusters, qbatch requires job submission access via the ``qsub``
command.

Environment variable defaults
-----------------------------

qbatch supports several environment variables to customize defaults for
your local system.

.. code:: sh

    $ export QBATCH_PPJ=12                   # requested processors per job
    $ export QBATCH_CHUNKSIZE=$QBATCH_PPJ    # commands to run per job
    $ export QBATCH_CORES=$QBATCH_PPJ        # commonds to run in parallel per job
    $ export QBATCH_MEM="0"                  # requested memory per job
    $ export QBATCH_MEMVARS="mem"            # memory request variable to set
    $ export QBATCH_SYSTEM="pbs"             # queuing system to use
    $ export QBATCH_NODES=1                  # (PBS-only) nodes to request per job
    $ export QBATCH_PE="smp"                 # (SGE-only) parallel environment name

These correspond to the same named options in the qbatch help output
above.

Some examples:
--------------

.. code:: sh

    # Submit an array job from a list of commands (one per line)
    # Generates a job script in ./.scripts/ and job logs appear in ./logs/
    $ qbatch commands.txt

    # Set the walltime for each job
    $ qbatch -w 3:00:00 commands.txt

    # Run 24 commands per job
    $ qbatch -c24 commands.txt

    # Request 1# Run 24 commands per job, but run 12 in parallel at a time
    $ qbatch -c24 -j12 commands.txt

    # Start jobs after successful completion of existing jobs with names starting with "stage1_"
    $ qbatch --afterok 'stage1_*' commands.txt

    # Pipe a list of commands to qbatch 
    $ parallel echo process.sh {} ::: *.dat | qbatch -

    # Run jobs locally with GNU Parallel, 12 commands in parallel
    $ qbatch -b local -j12 commands.txt

    # Many options don't make sense locally: chunking, individual vs array, nodes,
    # ppj, highmem, and afterok are ignored

.. |Travis CI build status| image:: https://travis-ci.org/pipitone/qbatch.svg?branch=master
   :target: https://travis-ci.org/pipitone/qbatch


