/*
 * Copyright (c) 2012-2013:  G-CSC, Goethe University Frankfurt
 * Author: Martin Rupp
 * 
 * This file is part of UG4.
 * 
 * UG4 is free software: you can redistribute it and/or modify it under the
 * terms of the GNU Lesser General Public License version 3 (as published by the
 * Free Software Foundation) with the following additional attribution
 * requirements (according to LGPL/GPL v3 §7):
 * 
 * (1) The following notice must be displayed in the Appropriate Legal Notices
 * of covered and combined works: "Based on UG4 (www.ug4.org/license)".
 * 
 * (2) The following notice must be displayed at a prominent place in the
 * terminal output of covered works: "Based on UG4 (www.ug4.org/license)".
 * 
 * (3) The following bibliography is recommended for citation and must be
 * preserved in all covered files:
 * "Reiter, S., Vogel, A., Heppner, I., Rupp, M., and Wittum, G. A massively
 *   parallel geometric multigrid solver on hierarchically distributed grids.
 *   Computing and visualization in science 16, 4 (2013), 151-164"
 * "Vogel, A., Reiter, S., Rupp, M., Nägel, A., and Wittum, G. UG4 -- a novel
 *   flexible software system for simulating pde based models on high performance
 *   computers. Computing and visualization in science 16, 4 (2013), 165-179"
 * 
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
 * GNU Lesser General Public License for more details.
 */

/**	\page pageugsubmit ugsubmit - Job Scheduling on Clusters

[TOC]

<hr>
\section secIntroduction Introduction
<hr>
 
 To use the scripts described here, you have to set up ug4's \ref pageBashTools .
 
\ref secugsubmit , \ref secuginfo and \ref secugcancel are scripts to unify the access to job schedulers over different clusters.<br>
Supported clusters are <tt> NecNehalem, Hermit, Jugene, Juqueen, mpi, cekon, moab </tt>.<br>

For this you can set some environment variables (here for bash)
\verbatim
export UGSUBMIT_TYPE=Hermit
export UGSUBMIT_EMAIL=mail@you.de
\endverbatim
Alternatively, you can use the arguments <tt>-cluster=Hermit</tt> and <tt>-email=mail\@you.de</tt> .<br>
Now you can run a job like this:

\verbatim
Hermit [~/ug4/bin] ugsubmit 64 -mail-all -walltime 00:30:00 --- ./ugshell -ex laplace.lua -numPreRefs 6 -numRefs 9
\endverbatim

and you get this output
\verbatim
ugsubmit 0.1. (c) Goethe-Center for Scientific Computing 2012

Using qsub on Hermit/Cray XE6
qsub -V ./bin/job.64.2012-03-07.10:27:34/job.sh
Submitting batch job to queue ...
Recieved PBSid: 
42851.sdb
\endverbatim

we can look if it really runs with uginfo:

\verbatim
Hermit [~/ug4/bin] uginfo
uginfo 0.1. (c) Goethe-Center for Scientific Computing 2012
Using Hermit
Job id                    Name             User            Time Use S Queue
42851.sdb                  ...3-07.10:27:34 rupp            00:00:01 C multi          
42853.sdb                  ...3-07.10:29:00 rupp                   0 R multi          
42854.sdb                  ...3-07.10:29:13 rupp                   0 R single 
\endverbatim


You can also write testscripts using a bunch of ugsubmits, and use them on a wide range of clusters 
like this:
\verbatim
#!/bin/bash
ugsubmit 1  --- ./ugshell -ex laplace.lua -numRefs 6
ugsubmit 16 --- ./ugshell -ex laplace.lua -numPreRefs 4 -numRefs 8
ugsubmit 64 --- ./ugshell -ex laplace.lua -numPreRefs 5 -numRefs 9
ugsubmit 256  -mail-end --- ./ugshell -ex laplace.lua -numPreRefs 6 -numRefs 10
ugsubmit 1024 -mail-end --- ./ugshell -ex laplace.lua -numPreRefs 7 -numRefs 11
\endverbatim

\note The Default Walltime is 00:10:00, thats 10 minutes. After that, your job will be canceled. 

<hr>
\section secOtherScripts Other Scripts, Improvements
<hr>

If you have other scripts which start job schedulers, please mail them to me, and tell me how
to make ugsubmit better. You have an option which you can't live without? Perhaps you can include it
into ugsubmit, then everyone can benefit. Also if you come to new clusters currently not
supported in ugsubmit, we can incorporate this into ugsubmit. Change it in the repro or write an email
to <a href="mailto:martin.rupp@gcsc.uni-frankfurt.de">martin.rupp@gcsc.uni-frankfurt.de</a> .

<hr>
\section secugsubmit ugsubmit
<hr>

ugsubmit can be located in <tt> /ug4/scripts/shell/ugsubmit </tt>. To make it available you have to 
<tt>source YOUR_UG4_PATH/ug4/scripts/shell/ugpath</tt>.

\verbatim
ugsubmit <npe> [optional Parameters] --- <binary> <arg1> <arg2> ...
\endverbatim

-        \<npe\>                  : number of processes
-        \<binary\>               : the binary
-        \<args\>                 : arguments for execution of binary

 optional Parameters are:
-        -test                  : do not actually execute, only write scripts (also sets -verbose)
-        -nppn \<nppn\>           : number of processes per node (optional, default maximum nr. on cluster)
-        -cluster \<type\>        : cluster to use. see below (or use env. variable UGSUBMIT_TYPE)
-        -name \<jobname\>        : a jobname to be used (optional, default "job")
-        -walltime \<hh:mm:ss\>   : max walltime for job to run (optional; default is 00:10:00)
-        -email \<emailadress\>   : emailadress. you can also use the environment variable UGSUBMIT_EMAIL.
-        -mail-start            : mail when job starts
-        -mail-end              : mail when job ends
-        -mail-error            : mail on error
-        -mail-all              : 
-        -verbose               : enable verbose mode
-        -tail                  : display the output (as tail -f. job runs even if you hit Ctrl-C)
-        -i                     : interactive (not fully supported)
-        -dir                   : directory to use

 
 Supported Cluster Types: NecNehalem, Hermit, Jugene, mpi, cekon, moab.
 
 Jugene Parameters
-        -Jugene-mode            : VN, DUAL or SMP. Default VN
-        -Jugene-mapfile         : Mapfile. Default TXYZ
-        -Jugene-verbose         : Verbose Level. Default 2

 Hermit Parameters
-        -Hermit-workspace       : Use the Hermit Workspace mechanism for I/O.
                                  https://wickie.hlrs.de/platforms/index.php/Workspace_mechanism

 descr: The script creates a new folder relative to the current path with the name
        '\<jobname\>.\<npe\>.(date)' (where \<npe\> is the number of processes. In this new directory
        a file named 'info.txt' is created, to identify the job. Then, depending
        on the selected cluster type, a shell script is created and submitted to the
        job scheduling system. ugsubmit also sets a symbolic link lastRun to the newly
        created directory
 
 exported environment variables:
 -       UGS_JOBNAME              : job name
 -       UGS_OUTDIR               : output directory
 -       UGS_NP                   : see \<npe\>
 -       UGS_NPPN                 : see -nppn
 -       UGS_MYDIR                : the directory ugsubmit is started from

<hr>
\section secuginfo uginfo
<hr>
uginfo displays job information of the current user on the cluster. 

<hr>
\section secugcancel ugcancel
ugcancel cancels a job. use \ref secuginfo to get a job id.
<hr>

*/
