/*
 * Copyright (c) 2012-2013:  G-CSC, Goethe University Frankfurt
 * Author: Martin Rupp
 * 
 * This file is part of UG4.
 * 
 * UG4 is free software: you can redistribute it and/or modify it under the
 * terms of the GNU Lesser General Public License version 3 (as published by the
 * Free Software Foundation) with the following additional attribution
 * requirements (according to LGPL/GPL v3 §7):
 * 
 * (1) The following notice must be displayed in the Appropriate Legal Notices
 * of covered and combined works: "Based on UG4 (www.ug4.org/license)".
 * 
 * (2) The following notice must be displayed at a prominent place in the
 * terminal output of covered works: "Based on UG4 (www.ug4.org/license)".
 * 
 * (3) The following bibliography is recommended for citation and must be
 * preserved in all covered files:
 * "Reiter, S., Vogel, A., Heppner, I., Rupp, M., and Wittum, G. A massively
 *   parallel geometric multigrid solver on hierarchically distributed grids.
 *   Computing and visualization in science 16, 4 (2013), 151-164"
 * "Vogel, A., Reiter, S., Rupp, M., Nägel, A., and Wittum, G. UG4 -- a novel
 *   flexible software system for simulating pde based models on high performance
 *   computers. Computing and visualization in science 16, 4 (2013), 165-179"
 * 
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
 * GNU Lesser General Public License for more details.
 */

/** \page pageUG4ParallelJuGene JuGene

\attention Please note that the BG/P system \em JuGene was replaced by the BG/Q system
named \em JuQueen in July 2012, see page \subpage pageUG4ParallelJuQueen for some
instructions to work on this successor of \em JuGene!
Although with this replacement not only the hardware has changed the following
informations on \em JuGene might remain of some use (especially some links to
out-sourced parts of the documentation) and is left here to the careful reader.
In addition maybe sometime a \ug4 user will have access to another BG/P machine.


- \ref secJuGene
- \ref secAccess_to_JuGenes_login_nodes
- \ref secConfiguration_of_ug4_for_JuGene
- \ref secWorking_with_ug4_on_JuGene
- \ref secVery_large_jobs_on_JuGene
- \ref secDebuggingOnJuGene

<hr>
\section secJuGene General Information about JuGene

<a href="http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUGENE/JUGENE_node.html">JuGene</a>
&mdash; the 
<a href="http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUGENE/Configuration/Configuration_node.html">72 racks Blue Gene/P</a>
(BG/P) system at <em>J&uuml;lich Supercomputing Centre (JSC)</em> (FZ J&uuml;lich)
in total provides 294.912 cores (288 Ki) and 144 Tbyte RAM.
The 73.728 compute nodes (CN) each have a quadcore Power 32-bit PC 450 running 
at 850 MHz, with 2 Gbyte of RAM. 

Half a rack (2048 cores) is called a \b midplane.
The \em JuGene system uses five different networks dedicated for various tasks 
and functionalities of the machine.
Relevant for us:
The 3-D torus network.
This is a point-to-point network &mdash; each of the CNs has six 
nearest-neighbour connections.

See more on the architecture of \em JuGene 
<a href="http://www.fz-juelich.de/SharedDocs/Downloads/IAS/JSC/EN/JUGENE/SlidesBGPArchitecture.pdf">here</a>.

More information about \em JuGene "in order to enable users of the system to 
achieve good performance of their applications" can be found in
<a href="http://www.prace-ri.eu/IMG/pdf/Best-practise-guide-JUGENE-v0-3.pdf">"PRACE Best-Practice Guide for JUGENE"</a>.

\note Note that the login nodes are running under <em>SuSE Linux Enterprise 
Server 10</em> (SLES 10), while the CNs are running a limited version of Linux 
called <em>Compute Node Kernel</em> (CNK).
Therefore its necessary to <strong>cross-compile</strong> for \em JuGene (cf. 
sec. \ref secCMake; sec. \ref secConfiguration_of_ug4_for_JuGene).


<hr>
\section secAccess_to_JuGenes_login_nodes Access to JuGene's Login Nodes
\em JuGene is reached via two so called <strong>front-end</strong> or 
<strong>login nodes</strong> (\c jugene3 and \c jugene4) for interactive access 
and the submission of batch jobs.

These login nodes are reached via
\code
ssh <user>@jugene.fz-juelich.de
\endcode
i.e., for login there is only a generic hostname, <tt>jugene.fz-juelich.de</tt>,
from which automatically a connection either to \c jugene3 or \c jugene4 will 
be established.

The front-end nodes have an identical environment, but multiple sessions of one 
user may reside on different nodes which must be taken into account when 
killing processes.

It is necessary to <strong>upload the SSH key</strong> of the machine from 
which to connect to one of JuGenes login nodes.
See 
<a href="http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUGENE/UserInfo/Logon.html">Logging on to JUGENE</a>
(also for \em X11 problems).

To be able to connect to \em JuGene from different machines maybe you find it 
useful to define one of \em GCSC's machines (e.g. \c speedo, \c quadruped, ...) 
as a "springboard" to one of \em JuGenes login nodes (so that you have to login 
to this machine first, then to \em JuGene), see \ref secSSHHopping.


<hr>
\section secConfiguration_of_ug4_for_JuGene Configuration of ug4 for JuGene

For \em JuGene you have to "cross compile" and to do so use a specific 
\ref secCMake_ToolchainFiles.
Start CMake like this
\code
cmake -DCMAKE_TOOLCHAIN_FILE=../cmake/toolchain/jugene.cmake ..
\endcode
or, for <strong>static builds</strong> which is the <strong>configuration of 
choice</strong> if you want to <strong>process very large jobs</strong>,
\code
cmake -DSTATIC=ON -DCMAKE_TOOLCHAIN_FILE=../cmake/toolchain/jugene_static.cmake ..
\endcode

See also \ref secVery_large_jobs_on_JuGene!

\note A <em>static build</em> where also all system libraries are linked 
statically need some additional "hand work" by now:
After configuration with \em CMake edit the following files by replacing all 
occurences of <tt>libXXXXXXX.so</tt> by <tt>libXXXXXXX.a</tt>
(has to be done only once):
\code
CMakeCache.txt,
ugbase/ug_shell/CMakeFiles/ugshell.dir/link.txt,
ugbase/ug_shell/CMakeFiles/ugshell.dir/build.make
\endcode

Or use this <tt>sed</tt> command:
\code
sed -i 's/\([[:alnum:]]*\).so/\1.a/g' CMakeCache.txt ugbase/ug_shell/CMakeFiles/ugshell.dir/link.txt ugbase/ug_shell/CMakeFiles/ugshell.dir/build.make 
\endcode

You can check your executable by running the (standard unix) <tt>ldd</tt> command ("list dynamic dependencies") on it:
\code
ldd ugshell
\endcode
Answer should be <tt>not a dynamic executable</tt> for a completely static 
build!

<strong>Debug builds</strong>:
Since the pre-installed <em>GCC 4.1.2</em> (in April 2012) is not able to compile
a "debug" build one should add the flag <tt>-DDEBUG_FORMAT=-gstabs</tt> to the
CMake call (cf. \ref secCMake_GCC412).

For debugging a parallel application on JuGene see \ref secDebuggingOnJuGene

<hr>
\section secWorking_with_ug4_on_JuGene Working with ug4 on JuGene

\subsection secBasic_job_handling Basic Job Handling
Please take the time and fill out the details in \c ugsubmit / \c uginfo / 
\c ugcancel (\ref pageugsubmit).

See 
<a href="http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUGENE/UserInfo/QuickIntroduction.html">Quick Introduction</a>
to job handling.
Also look at 
<a href="http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUGENE/UserInfo/mpirun.html">job/mpirun options</a>.

Here we only introduce some important details (everything else should become 
clear by examining the examples provided):
<ul>
  <li><strong>Jobs can be submitted</strong> by the <tt>llrun</tt> 
    (interactively) and the <tt>mpirun</tt> (as batch job; defined in a 
    \em LoadLeveler script) command.
    See below for some details.
  </li>
  <li>There are different <strong>execution modes</strong> which can be 
    specified by the <tt>mpirun</tt> and <tt>llrun</tt> parameter 
    <tt>-mode {VN | DUAL | SMP}</tt>:
    <ul>
      <li><strong>Quad Mode</strong> (a.k.a. "Virtual Mode"):
        All four cores run one MPI process.
        Memory/MPI Process = 1/4 CN RAM:
        <tt>-mode VN</tt>.
      </li>
      <li><strong>Dual Mode</strong>:
        Two cores run one MPI process (hybrid MPI/OpenMP).
        Memory/MPI Process = 1/2 CN RAM:
        <tt>-mode DUAL</tt>.
      </li>
      <li><strong>SMP Mode</strong> ("Symmetrical Multi Processing Mode"):
        All four cores run one MPI process (hybrid MPI/OpenMP).
        Memory/MPI Process = CN RAM:
        <tt>-mode SMP</tt>.
      </li>
    </ul>
    Note that in quad mode (using all 4 processors of a computing node) this 
    means each core has only ca. 512 Mbyte of RAM (474 Mbyte to be more 
    specific, since the CNK also needs some memory).
    
    Obviously "VN" is the preferred execution mode if large numbers of 
    processes should be achieved &mdash; and \ug4 works with VN mode (at least 
    up to ~64 Ki DoFs per process)!
  </li>
  <li>The <tt>mpirun</tt> parameter <tt>-mapfile</tt> specifies an order in 
    which the MPI processes are mapped to the CNs / the cores of the BG/P 
    partition reserved for the run.
    This order can either be specified by a permutation of the letters X,Y,Z 
    and T \em or the name of a mapfile in which the distribution of the tasks 
    is specified, <tt>-mapfile {&lt;mapping&gt;|&lt;mapfile&gt;}</tt>:
    <ul>
      <li><tt>&lt;mapping&gt;</tt> is a permutation of X, Y, Z and T.

        The standard mapping on \em JuGene is to place the tasks in "XYZT" 
        order, where X, Y, and Z are the torus coordinates of the nodes in the 
        partition and T is the number of the cores within each node (T=0,1,2,3).

        When the tasks are distributed across the nodes the first dimension is 
        increased first, i.e. for XYZT the first three tasks would be executed 
        by the nodes with the torus coordinates <tt>&lt;0,0,0,0&gt;</tt>, 
        <tt>&lt;1,0,0,0&gt;</tt> and <tt>&lt;2,0,0,0&gt;</tt>, which obviously 
        is not what we want for our simulation runs.
        For now we recommend <tt>-mapfile TXYZ</tt> which fills up a CN before 
        going to the next CN so that MPI processes working on adjacent 
        subdomains are placed closely in the 3-D torus.
      </li>
      <li><tt>&lt;mapfile&gt;</tt> is the name of a mapfile in which the 
        distribution of the tasks is specified:
        It contains a line of <tt>x y z t</tt> coordinates for each MPI process.
        See sec. 6. of the <em>Best-Practise guide</em> mentioned above for an 
        example and which LoadLeveler keywords to use.
      </li>
    </ul>
<!--
TODO: Some notes about "topology aware placing of MPI processes" might become relevant in the future.
-->
  </li>
  <li>If \ug4 was dynamically linked add to the <tt>mpirun</tt> parameters 
    <tt>-env LD_LIBRARY_PATH==/bgsys/drivers/ppcfloor/comm/lib/</tt>.
    \note This parameter is (obviously) not necessary for completely static builds!

  </li>
  <li><strong>"Modules"</strong> can be loaded by executing the <tt>module</tt> 
    command.

    E.g. <tt>module load lapack</tt> for LAPACK (see http://www.fz-juelich.de/ias/jsc/EN/Expertise/Support/Software/Libraries/LAPACKJugene.html
    for choosing one of several versions).

<!-- TODO: Nevertheless CMake says "Info: Not using Lapack. No package found"!? -->

    This is also for loading <strong>performance analysis tools</strong>, e.g. 
    <tt>module load scalasca</tt>, <tt>module load UNITE</tt> etc.
<!-- TODO: Please provide a link which explains a bit more !!! -->     
<!-- no experience with this stuff yet! -->
  </li>
  <li><a href="http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUGENE/UserInfo/llview.html"><tt>llview</tt></a>
    is a <strong>tool with a graphical X11 user interface</strong> for 
    displaying system status, running jobs, scheduling and prediction of the 
    start time of jobs (the latter can also be achieved by the <tt>llq</tt>
    command, see below).

<!--
    \todo Write more about <tt>llview</tt>. - There is not much more to say about llview - by us! (ih)
-->
  </li>
  <li><strong>Interactive jobs</strong> can be started with the 
    <a href="http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUGENE/UserInfo/LoadLevelerInteractive.html">llrun</a>
    command. \c llrun is invoked in the following way:
    \code
llrun [<llrun_options>] [<mpirun_options>] [<executable>]
    \endcode
    \c llrun replaces \c mpirun and can be used with the same command line options like \c mpirun.
    One \c llrun option which might be interesting:
    <tt> -w \<hh:mm:ss\></tt>: submit job with wallclock limit (Default: 00:30:00).
    For other options see e.g. http://www.prace-ri.eu/IMG/pdf/Best-practise-guide-JUGENE-v0-3.pdf, sec. 3.5.3.

    <strong>Example</strong>:
    \code
llrun -np 4 -mode VN -mapfile TXYZ -verbose 2 -exe ./ugshell_debug -args "-ex ../apps/scaling_tests/modular_scalability_test.lua -numPreRefs 3 -numRefs  7"
    \endcode

    \note Note the quotation marks around the executables arguments!


    Please note that <tt>llrun</tt> only allows jobs up to 256 
    (<tt>-mode SMP</tt>) / 512 (<tt>-mode DUAL</tt>) / 1024 (<tt>-mode VN</tt>) 
    MPI processes!
  </li>
  <li><strong>Batch Jobs</strong> are defined in so called <strong>"LoadLeveler 
    scripts"</strong> and submitted by the 
    <a href="http://www2.fz-juelich.de/jsc/jugene/usage/loadl/llsubmit/"><tt>llsubmit</tt></a>
    command to the <em>IBM Tivoli Workload Scheduler LoadLeveler</em> (TWS 
    LoadLeveler), typically in the directory where the ug4 executable resides:
    \code
llsubmit <cmdfile>
    \endcode
    <tt>&lt;cmdfile&gt;</tt> is a (plain Unix) shell script file (i.e., the 
    "LoadLeveler script"), which contains job definitions given by 
    <strong>"LoadLeveler keywords"</strong> (some important examples are 
    explained below).

    If <tt>llsubmit</tt> was able to submit the job it outputs a <strong>job 
    name</strong> (e.g. <tt>jugene4b.zam.kfa-juelich.de.298508</tt>) with which 
    a specific job can be identified in further commands, e.g. to cancel it 
    (see below).

    The <strong>output</strong> of the run (messages from Frontend end Backend 
    MPI &mdash; and the output of \ug4 &mdash; is written to a file in the 
    directory where <tt>llsubmit</tt> was executed and whose name begins with 
    the job name you have specified in the LoadLeveler script and ends with 
    <tt>&lt;job number&gt;.out</tt>.

    For some <strong>example LoadLeveler scripts</strong> used with \ug4 see 
    subdirectory <tt>scripts/shell/</tt>:

    <ul>
      <li><tt>ll_scale_gmg.x</tt> contains job definitions for a complete 
        scalability study for GMG in 2-D and 3-D.
      </li>
      <li><tt>ll_template.x</tt> also contains some documentation of LoadLeveler 
        and mpirun parameters.
      </li>
    </ul>
    (All <tt>mpirun</tt> commands therein are commented out &mdash; to performe 
    a specific run remove the comment sign.)

    Please change in your copy of one of these scripts at least the value of the 
    <tt>notify_user</tt> keyword before submitting a job ...

    See also this 
    <a href="http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUGENE/UserInfo/LoadLeveler.html">more recent JSC documentation</a>,
    and especially the
    <a href="http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUGENE/UserInfo/LoadLevelerSamples.html">Job File Samples</a>
    for more details.
  </li>
  <li><strong>LoadLeveler keywords</strong> are strings embedded in comments 
    beginning with the characters "# @".
<!--
    \todo Isn't there a better overview on the web?
-->
    Some selected keywords:
    <ul>
      <li><tt>job_type = bluegene</tt> specifies that the job is running on 
        <em>JuGene</em>'s CNs.
      </li>
      <li><tt>job_name</tt> specifies the name of the job, which will get part 
        of the name of the output file.
      </li>
      <li><tt>bg_size</tt> specifies the size of the BG/P partition reserved 
        for the job in <strong>number of compute nodes</strong>.

        That is, for <tt>&lt;NP&gt;</tt> MPI processes, <tt>bg_size</tt> must 
        be >= <tt>(&lt;NP&gt;)/4</tt>.

        Alternatively the size of a job can be defined by the <tt>bg_shape</tt> 
        keyword.

        See comments in the example LoadLeveler scripts for proper settings.
      </li>
      <li><tt>bg_connection</tt> specifies the "connection type", i.e. the 
        <strong>network topology</strong> used by a job.

        Connection type can be one in <tt>[TORUS| MESH | PREFER_TORUS]</tt>.
        Default is <tt>bg_connection = MESH</tt>.

        <tt>bg_connection = TORUS</tt> &mdash; utilising the 3-D torus network 
        &mdash; is the preferred topology for our jobs.
        For this <tt>bg_size</tt> (see above) must be >= 512.

        See also comments and usage in the example LoadLeveler scripts for 
        proper settings.
      </li>
    </ul>
    Please note that keywords must not be followed by comments in the same line! 

    A nice introduction to LoadLeveler command file syntax is given e.g. 
    <a href="https://docs.loni.org/wiki/LoadLeveler_Command_File_Syntax">here</a>.
  </li>
  <li><tt>llq</tt> is used to display the status of jobs (of a specified user) 
    in the queue/executed:
    \code
llq [-u  <userid>]
    \endcode

    The <strong>estimated start time</strong> of a job can be determined by
    <tt>llq -s \<job-id\></tt>
    (cf. https://docs.loni.org/wiki/Useful_LoadLeveler_Commands), where
    <tt>\<job-id\></tt> was determined by the previous command, e.g. for the
    job <tt>jugene4b.304384.0</tt>:

    \code
    llq -s 304384  
    
    ===== EVALUATIONS FOR JOB STEP jugene4b.zam.kfa-juelich.de.304384.0 =====
    
    Step state                       : Idle
    Considered for scheduling at     : Sun 01 Apr 2012 00:02:55 CEST
    Top dog estimated start time     : Sun 01 Apr 2012 16:38:55 CEST
    ...
    \endcode
<!--
    \todo The "top dog" time in the example above and the predicted start time (~18:44) displayed by \c llview differ!?
-->
  </li>
  <li><tt>llcancel</tt> is used to cancel a job (<tt>&lt;jobname&gt;</tt> as 
    displayed by <tt>llq</tt>):
    \code
llcancel <jobname>
    \endcode
  </li>
  <li>Debugging:
    See documentation to the tools 
    <tt>/bgsys/drivers/ppcfloor/tools/coreprocessor</tt>, <tt>gdbserver</tt> ...
  </li>
  <li>Available file systems.
    See
    <a href="http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUDGE/Userinfo/Access_Environment.html">JSC documentation</a>
    for more details.
  </li>
  <li>Querying <strong>Quota Status</strong>:
    \code
q_cpuquota <options>
    \endcode

    Useful options: 
    \arg <tt>-?</tt>                   usage information and all options .
    \arg <tt>-j &lt;jobstepid&gt;</tt> for a single job.
    \arg <tt>-t &lt;time&gt;</tt>      for all jobs in the specified time, e.g.
      <tt>q_cpuquota -t 23.11.2011 01.12.2011</tt>.
    \arg <tt>-d &lt;number&gt;</tt> for last number of days (positive integer).
  </li>
</ul>


<hr>
\subsection secVery_large_jobs_on_JuGene Very large Jobs on JuGene
<ul>
  <li>Although its possible on \em JuGene to create shared libraries and run 
    dynamically linked executables this is <strong>in general not 
    recommended</strong>, since loading of shared libraries can delay the 
    startup of such an application considerably, especially when using large 
    partitions (8 racks or more).
    See also
    <a href="http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUGENE/UserInfo/SharedLibraries.html">Shared Libraries and Dynamic Executables</a>.

    So, for <strong>very large jobs</strong> be sure to have \ug4 built as a 
    <strong>completely static executable</strong> (c.f. 
    \ref secConfiguration_of_ug4_for_JuGene) since otherwise loading of the 
    shared libraries consumes too much wall time!
  </li>
  <li><strong>Very large jobs</strong> (e.g. jobs larger than 32 racks) 
    normally run on <b>Tuesday</b> only.

    Exceptions to this rule are possible in urgent cases (please contact the SC 
    Support under <a href="sc@fz-juelich.de">sc@fz-juelich.de</a>).
  </li>
</ul>

<hr>
\subsection secDebuggingOnJuGene Debugging on JuGene
On JuGene the parallel debuggers \em TotalView and \em DDT are available.

Be sure to compile \ug4 as a debug build.

DDT uses X11 for its graphical user interface. Be sure to log in with an X window
forwarding enabled. This could mean using the <tt>-X</tt> or <tt>-Y</tt> option
to <tt>ssh</tt>.

<strong>Basic usage of \em DDT</strong>:
<ol>
<li> Load the appropriate module:
     \code
module load UNITE ddt
     \endcode
</li>
<li>Start DDT by typing <tt>ddt</tt>.</li>
<li>A "DDT" logo appears, wait a bit (or click on the logo) to get the welcome dialog box.</li>
<li>Click on "Run and Debug a Program" in the "Welcome" dialog box</li>
<li>Enter your \ug4 parameters (don't forget to enclose them by quotation marks),
    numbers of processes to run,
    a (hopefully) appropriate wall time to do all your debugging work
    (after "Queue submission Parameters"),
    "MPIRun parameters" (e.g. <tt>-mode VN -mapfile TXYZ -verbose 2</tt>) in the
    fields provided after clicking "Advanced\>\>" etc.,
    then click the "Submit" button.

    Wait until the job is launched (you might exercise some patience), DDT will
    catch the focus automatically when resources are available.</li>
</ol>
The rest should be quite self-explaining.

One can also immediately specify the application to debug and also its parameters
by typing <tt>ddt [\<your app\> \<parameters\>]</tt>.

<!--
TODO: As the textfield "Number of nodes (1 procs per node)" says, choosing e.g. 4 nodes
brings up a job with only 4 MPI processes - is it also possible to get 4 processes per
node, as we otherwise get using execution mode "VN" (<tt>-mode VN</tt>)?
-->

<strong>Basic usage of \em TotalView</strong>:
<ol>
<li> Load the appropriate module:
     \code
module load UNITE totalview
     \endcode
</li>
<li>Start TotalView by adding the <tt>-tv</tt> flag to your \c llrun call with which
    you normally would run your executable in interactive mode (see above).</li>
</ol>

<strong>Example debug sessions</strong>:
<ol>
<li>Start a \em DDT debugging session (immediately specifying the executable to debug):
     \code
ddt
     \endcode

     Specification of executable and parameters should also work - parameters are written in the appropriate field, but not recognised ....
     \code
ddt ./ugshell_debug -ex ../apps/scaling_tests/modular_scalability_test.lua -numRefs  7
     \endcode

     The mpirun parameters, e.g. <tt>-mapfile TXYZ -verbose 2</tt> can be placed in
     the fields accessible after clicking on the "Advanced" button.
     </li>
<li>Start a \em TotalView debugging session:
     \code
llrun -tv -np 4 -mode VN -mapfile TXYZ -verbose 2 -exe ./ugshell_debug -args "-ex ../apps/scaling_tests/modular_scalability_test.lua -numPreRefs 3 -numRefs  7"
     \endcode
     Use the \c llrun option <tt> -w \<hh:mm:ss\></tt> to specify a (hopefully)
     appropriate wall time to do all your debugging work.</li>
<!--
Remark: When executing \c llrun with '-tv' neither the <tt>-exe</tt> nor the <tt>-args</tt>
argument (see above) was necessary, and also no quotation marks.
-->
</ol>

For additional information (especially for debugging with TotalView) see
http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUGENE/UserInfo/ParallelDebugging.html.
*/
