/*
 * Copyright (c) 2012-2015:  G-CSC, Goethe University Frankfurt
 * Author: Martin Rupp
 * 
 * This file is part of UG4.
 * 
 * UG4 is free software: you can redistribute it and/or modify it under the
 * terms of the GNU Lesser General Public License version 3 (as published by the
 * Free Software Foundation) with the following additional attribution
 * requirements (according to LGPL/GPL v3 §7):
 * 
 * (1) The following notice must be displayed in the Appropriate Legal Notices
 * of covered and combined works: "Based on UG4 (www.ug4.org/license)".
 * 
 * (2) The following notice must be displayed at a prominent place in the
 * terminal output of covered works: "Based on UG4 (www.ug4.org/license)".
 * 
 * (3) The following bibliography is recommended for citation and must be
 * preserved in all covered files:
 * "Reiter, S., Vogel, A., Heppner, I., Rupp, M., and Wittum, G. A massively
 *   parallel geometric multigrid solver on hierarchically distributed grids.
 *   Computing and visualization in science 16, 4 (2013), 151-164"
 * "Vogel, A., Reiter, S., Rupp, M., Nägel, A., and Wittum, G. UG4 -- a novel
 *   flexible software system for simulating pde based models on high performance
 *   computers. Computing and visualization in science 16, 4 (2013), 165-179"
 * 
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
 * GNU Lesser General Public License for more details.
 */


/** \page pageUG4ParallelHermit Hermit

- \ref secHermit_Architecture
- \ref secHermit_General
- \ref secHermit_GCC
- \ref secHermit_Cray
- \ref secHermit_Workspace
- \ref secHermit_Debugging


<hr>
\section secHermit_Architecture Architecture
The <a href="https://wickie.hlrs.de/platforms/index.php/Cray_XE6">Cray XE6 ("Hermit")</a>.
In Installation step 1, the XE6 is a 3552 node cluster.
Each node is a Dual Socket
<a href="http://en.wikipedia.org/wiki/List_of_AMD_Opteron_microprocessors#Opteron_6200-series_.22Interlagos.22_.2832_nm.29">AMD Opteron 6276 (Interlagos)</a>
@ 2.3GHz 16 cores each, which results in 113.664 cores in total.
Normal nodes have 32 GB RAM, 480 special nodes have 64 GB (total 126 TB).
(<a href="https://wickie.hlrs.de/platforms/index.php/CRAY_XE6_Hardware_and_Architecture"> Architecture</a>)
That is 1 GB RAM for each process when running the maximum of 32 processes on a 
node. 
Current maximum number of cores for one job is 64000.
Speak to the administration for more nodes.


<hr>
\section secHermit_General General 

Be aware that hermit blocks all connections to the normal internet. You have to use \ref secSVNSSHTunneling to check out on Hermit, and use uginstall's -svnServer option (see \ref pageUginstall).

The Job Scheduler on Hermit is supported by \ref pageugsubmit "ugsubmit" for unified  job scheduling on all clusters.
You might want to use -Hermit-workspace .

- <a href="https://wickie.hlrs.de/platforms/index.php/DDT">Debugging with DDT</a>
- <a href="https://wickie.hlrs.de/platforms/index.php/Software_Development_Tools,_Compilers_%26_Libraries">Available software </a>
- <a href="https://wickie.hlrs.de/platforms/index.php/Module_command">The module command</a>
- <a href="https://wickie.hlrs.de/platforms/index.php/CRAY_XE6_Using_the_Batch_System">Batch system</a> (recommended is \ref pageugsubmit "ugsubmit")

\note You have to choose modules every time you log in (you might want to add 
your <tt>module load/swap</tt> commands into your <tt>.bashrc</tt> or similar ).

<hr>
\section secHermit_GCC GCC
First, look what modules are loaded
\verbatim
module list
\endverbatim

There is one which is named PrgEnv-cray or PrgEnv-*. Now you swap that to 
PrgEnv-gnu:
\verbatim
module swap PrgEnv-cray PrgEnv-gnu
\endverbatim

There's a one-liner for this task:
\verbatim
module swap $(module li 2>&1 | awk '/PrgEnv/{print $2}') PrgEnv-gnu
\endverbatim

Then you start cmake with a \ref secCMake_ToolchainFiles :
\verbatim
cmake -DCMAKE_TOOLCHAIN_FILE=../cmake/toolchain/hermit.cmake ..
\endverbatim


<hr>
\section secHermit_Cray Cray CC
The Cray C Compiler is not working at the moment because there is an internal 
compiler error in release mode.

Toolchain file is <tt>../cmake/toolchain/hermit.cmake</tt>, and the module is
PrgEnv-cray.
\verbatim
module swap $(module li 2>&1 | awk '/PrgEnv/{print $2}') PrgEnv-cray
cmake -DCMAKE_TOOLCHAIN_FILE=../cmake/toolchain/hermit.cmake ..
\endverbatim

<hr>
\section secHermit_Workspace Workspace Mechanism
Access to the user file system (anything in your home directory) from your 
running job is very slow on Hermit.
It is very noticable in runs with 1024 cores and even if you are accessing only 
small files once like your script files.
The
<a href="https://wickie.hlrs.de/platforms/index.php/Workspace_mechanism">Workspace Mechanism</a>
lets you create a directory on a specialized parallel file system.
First create a workspace in ug4:
\verbatim
ln -s `ws_allocate ug4ws 31` workspace
\endverbatim
Before each time you run ugshell, you'll have to run a script like the following:
\verbatim
mkdir -p workspace/ugcore
rsync -a -W --exclude=.git ugcore/scripts/ workspace/ugcore/scripts/
rsync -a -W --exclude=.svn apps/ workspace/apps/
export UG4_ROOT=workspace
\endverbatim
See also <a href="http://linux.die.net/man/1/rsync">rsync docu</a> (-a = subdirectories and keep date, -W = whole files).

Now start your job with ugsubmit from inside your workspace.

Be aware that also files written to can be damaged if you are not using the
Workspace mechanism. 
See also the "-dir" and "-Hermit-workspace" option in \ref pageugsubmit.

Note the export UG4_ROOT-part:
\verbatim
export UG4_ROOT=$runDir
\endverbatim
If this environment variable is not specified, \ug4 will look in <tt>../scripts/</tt> for scripts and
<tt>../data/</tt> for data, relative to the path of the binary.

\warning Be aware that your workspace time is limited. You'll have to use <tt>ws_extend</tt> at least every month to prevent your workspace from being deleted. This also includes your output,
so be sure to extend your workspace time and copy your result out of the workspace asap.


<hr>
\section secHermit_Debugging Debugging on Hermit
On Hermit the parallel debugger \em DDT is available.
See https://wickie.hlrs.de/platforms/index.php/DDT.

Be sure to compile \ug4 as a debug build.

DDT uses X11 for its graphical user interface. Be sure to log in with an X window
forwarding enabled. This could mean using the <tt>-X</tt> or <tt>-Y</tt> option
to <tt>ssh</tt>.

<strong>Basic usage of \em DDT</strong>:
<ol>
<li> Load the DDT module:
     \code
module load ddt
     \endcode
     </li>
<li>Start a job in <em>interactive mode</em> with
    \code
qsub -IX [other job options]
    \endcode
    <tt>-X</tt> enables X11 forwarding; <tt>-X</tt> exports all environment
     variables in qsub's  command environment to the job.</li>
<li>Start DDT by typing
    \code
ddt
    \endcode
    </li>
<li>Click on "Run and Debug a Program" in the "Welcome" dialog box, enter the
    executable to debug, numbers of processes to run, the executables parameters
    etc.,
    then click the "Submit" button.

    One can also immediately specify the application to debug and also its
    parameters by typing <tt>ddt [\<your app\> \<parameters\>]</tt></li>
</ol>
The rest should be quite self-explaining.

<strong>Example debug session</strong>:
<ol>
<li>Start interactive session:
    \code
qsub -I -V -X -l mppwidth=64,mppnppn=32,walltime=00:30:00
qsub: waiting for job 309598.sdb to start
    \endcode
    I.e, the session in this example lasts 30 minutes.
    </li>
<li>Start DDT with <tt>ugshell</tt> as executable and some \ug4 parameters:
    \code
ddt ./ugshell -ex ../apps/scaling_tests/modular_scalability_test.lua -numPreRefs 1 -hRedistFirstLevel 4 -hRedistStepSize 2 -hRedistNewProcsPerStep 4 -numRefs 8
    \endcode
<!--
ddt ./ugshell -ex ../apps/scaling_tests/modular_scalability_test.lua  -numPreRefs 4 -lsType feti -ds famg  -ns famg  -numPPSD   4 -numRefs  7
-->

    </li>
</ol>

Notes:
<ul>
<li><tt>-I</tt> declares that the job is to be run "interactively".</li>
<li><tt>-V</tt> declares that all environment variables are passed to the job.</li>
<li><tt>-X</tt> Enables X11 forwarding, necessary to get DDT's GUI as X11 window.</li>
<li>After the job is started with <tt>qsub</tt> you've entered a new shell,
    so you have to go (again) into \ug4's bin directory before starting the debugger.</li>
<li> When retrying the above procedere the following error message appeared after executing the <tt>qsub</tt> command (19092012):
\code
PrgEnv-cray/4.0.46(11):ERROR:150: Module 'PrgEnv-cray/4.0.46' conflicts with the currently loaded module(s) 'PrgEnv-gnu/4.0.46'
PrgEnv-cray/4.0.46(11):ERROR:102: Tcl command execution failed: conflict PrgEnv-gnu

ModuleCmd_Switch.c(172):ERROR:152: Module 'PrgEnv-cray' is currently not loaded
\endcode
This can seemingly be ignored.
</li>

<li>In the same test DDT complained that no licence file is found:
\code
ddt ./ugshell ...

Unable to obtain valid licence for DDT.
The licence file tried was specified using the (optional) environment
variable specifier DDT_LICENCE_FILE.
The file "/opt/cray/ddt/3.2/Licence" does not exist.

If you do not already have a licence for DDT, please visit the Allinea website
http://www.allinea.com/products/ddt/ to obtain an evaluation licence.
\endcode

According to the message above it was possible to specify the license file of an earlier version:
\code
nid03538:~ igcingo$ export DDT_LICENCE_FILE=/opt/cray/ddt/3.1/Licence
\endcode

So, when executing again the <tt>ddt</tt> command DDT eventually starts ...
</li>
</ul>
*/
