/*
 * Copyright (c) 2012-2014:  G-CSC, Goethe University Frankfurt
 * Author: Martin Rupp
 * 
 * This file is part of UG4.
 * 
 * UG4 is free software: you can redistribute it and/or modify it under the
 * terms of the GNU Lesser General Public License version 3 (as published by the
 * Free Software Foundation) with the following additional attribution
 * requirements (according to LGPL/GPL v3 §7):
 * 
 * (1) The following notice must be displayed in the Appropriate Legal Notices
 * of covered and combined works: "Based on UG4 (www.ug4.org/license)".
 * 
 * (2) The following notice must be displayed at a prominent place in the
 * terminal output of covered works: "Based on UG4 (www.ug4.org/license)".
 * 
 * (3) The following bibliography is recommended for citation and must be
 * preserved in all covered files:
 * "Reiter, S., Vogel, A., Heppner, I., Rupp, M., and Wittum, G. A massively
 *   parallel geometric multigrid solver on hierarchically distributed grids.
 *   Computing and visualization in science 16, 4 (2013), 151-164"
 * "Vogel, A., Reiter, S., Rupp, M., Nägel, A., and Wittum, G. UG4 -- a novel
 *   flexible software system for simulating pde based models on high performance
 *   computers. Computing and visualization in science 16, 4 (2013), 165-179"
 * 
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
 * GNU Lesser General Public License for more details.
 */

/** \page pageUG4ParallelCekon Cekon

\em Cekon is the smaller in-house cluster of the \em G-CSC.
It consists of 23 compute nodes with 4 cores per node, that is 92 computing cores.

<ul>
  <li>Configuration:
    Normally a run of \em CMake with "standard" parameters should do the job.

    There are some problems with the pre-installed <em>GCC 4.1.2</em>, see 
    \ref secCMake_GCC412 .
    You can also use <tt>icc</tt> as compiler.
    <br>
    
        
  </li>
  <li>
  You can also use llvm/clang as compiler: <br>
\verbatim
cmake -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER=clang ..
\endverbatim
	\note Clang only produces debug information in the Dwarf4-format, which is not supported by gdb on this machine. Use ddt instead.
  <br>  
  <li>To start <b>Jobs on Cekon</b>, use \c ugsubmit /   \c uginfo / \c ugcancel : <br>
   \ref pageugsubmit .

	To start a job with 16 processes and arguments <tt>-ex conv_diff/laplace.lua -numPreRef 3 -numRefs 9</tt>, use
\verbatim
ugsubmit 16 --- ugshell -ex conv_diff/laplace.lua -numPreRef 3 -numRefs 9 
\endverbatim
	For information about your jobs use <tt>uginfo</tt>, to kill job with id 2359 use <tt>ugcancel 2359</tt>
	Internally, ugsubmit will pass all the information to the appropriate Job Scheduler of the cluster (for cekon this is SLURM).   
  <li>
    If you want to do job scheduling, start your job with
    \verbatim
salloc -n 16 mpirun ugshell -ex conv_diff/laplace.lua -numPreRef 3 -numRefs 9
    \endverbatim
    \note Be aware that this will only run on Cekon, where ugsubmit-scripts are as easy to use and will run on a number of clusters.
        
    Please note that ony the <tt>salloc</tt> parameter <tt>-n</tt> reserves a 
    number of processes / cores of a job, while <tt>-N</tt> (capital \c N) a 
    number of nodes.
    See the <tt>salloc</tt> manual page for further details.<br>
  To display information about jobs already running (located in the 
    \em SLURM scheduling queue) use the <tt>squeue</tt> command.
    
  </li>
  <li>\b Debugging : If \em DDT is installed, simply type <tt>ddt</tt> in the 
    Unix shell to start the Debugger and fill in the job definition of the job 
    to be debugged in the GUI (X11 based) &mdash; everything should be quite 
    self-explanatory.
  </li>
</ul>

<hr>
<b>Troubleshooting</b>
<ul>
  <li>If you encounter an error like the following when submitting your job on
  \em cekon you might have ignored the general remark in section \ref secGeneral_Notes

\verbatim
salloc -n 64 ./ugshell -ex conv_diff/laplace.lua -numPreRefs 3 -numRefs 7
salloc: Granted job allocation 14437
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
    This will severely limit memory registrations.
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   cekon.gcsc.uni-frankfurt.de
  Local device: mthca0
--------------------------------------------------------------------------
\endverbatim

  </li>
</ul>


*/