/*
 * Copyright (c) 2012-2014:  G-CSC, Goethe University Frankfurt
 * Author: Martin Rupp
 * 
 * This file is part of UG4.
 * 
 * UG4 is free software: you can redistribute it and/or modify it under the
 * terms of the GNU Lesser General Public License version 3 (as published by the
 * Free Software Foundation) with the following additional attribution
 * requirements (according to LGPL/GPL v3 §7):
 * 
 * (1) The following notice must be displayed in the Appropriate Legal Notices
 * of covered and combined works: "Based on UG4 (www.ug4.org/license)".
 * 
 * (2) The following notice must be displayed at a prominent place in the
 * terminal output of covered works: "Based on UG4 (www.ug4.org/license)".
 * 
 * (3) The following bibliography is recommended for citation and must be
 * preserved in all covered files:
 * "Reiter, S., Vogel, A., Heppner, I., Rupp, M., and Wittum, G. A massively
 *   parallel geometric multigrid solver on hierarchically distributed grids.
 *   Computing and visualization in science 16, 4 (2013), 151-164"
 * "Vogel, A., Reiter, S., Rupp, M., Nägel, A., and Wittum, G. UG4 -- a novel
 *   flexible software system for simulating pde based models on high performance
 *   computers. Computing and visualization in science 16, 4 (2013), 165-179"
 * 
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
 * GNU Lesser General Public License for more details.
 */

//todo:	More prominent note on OpenMPI for all operating systems!

/** \page pageUG4SetupParallel Installation on Parallel Computers

[TOC]

Special instructions for the different parallel computers are on separate pages:
- \subpage pageUG4ParallelCekon
- \subpage pageUG4ParallelCesari
- \subpage pageUG4ParallelJuGene &mdash; R.I.P. &mdash; please use \em JuQueen!
- \subpage pageUG4ParallelJuQueen
- \subpage pageUG4ParallelHermit

Instructions for password-less login to the different clusters are on the page
about \ref pageUG4SSH (including \ref secSSHKeys, \ref secSSHHopping, \ref secSSHFS).

 See also \ref pageugsubmit.

<hr>
\section secGeneral_Notes General Notes

All examples are for running ug in parallel with <tt>&lt;NP&gt;</tt> processors 
and <tt>$UGARGS</tt> as arguments, where <tt>&lt;NP&gt;</tt> is a placeholder 
for the number of MPI processes and <tt>$UGARGS</tt> is an Unix environment 
variable which for example is defined by (Bash syntax; just to shorten command 
lines):
\verbatim
UGARGS="-ex ../scripts/tests/modular_scalability_test.lua -dim 2 -grid ../data/grids/unit_square_01/unit_square_01_quads_8x8.ugx"
\endverbatim

\warning Except for your own computer/workstation or explicitly stated, do 
<strong>NOT EVER</strong> use <tt>mpirun -np &lt;NP&gt; ugshell $UGARGS</tt> to 
start your job on a cluster!
The node you are logging into is only a login node, and you don't want to run 
your job on these.

<hr>
\section secSetupCluster Setting up your cluster
We have included a automatic configuration: It is called <b>ugconfig</b>, and is in ug4/scripts/shell/ugconfig. So for your .bashrc, we suggest
\code
source $HOME/ug4/trunk/scripts/shell/ugbash		% for completion etc.
source $HOME/ug4/trunk/scripts/shell/ugconfig	% for configuration
\endcode

ugconfig then automatically detects you cluster and loads standard modules like cmake, LAPACK or compilers.
With this, you can also use the next tool to help you set up: Instead of using cmake, you can use ugcmake-start, e.g.
\code
ugcmake-start -DDIM=2 -DCPU=1 -Damg=ON ..
\endcode
This will again automatically detect your cluster, and call cmake with the appropriate toolchain file and flags (see also \ref secCMake).

The aim of ugconfig and ugcmake-start is to make your life easier, so if they do not work, please tell, so we can fix that problem for you and others. 
 
<hr>
\section secCMake CMake, Toolchains, Compilers

\subsection secCMake_ToolchainFiles Toolchain File
On some systems (especially when the software is built for a different system 
than the one which does the build) it is necessary to change some configuration 
settings done by \em CMake (like compilers or flags to use) by a so called 
"toolchain file" (cf. for example 
<a href="http://www.vtk.org/Wiki/CMake_Cross_Compiling">CMake Cross Compiling</a>).

In this case run \em CMake like this
\verbatim
cmake -DCMAKE_TOOLCHAIN_FILE=<TOOLCHAINFILE> ..
\endverbatim

Toolchain files are located in <tt>cmake/toolchain</tt>.

\note if you already set other cmake settings before or e.g. tipped "cmake .." you first need to remove all files in your build-directory 
by the command "rm -rf *" and then call cmake with the toolchainfile by "cmake -DCMAKE_TOOLCHAIN_FILE=<TOOLCHAINFILE> .."!


\subsection secStaticBuild Static Builds 
ug4 is using external libraries and distributes some of its own codes into libraries (e.g. plugins).
There are two ways of dealing with libraries:
<ul>
<li> <em>dynamic linking</em>: When the program is started, the library code is loaded out of another file. Those other files are called <a href=http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html>shared libraries</a>. They usually end on .so or .so*.</li>
<li> <em>static linking</em>: The library code is written into the program binary when linking is done. When the program is started, all code is already loaded, since
it is <em>in</em> the program itself. Static libraries usually end on '.a'. See also <a href=https://en.wikipedia.org/wiki/Static_library>Static Libraries</a>.</li>
</ul>   
Although its possible on most clusters to create shared libraries and run 
dynamically linked executables this is <strong>in general not 
recommended</strong>, since loading of shared libraries can delay the 
startup of such an application considerably, especially when using large 
partitions (8 racks or more).
See for example  <a href="http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/UserInfo/SharedLibraries.html">Shared Libraries and Dynamic Executables on Juqueen</a>.

Start CMake like this
\code
cmake -DSTATIC_BUILD=ON -DCMAKE_TOOLCHAIN_FILE=../cmake/toolchain/juqueen.cmake ..
\endcode

If you're using a \ref secCMake_ToolchainFiles, use
\code
cmake -DSTATIC_BUILD=ON -DCMAKE_TOOLCHAIN_FILE=<TOOLCHAINFILE> ..
\endcode 

\warning You must use this line in a <em>clean</em> directory, without previous cmake files. Otherwise you will have linking problems with some files like MPI (error "attempted static link of dynamic object").
    

\subsection secCMake_OtherCompilers Setting compilers to use
You can specify other compilers than detected by \em CMake from the command 
line with
\verbatim
cmake -DCMAKE_C_COMPILER=cc -DCMAKE_CXX_COMPILER=CC ..
\endverbatim

Please see for compiler troubleshooting \ref troubleshooting_compiler_errors .

If you need to choose another compiler, please consider writing your own 
toolchain file, so others can benefit from your knowledge. 


<hr>
\section secInstallation_of_additional_software Installation of Additional Software

Unfortunately on some systems it turned out that especially the build tool 
\em CMake, absolutely necessary to configure \ug4 (cf. \ref pageUG4Install, 
\ref secInstallCMake), was not available.
In such cases you have to install the required software yourself (typically 
locally).
For some installation instructions &mdash; including those for \em CMake &mdash;
see \ref pageAdditionalSoftware .


<hr>
\section secWindows Windows
<!--
TODO: Write installation instructions for Windows clusters and parallel computing on Windows
-->


<hr>
\section secMacOSX MacOSX

After installation of a MPI implementation (e.g. 
<a href="http://www.open-mpi.org/">OpenMPI</a>, for example via 
<a href="http://www.macports.org">MacPorts</a>:
<tt>sudo port install openmpi</tt>) you can use
\verbatim
mpirun -np <NP> ugshell $UGARGS
\endverbatim
to run \ug4.

*/
