Installation of GC3Pie — gc3pie development (SVN $Revision$) documentation

### Navigation

- index
- modules |
- next |
- previous |
- gc3pie development (SVN $Revision$) documentation »
- User Documentation »

# Installation of GC3Pie

## Quick start

We provide an installation script which automatically try to install GC3pie in your home directory. The quick installation procedure has only been tested on variants of the GNU/Linux operating system. (However, the script should work on MacOSX as well, provided you follow the preparation steps outlined in the “MacOSX installation” section below.)

To install GC3Pie just type this at your terminal prompt:

    sh -c "$(wget -O- http://gc3pie.googlecode.com/svn/install.sh)"

If wget is not installed in your computer, you should use curl instead:

    sh -c "$(curl -s http://gc3pie.googlecode.com/svn/install.sh)"

The above command creates a directory `$HOME/gc3pie` and installs the latest release of GC3Pie and all its dependencies into it.

In case you have trouble running the installation script, please send an email to gc3pie@googlegroups.com asking for help. Include the full output of the script in your email, in order to help us to identify the problem.

## Non-standard installation options

The installation script accept a few options that select alternatives to the standard behavior. In order to use these options, you have to:

1. download the installation script into a file named `install.sh`:

       wget http://gc3pie.googlecode.com/svn/install.sh

2. run the command:

       sh ./install.sh [options]

   replacing the string `[options]` with the actual options you want to pass to the script.

The accepted options are as follows:

> `-d DIRECTORY`

> > Install GC3Pie in location `DIRECTORY` instead of `$HOME/gc3pie`

> `--overwrite`

> > Overwrite the destination directory if it already exists. Default behavior is to abort installation.

> `--develop`

> > Instead of installing the latest release of GC3Pie, it will install the development branch from the SVN repository.

> `--yes`

> > Run non-interactively, and assume a “yes” reply to every question.

> `-p PYTHON`

> > Uses the given `PYTHON` program as python interpreter. By default the installation script looks for a `python` binary in the standard `$PATH`.

> `--no-gc3apps`

> > Do not install any of the GC3Apps, e.g., `gcodeml`, `grosetta` and `ggamess`.

## Manual installation

In case you can’t or don’t want to use the automatic installation script, the following instructions will guide you through all the steps needed to manually install GC3Pie on your computer.

These instructions show how to install GC3Pie from the GC3 source repository into a separate python environment (called virtualenv). Installation into a virtualenv has two distinct advantages:

> - All code is confined in a single directory, and can thus be easily replaced/removed.
> - Better dependency handling: additional Python packages that GC3Pie depends upon can be installed even if they conflict with system-level packages.

0. Install software prerequisites:

   - On Debian/Ubuntu, install packages: `subversion`, `python-dev`, `python-profiler` and the C/C++ compiler:

         apt-get install subversion python-dev python-profiler gcc g++

   - On CentOS5, install packages `subversion` and `python-devel` and the C/C++ compiler:

         yum install subversion python-devel gcc gcc-c++

   - On other Linux distributions, you will need to install:

     - the `svn` command (from the SubVersion VCS)
     - Python development headers and libraries (for installing extension libraries written in C/C++)
     - the Python package `pstats` (it’s part of the Python standard library, but sometimes it needs separate installation)
     - a C/C++ compiler (this is usually installed by default).

   If you intend to use also the ARC backend (required for SMSCG), you need the NorduGrid/ARC binaries and a working `slcs-init` command installed on the same machine where GC3Pie is. You can find instructions for installing it at:

   > http://www.smscg.ch/WP/middleware/ARC1/smscg_ui_Installation.html

   Additional OS-specific installation details can be found at:

   > http://code.google.com/p/gc3pie/wiki/OSSpecificInstallDetails

1. If virtualenv is not already installed on your system, get the Python package and install it:

       wget http://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.7.tar.gz
       tar -xzf virtualenv-1.7.tar.gz && rm virtualenv-1.7.tar.gz
       cd virtualenv-1.7/

   If you are installing as root, the following command is all you need:

       python setup.py install

   If instead you are installing as a normal, unprivileged user, things get more complicated:

       export PYTHONPATH=$HOME/lib64/python:$HOME/lib/python:$PYTHONPATH
       export PATH=$PATH:$HOME/bin
       mkdir -p $HOME/lib/python
       python setup.py install --home $HOME

   You will also have to add the two export lines above to the:

   - $HOME/.bashrc file, if using the bash shell or to the
   - $HOME/.cshrc file, if using the tcsh shell.

   In any case, once virtualenv has been installed, you can exit its directory and remove it:

       cd ..
       rm -rf virtualenv-1.7

2. Create a virtualenv to host the GC3Pie installation, and `cd` into it:

       virtualenv --system-site-packages $HOME/gc3pie
       cd $HOME/gc3pie/
       source bin/activate

   In this step and in the following ones, the directory `$HOME/gc3pie` is going to be the installation folder of GC3Pie. You can change this to another directory path; any directory that’s writable by your Linux account will be OK.

   If you are installing system-wide as `root`, we suggest you install GC3Pie into `/opt/gc3pie` instead.

3. Check-out the `gc3pie` files in a `src/` directory:

       svn co http://gc3pie.googlecode.com/svn/branches/2.0/gc3pie src

4. Install the `gc3pie` in “develop” mode, so any modification pulled from subversion is immediately reflected in the running environment:

       cd src/
       env CC=gcc ./setup.py develop
       cd .. # back into the `gc3pie` directory

   This will place all the GC3Pie command into the `gc3pie/bin/` directory.

5. GC3Pie comes with driver scripts to run and manage large families of jobs from a few selected applications. These scripts are not installed by default because not everyone needs them.

   Run the following commands to install the driver scripts for the applications you need:

       # if you are insterested in GAMESS, do the following
       ln -s '../src/gc3apps/gamess/ggamess.py' bin/ggamess

       # if you are insterested in Rosetta, do the following
       ln -s '../src/gc3apps/rosetta/gdocking.py' bin/gdocking
       ln -s '../src/gc3apps/rosetta/grosetta.py' bin/grosetta

       # if you are insterested in Codeml, do the following
       ln -s '../src/gc3apps/codeml/gcodeml.py' bin/gcodeml

6. Before you can actually run the GC3Pie, you need to have a working configuration file; the ConfigurationFile Wiki page provides an explanation of the syntax.

   The example configuration file can be used as a startup point if you want to enable access to the SMSCG infrastructure. Before you can actually use this file, you will need to:

   - insert into it three values, for which we can NOT provide defaults:

   > 1. `aai_username`:

   >      This is the “username” you are asked for when accessing any SWITCHaai/Shibboleth web page, e.g., https://gc3-aai01.uzh.ch/secure/

   > 2. `idp`:

   >      Find this out with the command “slcs-info”: it prints a list of IdP (Identity Provider IDs) followed by the human-readable name of the associated institution. Pick the one that corresponds to you University. It is always the last two components of the University’s Internet domain name (e.g., “uzh.ch” or “ethz.ch”).

   > 3. `vo`:

   >      In order to use SMSCG, you must sign up to a VO (Virtual Organisation). One the words “life”, “earth”, “atlas” or “crypto” should be here. Find out more at: http://www.smscg.ch/www/user/

   > Please, remember to uncomment the three lines, too.

   - Enable the SMSCG resource in the `Examples` section of the file. To do this simply uncomment the following lines:

         # [resource/smscg]
         # enabled = false
         # name = smscg
         # type = arc0
         # auth = smscg
         # arc_ldap = ldap://giis.smscg.ch:2135/o=grid/mds-vo-name=Switzerland
         # max_cores_per_job = 256
         # max_memory_per_core = 2
         # max_walltime = 24
         # ncores = 8000
         # architecture = i686, x86_64

7. Now you can check your GC3Pie installation; just type the command:

       gc3utils --help

   and you should see the following output appear on your screen:

       Usage: gc3utils COMMAND [options]

       Command `gc3utils` is a unified front-end to computing resources.
       You can get more help on a specific sub-command by typing::
         gc3utils COMMAND --help
       where command is one of these:
         clean
         get
         info
         kill
         list
         resub
         stat
         tail

   If you get some errors, do not despair! The GC3Pie users mailing-list <gc3pie@googlegroups.com> is there to help you :-)

## Upgrade

If you used the installation script, the fastest way to upgrade is just to reinstall:

0. De-activate the current GC3Pie virtual environment:

       deactivate

   (If you get an error “command not found”, do not worry and proceed on to the next step; in case of other errors please stop here and report to the GC3Pie users mailing-list <mailto:gc3pie.googlegroups.com>.)

1. Move the `$HOME/gc3pie` directory to another location, e.g.:

       mv $HOME/gc3pie $HOME/gc3pie.OLD

2. Reinstall GC3Pie using the quick-install script (top of this page).

3. Once you have verified that your new installation is working, you can remove the `$HOME/gc3pie.OLD` directory.

If instead you installed GC3Pie using the “manual installation” instructions, then the following steps will update GC3Pie to the latest version in the code repository:

1. cd to the directory containing the GC3Pie virtualenv; assuming it is named `gc3pie` as in the above installation instructions, you can issue the commands:

       cd $HOME/gc3pie # use '/opt/gc3pie' if root

2. Activate the virtualenv:

       source bin/activate

3. Upgrade the gc3pie source and run the `setup.py` script again:

       cd src
       svn up
       env CC=gcc ./setup.py develop

Note: A major restructuring of the SVN repository took place in r1124 to r1126 (Feb. 15, 2011); if your sources are older than SVN r1124, these upgrade instructions will not work, and you must reinstall completely. You can check what version the SVN sources are, by running the svn info command in the src directory: watch out for the Revision: line.

## MacOSX Installation

Installation on MacOSX machines is possible, however there are still a few issues. Please let us know on the GC3Pie users mailing-list <mailto:gc3pie@googlegroups.com> if you need MacOSX support.

1. Standard usage of the installation script (i.e., with no options) works, but you have to use curl since wget is not installed by default.
2. In order to install GC3Pie you will need to install XCode and, in some of the MacOSX versions, also the Command Line Tools for XCode
3. Options can only be given in the abbreviated one-letter form (e.g., `-d`); the long form (e.g., `--directory`) will not work.
4. The shellcmd backend of GC3Pie depends on the GNU `time` command, which is not installed on MacOSX by default. This means that with a standard MacOSX installation the shellcmd resource will not work. However:
   - other resources, like pbs via ssh transport, will work.
   - you can install the GNU time command either via MacPorts, Fink, Homebrew or from the this url. After installing it you don’t need to update your `PATH` environment variable, it’s enough to set the `time_cmd` option in your GC3Pie configuration file.
5. At the time of this writing, the ARC client libraries are not available on MacOSX, so ARC backends will not work.

## HTML Documentation

HTML documentation for the GC3Libs programming interface can be read online at:

> http://gc3pie.googlecode.com/svn/branches/2.0/gc3pie/docs/html/index.html

If you installed GC3Pie manually, or if you installed it using the `install.sh` script with the `--develop` option, you can also access a local copy of the documentation from the sources:

    cd $HOME/gc3pie # or wherever the gc3pie virtualenv is installed
    cd src/docs
    make html

Note that you need the Python package Sphinx in order to build the documentation locally.

### Navigation

- index
- modules |
- next |
- previous |
- gc3pie development (SVN $Revision$) documentation »
- User Documentation »