Buildbot — ASE documentation

### Navigation

- index
- modules |
- ASE development »

# Buildbot

buildbot is a continuous integration system. The server called `buildmaster` defines and schedules processes of building and testing of software to be performed by so called builders on machines called `buildslaves` (or buildbots). One buildslave can have several builders associated (for example a given operating system version `buildslave` performs testing of several software branches `builders`). One builder can also be associated with several `buildslaves` (several, “identical” machines running the same operating system), in which case `buildmaster` will schedule the given builder to switch periodically between the `buildslaves`. At the time of writing (2012) buildbot does not offer satisfactory security mechanisms, and due to clear text passwords stored/transferred both on `buildmaster` and `buildslaves` external security measures like firewall, proxy, ssh/stunnel should be used.

The sections below describe the configuration of `buildmaster` and `buildslaves`.

## Configuration of buildmaster

The `buildmaster` should be running as unprivileged user (usually called `buildmaster`). The user and group need to be created.

In order to install buildbot on RHEL6 system you need:

    yum install python-twisted python-jinja2 python-tempita

Additional dependencies + buildbot itself need to be installed manually:

    wget https://downloads.sourceforge.net/project/sqlalchemy/sqlalchemy/0.7.9/SQLAlchemy-0.7.9.tar.gz
    wget http://sqlalchemy-migrate.googlecode.com/files/sqlalchemy-migrate-0.7.2.tar.gz

It is sufficient to unpack the files and set `PYTHONPATH`, and `PATH` accordingly. Buildbot may need to be patched for twisted compatibility http://www.npcglib.org/~stathis/blog/2012/10/20/bug-buildbot-dies-with-exceptions-importerror-cannot-import-name-noresource/

Proceed with buildbot configuration:

- create the master:

      buildbot create-master --relocatable python-ase

- reconfigure the master with doc/development/master.cfg:

      cp master.cfg python-ase
      buildbot checkconfig python-ase/master.cfg
      buildbot start python-ase

- consider setting up a crontab which starts `buildmaster` after the server reboot (this solution is not reliable, deploy rather init scripts - see below):

      # run every 5 minutes
      */5 * * * * sh ~/python-ase-restart.sh

  with the following `python-ase-restart.sh`:

      bot=python-ase
      # no pid file - assume buildbot is not running
      if ! test -f $bot/twistd.pid;
      then
          buildbot restart $bot || buildbot start $bot
      else
          # pid file exists but buildbot is not running
          pid=`cat $bot/twistd.pid`
          if test -z `ps -p $pid -o pid=`;
          then
              buildbot restart $bot
          fi
      fi

- or create a system V init script under `/etc/init.d`

      #!/bin/sh
      #
      # python-ase:	python-ase buildmaster
      #
      # chkconfig:	345 98 02
      # description:	python-ase buildmaster

      # LSB init-info
      ### BEGIN INIT INFO
      # Provides:          python-ase
      # Required-Start:    $network
      # Required-Stop:     $network
      # Default-Start:     2 3 4 5
      # Default-Stop:      0 1 6
      # Short-Description: python-ase buildmaster
      ### END INIT INFO

      # Source function library.
      if [ -e /etc/init.d/functions ]; then
      	. /etc/init.d/functions
      fi

      # LSB functions
      . /lib/lsb/init-functions

      # Check that networking is configured.
      [ "${NETWORKING}" = "no" ] && exit 0

      RUN_AS=buildmaster-username
      PYTHON_ASE_HOME=/home/$RUN_AS/python-ase
      test -d $PYTHON_ASE_HOME || exit 5

      LOGFILE=$PYTHON_ASE_HOME/../python-ase.log
      PIDFILE=$PYTHON_ASE_HOME/../python-ase.pid
      LOCKFILE=$PYTHON_ASE_HOME/../python-ase.lock

      start() {
      	echo -n $"Starting python-ase buildmaster: "
      	dostatus > /dev/null 2>&1
      	if [ $RETVAL -eq 0 ]
      	then
      		echo -n $"python-ase buildmaster already running"
      		log_failure_msg
      		RETVAL=1
      		return
      	fi
      	#su - $RUN_AS -s /bin/sh -c "exec nohup /bin/sh $PYTHON_ASE_HOME/../python-ase-start.sh >> $LOGFILE 2>&1 &"
              # don't produce log
      	su - $RUN_AS -s /bin/sh -c "exec nohup /bin/sh $PYTHON_ASE_HOME/../python-ase-start.sh >> /dev/null 2>&1 &"
      	RETVAL=$?
      	if [ $RETVAL -eq 0 ]
      	then
      		sleep 5
      		su - $RUN_AS -s /bin/sh -c "cat $PYTHON_ASE_HOME/twistd.pid > $PIDFILE"
      		su - $RUN_AS -s /bin/sh -c "touch $LOCKFILE"
      		log_success_msg
      	else
      		log_failure_msg
      	fi
      	return $RETVAL
      }

      stop() {
      	echo -n $"Shutting down python-ase buildmaster: "
      	kill $(su - $RUN_AS -s /bin/sh -c "cat $PIDFILE 2>/dev/null") > /dev/null 2>&1
      	RETVAL=$?
      	sleep 5
      	if [ $RETVAL -eq 0 ]
      	then
      		su - $RUN_AS -s /bin/sh -c "rm -f $PIDFILE $LOCKFILE"
      		log_success_msg
      	else
      		log_failure_msg
      	fi
      	return $RETVAL
      }

      restart() {
      	stop
      	start
      }

      condrestart() {
      	[ -f $LOCKFILE ] && restart || :
      }

      dostatus() {
      	kill -0 $(cat $PIDFILE 2>/dev/null) > /dev/null 2>&1
      	RETVAL=$?
      	if [ $RETVAL -eq 0 ]
      	then
      		echo "python-ase buildmaster (pid $(cat $PIDFILE 2>/dev/null)) is running..."
      	else
      		if [ -f $PIDFILE ]
      		then
      			echo "python-ase buildmaster dead but pid file exists"
      			RETVAL=1
      			return
      		fi
      		if [ -f $LOCKFILE ]
      		then
      			echo "python-ase buildmaster dead but subsys locked"
      			RETVAL=2
      			return
      		fi
      		echo "python-ase buildmaster is stopped"
      		RETVAL=3
      	fi
      }

      # See how we were called.
      case "$1" in
        start)
      	start
      	;;
        stop)
      	stop
      	;;
        status)
      	dostatus
      	;;
        restart|reload)
      	restart
      	;;
        condrestart)
      	condrestart
      	;;
        *)
      	echo $"Usage: $0 {start|stop|status|restart|reload|condrestart}"
      	exit 1
      esac

      exit $RETVAL

  The service is added with:

      chkconfig --add python-ase-buildmaster

  started with:

      service python-ase-buildmaster start

  end enabled for boot time with:

      chkconfig python-ase-buildmaster on

  See also an example of `systemd` script in the section below.

- consider protecting the `buildmaster` web interface by, e.g. apache reverse proxy (http://httpd.apache.org/docs/2.4/mod/mod_proxy.html). The basic configuration file may look like

      # EEE.EEE.EEE.EEE is the IP of your External apache server:
      # ase-buildbot.fysik.dtu.dk

      <VirtualHost EEE.EEE.EEE.EEE:80>
          ServerName ase-buildbot.fysik.dtu.dk
          ServerAlias ase-buildbot ase-buildbot.fysik.dtu.dk
          Redirect / https://ase-buildbot.fysik.dtu.dk/
      </VirtualHost>

      <VirtualHost EEE.EEE.EEE.EEE:443>
          ServerName ase-buildbot.fysik.dtu.dk
          ServerAlias ase-buildbot ase-buildbot.fysik.dtu.dk

          # important http://httpd.apache.org/docs/current/mod/mod_proxy.html
          ProxyRequests Off

          # III.III.III.III:8080 the IP and port of the Internal buildbot web server

          ProxyPass / http://III.III.III.III:8080/
          ProxyPassReverse / http://III.III.III.III:8080/

          <Proxy http://III.III.III.III:8080/*>
      	Options -Indexes
      	Order deny,allow
      	Deny from all
      	Allow from 127.0.0.1
      	Allow from ::1
      	#Allow from all
          </Proxy>

      SSLProxyEngine on

      # provide an authority signed certificate
      SSLCertificateFile /etc/pki/tls/certs/localhost.crt
      SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

      </VirtualHost>

      # vim: filetype=apache shiftwidth=4 tabstop=4 wrap!

## Installation and configuration of buildslaves

Some examples are given below. The configuration should be performed under an unprivileged `buildslave`, or buildbot user account.

### Installation

#### Fedora

Install with:

    yum install buildbot-slave

Perform Configuration now.

After configuring buildbot-slave, you can configure `systemd` service by creating `python-ase-fedora-18-x86_64-gcc-2.7.service` file under `/usr/lib/systemd/system`.

    [Unit]
    Description=buildslave python-ase-fedora+18+x86_64+gcc+2.7 stock
    After=network.target

    [Service]
    Type=forking
    User=buildslave-username
    Group=buildslave-groupname
    ExecStart=/usr/bin/sh /home/buildslave-username/python-ase-fedora+18+x86_64+gcc+2.7-start.sh
    Restart=always

    [Install]
    WantedBy=multi-user.target

`python-ase-fedora+18+x86_64+gcc+2.7-start.sh` simply exports the necessary environment variables (if needed) and starts `buildslave` (use the full path), e.g.:

    #!/bin/sh
    bot=/home/buildslave-username/python-ase-fedora+18+x86_64+gcc+2.7
    buildslave start $bot

Choose `User` and `Group` under which `buildslave` will be running. The service is started with:

    systemctl start python-ase-fedora-18-x86_64-gcc-2.7.service

In order to force the service to be started at boot time create a link:

    cd /etc/systemd/system/multi-user.target.wants
    ln -s /usr/lib/systemd/system/python-ase-fedora-18-x86_64-gcc-2.7.service .

#### OS X

Configure Homebrew, and:

    brew install subversion

Configure a virtualenv, and then:

    pip install numpy
    pip install buildbot-slave

Perform Configuration now.

#### RHEL5

Install recent `python-setuptools` in order to get `easy_install`:

    mkdir ~/buildbot-slave-el5
    export PATH=$HOME/buildbot-slave-el5:${PATH}
    export PYTHONPATH=$HOME/buildbot-slave-el5:${PYTHONPATH}
    wget http://pypi.python.org/packages/2.4/s/setuptools/setuptools-0.6c11-py2.4.egg
    sh setuptools-0.6c11-py2.4.egg --install-dir=$HOME/buildbot-slave-el5

then:

    easy_install --install-dir=$HOME/buildbot-slave-el5 zope.interface==3.6.7
    easy_install --install-dir=$HOME/buildbot-slave-el5 twisted==9.0.0  # ignore errors
    easy_install --install-dir=$HOME/buildbot-slave-el5 buildbot-slave

Perform Configuration now.

#### RHEL6

Install `build-slave` and dependencies:

    mkdir ~/buildbot-slave-el6
    export PATH=$HOME/buildbot-slave-el6:${PATH}
    export PYTHONPATH=$HOME/buildbot-slave-el6:${PYTHONPATH}
    easy_install --install-dir=$HOME/buildbot-slave-el6 buildbot-slave

Perform Configuration now.

#### RHEL7

Install `build-slave` and dependencies:

    mkdir ~/buildbot-slave-el7
    export PATH=$HOME/buildbot-slave-el7:${PATH}
    export PYTHONPATH=$HOME/buildbot-slave-el7:${PYTHONPATH}
    easy_install --install-dir=$HOME/buildbot-slave-el7 buildbot-slave

Perform Configuration now.

#### Windows

`build-slave` can be installed and configured to start as a service on Windows http://trac.buildbot.net/wiki/RunningBuildbotOnWindows. This involves several steps:

1. install Python(x,y) from https://code.google.com/p/pythonxy/wiki/Downloads

2. configure distutils to use mingw. First enable showing file extensions:

       Open a folder with IE -> Folder and search options-> View -> Folder Options:
       Check: Show hidden files, ...; uncheck: Hide extensions for known file types.

   Then, in notepad, create C:\python27\lib\distutils\distutils.cfg, containing:

       [build]
       compiler=mingw32

3. install `build-slave` on the command line:

       C:\python27\scripts\easy_install.exe buildbot-slave

4. create a local (domain `computer-name`) user that will run the `buildbot` service:

       control panel->administrative tools->computer management->local users and groups->users->new user: buildslave-username.
       Click the created user: member of: administrators->check names

5. grant `buildslave-username` the ability to run the services. Login as `computer-name\buildslave-username`: Run `secpol.msc` on the command line as administrator (task bar->cmd->right click: run as administrator):

   - Select the “Local Policies” folder
   - Select the “User Rights Assignment” folder
   - Double click “Log on as a service”
   - Use “Add User or Group...” to add your user here.

   Select the correct “from this location” (may require login as the current domain administrator) and Enter the object names: `computer-name\buildslave-username`.

6. on the command line install the `buildbot` service:

       buildbot_service.py --user computer-name\buildslave-username --password thepassword --startup auto install

7. start the service (for the moment it does not start any `buildslave`, because they are not configured yet):

       Start->Control Panel> Administrative Tools->Services->Buildbot (Start)

   There are additional steps mentioned in the buildbot wiki, but it seems just to work on Windows 7.

8. run regedit as administrator (type “regedit” on the command line) and add “directories” parameter of the String Value type, containing paths to all your `buildslave` instances (they will be configured in the Configuration section below):

       HKEY_LOCAL_MACHINE\System\CurrentControlSet\services\Buildbot->paramaters->new (String Value): directories C:\python-ase-windows+7+AMD64+msc+2.7;C:\proj2

9. configure `buildslave` instance as described in the Configuration section below and start the service again (point 7.). Test that `buildslave` comes online, and verify that the service starts after reboot.

### Configuration

After having installed the buildbot create a name which will identify your `buildslave`. Obtain the first part of the name for your `buildslave` by running doc/development/master.cfg:

    python master.cfg

This will return a string like `redhat+6+x86_64+gcc+2.6` (OS+OSversion+Bitness+Ccompiler+PythonVersion). Append it with something that identifies your `buildslave`, for example `gpu`. For a very special system you can use a name like `mycluster+x86_64+open64+2.5 gpu`. Note that ASE buildbot will perform verification of python version based on the `buildslave` name so stay consistent!

Generate a strong `buildslave` password with Creating an encrypted password for SVN access. Don’t reuse any valuable passwords. You don’t need to remember it, buildbot stores it plain text!

Create the `buildslave` with:

    buildslave create-slave python-ase-redhat+6+x86_64+gcc+2.6 buildbot.fysik.dtu.dk:ASE_BB_PORT "redhat+6+x86_64+gcc+2.6 gpu" "password"

ASE_BB_PORT is the port ASE buildbot uses for communication with the `buildslave`. You will have to tell us the name and password of your `buildslave`. Please contact ase-developers list Mailing Lists, but don’t send the name and password there!

Edit the `python-ase-redhat+6+x86_64+gcc+2.6/info/{admin,info}` files: describe your `buildslave` configuration relevant for the builder process in the `info` file.

Note that before starting the slave you need to perform an temporary svn checkout of ASE in order to accept the certificate permanently.

Start the `buildslave` with:

    buildslave start python-ase-redhat+6+x86_64+gcc+2.6

Don’t forget to configure a crontab job or a service as described in the previous sections.

By default all slaves run the continuous integration for the trunk. If you prefer your `buildslave` works also on one of the branches, write this in the email to ase-developers Mailing Lists.

### Navigation

- index
- modules |
- ASE development »