8. Dive into real world cdist — cdist 6.9.4 documentation

# 8. Dive into real world cdist

## 8.1. Introduction

This walkthrough shows real world cdist configuration example.

Sample target host is named test.ungleich.ch. Just replace test.ungleich.ch with your target hostname.

Our goal is to configure python application hosting. For writing sample application we will use Bottle WSGI micro web-framework. It will use PostgreSQL database and it will list items from items table. It will be served by uWSGI server. We will also use the Nginx web server as a reverse proxy and we want HTTPS. For HTTPS we will use Let's Encrypt certificate.

For setting up hosting we want to use cdist so we will write a new type for that. This type will:

- install required packages

- create OS user, user home directory and application home directory

- create PostgreSQL database

- configure uWSGI

- configure Let's Encrypt certificate

- configure nginx.

Our type will not create the actual python application. Its intention is only to configure hosting for specified user and project. It is up to the user to create his/her applications.

So let's start.

## 8.2. Creating type layout

We will create a new custom type. Let's call it __sample_bottle_hosting.

Go to ~/.cdist/type directory (create it if it does not exist) and create new type layout:

    cd ~/.cdist/type
    mkdir __sample_bottle_hosting
    cd __sample_bottle_hosting
    touch manifest gencode-remote
    mkdir parameter
    touch parameter/required

## 8.3. Creating __sample_bottle_hosting type parameters

Our type will be configurable through the means of parameters. Let's define the following parameters:

projectname

  name for the project, needed for uWSGI ini file

user

  user name

domain

  target host domain, needed for Let's Encrypt certificate.

We define parameters to make our type reusable for different projects, user and domain.

Define required parameters:

    printf "projectname\n" >> parameter/required
    printf "user\n" >> parameter/required
    printf "domain\n" >> parameter/required

For details on type parameters see Defining parameters.

## 8.4. Creating __sample_bottle_hosting type manifest

Next step is to define manifest (~/.cdist/type/__sample_bottle_hosting/manifest). We also want our type to currently support only Devuan. So we will start by checking target host OS. We will use os global explorer:

    os=$(cat "$__global/explorer/os")

    case "$os" in
        devuan)
            :
        ;;
        *)
            echo "OS $os currently not supported" >&2
            exit 1
        ;;
    esac

If target host OS is not Devuan then we print error message to stderr and exit. For other OS-es support we should check and change package names we should install, because packages differ in different OS-es and in different OS distributions like GNU/Linux distributions. There can also be a different configuration locations (e.g. nginx config directory could be in /usr/local tree). If we detected unsupported OS we should error out. cdist will stop configuration process and output error message.

### 8.4.1. Creating user and user directories

Then we create user and his/her home directory and application home directory. We will use existing cdist types __user and __directory:

    user="$(cat "$__object/parameter/user")"
    home="/home/$user"
    apphome="$home/app"

    # create user
    __user "$user" --home "$home" --shell /bin/bash
    # create user home dir
    require="__user/$user" __directory "$home" \
        --owner "$user" --group "$user" --mode 0755
    # create app home dir
    require="__user/$user __directory/$home" __directory "$apphome" \
        --state present --owner "$user" --group "$user" --mode 0755

First we define user, home and apphome variables. User is defined by type's user parameter. Here we use require which is cdist's way to define dependencies. User home directory should be created after user is created. And application home directory is created after both user and user home directory are created. For details on require see Dependencies.

### 8.4.2. Installing packages

Install required packages using existing __package type. Before installing package we want to update apt package index using __apt_update_index:

    # define packages that need to be installed
    packages_to_install="nginx uwsgi-plugin-python3 python3-dev python3-pip postgresql postgresql-contrib libpq-dev python3-venv uwsgi python3-psycopg2"

    # update package index
    __apt_update_index
    # install packages
    for package in $packages_to_install
        do require="__apt_update_index" __package $package --state=present
    done

Here we use shell for loop. It executes require="__apt_update_index" __package for each member in a list we define in packages_to_install variable. This is much nicer then having as many require="__apt_update_index" __package lines as there are packages we want to install.

For python packages we use __package_pip:

    # install pip3 packages
    for package in bottle bottle-pgsql; do
        __package_pip --pip pip3 $package
    done

### 8.4.3. Creating PostgreSQL database

Create PostgreSQL database using __postgres_database and __postgres_role for creating database user:

    #PostgreSQL db & user
    postgres_server=postgresql

    # create PostgreSQL db user
    require="__package/postgresql" __postgres_role $user --login --createdb
    # create PostgreSQL db
    require="__postgres_role/$user __package/postgresql" __postgres_database $user \
        --owner $user

### 8.4.4. Configuring uWSGI

Configure uWSGI using __file type:

    # configure uWSGI
    projectname="$(cat "$__object/parameter/projectname")"
    require="__package/uwsgi" __file /etc/uwsgi/apps-enabled/$user.ini \
                --owner root --group root --mode 0644 \
                --state present \
                --source - << EOF
    [uwsgi]
    socket = $apphome/uwsgi.sock
    chdir = $apphome
    wsgi-file = $projectname/wsgi.py
    touch-reload = $projectname/wsgi.py
    processes = 4
    threads = 2
    chmod-socket = 666
    daemonize=true
    vacuum = true
    uid = $user
    gid = $user
    EOF

We require package uWSGI present in order to create /etc/uwsgi/apps-enabled/$user.ini file. Installation of uWSGI also creates configuration layout: /etc/uwsgi/apps-enabled. If this directory does not exist then __file type would error. We also use stdin as file content source. For details see Input from stdin. For feeding stdin we use here-document (<< operator). It allows redirection of subsequent lines read by the shell to the input of a command until a line containing only the delimiter and a newline, with no blank characters in between (EOF in our case).

### 8.4.5. Configuring nginx for Let's Encrypt and HTTPS redirection

Next configure nginx for Let's Encrypt and for HTTP -> HTTPS redirection. For this purpose we will create new type __sample_nginx_http_letsencrypt_and_ssl_redirect and use it here:

    domain="$(cat "$__object/parameter/domain")"
    webroot="/var/www/html"
    __sample_nginx_http_letsencrypt_and_ssl_redirect "$domain" --webroot "$webroot"

### 8.4.6. Configuring certificate creation

After HTTP nginx configuration we will create Let's Encrypt certificate using __letsencrypt_cert type. For Let's Encrypt cert configuration ensure that there is a DNS entry for your domain. We assure that cert creation is applied after nginx HTTP is configured for Let's Encrypt to work:

    # create SSL cert
    require="__package/nginx __sample_nginx_http_letsencrypt_and_ssl_redirect/$domain" \
        __letsencrypt_cert --admin-email admin@test.ungleich.ch \
            --webroot "$webroot" \
            --automatic-renewal \
            --renew-hook "service nginx reload" \
            --domain "$domain" \
            "$domain"

### 8.4.7. Configuring nginx HTTPS server with uWSGI upstream

Then we can configure nginx HTTPS server that will use created Let's Encrypt certificate:

    # configure nginx
    require="__package/nginx __letsencrypt_cert/$domain" \
        __file "/etc/nginx/sites-enabled/https-$domain" \
        --source - --mode 0644 << EOF
    upstream _bottle {
        server unix:$apphome/uwsgi.sock;
    }

    server {
        listen 443;
        listen [::]:443;

        server_name $domain;

        access_log  /var/log/nginx/access.log;

        ssl on;
        ssl_certificate      /etc/letsencrypt/live/$domain/fullchain.pem;
        ssl_certificate_key  /etc/letsencrypt/live/$domain/privkey.pem;

        client_max_body_size 256m;

        location / {
            try_files \$uri @uwsgi;
        }

        location @uwsgi {
            include uwsgi_params;
            uwsgi_pass _bottle;
        }
    }
    EOF

Now our manifest is finished.

### 8.4.8. Complete __sample_bottle_hosting type manifest listing

Here is complete __sample_bottle_hosting type manifest listing, located in ~/.cdist/type/__sample_bottle_hosting/manifest:

    #!/bin/sh

    os=$(cat "$__global/explorer/os")

    case "$os" in
        devuan)
            :
        ;;
        *)
            echo "OS $os currently not supported" >&2
            exit 1
        ;;
    esac

    projectname="$(cat "$__object/parameter/projectname")"
    user="$(cat "$__object/parameter/user")"
    home="/home/$user"
    apphome="$home/app"
    domain="$(cat "$__object/parameter/domain")"

    # create user
    __user "$user" --home "$home" --shell /bin/bash
    # create user home dir
    require="__user/$user" __directory "$home" \
        --owner "$user" --group "$user" --mode 0755
    # create app home dir
    require="__user/$user __directory/$home" __directory "$apphome" \
        --state present --owner "$user" --group "$user" --mode 0755

    # define packages that need to be installed
    packages_to_install="nginx uwsgi-plugin-python3 python3-dev python3-pip postgresql postgresql-contrib libpq-dev python3-venv uwsgi python3-psycopg2"

    # update package index
    __apt_update_index
    # install packages
    for package in $packages_to_install
        do require="__apt_update_index" __package $package --state=present
    done
    # install pip3 packages
    for package in bottle bottle-pgsql; do
        __package_pip --pip pip3 $package
    done

    #PostgreSQL db & user
    postgres_server=postgresql

    # create PostgreSQL db user
    require="__package/postgresql" __postgres_role $user --login --createdb
    # create PostgreSQL db
    require="__postgres_role/$user __package/postgresql" __postgres_database $user \
        --owner $user
    # configure uWSGI
    require="__package/uwsgi" __file /etc/uwsgi/apps-enabled/$user.ini \
                --owner root --group root --mode 0644 \
                --state present \
                --source - << EOF
    [uwsgi]
    socket = $apphome/uwsgi.sock
    chdir = $apphome
    wsgi-file = $projectname/wsgi.py
    touch-reload = $projectname/wsgi.py
    processes = 4
    threads = 2
    chmod-socket = 666
    daemonize=true
    vacuum = true
    uid = $user
    gid = $user
    EOF

    # setup nginx HTTP for Let's Encrypt and SSL redirect
    domain="$(cat "$__object/parameter/domain")"
    webroot="/var/www/html"
    __sample_nginx_http_letsencrypt_and_ssl_redirect "$domain" --webroot "$webroot"

    # create SSL cert
    require="__package/nginx __sample_nginx_http_letsencrypt_and_ssl_redirect/$domain" \
        __letsencrypt_cert --admin-email admin@test.ungleich.ch \
            --webroot "$webroot" \
            --automatic-renewal \
            --renew-hook "service nginx reload" \
            --domain "$domain" \
            "$domain"

    # configure nginx
    require="__package/nginx __letsencrypt_cert/$domain" \
        __file "/etc/nginx/sites-enabled/https-$domain" \
        --source - --mode 0644 << EOF
    upstream _bottle {
        server unix:$apphome/uwsgi.sock;
    }

    server {
        listen 443;
        listen [::]:443;

        server_name $domain;

        access_log  /var/log/nginx/access.log;

        ssl on;
        ssl_certificate      /etc/letsencrypt/live/$domain/fullchain.pem;
        ssl_certificate_key  /etc/letsencrypt/live/$domain/privkey.pem;

        client_max_body_size 256m;

        location / {
            try_files \$uri @uwsgi;
        }

        location @uwsgi {
            include uwsgi_params;
            uwsgi_pass _bottle;
        }
    }
    EOF

## 8.5. Creating __sample_bottle_hosting type gencode-remote

Now define gencode-remote script: ~/.cdist/type/__sample_bottle_hosting/gencode-remote. After manifest is applied it should restart uWSGI and nginx services so that our configuration is active. Our gencode-remote looks like the following:

    echo "service uwsgi restart"
    echo "service nginx restart"

Our __sample_bottle_hosting type is now finished.

## 8.6. Creating __sample_nginx_http_letsencrypt_and_ssl_redirect type

Let's now create __sample_nginx_http_letsencrypt_and_ssl_redirect type:

    cd ~/.cdist/type
    mkdir __sample_nginx_http_letsencrypt_and_ssl_redirect
    cd __sample_nginx_http_letsencrypt_and_ssl_redirect
    mkdir parameter
    echo webroot > parameter/required
    touch manifest
    touch gencode-remote

Edit manifest:

    domain="$__object_id"
    webroot="$(cat "$__object/parameter/webroot")"
    # make sure we have nginx package
    __package nginx
    # setup Let's Encrypt HTTP acme challenge, redirect HTTP to HTTPS
    require="__package/nginx" __file "/etc/nginx/sites-enabled/http-$domain" \
        --source - --mode 0644 << EOF
    server {
        listen *:80;
        listen [::]:80;

        server_name $domain;

        # Let's Encrypt
        location /.well-known/acme-challenge/ {
            root $webroot;
        }

        # Everything else -> SSL
        location / {
            return 301 https://\$host\$request_uri;
        }
    }

    EOF

Edit gencode-remote:

    echo "service nginx reload"

## 8.7. Creating init manifest

Next create init manifest:

    cd ~/.cdist/manifest
    printf "__sample_bottle_hosting --projectname sample --user app --domain \$__target_host sample\n" > sample

Using this init manifest our target host will be configured using our __sample_bottle_hosting type with projectname sample, user app and domain equal to __target_host. Here the last positional argument sample is type's object id. For details on __target_host and __object_id see Environment variables (for reading) reference.

## 8.8. Configuring host

Finally configure test.ungleich.ch:

    cdist config -v -i ~/.cdist/manifest/sample test.ungleich.ch

After cdist configuration is successfully finished our host is ready.

## 8.9. Creating python bottle application

We now need to create Bottle application. As you remember from the beginning of this walkthrough our type does not create the actual python application, its intention is only to configure hosting for specified user and project. It is up to the user to create his/her applications.

Become app user:

    su -l app

### 8.9.1. Preparing database

We need to prepare database for our application. Create table and insert some items:

    psql -c "create table items (item varchar(255));"

    psql -c "insert into items(item) values('spam');"
    psql -c "insert into items(item) values('eggs');"
    psql -c "insert into items(item) values('sausage');"

### 8.9.2. Creating application

Next create sample app:

    cd /home/app/app
    mkdir sample
    cd sample

Create app.py with the following content:

    #!/usr/bin/env python3

    import bottle
    import bottle_pgsql

    app = application = bottle.Bottle()
    plugin = bottle_pgsql.Plugin('dbname=app user=app password=')
    app.install(plugin)

    @app.route('/')
    def show_index(db):
        db.execute('select * from items')
        items = db.fetchall() or []
        rv = '<html><body><h3>Items:</h3><ul>'
        for item in items:
            rv += '<li>' + str(item['item']) + '</li>'
        rv += '</ul></body></html>'
        return rv

    if __name__ == '__main__':
        bottle.run(app=app, host='0.0.0.0', port=8080)

Create wsgi.py with the following content:

    import os

    os.chdir(os.path.dirname(__file__))

    import app
    application = app.app

We have configured uWSGI with touch-reload = $projectname/wsgi.py so after we have changed our wsgi.py file uWSGI reloads the application.

Our application selects and lists items from items table.

### 8.9.3. Opening application

Finally try the application:

    http://test.ungleich.ch/

It should redirect to HTTPS and return:

### Items:

- spam
- eggs
- sausage

## 8.10. What's next?

Continue reading next sections ;)