Title:  Deploying Lamson And OneShotBlog

These instructions will teach you how to setup a completely clean Python 2.6 installation,
a virtualenv with lamson, and all the gear needed to run the "oneshotblog.com":http://oneshotblog.com/
software on your machine.  It then leads you through the nitty-gritty details of getting 
oneshotblog.com running on your machine and testing it.

Most of these instructions could be easily turned into an automated script, which may happen in
the future.  For now it is meant to teach you about the typically dirty details involved in setting
up a system for the first time.  It also tries to avoid various problems with different operating
systems, so let me know how it works for you.

h2. A Warning

Deploying server software is a notoriously nasty process, especially the first 10 or 20 times.
Most operating systems do their best to enforce completely arbitrary restrictions on your
file layouts and configurations, and every system has different arbitrary restrictions.

When you go through these instructions, make sure you stay awake and be ready to delve
into why a particular step might not work on your system.  There's a good chance you missed
something or that there's something just slightly different about your system that makes
the step not work.

For example, in the parts of this document where I setup "oneshotblog.com":http://oneshotblog.com/
I ran into a problem with "SpamBayes":http://spambayes.sourceforge.net/ dying because
it couldn't iterate a bsddb.  Problem is this works just fine in the exact same setup on a CentOS
machine and was only dying on a MacOSX machine that I later tested.  For whatever reason, the
exact same setup can't run SpamBayes on OSX, even though it can run with the stock Python 2.5 in
OSX.

To solve the problem I just had to show you how to disable SpamBayes in the oneshotblog.com code 
so you could test it.  That's just how deployment goes.  You get on a machine and start setting
things up and then 2/3 of the way through the configuration you find out that something doesn't
work.

Only choices are to work around the problem (like I did) or try to figure out why your machine
is different and fix it.


h2. Step 0: Setup A Workplace

You'll want a directory to do this in so that you don't screw up your machine.  Here's what I did:

<pre class="code">
$ mkdir deploy
$ cd deploy
$ export DEPLOY=$PWD
</pre>

That last bit is so you can refer to this deployment directory with $DEPLOY (which I'll
be using in the instructions from now on).


h2. Step 1: Get Python

Many operating systems have old versions of Python, and even though Lamson works with
2.6 or 2.5, you'll probably want to get 2.6 for your deployment.  If your OS has 2.6
available then go ahead and install it.

If it doesn't have the right Python version, then here's how you can install it from
source and use it as your default Python.  To do this, just punch in these commands:

<pre class="code">
$ mkdir source
$ cd source
$ wget http://www.python.org/ftp/python/2.6.2/Python-2.6.2.tgz
$ tar -xvf Python-2.6.2.tgz
$ cd Python-2.6.2
$ ./configure --prefix=$DEPLOY
$ make
$ make install
</pre>

After this you will have a bunch of new directories in $DEPLOY:

<pre class="code">
$ ls $DEPLOY
bin     include lib     share   source
</pre>

Finally, you just have to put this new bin directory into your $PATH:

<pre class="code">
$ export PATH=$DEPLOY/bin:$PATH
</pre>

... then you just try it out to make sure that you have the right one:

<pre class="code">
Python-2.6.2 $ which python
$DEPLOY/deploy/bin/python

Python-2.6.2 $ python 
Python 2.6.2 (r262:71600, Jun  8 2009, 00:44:56) 
[GCC 4.0.1 (Apple Inc. build 5490)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
</pre>

That's it, you'll now be able to use this Python when you need to run your Lamson server, and
setup a virtualenv (coming next) so that you're walled off from the rest of the system.

bq.  Operating system fanatics will scoff at putting the python install in this directory, so
if you want you can just install it to the default /usr/local on your system and deal with 
all the various clashes and conflicts you'll have, especially if you are on an MacOSX machine.

h2. Step 2: Install VirtualEnv

Now we need to create a "virtual environment" to install all your software.  To do this we'll
need easy_install installed to your $DEPLOY directory:

<pre class="code">
$ cd $DEPLOY/source
$ wget http://peak.telecommunity.com/dist/ez_setup.py
$ python ez_setup.py 
$ which easy_install
$DEPLOY/bin/easy_install
</pre>

As you can see, you now have a clean install of easy_install in your fresh $DEPLOY/bin directory
for you to use.  Now you need to install @virtualenv@:

<pre class="code">
$ easy_install --prefix $DEPLOY virtualenv
$ which virtualenv
$DEPLOY/bin/virtualenv
</pre>

bq. Make sure you use @--prefix $DEPLOY@ above or you'll install things into the default
system setup even though easy_install is clearly and obviously running from a Python in a
totally different location so easy_install should know that.


h2. Step 3: Create Your VirtualEnv

With that you are ready to setup your virtual environment which will house your
Lamson setup and fill it with the gear you need.

First up is getting your virtualenv created and activated:

<pre class="code">
$ cd $DEPLOY
$ virtualenv LAMSON
New python executable in LAMSON/bin/python
Installing setuptools............done.
$ cd LAMSON
$ . bin/activate
</pre>

That's pretty simple, and it tells you clearly that you are using the LAMSON
virtualenv.  It prepends that to your currently prompt, so your prompt may
look different.

After that we can use easy_install to install our packages to this 
LAMSON virtual env.  Keep in mind that these packages will be in $DEPLOY/LAMSON,
so they won't infect your regular $DEPLOY setup.

<pre class="code">
$ cd $DEPLOY/LAMSON
$ easy_install lamson
</pre>

After that, you have lamson installed and ready to go, and you can install
anything you want, but there is one catch:

bq. You *MUST* be in the $DEPLOY/LAMSON directory or easy_install barfs complaining
that the package is not there.

h2. Step 4: Making Sure It Works

All of this setup is pointless if you can't get back to it later, so exit your
terminal completely and start a new one so you can do this:

<pre class="code">
$ cd projects/lamson/deploy/
$ export DEPLOY=$PWD
$ export PATH=$DEPLOY/bin:$PATH
$ cd $DEPLOY/LAMSON
$ . bin/activate
(LAMSON) $ which python
$DEPLOY/deploy/LAMSON/bin/python
(LAMSON) $ which easy_install
$DEPLOY/deploy/LAMSON/bin/easy_install
(LAMSON) $ which lamson
$DEPLOY/deploy/LAMSON/bin/lamson
(LAMSON) $ cd $DEPLOY
(LAMSON) $ lamson help
</pre>

If you can do all that, then you know you've got the setup going, now you
just need a little shell script to kick this all into gear automatically:

<pre class="code prettyprint">
#!/bin/sh

export DEPLOY=$PWD
export PATH=$DEPLOY/bin:$PATH
cd $1
source bin/activate
cd $DEPLOY
</pre>

To use this script, you just do this:

<pre class="code">
$ cd projects/lamson/deploy
$ . activate LAMSON
</pre>

With that you have a fully ready to go setup that's not using your normal
system's Python at all, has Python 2.6 installed, a fully virtualenv, and
the start of your lamson setup.


h2. Step 5: Setting Up The OneShotBlog

Let's see if we can setup the OneShotBlog example from the Lamson source
the way it is on the "oneshotblog.com":http://oneshotblog.com site.  We'll
need a few more modules installed with easy_install:

<pre class="code">
$ cd $DEPLOY/LAMSON
$ easy_install markdown
$ easy_install mock
$ easy_install spambayes
</pre>


Let's grab the 0.9.3 source from "PyPI":http://pypi.python.org/pypi/lamson/0.9.3 so we can get at the OSB example source:

<pre class="code">
$ cd $DEPLOY/source
$ wget http://pypi.python.org/packages/source/l/lamson/lamson-0.9.3.tar.gz
$ tar -xzf lamson-0.9.3.tar.gz
$ cd lamson-0.9.3/examples/osb
</pre>

Now we hit a slight snag.  OSB is using "SpamBayes":http://spambayes.sourceforge.net/ to do
spam filtering, but you probably will have a broken setup and would need to configure a ton
of stuff to get it working.  For now we're, just going to cheat, since it looks like SpamBayes
has problems with trying to iterate the keys in a bsddb under *some* Python builds.  To avoid
the problem, we're just going to edit @app/handlers/comment.py@ to remove the line with @spam_filter@:

<pre class="code prettyprint">
@route("(user_id)-AT-(domain)-(post_name)-comment@(host)")
# DELETE THIS LINE IN app/handlers/comments.py
@spam_filter(SPAM['db'], SPAM['rc'], SPAM['queue'], next_state=SPAMMING)
def START(message, user_id=None, post_name=None, host=None, domain=None):
    comment.attach_headers(message, user_id, post_name, domain) 
    confirmation.send(relay, "comment", message, "mail/comment_confirm.msg", locals())
    return CONFIRMING
</pre>

The spam filtering does work, but SpamBayes is difficult to get working in such a small
test run.

You are now sitting in the OSB example code, so you can fire up the logger
server and run the unit tests to make sure everything is working:

<pre class="code">
$ mkdir logs
$ mkdir run
$ mkdir app/data/posts
$ lamson log
$ nosetests
</pre>


You should get two errors you can ignore for now:

<pre class="code">
..............
======================================================================
FAIL: handlers.comments_tests.test_spam_sent_by_unconfirmed_user
----------------------------------------------------------------------
Traceback (most recent call last):
    ...
-------------------- >> begin captured logging << --------------------
root: WARNING: Attempt to post to user 'spamtester@somehost.com' but user doesn't exist.
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: handlers.comments_tests.test_spam_sent_by_confirmed_user
----------------------------------------------------------------------
Traceback (most recent call last):
    ...
-------------------- >> begin captured stdout << ---------------------
run/posts count after dever 1
run/posts count after dever 2

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
root: WARNING: Attempt to post to user 'spamtester@somehost.com' but user doesn't exist.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 25 tests in 1.363s
</pre>

Those are just fine since you don't have PyEnchant installed and aren't
using the spam filtering.

h2. Step 6: Run OneShotBlog Example

Now you're running the logger server and have your unit tests going, and
hopefully you can fix anything that you run into by now.  All you need now
is to run the whole setup and try it out:

<pre class="code">
$ lamson start
$ lamson start -pid run/queue.pid -boot config.queue
$ lamson start -pid run/forward.pid -boot config.forward
</pre>

With all this gear running you should be able to look in the @logs/lamson.log@ 
and @logs/logger.log@ to see what's going on.  You'll see the following 
activity:

* Forwarding receiver taking mail that couldn't be delivered and forwarding it to the logger server.
* The queue receiver pulling messages off run/posts and either delivering them as comments or updating the index.
* The rest of lamson processing mail and doing its job of feeding these two or just sending emails.

bq. If you want to see the configuration for these two other servers look in @config/queue.py@ and
@config/forward.py@ or better yet, diff them against @config/boot.py@ to see what's really different.

You should also check to see that they are really running:

<pre class="code">
$ ps ax | grep lamson
29438   ??  S 0:05.78 python lamson log
29605   ??  S 0:00.63 python lamson start
29612   ??  S 0:00.19 python lamson start -pid run/queue.pid -boot config.queue
29617   ??  S 0:00.34 python lamson start -pid run/forward.pid -boot config.forward
</pre>

h2. Step 7: Playing With OneShotBlog

Now we get to play with it.  Lamson comes with a web server that you can run to do simple testing
so start up a second window/terminal and do this:

<pre class="code">
$ cd projects/lamson/deploy/
$ . activate LAMSON
(LAMSON) $ cd source/lamson-0.9.3/examples/osb/
(LAMSON) $ lamson web -basedir app/data
Starting server on 127.0.0.1:8888 out of directory 'app/data'
</pre>

Now hit "http://localhost:8888/":http://localhost:8888/ with your browser and see 
the junk left over from your test runs.  Most of those posts won't actually
exist, so let's make a fake one for now.

You can forget about this web server window for now, and go back to your 
LAMSON window to do this with mutt:

# mutt -F muttrc to get it going with a fake setup.
# Send an email to first.blog@oneshotblog.com  (m is the key).
# You'll get a confirmation back, reply to it.
# You'll get a welcome message, but this message isn't in the index yet.  You can go look at it directly though.
# Send *another* email, this time to my.new.post@oneshotblog.com.
# No confirmation this time, just a message saying it was completed.
# *Now* go look at the index (might take a few seconds, up to 10).
# Click on the post title and go look at it.
# Right click on the [send comment] link and copy the email address.
# Go back to mutt and send an email to that address, this will post a comment to that post.
# Reply to the comment confirmation email, this should be the only one you get.
# In about 10 seconds you'll see your comment show up.

With that you have fully tested out the OneShotBlog example.  All that remains would
be a full deployment in a for-real situation, which is what we'll do next.


h2. Step 8:  Running On Port 25 For Real

The only problem with testing out the OneShotBlog with your own email client
is that you need to trick your computer into thinking your localhost address
is also oneshotblog.com.  To do that, open your /etc/hosts file and make
whatever changes you need to have localhost be oneshotblog.com also.

You'll know you've got it right when you can point your browser at "oneshotblog.com":http://oneshotblog.com/
and see your @lamson web@ window display log messages showing you click around.

bq.  Remember to undo this or you might be annoyed later.

Next, you'll need to stop lamson and restart it to use port 25.  This will be a
problem if you have another server running on port 25, so make sure you turn 
that server off for now.

bq.  I hope you aren't doing this on a live site where that's a problem.

<pre class="code">
$ lamson stop
$ vim config/settings.py
</pre>

At this point you'll want to change the receiver_config to look like this in @config/settings.py@:

<pre class="code prettyprint">
receiver_config = {'host': 'localhost', 'port': 25}
</pre>

Then you'll want to restart lamson so that it drops privilege to your
user after grabbing that port.  Easiest way to find out what your
user id (uid) and group id (gid) are is to use Python:

<pre class="code prettyprint">
Python 2.6.2 (r262:71600, Jun  8 2009, 00:44:56) 
[GCC 4.0.1 (Apple Inc. build 5490)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.getuid()
500
>>> os.getgid()
500
>>> ^D
</pre>

This shows that I'm uid 500 and gid 500, so now I can start my server:

<pre class="code">
$ sudo lamson start -uid 500 -gid 500
$ sudo chown -R zedshaw ./
</pre>

That last command just makes sure that lamson didn't accidentally change
the permission of a stray file or queue to @root@.

With that you should be running your lamson server as your user rather
than as root, but still bound to port 25.  Here's how to check:

<pre class="code">
$ ps aux | grep lamson | grep root
</pre>

If you don't see anything printed out, then you're safe.  If you see something
running as root, then you've got some work to do.


h2. Conclusion

I'll leave it to you to actually get your mail client to talk to this OneShotBlog
and working.  If you have problems, look at all the files in @logs/@ and also
use the @lamson queue@ command to inspect the different queues in the @run/@ directory.

At this point, you should know enough about setting up a lamson server
and configuring a real application (warts and all).  You should also
have learned how to get a clean Python installation that you can 
use no matter what your native OS does to Python, even if it's
retarded and renames python to Python for no apparent reason.

This document is very fresh, so send me feedback on your experience
with running through it.  Make sure you tell me what system you are 
on and that you ran each command exactly when you do.


