Title: Python Milters

<!-- This is part of the Python Milter Applications documentation.
 Copyright (C) 2001,2002,2003,2004,2005-2011 Business Management Systems, Inc.
 See the file license.ht for copying conditions.  -->

<P ALIGN="CENTER"><A HREF="http://www.anybrowser.org/campaign/">
<IMG SRC="http://bmsi.com/art/brain1.gif"
ALT="Viewable With Any Browser" BORDER="0"></A>

<img src="http://bmsi.com/art/banner_4.gif" width="468" height="60" border="0"
	usemap="#banner_4" alt="Your vote?">
<map name="banner_4">
  <area shape="rect" coords="330,25,426,59"
  	href="http://www.schoolandstate.org/Knowledge/Drennan/SchoolWars.htm" alt="I Disagree">
  <area shape="rect" coords="234,28,304,57" href="http://www.honestEd.com/" alt="I Agree">
</map>
</P>

<h1 align=center>Sendmail/Postfix Milters in Python</h1>
<h4 align=center>by <a href="mailto:%75%72%6D%61%6E%65%40%6E%65%75%72%61l%61%63%63%65%73%73%2E%63%6F%6D">Jim Niemira</a>
  and <a href="mailto:%73%74%75%61%72%74%40%62%6D%73%69%2E%63%6F%6D">
  Stuart D. Gathman</a><br>
This web page is written by Stuart D. Gathman<br>and<br>sponsored by
<a href="http://www.bmsi.com">Business Management Systems, Inc.</a> <br>
(see <a href="license.html">LICENSE</a> for copying permissions for this
documentation)<br>
Last updated Aug 08, 2012</h4>
<table rules="none">
<tr><td>
<img src="Maxwells.gif" alt="Maxwell's Daemon: pymilter mascot" align="top">
Mascot by students of <a href="http://alphard.ethz.ch/hafner/lebl.htm">Christian Hafner</a>
</td>
<td>

See the <a href="faq.html">FAQ</a> | <a href="http://sourceforge.net/project/showfiles.php?group_id=139894">Download now</a> |
<a href="#support">Support</a> |
<a href="#overview">Overview</a> |
<a href="/python/dspam.html">pydspam</a> |
<a href="/libdspam/dspam.html">libdspam</a>
<p>
<a href="//www.sendmail.org/">Sendmail</a> introduced a
<a href="https://www.milter.org/developers/api/index"> new API</a> beginning with version 8.10 - libmilter.  Sendmail 8.12 officially released libmilter.
Version 8.12 seems to be more robust, and includes new privilege
separation features to enhance security.  Even better, sendmail 8.13
supports socket maps, which makes <a href="pysrs.html">pysrs</a> much more
efficient and secure.   
Sendmail 8.14 finally supports modifying 
MAIL FROM via the milter API, and a data callback allowing spam to
be rejected before beginning the DATA phase (even after accepting some
recipients).
<table><td>
<a href="//www.python.org">
<img src="python55.gif" align=left alt="A Python"></a>
Pymilter provides a milter module for <a href="//www.python.org">Python</a>
that implements a python interface to libmilter exploiting all its features.
</td></table>
<a href="http://www.postfix.org">
<img src="postfix-logo.jpg" align=left alt="A Postmark"> </a>
Now <a href="http://www.postfix.org">Postfix</a> also implements the
milter protocol, and you can program
<a href="http://www.postfix.org/MILTER_README.html">SMTP time filters</a>
for Postfix in Python.
</td></tr> </table>

<h3> What's New </h3>

<ul>
<li> milter 0.8.16 has dkim signing, and Authentication-Results header.
pymilter-0.9.7 has several improved diagnostics for milter programming errors.
<li> milter has dkim checking and logging in CVS.  Will use DKIM Pass 
for reputation tracking, and as an additional acceptable identity 
along with HELO, PTR, or SPF.
<li> pymilter-0.9.4 supports python-2.6
<li> pymilter-0.9.2 supports the negotiate, data, and unknown callbacks.
Protocol steps are automatically negotiated by the high-level Milter
package by annotating callback methods with @nocallback or @noreply.

<li> pymilter-0.9.1 supports CHGFROM, introduced with sendmail-8.14,
 and also supported by postfix-2.3.
</ul>

<h3><a name=support>Support</a></h3>

<ul>
<li> <a href="http://bmsi.com/mailman/listinfo/pymilter">pymilter mailing list</a>
<li> <a href="http://www.openspf.net/Forums">SPF forums and chat</a> for SPF questions
<li> IRC channel: <b>#dkim</b> on irc.perl.org for DKIM questions
<li> IRC channel: <b>#pymilter</b> on irc.freenode.net for pymilter questions
<p>
You may be required to register your user nickname (nick) and identify with
that nick. Otherwise, you may not be able to join or be heard on the IRC
channel. There is a page describing how to register your nick at
<a href="http://freenode.net">freenode.net</a>.
</ul>

<h3><a name=overview>Overview</a></h3>

To accomodate other open source projects using pymilter, this package has been
shedding modules which can be used by other packages.
<ul>
<li> The <a href="http://spidey2.bmsi.com/pymilter/">pymilter</a> package provides a robust toolkit for Python <a
href="#milter">milters</a> that wraps the C libmilter library.  There
is also a <a href="http://code.google.com/p/ppymilter/">
pure Python milter library</a> that implements the milter protocol in Python.
<li> The <a href="policy.html">milter</a> package provides the beginnings of a
general purpose mail filtering system written in Python.  It also includes
a simple spfmilter that supports policy by domain and spf result via
the sendmail access file.
<li> The <a href="pysrs.html">pysrs</a> package provides an <code>SRS</code>
library, <code>SES</code> library, a sendmail socketmap daemon implementing
SRS, and (Real Soon Now) an srsmilter daemon implementing SRS,
now that sendmail-8.14 supports CHGFROM and this is supported in pymilter-0.9.
<li> The <a href="http://cheeseshop.python.org/pypi/pyspf">pyspf</a> package
provides the <code>spf</code> module, a well tested implementation of the of
the <a href="http://openspf.net">SPF</a> protocol, which is useful for
detecting email forgery. 
<li> The <a href="pygossip.html">pygossip</a> package provides the
<code>gossip</code> library and server daemon for the GOSSiP protocol, which
exchanges reputation of qualified domains.  (Qualified in the milter package
means that example.com:PASS tracks a different reputation than
example.com:NEUTRAL.)
<li> The <a href="http://pydns.sourceforge.net/">pydns</a> package provides the
low level <code>DNS</code> library for python DNS lookups.  It is much smaller
and lighter than the more capable (and bigger)
<a href="http://www.dnspython.org/"> dnspython</a> library.  Low level lookups
are needed to find SPF and MX records for instance.  
<li> The <a href="dspam.html">pydspam</a> package wraps an old version of
libdspam for python.  The C API changed dramatically for new versions, and I
haven't gotten things updated yet.  Another content filter might be in order.
</ul>
<p>
At the lowest level, the <code>milter</code> module provides a thin wrapper around the
<a href="https://www.milter.org/developers/api/index">
sendmail libmilter API</a>.  This API lets you register callbacks for 
a number of events in the
<a href="http://www.cs.concordia.ca/~group/fig/public/email/relay/milter+ruleset-checks.html">process of sendmail receiving a message via SMTP</a>.  
These events include the initial connection from a MTA,
the envelope sender and recipients, the top level mail headers, and
the message body.  There are options to mangle all of these components
of the message as it passes through the milter.
<p>
At the next level, the <code>Milter</code> module (note the case difference)
provides a Python friendly object oriented wrapper for the low level API.  To
use the Milter module, an application registers a 'factory' to create an object
for each connection from a MTA to sendmail.  These connection objects
must provide methods corresponding to the libmilter callback events.
<p>
Each event method returns a code to tell sendmail whether to proceed
with processing the message.  This is a big advantage of milters over
other mail filtering systems.  Unwanted mail can be stopped in its
tracks at the earliest possible point.
<p>
The <code>Milter.Milter</code> class provides default implementations for event
methods that
do nothing, and also provides wrappers for the libmilter methods to mutate
the message.
<p>
The <code>mime</code> module provides a wrapper for the Python email package that
fixes some bugs, and simplifies modifying selected parts of a MIME message.
<p>
Finally, the bms.py application is both a sample of how to use the
Milter and spf modules, and the beginnings of a general purpose SPAM filtering,
wiretapping, SPF checking, and Win32 virus protecting milter.  It can
make use of the <a href="pysrs.html">pysrs</a> package when available for
SRS/SES checking and the <a href="dspam.html">pydspam</a> package for Bayesian
content filtering.  SPF checking
requires <a href="http://pydns.sourceforge.net/">
pydns</a>.  Configuration documentation is currently included as comments
in the <a href="milter.cfg">sample config file</a> for the bms.py milter.
See also the <a href="HOWTO">HOWTO</a> and <a href="logmsgs.html">
Milter Log Message Tags</a>.
<p>
Python milter is under GPL.  The authors can probably be convinced to
change this to LGPL if needed.

<h3>What is a <a name="milter">milter</a>?</h3>

Milters can run on the same machine as sendmail, or another machine.  The
milter can even run with a different operating system or processor than
sendmail.
Sendmail talks to the milter via a local or internet socket.
Sendmail keeps the
milter informed of events as it processes a mail connection.  At any 
point, the milter can cut the conversation short by telling sendmail
to ACCEPT, REJECT, or DISCARD the message.  After receiving a complete
message from sendmail, the milter can again REJECT or DISCARD it, but it
can also ACCEPT it with changes to the headers or body.

<h3> What can you do with a milter? </h3>

<menu>
<li> A milter can DISCARD or REJECT spam based based on algorithms scripted
in python rather than sendmail's cryptic "cf" language.
<li> A milter can alter or remove attachments from mail that are poisonous to
Windows.
<li> A milter can scan for viruses and clean them when detected.
<li> A milter scans outgoing as well as incoming mail.
<li> A milter can add and delete recipients to forward or secretly
copy mail.
<li> For more ideas, check the <a href="//www.milter.org">Milter Web Page</a>.
</menu>

<a href="https://www.milter.org/developers/api/index">
Documentation for the C API</a> is provided with sendmail.  
<a href="http://spidey2.bmsi.com/pymilter/">Documentation for
pymilter</a> is provided via Doxygen.  Miltermodule provides a thin python
wrapper for the C API.  Milter.py provides a simple OO wrapper on top of that.  
<p>
The Python milter package includes a sample milter that replaces dangerous 
attachments with a warning message, discards mail addressed to
MAILER-DAEMON, and demonstrates several SPAM abatement strategies.  
The MimeMessage class to do this used to be based on the
<code>mimetools</code> and <code>multifile</code> standard python packages.  
As of milter version 0.6.0, it is based on the email standard
python packages, which were derived from the 
<a href="http://sourceforge.net/projects/mimelib">mimelib</a> project.
The MimeMessage class patches several bugs in the email package,
and provides some backward compatibility.

<p>
The "defang" function of the sample milter was inspired by
<a href="http://www.roaringpenguin.com/mimedefang/">MIMEDefang</a>,
a Perl milter with flexible attachment processing options.  The latest
version of MIMEDefang uses an apache style process pool to avoid reloading
the Perl interpreter for each message.  This makes it fast enough for
production without using Perl threading.
<p>
<a href="http://sourceforge.net/projects/mailchecker">mailchecker</a> is
a Python project to provide flexible attachment processing for mail.  I
will be looking at plugging mailchecker into a milter.
<p>
<a href="http://software.libertine.org/tmda/">TMDA</a> is a Python project
to require confirmation the first time someone tries to send to your
mailbox.  This would be a nice feature to have in a milter.
<p>
There is also a <a href="http://www.milter.org/">Milter community website</a>
where milter software and gory details of the API are discussed.

<h3> Is a milter written in python efficient? </h3>

The python milter process is multi-threaded and startup cost is incurred
only once.  This is much more efficient than some implementations that
start a new interpreter for each connection.  Testing in a production
environment did not use a significant percentage of the CPU.  Furthermore,
python is easily extended in C for any step requiring expensive CPU
processing.
<p>
For example, the HTML parsing feature to remove scripts from HTML attachments
is rather CPU intensive in pure python.  Using the C replacement for sgmllib
greatly speeds things up.

<h3> Goals </h3>

<menu>
<li> Implement RRS - a backdoor for non-SRS forwarders.  User lists non-SRS 
     forwarder accounts (perhaps in <code>~/.forwarders</code>), and a util
     provides a special local alias for the user to give to the forwarder.
     Alias only works for mail from that forwarder.  Milter gets forwarder
     domain from alias and uses it to SPF check forwarder.  Requires
     milter to have read access to <code>~/.forwarders</code> or else
     a way for user to submit entries to milter database.
<li> The bms.py milter has too many features.  Create a framework where
     numerous small feature modules can be plugged together in the
     configuration.
<li> Create a pure python substitute for miltermodule and libmilter that
     implements the <a
href="http://www.duh.org/cvsweb.cgi/~checkout~/pmilter/doc/milter-protocol.txt?rev=1">
     libmilter protocol</a> in python.  Someone has already 
     <a href="http://code.google.com/p/ppymilter/">started this</a>
     at Google code.
<li> Find or write a faster implementation of sgmllib.  The 
     <a href="http://www.effbot.org/zone/sgmlop-index.htm">sgmlop package</a>
     is not very compatible with 
     <a href="http://www.python.org/doc/2.1.3/lib/module-sgmllib.html">
     Python-2.1 sgmllib</a>, but it is a start, and is supported in
     milter-0.4.5 or later.
<li> Implement all or most of the features of 
     <a href="http://www.roaringpenguin.com/mimedefang/">MIMEDefang</a>.
<li> Follow the official <a href="http://www.python.org/peps/pep-0008.html">
     Python coding standards</a> more closely.
<li> Make unit test code more like other python modules.
</menu>

<h3> Confirmed Installations </h3>

Please <a href="mailto:%73%74%75%61%72%74%40%62%6D%73%69%2E%63%6F%6D">email</a>
me if you do <i>not</i> successfully install milter.  The confirmed
installations are too numerous to list at this point.

<h2> Enough Already! </h2>

Nearly a dozen people have emailed me begging for a feature to copy
outgoing and/or incoming mail to a backup directory by user.  Ok, it
looks like this is a most requested feature.  In the meantime,
here are some things to consider:
<ul>
<li> The milter package (bms.py) supports the mail_archive option
in the <code>[wiretap]</code> section.  This is not by user, however.
<li> If you want to equivalent of a Bcc added to each message, this
is very easy to do in the python code for bms.py.   See below.
<li> If you want to copy to a file in a directory (thus avoiding having to
set up aliases), this is slightly more involved.  The bms.py milter already
copies the message to a temporary file for use in replacing the message body
when banned attachments are found.  You have to open a file, and copy the
Mesage object to it in eom().
<li> Finally, you are probably aware that most email clients already
keep a copy of outgoing mail?  Presumably there is a good reason for
keeping another copy on the server.
</ul>
<p>
To Bcc a message, call <code>self.add_recipient(rcpt)</code> in envfrom after
determining whether you want to copy (e.g. whether the sender is local).  For
example,
<pre>
  def envfrom(...
    ...
    if len(t) == 2:
      self.rejectvirus = t[1] in reject_virus_from
      if t[0] in wiretap_users.get(t[1],()):
	self.add_recipient(wiretap_dest)
      if t[1] == 'mydomain.com':
        self.add_recipient('&lt;copy-%s&gt;' % t[0])
      ...
</pre>
<p>
To make this a generic feature requires thinking about how the configuration
would look.  Feel free to make specific suggestions about config file
entries.  Be sure to handle both Bcc and file copies, and designating what
mail should be copied.  How should "outgoing" be defined?  Implementing it is
easy once the configuration is designed.

<hr>
<p>
<a href="http://validator.w3.org/check/referer">
<img border=0 src="http://bmsi.com/vh32.png" alt=" [ Valid HTML 3.2! ] " height=31 width=88></a>
<a href="http://www.redhat.com">
<img src="http://bmsi.com/art/powered_by.gif" width="88" height="31" alt=" [ Powered By Red Hat Linux ] " border="0"></a>
</p>
