## The EDDIE-Tool User's Manual

Contents:

- Introduction
- Installation
  - Downloading
  - Installing
- Command-Line Options
- Configuration
  - Config Files
  - Global Configurables
  - Configuration Format
  - Simple Configuration
  - Directives
    - Common Directive Arguments
    - Rule Format
    - Built-in Directives
    - Directive Details
  - Built-in Directives
  - Actions
  - Notification and Message objects
- Other Features
  - Console

### Introduction

The EDDIE-Tool (commonly just called EDDIE) is an agent for system, network and security monitoring. It is highly customizable and easily extendable. It has been designed to be as platform-independent as possible, with platform-specific code limited to a small group of modules, making it easily portable to new platforms. It is fully written in Python and the configuration has a Python "look-and-feel" to it, although no Python or coding skills are necessary to configure it.

This user's manual is specific for EDDIE-Tool versions 0.29 and above, as some significant changes were made to improve the configuration. These changes can be read here. The user's manual for earlier versions can be read here.

### Installation

#### Downloading

You need to download the following:

- Required:
  - EDDIE - http://eddie-tool.net/download/
  - Python 2.2.1+ - http://www.python.org/ (must support threads) - Python 2.3+ is recommended, but Eddie has been tested with Python versions 2.2.1 through to 2.4.x. On Windows Python 2.3 or newer is required.
- Optional:
  - Elvin 4 - http://elvin.dstc.edu.au/ - Elvin is the message system supported by EDDIE and some of its related tools to pass alerts, collected data, etc between them. It is not required for basic monitoring.
  - elvinrrd - http://www.psychofx.com/elvinrrd/ - the daemon for storing EDDIE-collected data into RRD databases.
  - estored - available soon - the daemon for storing EDDIE-collected data into databases.

#### Installing

Follow the QUICKSTART document (also located in the eddie/doc/ directory) or continue with the steps below.

- Python - first install Python by following the instructions included in the Python distribution. EDDIE is written in Python and will not work if Python is not available. Python must have been compiled with thread support as EDDIE is fully threaded. EDDIE has been tested with Python versions 2.2.1 through to 2.4.x under Linux, Solaris and HP-UX, OS X, FreeBSD, OpenBSD and Windows.
- EDDIE - un-tar EDDIE into your favorite directory, eg: /opt $ cd /opt $ gtar xvzf eddie-0.xx.tgz Edit the main EDDIE file, eddie.py in the bin directory $ vi eddie-0.xx/bin/eddie.py and change the first line to point to your Python interpreter, eg: #!/opt/python/bin/python Note: DO NOT remove the special '#!' characters.
- That should be all that is needed to get EDDIE up & running. You can test that EDDIE will work by running it, eg: $ /installdir/eddie-0.xx/bin/eddie.py You should see the version of EDDIE and your system type, eg: EDDIE v0.30
  systype: Linux/2.2.5-15/i586 You can see what systems are supported by EDDIE by looking in the eddie/lib/ directory. Besides common, the other directories will be specific to different operating systems (e.g., 'Linux', 'SunOS'). If your system is not supported yet you will have to wait for a port of EDDIE to your system or contact the developers to see about porting it yourself.
- If you see the above output with no major error messages then EDDIE is working. However, without any configuration it will not be doing anything. Now the hard^H^H^H^Hfun part, building the configuration. (Press [CTRL]-C to stop it.)

### Command-Line Options

- -h, --help: display the help text, then exit
- --version: display the EDDIE-tool version, then exit
- -c FILE, --config=FILE: specify the path to the configuration file
  The default is (RUNDIR)/../config/eddie.cf
- --showconfig: display the configuration, then exit
- -v, --verbose: display extra messages
- -d, --daemon: run in the background, and return the PID of the daemon process

### Configuration

#### Config files

- The EDDIE config files should be in /installdir/eddie-0.xx/config. EDDIE begins by looking for an eddie.cf file in this directory, ie: /installdir/eddie-0.xx/config/eddie.cf
- EDDIE is distributed with a config.sample directory containing example configuration files. You should look over these to get an idea of how the configuration works.
- You must first create a config directory, if one isn't there already, eg: $ mkdir /installdir/eddie-0.xx/config then create an eddie.cf file in this new directory. You can copy and change the config.sample/eddie.cf file if you like.
- It is common for the EDDIE Directives (the rules which EDDIE uses to collect data and perform actions) to be written in separate rules files. These are then INCLUDEd in the main eddie.cf config file. The rules files can be in the same directory as eddie.cf or can be in subdirectories under it. It is common for a complex setup to split the rules into multiple files grouped by commonality.

#### Global Configurables

The global configurables are usually in eddie.cf and are listed below:

- LOGFILE - the file to log to (this should be the first thing set so that any problems are logged to the right place).
- LOGLEVEL - how much detail to log to the above logfile.
- ADMIN - the administrator's email address.
- ADMINLEVEL - how much log detail to send to the admin.
- ADMIN_NOTIFY - how often to send admin log emails.
- NUMTHREADS - the maximum number of threads (including supporting threads and directive threads) EDDIE will attempt to limit itself to using. This should be a minimum of 5.
- SCANPERIOD - the default time to wait between checks; can be overridden by each directive. See Time Definition for definition of time format.
- CONSOLE_PORT - defines the tcp port EDDIE binds to to provide a console interface to the directive states; this defaults to 33343, and can be set to 0 to disable this feature (see Console).
- EMAIL_FROM - the From: address to be used by the email action. Will default to the user EDDIE is run as.
- EMAIL_REPLYTO - the Reply-To: address to be used by the email action. Defaults to empty string, "".
- SENDMAIL - the location of the sendmail binary which EDDIE uses to send all emails. Is usually '/usr/lib/sendmail' (Solaris) or '/usr/sbin/sendmail' (Redhat Linux). Defaults to '/usr/lib/sendmail'.
- SMTP_SERVERS - a comma-separated list of SMTP servers to use to make SMTP connections for email sending. Defaults to 'localhost'.
- ELVINURL - the URL of an Elvin server (if required).
- ELVINSCOPE - the scope of an Elvin server (if required).
- INTERPRETERS - commands classed as "interpreters" by process checking rules (see the PROC directive).
- WORKDIR - directory where Eddie can store temporary files. [Eddie 0.35+]
- RESCANCONFIGS - Flag indicating the desire to constantly scan (and reload) config file changes. Defaults to true.
- CLASS - a grouping of hosts which share the same directives.
- ALIAS - global aliases can be set here, if required.
- INCLUDE - include another configuration file; it is common to split the rules into different files in a large installation.

eddie.cf is well documented, so read through the file and modify the settings to suit your environment.

#### Configuration Format

The EDDIE configuration follows the standard Python code format. Where methods or child objects of an object are indicated by indenting them beneath the parent object definition, sub-objects or parameters of a directive object are similarly indicated by indenting them beneath the parent object definition. For example, a part of the configuration may look like:

        group testing:
            PING testping:
                host="10.0.0.1"
                numpings=10
                rule="not alive"
                action=email("chris", "%(host)s failed ping")

        FILE file1:
            file='/tmp/file1.tmp'
            scanperiod='2m'
            rule='not exists'
            action=ticker("%(file)s does not exist", timeout=1)
            act2ok=ticker("%(file)s now exists", timeout=1)

A config group called "testing" is defined, then the PING directive "testping" is configured inside this group because it is indented. Similarly, all testping's arguments are indented as they belong to the PING directive configuration. The second directive, FILE called "file1", is at the same indentation level as the group definition (i.e., not indented) and is therefore a global directive. Thus, all hosts using this example config would execute the FILE directive, but only those hosts in the "testing" group would execute the PING directive.
If you are used to Python coding this will be second nature to you. If you are not, it will not be hard to pick up.
The above example also introduces the format of directive definitions. Directives are the rules which do "something". More often than not, they will perform system or network checks of some sort. But they are very flexible and could be configured to do more than simple checks.
In any case, the format of directive definitions is:

                 DIRECTIVE name:
                     argument1=value1
                     [argument2=value2
                     ...]

where "DIRECTIVE" is the directive name, like PROC or FS, and "name" is the user-defined, unqie name of this directive object. The arguments customize the directive appropriately. Some arguments are directive-specific while others are common to all directives.
Example:

                 PROC test:
                    name='syslogd'
                    rule='not exists'
                    scanperiod='30s'
                    action=email("alert@my.domain","syslogd is not running")

This is an example definition of a PROC directive, called 'test'. It contains the PROC-specific argument, 'name'. 'rule', 'scanperiod' and 'action' are arguments which are common to all directives. Some arguments are optional while others are required, and errors will be raised if they are missing. In this example 'name' and 'rule' are required. 'scanperiod' and 'action' are optional.

#### Simple Configuration

An EDDIE configuration can be simple to get basic monitoring started quickly and made as complicated as required to perform advanced operations. A simple example rules file is shown below to monitor basic services on a host. This rules file, named simple.rules, would be placed in the same directory as eddie.cf and eddie.cf would contain the entry INCLUDE 'simple.rules' The file simple.rules contains

            # Process checks
            PROC syslogd:
                name='syslogd'
                rule='not exists'
                action=email('root', '%(name)s is not running on %(h)s')
            PROC inetd:
                name='inetd'
                rule='not exists'
                action=email('root', '%(name)s is not running on %(h)s')
            PROC sshd:
                name='sshd'
                rule='not exists'
                action=email('root', '%(name)s is not running on %(h)s')

            # Filesystem checks
            FS root:
                fs='/'
                rule='pctused > 90'
                action=email('root', '%(mountpt)s over 90%% on %(h)s')
            FS varlog:
                fs='/var/log'
                rule='pctused > 90'
                action=email('root', '%(mountpt)s over 90%% on %(h)s')

            # Service Port checks
            SP smtp_port:
                port='smtp'
                protocol='tcp'
                bindaddr='0.0.0.0'
                rule='not exists'
                action=email('root', '%(protocol)s/%(port)s on %(h)s is not listening')
            SP http_port:
                port='http'
                protocol='tcp'
                bindaddr='0.0.0.0'
                rule='not exists'
                action=email('root', '%(protocol)s/%(port)s on %(h)s is not listening')

            # System statistics checks
            SYS loadaverage:
                rule="loadavg1 > 3.00"
                scanperiod='1m'
                action=email('root', '%(h)s load-average > 3.00')

#### Directives

The directives are the configuration commands which tell EDDIE what to do. They are of the form:

        DIRECTIVE name:
            arg1=value1
            arg2=value2
            argn=valuen

Where "DIRECTIVE" is the name of the directive itself (see Built-in Directives); "name" is a user-defined name of the directive definition (the directive ID is usually constructed as "DIRECTIVE.name", e.g., "FS.root", and will appear in the logs, console, etc); "args" are arguments to define what the directive should do and how it should do it. Some arguments are common to all directives and others are specific to that type of directive.

##### Common Directive Arguments

- rule=<rule string> - a string defining a rule that will be evaluated (in a Python environment) using variables specific to the current directive. The rule should evaluate to 1 (true) or 0 (false). If the rule is true, the directive state will be set to "fail" and the action (if any) will be executed. If false, the state will be "ok" and the actelse (if any) will be executed. The rule argument may be optional to some directives if they provide a default rule.
- scanperiod=<time> - this changes how often the directive will be run (the default is the time period specified by SCANPERIOD in eddie.cf). See Time Definition for formats of <time>.
- numchecks=<integer> - the number of checks to perform before the state will be set to fail and actions called. The time between these "re-checks" is set by the checkwait argument.
- checkwait=<time> - the time between "re-checks" if the numchecks argument is greater than 1. See Time Definition for formats of <time>.
- action=<actionlist> - this is a list of actions (see Actions) to be performed if the directive enters the failed state. i.e., if the directive is performing a check and the check fails, these actions will be called.
- act2ok=<actionlist> - this is a list of actions (see Actions) to be performed when the directive state changes from failed to ok.
- actelse=<actionlist> - this is a list of actions (see Actions) to be performed when the directive state was ok and is still ok after running any checks.
- console=<display_string> - define what is output to EDDIE Console connections for the current directive. Set to None to hide this directive from the Console.
- checkdependson=<directive(s)> - don't perform the check if any of these directives are in a failed state. [Eddie 0.31+]
- actiondependson=<directive(s)> - skip action execution if any of these directives are in a failed state. [Eddie 0.31+]
- actionperiod=<time expression> - an expresion which defines how long to wait before calling the next consequetive action. The variable 't' is the current time between action calls, and defaults to the scanperiod after the first action call. The scan period is also available in the expression with the scanperiod variable. Example: actionperiod='t * 2' - double the time between consequetive action calls. [Eddie 0.31+]
- history=<integer> - request directive to remember this many previous data samples. Access them in rules using the terminology 'history[n].dataname', where n is how many samples ago, and dataname is the name of the data to retrieve. Example, alert if a filesystem grows by over 5% between checks:

              FS export00_grow:
                  fs='/export/00'
                  scanperiod='1m'
                  history = 1
                  rule='(pctused - history[1].pctused) > 5'
                  action=email('root','%(mountpt)s grew to %(pctused)s')

  Example 2, alert if average filesystem growth over last 3 sample periods is too high:

              FS export01_avg_grow:
                  fs='/export/01'
                  scanperiod='1m'
                  history = 3
                  rule='(pctused + history[1].pctused + history[2].pctused + history[3].pctused) / 3 > 10'
                  action=email('root','%(mountpt)s grew to %(pctused)s')

  Note that at startup, rules will not run until enough history data is available. So, in the previous example, the directive would wait for three scanperiods before it had enough history data (three sample periods) to be able to evaluate the rule. [Eddie 0.31+]
- excludehosts=<hostlist> - do not execute this directive on any of the specified hosts. Hosts are specified as a string of comma-separated hostnames. [Eddie 0.32+]
- actionmaxcalls=<integer> - define the maximum number of times actions will be called for a particular failure. [Eddie 0.32+]
- disabled=<integer> - set to 1 to force a directive to be disabled. If disabled, a directive still exists in the configuration, but no checks will be executed for it. [Eddie 0.33+]
- checktime=<expression> - if specified, the directive will only be executed if the expression evaluates to true. The expression can make use of the current time and day with the variables: day ('mon', 'tue', etc); time (HHMM); hour (0-23); minute (0-59); second (0-59). And the for shorthands, the fixed lists: weekdays ('mon' - 'fri'), weekend ('sat', 'sun').
  Some Examples:

                  checktime='day=="mon" or day=="tue"'
                  checktime='day in weekdays and hour > 18'

  [Eddie 0.33+]

##### Rule Format

The format of rules is very simple. For those familiar with Python, rules are simply Python expressions which are evaluated on-the-fly with the variables set by the directive at the time. For those unfamiliar with Python, the expressions are almost English-like using operators such as: not, and, or; and mathematical operators such as: ==, !=, >, <, >=, <=. Use these operators to evaluate the variables you are interested in. The whole expression should evaluate to 1 (i.e., true) if the actions set by the action argument should be executed. If it evaluates to 0 (i.e., false) only the actions set by the actelse argument are executed.

As rule expressions are evaluated in a Python environment, links to related Python documentation is provided below.

- Boolean Operations
- Comparisons
- String Methods

  ##### Built-in Directives

  The built-in directives are grouped roughly into categories and are as follows:

  System Monitoring:

  - COM - define custom rules.
  - FS - perform filesystem checks.
  - IF - perform checks on local network interfaces.
  - METASTAT - some tests for Solaris Disksuite.
  - NET - define a network statistics rule.
  - PID - perform checks on PID-files.
  - PROC - perform checks on active processes.
  - SP - define a local service port checking rule.
  - DBI - execute a database query.
  - STORE - define a data storage rule.
  - SYS - perform system statistics checks.
  - DISK - perform disk I/O statistics checks.
  - TAPE - perform tape I/O statistics checks.
  Network Monitoring:
  - HTTP - perform remote HTTP/HTTPS tests.
  - PING - network host ping testing.
  - POP3TIMING - time & check POP3 connections.
  - PORT - remote TCP port checking rule.
  - RADIUS - perform radius authentication tests.
  - SMTP - test and measure SMTP connections to servers.
  - SNMP - retrieve and test SNMP data from remote hosts and devices.
  Security Monitoring:
  - FILE - for testing file stats or changes.
  - LOGSCAN - define a log scanning rule.
  Note that there may be many more directives depending on the version of EDDIE or any new or optional directives which may have been added to the distribution. See the EDDIE-Tool developer's guide for more information on creating new directives.

  ##### Directive Details

  COM
  COM is a generic directive used to perform custom checks that other directives are not available for. It simply executes the given command in a sub-shell, and captures the stdout/stderr and return value for testing by the directive rule.
  Security note: if EDDIE is run as root, the config files should not be world-writable as obviously directives like COM can execute any commands on the system.
  COM-specific Arguments:

  - cmd=<command> (required):
    - specifies the command to be executed in a sub-shell.
    Example:

                  cmd="/bin/ls /tmp/*.tmp | wc -l"

  Rule Variables:
  - out (string)
    - the out variable contains the standard output (stdout) of the executed command string.
  - outfieldsn (int)
    - Number of output fields
  - outfieldn (auto-typed)
    - the standard output is also split (by whitespace) and stored in variables outfield1, outfield2, etc, to simplify rule creation.
  - err (string)
    - the err variable contains the standard error (stderr) of the executed command string.
  - ret (int)
    - the ret variable contains the return code of the executed command string.
  - Examples:

                  rule='out == "test"'          # true if stdout is just "test"
                  rule='out.find("test")'       # true if stdout contains "test"
                  rule='int(out) > 5'           # true if out (converted to an integer) is > 5
                  rule='int(ret) != 0'          # true if return value of the cmd is not 0
                  rule='int(outfield1) != 0'    # true if stdout field 3 is not 0

  Action Variables:
  - cmd (string)
    - the command string as specified by the cmd argument.
  - Plus all the rule variables as usual.
  - Examples:

                  action=email("alert", "the command '%(cmd)s' failed.")

  Directive Examples:

              # Check load average (the hard way, without using SYS)
              COM loadavg:
                  cmd="uptime | cut -d, -f4 | awk '{print $3}'"
                  rule="float(out) > 6.0"
                  action=email("alert", "Load on %(h)s is > 6.0")

              # Check number of netscapes running
              COM count_ns:
                  cmd="ps -ef | grep netscape | wc -l"
                  rule="int(out) > 3.0"
                  action=email("alert", "There are %(out)s netscapes running on %(h)s")

              # A variation on checking load average, using 'outfield' variables
              COM loadavg:
                  cmd="uptime | cut -d, -f4"
                  rule="float(outfield3) > 6.0"
                  action=ticker("Load on %(h)s is %(outfield3)s", timeout=1)

  FILE
  This is a directive for performing checks on files or changes to files. Rules can be written based on any changes to the file metadata, like modification date, size, ownership, permissions, etc. It can also pick up changes to the file itself, which can be useful as a security check.
  FILE-specific Arguments:

  - file=<file name> (required):
    - the file to be checked.
  - keepdiff=true | false (optional):
    - if true Eddie will generate diffs of changed files. Previous copies are stored in the WORKDIR directory. [Eddie 0.35+]
  - difftype=context | unified | full (optional):
    - the type of diff to generate. [Eddie 0.35+]
  - context_lines=<integer> (optional):
    - how many context lines to show around the changed lines in a diff. [Eddie 0.35+]
  Rule Variables:
  - exists (boolean)
    - will be True if the file exists; and False if not.
  - missing (boolean)
    - will be True if an already existing file has disappeared. [Eddie 0.35+]
  - mode (integer)
    - the mode (permissions) of the file.
  - ino (long integer)
    - pointer to the inode of the file.
  - dev (long integer)
    - pointer to the device the file resides on.
  - nlink (integer)
    - the number of links this file has.
  - uid (integer)
    - the uid of the file.
  - gid (integer)
    - the gid of the file.
  - size (long integer)
    - the file size.
  - atime (long integer)
    - time of last access.
  - mtime (long integer)
    - time of last modification.
  - ctime (long integer)
    - time of last status change.
  - md5 (string)
    - md5 hex digest of file contents.
  - perm (integer) [0.31+]
    - the permissions bits only of the file mode.
  - sticky (integer) [0.31+]
    - the sticky bits only of the file mode.
  - type (integer) [0.31+]
    - the file type bits only of the file mode.
  - issocket (boolean) [0.31+]
    - true if the file is a socket.
  - issymlink (boolean) [0.31+]
    - true if the file is a symbolic link.
  - isfile (boolean) [0.31+]
    - true if the file is a standard file.
  - isblockdevice (boolean) [0.31+]
    - true if the file is a block device.
  - isdir (boolean) [0.31+]
    - true if the file is a directory.
  - ischardevice (boolean) [0.31+]
    - true if the file is a character device.
  - isfifo (boolean) [0.31+]
    - true if the file is a FIFO.
  - lastvar
    - the previous value of any above variable (e.g., lastmode).
  - now (long integer)
    - contains the current time, useful for comparison against ctime/mtime/atime.
  Action Variables:
  - file (string)
    - the value of the file argument.
  - diff (string)
    - the diff between changed file and previous copy - if keepdiff is true. [Eddie 0.35+]
  - Plus all the rule variables as usual.
  Directive Examples:

              # Alert when /etc/passwd changes
              FILE passwd_change:
                  file='/etc/passwd'
                  rule='mtime != lastmtime'
                  action=email('alert','%(file)s has been modified.')

              # Alert when 'ps' changes
              FILE ps_change:
                  file='/bin/ps'
                  rule='md5 != lastmd5'
                  action=email('alert','%(file)s has changed.')

              # Alert if file not owned by root
              FILE file_root:
                  file='/usr/local/bin/testfile'
                  rule='uid != 0'
                  action=email('alert','%(file)s uid is %(uid)s.')

              ## Simple test that cron is working
              ## crontab should have an entry like:
              ##   0,15,30,45 * * * * /bin/touch /var/run/eddie/cron.test
              FILE cron_test:
                  file='/var/run/eddie/cron.test'
                  rule='exists and mtime < (now-15*60)'  # file modified over 15 minutes ago
                  action=email("alert", "Cron test failed.", "%(file)s mtime=%(mtime)s now=%(now)s")

              # Make sure this file isn't a symlink
              FILE check_file:
                  file='/etc/passwd'
                  rule='issymlink'
                  action=email('alert','%(file)s should not be a symlink !!')

              # Alert if a file disappears
              FILE file_missing:
                  file='/etc/passwd'
                  rule='missing'
                  action=email('alert','%(file)s has disappeared')

  FS
  The FS directive is used to perform checks on local filesystems. Alerting when the filesystem is full would be the most common use for this directive.
  FS-specific Arguments:

  - fs=<filesystem mount point> (required):
    - specifies the filesystem to perform the check on. The value, a string, is the mount point of the filesystem (e.g., "/", "/var/log").
    Example:

                  fs='/var/log'

  Rule Variables:
  - size (int)
    - the size of the filesystem in kBytes.
  - used (int)
    - the amount of the filesystem used, in kBytes.
  - avail (int)
    - the amount of the filesystem available, in kBytes.
  - pctused (float)
    - the percentage of filesystem used.
  - fsname (string)
    - the filesystem (device) name.
  - mountpt (string)
    - the filesystem mount point.
  - Examples:

                  rule='pctused > 95.0'  # true if filesystem over 95% full
                  rule='avail > 350000'  # true if less than about 350MB available

  Action Variables:
  - df (string)
    - contains the stats of the current filesystem in a "df"-like format.
  - Plus all the rule variables as usual.
  - Examples:

                  action=email("alert", "%(mountpt)s at %(pctused)s%% - %(avail)d of %(size)d remain")

  Directive Examples:

              # alert if / over 95% full
              FS root:
                  fs='/'
                  rule='pctused > 95'
                  action=email("alert", "%(mountpt)s is over 95%% full on %(h)s")

              # alert if /var has less than 100MB available
              FS var:
                  fs='/var'
                  rule='avail < 100*1024'
                  action=email("alert", "%(mountpt)s has only %(avail)dkB free on %(h)s")

  HTTP
  This is a directive for performing remote HTTP and HTTPS tests against web servers.
  The elapsed connection time is recorded, and all related connection variables are made available, such as response code, headers and returned message body, as well as error information if the connection failed.
  SSL-support must be compiled into Python for HTTPS connections.
  The POST method is not yet supported.
  HTTP-specific Arguments:

  - url=<URL> (required):
    - the URL to fetch.
  - request_timeout=<seconds> (optional):
    - how many seconds to wait for a HTTP(S) response before timing out the connection and returning an error (variable 'timedout' will then be set to 1). Requires Python 2.3+ (socket settimeout() functionality). [Eddie 0.33+]
  - server=<hostname_or_ip> (optional):
    - specify the server to connect to; useful if the address of the server differs from the address specified in the URL. For example, making a connection to a server with a different address than that specified in the HTTP host header. [Eddie 0.36+]
  - persist_cookies=<True | False> (optional):
    - option to specify whether to persist server-defined cookies on the client side. If enabled, Eddie HTTP checks will send back any cookies defined by the server, doing its best to obey expire times. Disabled by default. [Eddie 0.36+]
  Rule Variables:
  - failed (boolean)
    - will be 1 (true) if the connection failed; and 0 (false) if not.
  - time (float)
    - the elapsed time of the connection (whether or not it failed).
  - time_resolve (float)
    - the elapsed time to perform a name resolution on the URL hostname. [Eddie 0.33+]
  - time_connect (float)
    - the elapsed time to connect to the HTTP(S) server. [Eddie 0.33+]
  - time_request (float)
    - the elapsed time to send the HTTP(S) request (ie: GET /) to the server. [Eddie 0.33+]
  - time_response (float)
    - the elapsed time to receive the HTTP(S) response from the server. [Eddie 0.33+]
  - timedout (boolean)
    - will be 1 (true) if the connection failed due to a socket timeout (timeout can be set with `request_timeout` argument); and 0 (false) if not. [Eddie 0.33+]

    If failed:

  - exception (string)
    - the exception type.
  - errno (integer)
    - error code of failure.
  - errstr (string)
    - error message.

    If not failed:

  - status (integer)
    - HTTP response status (e.g., 200, 404, etc)
  - reason (string)
    - HTTP response status message (e.g., "OK", "Not Found", etc)
  - ok (boolean)
    - true if status is 2xx or 3xx.
  - length (integer)
    - length of message body.
  - version (integer)
    - HTTP version used (10 = HTTP/1.0; 11 = HTTP/1.1)
  - header (string)
    - the response header.
  - body (string)
    - contains HTTP message body.
  Action Variables:
  - url (string)
    - the value of the url argument.
  - hostname (string)
    - the hostname part of the URL.
  - file (string)
    - the file part of the URL.
  - Plus all the rule variables as usual.
  Directive Examples:

              # Check our web site is up.
              HTTP website:
                  url='http://www.my.domain.name/index.html'
                  rule='failed'
                  action=email('alert', '%(url)s failed', 'exception: %(exception)s\nerrno: %(errno)s\nerrstr: %(errstr)s')

              # Check a certain page hasn't disappeared.
              HTTP mypage:
                  url='http://www.my.domain.name/~fred/fred.html'
                  rule='failed or not ok'
                  action=email('fred', '%(url)s failed')

              # Store web site response time in RRD db.
              HTTP web_time:
                  url='http://our.website.com/'
                  rule='not failed'
                  scanperiod='5m'
                  action=elvinrrd('http-%(h)s_%(hostname)s', 'time=%(time)f')
                  actelse=email('alert', 'Connection failed to %(url)s')

  IF
  The IF directive provides a mechanism for testing network interfaces. Interfaces listed in "netstat -i" are available for testing. The test can be simply whether the interfaces exists on a host or not; or it can be a more complex rule based on various statistics about that interface.
  IF-specific Arguments:

  - name=<interface name> (required):
    - the name of the interface (e.g., "eth0" or "hme0").
  Rule Variables:
  - exists (boolean)
    - this variable will be 1 (true) if the interface exists, 0 (false) if not.
  - Other variables available in this name-space will be every piece of data about the interface that EDDIE can find, like "rx_bytes", "rx_packets", etc. The types of data available will be OS-dependent. You should examine the netstat.py module in the system-specific lib directory to see what is available.
  - exists (boolean)
  - Example:

                  rule='not exists'   # true if interface doesn't exist

  Action Variables:
  - name (string)
    - the interface name as specified by the name argument.
  Directive Examples:

              # alert if eth0 interface has disappeared
              IF ethexists:
                      name='eth0'
                      rule='not exists'
                      action=email('alert', 'interface %(name)s has disappeared on %(h)s')

              # alert if input packet errors are greater than 10% (Solaris)
              IF ierrs:
                      name='hme0'
                      rule="100.0*ierrs/ipkts > 10.0"
                      action=email('alert', 'input packet error > 10%% on %(name)s')

  LOGSCAN
  The LOGSCAN directive provides a facility to watch files for important messages. Every line in the file can be matched, or for a busy system, selective lines can be picked out using a regular expression pattern. Commonly the resulting lines are emailed to an admin, but any standard EDDIE action could also be performed with the results.
  This directive works by initially finding the end of the file on its first 'scan' and storing this location. On the second and subsequent scans, the directive will scan all the new lines of the file that have been added since the previous scan, and finish by storing the new location of the end-of-file. If, however, the file has truncated in size (i.e., perhaps a log rotation has occured) the directive will scan all lines from the start of the truncated file.
  Note: it is possible that some lines may be missed in between scanning the file and the file being truncated (or log rotation) if the scanperiod is not short enough. It is recommended that the scanperiod be short if the file is updated frequently (i.e., for a busy logfile).
  LOGSCAN-specific arguments:

  - file=<filename> (required):
    - the filename of the file to be scanned.
  - regex=<search string> (optional):
    - a string to use to filter scanned lines. The string can be some text to match anywhere on the line, or a more complex regular expression. If all lines are required, regex='.*' should be used. regex='.*' is now the default (as of Eddie 0.37) if regex and regfile are not specified.
  - regfile=<filename string> (optional):
    - a filename containing multiple regular expressions (one per line) which will be used to perform log file line matching. [Eddie 0.35+]
  - negate=<true or false> (optional):
    - if set to 'true', the regex filter is used to exclude matching lines from the scan.
  - rule=<rule string> (optional):
    - the rule argument is optional for this directive, and defaults to "matchedcount > 0". i.e., the rule is true if 1 or more scanned lines match the regex. This can, of course, be overridden by supplying a custom rule argument.
  Rule Variables:
  - linecount (integer)
    - the number of lines scanned (not necessarily the number of lines returned if a regex was used).
  - matchedcount (integer)
    - this variable contains the number of lines that were returned by the scan (i.e. only those that passed the regex filter). [Eddie 0.35+]
  - unmatchedcount (integer)
    - this variable contains the number of lines which were filtered out by the regex (and, possibly, negate) rules. [Eddie 0.35+]
  - lines (string)
    - the actual lines of the file that matched the regex are stored in this variable. The number of lines returned should always equal matchedcount.
  - Examples:

                  rule='matchedcount > 0'  # true if 1 or more lines matched (default rule)
                  rule='lines != ""'          # true if some lines matched (same result as above)

  Action Variables:
  - file (string)
    - the value of the file argument.
  - regex (string)
    - the value of the regex argument.
  - negate (boolean)
    - the value of the negate argument.
  - Plus all the rule variables as usual.
  - Examples:

                  action=email("alert", "%(matchedcount)d lines matched scan, they are:\n%(lines)s")

  Directive Examples:

              # Email all entries from /var/log/messages to alert every 12 hours.
              LOGSCAN messages:
                  file='/var/log/messages'
                  regex='.*'
                  scanperiod='12h'
                  action=email("alert", "%(h)s:%(file)s", "-- Logscan matched %(matchedcount)d lines: --\n%(lines)s")

              # Email lines from /var/log/httpd/error_log, and ignore "notice" messages
              LOGSCAN httpd_error:
                  file='/var/log/httpd/error_log'
                  regex='.*[notice].*'
                  negate=true
                  action=email("alert", "%(h)s:%(file)s", "-- Logscan matched %(matchedcount)d lines: --\n%(lines)s")

  METASTAT
  This directive, part of the Solaris directives module, allows simple checks to be performed on Solaris Disksuite devices. Currently it only checks whether any metadevices require maintenance, but will be expanded in the future.
  METASTAT-specific Arguments:

  - only the standard rule argument is currently required.
  Rule Variables:
  - need_maintenance (boolean)
    - this variable is 1 (true) if a metadevice requires maintenance; 0 (false) if not.
  - Example:

                  rule='need_maintenance'   # true if a metadevice needs maintenance

  Action Variables:
  - Just the rule variables as usual.
  Directive Examples:

          # Check if any metadevice requires maintenance
          METASTAT maintenance:
              rule='need_maintenance'
              action=email('alert', 'A metadevice on %(h)s requires maintenance')

  NET
  The NET directive provides an interface to the kernel network statistics usually provided by a call to 'netstat -s'. Simple or complex rules can be written using these statistics.
  Linux Note: network stats counters are now collected from '/proc/net/snmp'. Try a 'cat /proc/net/snmp' to see what counters are available.
  NET-specific arguments:

  - (rule): the rule is the only required argument for this directive.
  Rule Variables:
  - The variables available in this name-space will be many of the network statistic provided by the kernel. i.e., see the results of a 'netstat -s' call (or 'cat /proc/net/snmp' on Linux). The actual variables available will be OS-dependent. You should examine the stats_ctrs data collector class in the netstat.py module in the system-specific lib directory to see what is available.
  - Examples:

                      rule='TcpInErrs > 0'   # true if TcpInErrs counter greater than 0 (Linux)

  Action Variables:
  - All the rule variables as usual.
  - Examples:

                  action=email("alert", "There have been %(TcpInErrs)d TCP Input Errors on %(h)s")

  Directive Examples:

              # alert if any UDP input errors (Solaris)
              IF udpinerr:
                      rule="udpInErrors > 0"
                      action=email('alert', '%(h)s has had %(udpInErrors)s UDP input errors')

  PID
  The PID directive is used to perform simple checks using pid files which some program generate. The most basic check is whether the pid file exists or not, which can often indicate whether the program is running or not; the second most basic check makes sure the pid found in the pid file also belongs to a process in the process table.
  PID-specific Arguments:

  - pidfile=<pidfile> (required):
    - specifies the pid file to perform checks on. The directive expects the pid to be the first number found in the file. It ignores anything beyond this. This is simply a filename as a string. (e.g., "/var/run/sshd.pid"). Example:

                  pidfile="/var/run/syslog.pid"

  Rule Variables:
  - exists (boolean)
    - this variable is 1 (true) if a process exists in the pidfile exists; and 0 (false) if it does not.
  - running (boolean)
    - this variable is 1 (true) if the pid found in the pidfile (if it exists) is also a valid process in the process table; and 0 (false) if not.
  - Examples:

                  rule='not exists'                   # true if pidfile not found
                  rule='exists and not running' # true if pidfile exists but pid not valid running process

  Action Variables:
  - pidfile (string)
    - the value of the pidfile argument.
  - Plus all the rule variables as usual.
  - Examples:

                  action=email("alert", "the pid-file %(pidfile)s does not exist")

  Directive Examples:

              # alert if the sshd pid file doesn't exist
              PID sshdpid1:
                  pidfile='/var/run/sshd.pid'
                  rule='not exists'
                  action=email("alert", "sshd pid file not found on %(h)s")

              # alert if the sshd pid doesn't match the process table
              PID sshdpid2:
                  pidfile='/var/run/sshd.pid'
                  rule='exists and not running'
                  action=email("alert", "sshd pid not a valid process on %(h)s")

  PING
  This directive provides a facility for checking the availability of hosts on a network. It allows ICMP ping checks to be performed and rules and actions can be written based on whether the remote host is alive, packet loss and round trip times.
  PING-specific arguments:

  - host=<hostname> (required):
    - the address of the host to ping.
  - numpings=<number> (optional):
    - the number of pings to send (default is 5 pings).
  Rule Variables:
  - alive (boolean)
    - this variable will be 1 (true) if the host responded to pings; and 0 (false) otherwise. Note: if at least one ping response is received, the host is considered to be 'alive'.
  - mintriptime (float)
    - the minimum round-trip-time (seconds).
  - avgtriptime (float)
    - the average round-trip-time (seconds).
  - maxtriptime (float)
    - the maximum round-trip-time (seconds).
  - numpktstx (integer)
    - the number of pings transmitted.
  - numpktsrx (integer)
    - the number of pings received.
  - pktloss (float)
    - the percentage packet-loss.
  - Examples:

                  rule='not alive'         # true if the host did not respond
                  rule='pktloss > 50.0'    # true if greater than 50% packet loss
                  rule='avgtriptime > 1.5' # true if avg RTT greater than 1.5 seconds

  Action Variables:
  - host (string)
    - the value of the host argument.
  - numpings (integer)
    - the value of the numpings argument.
  - Plus all the rule variables as usual.
  - Examples:

                  action=email("alert", "avg RTT from %(h)s to %(host)s is %(avgtriptime)f")

  Directive Examples:

              # Alert if host not responding
              PING foo:
                  host="foo.domain.name"
                  rule="not alive"
                  action=email('alert', 'host foo is not responding to pings')

              # Alert via ticker if there is any packet-loss.
              PING badpings:
                  host='10.0.0.5'
                  numpings=20
                  rule='pktloss >= 0.0'
                  scanperiod='1m'
                  action=ticker("%(host)s packetloss=%(pktloss)0.1f%% avgrtt=%(avgtriptime)f sec")

  POP3TIMING
  The POP3TIMING directive is used to measure the performance of a POP3 server. EDDIE connects to the given POP3 server/port and logs in as the given user, then performs some standard commands before closing the connection. The time taken for each step of the connection are timed and stored in variables to be used by the action(s).
  Besides timing information, this directive can also be used to perform basic checks on a POP3 server. A variable is set if the connection fails, so simple rules can be written to test this.
  POP3TIMING-specific Arguments:

  - server=<hostname:port> (required):
    - the address of the POP3 server and an optional port number, separated by a colon. If no port number is supplied, the default (110) is used.
  - user=<username> (required):
    - the username to use when logging in to the POP3 server.
  - password=<password> (required):
    - the password to use when authenticating the above user. Note: the password will be in plain text, so the config file should have appropriate permissions set if necessary.
  - Examples:

                  server='10.0.0.12'
                  server='pop3.my.domain:10110'
                  user='fred'
                  password='foo'

  Rule Variables:
  - connected (boolean)
    - this variable will be 1 (true) if a successful POP3 connection was made; 0 (false) otherwise.
  - connecttime (float)
    - the elapsed time (seconds) for the initial connection to be established.
  - authtime (float)
    - the elapsed time (seconds) for the authentication part (USER and PASS commands) to be performed.
  - listtime (float)
    - the elapsed time (seconds) for a LIST command to be performed.
  - retrtime (float)
    - the elapsed time (seconds) for a RETR command to be performed.
  - Examples:

                  rule='not connected'     # true if connection failed
                  rule='connecttime > 2.0' # true if time to connect over 2 seconds
                  rule='connecttime+authtime+listtime+retrtime > 5.0' # true if whole session took over 5 seconds

  Action Variables:
  - server (string)
    - the server part of the server argument (not the port).
  - port (integer)
    - the TCP port number of the POP3 server.
  - username (string)
    - the value of the user argument.
  - password (string)
    - the value of the password argument.
  - Plus all the rule variables as usual.
  - Examples:

                  action=email("alert", "connection to %(server)s took %(connecttime)f seconds")

  Directive Examples:

              POP3TIMING pop3test:
                  server='pop3.domain.com'
                  user='fred'
                  password='foo'
                  rule='connected'
                  action=email('mary', 'host=%(server)s, username=%(username)s, connecttime=%(connecttime)f, authtime=%(authtime)f, listtime=%(listtime)f, retrtime=%(retrtime)f')
                  actelse=email('alert', 'POP3 connection to %(server)s failed')

  PORT
  The PORT directive tests remote TCP based services. The simplest test is to determine whether the service is accepting remote connections on a given TCP port.
  The test can be made more complex by defining send and expect strings. The send string will be sent to the remote host after connecting, and any reply will be matched against the expect string (a regular expression). The check fails if the result does not match.
  PORT-specific arguments:

  - host=<remote host name or IP> (required):
    - specifies the remote host name or IP address to connect to.
  - port=<remote TCP port number> (required):
    - the remote TCP port number to connect to (an integer).
  - send=<string to send> (optional):
    - a string to be sent to the remote port after connection is opened.
  - expect=<string to expect> (optional):
    - a string to expect in response from the remote TCP connection. The string can match a substring of the response.
  - expectrexp=<regexp string to expect> (optional):
    - a regular expression string to expect in response from the remote TCP connection. See Python re module documentation for more information. [Eddie 0.35+]
  Rule Variables:
  - alive (boolean)
    - will be set to 1 (true) if a TCP connection can be made; or 0 (false) if not.
  - recv (string)
    - the response received from the remote TCP connection (after the 'send' string is sent if any).
  - connect_time (float)
    - the total elapsed time (in seconds) for the connection to be made, any data sent/received, and closed again.
  - matched (boolean)
    - if an 'expect' argument was given, this variable will be set to 1 (true) if it matched the data received from the connection, and 0 (false) otherwise.
  - error (int)
    - if the connection failed, this variable will contain the error code.
  - errorstr (string)
    - if the connection failed, the explanation of the error as a string is stored in errorstr.
  - Examples:

                  rule='not alive'   # true if the connection could not be opened
                  rule='not matched' # true if the received string did not match the expect string
                  rule='connect_time > 0.5' # true if connect time greater than 0.5 seconds

  Directive Examples:

              # check that 10.0.0.5 is accepting connections on port 80
              PORT webcheck:
                      host='10.0.0.5'
                      port=80
                      rule='not alive'
                      action=email('alert', 'port 80 not responding on 10.0.0.5')

              # check that a host is accepting connections on port 25
              PORT smtpcheck:
                      host='ahost.domain.name'
                      port=25
                      expect='220 '
                      rule='not alive or not matched'
                      action=email('alert', 'port 25 problem on 10.0.0.5')

  PROC
  The PROC directive is used to perform process checks. In the simplest case it is used to check if a process is not running when it should be (or running when it should not be). More complex rules can also be written, using most of the process statistics such as memory-usage, owner, percentage cpu used, running time, etc.
  PROC-specific Arguments:

  - name=<process name> (required):
    - specifies the name of the process to perform the check on. The value is a string and is defined as the name of the program running, not including any path information or arguments. For example, if a process is shown as "/usr/sbin/syslogd -r", the value of name should be "syslogd" only.
    An exception is programs executed by some type of interpreter, like 'sh', 'perl', 'python', etc. Any program name defined in the global config INTERPRETERS is assumed to be an interpreter and the name in this case will be the first argument (excluding any path information). For example, a program shown in ps as "/usr/bin/perl /var/tmp/test -x" will match a name value of "test" provided that "perl" exists in the INTERPRETERS setting.
    Example:

                  name='syslogd'

  Rule Variables:
  - exists (boolean)
    - this variable is 1 (true) if a process exists in the process list with the same name as the name argument; or 0 (false) if not.
  - Other variables available in this name-space will be every piece of data about the process that EDDIE can find, like "uid", "ppid", "pcpu", "vsz", etc. The types of data available will be OS-dependent. You should examine the proc.py module in the system-specific lib directory to see what is available.
    The rule could be something like "vsz > 40000" which would be true if the memory footprint (vsz) of the program was over 40000 bytes.
  - Examples:

                  rule='not exists'   # true if process not running
                  rule='pcpu > 50.0'  # true if process using over 50% CPU

  Action Variables:
  - name (string)
    - the value of the name argument.
  - Plus all the rule variables as usual.
  - Examples:

                  action=email("alert", "the %(name)s process is not running")

  Directive Examples:

              # alert if cron is not running
              PROC cron:
                  name='cron'
                  rule='not exists'
                  action=email("alert", "cron is not running on %(h)s")

              # syslog has a memory leak - alert if using over 50MB
              PROC syslogmem:
                  name='syslogd'
                  rule='vsz > 50*1024'
                  action=email("alert", "%(name)s is using %(vsz)d kBytes")

  RADIUS
  The RADIUS directive provides a facility for performing radius authentication checks.
  RADIUS-specific Arguments:

  - server=<hostname:port> (required):
    - the address of the radius server, and optionally the UDP port number. If port is left out, the default (1645) is used.
  - secret=<secret> (required):
    - the server secret.
  - user=<username> (required):
    - the username to use for authentication.
  - password=<password> (required):
    - the password to use for authentication.
  Rule Variables:
  - passed (boolean)
    - this variables is 1 (true) if the radius authentication was accepted; and 0 (false) if not.
  - timing (float)
    - the elapsed time between sending the radius authentication request and receiving the response (whether or not it failed).
  Action Variables:
  - server (string)
    - the hostname of the radius server.
  - port (integer)
    - the port of the radius server.
  - secret (string)
    - the value of the secret argument.
  - username (string)
    - the value of the username argument.
  - password (string)
    - the value of the password argument.
  - Plus all the rule variables as usual.
  Directive Examples:

              RADIUS radtest:
                  server='radius.domain.name:1812'
                  secret='s3cr3t'
                  user='bob@domain.name'
                  password='b0bm@t3'
                  rule='not passed'
                  action='email("alert", "radius FAILED to %(host)s:%(port)d")'

  SP
  The SP directive is used to perform checks on listening service ports. These can be either TCP or UDP ports. The simplest use is to check if nothing is currently listening on the given port, protocol and bind address combination.
  SP-specific Arguments:

  - protocol=<protocol name> (required):
    - specifies the protocol type; currently only 'tcp' and 'udp' are supported here.
  - port=<port number or name> (required):
    - specifies the port number (or name as defined in /etc/services, which should be a string) to perform the check on.
  - bindaddr=<IP address> | any (required):
    - specifies the IP address this port should be bound to. Usually this is the wildcard address, '0.0.0.0'. Note, under Solaris '*' represents the wildcard address.
    New in 0.33+: if '*' is used it will be automatically converted to '0.0.0.0', same with the data collected from the system; so either format should be valid now.
    New in 0.35: if set to string "any" then the bind address will be ignored when matching. Only the port will be compared.
  Rule Variables:
  - exists (boolean)
    - this variable will be 1 (true) if the port/protocol/bindaddr combination exists in the netstat table and is in the "LISTENING" state; otherwise it will be 0 (false).
  - Example:

                  rule='not exists'   # true if port not listening for connections

  Action Variables:
  - port (string)
    - the port name or number as specified by the port argument.
  - protocol (string)
    - the protocol as specified by the protocol argument.
  - bindaddr (string)
    - the bind address as specified by the bindaddr argument.
  - Plus all the rule variables as usual.
  - Examples:

                  action=email("alert", "the port %(protocol)s/%(port)s bound to %(bindaddr)s is not listening on %(h)s")

  Directive Examples:

              # alert if nothing listening on http port
              SP http:
                  port='http'
                  protocol='tcp'
                  bindaddr='0.0.0.0'
                  rule='not exists'
                  action=email('alert', 'http port not bound to on %(h)s')

              # alert if nothing listening on tcp port 22 on 10.0.0.5
              SP sshport:
                  port=22
                  protocol='tcp'
                  bindaddr='10.0.0.5'
                  rule='not exists'
                  action=email('alert', '%(protocol)s port %(bindaddr)s:%(port)s not listening')

  DBI
  The DBI directive is used to perform database queries (typically SQL), and check the results.
  DBI-specific Arguments:

  - dbtype=<type> (required)
    - specifies the database type; use anything that Python supports, such as "pg" and "mysql".
    Note that you must have the associated <dbtype>db.py Python module installed.
  - host=<hostname>
    - specifies the database server.
  - port=<port#>
    - specifies the port that the database is listening on.
  - user=<user name> (required)
    - specifies the user to log in to the database as.
  - password=<passwd> (required)
    - specifies the user's database login password.
  - query=<SQL query>
    - specifies the single-row or multi-row query to execute.
  Rule Variables:
  - connected (boolean)
    - this variable will be true if the database could be connected to, and false otherwise.
  - results (int)
    - indicates the number of results (0 to #).
  - result1 .. resultN (auto-typed)
    - query result variables.
    If the query returns a single row, then the results are available in result1 .. resultN.
    If the query returns multiple rows, then the result1 .. resultN contains the first column of every result row.
  Action Variables:
  - errmsg (string)
    - the error message, if any, returned by the database connect or query.
  Directive Examples:

              # test that our postgresql server is alive and responding to requests properly
              DBI postgresql_check:
                  dbtype='pg'
                  host='localhost'
                  database='monitoring'
                  user='monitoring'
                  password='sshhh'
                  query='select * from monitoring'
                  rule='not connected or results != 1 or result1 != 42'
                  action=email(ALERT_EMAIL, 'PostgreSQL DB %(database)s failed test', 'Query: %(query)s\nConnected: %(connected)s\nError: %(errmsg)s')

              # alert if too many connections to the Postgres database
              DBI db_connections:
                  dbtype='pg'
                  host='localhost'
                  database='mydb'
                  user='pgsql'
                  password='sekrit'
                  query='select count(1) from pg_stat_activity'
                  rule='connected and results > 0 and result1 > 40'
                  action=email('alert', 'Database %(database)s on %(h)s: too many connections (currently %(result1)s)')
                  console='%(database)s on %(host)s : connections = %(result1)d'

  SMTP
  This directive makes a connection to an SMTP server and returns the elapsed response time.
  SMTP-specific Arguments:

  - host=<hostname:port> (required):
    - the remote host to poll. Specifying the port is optional, and defaults to 25.
  Rule Variables:
  - connected (boolean)
    - is True if the SMTP connection was successful; False if not.
  Action Variables:
  - host (string)
    - the hostname part of the host argument.
  - port (integer)
    - the port part of the host argument.
  - connecttime (float)
    - the elapsed time taken to connect to the server and retrieve a response (only if connection was successful).
  - rule (string)
    - the value of the rule argument.
  - Plus all the rule variables as usual.
  Directive Examples:

              SMTP smtp_test:
                  server='mail.mydomain.com'
                  rule='connected'
                  action=email('alert', "SMTP connection to %(server)s:%(port)s took %(connecttime)s secs")

  SNMP
  This directive provides an SNMP client to retrieve data from remote hosts and devices via the SNMP protocol. Multiple values can be retrieved in one call. Standard EDDIE rules can then perform tests on the retrieved data, or the data could be stored in RRD files using the elvinrrd action (for instance).
  SNMP-specific Arguments:

  - host=<hostname:port> (required):
    - the remote host to poll. Specifying the port is optional, and defaults to 161.
  - oid=<OIDs> (required):
    - one or more OIDs to fetch from host. Multiple OIDs should be separated by commas (,). Currently these must be the full OID in numeric format only.
  - community=<community string> (optional):
    - the community string of the remote SNMP server. Defaults to 'public' if not supplied.
  - maxretry=<integer> (optional):
    - the maximum number of retries before failing an SNMP connection. Defaults to 5.
  Rule Variables:
  - failed (boolean)
    - contains 1 (true) if the SNMP connection failed; 0 (false) if ok.
  - response (type dependent on type of OID)
    - contains the result of the OID, if only 1 OID was specified.
  - responsen (type dependent on type of OID)
    - contains the results of the OIDs, if 2 or more were specified. E.g., response1, response2, response3, etc, matching the specified OIDs in order.
  Action Variables:
  - host (string)
    - the hostname part of the host argument.
  - port (integer)
    - the port part of the host argument.
  - community (string)
    - the value of the community argument.
  - oid (string)
    - the value of the oid argument.
  - maxretry (integer)
    - the value of the maxretry argument.
  - rule (string)
    - the value of the rule argument.
  - Plus all the rule variables as usual.
  Directive Examples:

          # Fetch a counter from a device
          SNMP foo:
              host='alt1.domain.name'
              oid='1.3.6.1.4.1.1872.2.1.1.6.0'
              community='private'
              rule='response > 0'
              maxretry=10
              action=email('alert', 'Head for the lifeboats: %(snmpresponse)s')

          SNMP router_traffic:
              scanperiod='5m'
              host='10.0.0.1'
              oid='1.3.6.1.2.1.2.2.1.10.2, 1.3.6.1.2.1.2.2.1.16.2'
              community='special'
              rule='not failed'
              maxretry=10
              action=elvinrrd("net-router_BRI01", "ibytes=%(response1)s", "obytes=%(response2)s")

  STORE
  The STORE directive is still being developed and tested. It will be documented at a later date.

  SYS
  The SYS directive provides an interface to the kernel's system statistics. Simple or complex rules can be written using these statistics.
  SYS-specific arguments:

  - (rule): the rule is the only required argument for this directive.
  Rule Variables:
  - The variables available in this name-space will be many of the system statistics provided by the kernel. Many of these are OS-specific, but some common variables are 'loadavg1', 'loadavg5', and 'loadavg15'.
    See the Solaris-specific System Data section, or the system data collector class in the system.py module in the system-specific lib directory to see what is available.
  - Examples:

                      rule='loadavg1 > 2.0'   # true if 1-min load-average > 2.0

  Action Variables:
  - All the rule variables as usual.
  - Examples:

                 action=email("alert", "The loadavg on %(h)s is %(loadavg1)0.2f")

  Directive Examples:

              # alert if 1 minute load average > 2
              SYS loadavg1:
                      rule="loadavg1 > 2.0"
                      action=email('alert', '%(h)s has a loadavg1 of %(loadavg1)0.2f')

  DISK
  The DISK directive provides an interface to the kernel's disk I/O statistics. Simple or complex rules can be written using these statistics. This requires the data collector diskdevice:DiskStatistics (which is only available on Solaris and Win32 at time of writing).
  [Eddie 0.35+]
  Directive-specific arguments:

  - device: the disk device name (on Solaris this will be the name used by kstat, e.g. md20 or sd100)
  Rule Variables:
  - The variables available in this name-space will be many of the statistics provided by the kernel.
    On Solaris these are, at least: nread, nwritten, reads, writes. For more details look at the output of: `/usr/bin/kstat -p -c disk`
  Action Variables:
  - All the rule variables as usual.
  - Examples:

                 action=email("alert", "Disk %(device)s : rbytes=%(nread)s wbytes=%(nwritten)s")

  Directive Examples:

              # /dev/md/dsk/d20 == /var : send read/write counters to RRD
              DISK md20_thruput:
                  device='md20'
                  scanperiod='5m'
                  rule='True'        # always perform action
                  action='elvinrrd("disk-%(h)s_%(device)s", "rbytes=%(nread)s", "wbytes=%(nwritten)s")'

  TAPE
  The TAPE directive provides an interface to the kernel's tape I/O statistics. Simple or complex rules can be written using these statistics. This has almost exactly the same functionality as the DISK directive. This requires the data collector diskdevice:TapeStatistics (which is only available on Solaris at time of writing).
  [Eddie 0.35+]
  Directive-specific arguments:

  - device: the tape device name (on Solaris this will be the name used by kstat, e.g. st65)
  Rule Variables:
  - The variables available in this name-space will be many of the statistics provided by the kernel.
    On Solaris these are, at least: nread, nwritten, reads, writes. For more details look at the output of: `/usr/bin/kstat -p -c tape`
  Action Variables:
  - All the rule variables as usual.
  - Examples:

                 action=email("alert", "Tape %(device)s : rbytes=%(nread)s wbytes=%(nwritten)s")

  Directive Examples:

              # st65 == TAPE : send tape read/write counters to RRD
              TAPE st65_thruput:
                  device='st65'
                  scanperiod='5m'
                  rule='True'        # always perform action
                  action=elvinrrd("tape-%(h)s_%(device)s", "rbytes=%(nread)s", "wbytes=%(nwritten)s")

  #### Actions

  Actions are performed when rules
  - fail - execute specified action
  - change state from fail to success - execute specified act2ok
  - subsequent successes - execute specified actelse
  Actions currently include:

  - log - pass a message into a file, the eddie tty, or syslog
  - email - generate an email message
  - system - run a command
  - restart - run /etc/init.d/(daemon) start command
  - nice - change the priority of a process
  - eddielog - pass a message to the eddie log file
  - ticker - send a message to an Elvin listener as a ticker
  - page - send a message to a pager
  - elvindb - send a message to an Elvin listener
  - elvinrrd - send a message to a RRDtool listener via Elvin
  - netsaint - send a message to a NetSaint listener via Elvin

  ##### Action Details

  log
  log performs message logging to a file, the tty where eddie was executed, or to syslog. The where depends on the via, which is the second parameter. If via looks like "XXX.YYY", then it is assumed that syslog type logging is desired. If via begins with a "/", then it is assumed that logging to a file is desired. If via is the string "tty", then the message will go to the tty where eddie was executed. You may specify multiple vias by separating them with a ";", as in "FACILITY.LEVEL;/path/to/file1.txt;/path/to/file2.log".
  Format:

  - action=log(message, via)
    - message - the string to be logged
    - via - the method to use for logging
      - syslog: FACILITY.LEVEL - see man syslog(3) or syslog.h for a list of facilities and levels
      - file: /path/to/file - append message to the specified file
      - tty: "tty" - write the message to the eddie tty
  Action Examples:

              # generate a syslog notification using the LOG_DAEMON facility and LOG_ALERT level
              action=log("There is a problem on %(h)s", "DAEMON.ALERT")

              # append a message to a log file
              action=log("There is a problem on %(h)s", "/var/log/eddie_disk.log")

              # display a message on the tty that eddie was started on, and append to eddie.log
              action=log("There is a problem on %(h)s", "tty;/var/log/eddie.log")

  email
  email performs message emailing.
  How it goes about sending the email depends on your SENDMAIL and SMTP_SERVERS Eddie config options.
  Format:

  - action=email(address, subject, body)
    - address - the recipient list
    - subject - subject of the email
    - body - email body text
      Note that embedded "\n" strings will be converted to newlines.
  Action Examples:

              # generate an email alert
              action=email("me@mydomain.com,them@myotherdomain.com", "There is a problem on %(h)s", "Problem age: %(problemage)s")

  system
  system allows execution of operating system commands.
  Format:

  - action=system(command)
    - command - the command to execute, via Python's os.system() call.
  Action Examples:

              # run command to rotate the web log file
              action=system("rotate /var/log/web_log")

  restart
  Run /etc/init.d/(name) start command. Usually used to restart a dead daemon.
  Format:

  - action=restart(daemon)
    - daemon - the daemon to restart, which should have a file /etc/init.d/(daemon)
  Action Examples:

              # restart the httpd server
              action=restart("httpd")

  nice
  Change the "nice" value of a running process, either up or down. Note that in order to increase the nice level, eddie has to be running as super-user.
  The process acted upon is the current pid in the dictionary, so this action only works for PROC and PID directives.
  Format:

  - action=nice([relative, ] value)
    - relative - the direction to move the nice value, one of "+" or "-"
    - value - if relative was specified, then the magnitude of the change, otherwise the absolute "nice" level
  Action Examples:

              # change the execution of the process to take a little less time
              action=nice("+", 5)

              # de-prioritize the process
              action=nice(20)

  eddielog
  This action allows for logging messages to the log file that eddie is configured to use. Depending on the ADMINLEVEL setting, the message may also (eventually, depending on ADMIN_NOTIFY setting) get emailed to the ADMIN.
  Format:

  - action=eddielog(message [, loglevel])
    - message - the message to log
    - loglevel - "0" (informational) to "9" (critical). Defaults to "0"
  Action Examples:

              # generate an informational message to the eddie log file
              action=eddielog("Disk issue on %(h)s: used level is %(pctused)s%%")

              # generate a high-priority message to the eddie log file
              # (and probably to the ADMIN as well, eventually)
              action=eddielog("Disk issue on %(h)s: used level is %(pctused)s%%", 9)

  ticker
  Send a ticker-type message to an Elvin listener.
  Format:

  - action=ticker(message [, timeout])
    - message - the message to send
    - timeout - the timeout, in minutes. Defaults to 10 minutes
  Action Examples:

              # send a ticker-type message
              action=ticker("%(file)s does not exist", timeout=1)

  page
  Send a page to the specified recipients. Currently implemented as an email.
  Format:

  - action=page(pager, message)
    - pager - email address of the pager
    - message - message body to send
  Action Examples:

              # send a page to the ADMIN_PAGER alias
              action=pager(ADMIN_PAGER, "Host %(server)s is inaccessable")

              # send a page to a Sprint phone
              action=pager("734657XXXX@messaging.sprintpcs.com", "Host %(server)s is inaccessable")

  elvindb
  Send information to a database listener via Elvin. Data to insert in db can be specified in the data argument as 'col1=data1, col2=data2, col3=data3' or if data is not specified it will use values sent previously.
  Format:

  - action=elvindb(table [, data])
    - table - specifies the table name to store the data in on the elvindb server
    - data - comma-separated "name=value" pairs
  Action Examples:

              # send data to table "MYTABLE" via elvindb
              action=elvindb("MYTABLE", "host=%(h)s,load1=%(load1)s,load5=%(load5)s")

  elvinrrd
  Send information to a RRDtool database listener via Elvin.
  Format:

  - action=elvinrrd(key, data [, data ...])
    - key - the RRD name configured in the Elvin consumer to store the data
    - data - values to store in "name=value" pairs, where name is the RRD variable name and value is the valure to store there
  Action Examples:

              # send the one-minute load average every minute for this host
              SYS loadavg1_rrd:
                  rule='True'        # always true
                  scanperiod='1m'
                  action="elvinrrd('loadavg1-%(h)s', 'loadavg1=%(loadavg1)f')"

  netsaint
  Send information to a NetSaint listener via Elvin.
  Format:

  - action=netsaint(description, output, return_code)
    - description - short description
    - output - message output
    - return_code - return code to log
  Action Examples:

              # send the free memory size to the NetSaint consumer
              action=netsaint("EddieMem", "Free memory on %(h): %(memfree)s", 1)

  #### Notification and Message objects

  Notification objects define levels of actions to be performed. Usually, the higher the level, the more serious the actions will be. Later versions of EDDIE will use notification objects for advanced features like problem escalation.
  Message objects define messages to be used in actions like email or paging. They are grouped together to provide a common way to call them from notification objects.

  - N - define a Notification object, containing levels of actions.

                 N name:
                     Level 0:
                         action1, [action2, ...]
                     Level 1:
                         action1, [action2, ...]
                     Level n:
                         action1, [action2, ...]

    Levels should range from 0 to 9, with 9 being the most critical. E.g.:

                 N COMMONALERT:

                     # Info
                     Level 0:
                         email(INFO_EMAIL,INFO)

                     # Warning
                     Level 1:
                         email(ALERT_EMAIL,WARN)

                     # Alert
                     Level 2:
                         email(ALERT_EMAIL,ALERT),ticker(ALERT_P)

                     # Serious Alert
                     Level 3:
                         email(ALERT_EMAIL,ALERT),email(ONCALL_EMAIL,ALERT_P),ticker(ALERT_P)

    Actions are defined like function calls, and multiple actions are separated by commas. See Actions for more information.
  - M - define a message group. This simply allows you to group together messages and message subgroups.
  - MSG - define a message object. This allows you to define a message object. A message object has a name and one or more strings of text. The way the strings are used by actions depends on the action. For example, the email() action wants two strings, the first being the subject and the second being the body of the email.

                 M groupname:
                     MSG msgname1: "string1"
                         "string2"
                     MSG msgname2: "string3"
                         "string4"

    or

                 M groupname:
                     M subgroupname1:
                         MSG msgname1: "string1"
                             "string2"
                         MSG msgname2: "string3"
                             "string4"
                     M subgroupname2:
                         MSG msgname3: "string5"
                             "string6"

    E.g.:

                 # Define common messages.  These are used by the COMMONALERT notification object
                 M commonmsg:
                     # Define a subgroup of messages to be used by PROC directive actions
                     M proc:
                         # Warning-level message for email
                         MSG WARN: "Warning: %(name)s on %(h)s not running"
                             "The %(name)s process on %(h)s is not running"

                         # Warning-level message for paging or tickertape
                         MSG WARN_P: "Warn: The %(name)s daemon on %(h)s is not running." ""

                         # Alert-level message for email
                         MSG ALERT: "Alert: %(name)s on %(h)s not running"
                             """ALERT: The %(name)s daemon on %(h)s is not running.

                 %(problemage)s
                 %(problemfirstdetect)s
                 """

                         # Alert-level message for paging or tickertape
                         MSG ALERT_P: "ALERT: The %(name)s daemon on %(h)s is not running." ""

  ### Other Features

  #### Console

  EDDIE features a Console facility which provides live information about the active directives via a TCP connection. The TCP port used is set by the CONSOLE_PORT setting in eddie.cf and defaults to port 33343. Set this to 0 to disable this feature.

  By default every directive is shown in the Console output in the format "<ID> - <state>". This can be modified with the console directive argument, or the directive not shown at all by setting this argument to None.

  Substitution variables available to the console argument string are:

  - state - current state of directive (e.g., "ok", "fail")
  - lastchecktime - date/time of last directive execution
  - problemfirstdetect - date/time of current failure first detected (only if state is failed)
  - problemlastfail - date/time of current failure last detected (only if state is failed)
  - others - directive-specific variables normally available to actions

  Directive examples:

          # check root filesystem usage
          FS rootfs:    fs='/'
                        rule="pctused > 95"
                        action=email("root", "%(mountpt)s at %(pctused)s%%")
                        console='%(state)s %(pctused)s%%'

          # email me load average every 5mins
          SYS loadavg5: rule="True"
                        action=email('chris', '%(h)s loadavg5: %(sysloadavg5).02f')
                        scanperiod='5m'
                        console="loadavg5=%(sysloadavg5).02f"

          # store root filesystem data in RRD (don't show on Console)
          FS root_rrd:  fs='/'
                        rule="True"
                        scanperiod='5m'
                        action=elvinrrd("fs-%(h)s_root", "used=%(fsused)s", "size=%(fssize)s")
                        console=None

  Console example:

          $ telnet localhost 33343
          Trying 127.0.0.1...
          Connected to localhost.
          Escape character is '^]'.
          Eddie Console Gateway
          FS.rootfs - ok 33%
          SYS.loadavg5 - loadavg5=0.14
          Connection closed by foreign host.

  TODO: System-specific information...... (NOT FINISHED)

  Solaris:

        System stats from '/usr/bin/uptime':
            uptime          - time since last boot (string)
            users           - number of logged on users (int)
            loadavg1        - 1 minute load average (float)
            loadavg5        - 5 minute load average (float)
            loadavg15       - 15 minute load average (float)

        System counters from '/usr/bin/vmstat -s' (see vmstat(1M)):
            ctr_swap_ins                            - (long)
            ctr_swap_outs                           - (long)
            ctr_pages_swapped_in                    - (long)
            ctr_pages_swapped_out                   - (long)
            ctr_total_address_trans_faults_taken    - (long)
            ctr_page_ins                            - (long)
            ctr_page_outs                           - (long)
            ctr_pages_paged_in                      - (long)
            ctr_pages_paged_out                     - (long)
            ctr_total_reclaims                      - (long)
            ctr_reclaims_from_free_list             - (long)
            ctr_micro_hat_faults                    - (long)
            ctr_minor_as_faults                     - (long)
            ctr_major_faults                        - (long)
            ctr_copyonwrite_faults                  - (long)
            ctr_zero_fill_page_faults               - (long)
            ctr_pages_examined_by_the_clock_daemon  - (long)
            ctr_revolutions_of_the_clock_hand       - (long)
            ctr_pages_freed_by_the_clock_daemon     - (long)
            ctr_forks                               - (long)
            ctr_vforks                              - (long)
            ctr_execs                               - (long)
            ctr_cpu_context_switches                - (long)
            ctr_device_interrupts                   - (long)
            ctr_traps                               - (long)
            ctr_system_calls                        - (long)
            ctr_total_name_lookups                  - (long)
            ctr_toolong                             - (long)
            ctr_user_cpu                            - (long)
            ctr_system_cpu                          - (long)
            ctr_idle_cpu                            - (long)
            ctr_wait_cpu                            - (long)

        Process/memory stats from '/usr/bin/vmstat' (see vmstat(1M)):
            procs_running   - number of processes running (int)
            procs_blocked   - number of processes blocked (int)
            procs_waiting   - number of processes waiting (int)
            mem_swapfree    - amount of free swap (kB) (int)
            mem_free        - amount of free RAM (kB) (int)

  Linux:

        loadavg1              - 1min load average (float)
        loadavg5              - 5min load average (float)
        loadavg15             - 15min load average (float)
        ctr_uptime            - uptime in seconds (float)
        ctr_uptimeidle        - idle uptime in seconds (float)
        ctr_cpu_user          - total cpu in user space (int)
        ctr_cpu_nice          - total cpu in user nice space (int)
        ctr_cpu_system        - total cpu in system space (int)
        ctr_cpu_idle          - total cpu in idle thread (int)
        ctr_cpu%d_user        - per cpu in user space (e.g., cpu0, cpu1, etc) (int)
        ctr_cpu%d_nice        - per cpu in user nice space (e.g., cpu0, cpu1, etc) (int)
        ctr_cpu%d_system      - per cpu in system space (e.g., cpu0, cpu1, etc) (int)
        ctr_cpu%d_idle        - per cpu in idle thread (e.g., cpu0, cpu1, etc) (int)
        ctr_pages_in          - pages read in (int)
        ctr_pages_out         - pages written out (int)
        ctr_pages_swapin      - swap pages read in (int)
        ctr_pages_swapout     - swap pages written out (int)
        ctr_interrupts        - number of interrupts received (int)
        ctr_contextswitches   - number of context switches (int)
        ctr_processes         - number of processes started (I think?) (int)
        boottime              - time of boot (epoch) (int)

  HP-UX:

        System stats from '/usr/bin/uptime':
            uptime          - (string)
            users           - (int)
            loadavg1        - (float)
            loadavg5        - (float)
            loadavg15       - (float)

        System counters from '/usr/bin/vmstat -s' (see vmstat(1)):
            ctr_swap_ins                                    - (long)
            ctr_swap_outs                                   - (long)
            ctr_pages_swapped_in                            - (long)
            ctr_pages_swapped_out                           - (long)
            ctr_total_address_trans_faults_taken            - (long)
            ctr_page_ins                                    - (long)
            ctr_page_outs                                   - (long)
            ctr_pages_paged_in                              - (long)
            ctr_pages_paged_out                             - (long)
            ctr_reclaims_from_free_list                     - (long)
            ctr_total_page_reclaims                         - (long)
            ctr_intransit_blocking_page_faults              - (long)
            ctr_zero_fill_pages_created                     - (long)
            ctr_zero_fill_page_faults                       - (long)
            ctr_executable_fill_pages_created               - (long)
            ctr_executable_fill_page_faults                 - (long)
            ctr_swap_text_pages_found_in_free_list          - (long)
            ctr_inode_text_pages_found_in_free_list         - (long)
            ctr_revolutions_of_the_clock_hand               - (long)
            ctr_pages_scanned_for_page_out                  - (long)
            ctr_pages_freed_by_the_clock_daemon             - (long)
            ctr_cpu_context_switches                        - (long)
            ctr_device_interrupts                           - (long)
            ctr_traps                                       - (long)
            ctr_system_calls                                - (long)
            ctr_Page_Select_Size_Successes_for_Page_size_4K - (long)
            ctr_Page_Select_Size_Successes_for_Page_size_16K - (long)
            ctr_Page_Select_Size_Successes_for_Page_size_64K - (long)
            ctr_Page_Select_Size_Successes_for_Page_size_256K - (long)
            ctr_Page_Select_Size_Failures_for_Page_size_16K - (long)
            ctr_Page_Select_Size_Failures_for_Page_size_64K - (long)
            ctr_Page_Select_Size_Failures_for_Page_size_256K - (long)
            ctr_Page_Allocate_Successes_for_Page_size_4K    - (long)
            ctr_Page_Allocate_Successes_for_Page_size_16K   - (long)
            ctr_Page_Allocate_Successes_for_Page_size_64K   - (long)
            ctr_Page_Allocate_Successes_for_Page_size_256K  - (long)
            ctr_Page_Allocate_Successes_for_Page_size_64M   - (long)
            ctr_Page_Demotions_for_Page_size_16K            - (long)

  ### Appendix A

  Time Definition

  The format for specifying time is either:

  - Integer number = that number in seconds; or
  - A string of the form "<int>[smhdwcy]" (an integer followed by one of the specified characters) where the characters define the time scale:
    - s = seconds
    - m = minutes
    - h = hours
    - d = days
    - w = weeks
    - c = calendar months
    - y = years

  [ EDDIE-Tool Homepage ]

  © Chris Miles 2002-2005

  $Id: manual.html 910 2007-12-10 12:48:51Z chris $