:Author:    D Haynes
:Date:      June 2009

..  Titling
    ##++::==~~--''``

..  sectnum::
    :start: 1

HWIT Developer guide
====================

This explains how the HWIT library is used. It will interest you if:

*   You want to make your own HWIT forms
*   You need to write scripts which use the `hwit.core` library
*   You would like to contribute tests, code, or documentation to the
    HWIT project.

HWIT development
~~~~~~~~~~~~~~~~

The HWIT project is a `Free software`_ project. HWIT is written in
Python_. The code is developed on a Ubuntu_ GNU/Linux platform. It is
maintained in a Bazaar_ version control repository. The documentation
is written in reStructuredText_ and generated by the Sphinx_ package.

`hwit-core` needs Python 2.5 or later.

`hwit-console` - the HWIT CLI
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

`hwit-console` is a basic command line tool for `hwit.core`. The intention is
that whatever functionality is provided by `the hwit.core library`_ should be
exposed in a console session via this simple program.

..  automodule::    hwit.core.utils.hwit_console
    :synopsis:  The user application for the HWIT project
    :members:

Making your own form
~~~~~~~~~~~~~~~~~~~~

When you've used HWIT a few times, you might think of an idea for a form
of your own. The `hwit-console` can help you make one.

The method for making your own form proceeds as follows:

1. `Make a spreadsheet which describes your form`_
2. `Export the spreadsheet to a tab-separated values (TSV) file`_
3. `Feed the TSV file to HWIT`_

You keep the definitions of your forms as sheets in a spreadsheet
workbook. You can use Microsoft Excel_, OpenOffice Calc_, Gnumeric_,
or any other spreadsheet program.

Each row in your sheet describes a field in your form.

..  automodule::    hwit.core.generator

Feed the TSV file to HWIT
-------------------------

Issue the command::

    hwit-console -g < my_form_template.tsv > my_new_form.hwit

From the command line you may also supply one or more
``--tag=<group_id>:<field_name>`` options. The effect of these is to
place a value in the associated field (see `hwit.core.model`_ below).
The format of the value is such as to permit string substitution by another
program (a ``sed``, ``awk``, ``perl``, or ``python`` script, for example).
This feature is useful when you want to automatically prepopulate forms
with specific data.

Your HWIT installation contains example form templates for you to study.
Load the file ``test/ref/scout_camp.tsv`` or
``test/ref/gui_test-template.tsv`` into a spreadsheet program to learn more.

The hwit.core library
~~~~~~~~~~~~~~~~~~~~~

`hwit-core` provides an API to interact with HWIT forms. A Python script
imports the library as follows::

    from hwit.core.model import Container
    from hwit.core.context import StrictReadContext

hwit.core.context
-----------------

Contexts are the preferred method of reading a HWIT file. They allow
you to decide (and declare in your code) what your response is to errors
in the file.

First, you should open the HWIT file for reading and pass it as a file object
to the constructor of :class:`hwit.core.model.Container`. This container object is
then the means by which you create the necessary reading context.

In order that this works in version 2.5 of Python, you need to start your
script with the line::

    from __future__ import with_statement

..  autoclass:: hwit.core.context.PermissiveReadContext
    :members:

..  autoclass:: hwit.core.context.StrictReadContext
    :members:

hwit.core.model
---------------

..  automodule::    hwit.core.model
    :synopsis:  data model for HWIT containers

    The classes in this module allow navigation of a loaded HWIT
    document. The members documented here comprise the only API
    you should rely on.

..  autoclass::    hwit.core.model.Container
    :members: title, meta, groups, state, rule, read, write

..  autoclass::    hwit.core.model.Group
    :members: state, rule, get

..  autoclass::    hwit.core.model.Field
    :members: name, checks, get, set

hwit.checks
~~~~~~~~~~~

The checking methods available in `hwit.checks` are useful
both to developers and those who design forms. They provide a
method of validating user input against a single defined type.

..  table:: Validators in use with HWIT

    =========================   ================================================
    Type of data                Validator
    =========================   ================================================
    Truth value                 hwit.checks.common.booltyper
    Non-negative whole number   hwit.checks.common.numbertyper.isPositiveInteger
    Year, month, day            hwit.checks.common.datetimetyper.isISODate
    Hours, minutes, seconds     hwit.checks.common.datetimetyper.isISOTime
    Date, time                  hwit.checks.common.datetimetyper.isISODateTime
    Email address               hwit.checks.common.emailtyper
    Survey vote                 hwit.checks.common.scaletyper.isSevenValuedVote
    =========================   ================================================
   

hwit.checks.common
------------------

..  automodule::    hwit.checks.common
    :synopsis:  validation services
    :exclude-members: AllowsEmptyStrings, likeTyper
    :members:

hwit.core.utils
~~~~~~~~~~~~~~~

The source distribution of HWIT gives you extra goodies in the *utils*
directory. As well as being useful to the project, they provide examples
of how to use HWIT in a scripting environment.

happenstance.py
---------------

Happenstance is a *fuzzer*. This is a program for generating lots of
``.hwit`` files which are in some way broken. This is to help test that
HWIT deals with unusual data in a robust way.

..  program::   happenstance

..  cmdoption:: -h

    Show the help page

..  cmdoption:: -o

    Set an output directory

..  cmdoption:: -n

    Set the number of files to generate

triage.py
---------

Triage is designed to work hand-in-hand with happenstance.py_. It loads
each ``.hwit`` file at a time, and categorises each one according to the
error it causes. This helps to identify and prioritise flaws in the HWIT
code.
 
..  program::   triage

..  cmdoption:: -h

    Show the help page

..  cmdoption:: -d

    Run in debug mode. This runs HWIT within the Python debugger, to
    diagnose the exact cause of the error.

..  cmdoption:: -v

    Save extra information alongside each generated error.

..  cmdoption:: -o

    Set the directory in which to store the output.

ingest.py
---------

When you have a significant volume of business data, putting it into 
reporting database can facilitate structured query.
This is what *ingest.py* does.  The output is a sqlite_ file which
gives you industrial-strength relational storage and a highly performant
search capability.

..  program::   ingest

..  cmdoption:: -h

    Show the help page

..  cmdoption:: -v

    Run in verbose mode. This gives more information as the program runs

cluster.py
----------

This script demonstrates a basic technique for visualising a set of HWIT
files as a sequence of 2D images.

This is most appropriate when the fields hold numeric values. In such
cases, the pattern matching abilities of the brain can outperform
computer clustering algorithms.

Looking at these patterns, and resolving to a number of groups can be a
good start to investigating an unknown data set.

The output of this script is a ``.gp`` file, which you can view using
the freely available gnuplot_ program.

..  program::   cluster

..  cmdoption:: -h

    Show the help page

..  cmdoption:: -v

    Run in verbose mode. This gives more information as the program runs

How to contribute code to HWIT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

HWIT welcomes your contribution. There are various ways you can help:

*   Use HWIT and tell everyone how good it is
*   Use HWIT, discover a weakness, and tell us so we can fix it
*   Use HWIT, discover a weakness, fix it, and share your work with us

Since this is the `HWIT Developer guide`_ , we focus here on the final
case, and we assume you are a technical person who can write test cases
in Python_.

Write tests
-----------

Writing test cases is absolutely essential to the successful development
of HWIT. Once a bug is discovered, the *very first thing we do* is to
write some code (called a *test case*) which demonstrates the bug.

Recreating the bug like this is important because:

*   When the test passes, we know our fix works
*   Over time, the number of tests accumulates. They prevent us from
    accidentally reintroducing a bug in a future release

You can run all the current test cases by launching Python's `unittest`
with the ``-m`` option from the command line::

    python -m unittest discover

In order to write tests, you should be familiar with the Python_
documentation on the `unittest module`_ . This is the framework
which enables the testing of the HWIT code.

You should write your tests in a copy of the most recently available
snapshot.

Fix the bug
-----------

Your unit tests are a contribution in themselves. If you are able to
fix the bug you discovered then this is a real bonus. Please share
this with the project. First you `make a patch`_, then notify us by
email. 

Make a patch
------------

The command line tool `diff` and its cousin `patch` are standard on
Unix-like systems. Users of Microsoft Windows will find a substitute
called `diff.py` in their Python_ distribution.
 
`Diff` is used to find the differences between individual text files,
or entire directories thereof. What follows is a very brief treatment;
if you want to know more you should find a good `diff and patch tutorial`_
on the web.

To generate a diff of your changes, change the name of your modified
snapshot to a useful name, eg: ``hwit-core-0.01-r00161-mybugfix``. Next
you need an original copy of *the same snapshot*, without your changes.

From the command line, you generate a `diff` like this::

    diff -urN hwit-core-0.01-r00161 hwit-core-0.01-r00161-mybugfix > hwit-core-0.01-r00161-mybugfix.patch

The patch file will contain step-by-step instructions on how to convert
the standard snapshot to your local, modified version.

If you are forced to use the Python version of `diff` (because
you're on MS Windows and that's all there is), you can only
do it one file at a time, ie::

    C:\play>c:\Python26\Tools\Scripts\diff.py -u
    hwit-core-0.01-r00161\doc\hwit-guide.rst
    hwit-core-0.01-r00161-mybugfix\doc\hwit-guide.rst

Versioning
~~~~~~~~~~

HWIT's version numbering begins with a major identifier,
follows with a ``.``, then a minor identifier, then ``-r``
and the revision number, which will continue to increment monotonically.
Eg: `0.01-r00758` .

HWIT's revision number is the main differentiator between versions.
The minor numbers will increment when new features are added.

At some point in the future, when confidence in the code permits,
the major number will change from 0 to 1.

Gateways to version 1
---------------------

Completed
'''''''''

*   Unit tests to achieve 100% code coverage for core modules
*   Unified error handling for syntax errors
*   Tight control of, eg: forbidden tags in file format

Open tasks
''''''''''

*   Full internationalisation

References
~~~~~~~~~~

..  target-notes::

..  todolist::

..  footnotes
..  _Free software: http://www.gnu.org/philosophy/free-sw.html
..  _Python: http://www.python.org
..  _Ubuntu: http://www.ubuntu.com
..  _Bazaar: http://bazaar-vcs.org
..  _reStructuredText: http://docutils.sourceforge.net/rst.html
..  _Sphinx: http://sphinx.pocoo.org
..  _Excel: http://office.microsoft.com/excel 
..  _Calc: http://www.openoffice.org/product/calc.html
..  _Gnumeric: http://www.gnome.org/gnumeric
..  _unittest module: http://docs.python.org/library/unittest.html#module-unittest
..  _diff and patch tutorial: http://stephenjungels.com/jungels.net/articles/diff-patch-ten-minutes.html
..  _contact the project: tundish@thuswise.org
..  _sqlite: http://www.sqlite.org
..  _gnuplot: http://www.gnuplot.info
