WeaveQ
=======

WeaveQ: A program and module for pivoting and joining across collections of 
data, with special support for pivoting and joining across JSON files, CSV 
files and Elasticsearch resultsets.

Getting Started
---------------

**Prerequisites**

In order to install WeaveQ you will need:

- Python 2.7 or higher (including Python 3)
- pip, a Python package installer (the pip package is usually called 
  ``python-pip`` in Linux package repositories)

For development of WeaveQ, you will need:

- tox >= 2.7.0 (available from PyPi)
- six >= 1.5.2 (available from PyPi)
- pyparsing >= 2.2.0 (available from PyPi)
- elasticsearch >= 5.1.0 (available from PyPi)
- elasticsearch_dsl >= 5.1.0 (available from PyPi)
- Doxygen (optional, available from your OS package repository)
- Sphinx (optional, available from PyPi)

**Installing**

To install the latest stable WeaveQ, use pip (you may need to ``sudo``)::

    $ pip install weaveq

To install from source, run the ``setup.py`` file located in the WeaveQ 
repository root (you may need to ``sudo``)::

    $ python setup.py install

**Documentation**

Comprehensive WeaveQ documentation is located at: <https://readthedocs.org/projects/weaveq/>

To build the documentation in the WeaveQ repository locally, from the repository root::

    $ doxygen
    $ cd docs
    $ make html

In-code HTML documentation will be written to ``docs/code/html`` and 
user/developer HTML documentation will be written to ``docs/build/html``.

Running the Tests
-----------------

There are 3 types of WeaveQ tests:

1. **Unit:** Exercises individual classes.

2. **System:** Exercises unit integration, integration with external modules 
   and integration with Elasticsearch.

3. **Performance:** Measures average performance of pre-defined operations to 
   help gauge performance impact of code changes.

For the system tests to work correctly, you need to have an Elasticsearch 
instance accessible from your development system (a vanilla, single local 
node should be sufficient). You will have to configure the system tests to 
point to the Elasticsearch node - do this by entering the host and port in 
the ``tests/system/elastic_test_config.json`` file.

To run the unit and system tests, from the repository root run::

    $ tox
    
``tox`` is used to manage virtual environments for testing that WeaveQ installs 
and runs with multiple Python versions. To successfully test both Python 2.7 
and Python 3 compatibility, you need to have these interpreters installed on 
your system.

To run the performance tests, from the repository root run::

    $ python perf_tests.py

Note that WeaveQ compatibility with the following has not been tested in this 
release (though this doesn't necessarily mean it won't work):

- Python versions < 2.7.6 or > 3.4
- Operating systems other than Linux
- Elasticsearch versions < 5.1.2

Deployment
----------

In order to use Elasticsearch data sources in WeaveQ queries, you must supply 
a configuration file on the command line. This file specifies the 
addresses/host names of Elasticsearch nodes, authentication certificates and 
other settings.

If deploying to a controlled environment with an Elasticsearch instance that 
you want users to access by default, consider providing a pre-written 
configuration file (for example, in ``/etc/weaveq``) so that users can get up 
and running quickly. You may also want to alias the ``weaveq`` command to 
specify the default configuration without requiring users to specify the 
``--config`` option.

The configuration file format is documented at 
<https://readthedocs.org/projects/weaveq/>

Versioning
----------

WeaveQ versioning follows the SemVer specification: <http://semver.org/>

You can see the tagged WeaveQ versions at: 
<https://github.com/jamesmistry/weaveq/tags>

Licence
-------

This project is licensed under the MIT License - see the LICENSE file.


