================
Crate Test Layer
================

Basic Usage
-----------

This layer starts and stops a ``Crate`` instance on a given host, port,
a given crate node name and, optionally, a given cluster name::

    >>> from crate.testing.layer import CrateLayer
    >>> import random

    >>> port = 44219
    >>> transport_port = 44319

    >>> layer =  CrateLayer('crate',
    ...                     crate_home=crate_path(),
    ...                     host='127.0.0.1',
    ...                     port=port,
    ...                     transport_port=transport_port,
    ...                     cluster_name='my_cluster'
    ... )

The working directory is defined on layer instantiation.
It is sometimes required to know it before starting the layer::

    >>> layer.wdPath()
    '...crate_layer/crate'

Lets start the layer::

    >>> layer.start()

The urls of the crate servers to be instantiated can be obtained
via ``crate_servers``::

    >>> layer.crate_servers
    ['http://127.0.0.1:44219']


Now we can access the ``Crate`` instance on the defined port::

    >>> import urllib3
    >>> http = urllib3.PoolManager()

    >>> stats_uri = "http://127.0.0.1:{0}/".format(port)
    >>> response = http.request('GET', stats_uri)
    >>> response.status
    200


The layer can be shutdown using its ``stop()`` method::

    >>> layer.stop()

Dynamic HTTP Port
-----------------

It is also possible to define a port range instead of a static HTTP port for
the layer::

    >>> port = '44200-44299'
    >>> layer = CrateLayer('crate',
    ...                    crate_home=crate_path(),
    ...                    port=port,
    ... )
    >>> layer.start()
    >>> layer.crate_servers
    ['http://127.0.0.1:442...']
    >>> layer.stop()

Crate will start with the first available port in the given range and the test
layer obtains the chosen port from the startup logs of the Crate process.
Note, that this feature requires a logging configuration with at least loglevel
``INFO`` on ``http``.


Default Values
--------------

Starting a ``Crate`` layer leaving out optional parameters will apply the following defaults::

    >>> layer_defaults = CrateLayer('crate_defaults',
    ...                             crate_path()
    ... )

The default http port is the first free port in the range of ``4200-4299``,
the default transport port is the first free port in the range of ``4300-4399``,
the host defaults to ``127.0.0.1``::

    >>> layer_defaults.start()
    >>> layer_defaults.crate_servers
    ['http://127.0.0.1:4200']

    >>> layer_defaults.stop()

The command to call is ``bin/crate`` inside the ``crate_home`` path.
The default config file is ``config/crate.yml`` inside ``crate_home``.
The default cluster name will be auto generated using the HTTP port.


Additional Settings
-------------------

The ``Crate`` layer can be started with additional settings as well.
Add a dictionary for keyword argument ``settings`` which contains your settings.
Those additional setting will override settings given as keyword argument.

The settings will handed over to the ``Crate`` process with the ``-C`` flag.
So the setting ``threadpool.bulk.queue_size: 100`` becomes
the command line flag: ``-Cthreadpool.bulk.queue_size=100``::

    >>> custom_layer = CrateLayer(
    ...     'custom',
    ...     crate_path(),
    ...     port=44401,
    ...     settings = {
    ...         "cluster.graceful_stop.min_availability": "none",
    ...         "http.port": 44402
    ...     }
    ... )

    >>> custom_layer.start()
    >>> custom_layer.crate_servers
    ['http://127.0.0.1:44402']
    >>> '-Ccluster.graceful_stop.min_availability=none' in custom_layer.start_cmd
    True
    >>> custom_layer.stop()


Verbosity
---------

The test layer hides the standard output of Crate per default. To increase the
verbosity level the additional keyword argument ``verbose`` needs to be set
to ``True``::

    >>> layer =  CrateLayer('crate',
    ...                     crate_home=crate_path(),
    ...                     verbose=True
    ... )

    >>> layer.start()
    >>> layer.verbose
    True
    >>> layer.stop()

Environment variables
---------------------

It is possible to provide environment variables for the ``Crate`` testing
layer::

    >>> layer =  CrateLayer('crate',
    ...                     crate_home=crate_path(),
    ...                     env={"CRATE_HEAP_SIZE": "300m"}
    ... )

    >>> layer.start()

    >>> import json
    >>> sql_uri = layer.crate_servers[0] + "/_sql"
    >>> response = http.urlopen('POST', sql_uri,
    ...     body='{"stmt": "select heap[\'max\'] from sys.nodes"}')
    >>> json_response = json.loads(response.data.decode('utf-8'))
    >>> json_response["rows"][0][0]
    314572800

    >>> layer.stop()

Starting a Cluster
------------------

To start a cluster of ``Crate`` instances, give each instance the same
``cluster_name``. If you want to start instances on the same machine then
 use value ``_local_`` for ``host`` and give every node different ports::

    >>> cluster_layer1 = CrateLayer(
    ...     'crate1',
    ...     crate_path(),
    ...     host='_local_',
    ...     cluster_name='my_cluster',
    ... )
    >>> cluster_layer2 = CrateLayer(
    ...     'crate2',
    ...     crate_path(),
    ...     host='_local_',
    ...     cluster_name='my_cluster',
    ...     settings={"discovery.initial_state_timeout": "10s"}
    ... )

If we start both layers, they will, after a small amount of time, find each other
and form a cluster::

    >>> cluster_layer1.start()
    >>> cluster_layer2.start()

We can verify that by checking the number of nodes a node knows about::

    >>> import json
    >>> def num_cluster_nodes(crate_layer):
    ...     sql_uri = crate_layer.crate_servers[0] + "/_sql"
    ...     response = http.urlopen('POST', sql_uri, body='{"stmt":"select count(*) from sys.nodes"}')
    ...     json_response = json.loads(response.data.decode('utf-8'))
    ...     return json_response["rows"][0][0]

We might have to wait a moment before the cluster is finally created::

    >>> num_nodes = num_cluster_nodes(cluster_layer1)
    >>> import time
    >>> retries = 0
    >>> while num_nodes < 2:
    ...     time.sleep(1)
    ...     num_nodes = num_cluster_nodes(cluster_layer1)
    ...     retries += 1
    ...     if retries == 30:
    ...         break
    >>> num_nodes
    2

    >>> cluster_layer1.stop()
    >>> cluster_layer2.stop()

From Uri
--------

The CrateLayer can also be created by providing a URI that points to a Crate
tarball::

    >>> import urllib.request
    >>> import json
    >>> with urllib.request.urlopen('http://crate.io/versions.json') as response:
    ...     versions = json.loads(response.read().decode())
    ...     version = versions['crate_testing']

    >>> uri = 'https://cdn.crate.io/downloads/releases/crate-{}.tar.gz'.format(version)
    >>> tmpdir = tempfile.mkdtemp()
    >>> layer = CrateLayer.from_uri(
    ...             uri, name='crate-uri', http_port=42203, directory=tmpdir)
    >>> layer.setUp()

    >>> work_dir = os.path.join(tmpdir, 'crate-' + version)
    >>> os.path.exists(work_dir)
    True

    >>> layer.tearDown()

    >>> os.path.exists(work_dir)
    False
