==============
Tutorial
==============

RAET is written in python. RAET uses Stack objects.
There are currently two types of RAET Stacks.

- A RoadStack provides communication between IP hosts via UDP sockets
- A LaneStack provides communication between processes on the same host via Unix Domain Sockets (UXD) on unix or mail slots on windows.

-------------
RoadStack
-------------

Although a RoadStack has a flexible set of configuration parameters,
it tries to use intelligent defaults where ever possible.

The following python snippet shows how to create a RoadStack.

.. code-block:: python

    import raet
    alpha = raet.road.stacking.RoadStack()


This will create a stack with an autogenerated unique name and the default
host address which is listening on any local IP interface and port 7530.

.. code-block:: bash

    >>> alpha.ha
       ('0.0.0.0', 7530)

A RoadStack will persist or save to disk some of its parameters. This is so
that if the system goes down, it can resume using the persisted data.
The default location for the persisted data is in /tmp on unix. If the user
does not have write permissions to /tmp then the .raet folder in the user's home
directory is used.

.. code-block:: python
    >>> alpha.keep.dirpath
       '/Users/username/.raet/keep/alpha'

To clear the persisted data call:

.. code-block:: bash

    >>> alpha.keep.clearAllDir()

A different name or a host address can be assigned at creation time.
To create another RoadStack just provide it with a unique name and host address.

In order for two RoadStacks to communicate one with another. They need to be connected.
RAET uses elliptic curve cryptography to sign and encrypt messages.
This requires that the two RoadStacks exchange keys and perform a handshake.

Each RoadStack has an .auto accept attribute that governs how it will accept keys.
The auto accept has three modes, these are: never, once, and always. For the
purposes of this tutorial we will use the least restrictive mode which is 'always'.
The default is the most restrictive which is 'never'.

If two RoadStack have never been connected before and have not exchanged keys then
they may perform what is called a vacuous join handshake to exhange keys. The main
parameter, if True, will allow that RoadStack to accept a vacuous join handshake
from another RoadStack.

The first step for one RoadStack to initiate the formation of a connection with
another RoadStack is for that RoadStack to add a RemoteEstate to its list of estates.
In the RAET coding metaphor, an estate represents a host on a UDP channel.

Another important concept to understand about a RoadStack is that it uses a
non-blocking UDP socket. This means that the socket must be repeatedly polled.
Non-blocking sockets makes RAET compatible with non-blocking concurrency frameworks.
RAET is  intended to be used with Ioflo, a non blocking flow based programming framework.
But RAET can be used with other non-blocking I/O python frameworks especially those
that use yield (generators).  When not using a framework, One can easily service
the non-blocking I/O with a'busy loop'.

The following example code will create two roadstacks and perform a vacuous handshake
to exchange keys. It employs a busy loop to service the non-blocking io. At the
end it closes the UDP sockets and clears the persistence files.

In a real application the closing and clearing should not happen until exiting
the application.

.. code-block:: python

    '''
    RAET Tutorial Example
    '''
    import time

    import raet
    from raet.raeting import AutoMode

    def example1():
        alpha = raet.road.stacking.RoadStack(name='alpha',
                                             ha=('0.0.0.0', 7531),
                                             auto=raeting.autoModes.always)

        beta = raet.road.stacking.RoadStack(name='beta',
                                             ha=('0.0.0.0', 7532),
                                             main=True,
                                             auto=raeting.autoModes.always)

        remote = raet.road.estating.RemoteEstate(stack=alpha,
                                                 ha=beta.ha)

        alpha.addRemote(remote)

        alpha.join(uid=remote.uid, cascade=True)

        stacks = [alpha, beta]
        while True:
            for stack in stacks:
                stack.serviceAll()
                stack.store.advanceStamp(0.1)
            if all([not stack.transactions for stack in stacks]):
                break
            time.sleep(0.1)

        for stack in stacks:
            stack.server.close()  # close the UDP socket
            stack.keep.clearAllDir()  # clear persisted data

        print("Finished\n")

    if __name__ == "__main__":
        example1()


The RAET log should print the following to the console.

.. code-block::

    Joiner alpha. Do Join with estate_3181b42bb09b11e4a6136c400891de78 in 1 at 0.0
    Joinent beta. Added new remote name='alpha' nuid='2' fuid='2' ha='('127.0.0.1', 7531)' role='alpha'
    Joinent beta. Do Accept of alpha in 1 at 0.0
    Joiner alpha. Do Ack Accept, Done with beta in 1 at 0.1
    Allower alpha. Do Hello with beta in 2 at 0.1
    Joinent beta. Done with alpha in 1 at 0.1
    Allowent beta. Do Cookie with alpha in 2 at 0.1
    Allower alpha. Do Initiate with beta in 2 at 0.2
    Allowent beta. Do Ack Initiate with alpha in 2 at 0.2
    Allower alpha. Do Ack Final, Done with beta in 2 at 0.3
    Aliver alpha. Do Alive with beta in 3 at 0.3
    Allowent beta. Done with alpha in 2 at 0.3
    Alivent beta. Do ack alive with alpha in 3 at 0.3
    Alivent beta. Done with alpha in 3 at 0.3
    Aliver alpha. Done with beta in 3 at 0.4
    Finished

What the log shows is that 3 different transactions are completed successfully.
The first transaction is a Join that is composed of a Joiner on the alpha side and
a Joinent on the beta side. The next transaction is an Allow that is composed of
an Allower on the alpha side and an Allowent on the beta side. The final transaction
is an Alive that is composed of an Aliver on the alpha side and an Alivent on the
beta side.

The Join transaction exchanges the signing and long term encryption keys. The
Allow transaction performs a Curve-CP handshake to exchange the short term encryption
keys. The Alive transaction performs a heatbeat to verify that both sides are
communicative and is used to support a presence service in RAET.

In the following example, to stacks will first complete the handshake and each
send a message to each other. The messages will be signed and encrypted.

.. code-block:: python

    '''
    RAET Tutorial Example
    '''
    import time

    import raet
    from raet.raeting import AutoMode

    def example2():

        alpha = raet.road.stacking.RoadStack(name='alpha',
                                             ha=('0.0.0.0', 7531),
                                             auto=raeting.autoModes.always)

        beta = raet.road.stacking.RoadStack(name='beta',
                                            ha=('0.0.0.0', 7532),
                                            main=True,
                                            auto=raeting.autoModes.always)

        remote = raet.road.estating.RemoteEstate(stack=alpha,
                                                 ha=beta.ha)

        alpha.addRemote(remote)

        alpha.join(uid=remote.uid, cascade=True)

        stacks = [alpha, beta]
        while True:
            for stack in stacks:
                stack.serviceAll()
                stack.store.advanceStamp(0.1)
            if all([not stack.transactions for stack in stacks]):
                break
            time.sleep(0.1)

        print("Finished Handshake\n")

        msg =  {'subject': 'Example message alpha to beta',
                'content': 'The dict keys in this dict are not special any dict will do.',}

        alpha.transmit(msg, remote.uid)
        while True:
            for stack in stacks:
                stack.serviceAll()
                stack.store.advanceStamp(0.1)
            if all([not stack.transactions for stack in stacks]):
                break
            time.sleep(0.1)

        rx = beta.rxMsgs.popleft()
        print("{0}\n".format(rx))
        print("Finished Message alpha to beta\n")

        msg =  {'subject': 'Example message beta to alpha',
                'content': 'Messages are the core of raet.',}

        beta.transmit(msg, remote.uid)
        while True:
            for stack in stacks:
                stack.serviceAll()
                stack.store.advanceStamp(0.1)
            if all([not stack.transactions for stack in stacks]):
                break
            time.sleep(0.1)

        rx = alpha.rxMsgs.popleft()
        print("{0}\n".format(rx))
        print("Finished Message beta to alpha\n")

        for stack in stacks:
            stack.server.close()  # close the UDP socket
            stack.keep.clearAllDir()  # clear persisted data

        print("Finished\n")

    if __name__ == "__main__":
        example2()

The RAET log should print the following to the console.

.. code-block::

    Joiner alpha. Do Join with estate_d68ca540b0a011e4ba4e6c400891de78 in 1 at 0.0
    Joinent beta. Added new remote name='alpha' nuid='2' fuid='2' ha='('127.0.0.1', 7531)' role='alpha'
    Joinent beta. Do Accept of alpha in 1 at 0.0
    Joiner alpha. Do Ack Accept, Done with beta in 1 at 0.1
    Allower alpha. Do Hello with beta in 2 at 0.1
    Joinent beta. Done with alpha in 1 at 0.1
    Allowent beta. Do Cookie with alpha in 2 at 0.1
    Allower alpha. Do Initiate with beta in 2 at 0.2
    Allowent beta. Do Ack Initiate with alpha in 2 at 0.2
    Allower alpha. Do Ack Final, Done with beta in 2 at 0.3
    Aliver alpha. Do Alive with beta in 3 at 0.3
    Allowent beta. Done with alpha in 2 at 0.3
    Alivent beta. Do ack alive with alpha in 3 at 0.3
    Alivent beta. Done with alpha in 3 at 0.3
    Aliver alpha. Done with beta in 3 at 0.4
    Finished Handshake

    Messenger alpha. Do Message Segment 0 with beta in 4 at 0.5
    Messengent beta. Do Ack Done Message on Segment 0 with alpha in 4 at 0.5
    Messengent beta. Complete with alpha in 4 at 0.5
    Messenger alpha. Done with beta in 4 at 0.6
    ({u'content': u'The dict keys in this dict are not special any dict will do.', u'subject': u'Example message alpha to beta'}, u'alpha')

    Finished Message alpha to beta

    Messenger beta. Do Message Segment 0 with alpha in 1 at 0.7
    Messengent alpha. Do Ack Done Message on Segment 0 with beta in 1 at 0.8
    Messengent alpha. Complete with beta in 1 at 0.8
    Messenger beta. Done with alpha in 1 at 0.8
    ({u'content': u'Messages are the core of raet.', u'subject': u'Example message beta to alpha'}, u'beta')

    Finished Message beta to alpha

    Finished


The next example show communication between 3 stacks.

.. code-block::

    '''
    RAET Tutorial Example
    '''
    import time

    import ioflo

    from ioflo.base.consoling import getConsole
    console = getConsole()
    console.reinit(verbosity=console.Wordage.concise)

    import raet
    from raet import raeting
    from raet.raeting import AutoMode


    def serviceStacks(stacks, duration=1.0, period=0.1):
        '''
        Utility method to service queues. Call from test method.
        '''
        store = ioflo.base.storing.Store(stamp=0.0)
        timer = ioflo.base.aiding.StoreTimer(store=store, duration=duration)
        while not timer.expired:
            for stack in stacks:
                stack.serviceAll()
                stack.store.advanceStamp(period)

            store.advanceStamp(period)
            if all([not stack.transactions for stack in stacks]):
                break
            time.sleep(period)
        console.concise("Perceived service duration = {0} seconds\n".format(timer.elapsed))


    def example3():
        alpha = raet.road.stacking.RoadStack(name='alpha',
                                             ha=('0.0.0.0', 7531),
                                             main=True,
                                             auto=AutoMode.always.value)

        beta = raet.road.stacking.RoadStack(name='beta',
                                            ha=('0.0.0.0', 7532),
                                            main=True,
                                            auto=AutoMode.always.value)

        gamma = raet.road.stacking.RoadStack(name='gamma',
                                            ha=('0.0.0.0', 7533),
                                            main=True,
                                            auto=AutoMode.always.value)

        remote = raet.road.estating.RemoteEstate(stack=alpha,
                                                 name=beta.name,
                                                 ha=beta.ha)
        alpha.addRemote(remote)
        alpha.join(uid=remote.uid, cascade=True)

        remote = raet.road.estating.RemoteEstate(stack=alpha,
                                                 name=gamma.name,
                                                 ha=gamma.ha)
        alpha.addRemote(remote)
        alpha.join(uid=remote.uid, cascade=True)

        remote = raet.road.estating.RemoteEstate(stack=beta,
                                                 name=gamma.name,
                                                 ha=gamma.ha)
        beta.addRemote(remote)
        beta.join(uid=remote.uid, cascade=True)

        stacks = [alpha, beta, gamma]
        serviceStacks(stacks)
        print("Finished Handshakes\n")

        msg =  {'subject': 'Example message alpha to whoever',
                'content': 'Hi',}
        for remote in alpha.remotes.values():
            alpha.transmit(msg, remote.uid)

        msg =  {'subject': 'Example message beta to whoever',
                'content': 'Hello.',}
        for remote in beta.remotes.values():
            beta.transmit(msg, remote.uid)

        msg =  {'subject': 'Example message gamma to whoever',
                'content': 'Good Day',}
        for remote in gamma.remotes.values():
            gamma.transmit(msg, remote.uid)

        serviceStacks(stacks)
        print("Finished Messages\n")

        for stack in stacks:
            print("Stack {0} received:\n".format(stack.name))
            while stack.rxMsgs:
                msg, source = stack.rxMsgs.popleft()
                print("source = '{0}'.\nmsg= {1}\n".format(source, msg))


        for stack in stacks:
            stack.server.close()  # close the UDP socket
            stack.keep.clearAllDir()  # clear persisted data

        print("Finished\n")


    if __name__ == "__main__":
        example3()


With the console verbosity level set to concise the following log output is given:

.. code-block::

    Joiner alpha. Do Join with beta in 1 at 0.0
    Joiner alpha. Do Join with gamma in 1 at 0.0
    Joiner beta. Do Join with gamma in 1 at 0.0
    Joinent beta. Added new remote name='alpha' nuid='3' fuid='2' ha='('127.0.0.1', 7531)' role='alpha'
    Joinent beta. Do Accept of alpha in 1 at 0.0
    Joinent gamma. Added new remote name='alpha' nuid='2' fuid='3' ha='('127.0.0.1', 7531)' role='alpha'
    Joinent gamma. Do Accept of alpha in 1 at 0.0
    Joinent gamma. Added new remote name='beta' nuid='3' fuid='2' ha='('127.0.0.1', 7532)' role='beta'
    Joinent gamma. Do Accept of beta in 1 at 0.0
    Joiner alpha. Do Ack Accept, Done with beta in 1 at 0.1
    Allower alpha. Do Hello with beta in 2 at 0.1
    Joiner alpha. Do Ack Accept, Done with gamma in 1 at 0.1
    Allower alpha. Do Hello with gamma in 2 at 0.1
    Joiner beta. Do Ack Accept, Done with gamma in 1 at 0.1
    Allower beta. Do Hello with gamma in 2 at 0.1
    Joinent beta. Done with alpha in 1 at 0.1
    Allowent beta. Do Cookie with alpha in 2 at 0.1
    Joinent gamma. Done with alpha in 1 at 0.1
    Allowent gamma. Do Cookie with alpha in 2 at 0.1
    Joinent gamma. Done with beta in 1 at 0.1
    Allowent gamma. Do Cookie with beta in 2 at 0.1
    Allower alpha. Do Initiate with beta in 2 at 0.2
    Allower alpha. Do Initiate with gamma in 2 at 0.2
    Allower beta. Do Initiate with gamma in 2 at 0.2
    Allowent beta. Do Ack Initiate with alpha in 2 at 0.2
    Allowent gamma. Do Ack Initiate with alpha in 2 at 0.2
    Allowent gamma. Do Ack Initiate with beta in 2 at 0.2
    Allower alpha. Do Ack Final, Done with beta in 2 at 0.3
    Aliver alpha. Do Alive with beta in 3 at 0.3
    Allower alpha. Do Ack Final, Done with gamma in 2 at 0.3
    Aliver alpha. Do Alive with gamma in 3 at 0.3
    Allower beta. Do Ack Final, Done with gamma in 2 at 0.3
    Aliver beta. Do Alive with gamma in 3 at 0.3
    Allowent beta. Done with alpha in 2 at 0.3
    Alivent beta. Do ack alive with alpha in 3 at 0.3
    Alivent beta. Done with alpha in 3 at 0.3
    Allowent gamma. Done with alpha in 2 at 0.3
    Alivent gamma. Do ack alive with alpha in 3 at 0.3
    Alivent gamma. Done with alpha in 3 at 0.3
    Allowent gamma. Done with beta in 2 at 0.3
    Alivent gamma. Do ack alive with beta in 3 at 0.3
    Alivent gamma. Done with beta in 3 at 0.3
    Aliver alpha. Done with beta in 3 at 0.4
    Aliver alpha. Done with gamma in 3 at 0.4
    Aliver beta. Done with gamma in 3 at 0.4
    Perceived service duration = 0.5 seconds
    Finished Handshakes

    Messenger alpha. Do Message Segment 0 with beta in 4 at 0.5
    Messenger alpha. Do Message Segment 0 with gamma in 4 at 0.5
    Messengent beta. Do Ack Done Message on Segment 0 with alpha in 4 at 0.5
    Messengent beta. Complete with alpha in 4 at 0.5
    Messenger beta. Do Message Segment 0 with gamma in 4 at 0.5
    Messenger beta. Do Message Segment 0 with alpha in 1 at 0.5
    Messengent gamma. Do Ack Done Message on Segment 0 with alpha in 4 at 0.5
    Messengent gamma. Complete with alpha in 4 at 0.5
    Messengent gamma. Do Ack Done Message on Segment 0 with beta in 4 at 0.5
    Messengent gamma. Complete with beta in 4 at 0.5
    Messenger gamma. Do Message Segment 0 with alpha in 1 at 0.5
    Messenger gamma. Do Message Segment 0 with beta in 1 at 0.5
    Messenger alpha. Done with beta in 4 at 0.6
    Messengent alpha. Do Ack Done Message on Segment 0 with beta in 1 at 0.6
    Messengent alpha. Complete with beta in 1 at 0.6
    Messenger alpha. Done with gamma in 4 at 0.6
    Messengent alpha. Do Ack Done Message on Segment 0 with gamma in 1 at 0.6
    Messengent alpha. Complete with gamma in 1 at 0.6
    Messenger beta. Done with gamma in 4 at 0.6
    Messengent beta. Do Ack Done Message on Segment 0 with gamma in 1 at 0.6
    Messengent beta. Complete with gamma in 1 at 0.6
    Messenger beta. Done with alpha in 1 at 0.6
    Messenger gamma. Done with alpha in 1 at 0.6
    Messenger gamma. Done with beta in 1 at 0.6
    Perceived service duration = 0.2 seconds
    Finished Messages

    Stack alpha received:

    source = 'beta'.
    msg= {u'content': u'Hello.', u'subject': u'Example message beta to whoever'}

    source = 'gamma'.
    msg= {u'content': u'Good Day', u'subject': u'Example message gamma to whoever'}

    Stack beta received:

    source = 'alpha'.
    msg= {u'content': u'Hi', u'subject': u'Example message alpha to whoever'}

    source = 'gamma'.
    msg= {u'content': u'Good Day', u'subject': u'Example message gamma to whoever'}

    Stack gamma received:

    source = 'alpha'.
    msg= {u'content': u'Hi', u'subject': u'Example message alpha to whoever'}

    source = 'beta'.
    msg= {u'content': u'Hello.', u'subject': u'Example message beta to whoever'}

    Finished


If the console verbosity is set to terse as follows:

.. code-block::

    console.reinit(verbosity=console.Wordage.terse)


The console log outputs the following:


.. code-block::

    Finished Handshakes

    Finished Messages

    Stack alpha received:

    source = 'beta'.
    msg= {u'content': u'Hello.', u'subject': u'Example message beta to whoever'}

    source = 'gamma'.
    msg= {u'content': u'Good Day', u'subject': u'Example message gamma to whoever'}

    Stack beta received:

    source = 'alpha'.
    msg= {u'content': u'Hi', u'subject': u'Example message alpha to whoever'}

    source = 'gamma'.
    msg= {u'content': u'Good Day', u'subject': u'Example message gamma to whoever'}

    Stack gamma received:

    source = 'alpha'.
    msg= {u'content': u'Hi', u'subject': u'Example message alpha to whoever'}

    source = 'beta'.
    msg= {u'content': u'Hello.', u'subject': u'Example message beta to whoever'}

    Finished


-------------
LaneStack
-------------

The setup of a LaneStack is similar to that for a RoadStack. The biggest
difference is that because unix domain (UXD) sockets are reliable, LaneStack messages
do not need to authenticated, encrypted, not transactioned. Consequently
the setup is easier. UXD sockets, however, require a file be created for
the socket.


The following example shows how to create a pair of LandStacks and send
messages between them.


.. code-block::


    import os
    import shutil
    import time
    import tempfile

    import ioflo
    from ioflo.base.consoling import getConsole

    import raet

    console = getConsole()
    console.reinit(verbosity=console.Wordage.concise)

    def serviceStacks(stacks, duration=0.5, period=0.1):
        '''
        Utility method to service queues. Call from test method.
        '''
        store = ioflo.base.storing.Store(stamp=0.0)
        timer = ioflo.base.aiding.StoreTimer(store=store, duration=duration)
        while not timer.expired:
            for stack in stacks:
                stack.serviceAll()
                stack.store.advanceStamp(period)

            store.advanceStamp(period)
            time.sleep(period)

    def example1():
        tempDirpath = tempfile.mkdtemp(prefix="raet", suffix="base", dir='/tmp')
        baseDirpath = os.path.join(tempDirpath, 'lane')

        alpha = raet.lane.stacking.LaneStack(name='alpha',
                                             uid=1,
                                             lanename='zeus',
                                             sockdirpath=baseDirpath)

        beta = raet.lane.stacking.LaneStack(name='beta',
                                            uid=1,
                                            lanename='zeus',
                                            sockdirpath=baseDirpath)

        stacks = [alpha, beta]

        for stack in stacks:
            console.terse("LaneStack '{0}': UXD socket ha = '{1}'\n".format(stack.name, stack.ha))

        remote = raet.lane.yarding.RemoteYard(stack=alpha, ha=beta.ha)
        alpha.addRemote(remote)

        msg = dict(content="Hello this is a message from alpha to beta")
        alpha.transmit(msg, remote.uid)
        serviceStacks(stacks)

        while beta.rxMsgs:
            rxMsg, source = beta.rxMsgs.popleft()
            console.terse("Beta received from {0} message = '{1}'\n".format(source, rxMsg))

        remote = beta.remotes.values()[0]
        console.terse("Beta remotes has '{0}' at '{1}'\n".format(remote.name, remote.ha))

        remote = alpha.remotes.values()[0]
        console.terse("Alpha remotes has '{0}' at '{1}'\n".format(remote.name, remote.ha))

        beta.transmit(dict(content = "Hi from beta"))
        beta.transmit(dict(content = "Hi again from beta"))
        beta.transmit(dict(content = "Hi yet again from beta"))

        alpha.transmit(dict(content = "Hello from alpha"))
        alpha.transmit(dict(content = "Hello again from alpha"))
        alpha.transmit(dict(content = "Hello yet again from alpha"))

        serviceStacks(stacks)

        for stack in stacks:
            while stack.rxMsgs:
                rxMsg, source = stack.rxMsgs.popleft()
                console.terse("LaneStack {0} received from {1} message = '{2}'\n".format(stack.name, source, rxMsg))



        for stack in stacks:
            stack.server.close()

        shutil.rmtree(tempDirpath)

        print("Finished\n")



    if __name__ == "__main__":
        example1()



The console log will output the following

.. code-block::

    LaneStack 'alpha': UXD socket ha = '/tmp/raetHdl6ZRbase/lane/zeus.alpha.uxd'
    LaneStack 'beta': UXD socket ha = '/tmp/raetHdl6ZRbase/lane/zeus.beta.uxd'
    Beta received from alpha message = '{u'content': u'Hello this is a message from alpha to beta'}'
    Beta remotes has 'alpha' at '/tmp/raetHdl6ZRbase/lane/zeus.alpha.uxd'
    Alpha remotes has 'beta' at '/tmp/raetHdl6ZRbase/lane/zeus.beta.uxd'
    LaneStack alpha received from beta message = '{u'content': u'Hi from beta'}'
    LaneStack alpha received from beta message = '{u'content': u'Hi again from beta'}'
    LaneStack alpha received from beta message = '{u'content': u'Hi yet again from beta'}'
    LaneStack beta received from alpha message = '{u'content': u'Hello from alpha'}'
    LaneStack beta received from alpha message = '{u'content': u'Hello again from alpha'}'
    LaneStack beta received from alpha message = '{u'content': u'Hello yet again from alpha'}'
    Finished
