Turning a PALM master into a microservice — pylm 0.14.1 documentation

# Turning a PALM master into a microservice

The low level API also includes parts that can be used to turn a master server into a more classical microserver in the form of an HTTP server. The goal would be to offer a gateway to a PALM cluster with the HTTP protocol, The master (and now microservice too) can connect to as many workers as it is needed, just like a Master or a Hub, while serving to several HTTP clients.

Note

One caveat. The HttpGateway part spawns a thread for every client connection so don’t rely on it for dealing with thousands of concurrent connections.

The components are the `pylm.parts.gateways.GatewayRouter`, `pylm.parts.gateways.GatewayDealer` and `pylm.parts.gateways.HttpGateway`. They can be used in the following fashion to wire a master to listen to an HTTP connection, that is served from the HttpGateway part.

The whole example can be implemented as follows.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    from pylm.parts.servers import ServerTemplate
    from pylm.parts.services import WorkerPullService, WorkerPushService, \
        CacheService
    from pylm.parts.gateways import GatewayDealer, GatewayRouter, HttpGateway

    server = ServerTemplate()

    worker_pull_address = 'tcp://127.0.0.1:5557'
    worker_push_address = 'tcp://127.0.0.1:5558'
    db_address = 'tcp://127.0.0.1:5559'

    server.register_inbound(GatewayRouter,
                            'gateway_router',
                            'inproc://gateway_router',
                            route='WorkerPush')
    server.register_outbound(GatewayDealer,
                             'gateway_dealer',
                             listen_address='inproc://gateway_router')
    server.register_bypass(HttpGateway,
                           name='HttpGateway',
                           listen_address='inproc://gateway_router',
                           hostname='localhost',
                           port=8888)
    server.register_inbound(WorkerPullService, 'WorkerPull', worker_pull_address,
                            route='gateway_dealer')
    server.register_outbound(WorkerPushService, 'WorkerPush', worker_push_address)
    server.register_bypass(CacheService, 'Cache', db_address)

    server.preset_cache(name='server',
                        db_address=db_address,
                        worker_pull_address=worker_pull_address,
                        worker_push_address=worker_push_address)

    if __name__ == '__main__':
        server.start()

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    import sys

    from pylm.servers import Worker

    class MyWorker(Worker):
        def function(self, message):
            return b'acknowledged'

    server = MyWorker(sys.argv[1], db_address='tcp://127.0.0.1:5559')

    if __name__ == '__main__':
        server.start()

    1
    2
    3
    import requests

    print(requests.get('http://localhost:8888/function').content)

Note that the client is calling the path `/function` of the server, that is mapped to the `function` method of the worker. This means that the body of the HTTP message is precisely the message you wan to send down the pipeline.

In this example, the GatewayDealer only pipes the output of the workers back to the GatewayRouter and the HttpGateway, but remember that every outbound component has a `route` argument that allows you to multiplex the output stream.