Adapting your components to the pylm registry — pylm 0.14.1 documentation

# Adapting your components to the pylm registry

If you plan to run moderately complex clusters of PALM components, you have to take a look at the pylm registry. The registry is a centralized service that manages the configuration, the execution and the monitoring of components.

The central registry is a web service that stores the following things:

- The configuration file of the cluster
- The status of the configuration of the cluster, useful to check if enough servers have been added to it.
- The output of all the components that were launched with the runner, a script provided by the registry.

To use the capabilities of the registry you have to turn your components in executable scripts in a particular way. Despite you can force the runner to run almost anything, we recommend you to follow these simple guidelines.

- Turn each component into an executable script. It may be the usual python script starting with the shebang or an entry point in your `setup.py`.
- Use `argparse.ArgumentParser` to let the script get the runtime arguments
- To allow testing your script, it is a good practice to define a main function that then is called with the usual `if __name__...`.

What follows is a simple example that adapts the components of a simple parallel server that can be found here (Simple parallel server and client communication).

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    from pylm.servers import Master
    from argparse import ArgumentParser

    def parse_arguments():
        parser = ArgumentParser()
        parser.add_argument('--name', type=str,
                            help="Name of the component", required=True)
        parser.add_argument('--pull', type=str,
                            help="Tcp address of the pull service",
                            default='tcp://127.0.0.1:5555')
        parser.add_argument('--pub', type=str,
                            help="Tcp address of the pub service",
                            default='tcp://127.0.0.1:5556')
        parser.add_argument('--wpush', type=str,
                            help="Tcp address of the push-to-workers service",
                            default='tcp://127.0.0.1:5557')
        parser.add_argument('--wpull', type=str,
                            help="Tcp address of the pull-from-workers service",
                            default='tcp://127.0.0.1:5558')
        parser.add_argument('--db', type=str,
                            help="Tcp address of the cache service",
                            default='tcp://127.0.0.1:5559')

        return parser.parse_args()

    def main():
        args = parse_arguments()
        server = Master(name=args.name,
                        pull_address=args.pull,
                        pub_address=args.pub,
                        worker_pull_address=args.wpull,
                        worker_push_address=args.wpush,
                        db_address=args.db)
        server.start()

    if __name__ == '__main__':
        main()

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    from pylm.servers import Worker
    from argparse import ArgumentParser
    from uuid import uuid4

    class MyWorker(Worker):
        def foo(self, message):
            return self.name.encode('utf-8') + b' processed' + message

    def parse_arguments():
        parser = ArgumentParser()
        parser.add_argument('--name', type=str, help='Name of this worker '
                                                     'component',
                            default=str(uuid4()))
        parser.add_argument('--db', type=str,
                            help='Address for the db socket of the master '
                                 'component',
                            default='tcp://127.0.0.1:5559')

        return parser.parse_args()

    def main():
        args = parse_arguments()
        server = MyWorker(args.name, args.db)
        server.start()

    if __name__ == '__main__':
        main()

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    from pylm.clients import Client
    from itertools import repeat
    from argparse import ArgumentParser

    def parse_arguments():
        parser = ArgumentParser()
        parser.add_argument('--server', type=str,
                            help="Name of the component you want to connect to",
                            required=True)
        parser.add_argument('--function', type=str,
                            help="Name of the function you want to call",
                            required=True)
        parser.add_argument('--db', type=str,
                            help="tcp address of the cache service of the master "
                                 "component",
                            default='tcp://127.0.0.1:5559')
        return parser.parse_args()

    def main():
        args = parse_arguments()
        client = Client(args.server, args.db)

        for response in client.job('.'.join([args.server, args.function]),
                                   repeat(b' a message', 10),
                                   messages=10):
            print(response)

    if __name__ == '__main__':
        main()

From the testing point of view, there is little difference on how to run the master:

    $> python master.py --name foo
    2017-02-01 10:11:41,485 - root - INFO - Starting the router
    2017-02-01 10:11:41,485 - root - INFO - Starting inbound part Pull
    2017-02-01 10:11:41,485 - root - INFO - Starting inbound part WorkerPull
    2017-02-01 10:11:41,485 - root - INFO - Starting outbound part WorkerPush
    2017-02-01 10:11:41,485 - root - INFO - Starting outbound part Pub
    2017-02-01 10:11:41,485 - root - INFO - Starting bypass part Cache
    2017-02-01 10:11:41,485 - root - INFO - Launch router
    2017-02-01 10:11:41,485 - root - INFO - Inbound Pull connects to WorkerPush
    2017-02-01 10:11:41,486 - root - INFO - b'Pull' successfully started
    2017-02-01 10:11:41,486 - root - INFO - Inbound WorkerPull connects to Pub
    2017-02-01 10:11:41,486 - root - INFO - b'WorkerPull' successfully started
    2017-02-01 10:11:41,486 - root - INFO - b'WorkerPush' successfully started
    2017-02-01 10:11:41,487 - root - INFO - Outbound WorkerPush connects to exterior
    2017-02-01 10:11:41,487 - root - INFO - b'Pub' successfully started
    2017-02-01 10:11:41,488 - root - INFO - Outbound Pub connects to exterior

The worker:

    $> python worker.py
    2017-02-01 10:12:16,674 - e29029... - INFO - Got worker push address: ...
    2017-02-01 10:12:16,674 - e29029... - INFO - Got worker pull address: ...

And the client in the form of a launcher:

    $> python launcher.py --server test --function foo
    2017-02-01 10:12:18,394 - INFO - Fetching configuration from the server
    2017-02-01 10:12:18,394 - INFO - CLIENT 29796938-e3d7-4f9a-b69b...
    2017-02-01 10:12:18,395 - INFO - CLIENT 29796938-e3d7-4f9a-b69b...
    b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
    b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
    b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
    b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
    b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
    b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
    b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
    b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
    b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'
    b'e29029a3-6943-4797-a8c2-6005134d8228 processed a message'

With the addition that now the components are ready to be run with the registry.