:orphan:

===================
Storlets Invocation
===================

Once the storlet and its dependencies are deployed the storlet is ready for invocation.
Storlets can be invoked in 2 ways:

1. Invocation upon GET. In this case the user gets a transformation of the object residing in the store (as opposed to the actual object). Typical use case for GET is anonimization, where the user might not have access to a certain data unless it is being anonymized by some storlet.
2. Invocation upon PUT. In this case the data kept in the store is a transformation of the object uploaded by the user (as opposed to the actual uploaded data or metadata). A typical use case is metadata enrichment, where a Storlet extracts format specific metadata from the uploaded data and adds it as Swift metadata.

Invocation involves adding an extra header to the Swift original
PUT/GET requests. Following our Identity Storlet example given in <https://opendev.org/openstack/storlets/blob/master/doc/source/writing_and_deploying_storlets.rst>, here are invocation examples. This time the examples make use of the python swift client.

Invocation on GET
=================

The code below shows the invocation. Some notes:

#. There are invocations with and without a parameter controlling whether the
   get42 binary dependency is to be called. Note the difference in the response
   headers where one shows the execution result and the other does not.
#. Note the X-Run-Storlet header. being added to the call.
#. Note the X-Generate-Log storlet that causes a log file to be created.
   The execution results below show the log retrieval.

::

    from swiftclient import client as c
    
    def get_processed_object(url, token, storlet_name, container_name, object_name, invoke_get42 = False):
        headers = {'X-Run-Storlet': storlet_name,
                   'X-Storlet-Generate-Log' : 'True'}
        if (invoke_get42 == True):
            querystring = 'execute=true'
        else:
            querystring = None
    
        response_headers, object_content = c.get_object(url,
                                              token,
                                              container_name,
                                              object_name,
                                              query_string = querystring,
                                              response_dict=dict(),
                                              headers = headers)
        print response_headers
        print object_content
    
    AUTH_IP = '127.0.0.1'
    AUTH_PORT = '5000'
    ACCOUNT = 'service'
    USER_NAME = 'swift'
    PASSWORD = 'passw0rd'
    os_options = {'tenant_name': ACCOUNT}
    
    url, token = c.get_auth("http://" + AUTH_IP + ":" + AUTH_PORT + "/v2.0", ACCOUNT +":"+USER_NAME, PASSWORD, os_options = os_options, auth_version="2.0")
    print 'Identity Storlet invocation without calling get42'
    get_processed_object(url, token, 'identitystorlet-1.0.jar', 'myobjects', 'source.txt')
    print 'Identity Storlet invocation instructing to call get42'
    get_processed_object(url, token, 'identitystorlet-1.0.jar', 'myobjects', 'source.txt', True)



Here is the result of the running the above python script:

::

    eranr@lnx-ccs8:/tmp$ python get_object_with_storlet.py
    Identity Storlet invocation without calling get42
    {
        'x-object-meta-x-object-meta-testkey': 'tester', 
        'transfer-encoding': 'chunked', 
        'accept-ranges': 'bytes', 
        'x-object-meta-testkey': 'tester', 
        'last-modified': 'Tue, 30 Sep 2014 22:07:42 GMT', 
        'etag': '8ca2a24dbd9779d462c66866c0fb90c3', 
        'x-timestamp': '1412114861.90504', 
        'x-trans-id': 'tx464a488a618e44b5b763d-00542baa25', 
        'date': 'Wed, 01 Oct 2014 07:15:50 GMT', 
        'x-object-meta-type': 'SBUS_FD_INPUT_OBJECT', 
        'content-type': 'application/octet-stream'
    }
    Some content to copy
    Identity Storlet invocation instructing to call get42
    {
        'x-object-meta-execution result': '42', 
        'x-object-meta-x-object-meta-testkey': 'tester', 
        'transfer-encoding': 'chunked', 
        'accept-ranges': 'bytes', 
        'x-object-meta-testkey': 'tester', 
        'last-modified': 'Tue, 30 Sep 2014 22:07:42 GMT', 
        'etag': '8ca2a24dbd9779d462c66866c0fb90c3', 
        'x-timestamp': '1412114861.90504', 
        'x-trans-id': 'tx12a4f2a168804dcabf8fc-00542baa26', 
        'date': 'Wed, 01 Oct 2014 07:15:50 GMT', 
        'x-object-meta-type': 'SBUS_FD_INPUT_OBJECT', 
        'content-type': 'application/octet-stream'
    }
    Some content to copy

We now show a download of the log file generated per the X-Storlet-Generate-Log header.
Again, we use the swift client assuming we have the appropriate environment variables in place.

Note that the log reflects the two invocations done above.

::

    eranr@lnx-ccs8:/tmp$ swift download storletlog identitystorlet.log
    identitystorlet.log [headers 0.243s, total 0.243s, 0.001 MB/s]
    eranr@lnx-ccs8:/tmp$ cat identitystorlet.log
    About to invoke storlet
    IdentityStorlet Invoked
    Storlet invocation done
    About to invoke storlet
    IdentityStorlet Invoked
    Exec = /home/swift/identitystorlet/get42
    Exit code = 42
    Storlet invocation done

Invocation on PUT
=================

the code below shows the invocation. Some notes:

#. As with the GET example there are invocations with and without a parameter controlling whether the get42 binary dependency is to be called. After each put we do a GET and print the response headers to show the difference between the invocations. See below.
#. As with the GET example we add the X-Run-Storlet header.
#. This time we do not add the X-Generate-Log header, which is the recommended way, as it saves a creation of an object.

::

    import random
    import string
    from swiftclient import client as c
    
    def put_processed_object(url, token, storlet_name, container_name, object_name, file_name_to_upload, invoke_get42 = False):
        headers = {'X-Run-Storlet': storlet_name,
                   'X-Storlet-Generate-Log' : 'True'}
        if (invoke_get42 == True):
            querystring = 'execute=true'
        else:
            querystring = None

        fileobj = open(file_name_to_upload,'rb')
        random_md = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(32))
        headers = {'X-Run-Storlet': 'identitystorlet-1.0.jar', 'X-Object-Meta-Testkey' : random_md}
        c.put_object(url,
                     token,
                     container_name,
                     object_name,
                     fileobj,
                     headers = headers,
                     query_string = querystring,
                     response_dict=dict())
    
        resp_headers, saved_content = c.get_object(
                    url,
                    token,
                    container_name,
                    object_name,
                    response_dict=dict())
        print resp_headers
    
    AUTH_IP = '127.0.0.1'
    AUTH_PORT = '5000'
    ACCOUNT = 'service'
    USER_NAME = 'swift'
    PASSWORD = 'passw0rd'
    os_options = {'tenant_name': ACCOUNT}
    
    url, token = c.get_auth("http://" + AUTH_IP + ":" + AUTH_PORT + "/v2.0", ACCOUNT +":"+USER_NAME, PASSWORD, os_options = os_options, auth_version="2.0")
    print 'Identity Storlet invocation without calling get42'
    put_processed_object(url, token, 'identitystorlet-1.0.jar', 'myobjects', 'source.txt', '/tmp/source.txt')
    print 'Identity Storlet invocation instructing to call get42'
    put_processed_object(url, token, 'identitystorlet-1.0.jar', 'myobjects', 'source.txt', '/tmp/source.txt' , True)



Here is the result of the running the above python script:

::

    eranr@lnx-ccs8:/tmp$ python put_object_with_storlet.py
    Identity Storlet invocation without calling get42
    {
        'content-length': '1024', 
        'x-object-meta-x-object-meta-testkey': '1185FZ5FPQ1WXS9IDT4TZZB6GYAQQ0WL', 
        'accept-ranges': 'bytes', 
        'x-object-meta-testkey': '1185FZ5FPQ1WXS9IDT4TZZB6GYAQQ0WL', 
        'last-modified': 'Wed, 01 Oct 2014 07:48:56 GMT', 
        'etag': '7575c5b098f45ccabce1c3f7fc906eb9', 
        'x-timestamp': '1412149735.87168', 
        'x-trans-id': 'tx9a27ba91bee34a8ca9f0c-00542bb1e7', 
        'date': 'Wed, 01 Oct 2014 07:48:55 GMT', 
        'x-object-meta-type': 'SBUS_FD_INPUT_OBJECT', 
        'content-type': 'text/plain'
    }
    Identity Storlet invocation instructing to call get42
    {
        'x-object-meta-execution result': '42', 
        'content-length': '1024', 
        'x-object-meta-x-object-meta-testkey': '54YA1EDTTODMBUJOYCHEGSOQQPV0180L', // This looks like a bug
        'accept-ranges': 'bytes', 
        'x-object-meta-testkey': '54YA1EDTTODMBUJOYCHEGSOQQPV0180L', 
        'last-modified': 'Wed, 01 Oct 2014 07:48:56 GMT', 
        'etag': '7575c5b098f45ccabce1c3f7fc906eb9', 
        'x-timestamp': '1412149735.97100', 
        'x-trans-id': 'txde8619a966c14b0c99d97-00542bb1e8', 
        'date': 'Wed, 01 Oct 2014 07:48:56 GMT', 
        'x-object-meta-type': 'SBUS_FD_INPUT_OBJECT', 
        'content-type': 'text/plain'
    }
