Metadata-Version: 2.1
Name: custom-image-builder
Version: 1.0.0
Summary: A python package that enables user to build their custom singularity image on HPC cluster
Author: ritwik-deshpande
Author-email: ritwikdeshpande01@gmail.com
Requires-Python: >=3.9,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: Jinja2 (==3.1.2)
Requires-Dist: MarkupSafe (==2.1.3)
Requires-Dist: PyJWT (==2.8.0)
Requires-Dist: certifi (==2023.7.22)
Requires-Dist: cffi (==1.15.1)
Requires-Dist: charset-normalizer (==3.2.0)
Requires-Dist: cryptography (==41.0.3)
Requires-Dist: globus-compute-common (==0.2.0)
Requires-Dist: globus-compute-sdk (==2.3.2)
Requires-Dist: globus-sdk (==3.27.0)
Requires-Dist: idna (==3.4)
Requires-Dist: packaging (==23.1)
Requires-Dist: pika (==1.3.2)
Requires-Dist: pycparser (==2.21)
Requires-Dist: pydantic (==1.10.12)
Requires-Dist: requests (==2.31.0)
Requires-Dist: tblib (==1.7.0)
Requires-Dist: texttable (==1.6.7)
Requires-Dist: typing_extensions (==4.7.1)
Requires-Dist: urllib3 (==2.0.4)
Requires-Dist: websockets (==10.3)
Description-Content-Type: text/markdown

# Building a singular container for HPC using globus-compute

## Context
* One of the executions configurations of [globus-compute](https://www.globus.org/compute) requires a registered container which is spun up to execute the user function on the HPC.

* HPCs do not run docker containers(due to security reasons as discussed [here](https://docs.vscentrum.be/software/singularity.html)) and support only an apptainer/singularity image.

* Installing the apptainer setup to build the singularity image locally is not a straightforward process especially on windows and mac systems as discussed in the [documentation](https://apptainer.org/docs/admin/main/installation.html).

Using this python library the user can specify their custom image specification to build an apptainer/singularity image 
which would be used to in-turn to run their functions on globus-compute. The library registers the container and 
returns the container id which would be used by the globus-compute executor to execute the user function.


## Prerequisite.
A [globus-compute-endpoint](https://globus-compute.readthedocs.io/en/latest/endpoints.html) setup on HPC cluster. 



## Example

Consider the following use-case where the user wants to execute a pandas operation on HPC using globus-compute.
They need a singularity image which would be used by the globus-compute executor. The library can be leveraged as follows:
```python
from custom_image_builder import build_and_register_container

tutorial_endpoint = "01e21ddf-6eb4-41db-8e1d-2bcfe0c8314f"
container_id = build_and_register_container(endpoint_id=tutorial_endpoint,
                                            image_file_name="my-test-image", 
                                            base_image_type="docker", 
                                            base_image="python:3.8",
                                            pip_packages=["pandas"])

print("Container id ", container_id)

from globus_compute_sdk import Executor

# User function runs on the HPC node
def transform():
    import pandas as pd
    data = {
        'City': ['New York', 'San Francisco', 'Los Angeles']
    }
    return pd.DataFrame(data)


with Executor(endpoint_id=tutorial_endpoint,
              container_id=container_id) as ex:
    fut = ex.submit(transform)
    

print(fut.result())

```

