# Virt-S3 🪣

A Virtualitics utility package to handle file I/O with Object Storage Systems like AWS S3 and Minio. 

With versatility in mind, `virt-s3` was designed to be a relatively lightweight package that can either used independently or in conjunction with the larger Virtualitics AI platform. The `virt-s3` module includes two primary submodules `s3` and `fs` that implement each API function of the `virt-s3` module specific to the target system: either S3/S3-like systems or local file systems.

We hope that you can use it, break it, and even help us improve it!

## Table of Contents
1. [Prerequisites](#Prerequisites)
2. [Example Usage](#Example-Usage)
3. [Architecture](#Architecture)
4. [Getting Started](#Getting-Started)
5. [Documentation](#Code-Documentation)


## Prerequisites
- Requires python>=3.11
- Local File System features currently only support posix `/` pathing (Linux, Mac, etc.)
    * Support for Windows `\` pathing **[Coming Soon]**


## Example Usage

### Writing a File
```python
import virt_s3
import pandas as pd
from io import BytesIO

# ENV variable `Local_FS` = '1' or '0' (local file system or S3)
params = virt_s3.get_default_params()

df = pd.DataFrame([{'a': 1, 'b': 2}])
buffer = BytesIO()
df.to_csv(buffer, index=None)

with virt_s3.SessionManager(params=params) as session:
    virt_s3.create_bucket('test-bucket', params=params, client=session)
    path = f"fixture/data/data.csv"
    saved_key = virt_s3.upload_data(buffer.getbuffer(), path, params=params, client=session)
```

### Reading a File
```python
import virt_s3
import pandas as pd

# ENV variable `Local_FS` = '1' or '0' (Local file system or S3)
params = virt_s3.get_default_params()

# use context manager to manage session scope
with virt_s3.SessionManager(params=params) as session:
    data = virt_s3.get_file(saved_key, bytes_io=True, params=params, client=session)
    df = pd.read_csv(data)
```

## Architecture

`Virt-S3` can be run on a local machine or from within a docker container. Additionally, it includes a variety of ways to interact with Object Storage Systems like AWS S3 and Minio in different hosting environments along with support for local file system access on host machine within docker container.

This versatility along with its lightweight set of dependencies allows `virt-s3` to be easily installed and used in various types of environments.


## Getting Started
1. Create a fresh virtual environment with python >= 3.11

2. Install the necessary dependencies

Basic Install (No Extras)
```bash
$ pip install virt-s3
```

Install with Single Extra
```bash
$ pip install "virt-s3[s3]"
```

Install with Multiple Extras
```bash
$ pip install "virt-s3[s3,dataframe,image]"
```

- **The Following Extras are Available:**
    * **s3** = installs dependencies required to interact with object stores like Minio/S3 (primarily relying on `boto3`)
    * **dataframe** = installs dependencies required for using `numpy`, `pandas`, and `pyarrow` dataframe/parquet operations
    * **image** = installs dependencies required to utilize image operations (e.g. get file as an image)

- **e.g.** If you want to use `virt_s3`, but can't install `pandas` or `pyarrow` in your restricted environment, then you can simply install `virt_s3` without the `dataframe` extra dependencies. You won't be able to use `virt_s3.extras.CSVFileValidator`, `virt_s3.extras.ParquetFileValidator`, `read_parquet_file_df`, and `write_parquet_file_df` but these are also not necessarily core functions of the library (therefore extras).

3. Make sure the following environment variables are set

```.env
#########################################
# Required Custom Environment Variables #
#########################################
LOCAL_FS_USER=<your username>
# use the local fs mirror or s3/minio: 1 = True, 0 = False
LOCAL_FS=0
LOCAL_FS_ROOT_DIR=</path/to/your/data/dir/>

########################################################
# Required Virtualitics Platform Environment Variables #
########################################################
# e.g. http://mock-s3:9000 or http://localhost:9000
S3_URL=<your s3/minio url>
S3_DEFAULT_BUCKET=test-buck<your bucket name>
AWS_SECRET_ACCESS_KEY=<your aws secret access key>
AWS_ACCESS_KEY_ID=<your aws access key id>
# e.g. us-east-1
AWS_REGION=<your aws region>
```

- Note: `S3_URL` can be replaced with a localhost url (e.g. http://localhost:9000) if not being run within a docker container

4. Run the above [example usage](#Example-Usage)

## Code Documentation

- [Full Module Table of Contents](modules.md)
- [Full Module API Specs](virt_s3.md)

| API |  Description  |
|-----|---------------|
|[`PredictConnectionStoreParams`](virt_s3.md#class-virt_s3predictconnectionstoreparamsuser_id-store_name-store_owner)| Dataclass for Predict Connection Store Parameters |
|[`S3Params`](virt_s3.md#class-virt_s3s3paramsbucket_name-endpoint_url-aws_access_key_idnone-aws_secret_access_keynone-region_namenone-profile_namenone-connection_store_paramsnone)| Dataclass for S3 Boto3 Connection Parameters |
|[`SessionManager`](virt_s3.md#class-virt_s3sessionmanagerparamsnone-clientnone-keep_openfalse-session_client_kwargs)| General Session Context Manager for virt_s3 repo |
|[`TransferConfig`](virt_s3.md#class-virt_s3transferconfigmultipart_threshold8388608-max_concurrency10-multipart_chunksize8388608-num_download_attempts5-max_io_queue100-io_chunksize262144-use_threadstrue-max_bandwidthnone-preferred_transfer_clientauto)| [boto3.s3.TransferConfig](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3.html) used to configure higher throughput upload/download functions |
|[`LocalFSParams`](virt_s3.md#class-virt_s3localfsparamsuse_local_fs-root_dir-user-bucket_name)| Dataclass for Using Local File System for all S3 Calls |
|[`ImageFormatType`](virt_s3.md#class-virt_s3imageformattypevalue-namesnone--modulenone-qualnamenone-typenone-start1-boundarynone)| Enum class type for Image Format Types |
|[`get_default_params()`](virt_s3.md#virt_s3get_default_paramsuse_localfalse-use_s3false)| Function to get default parameters to use for all functions (default behavior is based off of ENV variables) |
|[`get_session_client()`](virt_s3.md#virt_s3get_session_clientparamsnone-kwargs)| Function to get session client based on passed in `S3Params` or `LocalFSParams`|
|[`create_bucket()`](virt_s3.md#virt_s3create_bucketbucket_name-paramsnone-clientnone)| Function to create a bucket to read and write from |
|[`get_file_chunked()`](virt_s3.md#virt_s3get_file_chunkedfpath-paramsnone-clientnone-kwargs)| Function to get a file using a chunking loop. This can be useful when trying to retrieve very large files |
|[`get_file()`](virt_s3.md#virt_s3get_filefpath-paramsnone-clientnone-bytes_iofalse-kwargs)| Function to retrieve specified file as in-memory object|
|[`get_image()`](virt_s3.md#virt_s3get_imagefpath-paramsnone-clientnone-img_formatnone-bytes_iofalse-kwargs)| Function to get image from either s3 or local file system |
|[`get_files_generator()`](virt_s3.md#virt_s3get_files_generatorfpath_li-paramsnone-clientnone-bytes_iofalse-kwargs)| Generator function to quickly loop through reading a list of keys or file paths |
|[`get_files_batch()`](virt_s3.md#virt_s3get_files_batchfpath_li-paramsnone-clientnone-max_concurrency0-bytes_iofalse-kwargs)| Function to get list of file paths or key paths in batch |
|[`list_dirs()`](virt_s3.md#virt_s3list_dirsdir_path-paramsnone-clientnone-delimiter-kwargs)| Function to list valid 'folders' within 'bucket'|
|[`get_valid_file_paths()`](virt_s3.md#virt_s3get_valid_file_pathsdir_path-ignoreds_store-filter_extensions-paramsnone-clientnone-kwargs)| Function to get list of valid file paths or keys within particular directory of bucket |
|[`file_exists()`](virt_s3.md#virt_s3file_existsfpath-paramsnone-clientnone-kwargs)| Function to see if key or file path exists in bucket |
|[`upload_data()`](virt_s3.md#virt_s3upload_datadata-fpath-paramsnone-clientnone-kwargs)| Function to upload data to S3 or local file system |
|[`delete_file()`](virt_s3.md#virt_s3delete_filefpath-paramsnone-clientnone-kwargs)| Function to delete a file from s3 or local file system |
|[`delete_files_by_dir()`](virt_s3.md#virt_s3delete_files_by_dirdir_path-paramsnone-clientnone-kwargs)| Function to delete all files and subdirectories, etc. in a given folder |
|[`archive_zip_as_buffer()`](virt_s3.md#virt_s3archive_zip_as_bufferdata_bytes_dict)| Function to create a zip archive from dictionary of expected archive filepaths and data bytes|
|[`extract_compressed_file()`](virt_s3.md#virt_s3extract_compressed_filefpath-extracted_dir_prefixnone-paramsnone-clientnone-kwargs) | Function to extract zip file contents into bucket |
|[`format_bytes()`](virt_s3.md#virt_s3utilsformat_bytesnum_bytes) | Funtion to take as input a number of bytes and return a formatted string for B, KB, MB, GB |
|[`read_parquet_file_df()`](virt_s3.md#virt_s3read_parquet_file_dffpath-paramsnone-clientnone-engineauto-columnsnone-filtersnone-use_nullable_dtypesfalse-download_kwargs)| Convenience function to read parquet file as pandas DataFrame |
|[`write_parquet_file_df()`](virt_s3.md#virt_s3write_parquet_file_dfdf-save_fpath-paramsnone-clientnone-engineauto-compressionnone-indexnone-partition_colsnone-upload_kwargs)| Convenience function to write pandas DataFrame to parquet file |