Metadata-Version: 2.4
Name: sdpk
Version: 0.0.2
Summary: Small DPK - an extended fork of Data Preparation Toolkit Library for Ray and Python
Author-email: Boris Lublinsky <blublinsky@hotmail.com>
License: Apache-2.0
Keywords: data,data preprocessing,data preparation,llm,generative,ai
Requires-Python: <3.13,>=3.10
Description-Content-Type: text/markdown
Requires-Dist: numpy<1.29.0
Requires-Dist: pyarrow==16.1.0
Requires-Dist: boto3==1.34.69
Requires-Dist: argparse
Requires-Dist: mmh3
Requires-Dist: psutil
Requires-Dist: polars
Requires-Dist: huggingface-hub>=0.25.2
Requires-Dist: ray[default]>=2.36.1
Requires-Dist: fastapi>=0.110.2
Requires-Dist: pillow>=10.3.0
Requires-Dist: twine
Requires-Dist: pytest>=7.3.2
Requires-Dist: pytest-dotenv>=0.5.2
Requires-Dist: pytest-env>=1.0.0
Requires-Dist: pre-commit>=3.3.2
Requires-Dist: pytest-cov>=4.1.0
Requires-Dist: pytest-mock>=3.10.0
Requires-Dist: ruff
Requires-Dist: moto==5.0.5
Requires-Dist: markupsafe==2.0.1

# Data Processing Library
This provides a python framework for developing _transforms_
on data stored in files and running them in a pure Python or [ray](https://www.ray.io/) cluster.
Data files may be stored in the local file system or  COS/S3.
For more details see the [documentation](../README.md).

## Library Artifact Build and Publish
To build the library
```shell
make build_dist
```
To publish it to PyPi
```shell
make publish
```

To up the version number, edit the Makefile to change VERSION and rerun
the above.  This will require committing both the `Makefile` and the
autotmatically updated `pyproject.toml` file.



