Metadata-Version: 2.4
Name: ff-storage
Version: 0.2.2
Summary: Fenixflow storage package for database and file operations
Author-email: Ben Moag <dev@fenixflow.com>
Maintainer-email: Fenixflow Team <dev@fenixflow.com>
License: MIT
Project-URL: Homepage, https://fenixflow.com
Project-URL: Repository, https://gitlab.com/fenixflow/fenix-packages
Project-URL: Documentation, https://gitlab.com/fenixflow/fenix-packages/-/tree/main/ff-storage
Project-URL: Bug Tracker, https://gitlab.com/fenixflow/fenix-packages/-/issues
Project-URL: Changelog, https://gitlab.com/fenixflow/fenix-packages/-/blob/main/ff-storage/CHANGELOG.md
Keywords: storage,database,postgresql,mysql,s3,object-storage,file-storage,migrations,fenixflow
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Database
Classifier: Topic :: System :: Filesystems
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: psycopg2-binary>=2.9
Requires-Dist: asyncpg>=0.29
Requires-Dist: pymysql>=1.1
Requires-Dist: mysql-connector-python>=8.0
Requires-Dist: boto3>=1.34
Requires-Dist: azure-storage-blob>=12.19
Requires-Dist: aiofiles>=23.0.0
Requires-Dist: aioboto3>=12.0.0
Provides-Extra: dev
Requires-Dist: pytest>=7.4; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21; extra == "dev"
Requires-Dist: black>=23.0; extra == "dev"
Requires-Dist: ruff>=0.1; extra == "dev"
Requires-Dist: moto[s3]>=5.0.0; extra == "dev"
Requires-Dist: aioboto3>=12.0.0; extra == "dev"

# ff-storage

Database and object storage operations for Fenixflow applications.

## Features

- **Database Connections**: PostgreSQL and MySQL with connection pooling
- **Multi-Database Support**: Consistent API for PostgreSQL and MySQL
- **Migration Management**: Simple SQL file-based migrations
- **Object Storage**: Async local filesystem and S3/S3-compatible storage
- **Streaming Support**: Read/write large files without loading into memory
- **Atomic Operations**: Safe file writes with temp file + rename
- **Metadata Management**: Store and retrieve metadata with objects
- **Base Models**: Dataclass-based models with UUID and timestamp support
- **Query Builder**: SQL query construction utilities

## Installation

```bash
# From GitLab
pip install git+https://gitlab.com/fenixflow/fenix-packages.git@main#subdirectory=ff-storage

# Local development
pip install -e .
```

## Usage

### PostgreSQL Connection

```python
from ff_storage import PostgresPool

# Create connection pool
db = PostgresPool(
    dbname="fenix_db",
    user="fenix",
    password="password",
    host="localhost",
    port=5432,
    pool_size=20
)

# Connect and execute query
db.connect()
results = db.read_query("SELECT * FROM documents WHERE status = %s", {"status": "active"})

# Execute with RETURNING
new_id = db.execute_query(
    "INSERT INTO documents (title) VALUES (%s) RETURNING id",
    {"title": "New Document"}
)

# Return connection to pool
db.close_connection()
```

### MySQL Connection

```python
from ff_storage import MySQLPool

# Create connection pool
db = MySQLPool(
    dbname="fenix_db",
    user="root",
    password="password",
    host="localhost",
    port=3306,
    pool_size=10
)

# Connect and execute query
db.connect()
results = db.read_query("SELECT * FROM documents WHERE status = %s", {"status": "active"})

# Execute INSERT (returns last insert ID)
new_id = db.execute_query(
    "INSERT INTO documents (title) VALUES (%s)",
    {"title": "New Document"}
)

# Check open connections
open_conns = db.get_open_connections()

# Return connection to pool
db.close_connection()
```

### Migrations

```python
from ff_storage.db.migrations import MigrationManager

# Setup migration manager
manager = MigrationManager(db_connection, "./migrations")

# Run all pending migrations
manager.migrate()
```

### Object Storage

```python
from ff_storage import LocalObjectStorage, S3ObjectStorage
import asyncio

async def main():
    # Local filesystem storage
    local = LocalObjectStorage("/var/data/documents")
    await local.write("docs/report.pdf", pdf_bytes, {"content-type": "application/pdf"})
    data = await local.read("docs/report.pdf")
    exists = await local.exists("docs/report.pdf")
    files = await local.list_keys(prefix="docs/")
    
    # S3 storage (AWS or S3-compatible)
    s3 = S3ObjectStorage(
        bucket="fenix-documents",
        region="us-east-1"
    )
    await s3.write("docs/report.pdf", pdf_bytes)
    data = await s3.read("docs/report.pdf")
    
    # Stream large files
    async for chunk in s3.read_stream("large_file.bin", chunk_size=8192):
        process_chunk(chunk)

asyncio.run(main())
```

## Database Classes

### SQL Base Class
Abstract base providing interface for all SQL operations:
- `connect()`: Establish connection
- `read_query()`: Execute SELECT queries
- `execute()`: Execute INSERT/UPDATE/DELETE
- `execute_query()`: Execute with RETURNING
- `execute_many()`: Batch operations
- Transaction management methods

### PostgreSQL
- `Postgres`: Direct connection without pooling
- `PostgresPool`: Connection pooling for production use

### MySQL
- `MySQL`: Direct connection without pooling
- `MySQLPool`: Connection pooling for production use

## Testing

```bash
# Run tests
pytest tests/

# With coverage
pytest --cov=ff_storage tests/
```

## License

Proprietary - Fenixflow Internal Use Only
