# zodb-s3blobs

Store ZODB blobs in S3-compatible object storage.

## Features

- Wraps any ZODB base storage (FileStorage, RelStorage, MappingStorage, ...)
- Works with any S3-compatible service (AWS S3, MinIO, Ceph, DigitalOcean Spaces)
- Local LRU filesystem cache for fast reads
- Full ZODB two-phase commit integration (transactional safety)
- ZConfig integration for `zope.conf` configuration
- Supports MVCC storages (`new_instance()`)
- Garbage collection of orphaned S3 objects during `pack()`

## Installation

```bash
pip install zodb-s3blobs
```

## Configuration via zope.conf

Add `%import zodb_s3blobs` and use the `<s3blobstorage>` section wrapping any base storage.

### With FileStorage

```xml
%import zodb_s3blobs

<zodb_db main>
    <s3blobstorage>
        bucket-name my-zodb-blobs
        s3-endpoint-url http://minio:9000
        s3-access-key $S3_ACCESS_KEY
        s3-secret-key $S3_SECRET_KEY
        cache-dir /var/cache/zodb-s3-blobs
        cache-size 2GB
        <filestorage>
            path /var/lib/zodb/Data.fs
        </filestorage>
    </s3blobstorage>
</zodb_db>
```

ZConfig expands `$VARIABLE` and `${VARIABLE}` from the process environment.
For production, consider omitting `s3-access-key` and `s3-secret-key` entirely
and relying on the boto3 credential chain (IAM roles, instance profiles,
`~/.aws/credentials`, or the `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`
environment variables).

### With RelStorage

When wrapping RelStorage, `zodb-s3blobs` overrides RelStorage's blob handling.
Blobs go to S3 instead of the `blob_chunk` table. RelStorage still handles object data (pickles) in the RDBMS.

```xml
%import zodb_s3blobs

<zodb_db main>
    <s3blobstorage>
        bucket-name my-zodb-blobs
        cache-dir /var/cache/zodb-s3-blobs
        cache-size 2GB
        <relstorage>
            <postgresql>
                dsn dbname='zodb' user='zodb' host='localhost'
            </postgresql>
        </relstorage>
    </s3blobstorage>
</zodb_db>
```

**Note:** When wrapping RelStorage, `zodb-s3blobs` automatically enables
RelStorage's `LOCK_EARLY` mode (equivalent to `RELSTORAGE_LOCK_EARLY=1`). This
causes the database commit lock to be held slightly longer — through the S3 blob
upload — to ensure transactional consistency. This is the same behavior as
RelStorage 2.x and is required so that blob uploads complete before the
transaction commits.

## Configuration Reference

| Parameter | Default | Description |
|-----------|---------|-------------|
| `bucket-name` | *(required)* | S3 bucket name |
| `s3-prefix` | `""` | Key prefix in bucket |
| `s3-endpoint-url` | `None` | For MinIO, Ceph, etc. |
| `s3-region` | `None` | AWS region |
| `s3-access-key` | `None` | Uses boto3 credential chain if omitted. Use `$ENV_VAR` substitution — never hardcode credentials. |
| `s3-secret-key` | `None` | Uses boto3 credential chain if omitted. Use `$ENV_VAR` substitution — never hardcode credentials. |
| `s3-use-ssl` | `true` | Whether to use SSL for S3 connections |
| `s3-addressing-style` | `auto` | S3 addressing style: `path`, `virtual`, or `auto` |
| `s3-sse-customer-key` | `None` | Base64-encoded 256-bit key for SSE-C encryption. Requires SSL. |
| `cache-dir` | *(required)* | Local cache directory path |
| `cache-size` | `1GB` | Maximum local cache size |

## How It Works

`zodb-s3blobs` uses the same proxy/wrapper pattern as ZODB's built-in `BlobStorage`. It wraps any base storage via `__getattr__` and explicitly overrides all blob methods so they always take precedence.

### Two-Phase Commit Flow

1. **`storeBlob`**: Object data (pickle) is stored in the base storage. The blob file is staged locally.
2. **`tpc_vote`**: Staged blobs are uploaded to S3. If any upload fails, the transaction aborts cleanly.
3. **`tpc_finish`**: No S3 operations (this method must not fail per ZODB contract). Staged files are moved into the local cache.
4. **`tpc_abort`**: Uploaded S3 objects are deleted (best-effort). Local staged files are cleaned up.

### S3 Key Layout

```
blobs/{oid_hex}/{tid_hex}.blob
```

With a configured prefix: `{prefix}/blobs/{oid_hex}/{tid_hex}.blob`

### Local Cache

The local filesystem cache provides fast reads after the first access. It uses LRU eviction with a background daemon thread that removes the oldest files (by access time) when the total size exceeds the configured maximum. The cache is required -- S3 latency makes direct access impractical for ZODB's synchronous access patterns.

### Garbage Collection

During `pack()`, the base storage is packed first, then S3 is scanned for keys referencing OIDs that are no longer reachable. Orphaned keys are deleted. This also cleans up any objects left behind by failed abort operations.

### S3 Bucket Security

Ensure your S3 bucket has appropriate access controls (Block Public Access enabled, restrictive bucket policy). The minimum IAM policy required by `zodb-s3blobs`:

```json
{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": [
            "s3:GetObject",
            "s3:PutObject",
            "s3:DeleteObject",
            "s3:ListBucket"
        ],
        "Resource": [
            "arn:aws:s3:::BUCKET_NAME",
            "arn:aws:s3:::BUCKET_NAME/*"
        ]
    }]
}
```

### Encryption at Rest (SSE-C)

`zodb-s3blobs` supports SSE-C (Server-Side Encryption with Customer-Provided Keys).
The S3 service encrypts/decrypts data using your key but never stores it.
Works with AWS S3, Hetzner Object Storage, MinIO (with KES), and other S3-compatible services.

> **Warning — AWS SSE-C deprecation (April 2026):** AWS will disable SSE-C by default
> on new S3 buckets starting April 2026. Existing buckets are unaffected. For new buckets,
> you must explicitly enable SSE-C in the bucket policy, or consider migrating to SSE-KMS.
> If you receive 403 errors with SSE-C configured, this is the likely cause.
> See the [AWS announcement](https://aws.amazon.com/blogs/storage/advanced-notice-amazon-s3-to-disable-the-use-of-sse-c-encryption-by-default-for-all-new-buckets-and-select-existing-buckets-in-april-2026/) for details.

Generate a 256-bit key:

```bash
python -c "import base64, os; print(base64.b64encode(os.urandom(32)).decode())"
```

Configure via environment variable:

```xml
s3-sse-customer-key $S3_SSE_KEY
```

**Important:** If you lose the key, encrypted data is irrecoverable. SSL is required (enforced at startup).

**Security note:** The SSE-C key is held in process memory for the lifetime of the storage instance. In long-running servers, consider using IAM-based encryption (SSE-KMS) instead if memory exposure is a concern. Python's string handling makes secure memory clearing impractical.

## Using with MinIO (dev setup)

**Warning:** The credentials below are MinIO defaults for local development only. Never use default credentials in production.

```yaml
# docker-compose.yml
services:
  minio:
    image: minio/minio
    command: server /data --console-address ":9001"
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
```

Create the bucket:

```bash
mc alias set local http://localhost:9000 minioadmin minioadmin
mc mb local/zodb-blobs
```

## Development

```bash
git clone https://github.com/bluedynamics/zodb-s3blobs.git
cd zodb-s3blobs
uv venv
uv pip install -e ".[test]"
pytest
```

For **reproducible deployments** (production), pin dependencies with a lockfile:

```bash
uv pip compile pyproject.toml -o requirements.txt
uv pip install -r requirements.txt
```

## Source Code and Contributions

The source code is managed in a Git repository, with its main branches hosted on GitHub.
Issues can be reported there too.

We'd be happy to see many forks and pull requests to make this package even better.
We welcome AI-assisted contributions, but expect every contributor to fully understand and be able to explain the code they submit.
Please don't send bulk auto-generated pull requests.

Maintainers are Jens Klein and the BlueDynamics Alliance developer team.
We appreciate any contribution and if a release on PyPI is needed, please just contact one of us.
We also offer commercial support if any training, coaching, integration or adaptations are needed.

## License

ZPL-2.1
