Metadata-Version: 2.4
Name: muleline
Version: 0.1.0
Summary: Universal file sync engine. Chunked resumable uploads, SHA-256 dedup, storage backend connectors.
Project-URL: Homepage, https://muleline.win
Project-URL: Repository, https://gitlab.com/ArchonAGI/muleline
Author-email: ArchonAGI <fredwojo@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: backup,chunked,dedup,nas,photo,s3,storage,sync,upload
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: System :: Archiving
Requires-Python: >=3.9
Requires-Dist: fastapi>=0.100
Requires-Dist: pyjwt>=2.0
Requires-Dist: python-multipart>=0.0.5
Provides-Extra: all
Requires-Dist: boto3>=1.28; extra == 'all'
Requires-Dist: dropbox>=12.0; extra == 'all'
Requires-Dist: google-api-python-client>=2.0; extra == 'all'
Requires-Dist: google-auth>=2.0; extra == 'all'
Requires-Dist: paramiko>=3.0; extra == 'all'
Requires-Dist: webdavclient3>=3.14; extra == 'all'
Provides-Extra: clip
Requires-Dist: open-clip-torch>=2.20; extra == 'clip'
Requires-Dist: torch>=2.0; extra == 'clip'
Provides-Extra: dev
Requires-Dist: httpx>=0.24; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.1; extra == 'dev'
Provides-Extra: dropbox
Requires-Dist: dropbox>=12.0; extra == 'dropbox'
Provides-Extra: gdrive
Requires-Dist: google-api-python-client>=2.0; extra == 'gdrive'
Requires-Dist: google-auth>=2.0; extra == 'gdrive'
Provides-Extra: s3
Requires-Dist: boto3>=1.28; extra == 's3'
Provides-Extra: server
Requires-Dist: stripe>=8.0; extra == 'server'
Requires-Dist: uvicorn[standard]>=0.23; extra == 'server'
Provides-Extra: sftp
Requires-Dist: paramiko>=3.0; extra == 'sftp'
Provides-Extra: webdav
Requires-Dist: webdavclient3>=3.14; extra == 'webdav'
Description-Content-Type: text/markdown

# Muleline

**Self-hosted file sync and photo management. One container, SQLite, no nonsense.**

[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.9%2B-blue)](https://python.org)
[![Framework](https://img.shields.io/badge/framework-FastAPI-009688)](https://fastapi.tiangolo.com)
[![GitLab](https://img.shields.io/badge/gitlab-ArchonAGI%2Fmuleline-orange)](https://gitlab.com/ArchonAGI/muleline)

---

## What is Muleline?

Muleline is a self-hosted file sync and photo management engine built for developers who want to own their infrastructure without the operational overhead of Immich, Nextcloud, or PhotoPrism.

Immich requires 4 containers, PostgreSQL, Redis, and a machine learning service before you can upload a single photo. Muleline requires Python and a directory. It ships as a `pip install`, mounts as a FastAPI router in your existing app, or runs standalone. All state lives in SQLite. No Redis, no Postgres, no Docker Compose sprawl.

It also goes where Immich cannot: store files on S3, Cloudflare R2, Google Drive, Dropbox, SFTP, or WebDAV — or implement your own backend in five methods.

---

<!-- screenshot -->
<!-- Add a screenshot here: docs/screenshot.png -->

---

## Key Features

- **Photo timeline with EXIF metadata** — dates, GPS coordinates, dimensions, and MIME type extracted automatically on upload; query and sort by any field
- **Chunked resumable uploads** — TUS-inspired protocol, 5MB chunks; disconnect mid-upload and resume exactly where you left off
- **SHA-256 deduplication** — hash-checked before upload initiates; duplicate files are detected in one round-trip without transferring bytes
- **Automatic thumbnails** — 400px JPEG thumbnails generated and cached on first access; served independently from the source file
- **Multi-device sync with manifest diff** — client sends a list of file hashes, server returns exactly what to upload and what to download; no redundant transfers
- **Device management** — register phones, tablets, desktops; track last sync time and platform per device
- **Sync event log** — every upload, download, and error recorded for auditing and debugging
- **6 storage backends** — local filesystem, S3/R2/MinIO, WebDAV, SFTP, Dropbox, Google Drive (details below)
- **Pluggable custom backend** — 5-method abstract interface; point Muleline at any storage system in under 50 lines
- **Mountable FastAPI router** — 3 lines to add Muleline sync endpoints to an existing app; no separate service, no new port
- **Works without FastAPI** — the engine is a plain Python class; use it directly in scripts, tests, or other frameworks
- **Single SQLite database, zero external dependencies** — WAL mode, no connection pooling setup, no migration tools required
- **MIT licensed** — use it, fork it, ship it in commercial products

---

## Muleline vs. the Alternatives

| | Muleline | Immich | Nextcloud | PhotoPrism |
|---|---|---|---|---|
| Containers required | 1 (or 0) | 4+ | 1 | 1 |
| Database | SQLite | PostgreSQL | MySQL/Postgres | SQLite/MySQL |
| Message queue | none | Redis | none | none |
| Install | `pip install` | Docker Compose | Docker Compose | Docker Compose |
| Mountable in your app | yes | no | no | no |
| Storage backends | 6 + custom | local only | local + plugins | local only |
| Language | Python/FastAPI | TypeScript/NestJS | PHP | Go |
| License | MIT | AGPL-3.0 | AGPL-3.0 | AGPL-3.0 |

---

## Quick Start

```bash
pip install muleline[server]
```

```python
# server.py
from fastapi import FastAPI
from muleline.engine import SyncEngine
from muleline.storage import LocalStorage
from muleline.db import run_migrations
import muleline.router as sync_router

run_migrations()
engine = SyncEngine(storage=LocalStorage("./data/uploads"))
sync_router.init(engine)

app = FastAPI()
app.include_router(sync_router.router)
```

```bash
uvicorn server:app --port 8000
```

All sync endpoints are now live at `http://localhost:8000/sync/`. Visit `/docs` for the auto-generated API reference.

---

## Storage Backends

| Backend | Use case | Install |
|---|---|---|
| `LocalStorage` | Local filesystem, NAS mount, any path | included |
| `S3Storage` | AWS S3, Cloudflare R2, MinIO, Backblaze B2 | `pip install muleline[s3]` |
| `WebDAVStorage` | Synology NAS, Nextcloud, any WebDAV server | `pip install muleline[webdav]` |
| `SFTPStorage` | Any SSH server | `pip install muleline[sftp]` |
| `DropboxStorage` | Dropbox Business, personal | `pip install muleline[dropbox]` |
| `GoogleDriveStorage` | Google Drive, Shared Drives | `pip install muleline[gdrive]` |
| `StorageBackend` (base) | Anything else — implement 5 methods | included |

Install everything at once:

```bash
pip install muleline[all]
```

### S3 / R2 / MinIO

```python
from muleline.storage import S3Storage

storage = S3Storage(
    bucket="my-bucket",
    endpoint_url="https://abc.r2.cloudflarestorage.com",  # omit for AWS
    access_key="...",
    secret_key="...",
)
```

### Custom Backend

```python
from muleline.storage.base import StorageBackend

class MyNASStorage(StorageBackend):
    def save(self, key: str, content: bytes, mime_type: str = "") -> None: ...
    def read(self, key: str) -> bytes | None: ...
    def delete(self, key: str) -> None: ...
    def exists(self, key: str) -> bool: ...
    def url(self, key: str, expires: int = 3600) -> str: ...
```

---

## API Reference

All endpoints are mounted under `/sync/`. Authentication is handled by your middleware — Muleline reads `request.state.user` (dict with `id` or `sub`) or `request.state.owner_id`.

### Upload

| Method | Path | Description |
|---|---|---|
| POST | `/sync/upload/init` | Start a chunked upload session. Returns `upload_id`, `chunk_size`, `total_chunks`. Dedup check runs here — if the hash already exists, no upload is needed. |
| POST | `/sync/upload/{id}/chunk` | Upload one chunk. Body is `multipart/form-data` with `chunk_index` and `chunk` file. |
| GET | `/sync/upload/{id}/status` | Check progress. Returns `received_chunks`, `total_chunks`, list of received indexes. |
| POST | `/sync/upload/{id}/complete` | Assemble chunks, verify SHA-256, extract EXIF, persist to storage. |
| DELETE | `/sync/upload/{id}` | Abort and clean up chunk temp files. |

### Files

| Method | Path | Description |
|---|---|---|
| GET | `/sync/files` | List files. Query params: `sort` (newest/oldest/name), `mime` (filter prefix), `limit`, `offset`. |
| GET | `/sync/files/{id}` | Get file metadata including EXIF, GPS, dimensions. |
| GET | `/sync/files/{id}/serve` | Serve the raw file with correct MIME type. |
| GET | `/sync/files/{id}/thumb` | Serve 400px JPEG thumbnail (generated and cached on first access). |
| DELETE | `/sync/files/{id}` | Delete file from database and storage backend. |

### Devices & Sync

| Method | Path | Description |
|---|---|---|
| POST | `/sync/devices/register` | Register or update a device (upsert on `device_id`). |
| GET | `/sync/devices` | List all devices for the current user. |
| POST | `/sync/manifest` | Send client file hashes, get back `to_upload` and `to_download` arrays. |
| POST | `/sync/log` | Record a sync event (upload, download, error) with byte count. |
| GET | `/sync/status` | Sync overview: devices, recent activity, total files and bytes. |
| POST | `/sync/cleanup` | Remove expired pending uploads and their temp files. |
| GET | `/sync/server/info` | Server version, uptime, total files, storage backend type. |

---

## Configuration

Muleline reads from environment variables at startup.

| Variable | Default | Description |
|---|---|---|
| `MULELINE_DB_PATH` | `./data/muleline.db` | SQLite database file path |
| `MULELINE_CHUNK_DIR` | `./data/chunks` | Temporary directory for upload chunks |
| `MULELINE_THUMB_DIR` | `./data/thumbs` | Thumbnail cache directory |
| `MULELINE_CHUNK_SIZE` | `5242880` (5MB) | Chunk size in bytes |
| `MULELINE_MAX_UPLOAD_SIZE` | `524288000` (500MB) | Maximum file size per upload |
| `MULELINE_PENDING_TTL` | `86400` (24h) | Seconds before incomplete uploads expire |
| `MULELINE_MAX_PENDING` | `10` | Max concurrent uploads per user |

---

## Architecture

**Sync flow, from client to storage:**

1. Client hashes all local files (SHA-256).
2. Client calls `POST /sync/manifest` with the hash list.
3. Server diffs against its own database and returns two lists: what to upload, what to download.
4. For each file to upload, client calls `POST /sync/upload/init`. If the hash already exists for that user, the server returns immediately — zero bytes transferred.
5. Client splits the file into 5MB chunks and POSTs each to `/sync/upload/{id}/chunk`. Order does not matter; any chunk can be retried independently.
6. Client calls `POST /sync/upload/{id}/complete`. Server assembles chunks in order, verifies the full-file SHA-256, extracts EXIF metadata, and writes to the configured storage backend.
7. Thumbnails are generated lazily on first `/thumb` request and cached to disk.

**State management:**

All state lives in one SQLite file (`muleline.db`) with five tables: `files`, `pending_uploads`, `devices`, `sync_log`, and `migrations`. No ORM, no migration framework — raw SQL with WAL mode enabled.

**Embedding in your app:**

```python
# Your existing FastAPI app
app = FastAPI()

# Add Muleline in 3 lines
run_migrations()
sync_router.init(SyncEngine(storage=LocalStorage("./uploads")))
app.include_router(sync_router.router, prefix="/media")
```

Muleline does not own the app. It is a module in yours.

---

## Roadmap

- Mobile app — iOS and Android clients with background sync
- Desktop sync client — watched folder, automatic upload on change
- Albums and sharing — named collections, public share links with optional expiry
- Web UI — dark-themed browser interface for browsing and managing files
- Face recognition — cluster photos by person using local ML (no cloud)
- Map view — browse photos by GPS location
- End-to-end encryption — client-side encryption before upload; server stores ciphertext only
- Multi-user support — tenant isolation, admin panel
- PyPI release — `pip install muleline` from the public registry

---

## Contributing

Muleline is MIT licensed and open to contributions. The codebase is small by design — 16 files, roughly 1,100 lines. If you can read Python you can understand the whole thing in an afternoon.

To run locally:

```bash
git clone https://gitlab.com/ArchonAGI/muleline
cd muleline
pip install -e ".[dev,server]"
python example.py          # runs on port 8510
pytest                     # run the test suite
```

Open an issue before a large PR. Small focused changes merge fastest.

---

## License

MIT. See [LICENSE](LICENSE).

Copyright (c) 2026 ArchonAGI
