---
id: bulk-upload
title: Bulk Upload
sidebar_position: 5
---

# Bulk Upload

High-performance file upload using presigned URLs for direct-to-storage transfers.

## Overview

Bulk upload bypasses the API server for file data, uploading directly to cloud storage (S3, GCS, Azure) using presigned URLs. This dramatically improves performance for uploading thousands of files.

**Performance comparison:**

| Scenario | Standard Upload | Bulk Upload |
|----------|-----------------|-------------|
| 1000 x 5MB files | ~10-15 min | ~1-2 min |
| 5000 x 2MB files | ~30+ min | ~3-5 min |
| 10000 x 1MB files | ~45+ min | ~5-8 min |

---

## How It Works

```
SDK                              API                         Storage
 |                                |                                |
 |-- POST /bulk_upload/           |                                |
 |   presigned_upload/            |                                |
 |   [{filename, size}, ...]  -->|                                |
 |                                |                                |
 |<-- [{presigned_url,            |                                |
 |      file_key}, ...]  --------|                                |
 |                                |                                |
 |=== PUT directly (parallel) ===============================>|
 |=== PUT directly (parallel) ===============================>|
 |                                |                                |
 |-- POST /bulk_upload/           |                                |
 |   confirm_upload/              |                                |
 |   [{file_key, checksum}]  --->|-- verify files exist -------->|
 |                                |-- create DataFile records      |
 |<-- DataFile records  ---------|                                |
```

**Benefits:**
- Direct upload to storage, bypassing API server bottleneck
- Parallel uploads (32+ concurrent by default)
- No rate limiting on file data
- Automatic multipart for large files (>5GB)

---

## Basic Usage

```python
from synapse_sdk.clients.backend import BackendClient

client = BackendClient(
    base_url="https://api.synapse.sh",
    api_token="your-token"
)

# 1. Request presigned URLs
files_info = [
    {'filename': 'image1.jpg', 'size': 1024000, 'content_type': 'image/jpeg'},
    {'filename': 'image2.jpg', 'size': 2048000, 'content_type': 'image/jpeg'},
]

presigned = client.request_presigned_upload(files=files_info)

# 2. Upload files directly to storage
import httpx
from pathlib import Path

for file_info, local_path in zip(presigned['files'], file_paths):
    with open(local_path, 'rb') as f:
        httpx.put(
            file_info['presigned_url'],
            content=f.read(),
            headers={'Content-Type': file_info.get('content_type', 'application/octet-stream')}
        )

# 3. Confirm uploads
confirmed = client.confirm_presigned_upload(
    files=[
        {
            'file_key': f['file_key'],
            'checksum': compute_md5(path),
            'size': path.stat().st_size,
        }
        for f, path in zip(presigned['files'], file_paths)
    ]
)

print(f"Created {len(confirmed['created'])} DataFile records")
```

---

## Parallel Upload Example

```python
import asyncio
import hashlib
from pathlib import Path

import httpx
from synapse_sdk.clients.backend import BackendClient


async def upload_files_fast(
    client: BackendClient,
    file_paths: list[Path],
    max_concurrent: int = 32,
) -> list[dict]:
    """Upload files using presigned URLs with parallel execution."""

    # Prepare file info
    files_info = [
        {
            'filename': path.name,
            'size': path.stat().st_size,
            'content_type': guess_content_type(path),
        }
        for path in file_paths
    ]

    # Request presigned URLs
    presigned = client.request_presigned_upload(files=files_info)

    # Upload in parallel
    semaphore = asyncio.Semaphore(max_concurrent)

    async def upload_one(file_info: dict, local_path: Path) -> dict:
        async with semaphore:
            async with httpx.AsyncClient() as http:
                with open(local_path, 'rb') as f:
                    content = f.read()

                await http.put(
                    file_info['presigned_url'],
                    content=content,
                    headers={'Content-Type': file_info.get('content_type', 'application/octet-stream')},
                )

                return {
                    'file_key': file_info['file_key'],
                    'checksum': hashlib.md5(content).hexdigest(),
                    'size': len(content),
                }

    upload_results = await asyncio.gather(*[
        upload_one(f, p)
        for f, p in zip(presigned['files'], file_paths)
    ])

    # Confirm uploads
    return client.confirm_presigned_upload(files=list(upload_results))


def guess_content_type(path: Path) -> str:
    """Guess content type from file extension."""
    import mimetypes
    mime_type, _ = mimetypes.guess_type(str(path))
    return mime_type or 'application/octet-stream'


# Usage
async def main():
    client = BackendClient(base_url="...", api_token="...")
    file_paths = list(Path('/data/images').glob('*.jpg'))

    result = await upload_files_fast(client, file_paths)
    print(f"Uploaded {len(result['created'])} files")

asyncio.run(main())
```

---

## Large File Handling (Multipart)

Files larger than 5GB automatically use multipart upload:

```python
# Request presigned URLs for large file
presigned = client.request_presigned_upload(
    files=[{'filename': 'large_dataset.zip', 'size': 10 * 1024**3}]  # 10GB
)

file_info = presigned['files'][0]

if file_info['multipart']:
    # Multipart upload
    multipart = file_info['multipart']
    part_size = multipart['part_size']  # 100MB default
    parts = []

    with open('large_dataset.zip', 'rb') as f:
        for part in multipart['parts']:
            chunk = f.read(part_size)
            response = httpx.put(part['presigned_url'], content=chunk)

            parts.append({
                'part_number': part['part_number'],
                'etag': response.headers['ETag'],
            })

    # Confirm with part ETags
    client.confirm_presigned_upload(
        files=[{
            'file_key': file_info['file_key'],
            'checksum': 'overall-md5',
            'size': 10 * 1024**3,
            'multipart': {
                'upload_id': multipart['upload_id'],
                'parts': parts,
            },
        }]
    )
else:
    # Single PUT upload
    with open('file.zip', 'rb') as f:
        httpx.put(file_info['presigned_url'], content=f.read())
```

---

## Integration with Data Collections

```python
from pathlib import Path
from synapse_sdk.clients.backend import BackendClient


def upload_to_collection_fast(
    client: BackendClient,
    collection_id: int,
    file_paths: list[Path],
) -> None:
    """Upload files to a data collection using bulk upload."""

    # Upload files
    files_info = [
        {'filename': p.name, 'size': p.stat().st_size}
        for p in file_paths
    ]

    presigned = client.request_presigned_upload(files=files_info)

    # Direct uploads
    uploaded = []
    for file_info, local_path in zip(presigned['files'], file_paths):
        with open(local_path, 'rb') as f:
            content = f.read()
            httpx.put(file_info['presigned_url'], content=content)

            uploaded.append({
                'file_key': file_info['file_key'],
                'checksum': hashlib.md5(content).hexdigest(),
                'size': len(content),
            })

    # Confirm and get DataFile records
    confirmed = client.confirm_presigned_upload(files=uploaded)

    # Create data units
    data_units = []
    for data_file, local_path in zip(confirmed['created'], file_paths):
        data_units.append({
            'files': {'primary': {'checksum': data_file['checksum']}},
            'data_collection': collection_id,
            'meta': {
                'origin_file_stem': local_path.stem,
                'origin_file_extension': local_path.suffix,
            },
        })

    # Batch create data units
    for i in range(0, len(data_units), 1000):
        batch = data_units[i:i + 1000]
        client.create_data_units(batch)
```

---

## Progress Tracking

```python
from tqdm import tqdm


async def upload_with_progress(
    client: BackendClient,
    file_paths: list[Path],
) -> dict:
    """Upload with progress bar."""

    files_info = [
        {'filename': p.name, 'size': p.stat().st_size}
        for p in file_paths
    ]

    presigned = client.request_presigned_upload(files=files_info)
    uploaded = []

    with tqdm(total=len(file_paths), desc="Uploading") as pbar:
        for file_info, local_path in zip(presigned['files'], file_paths):
            with open(local_path, 'rb') as f:
                content = f.read()

            httpx.put(file_info['presigned_url'], content=content)

            uploaded.append({
                'file_key': file_info['file_key'],
                'checksum': hashlib.md5(content).hexdigest(),
                'size': len(content),
            })

            pbar.update(1)

    return client.confirm_presigned_upload(files=uploaded)
```

---

## Error Handling

```python
from synapse_sdk.exceptions import ValidationError, ServerError


def safe_bulk_upload(client, files_info, file_paths):
    """Bulk upload with error handling."""

    try:
        presigned = client.request_presigned_upload(files=files_info)
    except ValidationError as e:
        if 'presigned uploads' in str(e):
            # Storage doesn't support presigned URLs, fall back
            print("Falling back to standard upload...")
            return standard_upload(client, file_paths)
        raise

    uploaded = []
    failed_uploads = []

    for file_info, local_path in zip(presigned['files'], file_paths):
        try:
            with open(local_path, 'rb') as f:
                content = f.read()

            response = httpx.put(
                file_info['presigned_url'],
                content=content,
                timeout=60.0,
            )
            response.raise_for_status()

            uploaded.append({
                'file_key': file_info['file_key'],
                'checksum': hashlib.md5(content).hexdigest(),
                'size': len(content),
            })
        except Exception as e:
            failed_uploads.append({
                'path': str(local_path),
                'error': str(e),
            })

    if uploaded:
        confirmed = client.confirm_presigned_upload(files=uploaded)
    else:
        confirmed = {'created': [], 'failed': []}

    return {
        'confirmed': confirmed,
        'upload_failures': failed_uploads,
    }
```

---

## Provider Support

| Storage Provider | Presigned Upload | Multipart |
|------------------|------------------|-----------|
| S3 / MinIO | Yes | Yes |
| Google Cloud Storage | Yes | Yes |
| Azure Blob Storage | Yes | Yes |
| Local Filesystem | No | No |
| SFTP | No | No |

When storage doesn't support presigned uploads, fall back to standard API-based upload.

---

## API Reference

### `request_presigned_upload()`

Request presigned URLs for file uploads.

```python
presigned = client.request_presigned_upload(
    files=[
        {'filename': 'file.jpg', 'size': 1024, 'content_type': 'image/jpeg'},
    ],
    expiration=3600,  # URL validity in seconds
)
```

**Parameters:**
- `files` (list[dict]): File metadata with `filename`, `size`, optional `content_type`
- `expiration` (int): URL expiration time in seconds (default: 3600)

**Returns:**
```python
{
    'files': [
        {
            'filename': 'file.jpg',
            'file_key': 'tenants/1/files/ab/cd/uuid.jpg',
            'presigned_url': 'https://s3.../...',  # None for multipart
            'multipart': None,  # or multipart info for large files
        }
    ],
    'expires_at': '2024-01-01T13:00:00Z',
}
```

### `confirm_presigned_upload()`

Confirm uploaded files and create DataFile records.

```python
confirmed = client.confirm_presigned_upload(
    files=[
        {'file_key': '...', 'checksum': 'md5...', 'size': 1024},
    ]
)
```

**Parameters:**
- `files` (list[dict]): Uploaded file info with `file_key`, `checksum`, `size`, optional `multipart`

**Returns:**
```python
{
    'created': [{'id': 1, 'file': '...', 'checksum': '...', 'size': 1024}],
    'failed': [{'file_key': '...', 'error': '...'}],
}
```

---

## See Also

- [DataCollectionClientMixin](../../api/clients/data-collection-mixin.md) - Standard data collection operations
- [Storage Providers](./storage.md) - Storage configuration
- [File Utilities](./file.md) - File handling utilities
