# API Reference

Complete reference for the `vcm_file_uploader` public API (v0.1.0).

All symbols listed below are importable from the top-level package:

```python
from vcm_file_uploader import STSUploadSession, UploadPlan, ManifestWriter, ...
```

---

## Core Classes

### `STSUploadSession`

Job-bound upload session backed by temporary STS credentials. Uploads files to a scoped S3 prefix using boto3 TransferManager with configurable retries.

Supports the context manager protocol (`with` statement).

```python
class STSUploadSession:
    def __init__(
        self,
        aws_access_key_id: str,
        aws_secret_access_key: str,
        aws_session_token: str,
        region: str,
        bucket: str,
        prefix: str,
        transfer_config: TransferConfig | None = None,
        max_retries: int = 5,
        retry_mode: str = "adaptive",
    ) -> None: ...
```

**Parameters:**

| Name | Type | Default | Description |
|---|---|---|---|
| `aws_access_key_id` | `str` | — | AWS access key ID |
| `aws_secret_access_key` | `str` | — | AWS secret access key |
| `aws_session_token` | `str` | — | AWS session token from STS |
| `region` | `str` | — | AWS region (e.g. `"us-east-1"`) |
| `bucket` | `str` | — | S3 bucket name |
| `prefix` | `str` | — | S3 key prefix (normalized with trailing `/`) |
| `transfer_config` | `TransferConfig \| None` | `None` | Custom transfer configuration |
| `max_retries` | `int` | `5` | Maximum retry attempts for transient S3 errors |
| `retry_mode` | `str` | `"adaptive"` | boto3 retry mode (`"adaptive"`, `"standard"`, `"legacy"`) |

**Attributes:**

| Name | Type | Description |
|---|---|---|
| `bucket` | `str` | S3 bucket name |
| `prefix` | `str` | Normalized S3 key prefix (with trailing `/`) |

#### `upload_file(local_path, key, *, extra_args=None)`

Upload a single file to S3.

```python
def upload_file(
    self,
    local_path: str,
    key: str,
    *,
    extra_args: dict | None = None,
) -> UploadResult: ...
```

| Parameter | Type | Default | Description |
|---|---|---|---|
| `local_path` | `str` | — | Path to local file |
| `key` | `str` | — | S3 object key (must start with `prefix`) |
| `extra_args` | `dict \| None` | `None` | Extra arguments for boto3 upload |

**Returns:** `UploadResult`

**Raises:** `PrefixError` — if `key` does not start with the configured prefix.

#### `upload_files(plan)`

Upload all files from an `UploadPlan`. Files are uploaded in descending size order. Continues on individual file failures.

```python
def upload_files(self, plan: UploadPlan) -> UploadSummary: ...
```

| Parameter | Type | Description |
|---|---|---|
| `plan` | `UploadPlan` | Upload plan containing files to upload |

**Returns:** `UploadSummary`

#### `close()`

Clean up session resources.

```python
def close(self) -> None: ...
```

#### Context Manager

```python
with STSUploadSession(...) as session:
    session.upload_files(plan)
# session.close() called automatically
```

---

### `UploadPlan`

Lightweight wrapper around the JSON upload plan produced by `BackupWatchman`. Provides validated, ordered iteration over files (largest first).

```python
class UploadPlan:
    def __init__(
        self,
        files: list[UploadFileEntry],
        deletions: list[str] | None = None,
        timestamp: str | None = None,
    ) -> None: ...
```

**Parameters:**

| Name | Type | Default | Description |
|---|---|---|---|
| `files` | `list[UploadFileEntry]` | — | Files to upload (sorted largest first internally) |
| `deletions` | `list[str] \| None` | `None` | File paths to delete |
| `timestamp` | `str \| None` | `None` | Plan generation timestamp |

**Attributes:**

| Name | Type | Description |
|---|---|---|
| `files` | `list[UploadFileEntry]` | Files sorted by size (largest first) |
| `deletions` | `list[str]` | Deletion paths |
| `timestamp` | `str \| None` | Plan timestamp |

#### `UploadPlan.from_json(path)` (classmethod)

Load and validate an upload plan from a JSON file.

```python
@classmethod
def from_json(cls, path: str | Path) -> UploadPlan: ...
```

**Raises:** `UploadPlanError` — if the file cannot be read or contains invalid JSON.

#### `UploadPlan.from_dict(data)` (classmethod)

Load and validate an upload plan from a dictionary.

```python
@classmethod
def from_dict(cls, data: dict) -> UploadPlan: ...
```

**Raises:** `UploadPlanError` — if the dictionary is invalid (missing required fields, wrong types, etc.).

**Validation rules:**
- Top-level must contain `"files"` key with a list value
- Each file entry requires: `path`, `full_path`, `size`
- `size` must be a non-negative number

#### Properties

| Property | Type | Description |
|---|---|---|
| `total_bytes` | `int` | Sum of all file sizes |
| `multipart_count` | `int` | Number of files with `multipart=True` |

#### Iteration

`UploadPlan` supports `len()` and iteration in descending size order:

```python
plan = UploadPlan.from_json("plan.json")
print(len(plan))       # number of files
for entry in plan:     # UploadFileEntry objects, largest first
    print(entry.path, entry.size)
```

---

### `ManifestWriter`

Builds and uploads run manifests to S3. Records uploaded objects with sizes, timestamps, and compression metadata.

```python
class ManifestWriter:
    def __init__(
        self,
        session: STSUploadSession,
        run_id: str | None = None,
    ) -> None: ...
```

**Parameters:**

| Name | Type | Default | Description |
|---|---|---|---|
| `session` | `STSUploadSession` | — | Upload session for uploading the manifest |
| `run_id` | `str \| None` | `None` | Run identifier (auto-generated 12-char hex if not provided) |

**Attributes:**

| Name | Type | Description |
|---|---|---|
| `run_id` | `str` | Run identifier |

#### `add_result(result)`

Record a single upload result. Only records successful uploads (failures are skipped).

```python
def add_result(self, result: UploadResult) -> None: ...
```

#### `add_summary(summary)`

Record all successful results from an `UploadSummary`.

```python
def add_summary(self, summary: UploadSummary) -> None: ...
```

#### `to_dict()`

Return the manifest as a dictionary.

```python
def to_dict(self) -> dict: ...
```

**Returns:**

```python
{
    "run_id": str,
    "created_at": str,        # ISO 8601 UTC timestamp
    "bucket": str,
    "prefix": str,
    "object_count": int,
    "objects": [
        {
            "path": str,          # local file path
            "uploaded_key": str,   # S3 key
            "size": int,
            "uploaded_at": str,    # ISO 8601 timestamp
            "compression": {...},  # optional CompressionInfo dict
        },
        ...
    ]
}
```

#### `write_manifest()`

Serialize the manifest to JSON and upload to S3.

```python
def write_manifest(self) -> UploadResult: ...
```

**Returns:** `UploadResult` for the manifest upload itself.

The manifest is uploaded to: `{prefix}manifests/uploader_manifest_{run_id}.json`

#### Properties

| Property | Type | Description |
|---|---|---|
| `manifest_key` | `str` | Full S3 key where the manifest will be uploaded |

---

### `LogSegmenter`

Tracks growing ASCII logs and uploads new segments. Each file is tracked by its last uploaded byte offset in JSONL state.

```python
class LogSegmenter:
    def __init__(self, segments_store: JsonlStore) -> None: ...
```

**Parameters:**

| Name | Type | Description |
|---|---|---|
| `segments_store` | `JsonlStore` | JSONL store for persisting segment state |

#### `get_last_offset(path)`

Get the last uploaded byte offset for a file.

```python
def get_last_offset(self, path: str) -> int: ...
```

**Returns:** `int` — last uploaded offset, or `0` if file has not been tracked.

#### `get_new_segments(path, segment_size=0)`

Detect new bytes since last upload and return segments.

```python
def get_new_segments(self, path: str, segment_size: int = 0) -> list[Segment]: ...
```

| Parameter | Type | Default | Description |
|---|---|---|---|
| `path` | `str` | — | File path to segment |
| `segment_size` | `int` | `0` | Target segment size in bytes (`0` = single segment for all new bytes) |

**Returns:** `list[Segment]` — segments representing new data. Empty if file hasn't grown.

#### `extract_segment(segment)`

Extract a segment's bytes to a temporary file.

```python
def extract_segment(self, segment: Segment) -> str: ...
```

**Returns:** `str` — path to temporary file. Caller is responsible for cleanup.

#### `upload_segment(segment, session, s3_key)`

Extract, upload, and update state for a single segment.

```python
def upload_segment(
    self,
    segment: Segment,
    session: STSUploadSession,
    s3_key: str,
) -> UploadResult: ...
```

| Parameter | Type | Description |
|---|---|---|
| `segment` | `Segment` | Segment to upload |
| `session` | `STSUploadSession` | Upload session |
| `s3_key` | `str` | S3 key for the segment |

**Returns:** `UploadResult` — automatically updates JSONL state on success.

---

### `BackupWatchman`

Monitors a directory, diffs against previous state, and produces upload plans. Each scan cycle: scan → diff → plan → rotate manifests.

```python
class BackupWatchman:
    def __init__(
        self,
        watch_dir: str | Path,
        output_dir: str | Path,
        *,
        rules: ScanRules | None = None,
        hash_policy: HashPolicy | None = None,
        categorize: Callable[[str], str] | None = None,
        job_id: str | None = None,
        multipart_threshold: int = 50 * 1024 * 1024,
    ) -> None: ...
```

**Parameters:**

| Name | Type | Default | Description |
|---|---|---|---|
| `watch_dir` | `str \| Path` | — | Directory to monitor |
| `output_dir` | `str \| Path` | — | Directory for manifests and plan files |
| `rules` | `ScanRules \| None` | `None` | File scan rules (defaults to all files: `**/*`) |
| `hash_policy` | `HashPolicy \| None` | `None` | Hash computation policy |
| `categorize` | `Callable[[str], str] \| None` | `None` | Function mapping file paths to category strings |
| `job_id` | `str \| None` | `None` | Job identifier for manifests |
| `multipart_threshold` | `int` | `52428800` (50 MB) | Size threshold for marking files as multipart |

> **Note:** `ScanRules` and `HashPolicy` are from the `file_watchman` package.

#### `scan()`

Run a single scan-diff-plan cycle.

```python
def scan(self) -> dict: ...
```

**Returns:** Upload plan dictionary:

```python
{
    "files": [
        {
            "path": str,          # relative path
            "full_path": str,     # absolute path
            "size": int,
            "sha256": str | None,
            "category": str | None,
            "reason": str,        # "new", "modified", etc.
            "multipart": bool,
        },
        ...  # sorted by size, largest first
    ],
    "deletions": [str, ...],
    "summary": {
        "upload_count": int,
        "upload_bytes": int,
        "deletion_count": int,
        "multipart_count": int,
    },
    "timestamp": str,       # UTC timestamp (YYYYMMDDTHHmmss)
    "manifest": dict,       # serialized file_watchman manifest
}
```

**Side effects:**
- Writes upload plan JSON to `output_dir/upload_{timestamp}.json`
- Rotates manifest files in `output_dir`

---

## Data Types

### `TransferConfig`

Frozen dataclass configuring boto3 S3 TransferManager.

```python
@dataclass(frozen=True)
class TransferConfig:
    multipart_threshold: int = 134217728   # 128 MiB
    multipart_chunksize: int = 134217728   # 128 MiB
    max_concurrency: int = 8
```

| Field | Type | Default | Description |
|---|---|---|---|
| `multipart_threshold` | `int` | `134217728` (128 MiB) | File size threshold for multipart uploads |
| `multipart_chunksize` | `int` | `134217728` (128 MiB) | Size of each multipart chunk |
| `max_concurrency` | `int` | `8` | Maximum concurrent upload threads |

---

### `UploadFileEntry`

Frozen dataclass representing a single file entry from an upload plan.

```python
@dataclass(frozen=True)
class UploadFileEntry:
    path: str
    full_path: str
    size: int
    category: str | None = None
    sha256: str | None = None
    reason: str = "unknown"
    multipart: bool = False
```

| Field | Type | Default | Description |
|---|---|---|---|
| `path` | `str` | — | Relative path to the file |
| `full_path` | `str` | — | Absolute path to the file |
| `size` | `int` | — | File size in bytes |
| `category` | `str \| None` | `None` | Optional category (e.g. `"log"`) |
| `sha256` | `str \| None` | `None` | Optional SHA-256 checksum |
| `reason` | `str` | `"unknown"` | Why the file was included (`"new"`, `"modified"`, etc.) |
| `multipart` | `bool` | `False` | Whether the file requires multipart upload |

---

### `CompressionInfo`

Frozen dataclass with compression metadata.

```python
@dataclass(frozen=True)
class CompressionInfo:
    algorithm: str = "gzip"
    level: int = 6
    original_bytes: int = 0
    compressed_bytes: int = 0
```

| Field | Type | Default | Description |
|---|---|---|---|
| `algorithm` | `str` | `"gzip"` | Compression algorithm |
| `level` | `int` | `6` | Compression level (0-9 for gzip) |
| `original_bytes` | `int` | `0` | Original uncompressed size |
| `compressed_bytes` | `int` | `0` | Size after compression |

---

### `UploadResult`

Mutable dataclass representing the result of a single file upload.

```python
@dataclass
class UploadResult:
    local_path: str
    s3_key: str
    size: int
    success: bool
    error: str | None = None
    compression: CompressionInfo | None = None
    uploaded_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
```

| Field | Type | Default | Description |
|---|---|---|---|
| `local_path` | `str` | — | Path to the local file |
| `s3_key` | `str` | — | S3 object key |
| `size` | `int` | — | Uploaded file size in bytes (0 on failure) |
| `success` | `bool` | — | Whether the upload succeeded |
| `error` | `str \| None` | `None` | Error message if upload failed |
| `compression` | `CompressionInfo \| None` | `None` | Compression info if file was compressed |
| `uploaded_at` | `datetime` | now (UTC) | Timestamp when upload completed |

---

### `UploadSummary`

Mutable dataclass summarizing an `upload_files()` run.

```python
@dataclass
class UploadSummary:
    results: list[UploadResult] = field(default_factory=list)
```

**Properties:**

| Property | Type | Description |
|---|---|---|
| `succeeded` | `list[UploadResult]` | Successful uploads |
| `failed` | `list[UploadResult]` | Failed uploads |
| `total_bytes` | `int` | Total bytes uploaded (successful only) |
| `all_succeeded` | `bool` | `True` if all uploads succeeded and list is non-empty |
| `exit_code` | `int` | Exit code for programmatic callers (see below) |

**Exit code semantics:**

| Code | Meaning |
|---|---|
| `0` | All uploads succeeded |
| `1` | Partial failure — some succeeded, some failed |
| `2` | Total failure — all uploads failed or none attempted |

```python
summary = session.upload_files(plan)
sys.exit(summary.exit_code)
```

---

### `Segment`

Frozen dataclass representing a byte range of a growing file to upload.

```python
@dataclass(frozen=True)
class Segment:
    path: str
    offset: int
    length: int
    segment_index: int
```

| Field | Type | Description |
|---|---|---|
| `path` | `str` | Path to the file |
| `offset` | `int` | Starting byte offset |
| `length` | `int` | Number of bytes in segment |
| `segment_index` | `int` | Segment index number |

**Properties:**

| Property | Type | Description |
|---|---|---|
| `segment_key_suffix` | `str` | S3 key suffix, e.g. `"logs/amber.mdout.seg0003"` |

---

## State Management

### `JsonlStore`

Append-only JSONL store with last-write-wins semantics. Each entry must contain a key field. On read, the latest entry for each key wins. Compaction deduplicates the file on disk.

```python
class JsonlStore:
    def __init__(self, path: str | Path, key_field: str = "path") -> None: ...
```

**Parameters:**

| Name | Type | Default | Description |
|---|---|---|---|
| `path` | `str \| Path` | — | Path to the JSONL file |
| `key_field` | `str` | `"path"` | Field name used as the deduplication key |

**Attributes:**

| Name | Type | Description |
|---|---|---|
| `path` | `Path` | File path |
| `key_field` | `str` | Key field name |

#### `append(entry)`

Append a single entry. Creates parent directories if needed.

```python
def append(self, entry: dict) -> None: ...
```

**Raises:** `ValueError` — if entry is missing the required key field.

#### `read_all()`

Read all entries, returning the latest per key (last-write-wins).

```python
def read_all(self) -> dict[str, dict]: ...
```

**Returns:** `dict[str, dict]` — mapping of keys to their latest entries. Malformed lines are skipped with a warning.

#### `get(key)`

Get the latest entry for a given key.

```python
def get(self, key: str) -> dict | None: ...
```

**Returns:** `dict | None` — latest entry, or `None` if not found.

#### `compact()`

Deduplicate the file on disk, keeping only the latest entry per key. Uses atomic rename for crash safety.

```python
def compact(self) -> int: ...
```

**Returns:** `int` — number of entries after compaction. Deletes the file if empty.

---

### `StateManager`

Manages JSONL state files for the uploader. Provides three pre-configured stores.

```python
class StateManager:
    def __init__(self, state_dir: str | Path) -> None: ...
```

**Parameters:**

| Name | Type | Description |
|---|---|---|
| `state_dir` | `str \| Path` | Directory to store state files |

**Attributes:**

| Name | Type | File | Key Field | Description |
|---|---|---|---|---|
| `state_dir` | `Path` | — | — | State directory path |
| `files` | `JsonlStore` | `files.jsonl` | `"path"` | Tracks uploaded files |
| `segments` | `JsonlStore` | `ascii_segments.jsonl` | `"segment_key"` | Tracks log segments |
| `runs` | `JsonlStore` | `runs.jsonl` | `"run_id"` | Tracks upload runs |

#### `compact_all()`

Compact all state files.

```python
def compact_all(self) -> dict[str, int]: ...
```

**Returns:** `dict[str, int]` — entry counts per store after compaction:

```python
{"files": int, "segments": int, "runs": int}
```

---

## Compression Functions

### `maybe_compress(path, *, category=None, threshold=67108864, level=6, check_stability=True)`

Conditionally compress an ASCII file if it meets size, type, and stability criteria.

```python
def maybe_compress(
    path: str,
    *,
    category: str | None = None,
    threshold: int = 67108864,       # 64 MiB
    level: int = 6,
    check_stability: bool = True,
) -> tuple[str, CompressionInfo] | None: ...
```

| Parameter | Type | Default | Description |
|---|---|---|---|
| `path` | `str` | — | File to consider for compression |
| `category` | `str \| None` | `None` | If `"log"`, skips ASCII detection |
| `threshold` | `int` | `67108864` (64 MiB) | Minimum file size |
| `level` | `int` | `6` | Gzip compression level (0-9) |
| `check_stability` | `bool` | `True` | Whether to verify the file is not actively growing |

**Returns:** `tuple[str, CompressionInfo] | None` — `(compressed_path, info)` if compressed, `None` if criteria not met.

**Compression criteria:**
1. File size >= `threshold`
2. File is ASCII text (no null bytes, >= 95% printable bytes) **or** `category == "log"`
3. File is stable (size unchanged over 2-second interval) if `check_stability` is `True`

```python
result = maybe_compress("/data/logs/output.log")
if result:
    compressed_path, info = result
```

---

### `compress_file(path, *, output_path=None, level=6)`

Compress a file with gzip.

```python
def compress_file(
    path: str,
    *,
    output_path: str | None = None,
    level: int = 6,
) -> tuple[str, CompressionInfo]: ...
```

| Parameter | Type | Default | Description |
|---|---|---|---|
| `path` | `str` | — | File to compress |
| `output_path` | `str \| None` | `None` | Output path (defaults to `path + ".gz"`) |
| `level` | `int` | `6` | Gzip compression level (0-9) |

**Returns:** `tuple[str, CompressionInfo]` — `(compressed_path, info)`.

---

### `is_ascii_file(path, sample_size=8192)`

> **Note:** Not exported in `__all__` but accessible via `vcm_file_uploader.compress.is_ascii_file`.

Detect if a file is ASCII/text by sampling its content.

```python
def is_ascii_file(path: str, sample_size: int = 8192) -> bool: ...
```

**Detection logic:**
1. Read the first `sample_size` bytes
2. Reject if any null bytes are found (strong binary indicator)
3. Accept if >= 95% of bytes are printable ASCII (32-126) or whitespace (tab, newline, carriage return)

---

## Helper Functions

### `create_transfer_config(config=None)`

> **Note:** Not exported in `__all__` but accessible via `vcm_file_uploader.transfer.create_transfer_config`.

Create a boto3 `TransferConfig` from a vcm `TransferConfig`.

```python
def create_transfer_config(config: TransferConfig | None = None) -> BotoTransferConfig: ...
```

| Parameter | Type | Default | Description |
|---|---|---|---|
| `config` | `TransferConfig \| None` | `None` | Custom configuration (uses defaults if `None`) |

**Returns:** `boto3.s3.transfer.TransferConfig`

---

## Exceptions

### `PrefixError`

Raised when an S3 key falls outside the allowed prefix. Subclass of `ValueError`.

```python
class PrefixError(ValueError): ...
```

Defined in `vcm_file_uploader.session`. Access via:

```python
from vcm_file_uploader.session import PrefixError
```

### `UploadPlanError`

Raised when an upload plan is invalid (malformed JSON, missing required fields, etc.). Subclass of `Exception`.

```python
class UploadPlanError(Exception): ...
```

Defined in `vcm_file_uploader.plan`. Access via:

```python
from vcm_file_uploader.plan import UploadPlanError
```

---

## Constants

### Session defaults (`vcm_file_uploader.session`)

| Constant | Value | Description |
|---|---|---|
| `DEFAULT_MAX_RETRIES` | `5` | Default maximum retry attempts |
| `DEFAULT_RETRY_MODE` | `"adaptive"` | Default boto3 retry mode |

### Compression defaults (`vcm_file_uploader.compress`)

| Constant | Value | Description |
|---|---|---|
| `DEFAULT_COMPRESS_THRESHOLD` | `67108864` (64 MiB) | Minimum file size for compression |
| `DEFAULT_GZIP_LEVEL` | `6` | Default gzip compression level |
| `STABILITY_WAIT` | `2.0` | Seconds between file size checks |

### Watchman defaults (`vcm_file_uploader.watchman`)

| Constant | Value | Description |
|---|---|---|
| `DEFAULT_MULTIPART_THRESHOLD` | `52428800` (50 MB) | Size threshold for multipart marking |

### TransferConfig defaults (`vcm_file_uploader._types`)

| Field | Value | Description |
|---|---|---|
| `multipart_threshold` | `134217728` (128 MiB) | Multipart upload threshold |
| `multipart_chunksize` | `134217728` (128 MiB) | Multipart chunk size |
| `max_concurrency` | `8` | Maximum concurrent uploads |
