# Ghost Remover

AI-powered video inpainting engine. Remove watermarks, logos, and objects from video using SAM 2 + ProPainter.

## Features

- **SAM 2 tracking** - Segment anything with text, box, or point prompts via Grounding DINO + SAM 2
- **ProPainter inpainting** - Temporal-aware video inpainting with optical flow completion
- **Two-pass architecture** - Pass 1: tracking + mask generation. Pass 2: inpainting + streaming encode
- **Chunked processing** - Handles arbitrarily long videos with overlap blending
- **MPS optimized** - Native Apple Silicon support with automatic CPU fallback for unsupported ops
- **Checkpoint resume** - Interrupt and resume without losing progress
- **Batch processing** - Process multiple videos in one command
- **Quality metrics** - Temporal consistency, pixel integrity (PSNR), and style (LPIPS) checks

## v0.3 Performance Upgrades

| Optimization | Effect |
|---|---|
| **Eliminate redundant ffprobe** | `frames_to_numpy` accepts `video_info` param, skips re-probing |
| **Single decode in Pass 1** | Reuses extracted JPEGs for re-detection fallback instead of second FFmpeg decode |
| **Float16 mixed precision** | `torch.autocast("mps", float16)` for ProPainter inference — ~1.5x faster, half memory |
| **Chunk prefetch** | Background thread decodes next chunk while current chunk is being inpainted |
| **Vectorized blending** | Overlap blending uses numpy batch ops instead of per-frame Python loop |
| **OOM auto-retry** | On memory error: split chunk in half → retry → downscale fallback → original frames |
| **FFmpeg timeout** | All subprocess calls have timeout protection (120-300s) |
| **Frame count validation** | Warns when decoded frame count doesn't match expected |

## Installation

```bash
# Create conda environment
conda env create -f environment.yml
conda activate ghost-remover

# Clone ProPainter into third_party/
git clone https://github.com/sczhou/ProPainter.git third_party/ProPainter
bash setup_propainter.sh

# Download SAM 2 checkpoints
mkdir -p checkpoints
# See https://github.com/facebookresearch/segment-anything-2#download-checkpoints
```

## Usage

### CLI

```bash
# Basic - remove watermark by text prompt
ghost-remover remove --video input.mp4 --target "watermark"

# With bounding box
ghost-remover remove --video input.mp4 --target "logo" --box 100,50,300,150

# Dry-run (preview mask overlay only)
ghost-remover remove --video input.mp4 --target "watermark" --dry-run

# Batch processing
ghost-remover remove --video a.mp4 b.mp4 c.mp4 --target "watermark"

# Hardware-accelerated encoding (VideoToolbox)
ghost-remover remove --video input.mp4 --target "watermark" --hw-accel

# Adjust sensitivity (0.1=tight, 1.0=aggressive)
ghost-remover remove --video input.mp4 --target "watermark" --sensitivity 0.7
```

### Python API

```python
from main import GhostRemoverPipeline

config = {
    "video_path": "input.mp4",
    "target": "watermark",
    "sensitivity": 0.5,
    "sam_model_size": "large",
}

pipeline = GhostRemoverPipeline(config)
output = pipeline.run()
pipeline.cleanup()
```

## Architecture

```
ghost-remover/
  core/
    ffmpeg_handler.py   # Video decode/encode, streaming pipe I/O
    tracker_sam3.py     # SAM 2 segmentation + video tracking
    inpainter_pp.py     # ProPainter inpainting engine
  utils/
    gpu_manager.py      # MPS memory monitoring + chunk size estimation
    video_chunker.py    # Chunk planning + overlap blending
    checkpoint.py       # Interrupt recovery
    quality_metrics.py  # Temporal/pixel/style quality checks
  api/
    skill_interface.py  # CLI + JSON API
  main.py               # Two-pass pipeline orchestrator
```

### Pipeline Flow

```
Input Video
  │
  ├─ Pass 1: SAM 2 Tracking (5-50%)
  │    ├─ First frame segmentation (text/box/point prompt)
  │    ├─ Per-chunk mask propagation
  │    ├─ Drift detection + re-detection
  │    └─ Packed mask storage (.npz)
  │
  ├─ Pass 2: ProPainter Inpainting (50-95%)
  │    ├─ RAFT optical flow (fp16)
  │    ├─ Bidirectional flow completion
  │    ├─ Image propagation
  │    ├─ Transformer inpainting (sliding window)
  │    ├─ Overlap blending (vectorized)
  │    └─ Streaming FFmpeg encode
  │
  └─ Finalize (95-100%)
       ├─ Audio merge
       └─ Quality check
```

## Tests

```bash
# Run all tests (134 tests)
python3 -m pytest tests/ghost-remover/ -v
```

## Requirements

- Python >= 3.11
- PyTorch >= 2.2.0 (MPS support)
- FFmpeg
- SAM 2 (segment-anything-2)
- ProPainter (third_party/)
- Grounding DINO (via HuggingFace transformers, optional for text prompts)

## Version

**v0.3.0** — 134 tests passing

## License

MIT
