# Sign Language Toolkit (SLTK)

A research toolkit for sign language video analysis: workspace management, pose extraction, automatic segmentation, ELAN annotation editing, and a REST API serving a React annotation workstation.

## Installation

```bash
# Core (data loading, formats, ELAN I/O)
pip install -e .

# With pose extraction
pip install -e ".[mediapipe]"

# With web API + frontend
pip install -e ".[api]"

# Everything
pip install -e ".[all]"

# Development (includes pytest, black, ruff, mypy)
pip install -e ".[dev]"
```

Requires Python 3.10+.

## Quick Start

### Running the API

```bash
# Start FastAPI backend (port 8000) + Vite frontend (port 5173)
bash scripts/run_dev.sh

# Or run the backend only
uvicorn sltk.api.main:app --host 0.0.0.0 --port 8000
```

Interactive docs at `http://localhost:8000/docs` (Swagger UI).

### Python Library

```python
from sltk.data import PoseSequence, Segment, SegmentList
from sltk.io import read_eaf, write_eaf

# Load poses from H5 file
poses = PoseSequence.load("video_wilor.h5", format="wilor", fps=25)

# Load ELAN annotations
segments = read_eaf("annotations.eaf", tiers=["Gloss"])

# Create segments and export to ELAN
new_segments = SegmentList([
    Segment(start=0.0, end=1.5, label="HELLO", tier="Gloss"),
    Segment(start=1.5, end=3.0, label="WORLD", tier="Gloss"),
])
write_eaf(new_segments, "output.eaf", video_path="source.mp4")
```

### CLI

```bash
sltk convert input.npy output.h5 --from mediapipe --to wilor --fps 25
sltk evaluate predictions.txt references.txt --task translation
sltk to-elan segments.json --video source.mp4 --output annotations.eaf
sltk from-elan annotations.eaf --output segments.json --tier Gloss
```

---

## Processing Pipeline

SLTK provides a three-stage pipeline for processing sign language videos: **pose extraction** (video → H5), **segmentation** (H5 → sign boundaries), and **spotting** (segments → gloss labels). Each stage can be run independently.

### Overview

```
Video (.mp4)
    │
    ├─► 1. Pose Extraction ──► {stem}_wilor.h5
    │       (WiLoR hand model: MANO params, 3D keypoints)
    │
    └─► 2. Segmentation ──► {stem}_segments.eaf / .json
    │       (Transformer BIO labelling: OUT/IN/BEGIN)
    │
    └─► 3. Spotting ──► {stem}_spotted.eaf
            (SignRep: match segments to dictionary glosses)
```

### Stage 1: Pose Extraction (Video → H5)

Extract hand poses from video using WiLoR. This produces an H5 file containing MANO rotation matrices and 3D keypoints per frame.

**Python:**

```python
from sltk.extraction.wilor import WiLoRExtractor, WiLoRConfig

config = WiLoRConfig(
    checkpoint_path="path/to/wilor_final.ckpt",
    detector_path="path/to/detector.pt",
)
extractor = WiLoRExtractor(config)
extractor.load_model()
result = extractor.extract_from_video("video.mp4")
# Saves to video_wilor.h5
```

**API:**

```bash
# Start extraction job (runs in background)
curl -X POST http://localhost:8000/api/extraction/start \
  -H "Content-Type: application/json" \
  -d '{
    "video_path": "/data/video.mp4",
    "output_root": "/data/output",
    "config": {"enable_wilor": true, "device": "cuda"}
  }'

# Poll status
curl http://localhost:8000/api/extraction/status/{job_id}
```

**H5 file structure (WiLoR):**

```
video_wilor.h5
  attrs: fps, num_frames, resolution, extractor
  frame_idx:      (num_frames, 2)       # (start_idx, count) per frame
  kpts_3d:        (num_detections, 21, 3)
  right:          (num_detections,)      # True = right hand
  mano/
    hand_pose:    (num_detections, 15, 3, 3)   # rotation matrices
    global_orient:(num_detections, 1, 3, 3)
```

**Weight resolution** — model checkpoints are found in this priority order:

1. Explicit path argument
2. Environment variable (`SLTK_WILOR_CHECKPOINT`, `SLTK_WILOR_DETECTOR`)
3. Bundled at `sltk/weights/wilor/`

MediaPipe and NLF extractors are also available for body/face poses — see `sltk/extraction/`.

### Stage 2: Segmentation (H5 → Segments)

The segmenter v2 is a Transformer that reads WiLoR H5 files and predicts per-frame BIO labels (0=OUT, 1=IN, 2=BEGIN), identifying where signs start and end.

**If you already have H5 files**, this is where you start.

**Python:**

```python
from sltk.segmentation.runner import segment_h5
from sltk.segmentation.output import OutputFormat

# Segment a single H5 file → JSON output
segment_h5(
    "video_wilor.h5",
    output_path="video_segments.json",
    output_format=OutputFormat.JSON,
    fps=25.0,
)

# Segment a single H5 file → ELAN output
segment_h5(
    "video_wilor.h5",
    output_path="video_segments.eaf",
    output_format=OutputFormat.ELAN,
    fps=25.0,
    media_path="video.mp4",  # links video in the EAF file
)

# Segment an entire directory of H5 files
segment_h5(
    "/data/poses/",
    output_path="/data/segments/output.json",
    output_format=OutputFormat.JSON,
    fps=25.0,
)
```

**Lower-level control:**

```python
from sltk.segmentation.runner import get_runner
from sltk.segmentation.h5_loader import h5_to_features
from sltk.segmentation.postprocess import extract_segments

# Load features from H5 (converts MANO rotations → 192-dim features)
features = h5_to_features("video_wilor.h5")  # shape: (num_frames, 192)

# Get the inference runner (singleton, loads checkpoint once)
runner = get_runner()

# Predict BIO labels
labels = runner.predict(features)  # shape: (num_frames,) with values 0/1/2

# Extract segment boundaries as (start_frame, end_frame) tuples
segments = extract_segments(labels)
# [(12, 45), (50, 82), (90, 120), ...]
```

**API:**

```bash
# Segment a single H5 file
curl -X POST http://localhost:8000/api/segmentation/segment \
  -H "Content-Type: application/json" \
  -d '{"h5_path": "/data/video_wilor.h5", "fps": 25.0}'

# Segment a directory (batch)
curl -X POST http://localhost:8000/api/segmentation/segment/batch \
  -H "Content-Type: application/json" \
  -d '{
    "directory": "/data/poses/",
    "fps": 25.0,
    "output_path": "/data/segments/",
    "output_format": "json"
  }'
```

**JSON output format:**

```json
{
  "video_name": {
    "fps": 25.0,
    "num_frames": 3000,
    "segments": [
      {"start_frame": 12, "end_frame": 45, "start_sec": 0.48, "end_sec": 1.80},
      {"start_frame": 50, "end_frame": 82, "start_sec": 2.00, "end_sec": 3.28}
    ]
  }
}
```

**ELAN output:** creates a tier `{video_name}_segmentation` with each segment labelled `"SIGN"`, authored by `segmenter_v2`.

**Checkpoint resolution:** set `SLTK_SEGMENTOR_CHECKPOINT` or place `segmentor_v2.ckpt` in `sltk/weights/segmentor/`.

### Stage 3: Spotting (Segments → Gloss Labels)

Spotting uses SignRep to extract 768-dim visual features from video frames, then matches each detected segment against a **dictionary** of known sign features to produce ranked gloss predictions.

**Prerequisites:**

- A segmented video (from Stage 2) with known segment boundaries
- A dictionary of sign features — `.npz` files with key `best_latent`, one per sign, typically stored at `/vol/research/SignFeaturePool/features2/{dataset}/{method}/`

**Python — full pipeline:**

```python
from sltk.embedding.pipeline import SignRepPipeline

pipeline = SignRepPipeline()

# Step 1: Extract dense features from the full video (sliding 16-frame windows)
continuous = pipeline.extract_continuous("video.mp4", stride=4)
# continuous.features: (L, 768) L2-normalized

# Step 2: Load dictionary features
dictionary = pipeline.load_dictionary(
    ["/data/dictionaries/bsldict/signrep/"],
    feature_key="best_latent",
)

# Step 3: Define segments (from Stage 2 output, or load from JSON/EAF)
segments = [
    {"segment_id": 0, "start_frame": 12, "end_frame": 45},
    {"segment_id": 1, "start_frame": 50, "end_frame": 82},
]

# Step 4: Spot — match each segment against dictionary
result = pipeline.spot(
    features=continuous,
    segments=segments,
    dictionary=dictionary,
    top_k=10,
    segment_pooling="max",  # "max", "mean", or "softmax_weighted"
)

# Each spotted segment has ranked gloss matches
for seg in result.segments:
    print(f"Segment {seg.start_ms}ms–{seg.end_ms}ms:")
    for gl in seg.top_glosses:
        print(f"  Rank {gl['rank']}: {gl['gloss']} ({gl['similarity']:.3f})")
```

**Python — one-shot from video:**

```python
from sltk.embedding.pipeline import SignRepPipeline

pipeline = SignRepPipeline()
result = pipeline.spot_from_video(
    video_path="video.mp4",
    segments_json="video_segments.json",  # from Stage 2
    dictionary_dirs=["/data/dictionaries/bsldict/signrep/"],
    top_k=20,
    stride=4,
)
```

**Save results as ELAN:**

```python
from sltk.segmentation.output import save_spotted_elan

save_spotted_elan(
    result,
    output_path="video_spotted.eaf",
    fps=25.0,
    media_path="video.mp4",
)
```

This creates an EAF file with tiers `Rank-1` through `Rank-N` (gloss labels) and `Score-1` through `Score-N` (similarity scores), authored by `signrep_spotter`.

**API:**

```bash
# Extract continuous features (cached server-side for 30 min)
curl -X POST http://localhost:8000/api/signrep/continuous/extract \
  -H "Content-Type: application/json" \
  -d '{"video_path": "/data/video.mp4", "stride": 4}'
# Returns: {"features_id": "abc123", ...}

# Spot glosses using cached features
curl -X POST http://localhost:8000/api/signrep/spot \
  -H "Content-Type: application/json" \
  -d '{
    "features_id": "abc123",
    "segments": [{"segment_id": 0, "start_frame": 12, "end_frame": 45}],
    "dictionary_dirs": ["/data/dictionaries/bsldict/signrep/"],
    "top_k": 10
  }'
```

**Checkpoint:** set `SLTK_SIGNREP_CHECKPOINT` or place `ckpt.pt` in `sltk/weights/signrep/`.

### End-to-End: Processing API

The processing API combines segmentation and spotting into a single background job. It expects WiLoR H5 files to already exist alongside the videos.

**API:**

```bash
# Segmentation only
curl -X POST http://localhost:8000/api/processing/submit \
  -H "Content-Type: application/json" \
  -d '{
    "video_paths": ["/data/video1.mp4", "/data/video2.mp4"],
    "type": "segments",
    "fps": 25.0
  }'

# Segmentation + spotting
curl -X POST http://localhost:8000/api/processing/submit \
  -H "Content-Type: application/json" \
  -d '{
    "video_paths": ["/data/video1.mp4", "/data/video2.mp4"],
    "type": "spots",
    "dictionary_dirs": ["/data/dictionaries/bsldict/signrep/"],
    "top_k": 5,
    "fps": 25.0,
    "workspace": "my_workspace"
  }'

# Poll job status
curl http://localhost:8000/api/processing/status/{job_id}

# Download output EAF
curl -O http://localhost:8000/api/processing/output/{job_id}/video1_spotted.eaf
```

**H5 file lookup:** the processing API searches for `{stem}_wilor.h5` next to the video, in a `poses/` subdirectory, or in a `{stem}/` subdirectory. If no H5 is found, the video is skipped.

**Output files:**

| Type | Output file | Description |
|------|-------------|-------------|
| `segments` | `{stem}_segments.eaf` | Sign boundaries (SIGN labels) |
| `spots` | `{stem}_segments.eaf` + `{stem}_spotted.eaf` | Boundaries + ranked gloss labels |

When a `workspace` is specified, output EAF files are auto-ingested into the corpus database.

### Building a Dictionary

Before spotting, you need a dictionary of sign features. Extract one feature per isolated sign video:

```python
from sltk.embedding.pipeline import SignRepPipeline

pipeline = SignRepPipeline()

# Single sign video → 768-dim feature
result = pipeline.extract_dictionary("isolated_sign.mp4", method="middle")
result.save_npz("dictionary/HELLO.npz")
```

**Batch extraction via the API:**

```bash
curl -X POST http://localhost:8000/api/signrep/dictionary/batch/job \
  -H "Content-Type: application/json" \
  -d '{
    "video_dir": "/data/isolated_signs/",
    "output_dir": "/data/dictionary/",
    "method": "middle"
  }'
```

Each `.npz` file is named after the gloss (e.g., `HELLO.npz`) and contains a `best_latent` key with the 768-dim feature vector.

---

## Architecture

```
sltk/
├── api/                  # FastAPI REST API (16 routers, 88+ endpoints)
│   ├── main.py           # App init, middleware, router registration
│   ├── models.py         # Pydantic request/response schemas
│   ├── routers/          # Route handlers (see API Reference below)
│   ├── dependencies.py   # Path validation, security
│   └── security.py       # Security headers, CORS
├── io/                   # File I/O
│   ├── elan.py           # ELAN .eaf read/write
│   ├── elan_roundtrip.py # XML-preserving ELAN editing (round-trip safe)
│   ├── h5.py             # HDF5 pose data I/O
│   └── safe_write.py     # Atomic file operations
├── extraction/           # Pose extraction (MediaPipe, WiLoR, NLF)
├── segmentation/         # Transformer-based sign segmentation
├── visualization/        # Skeleton overlay video generation
├── processing/           # Feature computation, normalization
├── analysis/             # Clustering, embeddings, statistics
├── data/                 # Core types (PoseSequence, Segment, Sample)
│   └── datasets/         # Dataset loaders (BOBSL, How2Sign, BSLCP, etc.)
└── config.py             # Configuration and environment
frontend/                 # React/Vite annotation workstation
scripts/
└── run_dev.sh            # Dev server launcher
```

---

## API Reference

Base URL: `http://localhost:8000`

### Workspace Management — `/api/workspace`

Multi-workspace system for organizing videos and annotation files. Persists to `~/.sltk/workspaces.json`.

| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/list` | List all workspaces |
| `POST` | `/create` | Create a new workspace |
| `POST` | `/switch` | Switch active workspace |
| `POST` | `/scan` | Scan directory for videos + ELAN files |
| `GET` | `/status` | Current workspace status |
| `PUT` | `/rename` | Rename workspace |
| `DELETE` | `/clear` | Clear active workspace |
| `DELETE` | `/{name}` | Delete workspace by name |
| `PATCH` | `/match` | Override video-ELAN matching |
| `POST` | `/rescan` | Rescan for new files |

### Videos — `/api/videos` `/api/video`

| Method | Endpoint | Description |
|--------|----------|-------------|
| `POST` | `/videos/discover` | Discover videos in directory (recursive) |
| `GET` | `/videos/info` | Video metadata (fps, resolution, duration) |
| `GET` | `/video/stream` | Stream video with HTTP Range support |

### Audio — `/api/audio`

| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/waveform` | Extract waveform peaks. Params: `path`, `samples` (default 8000) |

### Extraction — `/api/extraction`

Pose extraction jobs (MediaPipe, WiLoR, NLF/SMPL-X).

| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/status/{job_id}` | Poll extraction progress |
| `POST` | `/cancel/{job_id}` | Cancel extraction |
| `GET` | `/jobs` | List all extraction jobs |
| `GET` | `/logs/{job_id}` | Job logs (last 1000 entries) |

### Poses — `/api/poses`

| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/load` | Load pose data from H5 file |
| `GET` | `/metadata` | Pose metadata (frames, keypoints, format) |
| `GET` | `/frame` | Single frame pose data |
| `GET` | `/statistics` | Pose statistics |

### Visualization — `/api/visualization`

Skeleton overlay video generation from H5 pose data.

| Method | Endpoint | Description |
|--------|----------|-------------|
| `POST` | `/generate` | Generate overlay. Body: `{video_path, h5_path, viz_type}` |
| `GET` | `/status/{job_id}` | Poll generation progress |
| `GET` | `/check` | Check if cached overlay exists |

`viz_type`: `"mediapipe"`, `"wilor"`, or `"nlf"`

### Datasets — `/api/datasets`

| Method | Endpoint | Description |
|--------|----------|-------------|
| `POST` | `/connect` | Register dataset connection |
| `GET` | `/connections` | List connected datasets |
| `GET` | `/list` | List available datasets |
| `GET` | `/{name}/videos` | Videos in dataset |
| `DELETE` | `/connections/{name}` | Remove connection |

### Features — `/api/features`

| Method | Endpoint | Description |
|--------|----------|-------------|
| `POST` | `/detect` | Detect features in video |
| `GET` | `/scan` | Scan for feature files |
| `GET` | `/datasets/{name}/features/summary` | Feature summary for dataset |

### Analysis — `/api/analysis`

Research-oriented endpoints for vocabulary, statistics, and linguistic analysis.

| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/vocabulary` | Extract vocabulary from dataset |
| `POST` | `/batch/statistics` | Batch statistical analysis |
| `POST` | `/research/vocabulary-mapping` | Map glosses across datasets |
| `POST` | `/research/compare-datasets` | Compare two datasets |
| `POST` | `/research/find-gloss-examples` | Find gloss examples |
| `POST` | `/linguistic/concordance` | Gloss concordance |
| `POST` | `/linguistic/cooccurrence` | Co-occurrence analysis |
| `POST` | `/linguistic/ngrams` | N-gram frequency |
| `POST` | `/linguistic/duration-analysis` | Duration statistics |

### Embeddings — `/api/embeddings`

| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/status/{dataset}` | Embedding generation status |
| `DELETE` | `/cache/{dataset}` | Clear embeddings |
| `GET` | `/signrep/status` | SignRep model status |

### Linguistics — `/api/linguistics`

Inter-rater reliability and phonological analysis.

| Method | Endpoint | Description |
|--------|----------|-------------|
| `POST` | `/reliability/kappa` | Cohen's kappa |
| `POST` | `/reliability/krippendorff` | Krippendorff's alpha |
| `POST` | `/reliability/boundary-agreement` | Boundary agreement |
| `POST` | `/reliability/confusion-matrix` | Confusion matrix |
| `POST` | `/phonological-form` | Extract phonological form |
| `POST` | `/phonological-distance` | Phonological distance |

### Jobs — `/api/jobs`

| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/status` | Job system status |
| `GET` | `/gpu` | GPU status and memory |
| `GET` | `/list` | List active jobs |
| `POST` | `/{job_id}/cancel` | Cancel job |

### Settings — `/api/settings`

| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/` | Get app settings |
| `POST` | `/` | Update settings |
| `GET` | `/info` | System info |
| `GET` | `/system/weights` | Model weights info |

---

## Middleware & Security

The API applies the following middleware (in order):

1. **GZip** — compresses responses >500 bytes
2. **Security headers** — `X-Frame-Options: DENY`, `X-Content-Type-Options: nosniff`, CSP, Permissions-Policy
3. **CORS** — configurable via `SLTK_CORS_ORIGINS` env var (default: `localhost:5173,localhost:3000`)
4. **Path validation** — whitelist check against `SLTK_ALLOWED_PATHS`, directory traversal prevention

## Configuration

Environment variables (set in `.env` or shell):

| Variable | Description | Default |
|----------|-------------|---------|
| `SLTK_CORS_ORIGINS` | Allowed CORS origins (comma-separated) | `http://localhost:5173,http://localhost:3000` |
| `SLTK_ALLOWED_PATHS` | Allowed filesystem paths for API access | `/vol/research,/home` |
| `SLTK_RESEARCH_DATA_ROOT` | Root for research data | `/vol/research` |
| `SLTK_DATASETS_ROOT` | Root for raw datasets | `/vol/research/datasets` |
| `SLTK_FEATURE_ROOT` | Root for extracted features | `/vol/research/SignFeaturePool/features2` |
| `SLTK_NLF_MODEL_PATH` | Path to NLF model weights | — |
| `SLTK_WILOR_MODEL_PATH` | Path to WiLoR model weights | — |
| `SLTK_SIGNREP_CHECKPOINT` | Path to SignRep checkpoint | — |
| `SLTK_SEGMENTOR_PATH` | Path to segmentor checkpoint | — |

## Supported Pose Formats

| Format | Joints | Description |
|--------|--------|-------------|
| **MediaPipe** | 33 body + 21x2 hands + 468 face | Holistic pose estimation |
| **WiLoR** | 21 per hand | MANO hand model with rotation matrices |
| **NLF/SMPL-X** | 55 joints | Full body with axis-angle rotations |

All stored as HDF5 (`.h5`) files.

## Testing

```bash
# Run full suite (1429 tests)
pytest

# With coverage
pytest --cov=sltk --cov-report=html

# Specific markers
pytest -m api          # API tests only
pytest -m "not slow"   # Skip slow tests
pytest -m gpu          # GPU tests only
```

## CI

GitHub Actions runs on every push/PR to `main`:
- **Lint**: black + ruff
- **Type check**: mypy
- **Tests**: pytest across Python 3.10, 3.11, 3.12 (coverage threshold: 40%)
- **Frontend**: npm build

## License

MIT
