# Video Backbone

The video backbone is the core visual encoder. It processes camera frames and language instructions into rich feature representations that the action heads decode into robot commands.

---

## Supported Models

| Model | Parameters | Hidden Size | VRAM (4-bit) | Recommended For |
|-------|-----------|-------------|-------------|----------------|
| `Qwen/Qwen2.5-VL-3B-Instruct` | 3B | 2048 | ~4 GB | Jetson, fast prototyping |
| `Qwen/Qwen2.5-VL-7B-Instruct` | 7B | 3584 | ~8 GB | Production, A100/RTX 4090 |
| `Qwen/Qwen2.5-Omni-3B` | 3B | 2048 | ~4 GB | Audio-native multimodal |
| `Qwen/Qwen2.5-Omni-7B` | 7B | 3584 | ~8 GB | Full multimodal |
| `nvidia/Cosmos-Reason2-2B` | 2B | 1536 | ~3 GB | Edge, physical reasoning |
| `nvidia/Cosmos-Reason2-8B` | 8B | 4096 | ~10 GB | Complex scenarios |

---

## Configuration

```python
from neon.model.video_backbone import BackboneConfig, VideoBackbone

config = BackboneConfig(
    model_id="Qwen/Qwen2.5-VL-3B-Instruct",
    torch_dtype="bfloat16",
    device_map="auto",
    attn_implementation="eager",  # Use "flash_attention_2" if installed
    fps=4,                         # Video frame rate
    max_frames=16,                 # Max frames per video clip
    image_size=224,
    freeze_backbone=True,          # Freeze for action head training
    load_in_4bit=True,             # 4-bit quantization
    use_double_quant=True,         # Double quantization for extra savings
)
```

### Key Parameters

`freeze_backbone`
: **True (default)** — Only action heads train. Fast, data-efficient. Recommended unless you have >100K episodes.
: **False** — Backbone also trains (with LoRA). Needs 40+ GB VRAM and lots of data.

`load_in_4bit`
: Quantizes backbone weights to 4-bit NF4. Reduces VRAM by ~4× with minimal quality loss. Essential for training on consumer GPUs.

`fps`
: How many frames per second when processing video input. Higher = more temporal detail, more compute. 4 FPS is a good default for 50 Hz control.

---

## Loading and Encoding

The backbone loads lazily — weights aren't downloaded until you call `load()`:

```python
backbone = VideoBackbone(config)
# Nothing downloaded yet

backbone.load()
# Now the model is in GPU memory

# Encode a single image + instruction
result = backbone.encode(
    images=[pil_image],
    text="Pick up the red cup",
)
# result["hidden_states"]: (1, seq_len, hidden_size) — full token features
# result["pooled"]:        (1, hidden_size) — mean-pooled representation
```

### Video Input

For temporal reasoning, pass multiple frames:

```python
result = backbone.encode(
    video_frames=[frame_t3, frame_t2, frame_t1, frame_t0],
    text="Pick up the moving ball",
)
# The backbone processes all 4 frames with temporal attention
```

---

## How Features Flow to Action Heads

```
backbone.encode(images, text)
         │
         ▼
    hidden_states: (1, seq_len, 2048)  ← Full token-level features
    pooled:        (1, 2048)           ← Mean across all tokens
         │
         ▼ (pooled is used)
    concat with proprioception and audio features
         │
         ▼
    fusion: Linear(concat_dim → 2048) + GELU
         │
         ▼
    action_head: (1, 2048) → (1, 16, 17)  ← Action chunk
```

The pooled representation compresses all visual and language understanding into a single vector. This is then fused with proprioception (current joint states) and optionally audio features before being decoded into actions.

!!! note "Why pooled instead of full hidden states?"
    Token-level features carry rich information but are variable-length. Pooling to a fixed-size vector simplifies the action head architecture. Future work: cross-attention between action queries and backbone tokens (like DETR).

---

## LoRA Fine-Tuning (Optional)

When the backbone is unfrozen, Neon applies LoRA to avoid full fine-tuning:

```python
config = NeonConfig(
    backbone=BackboneConfig(
        model_id="Qwen/Qwen2.5-VL-7B-Instruct",
        freeze_backbone=False,  # Unfreeze for LoRA
    ),
)

# In TrainConfig:
train_config = TrainConfig(
    use_lora=True,
    lora_r=32,
    lora_alpha=64,
    lora_target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)
```

LoRA adds ~0.5% trainable parameters to the backbone while preserving pre-trained capabilities. This is only needed when:

- You have domain-specific visual environments (e.g., underwater, industrial)
- You have 100K+ episodes of training data
- You have A100-class GPUs with 40+ GB VRAM
