# Why Video Models?

Most Vision-Language-Action (VLA) models use **image encoders** — CLIP, SigLIP, DINOv2 — that process a single frame at a time. Neon takes a fundamentally different approach: it uses **video foundation models** as the visual backbone.

This page explains why, and what it changes.

---

## The Problem with Image Encoders

An image encoder sees a frozen moment. When you show it a robot arm reaching toward a cup, it sees *a robot arm near a cup*. It doesn't see:

- The **direction** the arm is moving
- The **velocity** of the approach
- Whether the cup is **sliding** or stationary
- What **just happened** in the previous frame
- What is **about to happen** next

Standard VLAs compensate by stacking multiple frames, adding positional embeddings, or using recurrence. But they're bolting temporal reasoning onto a model that was never designed for it.

---

## What Video Models Already Know

Video foundation models like [Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) and [Cosmos-Reason2](https://github.com/NVIDIA/Cosmos-Reason2) are pre-trained on millions of hours of video. They already understand:

| Capability | Why it matters for robotics |
|------------|----------------------------|
| **Object permanence** | Track objects through occlusion — the cup is still there behind the robot's arm |
| **Physics intuition** | Predict how objects fall, slide, stack — essential for manipulation |
| **Motion prediction** | "What happens next" is literally what video models are trained to do |
| **Temporal coherence** | Natural understanding of smooth trajectories, not jerky frame-by-frame reasoning |
| **Spatial reasoning** | Depth from motion parallax, 3D layout from temporal cues |

```mermaid
graph TD
    subgraph "Image Encoder (CLIP)"
        I1["Frame t"] --> IE["Image<br/>Encoder"]
        IE --> F1["Static features"]
    end

    subgraph "Video Model (Qwen2.5-VL)"
        V1["Frame t-3"] --> VE["Video<br/>Encoder"]
        V2["Frame t-2"] --> VE
        V3["Frame t-1"] --> VE
        V4["Frame t"] --> VE
        VE --> F2["Temporal features<br/>(motion · physics · prediction)"]
    end

    style IE fill:#666,color:#fff
    style VE fill:#e65100,color:#fff
```

---

## Two Backbone Options

Neon supports two video backbone families:

### Qwen2.5-VL (Default)

Alibaba's Vision-Language model with native video understanding. Processes interleaved images and video with temporal attention.

| Variant | Parameters | VRAM (4-bit) | Best for |
|---------|-----------|-------------|----------|
| `Qwen2.5-VL-3B` | 3B | ~4 GB | Jetson Orin, fast iteration |
| `Qwen2.5-VL-7B` | 7B | ~8 GB | Production quality |

**Strengths:** Language grounding, instruction following, rich spatial understanding.

### Cosmos-Reason2

NVIDIA's Physical AI reasoning model, built on top of Qwen3-VL but fine-tuned specifically for physical world reasoning.

| Variant | Parameters | VRAM (4-bit) | Best for |
|---------|-----------|-------------|----------|
| `Cosmos-Reason2-2B` | 2B | ~3 GB | Edge deployment |
| `Cosmos-Reason2-8B` | 8B | ~10 GB | Complex physical reasoning |

**Strengths:** Physics-grounded predictions, world model understanding, sim-to-real transfer.

---

## How Neon Uses the Backbone

The backbone is **frozen** during training by default. Neon treats it as a feature extractor:

```
Video frames + text instruction
       │
       ▼
┌──────────────────┐
│  Frozen Backbone  │  ← Pre-trained video foundation model
│  (3B-7B params)   │     Already understands motion & physics
└────────┬─────────┘
         │
    Pooled features
    (batch, 2048-3584)
         │
         ▼
┌──────────────────┐
│  Trainable Heads  │  ← Only these are trained (~2M params)
│  Action Decoder   │     Arms, locomotion, grippers
└──────────────────┘
```

**Why freeze the backbone?**

1. **Parameter efficiency** — Train 2M parameters instead of 3B
2. **Data efficiency** — Fewer trainable params = less data needed to converge
3. **Speed** — Backbone forward pass doesn't compute gradients
4. **Stability** — Pre-trained features are already excellent; fine-tuning risks catastrophic forgetting

If you have enough data (>100K episodes) and compute (A100+), you can optionally apply **LoRA** to the backbone for domain-specific fine-tuning.

---

## Compared to Other Approaches

| Approach | Visual Encoder | Temporal Handling | Used By |
|----------|---------------|-------------------|---------|
| RT-2 | ViT (image) | Frame stacking | Google |
| Octo | ResNet (image) | Transformer history | Berkeley |
| GR00T N1 | Eagle-2 (image) | Dual-system architecture | NVIDIA |
| OpenVLA | SigLIP (image) | Single frame | Stanford |
| **Neon** | **Qwen2.5-VL (video)** | **Native temporal attention** | — |

The key difference: Neon doesn't need to *learn* temporal reasoning from robot data. The video backbone already has it. We just need to teach it *what actions to take* given what it sees.
