# Architecture

How Neon is structured internally.

---

## Package Structure

```
neon/
├── model/
│   ├── video_backbone.py       # Qwen2.5-VL / Cosmos-Reason2 adapter
│   ├── action_heads.py         # Parameter Golf v2 decoders (ReLU², RMSNorm, soft-cap)
│   ├── audio.py                # AudioEncoder (Whisper/Omni), SpeechResponseHead, PersonaPlex
│   └── neon_vla.py             # NeonVLA — the complete model pipeline
├── data/
│   ├── action_space.py         # G1 29-DoF joint definitions, normalization, modes
│   ├── data_soup.py            # Multi-source mixing (7 types), cross-embodiment mapping
│   └── relative_actions.py     # Cosmos-Predict2.5 relative EE actions
├── training/
│   ├── config.py               # TrainConfig + 4 presets (arms, wholebody, cosmos, edge)
│   └── train.py                # NeonTrainer — full training loop
├── inference/
│   ├── server.py               # HTTP inference server with temporal smoothing
│   └── g1_controller.py        # Unitree G1 SDK interface, safety limits
├── tools/
│   └── neon_tool.py            # Strands Agent tool integration
└── tests/                      # 80 tests across 5 files
```

---

## Component Hierarchy

```mermaid
graph TD
    NV["NeonVLA<br/><em>Complete model</em>"]
    NV --> VB["VideoBackbone<br/><em>Qwen2.5-Omni / Cosmos</em>"]
    NV --> AE["AudioEncoder<br/><em>Whisper / Mel / Omni</em>"]
    NV --> PE["ProprioceptionEncoder<br/><em>MLP: joints → features</em>"]
    NV --> FUS["Fusion Layer<br/><em>Linear + ReLU²</em>"]
    NV --> AH["G1ActionHead"]
    NV --> SH["SpeechResponseHead"]

    AH --> ACH["ActionChunkingHead<br/><em>Arms (14 DoF)</em><br/>RMSNorm + ReLU² + SoftCap"]
    AH --> ACH2["ActionChunkingHead<br/><em>Locomotion (3 DoF)</em><br/>RMSNorm + ReLU² + SoftCap"]
    AH --> MLP["MLPDecoder<br/><em>Torso + Head (3 DoF)</em><br/>Skip + ResidualScale"]
    AH --> MLP2["MLPDecoder<br/><em>Legs (12 DoF)</em><br/>Skip + ResidualScale"]

    style NV fill:#e65100,color:#fff
    style VB fill:#1565c0,color:#fff
    style AE fill:#1565c0,color:#fff
    style AH fill:#1b5e20,color:#fff
```

---

## Data Flow — Training

```mermaid
sequenceDiagram
    participant DS as DataSoupDataset
    participant VB as Video Backbone (frozen)
    participant PE as Proprio Encoder
    participant AE as Audio Encoder
    participant FUS as Fusion (ReLU²)
    participant AH as Action Heads (v2)
    participant OPT as Adam (β₁=0.85)

    DS->>VB: images + text
    VB->>FUS: visual-language features (2048)
    DS->>PE: joint states
    PE->>FUS: proprio features (256)
    DS->>AE: audio waveform (optional)
    AE->>FUS: audio features (2048)
    FUS->>AH: fused features (2048)
    AH->>AH: RMSNorm → ReLU² → Skip → SoftCap
    Note over AH: MSE loss vs target actions
    AH->>OPT: gradients (clip=0.3)
    OPT->>AH: parameter update
```

---

## Data Flow — Inference

```mermaid
sequenceDiagram
    participant CAM as Camera
    participant MIC as Microphone
    participant SDK as Joint States
    participant NV as NeonVLA
    participant PP as PersonaPlex
    participant G1 as G1 Robot

    CAM->>NV: camera frame
    MIC->>NV: spoken command (16kHz)
    SDK->>NV: current joint positions
    NV->>NV: backbone → fusion → action heads
    NV->>G1: action chunk (16 steps × 17 DoF)
    NV->>PP: speech text
    PP->>G1: audio playback
```

---

## Data Pipeline

```mermaid
graph LR
    subgraph "7 Source Types"
        LR["LeRobot"]
        AG["Agibot"]
        DG["DreamGen"]
        COS["Cosmos DreamGen"]
        OXE["OXE"]
        VC["Voice Commands"]
        S4D["Stereo4D"]
    end

    subgraph "Processing"
        MAP["ActionMapper<br/>Cross-embodiment"]
        REL["Relative Actions<br/>Cosmos EE deltas"]
        NORM["Normalize<br/>to G1 space"]
    end

    LR --> MAP
    AG --> MAP
    DG --> MAP
    COS --> REL
    OXE --> MAP
    MAP --> NORM
    REL --> NORM
    VC --> SOUP
    S4D --> SOUP
    NORM --> SOUP["DataSoup<br/>Weighted mixing"]

    style SOUP fill:#e65100,color:#fff
```

---

## Configuration Cascade

```mermaid
graph LR
    TC["TrainConfig"] --> NC["NeonConfig"]
    TC --> DSC["DataSoupConfig"]
    TC --> HYP["adam_β₁=0.85<br/>grad_clip=0.3"]
    NC --> BC["BackboneConfig"]
    NC --> AHC["ActionHeadConfig<br/>ReLU², RMSNorm, SoftCap"]
    NC --> AC["AudioConfig"]
    DSC --> SRC["DataSourceConfig[]<br/>7 source types"]

    style TC fill:#333,color:#fff
    style AHC fill:#e65100,color:#fff
```

---

## Design Principles

### 1. Frozen Backbone, Trainable Heads

The video backbone is a 3-7B parameter model pre-trained on millions of hours of video. We freeze it and only train the ~2M parameter action decoder. This means:

- Train with 100× less data than end-to-end approaches
- No catastrophic forgetting of visual understanding
- Action heads benefit from Parameter Golf v2 optimizations (every param counts)

### 2. Progressive Scaling

Start with `arms_only` (17 DoF), scale to `whole_body` (29 DoF) as you get more data and compute. Each control mode adds joints without changing existing heads.

### 3. Cross-Embodiment Actions

Cosmos relative actions (`rel_xyz = R₁ᵀ·(xyz₂-xyz₁)`) mean data from any robot can train a G1. Same physical movement = same numbers.

### 4. Modular Multimodality

Audio is optional. Speech output is optional. Every modality can be independently enabled, disabled, or swapped. The fusion layer handles variable input dimensionality.

### 5. Safety by Default

The G1 controller enforces joint limits, velocity limits, and locomotion limits before every command. Safety checks cannot be bypassed in the default configuration.
