# Neon — Autonomous Agent Playbook

> *"Teaching robots to see time."*

This is the operating manual for any AI agent working on Neon. Read it completely before taking any action.

---

## Identity

**Project**: Neon — Open-source Vision-Language-Action model for humanoid whole-body control
**Repository**: `cagataycali/neon`
**Package**: `neon-vla` (PyPI)
**Docs**: https://cagataycali.github.io/neon
**Creator**: @cagataycali (Cagatay Cali)
**License**: MIT

---

## The One-Sentence Idea

**Video foundation models already understand motion — we connect them to robot bodies through tiny, elegant action decoders.**

---

## Architecture (Know This Cold)

```
Camera Frames ─┐
                ├─→ Video Backbone (frozen, 3-7B) ─┐
Voice/Text ────┘                                    │
                                                    ├─→ Feature Fusion ─→ Action Heads (~6M) ─→ 29 DoF Actions
Audio (16kHz) ──→ Whisper Encoder ─────────────────┘                                            └─→ Speech Out
                                                    │
Joint States ───→ Proprio Encoder (MLP) ───────────┘
```

Key files:
| File | What | Params |
|------|------|--------|
| `neon/model/neon_vla.py` | Full VLA pipeline | — |
| `neon/model/video_backbone.py` | Qwen2.5-Omni / Cosmos adapter | 3-7B (frozen) |
| `neon/model/action_heads.py` | Parameter Golf v2 decoders | ~6M (trainable) |
| `neon/model/audio.py` | Whisper encoder + PersonaPlex TTS | ~39M (frozen) |
| `neon/data/action_space.py` | G1 29-DoF joint definitions | — |
| `neon/data/data_soup.py` | Multi-source dataset mixing | — |
| `neon/data/relative_actions.py` | Cosmos relative EE actions | — |
| `neon/training/config.py` | 4 training presets | — |
| `neon/training/train.py` | NeonTrainer | — |
| `neon/inference/server.py` | HTTP inference server | — |
| `neon/inference/g1_controller.py` | Unitree SDK interface | — |

---

## Datasets — What Neon Needs

### Currently Integrated (Data Soup)

| Source | Type | HuggingFace ID | What | Episodes |
|--------|------|----------------|------|----------|
| **LeRobot Bridge** | `lerobot` | `lerobot/bridge_orig` | Tabletop manipulation (Franka) | 100K+ |
| **LeRobot DROID** | `lerobot` | `lerobot/droid` | Diverse manipulation | 70K+ |
| **Agibot-World** | `agibot` | `lerobot/xvla-agibot-world` | Bimanual manipulation | 1M+ |
| **Voice Commands** | `voice_commands` | `cagataydev/vlm-voice-commands` | 50K natural language robot instructions | 50K |
| **Voice Audio** | `voice_commands` | `cagataydev/vlm-voice-audio` | Spoken audio samples paired with text | 500+ |
| **Stereo4D** | `stereo4d` | `cagataydev/strands-kitchen-stereo4d` | Kitchen stereo video (depth priors) | ~10 |
| **GR00T-Dreams** | `dreamgen` | `NVIDIA/GR00T-Dreams-GR1` | Synthetic humanoid demonstrations | — |

### Datasets We Should Collect / Integrate Next

| Priority | Dataset | Why | How to Get |
|----------|---------|-----|------------|
| 🔴 HIGH | **G1 teleoperation** | Ground truth for OUR robot | Record via `g1_controller.py` with joystick/VR |
| 🔴 HIGH | **Cosmos Bridge** | Relative action training data | `nvidia/Cosmos-Predict2.5` synthetic pipeline |
| 🟡 MED | **Open X-Embodiment** | Massive cross-embodiment mix | `google/open-x-embodiment` (multiple subsets) |
| 🟡 MED | **DROID-100** | High-quality bimanual | `lerobot/droid_100` |
| 🟡 MED | **RH20T** | Real humanoid teleoperation | `RH20T/rh20t` |
| 🟢 LOW | **Language-Table** | Language grounding | `google/language-table` |
| 🟢 LOW | **Calvin** | Long-horizon manipulation | `calvin/calvin` |

### How to Add a New Dataset

1. Add a `DataSourceConfig` in `neon/data/data_soup.py`
2. Implement a `_load_<type>` method if new source type
3. Add embodiment mapping in `ActionMapper.EMBODIMENT_MAPS`
4. Add to a training config preset in `neon/training/config.py`
5. Test: `pytest tests/test_data.py -v`

---

## Training — How to Train Neon

### On HuggingFace (Recommended)

```bash
# Arms-only on A100 (cheapest useful config)
pip install neon-vla
python -c "
from neon.training.config import default_arms_only_config
from neon.training.train import train
train(default_arms_only_config())
"
```

Or via HF Jobs:
```bash
hf jobs uv run --flavor a100-large --secrets HF_TOKEN --timeout 8h -- \
    python -m neon.training.train --backbone Qwen/Qwen2.5-Omni-7B --mode arms_only
```

### At Home (Consumer GPU)

| GPU | Config | What Works |
|-----|--------|------------|
| **RTX 3090 (24GB)** | `edge_3b` | 3B backbone + arms_only, batch=2 |
| **RTX 4090 (24GB)** | `edge_3b` | 3B backbone + arms_only, batch=4 |
| **A100 (40GB)** | `default_arms_only` | 7B backbone + arms_only, batch=4 |
| **A100 (80GB)** | `default_wholebody` | 7B backbone + whole_body, batch=4 |
| **L40S (46GB)** | `cosmos_physics` | 8B Cosmos backbone, batch=2 |
| **Jetson Orin (32GB)** | Inference only | 3B backbone, 50ms latency |

Key settings for consumer GPUs:
```python
config = edge_3b_config()
config.batch_size = 2          # Lower for less VRAM
config.gradient_accumulation_steps = 16  # Compensate with accum
config.model.backbone.load_in_4bit = True  # Always 4-bit on consumer
```

### Training Presets

| Preset | Backbone | Mode | GPU | Use Case |
|--------|----------|------|-----|----------|
| `default_arms_only` | Qwen2.5-Omni-7B | arms_only (17 DoF) | A100 | Tabletop manipulation |
| `default_wholebody` | Qwen2.5-Omni-7B | whole_body (29 DoF) | A100 80GB | Full locomotion + manipulation |
| `cosmos_physics` | Cosmos-Reason2-8B | arms_only | A100 | Physics-heavy, sim2real |
| `edge_3b` | Qwen2.5-Omni-3B | arms_only (17 DoF) | RTX 3090/L4 | Edge/home deployment |

### Training Lessons (Non-Negotiable)

1. **Always `load_in_4bit=True`** — backbone at full precision OOMs even on A100
2. **Never unfreeze backbone on <80GB VRAM** — 7B gradient memory is ~28GB
3. **`push_to_hub=True` always** — HF Jobs ephemeral storage vanishes
4. **`paged_adamw_8bit`** — saves ~30% VRAM vs standard AdamW
5. **β₁=0.85, grad_clip=0.3** — from Parameter Golf, critical for small heads
6. **flash_attention_2 auto-detection** — our code falls back to eager if not installed
7. **First step is slow (1-3 min)** — gradient checkpointing recomputes, normal after

---

## Hardware

| Device | Role | Specs | Always On |
|--------|------|-------|:---------:|
| **Thor** 🤖 | Inference + G1 control | Jetson Orin, 32GB, CUDA | ✅ |
| **EC2** ☁️ | Training | L40S, 46GB VRAM | on-demand |
| **MacBook** 💻 | Development, tests | M3 Max, MPS | when open |
| **HF Jobs** 🏋️ | QLoRA fine-tuning | A100/L4 | per-job |

---

## Testing

```bash
# Full test suite (80 tests, no GPU needed)
pytest -v --tb=short

# Specific areas
pytest tests/test_model.py -v          # 22 tests — ReLU², RMSNorm, action heads, NeonVLA
pytest tests/test_data.py -v           # 11 tests — data soup, action mapper
pytest tests/test_audio.py -v          # 16 tests — Whisper, PersonaPlex
pytest tests/test_relative_actions.py -v  # 12 tests — Cosmos relative actions
pytest tests/test_action_space.py -v   # 19 tests — G1 joints, normalization

# Lint + type check
ruff check .
mypy neon --ignore-missing-imports
```

**All tests must pass on CPU without GPU or backbone weights.**

---

## CI/CD

| Workflow | Trigger | What |
|----------|---------|------|
| `ci.yml` | push/PR | Python 3.10-3.13, lint + mypy + pytest + build verify |
| `publish.yml` | GitHub release | OIDC → PyPI (fallback: PYPI_TOKEN) |
| `paper.yml` | `paper/` changes | Build PDFs, attach to release |
| `docs.yml` | push to main | MkDocs Material → GitHub Pages |

---

## Code Standards

1. **Type hints everywhere** — `mypy neon` must pass
2. **Docstrings on every public function** — numpy-style
3. **Raise on fatal errors** — never `logger.warning()` and continue silently
4. **Lazy imports** — heavy deps (torch, transformers) import on USE not IMPORT
5. **`ruff check .` clean** — no lint warnings
6. **80 tests minimum** — add tests for every new feature
7. **Comment non-obvious lines** — tensor shapes, magic numbers, why not just how

---

## Agent Capabilities

When working autonomously, an agent on this project can:

### ✅ Allowed
- Run tests (`pytest`)
- Run lint/type checks (`ruff`, `mypy`)
- Edit any source file in `neon/`, `tests/`, `docs/`, `paper/`
- Create new training configs in `neon/training/config.py`
- Add new data sources to `neon/data/data_soup.py`
- Launch HF training jobs via `hf_train` tool
- Push to `cagataycali/neon` (main branch)
- Build and verify the package
- Update docs and paper
- Collect/process datasets on HuggingFace

### ⛔ Not Allowed
- Delete existing tests
- Remove safety limits from `g1_controller.py`
- Unfreeze backbone without explicit approval
- Push model weights >2GB to the repo (use HF Hub)
- Modify joint limits in `action_space.py` without URDF verification

---

## Autonomous Training Workflow

When told to "train neon" or "improve neon", follow this sequence:

```
1. Check: What data sources are available?
   → List HuggingFace datasets, check data_soup.py configs

2. Check: What's the current model performance?
   → Look for training_stats.json in output dirs
   → Check HuggingFace model cards for latest metrics

3. Decide: Which config to use?
   → edge_3b for quick iteration (L4, ~2h)
   → default_arms_only for production (A100, ~6h)
   → cosmos_physics for physics tasks (A100, ~8h)

4. Launch training:
   hf_train(action="launch",
            model="Qwen/Qwen2.5-Omni-7B",
            dataset="<dataset_id>",
            output="cagataydev/neon-g1-v<next>",
            flavor="a100-large")

5. Monitor:
   hf_train(action="logs", job_id="<id>")

6. After training:
   → Update README with new model link
   → Run full test suite
   → Tag a release if loss improved
```

---

## Roadmap

| Phase | Status | What |
|-------|--------|------|
| **v0.1** | ✅ Done | Core architecture, action heads, data soup, 80 tests |
| **v0.2** | 🔄 Active | First trained weights, HF model card, edge deployment |
| **v0.3** | 📋 Planned | G1 hardware integration, real-world eval |
| **v0.4** | 📋 Planned | Multi-camera support, depth estimation |
| **v0.5** | 📋 Planned | RL fine-tuning, sim2real via Cosmos DreamGen |
| **v1.0** | 🎯 Goal | Autonomous G1 manipulation from voice commands |

---

## Key Publications

- `paper/neon.tex` — 6-page technical report (full math, proofs, pseudocode)
- `paper/soul.tex` — 1-page manifesto ("Teaching Robots to See Time")

Build with: `cd paper && pdflatex neon.tex && pdflatex soul.tex`

---

## Quick Reference

```python
# Import
from neon.model.neon_vla import NeonVLA, NeonConfig

# Create model
model = NeonVLA(NeonConfig(control_mode="arms_only"))
model.load_backbone()

# Predict
output = model.predict(
    image=camera_frame,
    instruction="Pick up the red cup",
    proprioception=joint_states,
)
# output.actions     → (16, 17) normalized
# output.raw_actions → (16, 17) denormalized joint positions
# output.upper_body  → (16, 14) arm positions
# output.locomotion  → (16, 3) velocity commands
```

---

*"The difference between seeing a photograph and watching a video is the difference between knowing and understanding."*
