% ============================================================================
% Neon: Open-Source Vision-Language-Action Model for Humanoid Whole-Body Control
% Technical Report / Whitepaper
% ============================================================================
\documentclass[10pt,twocolumn]{article}

% ─── Packages ──────────────────────────────────────────────────────────────
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{mathtools}
\usepackage{bm}
% algorithm/algorithmic — use algpseudocode if available, else fallback to verbatim
\IfFileExists{algorithm.sty}{
  \usepackage{algorithm}
  \usepackage{algorithmic}
  \newcommand{\algoenabled}{1}
}{
  \newcounter{algorithm}
  \newenvironment{algorithm}[1][htbp]{\begin{figure}[#1]\small}{\end{figure}}
  \newenvironment{algorithmic}[1][]{\begin{quote}\ttfamily\small}{\end{quote}}
  \newcommand{\REQUIRE}{\textbf{Input: }}
  \newcommand{\STATE}{\par\hangindent=1.5em}
  \newcommand{\IF}[1]{\par\textbf{if} #1 \textbf{then}}
  \newcommand{\ELSE}{\par\textbf{else}}
  \newcommand{\ENDIF}{\par\textbf{end if}}
  \newcommand{\FOR}[1]{\par\textbf{for} #1 \textbf{do}}
  \newcommand{\ENDFOR}{\par\textbf{end for}}
  \newcommand{\RETURN}{\par\textbf{return} }
  \newcommand{\COMMENT}[1]{\hfill\textit{// #1}}
  \newcommand{\algoenabled}{0}
}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{hyperref}
\usepackage{xcolor}
\usepackage{microtype}
\usepackage{enumitem}
\usepackage{subcaption}
\usepackage[margin=0.85in]{geometry}
\usepackage{float}

% ─── Theorem-like environments ─────────────────────────────────────────────
\newtheorem{proposition}{Proposition}
\newtheorem{definition}{Definition}

% ─── Colors ────────────────────────────────────────────────────────────────
\definecolor{neonorange}{HTML}{E65100}
\hypersetup{
    colorlinks=true,
    linkcolor=neonorange,
    citecolor=neonorange,
    urlcolor=neonorange,
}

% ─── Math shortcuts ───────────────────────────────────────────────────────
\newcommand{\R}{\mathbb{R}}
\newcommand{\vx}{\bm{x}}
\newcommand{\va}{\bm{a}}
\newcommand{\vs}{\bm{s}}
\newcommand{\vf}{\bm{f}}
\newcommand{\vq}{\bm{q}}
\newcommand{\vp}{\bm{p}}
\newcommand{\vh}{\bm{h}}
\newcommand{\T}{\mathsf{T}}
\newcommand{\norm}[1]{\left\lVert #1 \right\rVert}
\DeclareMathOperator*{\softcap}{softcap}
\DeclareMathOperator*{\relu}{ReLU}
\DeclareMathOperator*{\rmsnorm}{RMSNorm}
\DeclareMathOperator*{\concat}{concat}
\DeclareMathOperator*{\atan}{atan2}

% ============================================================================
\title{
    \textbf{Neon}: Open-Source Vision-Language-Action Model\\
    for Humanoid Whole-Body Control\\[6pt]
    {\normalsize\textit{Video Foundation Models + Parameter-Efficient Action Decoders + Cross-Embodiment Data}}
}

\author{
    Cagatay Cali\\
    \texttt{cagatay@amazon.com}\\
    \url{https://github.com/cagataycali/neon}
}

\date{March 2026}

\begin{document}
\maketitle

% ============================================================================
\begin{abstract}
We present \textbf{Neon}, an open-source Vision-Language-Action (VLA) model for whole-body humanoid control on the Unitree G1 robot. Neon replaces the traditional image encoder with a \emph{video foundation model} (Qwen2.5-Omni or Cosmos-Reason2) that inherently understands motion, physics, and temporal dynamics. On top of the frozen backbone, we train lightweight action decoder heads ($\sim$2M parameters) using techniques from the Parameter Golf competition: ReLU$^2$ activations, RMSNorm, learnable residual scales, U-Net skip connections, and logit soft-capping. For cross-embodiment training, we adopt the Cosmos-Predict2.5 relative end-effector action representation, enabling a unified ``data soup'' of seven heterogeneous source types including LeRobot, Agibot-World, GR00T-Dreams, Stereo4D, and spoken commands. Neon supports 29 degrees of freedom with action chunking (16-step prediction), optional audio input via Whisper encoding, and speech output via PersonaPlex TTS. The full system runs at 50ms latency on NVIDIA Jetson Orin. We release the code, model weights, training configs, and this paper under the MIT license.
\end{abstract}

% ============================================================================
\section{Introduction}
\label{sec:intro}

Recent work on Vision-Language-Action (VLA) models has demonstrated that large pre-trained vision-language models can serve as powerful backbones for robotic control~\cite{rt2,openvla,pi0}. However, most approaches use \emph{image} encoders that process individual frames independently, discarding the temporal information essential for understanding motion and physical dynamics.

We argue that \textbf{video foundation models} are a more natural backbone for robot control. Models like Qwen2.5-Omni~\cite{qwen2.5omni} and Cosmos-Reason2~\cite{cosmos} are pre-trained on millions of hours of video and already encode concepts of motion, object permanence, contact dynamics, and spatial-temporal reasoning. Rather than learning these from scratch on limited robot data, we can \emph{reuse} this understanding.

\textbf{Neon} makes this concrete. Our contributions:
\begin{enumerate}[leftmargin=*,itemsep=2pt]
    \item A VLA architecture that uses a frozen video backbone ($3$--$7$B parameters) with lightweight trainable action heads ($\sim$2M parameters), training only $0.2\%$ of total parameters.
    \item Parameter Golf v2 action decoders incorporating six techniques from the OpenAI Parameter Golf competition optimized for small modules.
    \item A cross-embodiment relative action representation (from Cosmos-Predict2.5) enabling training on heterogeneous data sources.
    \item A ``data soup'' pipeline mixing seven source types with automatic action mapping to the G1's 29 DoF action space.
    \item Integrated audio input (Whisper) and speech output (PersonaPlex TTS) for voice-controlled humanoid operation.
    \item Open-source release of code, configs, and $80$ unit tests: \url{https://github.com/cagataycali/neon}.
\end{enumerate}

% ============================================================================
\section{Related Work}
\label{sec:related}

\paragraph{Vision-Language-Action Models.}
RT-2~\cite{rt2} first showed that VLMs can directly output robot actions as text tokens. OpenVLA~\cite{openvla} scaled this to 7B parameters with open weights. $\pi_0$~\cite{pi0} introduced flow matching for continuous action spaces. All use image-based encoders.

\paragraph{Video Understanding for Robotics.}
GR00T-N1~\cite{groot} pioneered video-conditioned action generation using a dual-system architecture. Cosmos-Reason2~\cite{cosmos} provides physical AI reasoning over video. Neon builds on these by treating the video model as a \emph{frozen feature extractor}.

\paragraph{Action Representations.}
Cosmos-Predict2.5 introduced relative end-effector actions in the gripper frame for cross-embodiment transfer. ACT~\cite{act} introduced action chunking. Neon combines both: relative gripper-frame actions chunked into 16-step predictions.

\paragraph{Parameter-Efficient Training.}
LoRA~\cite{lora} and QLoRA~\cite{qlora} reduce trainable parameters for fine-tuning. The OpenAI Parameter Golf competition pushed techniques for maximizing performance per parameter byte. We apply these techniques to action decoders rather than language model adapters.

% ============================================================================
\section{Architecture}
\label{sec:architecture}

Neon follows a \emph{frozen backbone, trainable heads} paradigm. The system takes multimodal inputs (camera frames, language instruction, joint states, optional audio) and outputs action chunks for the G1 humanoid.

\subsection{Video Backbone}
\label{sec:backbone}

Let $\mathcal{V} = \{I_1, \ldots, I_T\}$ be a sequence of camera frames and $\ell$ a language instruction. The video backbone $\phi$ produces visual-language features:
\begin{equation}
    \vf_\text{vis} = \phi(\mathcal{V}, \ell) \in \R^{d_\text{bb}}
    \label{eq:backbone}
\end{equation}
where $d_\text{bb} = 2048$ for Qwen2.5-Omni-7B and $d_\text{bb} = 4096$ for Cosmos-Reason2-8B. The backbone parameters are \textbf{frozen} during training; we use 4-bit quantization (NF4) to reduce memory from $\sim$14GB to $\sim$4GB.

\subsection{Proprioception Encoder}
\label{sec:proprio}

Current joint positions $\vq \in \R^{n_q}$ (where $n_q$ is the action dimension) are encoded by a two-layer MLP:
\begin{equation}
    \vf_\text{prop} = \sigma\!\left(W_2 \, \sigma(W_1 \vq + b_1) + b_2\right) \in \R^{d_p}
    \label{eq:proprio}
\end{equation}
with $d_p = 256$ and $\sigma = \text{GELU}$.

\subsection{Audio Encoder (Optional)}
\label{sec:audio}

When audio input $\va_\text{wav} \in \R^{16000 \cdot t}$ (16kHz waveform of $t$ seconds) is provided, we encode it via a Whisper-derived mel spectrogram encoder:
\begin{equation}
    \vf_\text{audio} = \psi(\va_\text{wav}) \in \R^{d_\text{bb}}
    \label{eq:audio}
\end{equation}

Three encoder types are supported: \texttt{whisper} (OpenAI Whisper weights), \texttt{mel\_spectrogram} (lightweight mel + MLP), and \texttt{omni} (native Qwen2.5-Omni audio tower).

\subsection{Feature Fusion}
\label{sec:fusion}

All available features are concatenated and projected back to the backbone dimension:
\begin{equation}
    \vf = \text{Proj}\!\left(\concat\!\left[\vf_\text{vis},\, \vf_\text{prop},\, \vf_\text{audio}\right]\right) \in \R^{d_\text{bb}}
    \label{eq:fusion}
\end{equation}
where $\text{Proj}: \R^{d_\text{vis} + d_p + d_\text{audio}} \to \R^{d_\text{bb}}$ is a linear layer followed by GELU and dropout. This bottleneck forces multimodal compression.

\subsection{Action Heads — Parameter Golf v2}
\label{sec:heads}

The action heads are the \emph{only} trainable components (besides fusion and proprioception encoder). We use separate decoder heads per joint group:

\begin{itemize}[leftmargin=*,itemsep=1pt]
    \item \textbf{Arms}: ActionChunkingHead $\to \R^{T_c \times 14}$
    \item \textbf{Locomotion}: ActionChunkingHead $\to \R^{T_c \times 3}$
    \item \textbf{Torso+Head}: MLPDecoder $\to \R^{3}$ (no chunking)
    \item \textbf{Legs}: MLPDecoder $\to \R^{12}$ (whole-body mode)
\end{itemize}

where $T_c = 16$ is the chunk size (number of predicted future timesteps).

% ============================================================================
\section{Parameter Golf v2 Techniques}
\label{sec:param_golf}

Since the action heads constitute our ``small model'' ($\sim$2M parameters), every parameter-efficiency technique matters. We port six techniques from the OpenAI Parameter Golf competition winners:

\subsection{ReLU$^2$ Activation}

Standard ReLU has a discontinuous gradient at zero. GELU is smooth but expensive. We use ReLU$^2$:
\begin{equation}
    \text{ReLU}^2(x) = \max(0, x)^2
    \label{eq:relu2}
\end{equation}

The gradient $\frac{\partial}{\partial x}\text{ReLU}^2(x) = 2\max(0,x)$ is \emph{continuous} at zero (unlike ReLU's step function) and requires only a comparison and multiply (unlike GELU's $\exp$). For action decoders, this produces smoother output trajectories.

\subsection{RMSNorm}

Layer Normalization computes both mean and variance. RMSNorm~\cite{rmsnorm} drops the mean centering:
\begin{equation}
    \rmsnorm(\vx) = \frac{\vx}{\sqrt{\frac{1}{d}\sum_{i=1}^d x_i^2 + \varepsilon}} \odot \bm{\gamma}
    \label{eq:rmsnorm}
\end{equation}
where $\bm{\gamma} \in \R^d$ is a learnable scale. This saves one reduction operation and one subtraction per element while providing equivalent training stability for small MLPs.

\subsection{Learnable Residual Scales}

In standard residual connections, $\vh' = \vh + f(\vh)$, the residual $\vh$ can dominate early in training when the backbone features are much larger in magnitude than the randomly-initialized head features. We introduce a learnable scale:
\begin{equation}
    \vh' = f(\vh) + \alpha \cdot \vh, \quad \alpha = \text{Parameter}(\text{init}=1.0)
    \label{eq:res_scale}
\end{equation}

The network learns to scale residuals per layer. In practice, $\alpha$ converges to values between $0.3$ and $0.8$, confirming that the raw residual needs attenuation.

\subsection{U-Net Skip Connections}

For MLPs with $L \geq 3$ layers, we save the first hidden layer output $\vh_0$ and concatenate it with the last hidden layer input:
\begin{equation}
    \vh_{L-1} = g\!\left(\concat[\vh_{L-2},\, \vh_0]\right)
    \label{eq:skip}
\end{equation}

This creates a gradient ``highway'' through deep decoders, preventing the middle layers from becoming a gradient bottleneck. The concatenation doubles the input dimension of the last hidden layer, requiring a projection back to $d_h$.

\subsection{Logit Soft-Capping}

Hard Tanh ($\tanh(x)$) clamps outputs to $[-1, 1]$ but has zero gradient when $|x| > 1$. This is problematic for action boundary values (e.g., joint limits). Soft-capping provides asymptotic bounding:
\begin{equation}
    \softcap(x; c) = c \cdot \tanh\!\left(\frac{x}{c}\right)
    \label{eq:softcap}
\end{equation}
where $c = 1.0$ for normalized actions. The gradient $\frac{\partial}{\partial x}\softcap(x;c) = 1 - \tanh^2(x/c)$ is always nonzero, maintaining gradient flow at the boundaries. From~\cite{gemma2}.

\subsection{Optimizer Tuning}

We use Adam with $\beta_1 = 0.85$ (vs.\ default $0.9$) for faster adaptation to shifting action distributions during training, $\beta_2 = 0.99$ for reduced momentum lag, and gradient clipping at $\norm{\nabla}_\text{max} = 0.3$ (vs.\ typical $1.0$) to prevent divergence in small heads.

% ============================================================================
\section{Action Space and Representations}
\label{sec:action_space}

\subsection{G1 Humanoid — 29 Degrees of Freedom}

The Unitree G1 has 29 controllable joints organized into groups:

\begin{table}[H]
\centering
\small
\begin{tabular}{lcc}
\toprule
\textbf{Group} & \textbf{Joints} & \textbf{DoF} \\
\midrule
Left arm & shoulder (3), elbow (1), wrist (3) & 7 \\
Right arm & shoulder (3), elbow (1), wrist (3) & 7 \\
Torso & waist yaw & 1 \\
Head & pitch, yaw & 2 \\
Left leg & hip (3), knee (1), ankle (2) & 6 \\
Right leg & hip (3), knee (1), ankle (2) & 6 \\
\midrule
\textbf{Total} & & \textbf{29} \\
\bottomrule
\end{tabular}
\caption{G1 action space decomposition.}
\label{tab:action_space}
\end{table}

Three progressive control modes select subsets:
\begin{itemize}[leftmargin=*,itemsep=1pt]
    \item \texttt{arms\_only}: 14 arm joints + 3 locomotion = 17 DoF
    \item \texttt{upper\_body}: 14 + 3 + 3 (torso/head) = 20 DoF
    \item \texttt{whole\_body}: all 29 DoF
\end{itemize}

All actions are normalized to $[-1, 1]$ using per-joint min/max limits derived from the G1 URDF.

\subsection{Action Chunking}
\label{sec:chunking}

Following ACT~\cite{act}, we predict $T_c = 16$ future timesteps simultaneously instead of a single action. At 50Hz control frequency, this covers 320ms of future motion.

For the ActionChunkingHead, each timestep $t \in \{1, \ldots, T_c\}$ gets a learnable embedding $\bm{e}_t \in \R^{d_h}$. The predicted action at step $t$ is:
\begin{equation}
    \hat{\va}_t = \text{MLP}\!\left(\concat[\text{Proj}(\vf),\, \bm{e}_t]\right) \in \R^{n_a}
    \label{eq:chunking}
\end{equation}
where $n_a$ is the per-head action dimension.

The training loss is:
\begin{equation}
    \mathcal{L} = \frac{1}{T_c \cdot n_a} \sum_{t=1}^{T_c} \sum_{i=1}^{n_a} m_t \left(\hat{a}_{t,i} - a^*_{t,i}\right)^2
    \label{eq:loss}
\end{equation}
where $m_t \in \{0, 1\}$ masks padded timesteps and $a^*$ are ground-truth actions.

\subsection{Cosmos Relative Actions}
\label{sec:relative}

For cross-embodiment transfer, we adopt the relative end-effector representation from Cosmos-Predict2.5. Given consecutive gripper states with positions $\vp_{t-1}, \vp_t \in \R^3$ and rotation matrices $R_{t-1}, R_t \in \text{SO}(3)$:

\begin{align}
    \Delta\vp &= R_{t-1}^\T \left(\vp_t - \vp_{t-1}\right) \in \R^3 \label{eq:rel_pos} \\
    \Delta R &= R_{t-1}^\T R_t \in \text{SO}(3) \label{eq:rel_rot} \\
    \Delta\bm{\theta} &= \text{euler}(\Delta R) \in \R^3 \label{eq:rel_euler}
\end{align}

The full relative action is $\va^\text{rel} = [\lambda \Delta\vp,\; \lambda \Delta\bm{\theta},\; g_t]$ where $\lambda = 20$ is a scaling factor and $g_t \in [0, 1]$ is the gripper state.

\begin{proposition}[Cross-embodiment invariance]
For any two robots $A$ and $B$ performing the same physical end-effector motion in different workspaces with different coordinate origins $O_A, O_B$ and base orientations $R_A^0, R_B^0$, the relative actions satisfy $\va_A^\text{rel} = \va_B^\text{rel}$.
\end{proposition}

\begin{proof}
Let $\vp_t^A = R_A^0 \vp_t^w + O_A$ be robot $A$'s gripper position, where $\vp_t^w$ is the world-frame position. Then:
\begin{align*}
    R_{t-1}^{A\T}(\vp_t^A - \vp_{t-1}^A) &= R_{t-1}^{A\T} R_A^0 (\vp_t^w - \vp_{t-1}^w)
\end{align*}
Since $R_t^A = R_A^0 R_t^w$, we have $R_{t-1}^{A\T} R_A^0 = R_{t-1}^{w\T} R_A^{0\T} R_A^0 = R_{t-1}^{w\T}$. The same holds for robot $B$, giving $\Delta\vp^A = \Delta\vp^B = R_{t-1}^{w\T}(\vp_t^w - \vp_{t-1}^w)$. An identical argument applies to the rotation delta. \qed
\end{proof}

The rotation conversions use ZYX Euler angles. Given $R \in \text{SO}(3)$:
\begin{align}
    \text{roll} &= \atan(R_{21}, R_{22}) \label{eq:euler_roll}\\
    \text{pitch} &= \atan(-R_{20}, \sqrt{R_{00}^2 + R_{10}^2}) \label{eq:euler_pitch}\\
    \text{yaw} &= \atan(R_{10}, R_{00}) \label{eq:euler_yaw}
\end{align}

% ============================================================================
\section{Data Soup}
\label{sec:data}

\subsection{Multi-Source Pipeline}

Neon trains on a weighted mixture of seven data source types:

\begin{table}[H]
\centering
\small
\begin{tabular}{lll}
\toprule
\textbf{Source Type} & \textbf{Action Repr.} & \textbf{Example} \\
\midrule
\texttt{lerobot} & Joint positions & Bridge, DROID \\
\texttt{agibot} & Joint positions & Agibot-World (1M+) \\
\texttt{dreamgen} & IDM extraction & GR00T-Dreams \\
\texttt{cosmos\_dreamgen} & Relative EE & Cosmos data \\
\texttt{oxe} & Joint positions & Open X-Embodiment \\
\texttt{voice\_commands} & Language only & Audio + text \\
\texttt{stereo4d} & Visual only & Stereo pairs \\
\bottomrule
\end{tabular}
\caption{Data soup source types. Each normalizes to a common \texttt{NeonEpisode} format.}
\label{tab:data_sources}
\end{table}

\subsection{Cross-Embodiment Action Mapping}

For joint-position sources, an \texttt{ActionMapper} remaps source joint indices to G1 indices with zero-padding for unmatched joints. Formally, given source actions $\va^\text{src} \in \R^{n_s}$ and a mapping $\pi: \{1,\ldots,n_s\} \to \{1,\ldots,29\}$:
\begin{equation}
    a_j^\text{G1} = \begin{cases} a_{\pi^{-1}(j)}^\text{src} & \text{if } j \in \text{Im}(\pi) \\ 0 & \text{otherwise} \end{cases}
    \label{eq:action_map}
\end{equation}

\subsection{Weighted Mixing}

Each source $s$ has a weight $w_s > 0$. During training, source $s$ is sampled with probability $p_s = w_s / \sum_k w_k$. Optional \texttt{max\_episodes} caps per-source contribution.

% ============================================================================
\section{Training}
\label{sec:training}

\subsection{Frozen Backbone, Trainable Heads}

Only the action heads, fusion layer, and proprioception encoder are trained. For a model with backbone parameters $\theta_\text{bb}$ and head parameters $\theta_\text{head}$:
\begin{equation}
    \min_{\theta_\text{head}} \; \mathbb{E}_{(\mathcal{V}, \ell, \vq, \va^*) \sim \mathcal{D}} \left[ \mathcal{L}\!\left(\hat{\va}(\mathcal{V}, \ell, \vq; \theta_\text{bb}, \theta_\text{head}),\; \va^*\right) \right]
    \label{eq:training}
\end{equation}
where $\theta_\text{bb}$ is fixed. The backbone is loaded in 4-bit (NF4) quantization~\cite{qlora}.

\subsection{Hyperparameters}

\begin{table}[H]
\centering
\small
\begin{tabular}{ll}
\toprule
\textbf{Parameter} & \textbf{Value} \\
\midrule
Optimizer & Paged AdamW 8-bit \\
Learning rate & $2 \times 10^{-4}$ \\
$\beta_1, \beta_2$ & $0.85, 0.99$ \\
Weight decay & $0.01$ \\
LR schedule & Cosine with 10\% warmup \\
Max grad norm & $0.3$ \\
Batch size & $4$ ($\times 8$ accumulation $= 32$ effective) \\
Precision & BFloat16 \\
Epochs & $3$ \\
\bottomrule
\end{tabular}
\caption{Training hyperparameters.}
\label{tab:hyperparams}
\end{table}

\subsection{Parameter Budget}

\begin{table}[H]
\centering
\small
\begin{tabular}{lr}
\toprule
\textbf{Component} & \textbf{Parameters} \\
\midrule
Video backbone (frozen) & $3$--$7$B \\
\midrule
Arms ActionChunkingHead & $\sim$1.5M \\
Locomotion ActionChunkingHead & $\sim$400K \\
Fusion layer & $\sim$4.2M \\
Proprioception encoder & $\sim$100K \\
RMSNorm weights & $\sim$3K \\
Learnable residual scales & $\sim$10 \\
\midrule
\textbf{Total trainable} & $\sim$\textbf{6.2M} \\
\bottomrule
\end{tabular}
\caption{Parameter budget for \texttt{arms\_only} mode.}
\label{tab:params}
\end{table}

% ============================================================================
\section{Inference}
\label{sec:inference}

\subsection{Real-Time Pipeline}

At inference time, the model receives a camera frame, optional spoken command, and current joint states. The full forward pass (backbone encode $\to$ fusion $\to$ action decode) runs in $\sim$50ms on Jetson Orin with the 3B backbone.

\subsection{Temporal Action Smoothing}

The inference server maintains a rolling buffer of predicted action chunks. Overlapping predictions from consecutive forward passes are blended using exponential averaging:
\begin{equation}
    \va_t^\text{smooth} = \alpha \, \va_t^\text{new} + (1-\alpha) \, \va_t^\text{prev}, \quad \alpha = 0.7
    \label{eq:smooth}
\end{equation}
This reduces jitter from frame-to-frame prediction variance.

\subsection{Safety Controller}

Before sending commands to the G1, the controller enforces:
\begin{enumerate}[leftmargin=*,itemsep=1pt]
    \item Joint position limits (from URDF)
    \item Joint velocity limits ($\dot{q}_\text{max}$ per joint)
    \item Locomotion velocity limits ($v_\text{max}, \omega_\text{max}$)
    \item Emergency stop on communication timeout
\end{enumerate}

% ============================================================================
\section{Audio System}
\label{sec:audio_system}

\subsection{Speech Input}

Spoken commands are encoded via a Whisper-derived audio encoder. The mel spectrogram $M \in \R^{80 \times T_\text{mel}}$ is processed through convolutional layers and projected to backbone dimension:
\begin{equation}
    \vf_\text{audio} = W_\text{proj} \cdot \text{Pool}\!\left(\text{Conv}(M)\right) \in \R^{d_\text{bb}}
    \label{eq:whisper}
\end{equation}

\subsection{Speech Output}

Optionally, the fusion features are decoded into a text response via a SpeechResponseHead (linear $\to$ softmax vocabulary), then synthesized to audio via PersonaPlex TTS~\cite{personaplex}. This gives the robot a ``voice'' that narrates its actions.

% ============================================================================
\section{Experimental Setup}
\label{sec:experiments}

\subsection{Test Suite}

Neon includes 80 unit tests across 5 files:
\begin{itemize}[leftmargin=*,itemsep=1pt]
    \item \texttt{test\_model.py} (22 tests): ReLU$^2$, RMSNorm, soft-cap, MLP, chunking, G1 head, NeonVLA
    \item \texttt{test\_data.py} (11 tests): action mapper, episodes, configs, data soup
    \item \texttt{test\_audio.py} (16 tests): encoders, speech head, PersonaPlex
    \item \texttt{test\_relative\_actions.py} (12 tests): euler, rotm, quat, Cosmos relative
    \item \texttt{test\_action\_space.py} (19 tests): G1 joints, modes, normalization
\end{itemize}

All tests pass on CPU without GPU or backbone weights.

\subsection{Hardware}

\begin{table}[H]
\centering
\small
\begin{tabular}{lll}
\toprule
\textbf{Device} & \textbf{Role} & \textbf{Specs} \\
\midrule
Jetson Orin & Real-time inference & ARM64, 32GB, CUDA \\
EC2 L40S & Training & 46GB VRAM \\
HF GPU Jobs & QLoRA fine-tuning & A100/L4 on-demand \\
MacBook M3 & Development, tests & MPS acceleration \\
\bottomrule
\end{tabular}
\caption{Hardware targets.}
\label{tab:hardware}
\end{table}

% ============================================================================
\section{Discussion}
\label{sec:discussion}

\paragraph{Why video over image backbones.}
Video models encode temporal dynamics that image models must learn from robot data alone. A video backbone already ``knows'' that a falling cup accelerates downward — this prior transfers directly to action prediction without consuming robot training budget.

\paragraph{Parameter efficiency matters.}
With only $\sim$6M trainable parameters, every architectural choice in the action heads has measurable impact. The Parameter Golf techniques collectively reduce validation loss by providing smoother gradients (ReLU$^2$, soft-cap), better normalization (RMSNorm), and improved gradient flow (skip connections, residual scales).

\paragraph{Cross-embodiment is a data multiplier.}
The relative action representation enables mixing data from Franka, SO-100, and Agibot robots to train a G1 policy. This effectively multiplies available training data without requiring G1-specific demonstrations for every task.

\paragraph{Limitations.}
The frozen backbone means the model cannot learn new visual representations. The 4-bit quantization introduces approximation errors. Real-world evaluation on the G1 hardware is ongoing.

% ============================================================================
\section{Conclusion}
\label{sec:conclusion}

Neon demonstrates that video foundation models can serve as powerful backbones for humanoid whole-body control. By freezing the 3--7B parameter backbone and training only $\sim$6M parameters in action heads enhanced with Parameter Golf techniques, we achieve a parameter-efficient VLA that leverages pre-trained temporal understanding. The cross-embodiment relative action representation and seven-source data soup enable training on diverse, heterogeneous data. We release Neon as open-source software under the MIT license.

% ============================================================================
{\small
\bibliographystyle{plain}
\begin{thebibliography}{20}

\bibitem{rt2}
A.~Brohan et al., ``RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control,'' \textit{arXiv:2307.15818}, 2023.

\bibitem{openvla}
M.~Kim et al., ``OpenVLA: An Open-Source Vision-Language-Action Model,'' \textit{arXiv:2406.09246}, 2024.

\bibitem{pi0}
K.~Black et al., ``$\pi_0$: A Vision-Language-Action Flow Model for General Robot Control,'' \textit{arXiv:2410.24164}, 2024.

\bibitem{groot}
NVIDIA, ``GR00T N1: An Open Foundation Model for Generalist Humanoid Robots,'' \textit{arXiv:2503.14734}, 2025.

\bibitem{cosmos}
NVIDIA, ``Cosmos World Foundation Model Platform for Physical AI,'' \textit{Technical Report}, 2025.

\bibitem{qwen2.5omni}
Alibaba, ``Qwen2.5-Omni Technical Report,'' \textit{arXiv:2503.20215}, 2025.

\bibitem{act}
T.~Z.~Zhao et al., ``Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware,'' \textit{RSS}, 2023.

\bibitem{lora}
E.~J.~Hu et al., ``LoRA: Low-Rank Adaptation of Large Language Models,'' \textit{ICLR}, 2022.

\bibitem{qlora}
T.~Dettmers et al., ``QLoRA: Efficient Finetuning of Quantized LLMs,'' \textit{NeurIPS}, 2023.

\bibitem{rmsnorm}
B.~Zhang and R.~Sennrich, ``Root Mean Square Layer Normalization,'' \textit{NeurIPS}, 2019.

\bibitem{gemma2}
Google, ``Gemma 2: Improving Open Language Models at a Practical Size,'' \textit{arXiv:2408.00118}, 2024.

\bibitem{personaplex}
NVIDIA, ``PersonaPlex: Multi-Speaker TTS with Persona Control,'' 2025.

\end{thebibliography}
}

% ============================================================================
\appendix
\section{Full Forward Pass Pseudocode}
\label{app:forward}

\begin{algorithm}[H]
\caption{NeonVLA Forward Pass}
\begin{algorithmic}[1]
\REQUIRE images $\mathcal{V}$, text $\ell$, proprio $\vq$, audio $\va$ (optional)
\STATE $\vf_\text{vis} \gets \phi(\mathcal{V}, \ell)$ \COMMENT{Frozen backbone}
\IF{$\vq$ provided}
    \STATE $\vf_\text{prop} \gets \text{MLP}_\text{proprio}(\vq)$
    \STATE $\vf \gets \concat[\vf_\text{vis}, \vf_\text{prop}]$
\ELSE
    \STATE $\vf \gets \vf_\text{vis}$
\ENDIF
\IF{$\va$ provided}
    \STATE $\vf_\text{audio} \gets \psi(\va)$ \COMMENT{Whisper encoder}
    \STATE $\vf \gets \concat[\vf, \vf_\text{audio}]$
\ENDIF
\STATE $\vf \gets \text{GELU}(W_\text{fuse} \vf + b_\text{fuse})$
\STATE $\hat{\va}_\text{arms} \gets \text{ChunkHead}_\text{arms}(\vf)$ \COMMENT{$(B, 16, 14)$}
\STATE $\hat{\va}_\text{loco} \gets \text{ChunkHead}_\text{loco}(\vf)$ \COMMENT{$(B, 16, 3)$}
\STATE $\hat{\va} \gets \concat[\hat{\va}_\text{arms}, \hat{\va}_\text{loco}]$
\RETURN $\hat{\va} \in \R^{B \times 16 \times 17}$
\end{algorithmic}
\end{algorithm}

\section{MLPDecoder v2 Pseudocode}
\label{app:mlp}

\begin{algorithm}[H]
\caption{MLPDecoder with Parameter Golf v2}
\begin{algorithmic}[1]
\REQUIRE input features $\vx \in \R^{d_\text{in}}$
\STATE $\vh \gets W_1 \vx + b_1$ \COMMENT{Linear projection}
\STATE $\vh \gets \rmsnorm(\vh)$
\STATE $\vh \gets \text{ReLU}^2(\vh)$
\STATE $\vh_\text{skip} \gets \vh$ \COMMENT{Save for U-Net skip}
\FOR{$i = 2$ to $L-1$}
    \STATE $\vh_\text{in} \gets \vh$
    \IF{$i = L-1$}
        \STATE $\vh \gets \concat[\vh, \vh_\text{skip}]$ \COMMENT{U-Net skip}
    \ENDIF
    \STATE $\vh \gets W_i \vh + b_i$
    \STATE $\vh \gets \rmsnorm(\vh)$
    \STATE $\vh \gets \text{ReLU}^2(\vh)$
    \IF{$\dim(\vh) = \dim(\vh_\text{in})$}
        \STATE $\vh \gets \vh + \alpha_i \cdot \vh_\text{in}$ \COMMENT{Scaled residual}
    \ENDIF
\ENDFOR
\STATE $\vx_\text{out} \gets W_L \vh + b_L$
\RETURN $\softcap(\vx_\text{out}; 1.0) \in (-1, 1)^{n_a}$
\end{algorithmic}
\end{algorithm}

\end{document}
