# Audio In/Out

Neon can **hear** spoken commands and **speak** back while executing actions. The robot has ears (Whisper encoder) and a voice (PersonaPlex TTS).

---

## Why Audio?

Text instructions are great for benchmarks. Real humans talk to robots.

- **"Hey, grab that blue thing over there"** — spoken while pointing
- **"Stop!"** — urgent, needs immediate processing
- **"What do you see?"** — the robot should answer verbally

Audio is also **multimodal context**. The sound of a motor stalling, glass breaking, or a timer beeping carries information that text can't capture.

---

## Architecture

```mermaid
graph TD
    subgraph Audio Input
        MIC["🎤 Microphone<br/>16kHz waveform"] --> WE["Whisper Encoder<br/>(frozen, ~39M params)"]
        WE --> PROJ["Projection Layer<br/>audio_dim → backbone_dim"]
    end

    subgraph Visual Input
        CAM["📹 Camera"] --> VB["Video Backbone"]
        TXT["📝 Text"] --> VB
    end

    subgraph Fusion
        PROJ --> FUS["Feature Fusion<br/>(concat + linear)"]
        VB --> FUS
        PE["🦾 Proprioception"] --> FUS
    end

    subgraph Output
        FUS --> AH["Action Heads → 🤖 Joints"]
        FUS --> SH["Speech Head → 🔊 PersonaPlex"]
    end

    style WE fill:#1565c0,color:#fff
    style VB fill:#e65100,color:#fff
    style FUS fill:#333,color:#fff
```

---

## Audio Input: Three Strategies

### 1. Whisper Encoder (Default)

OpenAI's Whisper model has a robust encoder that converts speech to features. We use only the encoder (not the decoder) as a feature extractor:

```python
from neon.model.audio import AudioConfig, AudioEncoder

config = AudioConfig(
    encoder_type="whisper",
    whisper_model="openai/whisper-base",  # 39M params
    freeze_encoder=True,                   # Don't train Whisper
    sample_rate=16000,                     # Whisper expects 16kHz
)

encoder = AudioEncoder(config, backbone_hidden_size=2048)

# Encode a spoken command
audio = load_wav("pick_up_the_cup.wav")  # (samples,) at 16kHz
features = encoder.encode(audio)          # (1, 2048) — same dim as backbone
```

**Why Whisper?** It's pre-trained on 680K hours of speech. It understands accents, noise, and real-world audio. Freezing it means zero additional training cost.

### 2. Mel Spectrogram CNN (Lightweight)

For edge deployment where even Whisper-base is too heavy:

```python
config = AudioConfig(encoder_type="mel", audio_hidden_size=512)
encoder = AudioEncoder(config, backbone_hidden_size=2048)
```

This builds a tiny CNN (80 mel bins → 256 → 512 → pool) with ~200K parameters. Less accurate but runs in 1ms on Jetson.

### 3. Omni Mode (Native)

When using `Qwen2.5-Omni` as the backbone, audio is processed natively alongside video — no separate encoder needed:

```python
config = AudioConfig(encoder_type="omni")
# Audio is passed directly to the backbone's multimodal processor
```

This is the best quality option since the backbone fuses audio, video, and text in a single attention mechanism.

---

## Audio Output: PersonaPlex TTS

The robot speaks using [PersonaPlex](https://huggingface.co/nvidia/BigVGAN), which runs on the Jetson Orin (Thor). Eight voice options give the robot personality:

| Voice | Type | Best For |
|-------|------|----------|
| `NATM0`–`NATM3` | Natural male | Default robot voice |
| `NATF0`–`NATF3` | Natural female | Alternative voice |

### Speech Response Head

The model predicts **when and what** to say:

```python
class SpeechResponseHead(nn.Module):
    # Predicts 5 response types from the same features used for actions
    response_types = ["narrate", "confirm", "warn", "ask", "silent"]
```

| Type | Example | When |
|------|---------|------|
| `narrate` | "I see the red cup. Reaching for it now." | During execution |
| `confirm` | "Got it. Working on it." | After receiving command |
| `warn` | "I can't reach that safely." | Safety concern |
| `ask` | "Which cup do you mean?" | Ambiguous instruction |
| `silent` | — | No speech needed |

### Usage

```python
output = model.predict(
    image=camera_frame,
    instruction="Pick up the cup",
    audio=microphone_waveform,  # Spoken command (numpy, 16kHz)
    speak=True,                  # Generate speech response
)

# output.actions → joint commands (as usual)
# output.speech_path → "/tmp/neon_speech_xxx.wav" (played on robot's speaker)
```

---

## Training Data

Two HuggingFace datasets provide audio training data:

| Dataset | Rows | Format | Content |
|---------|------|--------|---------|
| [`cagataydev/vlm-voice-audio`](https://huggingface.co/datasets/cagataydev/vlm-voice-audio) | 1,000 | Audio (44.1kHz WAV) + text | PersonaPlex-synthesized spoken robot commands in 8 voices |
| [`cagataydev/vlm-voice-commands`](https://huggingface.co/datasets/cagataydev/vlm-voice-commands) | 50,000 | Text only | Robot commands across 10 categories |

### Categories

The commands cover the full range of robot interactions:

| Category | Count | Example |
|----------|-------|---------|
| `pick_place` | 222 | "Pick up the black jar and place it on the box" |
| `manipulation` | 149 | "Carefully twist the lid off the bottle" |
| `navigation` | 113 | "Take the path through the left side of the table" |
| `multistep` | 109 | "First grab the sponge, then wipe the counter, then put it away" |
| `observation` | 105 | "What color is the object in front of you?" |
| `spatial` | 78 | "Trace a line from the sponge to the book" |
| `household` | 77 | "Please sweep the floor near the fridge" |
| `safety` | 69 | "Stop immediately and back away" |
| `conversational` | 42 | "How are you doing today?" |
| `context_rich` | 36 | "Remember where you put the screwdriver earlier" |

---

## Configuration

Enable audio in `NeonConfig`:

```python
config = NeonConfig(
    control_mode="arms_only",
    audio={
        "enabled": True,
        "encoder_type": "whisper",           # "whisper", "mel", or "omni"
        "whisper_model": "openai/whisper-base",
        "freeze_encoder": True,
        "enable_speech_output": True,
        "personaplex_voice": "NATM1",        # Default robot voice
        "personaplex_host": "http://192.168.1.151:8400",  # Thor
    },
)
```

Set `audio=None` (default) to disable audio entirely — backward compatible with existing configs.
