# Architecture

## System Overview

RoohAI is structured as three layers: the **transport layer** (browser + FastAPI with dual WebSocket/WebRTC transport), the **Rooh pipeline** (STT → LLM → TTS orchestration with hooks and registry), and the **model layer** (swappable STT, LLM, TTS, and VAD implementations).

![RoohAI Architecture](architecture.drawio.svg)

## Core Framework

### Base Classes (`roohai/base.py`)

Four abstract base classes define the interface for all models:

```python
class STTModel:
    def load() -> None
    def transcribe(audio: np.ndarray, sample_rate: int) -> str
    def unload() -> None
    def is_loaded -> bool  # property

class TTSModel:
    def load() -> None
    def synthesize(text: str) -> tuple[np.ndarray, int]
    def unload() -> None
    def is_loaded -> bool

class LLMModel:
    def load() -> None
    async def chat(message, history, system_prompt, session_id) -> str
    async def chat_stream(message, history, system_prompt, session_id) -> AsyncIterator[str]
    def unload() -> None
    def is_loaded -> bool

class VADModel:
    def load() -> None
    def is_speech(audio_chunk: np.ndarray, sample_rate: int) -> float  # 0.0-1.0
    def unload() -> None
    def is_loaded -> bool
```

Each model class has a `META` dictionary that describes its display name, kind (local/cloud), pip dependencies, HuggingFace models to download, and configurable fields.

### Registry (`roohai/registry.py`)

The registry is a singleton that maps names to model classes with default kwargs:

```python
from roohai.registry import registry

# Register a model
registry.register_stt("whisper", WhisperSTT, model_id="openai/whisper-tiny")

# Create an instance
stt = registry.create_stt("whisper")  # Uses defaults
stt = registry.create_stt("whisper", model_id="openai/whisper-base")  # Override

# List available models
registry.list_stt()  # ["whisper", "wav2vec2", "deepgram"]

# Full catalog with META info
registry.catalog()  # {"stt": [...], "tts": [...], "llm": [...], "vad": [...]}
```

### Rooh (`roohai/pipeline.py`)

The `Rooh` class orchestrates the full STT -> LLM -> TTS flow:

```python
pipeline = Rooh("config.yaml")
pipeline.load()  # Loads all configured models

# Full pipeline
transcription, response, audio_bytes = await pipeline.process_audio(wav_bytes)

# Individual steps
text = await pipeline.transcribe(wav_bytes)
response = await pipeline.chat(text, session_id="abc-123")
audio = await pipeline.synthesize(response)

# Streaming LLM (yields tokens as they arrive)
async for chunk in pipeline.chat_stream(text, session_id="abc-123"):
    print(chunk, end="")

# System prompt (set per-agent, passed to LLM on every call)
pipeline.set_system_prompt("You are a helpful voice assistant.")

# LLM hooks (custom LLM logic, receives session_id)
pipeline.set_llm_hook(my_hook, stream_hook=my_stream_hook)

# Hot-swap a model
pipeline.swap_model("tts", "cartesia")

# Check streaming support
pipeline.stt_supports_streaming  # True if using Deepgram
```

## Server Layer

### Shared Logic (`server/voice_common.py`)

Both transport paths share core voice pipeline logic extracted into `voice_common.py`:

- **`BargeInState`** — Per-connection state machine tracking generation and playback phases
- **VAD constants** — `CHUNK_SIZE` (512 samples), `SPEECH_THRESHOLD` (0.5), `SILENCE_CHUNKS_TO_END` (30), `MIN_SPEECH_CHUNKS` (10)
- **`split_into_sentences()`** — Sentence boundary splitting for TTS chunking
- **`synthesize_chunks()`** — Transport-agnostic TTS chunking that accepts a `send_audio` callback
- **`generate_and_synthesize()`** — Single entry point that dispatches to streaming or batch based on `pipeline.llm_streaming`. Passes `session_id` through to the LLM.
- **`stream_and_synthesize()`** — Consumes `pipeline.chat_stream()`, sends incremental `llm_response_chunk` messages to the client, detects sentence boundaries (`.!?` followed by space), and starts TTS on each completed sentence while the LLM is still generating. This overlaps LLM generation and TTS synthesis for lower perceived latency.
- **`PipelineMetrics`** — Dataclass tracking per-turn timing: `stt_ms`, `llm_ms`, `llm_first_token_ms` (streaming only), `tts_ms`, `total_ms`. Sent to client as a `metrics` message after each turn.

### WebSocket Streaming (`server/streaming.py`)

The streaming handler supports two STT paths chosen automatically based on the active model:

#### Streaming STT Path (Deepgram)

```
Client                    Server                    Deepgram
  │                         │                         │
  │─── audio_chunk ────────>│─── send_chunk ─────────>│
  │                         │                         │
  │                         │<── interim transcript ──│
  │<── interim_transcription│                         │
  │                         │                         │
  │                         │<── utterance_end ───────│
  │<── transcription ───────│                         │
  │<── llm_response_chunk ──│  (LLM streaming)       │
  │<── llm_response_chunk ──│  (tokens arrive)       │
  │<── audio ───────────────│  (TTS on 1st sentence) │
  │<── llm_response_chunk ──│  (more tokens)         │
  │<── audio ───────────────│  (TTS on 2nd sentence) │
  │<── llm_response_done ───│  (stream complete)     │
  │<── audio ───────────────│  (final TTS chunk)     │
```

#### Batch STT Path (Whisper, Wav2Vec2)

```
Client                    Server
  │                         │
  │─── audio_chunk ────────>│  (VAD accumulates speech)
  │<── vad: speech_start ───│
  │─── audio_chunk ────────>│
  │─── audio_chunk ────────>│
  │<── vad: speech_end ─────│  (silence detected)
  │                         │  STT → LLM (streaming) → TTS
  │<── transcription ───────│
  │<── llm_response_chunk ──│  (tokens stream in)
  │<── audio ───────────────│  (TTS starts on first sentence)
  │<── llm_response_done ───│
  │<── audio ───────────────│
```

### Barge-In / Interrupt System

The barge-in system allows users to interrupt the AI at any point during generation or audio playback.

**State machine (`_BargeInState`):**

```
                ┌──────────────┐
                │    Idle      │  is_generating=false
                │              │  playback_pending=false
                └──────┬───────┘
                       │ new utterance
                       ▼
                ┌──────────────┐
                │  Generating  │  is_generating=true
                │              │  playback_pending=true
                └──────┬───────┘
                       │ server finishes LLM+TTS
                       ▼
                ┌──────────────┐
                │  Playback    │  is_generating=false
                │  Pending     │  playback_pending=true
                └──────┬───────┘
                       │ client sends playback_done
                       ▼
                ┌──────────────┐
                │    Idle      │
                └──────────────┘

At ANY state where is_active=true (Generating OR Playback Pending):
  → VAD detects speech → send "interrupt" → cancel generation → stop playback
```

**How it works:**

1. VAD runs on every audio chunk in the main WebSocket loop
2. If speech is detected while `is_active` is true, the server:
   - Sets the `cancel_event` (stops in-progress TTS synthesis)
   - Sends `{"type": "interrupt"}` to the client
3. The client receives the interrupt and:
   - Pauses the audio player
   - Clears the audio queue
   - Shows a brief "Interrupted" visual cue
4. Deepgram continues processing the user's new speech
5. When the new utterance completes, a fresh generation starts

**Why `playback_pending`?** The server pushes all audio chunks to the client in ~2-3 seconds, but the client plays them over 10-20 seconds. Without tracking client-side playback, the server would consider itself idle while the user is still hearing the AI speak.

### WebRTC Transport (`server/tracks.py` + `server/webrtc.py`)

The WebRTC path provides lower-latency audio transport using native browser WebRTC with Opus codec.

#### Connection Flow

```
Browser                           Server
  │                                 │
  │── getUserMedia (mic) ──────────>│
  │── RTCPeerConnection ──────────>│
  │── createDataChannel("status") ─>│
  │── createOffer ─────────────────>│
  │                                 │
  │── POST /api/webrtc/offer ──────>│  setRemoteDescription (fires on_track)
  │                                 │  on_track: create AudioProcessorTrack
  │                                 │            addTrack (→ sendrecv SDP)
  │                                 │            start _process_loop
  │                                 │  createAnswer
  │<── SDP answer ──────────────────│
  │                                 │
  │── setRemoteDescription ────────>│
  │   (Opus audio flows both ways)  │
  │                                 │
  │<══ remote track (TTS audio) ═══>│  recv() returns TTS or silence
  │══> local track (mic audio) ════>│  _process_loop receives frames
```

#### AudioProcessorTrack

The `AudioProcessorTrack` extends aiortc's `MediaStreamTrack` and handles:

1. **Inbound audio** — Receives Opus-decoded 48kHz stereo frames via `_process_loop()`
2. **Stereo de-interleaving** — Packed s16 stereo `(1, samples*2)` → mono float32
3. **Downsampling** — 48kHz → 16kHz via simple decimation (`samples[::3]`)
4. **Silero VAD** — Speech detection on 512-sample chunks at 16kHz
5. **Pre-speech buffer** — Keeps last ~200ms (6 chunks) before speech starts
6. **Pipeline execution** — STT → LLM → TTS with barge-in cancel support
7. **Outbound audio** — `recv()` returns TTS frames or silence at 20ms intervals

#### recv() Pacing

aiortc calls `recv()` in a tight loop. Without wall-clock pacing, the event loop starves and nothing else runs. The track uses the same approach as aiortc's base `AudioStreamTrack`:

```python
# Sleep until 20ms has elapsed since the last frame
wait = self._recv_start + (self._recv_timestamp / SAMPLE_RATE) - time.time()
if wait > 0:
    await asyncio.sleep(wait)
```

#### Audio Format

WebRTC uses the Opus codec, which requires packed `s16` (not planar `s16p`):
- **Silence frames:** `np.zeros((1, 960 * 2), dtype=np.int16)` — shape `(1, FRAME_SAMPLES * 2)`
- **TTS frames:** Mono audio duplicated to stereo via interleaving: `interleaved[0::2] = mono; interleaved[1::2] = mono`
- **Time base:** `Fraction(1, 48000)` (PyAV requires a `Fraction`, not a string)

#### Data Channel Protocol

The WebRTC path uses an RTCDataChannel named `"status"` for JSON messages (same format as WebSocket):

**Server → Client:**

| Type | Description |
|------|-------------|
| `status` | Pipeline state (listening/transcribing/thinking/speaking/paused) |
| `interim_transcription` | Live partial transcription |
| `transcription` | Final committed transcription |
| `llm_response` | AI response text (non-streaming fallback) |
| `llm_response_chunk` | Incremental LLM token (streaming) |
| `llm_response_done` | Full LLM response text (streaming complete) |
| `interrupt` | Stop playback (barge-in) |
| `playback_complete` | All TTS frames enqueued — client should send `playback_done` after hearing them |
| `error` | Error description |

**Client → Server:**

| Type | Description |
|------|-------------|
| `playback_done` | Client finished hearing all TTS audio |
| `pause_audio` | Pause audio capture (keep connection alive) |
| `resume_audio` | Resume audio capture |
| `ping` | Keep-alive |

#### Streaming STT over WebRTC

When the active STT model supports streaming (e.g., Deepgram), the WebRTC track:
1. Calls `stt.start_stream()` when the data channel connects
2. Forwards 16kHz PCM16 chunks via `stt.send_chunk()` on every frame
3. Deepgram fires `on_utterance_end` → pipeline runs LLM + TTS (skips STT step)
4. VAD still runs for barge-in detection during playback

When using batch STT (Whisper, Wav2Vec2), the track accumulates audio during speech and sends it all at once when silence is detected.

### WebSocket Message Types

#### Client -> Server

| Type | Fields | Description |
|------|--------|-------------|
| `audio_chunk` | `audio` (base64) | Raw PCM16 audio at 16kHz |
| `playback_done` | — | Client finished playing all audio |
| `pause_audio` | — | Pause audio capture (connection stays alive) |
| `resume_audio` | — | Resume audio capture |
| `ping` | — | Keep-alive |

#### Server -> Client

| Type | Fields | Description |
|------|--------|-------------|
| `session` | `session_id` | Unique connection ID (sent on connect) |
| `status` | `state`, `vad` | Pipeline state (listening/transcribing/thinking/speaking/paused) |
| `interim_transcription` | `text` | Live partial transcription (replaced on each update) |
| `transcription` | `text` | Final committed user transcription |
| `llm_response_chunk` | `text` | Streaming AI response token (progressive display) |
| `llm_response_done` | `text` | Streaming complete (full accumulated text) |
| `llm_response` | `text` | Non-streaming AI response (full text at once) |
| `audio` | `audio` (base64 WAV) | TTS audio chunk |
| `interrupt` | — | Stop playback immediately (barge-in) |
| `metrics` | `stt_ms`, `llm_ms`, `llm_first_token_ms`, `tts_ms`, `total_ms` | Pipeline timing for the turn |
| `error` | `message` | Error description |
| `pong` | — | Keep-alive response |

## Frontend

### Audio Capture

The frontend uses an `AudioWorklet` processor (`audio-processor.js`) to capture microphone audio:

1. `AudioContext` opens at 48kHz (browser native)
2. The worklet downsamples to 16kHz using linear interpolation
3. PCM16 chunks are sent as base64 over the WebSocket
4. Audio flows continuously — the worklet never stops sending (required for barge-in)
5. The worklet supports `pause`/`resume` messages from the main thread to stop/restart audio processing without tearing down the connection

### Audio Playback

TTS audio arrives as one or more WAV chunks. The client queues them for sequential playback:

```
Server sends: audio chunk 1, chunk 2, chunk 3
Client queue: [chunk1, chunk2, chunk3]
Playback:     chunk1 → chunk2 → chunk3 → send "playback_done"
```

On interrupt, `stopPlayback()` clears the queue and pauses immediately.

### Mic Pause/Resume

The mic button toggles between three states: **disconnected** → **listening** → **paused** → **listening** (cycle). A separate disconnect button (X icon) performs full teardown.

- **Pause:** Stops the AudioWorklet from sending chunks, mutes the mic track, sends `pause_audio` to the server. The WebSocket/WebRTC connection stays alive.
- **Resume:** Restarts audio processing, unmutes the mic track, sends `resume_audio` to the server.
- **Disconnect:** Full teardown — closes WebSocket/WebRTC, stops all tracks, releases mic.

This preserves conversation history and connection state across pause/resume cycles.

## Agent System

Agents are named configurations stored as YAML files in `agents/`:

```yaml
name: my-agent
status: active
transport: webrtc          # "websocket" (default) or "webrtc"
llm_streaming: true        # true (default) = stream tokens, false = wait for full response
system_prompt: You are a helpful voice assistant. Be concise and friendly.
pipeline:
  stt: deepgram
  llm: bedrock-claude
  tts: cartesia
  vad: silero
models:
  deepgram:
    class: deepgram
    type: stt
    model: nova-3
    language: en
  bedrock-claude:
    class: bedrock-claude
    type: llm
    model_id: anthropic.claude-3-haiku-20240307-v1:0
    region: us-east-1
  cartesia:
    class: cartesia
    type: tts
    model_id: sonic-2
  silero:
    class: silero
    type: vad
```

Secrets (API keys) are stored separately in `~/.roohai/secrets.yaml` with restricted file permissions, never in the agent YAML.

Only one agent is active at a time. Activating an agent hot-swaps all pipeline models.
