# Models Reference

Every model in RoohAI is a subclass of one of the four base types and exposes a `META` dictionary describing its capabilities, dependencies, and configurable fields.

---

## Speech-to-Text (STT)

### Whisper

| Property | Value |
|----------|-------|
| Class | `WhisperSTT` |
| Registry name | `whisper` |
| Kind | Local |
| Dependencies | `transformers>=4.46.0,<5.0.0`, `torch`, `sentencepiece` |
| Download size | ~150 MB |
| Streaming | No |

OpenAI's open-source speech recognition model. Accurate across many languages, runs entirely locally.

**Configuration:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `model_id` | text | `openai/whisper-tiny` | HuggingFace model ID. Options: `openai/whisper-tiny`, `openai/whisper-base`, `openai/whisper-small` |
| `language` | text | `en` | Language code |

**Example config:**

```yaml
whisper-tiny:
  type: stt
  class: whisper
  model_id: openai/whisper-tiny
  language: en
```

---

### Wav2Vec2

| Property | Value |
|----------|-------|
| Class | `Wav2Vec2STT` |
| Registry name | `wav2vec2` |
| Kind | Local |
| Dependencies | `transformers>=4.46.0,<5.0.0`, `torch` |
| Download size | ~360 MB |
| Streaming | No |

Facebook's self-supervised speech recognition. English-only, runs locally.

**Configuration:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `model_id` | text | `facebook/wav2vec2-base-960h` | HuggingFace model ID |

**Example config:**

```yaml
wav2vec2:
  type: stt
  class: wav2vec2
  model_id: facebook/wav2vec2-base-960h
```

---

### Deepgram

| Property | Value |
|----------|-------|
| Class | `DeepgramSTT` |
| Registry name | `deepgram` |
| Kind | Cloud |
| Dependencies | `deepgram-sdk>=6.0` |
| Download size | 0 (cloud API) |
| Streaming | **Yes** |

Cloud-based streaming speech recognition with real-time interim results and utterance boundary detection. Recommended for production use.

**Configuration:**

| Field | Type | Default | Required | Description |
|-------|------|---------|----------|-------------|
| `api_key` | secret | — | Yes | Deepgram API key (env: `DEEPGRAM_API_KEY`) |
| `model` | select | `nova-3` | No | Recognition model (`nova-3`, `nova-2`) |
| `language` | text | `en` | No | Language code |

**Example config:**

```yaml
deepgram:
  type: stt
  class: deepgram
  model: nova-3
  language: en
  # api_key stored in ~/.roohai/secrets.yaml
```

**Streaming features:**
- Interim transcriptions (updated in real time as user speaks)
- `speech_final` detection (natural pause = end of turn)
- `utterance_end` events (silence timeout, default 1500ms)
- Smart formatting and punctuation

---

### NVIDIA Canary-Qwen

| Property | Value |
|----------|-------|
| Class | `NvidiaCanarySTT` |
| Registry name | `nvidia-canary` |
| Kind | Local |
| Dependencies | `nemo_toolkit[asr]`, `torch`, `soundfile` |
| Download size | ~5000 MB |
| Streaming | No |

NVIDIA's Canary-Qwen 2.5B ASR model via NeMo toolkit. State-of-the-art accuracy, requires CUDA GPU.

**Configuration:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `model_id` | text | `nvidia/canary-qwen-2.5b` | Pretrained model name |
| `language` | text | `en` | Language code |

**Example config:**

```yaml
nvidia-canary:
  type: stt
  class: nvidia-canary
  model_id: nvidia/canary-qwen-2.5b
  language: en
```

> **Note:** Requires a CUDA-capable GPU. The NeMo toolkit will be installed automatically during agent preparation.

---

### NVIDIA Parakeet

| Property | Value |
|----------|-------|
| Class | `NvidiaParakeetSTT` |
| Registry name | `nvidia-parakeet` |
| Kind | Local |
| Dependencies | `nemo_toolkit[asr]`, `torch`, `soundfile` |
| Download size | ~600 MB |
| Streaming | No |

NVIDIA's Parakeet TDT 0.6B — a smaller, faster ASR model. Good accuracy-to-speed ratio.

**Configuration:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `model_id` | text | `nvidia/parakeet-tdt-0.6b-v2` | Pretrained model name |
| `language` | text | `en` | Language code |

**Example config:**

```yaml
nvidia-parakeet:
  type: stt
  class: nvidia-parakeet
  model_id: nvidia/parakeet-tdt-0.6b-v2
  language: en
```

---

## Language Models (LLM)

### Amazon Bedrock (Claude)

| Property | Value |
|----------|-------|
| Class | `BedrockLLM` |
| Registry name | `bedrock` |
| Kind | Cloud |
| Dependencies | `boto3` |
| Download size | 0 (cloud API) |

Anthropic Claude via Amazon Bedrock. Supports environment-based auth (IAM roles, SSO) or explicit access keys.

**Configuration:**

| Field | Type | Default | Required | Description |
|-------|------|---------|----------|-------------|
| `auth_mode` | select | `environment` | No | `environment` (AWS CLI/SSO) or `access_keys` |
| `aws_access_key_id` | secret | — | If `access_keys` | AWS access key (env: `AWS_ACCESS_KEY_ID`) |
| `aws_secret_access_key` | secret | — | If `access_keys` | AWS secret key (env: `AWS_SECRET_ACCESS_KEY`) |
| `region` | text | `us-east-1` | No | AWS region |
| `model_id` | text | `anthropic.claude-3-haiku-20240307-v1:0` | No | Bedrock model ID |

**Example config:**

```yaml
bedrock-claude:
  type: llm
  class: bedrock
  model_id: anthropic.claude-3-haiku-20240307-v1:0
  region: us-east-1
  auth_mode: environment
```

**Bedrock API details:**
- Uses API version `bedrock-2023-05-31`
- System prompt: "You are a helpful voice assistant. Keep responses concise and conversational."
- Supports conversation history

---

### Local LLM (TinyLlama)

| Property | Value |
|----------|-------|
| Class | `LocalLLM` |
| Registry name | `local` |
| Kind | Local |
| Dependencies | `transformers>=4.46.0,<5.0.0`, `torch`, `accelerate` |
| Download size | ~2200 MB |

Run a HuggingFace language model locally. Uses the `text-generation` pipeline with chat template support.

**Configuration:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `model_id` | text | `TinyLlama/TinyLlama-1.1B-Chat-v1.0` | HuggingFace model ID |

**Example config:**

```yaml
local-llm:
  type: llm
  class: local
  model_id: TinyLlama/TinyLlama-1.1B-Chat-v1.0
```

> **Note:** Local LLMs are significantly slower than cloud APIs, especially on CPU. A GPU is recommended for reasonable response times.

---

## Text-to-Speech (TTS)

### SpeechT5

| Property | Value |
|----------|-------|
| Class | `SpeechT5TTS` |
| Registry name | `speecht5` |
| Kind | Local |
| Dependencies | `transformers>=4.46.0,<5.0.0`, `torch`, `sentencepiece` |
| Download size | ~600 MB |

Microsoft's unified speech-text model. Generates speech locally with a HiFi-GAN vocoder.

**Configuration:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `model_id` | text | `microsoft/speecht5_tts` | TTS model ID |
| `vocoder` | text | `microsoft/speecht5_hifigan` | Vocoder model ID |

**Example config:**

```yaml
speecht5:
  type: tts
  class: speecht5
  model_id: microsoft/speecht5_tts
  vocoder: microsoft/speecht5_hifigan
```

> **Note:** Uses random speaker embeddings (`torch.randn(1, 512)`) since the standard speaker embedding dataset is currently unavailable.

---

### Bark

| Property | Value |
|----------|-------|
| Class | `BarkTTS` |
| Registry name | `bark` |
| Kind | Local |
| Dependencies | `transformers>=4.46.0,<5.0.0`, `torch` |
| Download size | ~1800 MB |

Suno's generative text-to-audio model. Produces expressive, natural-sounding speech. Slower but higher quality than SpeechT5.

**Configuration:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `model_id` | text | `suno/bark-small` | HuggingFace model ID |

**Example config:**

```yaml
bark:
  type: tts
  class: bark
  model_id: suno/bark-small
```

---

### Piper

| Property | Value |
|----------|-------|
| Class | `PiperTTS` |
| Registry name | `piper` |
| Kind | Local |
| Dependencies | `piper-tts>=1.4` |
| Download size | ~60 MB |

Fast, lightweight local TTS engine. Downloads voice models automatically on first use. Excellent speed-to-quality ratio.

**Configuration:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `voice` | select | `en_US-lessac-medium` | Voice model |

**Available voices:**

| Value | Description |
|-------|-------------|
| `en_US-lessac-medium` | Lessac (US English, Medium quality) |
| `en_US-amy-medium` | Amy (US English, Medium quality) |
| `en_US-ryan-medium` | Ryan (US English, Medium quality) |
| `en_US-libritts_r-medium` | LibriTTS-R (US English, Medium quality) |
| `en_GB-alan-medium` | Alan (British English, Medium quality) |
| `en_GB-cori-medium` | Cori (British English, Medium quality) |

**Example config:**

```yaml
piper:
  type: tts
  class: piper
  voice: en_US-lessac-medium
```

---

### Cartesia

| Property | Value |
|----------|-------|
| Class | `CartesiaTTS` |
| Registry name | `cartesia` |
| Kind | Cloud |
| Dependencies | `cartesia` |
| Download size | 0 (cloud API) |

Cloud-based TTS with fast, natural-sounding voices. Low latency, high quality. Recommended for production.

**Configuration:**

| Field | Type | Default | Required | Description |
|-------|------|---------|----------|-------------|
| `api_key` | secret | — | Yes | Cartesia API key (env: `CARTESIA_API_KEY`) |
| `model_id` | text | `sonic-2` | No | TTS model |
| `voice_id` | text | `a0e99841-...` | No | Voice identifier |
| `language` | text | `en` | No | Language code |
| `sample_rate` | text | `24000` | No | Output sample rate (8000-48000) |

**Example config:**

```yaml
cartesia:
  type: tts
  class: cartesia
  model_id: sonic-2
  voice_id: a0e99841-438c-4a64-b679-ae501e7d6091
  language: en
  sample_rate: 24000
  # api_key stored in ~/.roohai/secrets.yaml
```

---

## Voice Activity Detection (VAD)

### Silero VAD

| Property | Value |
|----------|-------|
| Class | `SileroVAD` |
| Registry name | `silero` |
| Kind | Local |
| Dependencies | `silero-vad>=5.1`, `torch` |
| Download size | ~2 MB |

Lightweight, accurate voice activity detection. Standard choice for all configurations.

**Configuration:** None required.

**Example config:**

```yaml
silero:
  type: vad
  class: silero
```

**Additional method:**

```python
# Get speech segments with timestamps
segments = vad.get_speech_segments(audio, sample_rate=16000)
# Returns: [{"start": 0.5, "end": 2.1}, {"start": 3.0, "end": 4.5}]
```

---

