# Audio Generation Cross-Provider Research

*Generated by /research-package on 2025-11-12*

## Executive Summary

- **Providers researched**: 4
- **Providers supporting audio-generation**: 4 (OpenAI, Google Gemini, ElevenLabs, Amazon Polly)
- **Streaming support**: Yes (3/4 providers, Gemini unclear)
- **Common input**: Text prompt with voice/model selection
- **Common output**: Binary audio data (various formats)

## Providers Supporting Audio Generation

| Provider | Models | Streaming | Documentation |
|----------|--------|-----------|---------------|
| OpenAI | tts-1, tts-1-hd, gpt-4o-mini-tts | Yes | https://platform.openai.com/docs/guides/text-to-speech |
| Google Gemini | gemini-2.5-flash-preview-tts, gemini-2.5-pro-preview-tts | Not documented | https://ai.google.dev/gemini-api/docs/speech-generation |
| ElevenLabs | Multiple TTS models | Yes | https://elevenlabs.io/docs/api-reference/text-to-speech |
| Amazon Polly | standard, neural, long-form, generative | Yes | https://docs.aws.amazon.com/polly/latest/dg/API_SynthesizeSpeech.html |

## Streaming Support Analysis

**Verdict**: Yes (3/4 providers confirmed)

### Provider Details

**OpenAI**:
- "The Speech API provides support for real-time audio streaming using chunk transfer encoding"
- Audio can be played before the full file has been generated

**Google Gemini**:
- No explicit streaming support documented in TTS API
- Live API supports real-time audio but is for conversational contexts, not TTS
- Returns complete base64-encoded audio in single response

**ElevenLabs**:
- Dedicated "Stream speech" endpoint
- Multiple streaming variants: WebSocket streaming, Standard streaming, Streaming with timing

**Amazon Polly**:
- Response includes `AudioStream` field
- Streams binary audio data as it's generated

**Recommendation**: Implement streaming support for audio-generation capability (3/4 providers support it).

## Usage Patterns

### OpenAI Usage Fields

OpenAI bills by characters but does not return explicit usage metrics in the API response. Pricing:
- TTS-1: $0.015 per 1,000 characters
- TTS-1-HD: $0.030 per 1,000 characters

Example request:
```json
POST /v1/audio/speech

{
  "model": "tts-1",
  "input": "Hello, this is a test.",
  "voice": "alloy",
  "response_format": "mp3",
  "speed": 1.0
}
```

Response: Binary audio stream (no usage metadata in response body)

### Google Gemini Usage Fields

Gemini returns only `inlineData.data` (base64-encoded PCM audio). No explicit usage tracking fields.

Example request:
```json
POST /v1beta/models/gemini-2.5-flash-preview-tts:generateContent

{
  "contents": [{
    "parts": [{
      "text": "Say cheerfully: Have a wonderful day!"
    }]
  }],
  "generationConfig": {
    "responseModalities": ["AUDIO"],
    "speechConfig": {
      "voiceConfig": {
        "prebuiltVoiceConfig": {
          "voiceName": "Kore"
        }
      }
    }
  }
}
```

Example response:
```json
{
  "candidates": [{
    "content": {
      "parts": [{
        "inlineData": {
          "mimeType": "audio/pcm",
          "data": "base64_encoded_audio_data..."
        }
      }]
    }
  }]
}
```

### ElevenLabs Usage Fields

ElevenLabs tracks usage through history/logging features but does not return explicit metrics in the speech generation response.

Example request:
```json
POST /v1/text-to-speech/{voice_id}

{
  "text": "Hello, this is a test.",
  "model_id": "eleven_monolingual_v1",
  "voice_settings": {
    "stability": 0.5,
    "similarity_boost": 0.75
  }
}
```

Response: Binary audio stream (application/octet-stream)

### Amazon Polly Usage Fields

Amazon Polly returns usage metadata in response headers:

Example request:
```json
POST /v1/speech

{
  "Text": "Hello, this is a test.",
  "VoiceId": "Joanna",
  "OutputFormat": "mp3",
  "Engine": "neural"
}
```

Response headers:
```
ContentType: audio/mpeg
x-amzn-RequestCharacters: 23
```

Response body: Binary AudioStream

### Cross-Provider Usage Analysis

| Field | Type | OpenAI | Gemini | ElevenLabs | Amazon Polly | Notes |
|-------|------|--------|--------|------------|--------------|-------|
| characters_synthesized | int | ❌ (billed) | ❌ | ❌ | ✅ x-amzn-RequestCharacters | Only Amazon Polly |
| audio_format | str | ❌ (in request) | ✅ mimeType | ❌ (in request) | ✅ ContentType | Returned as MIME type |

### Proposed AudioGenerationUsage

```python
class AudioGenerationUsage(Usage):
    """Audio generation usage metrics.

    All fields optional since providers vary significantly.
    Most providers bill by characters but don't return usage in response.
    """
    characters_synthesized: int | None = None  # Amazon Polly only (x-amzn-RequestCharacters)
```

**Rationale**:
- Only Amazon Polly returns character count in response headers
- OpenAI, Gemini, and ElevenLabs bill by characters but don't expose usage metrics
- Field is optional since 3/4 providers don't report it
- Implementation can calculate from input text length when provider doesn't report

## Finish Reasons

### Analysis

Audio generation is fundamentally different from text generation streaming:
- Text generation: Incremental token-by-token with stop conditions (max_tokens, stop_sequence, content_filter)
- Audio generation: Single complete audio file or stream chunks without intermediate stop conditions

### Provider Behavior

**OpenAI**: No finish_reason concept - request completes successfully or errors
**Gemini**: No finish_reason concept - returns complete audio or error
**ElevenLabs**: No finish_reason concept - returns complete audio or error
**Amazon Polly**: No finish_reason concept - returns complete audio or error

### Verdict

Audio generation does **not** use the `FinishReason` pattern. The operation either:
1. Completes successfully (returns full audio)
2. Fails with an error (HTTP error codes)

**Recommendation**: Skip `AudioGenerationFinishReason` class. Not applicable to this capability.

## Input/Output Pattern Analysis

### Input Pattern Analysis

**OpenAI**:
- `input` (string, required): Text to convert to speech
- `model` (string, required): tts-1, tts-1-hd, gpt-4o-mini-tts
- `voice` (string, required): alloy, echo, fable, onyx, nova, shimmer, etc.
- `speed` (float, optional): 0.25 to 4.0
- `response_format` (string, optional): mp3, opus, aac, flac, wav, pcm

**Google Gemini**:
- `contents[].parts[].text` (string, required): Text to synthesize
- `generationConfig.responseModalities` (array, required): ["AUDIO"]
- `generationConfig.speechConfig.voiceConfig` (object, required): Voice configuration
- Model specified in endpoint path

**ElevenLabs**:
- `text` (string, required): Content to convert
- `model_id` (string, required): Model identifier
- `voice_id` (path param, required): Voice to use
- `voice_settings` (object, optional): stability, similarity_boost, style, speed

**Amazon Polly**:
- `Text` (string, required): Input text
- `VoiceId` (string, required): Voice identifier
- `Engine` (string, optional): standard, neural, long-form, generative
- `OutputFormat` (string, required): mp3, ogg_vorbis, pcm, json
- `SampleRate` (string, optional): Audio frequency

**Common Pattern**: All providers require:
1. Text input (string)
2. Voice selection (various formats)
3. Output format specification

**Proposed Input**: `prompt: str`

**Rationale**:
- Text is the universal input across all providers
- Voice/model/format are parameters, not part of core input
- Follows existing pattern from text-generation (prompt: str)
- Keeps capability interface simple and unified

### Output Pattern Analysis

**OpenAI**:
- Binary audio stream
- Content-Type varies by format (audio/mpeg, audio/opus, etc.)

**Google Gemini**:
- Base64-encoded PCM audio in `candidates[0].content.parts[0].inlineData.data`
- `mimeType` field: "audio/pcm"
- Sample rate: 24kHz, 16-bit, mono

**ElevenLabs**:
- Binary audio stream (application/octet-stream)
- Format determined by output_format parameter

**Amazon Polly**:
- `AudioStream`: Binary audio data
- `ContentType` header: MIME type
- `RequestCharacters` header: Usage tracking

**Common Pattern**: All providers return binary audio data with MIME type indication

**Proposed Output**: `AudioArtifact`

**Rationale**:
- Audio is binary data with MIME type (matches `ImageArtifact` pattern)
- `AudioArtifact` supports: `data` (bytes), `url` (str), `path` (str), `mime_type`
- Enables consistent handling across providers
- Allows response transformation (base64 decode for Gemini, direct binary for others)

## Recommended Package Structure

```python
from celeste.artifacts import AudioArtifact
from celeste.io import Input, Output, Usage, Chunk

class AudioGenerationInput(Input):
    """Input for audio generation operations."""
    prompt: str

class AudioGenerationUsage(Usage):
    """Audio generation usage metrics.

    All fields optional since providers vary significantly.
    """
    characters_synthesized: int | None = None  # Provider-reported character count

class AudioGenerationOutput(Output[AudioArtifact]):
    """Audio generation output with binary audio artifact."""
    pass

# Streaming support (since 3/4 providers support it)
class AudioGenerationChunk(Chunk[bytes]):
    """Audio generation stream chunk.

    Audio streaming sends raw bytes without finish_reason.
    """
    usage: AudioGenerationUsage | None = None  # Final chunk only
```

## API Parameters Analysis

### Universal Parameters (All 4 providers)

| Parameter | OpenAI | Gemini | ElevenLabs | Amazon Polly | Celeste Name |
|-----------|--------|--------|------------|--------------|--------------|
| Text input | `input` | `contents[].parts[].text` | `text` | `Text` | `prompt` |
| Voice | `voice` | `speechConfig.voiceConfig.prebuiltVoiceConfig.voiceName` | `voice_id` | `VoiceId` | `voice` |
| Model | `model` | (in endpoint path) | `model_id` | `Engine` | `model` |
| Audio format | `response_format` | (fixed: PCM) | `output_format` | `OutputFormat` | `audio_format` |

### Common Parameters (3/4 providers)

| Parameter | OpenAI | Gemini | ElevenLabs | Amazon Polly | Celeste Name |
|-----------|--------|--------|------------|--------------|--------------|
| Speed/Rate | `speed` (0.25-4.0) | (prompt-based) | `voice_settings.speed` | ❌ | `speed` |
| Sample rate | ❌ (fixed by format) | 24kHz (fixed) | `output_format` | `SampleRate` | (provider-specific) |

### Provider-Specific Parameters

**OpenAI**:
- None (simple API with minimal parameters)

**Google Gemini**:
- `multiSpeakerVoiceConfig` - Multi-speaker conversations (up to 2 speakers)
- Style/tone controlled via natural language in text prompt
- 30 prebuilt voices available

**ElevenLabs**:
- `voice_settings.stability` (0.0-1.0)
- `voice_settings.similarity_boost` (0.0-1.0)
- `voice_settings.style` (0.0-1.0)
- `voice_settings.use_speaker_boost` (boolean)
- `optimize_streaming_latency` (0-4)

**Amazon Polly**:
- `TextType` (text, ssml)
- `LexiconNames` (pronunciation lexicons)
- `SpeechMarkTypes` (sentence, ssml, viseme, word)

### Proposed Parameters

```python
from celeste.core import Parameters
from typing import TypedDict
from typing_extensions import Unpack

class AudioGenerationParameters(Parameters):
    """Audio generation parameters.

    Universal parameters supported across providers.
    """
    voice: str  # Voice identifier (provider-specific)
    audio_format: str  # mp3, wav, opus, pcm, etc.
    speed: float  # Playback speed multiplier (0.25-4.0 typical range)
```

**Provider-Specific Mappers**:
- `VoiceMapper` - Maps Celeste voice names to provider voice IDs
- `AudioFormatMapper` - Maps Celeste format names to provider formats
- `SpeedMapper` - Maps speed parameter to provider-specific fields

## Key Architectural Patterns

### Gemini-Specific Considerations

Gemini uses the unified `generateContent` API with response modality configuration:

```python
# Request transformation
request = {
    "contents": [{"parts": [{"text": inputs.prompt}]}],
    "generationConfig": {
        "responseModalities": ["AUDIO"],
        "speechConfig": {
            "voiceConfig": {
                "prebuiltVoiceConfig": {
                    "voiceName": parameters.get("voice", "Kore")
                }
            }
        }
    }
}

# Response transformation
def _parse_content(self, response_data: dict) -> AudioArtifact:
    """Parse Gemini audio response.

    Gemini returns base64-encoded PCM data that needs decoding.
    """
    inline_data = response_data["candidates"][0]["content"]["parts"][0]["inlineData"]
    audio_bytes = base64.b64decode(inline_data["data"])
    mime_type = inline_data.get("mimeType", "audio/pcm")

    return AudioArtifact(data=audio_bytes, mime_type=mime_type)
```

### Multi-Format Support

Providers return different audio formats:
- **OpenAI**: mp3, opus, aac, flac, wav, pcm (client-specified)
- **Gemini**: PCM only (24kHz, 16-bit, mono)
- **ElevenLabs**: mp3, opus, pcm, μ-law, a-law (client-specified)
- **Amazon Polly**: mp3, ogg_vorbis, pcm (client-specified)

Client implementations must handle format-specific encoding in `AudioArtifact.mime_type`.

## Next Steps

1. **`/create-package audio-generation`** → Create package structure from this research
2. **`/research-provider openai audio-generation`** → OpenAI-specific implementation details
3. **`/implement-provider openai audio-generation`** → Implement OpenAI provider
4. **`/add-parameters openai audio-generation all`** → Add all parameter mappers
5. Repeat steps 2-4 for Gemini, ElevenLabs, and Amazon Polly

## Key Design Decisions

1. **Input**: Simple `prompt: str` pattern (text to synthesize)
2. **Output**: `AudioArtifact` for binary audio data with MIME type
3. **Usage**: Single optional field `characters_synthesized` (only Amazon Polly reports)
4. **Streaming**: Supported (3/4 providers) with `AudioGenerationChunk` yielding raw bytes
5. **FinishReason**: Not applicable (audio generation completes atomically)
6. **Parameters**: Universal parameters (voice, audio_format, speed) with provider-specific mappers
7. **Gemini Integration**: Use `generateContent` API with `responseModalities: ["AUDIO"]`
8. **Format Handling**: `AudioArtifact.mime_type` tracks format (mp3, pcm, opus, etc.)
