# Ondine - Complete Technical Reference

**Version**: 1.0.0
**Last Updated**: October 18, 2025
**Purpose**: Comprehensive technical documentation of every component, class, design decision, and relationship in the Ondine LLM Dataset Engine.

**Quick Navigation**:
- **Architecture Overview & Diagrams**: See [`ARCHITECTURE.md`](ARCHITECTURE.md) (auto-generated from `architecture/model.yaml`)
- **Design Decisions**: See [`architecture/decisions/`](architecture/decisions/) (ADRs)
- **Implementation Details**: This document (TECHNICAL_REFERENCE.md)

**Note**: This document provides detailed implementation information. For structural relationships and visual diagrams, see `ARCHITECTURE.md`. For design rationale and trade-offs, see the ADRs.

---

## Table of Contents

- [Part 1: Architecture Overview](#part-1-architecture-overview)
- [Part 2: External Dependencies](#part-2-external-dependencies)
- [Part 3: Layer 0 - Core Utilities](#part-3-layer-0---core-utilities)
- [Part 4: Core Models & Specifications](#part-4-core-models--specifications)
- [Part 5: Layer 1 - Infrastructure Adapters](#part-5-layer-1---infrastructure-adapters)
- [Part 6: Layer 2 - Processing Stages](#part-6-layer-2---processing-stages)
- [Part 7: Layer 3 - Orchestration Engine](#part-7-layer-3---orchestration-engine)
- [Part 8: Layer 4 - High-Level API](#part-8-layer-4---high-level-api)
- [Part 9: Configuration System](#part-9-configuration-system)
- [Part 10: CLI Interface](#part-10-cli-interface)
- [Part 11: Framework Integrations](#part-11-framework-integrations)
- [Part 12: Execution Flows](#part-12-execution-flows)
- [Part 13: Data Flows](#part-13-data-flows)
- [Part 14: Extension Points](#part-14-extension-points)

---

# Part 1: Architecture Overview

## 1.1 System Architecture

Ondine follows a **5-layer architecture**:

```mermaid
graph TB
    subgraph "Layer 4: High-Level API"
        API[Pipeline]
        Builder[PipelineBuilder]
        Composer[PipelineComposer]
        Processor[DatasetProcessor]
    end

    subgraph "Layer 3: Orchestration"
        Executor[PipelineExecutor]
        Strategy[ExecutionStrategy]
        SyncExec[SyncExecutor]
        AsyncExec[AsyncExecutor]
        StreamExec[StreamingExecutor]
        Context[ExecutionContext]
        StateManager[StateManager]
        Observers[Observers]
    end

    subgraph "Layer 2: Processing Stages"
        DataLoader[DataLoaderStage]
        PromptFormatter[PromptFormatterStage]
        BatchAgg[BatchAggregatorStage]
        LLMInvocation[LLMInvocationStage]
        BatchDisagg[BatchDisaggregatorStage]
        ResponseParser[ResponseParserStage]
        ResultWriter[ResultWriterStage]
    end

    subgraph "Layer 1: Infrastructure Adapters"
        LLMClient[LLM Client]
        DataIO[Data I/O]
        Checkpoint[Checkpoint Storage]
    end

    subgraph "Layer 0: Utilities"
        Retry[RetryHandler]
        RateLimit[RateLimiter]
        Cost[CostTracker]
        Budget[BudgetController]
        Logging[Logging Utils]
    end

    API --> Executor
    Builder --> API
    Executor --> Strategy
    Strategy --> SyncExec
    Strategy --> AsyncExec
    Strategy --> StreamExec
    Executor --> Context
    Executor --> StateManager
    Executor --> Observers
    Executor --> DataLoader
    DataLoader --> PromptFormatter
    PromptFormatter --> BatchAgg
    BatchAgg --> LLMInvocation
    LLMInvocation --> BatchDisagg
    BatchDisagg --> ResponseParser
    ResponseParser --> ResultWriter
    LLMInvocation --> LLMClient
    DataLoader --> DataIO
    ResultWriter --> DataIO
    StateManager --> Checkpoint
    LLMInvocation --> Retry
    LLMInvocation --> RateLimit
    LLMInvocation --> Cost
    Executor --> Budget
```

### Layer Responsibilities

| Layer | Directory | Purpose | Dependencies |
|-------|-----------|---------|--------------|
| **Layer 0** | `utils/` | Cross-cutting concerns (retry, rate limiting, cost tracking) | External libraries only |
| **Layer 1** | `adapters/` | External system integrations (LLM providers, file I/O) | Layer 0 + external APIs |
| **Layer 2** | `stages/` | Data transformation logic (load, format, batch, invoke, parse, write) | Layers 0-1 |
| **Layer 3** | `orchestration/` | Execution control and state management | Layers 0-2 |
| **Layer 4** | `api/` | User-facing interfaces (Pipeline, Builder) | All layers |

### Key Design Principles

1. **Dependency Rule**: Dependencies only point inward (higher layers depend on lower layers, never the reverse)
2. **Focused Components**: Each component has one clear purpose
3. **Extensible**: Open for extension, closed for modification
4. **Abstraction**: Depend on abstractions, not concretions
5. **Simple**: Keep it simple

## 1.2 Design Patterns Catalog

| Pattern | Where Used | Purpose | Implementation |
|---------|------------|---------|----------------|
| **Facade** | `Pipeline` | Simplify complex subsystem | Hides orchestration complexity |
| **Builder** | `PipelineBuilder` | Fluent construction API | Chainable method calls |
| **Strategy** | `ExecutionStrategy` | Pluggable execution modes | `SyncExecutor`, `AsyncExecutor`, `StreamingExecutor` |
| **Template Method** | `PipelineStage.execute()` | Standardized stage flow | Base class with hooks |
| **Observer** | `ExecutionObserver` | Monitoring hooks | `ProgressObserver`, `CostObserver`, `LoggingObserver` |
| **Adapter** | `LLMClient`, `DataReader` | Interface translation | Wrap external libraries |
| **Factory** | `ParserFactory` | Object creation | Create parsers by type |
| **Composite** | `PipelineComposer` | Tree structure | Compose pipelines |
| **Singleton** | Config instances | Single instance | Via module-level state |
| **Chain of Responsibility** | Stage pipeline | Sequential processing | Each stage processes then passes |
| **Protocol** | `TextCleaner` | Structural typing | Duck typing with validation |
| **Memento** | `ExecutionContext` | State capture | Serializable state |

## 1.3 Thread Safety Strategy

### Thread-Safe Components

| Component | Mechanism | Reason |
|-----------|-----------|--------|
| `CostTracker` | `threading.Lock` | Shared cost accumulation |
| `RateLimiter` | `threading.Lock` | Token bucket state |
| `CheckpointStorage` | File-based locks | Concurrent writes |
| `LLMInvocationStage` | `ThreadPoolExecutor` | Concurrent LLM calls |

### Thread-Unsafe Components (By Design)

| Component | Why Not Thread-Safe |
|-----------|-------------------|
| `PipelineBuilder` | Construction phase only, not shared |
| `Pipeline` (single exec) | Each execution is sequential |
| `DataLoaderStage` | Reads once, no shared state |

---

# Part 2: External Dependencies

## 2.1 Dependency Catalog

### Production Dependencies (20 libraries)

| Library | Version | Category | License | Why Chosen | Alternatives Considered |
|---------|---------|----------|---------|------------|------------------------|
| **litellm** | >=1.80.0 | LLM | MIT | Native multi-provider integration (100+ providers), Router for load balancing, automatic cost tracking, async-first | LlamaIndex wrappers (removed), LangChain (more complex), direct APIs (no abstraction) |
| **instructor** | >=1.0.0 | LLM | MIT | Type-safe structured output with Pydantic models, auto-retry on validation errors, JSON mode + function calling support | Manual JSON parsing (error-prone), LlamaIndex structured_predict (removed) |
| **llama-index** | >=0.12.0 | Future RAG | MIT | Preserved for future RAG capabilities (vector stores, query engines, embeddings). Currently unused. | Will be leveraged when RAG features are added |
| **pandas** | >=2.0.0 | Data | BSD-3 | Industry standard, rich API | Polars (less mature ecosystem) |
| **polars** | >=0.20.0 | Data | MIT | Fast Parquet reading | Pandas (slower for large files) |
| **pydantic** | >=2.0.0 | Validation | MIT | Validation + serialization + type hints | dataclasses (no validation), marshmallow (complex) |
| **python-dotenv** | >=1.0.0 | Config | BSD-3 | Simple .env loading | os.environ (manual) |
| **tqdm** | >=4.66.0 | UI | MPL/MIT | Simple, widely used progress bars | rich.progress (overkill), progressbar2 |
| **tenacity** | >=8.2.0 | Reliability | Apache-2.0 | Flexible retry logic | `backoff` (simpler but less flexible) |
| **openpyxl** | >=3.1.0 | Data | MIT | Excel file support | xlrd (deprecated), pyexcel |
| **pyarrow** | >=15.0.0 | Data | Apache-2.0 | Fast Parquet I/O | fastparquet (slower) |
| **tiktoken** | >=0.5.0 | LLM | MIT | Fast token counting, OpenAI-native | transformers (slower), estimate (inaccurate) |
| **structlog** | >=24.0.0 | Logging | MIT/Apache | Structured logging, JSON output | standard logging (unstructured) |
| **jinja2** | >=3.1.0 | Templating | BSD-3 | Powerful prompt templating | string.Template (too simple), mako |
| **prometheus-client** | >=0.20.0 | Monitoring | Apache-2.0 | Industry standard metrics | Custom metrics (reinvent wheel) |
| **click** | >=8.1.0 | CLI | BSD-3 | Decorator-based, simple | argparse (verbose), typer (overkill) |
| **rich** | >=13.0.0 | CLI | MIT | Beautiful tables and formatting | colorama (basic), termcolor |

### Dev Dependencies (8 libraries)

| Library | Purpose | Why Chosen |
|---------|---------|------------|
| **pytest** | Testing framework | Industry standard, rich plugins |
| **pytest-cov** | Coverage reporting | Integrates with pytest |
| **pytest-asyncio** | Async test support | Test async executors |
| **black** | Code formatting | Opinionated, consistent |
| **ruff** | Fast linting | Faster than flake8/pylint |
| **mypy** | Type checking | Catch type errors |
| **ipython** | Interactive shell | Better REPL |
| **jupyter** | Notebooks | Interactive exploration |

### Optional Dependencies (1 group)

| Group | Libraries | Purpose | Platform |
|-------|-----------|---------|----------|
| **mlx** | mlx>=0.29.0, mlx-lm>=0.28.0 | Apple Silicon local inference | macOS only (M1/M2/M3/M4) |

**Installation**: `pip install ondine[mlx]`

## 2.2 Dependency Graph

```mermaid
graph TD
    Ondine[Ondine Core]

    Ondine --> LiteLLM[litellm]
    Ondine --> Instructor[instructor]
    Ondine -.-> LI[llama-index]
    Ondine --> Pandas
    Ondine --> Pydantic
    Ondine --> Structlog
    Ondine --> Click
    Ondine --> Rich

    LiteLLM --> Providers[100+ Providers API]
    Instructor --> LiteLLM

    Pandas --> Openpyxl[openpyxl]
    Pandas --> Polars

    Ondine --> Tiktoken[tiktoken]
    Ondine --> Tenacity[tenacity]
    Ondine --> Tqdm[tqdm]
    Ondine --> Jinja2[jinja2]
    Ondine --> Prometheus[prometheus-client]
    Ondine --> Dotenv[python-dotenv]

    Ondine -.-> MLX[mlx + mlx-lm]
    MLX -.-> MLXMetal[mlx-metal]

    Polars --> PyArrow[pyarrow]

    style MLX stroke-dasharray: 5 5
    style MLXMetal stroke-dasharray: 5 5
    style LI stroke-dasharray: 5 5
```

**Notes**:
- Dashed lines indicate optional/future dependencies
- `llama-index` preserved for future RAG (currently unused)
- `mlx` for Apple Silicon only (`pip install ondine[mlx]`)

### Critical Dependencies (Cannot Be Removed)

1. **litellm** - Core LLM dependency providing:
   - **100+ provider support** (OpenAI, Anthropic, Groq, Azure, Cohere, AI21, etc.)
   - **Native async** via `litellm.acompletion()`
   - **Router** for load balancing and failover
   - **Automatic cost tracking** via `litellm.completion_cost()`
   - **Redis caching** for response deduplication
2. **instructor** - Structured output dependency providing:
   - **Type-safe Pydantic models** with automatic validation
   - **Auto-retry** on validation errors (max_retries=3)
   - **Dual-path support**: JSON mode (Groq) + function calling (OpenAI/Anthropic)
3. **llama-index** - Future RAG dependency:
   - **Currently unused** (preserved for future RAG integration)
   - Will provide: vector stores, embeddings, query engines, retrieval
4. **pandas** - Data manipulation backbone, used throughout
5. **pydantic** - Configuration validation, type safety
6. **structlog** - Structured logging, observability

### Optional Dependencies (Can Be Removed)

1. **polars** - Only for fast Parquet reading, pandas can handle it (slower)
2. **prometheus-client** - Only for metrics export, can be disabled
3. **rich** - Only for CLI pretty printing, can fall back to basic output
4. **jinja2** - Only for advanced templating, can use string.format()
5. **mlx + mlx-lm** - Only for Apple Silicon local inference, use cloud providers instead

---

# Part 5: Layer 1 - Infrastructure Adapters (LLM Providers)

## 5.1 LLM Provider Overview

Ondine supports 100+ LLM providers through **native LiteLLM integration**, using the **Adapter pattern** for clean abstraction and framework swappability.

### Architecture Evolution (v1.4.0)

**Before (v1.3.x)**: LlamaIndex wrappers
```
User → PipelineBuilder → Factory → LlamaIndex Wrapper → LiteLLM → Provider API
```

**After (v1.4.0)**: Native LiteLLM + Instructor
```
User → PipelineBuilder → UnifiedLiteLLMClient → LiteLLM (direct) → Provider API
                                                → Instructor (structured output)
```

**Benefits:**
- ✅ Direct `litellm.acompletion()` access (async-first)
- ✅ Router for load balancing (automatic failover)
- ✅ Redis caching (avoid duplicate calls)
- ✅ Instructor for structured output (type-safe Pydantic)
- ✅ Cleaner code (-673 lines of wrappers)
- ✅ 100+ providers supported (vs 5 before)

### Supported Providers

**All 100+ LiteLLM providers supported**, including:

| Provider | Category | Platform | Cost | Use Case |
|----------|----------|----------|------|----------|
| **OpenAI** | Cloud API | All | $$ | Production, high quality, GPT-4 |
| **Azure OpenAI** | Cloud API | All | $$ | Enterprise, compliance, private deployments |
| **Anthropic** | Cloud API | All | $$$ | Claude models, 200K context |
| **Groq** | Cloud API | All | Free tier | Ultra-fast inference, development |
| **Together.AI** | Cloud API | All | $ | Affordable, fast, open models |
| **Cohere** | Cloud API | All | $$ | Embed, rerank, command models |
| **AI21** | Cloud API | All | $$$ | Jurassic models |
| **Mistral** | Cloud API | All | $$ | European provider, open models |
| **Ollama** | Local | All | Free | Privacy, offline, self-hosted |
| **vLLM** | Self-hosted | All | Free | Custom deployments, full control |
| **MLX** | Local | macOS (Apple Silicon) | Free | Apple Silicon local inference, offline |
| **+90 more** | Various | Various | Varies | See: https://docs.litellm.ai/docs/providers |

### Provider Selection Guide

**Choose OpenAI if**: Production quality, mature ecosystem, GPT-4
**Choose Azure if**: Enterprise compliance, private deployments, Managed Identity
**Choose Anthropic if**: Claude models, 200K context, thinking models
**Choose Groq if**: Ultra-fast inference (500+ tokens/sec), free tier
**Choose Together.AI if**: Affordable cloud inference, open models
**Choose Ollama if**: Privacy, offline, local deployment
**Choose MLX if**: Apple Silicon Mac, free, private, offline

---

## 5.2 UnifiedLiteLLMClient (Core LLM Adapter)

### Purpose
Single unified client for all cloud-based LLM providers using native LiteLLM integration. Replaced 673 lines of LlamaIndex wrapper code with direct `litellm.acompletion()` calls in v1.4.0 migration.

### Class: `UnifiedLiteLLMClient`

**Location**: `ondine/adapters/unified_litellm_client.py` (365 lines)

**Inheritance**: `LLMClient` (Adapter pattern)

**Platform**: All (cloud-based providers)

**Responsibility**: Wrap ALL LiteLLM features in clean abstraction for framework swappability

### Key Features

1. **Native Async**: Direct `litellm.acompletion()` integration (async-first design)
2. **Instructor Integration**: Type-safe structured output with Pydantic models
3. **Router Support**: Load balancing across multiple providers/deployments
4. **Redis Caching**: Automatic response caching via `litellm.cache`
5. **Prefix Caching**: Automatic detection and logging of cached tokens
6. **100+ Providers**: Supports all LiteLLM providers (OpenAI, Anthropic, Groq, Azure, Cohere, etc.)
7. **Auto-Cost**: Automatic cost tracking via `litellm.completion_cost()`

### Architecture

```python
class UnifiedLiteLLMClient(LLMClient):
    """
    Unified client for all cloud LLM providers via LiteLLM.

    Features:
    - litellm.acompletion() for async API calls
    - litellm.Router for load balancing
    - litellm.cache for Redis caching
    - litellm.completion_cost() for cost tracking
    - Instructor for structured output

    Supports: 100+ providers
    """

    def __init__(self, spec: LLMSpec):
        # Setup environment variables for LiteLLM
        # Initialize Router if configured
        # Initialize Redis cache if configured

    async def ainvoke(self, prompt: str, **kwargs) -> LLMResponse:
        # Primary async method using litellm.acompletion()
        # Handles Router and caching automatically

    def invoke(self, prompt: str, **kwargs) -> LLMResponse:
        # Sync wrapper around ainvoke()

    async def structured_invoke_async(
        self, prompt: str, output_cls: type[BaseModel], **kwargs
    ) -> LLMResponse:
        # Structured output via Instructor
        # Auto-detects mode (JSON vs function calling)

    def structured_invoke(
        self, prompt: str, output_cls: type[BaseModel], **kwargs
    ) -> LLMResponse:
        # Sync wrapper for structured output
```

### Supported Providers

**100+ providers via LiteLLM**, examples:

```python
# OpenAI
model="gpt-4o-mini"  # or "openai/gpt-4o-mini"

# Azure
model="azure/your-deployment-name"

# Anthropic
model="claude-3-5-haiku-20241022"

# Groq
model="groq/llama-3.3-70b-versatile"

# Together.AI
model="together_ai/meta-llama/Meta-Llama-3.1-70B"

# Cohere, Mistral, AI21, Bedrock, Vertex AI, etc...
# See: https://docs.litellm.ai/docs/providers
```

### Async-First Design

**Primary Method**: `ainvoke()` (native async)
```python
async def ainvoke(self, prompt: str, **kwargs) -> LLMResponse:
    response = await litellm.acompletion(
        model=self.model_identifier,
        messages=self._build_messages(prompt, kwargs),
        temperature=self.spec.temperature,
        max_tokens=self.spec.max_tokens,
    )
    return self._parse_response(response, prompt)
```

**Why Async-First**:
- ✅ True non-blocking I/O (not thread-based)
- ✅ Enables AsyncExecutor for concurrency
- ✅ Future Router async support
- ✅ Matches LiteLLM's native async API

### Structured Output with Instructor

**Dual-Path Strategy** with auto-detection:

1. **Mode.TOOLS** - Native function calling (OpenAI, Anthropic)
2. **Mode.JSON** - JSON mode (Groq, safe fallback)

**Auto-Detection**:
```python
# Groq → JSON mode
if "groq" in model.lower():
    mode = instructor.Mode.JSON

# OpenAI/Anthropic → TOOLS mode
elif any(p in model.lower() for p in ["gpt", "claude"]):
    mode = instructor.Mode.TOOLS

# Default → JSON mode (safest)
else:
    mode = instructor.Mode.JSON
```

**Example**:
```python
from pydantic import BaseModel

class BaconProduct(BaseModel):
    cleaned_description: str
    pack_size: float | None

# Automatic validation + retry
response = client.structured_invoke(
    prompt="Clean: Bacon, 16 oz pack",
    output_cls=BaconProduct
)
# Returns validated BaconProduct instance
```

### Router for Load Balancing

**Enable via PipelineBuilder**:
```python
.with_router(
    model_list=[
        {
            "model_name": "fast-llm",
            "litellm_params": {
                "model": "groq/llama-3.3-70b-versatile",
                "api_key": os.getenv("GROQ_API_KEY")
            }
        },
        {
            "model_name": "fast-llm",  # Same name = fallback
            "litellm_params": {
                "model": "openai/gpt-4o-mini",
                "api_key": os.getenv("OPENAI_API_KEY")
            }
        }
    ],
    routing_strategy="simple-shuffle"
)
```

**Router Features** (via LiteLLM):
- Load balancing across deployments
- Automatic failover on errors
- Built-in retries (exponential backoff)
- Cooldowns for failing providers
- RPM/TPM limits per deployment
- Redis for distributed state

**Docs**: https://docs.litellm.ai/docs/routing

### Redis Caching

**Enable via PipelineBuilder**:
```python
.with_redis_cache(
    redis_url="redis://localhost:6379",
    ttl=3600  # 1 hour
)
```

**How it works**:
- LiteLLM caches based on: (model, messages, temperature)
- Cache hit: Returns cached response (zero cost!)
- Distributed across processes
- TTL-based expiration

**Docs**: https://docs.litellm.ai/docs/caching

### Prefix Caching Detection

**Automatic logging of cached tokens**:
```python
# System messages sent separately:
messages = [
    {"role": "system", "content": "Long instructions (1024+ tokens)"},
    {"role": "user", "content": prompt}
]

# If provider caches system message:
# [INFO] ✅ Cache hit! 1024/2048 tokens cached (50%)
```

**Provider Support**:
- OpenAI: Auto-caches system messages >1024 tokens (50% discount)
- Anthropic: Auto-caches system messages (90% discount)
- Groq: Model-specific support

**Docs**: https://docs.litellm.ai/docs/completion/prompt_caching

### Cost Tracking

**Automatic via LiteLLM**:
```python
cost = litellm.completion_cost(
    model=self.model_identifier,
    prompt=prompt_text,
    completion=response_text
)
```

**Features**:
- Up-to-date pricing (LiteLLM maintains pricing DB)
- Automatic detection (no manual config needed)
- Manual override available

### Testing Coverage

**Unit Tests** (27 tests):
- Async/sync invoke
- All providers (OpenAI, Groq, Anthropic, Azure)
- Cost calculation
- Token estimation
- Router initialization
- Cache initialization

**Integration Tests** (8 tests):
- E2E with real APIs (OpenAI, Groq, Anthropic)
- Structured output validation
- Prefix caching verification
- Router failover

**Test Results**: 461/461 unit tests passing, 103/103 integration tests passing

### Migration from v1.3.x

**Old (LlamaIndex wrapper)**:
```python
from ondine.adapters.litellm_client import LiteLLMClient
# Used LlamaIndex's LiteLLM wrapper
```

**New (Native LiteLLM)**:
```python
from ondine.adapters.unified_litellm_client import UnifiedLiteLLMClient
# Uses litellm.acompletion() directly
```

**Backward Compatibility**: ✅ 100% - factory routes automatically, user code unchanged

---

## 5.6 OpenAI-Compatible Provider (Deprecated)

**Note**: As of v1.4.0, custom OpenAI-compatible endpoints (Ollama, vLLM, Together.AI) are handled by `UnifiedLiteLLMClient` using LiteLLM's native `base_url` support.

**Migration**:
```python
# Old (v1.3.x)
.with_llm(
    provider="openai_compatible",
    model="llama-3.1-70b",
    base_url="http://localhost:11434/v1"
)

# New (v1.4.0) - Still works! Routed to UnifiedLiteLLMClient
.with_llm(
    provider="openai_compatible",
    model="llama-3.1-70b",
    base_url="http://localhost:11434/v1"
)
```

**`OpenAICompatibleClient` class removed** - functionality now in `UnifiedLiteLLMClient`

---

## 5.7 Azure OpenAI Provider (Deprecated)

**Note**: As of v1.4.0, Azure OpenAI is handled by `UnifiedLiteLLMClient` using LiteLLM's native Azure support.

**Migration**:
```python
# Old (v1.3.x)
from ondine.adapters.llm_client import AzureOpenAIClient

# New (v1.4.0) - Handled by UnifiedLiteLLMClient
# Just use standard builder API:
.with_llm(
    provider="azure_openai",
    model="gpt-4",
    azure_endpoint="https://your-endpoint.openai.azure.com/",
    azure_deployment="your-deployment"
)
```

**Azure Managed Identity**:
- Still supported via `AZURE_AD_TOKEN` environment variable
- LiteLLM handles authentication
- Set `AZURE_AD_TOKEN` before running pipeline

**`AzureOpenAIClient` class removed** - functionality now in `UnifiedLiteLLMClient`

---

## 5.8 OpenAI-Compatible Provider (Custom APIs)

### Purpose
Enable integration with any LLM API that implements the OpenAI chat completions format.

### Class: `OpenAICompatibleClient`

**Inheritance**: `LLMClient` (Adapter pattern)

**Platform**: All (platform-agnostic)

**Responsibility**: Connect to custom OpenAI-compatible API endpoints

**Supports**:
- **Ollama** (local LLM server)
- **vLLM** (self-hosted inference)
- **Together.AI** (cloud API)
- **Anyscale** (cloud API)
- **LocalAI** (self-hosted)
- **Any custom OpenAI-compatible API**

### Configuration

**Required Fields**:
- `provider: openai_compatible`
- `base_url`: Custom API endpoint URL
- `model`: Model identifier

**Optional Fields**:
- `provider_name`: Custom name for logging/metrics
- `api_key`: Authentication (or use env var, or "dummy" for local)
- `input_cost_per_1k_tokens`: Custom pricing
- `output_cost_per_1k_tokens`: Custom pricing

### Architecture

```python
class OpenAICompatibleClient(LLMClient):
    def __init__(self, spec: LLMSpec):
        # Validate base_url is provided
        # Initialize OpenAI client with custom base_url
        # Use provider_name for metrics

    def invoke(self, prompt: str, **kwargs) -> LLMResponse:
        # Call custom API using OpenAI format
        # Return standardized LLMResponse
```

**Design Decision: Reuse OpenAI Client**

Instead of reimplementing HTTP calls, leverage llama-index's OpenAI client with custom `api_base`:

```python
self.client = OpenAI(
    model=spec.model,
    api_key=api_key or "dummy",
    api_base=spec.base_url,  # Custom URL
)
```

**Rationale**:
- DRY: Reuse existing, well-tested code
- Reliability: llama-index handles edge cases
- Compatibility: Ensures OpenAI format compliance
- Maintainability: Updates to OpenAI client benefit us

### Example Configurations

#### Local Ollama (Free)
```yaml
llm:
  provider: openai_compatible
  provider_name: "Ollama-Local"
  model: llama3.1:70b
  base_url: http://localhost:11434/v1
  # No API key needed
  input_cost_per_1k_tokens: 0.0
  output_cost_per_1k_tokens: 0.0
```

#### Together.AI (Cloud)
```yaml
llm:
  provider: openai_compatible
  provider_name: "Together.AI"
  model: meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo
  base_url: https://api.together.xyz/v1
  api_key: \${TOGETHER_API_KEY}
  input_cost_per_1k_tokens: 0.0006
  output_cost_per_1k_tokens: 0.0006
```

#### Self-Hosted vLLM
```yaml
llm:
  provider: openai_compatible
  provider_name: "vLLM-Custom"
  model: meta-llama/Llama-3.1-70B-Instruct
  base_url: http://your-vllm-server:8000/v1
  input_cost_per_1k_tokens: 0.0  # Self-hosted = free
  output_cost_per_1k_tokens: 0.0
```

### Validation

**Pydantic Model Validator**:
```python
@model_validator(mode="after")
def validate_provider_requirements(self) -> "LLMSpec":
    if self.provider == LLMProvider.OPENAI_COMPATIBLE and self.base_url is None:
        raise ValueError("base_url required for openai_compatible provider")
    return self
```

**Design**: Fail fast with clear error message

### Token Estimation

Uses tiktoken with cl100k_base encoding (approximation):
```python
self.tokenizer = tiktoken.get_encoding("cl100k_base")
tokens = len(self.tokenizer.encode(text))
```

**Tradeoff**: Not model-specific, but consistent and fast

### Performance Characteristics

**Depends on backend**:
- **Ollama local**: ~10-20 tokens/sec (70B model, single GPU)
- **vLLM multi-GPU**: ~50-80 tokens/sec (70B model, tensor parallel)
- **Together.AI cloud**: ~30-50 tokens/sec (network latency)

### Authentication Handling

**Priority order**:
1. `spec.api_key` (explicit in config)
2. `OPENAI_COMPATIBLE_API_KEY` env var
3. `"dummy"` fallback (for local APIs without auth)

**Design**: Flexible authentication for different scenarios

### Provider Naming

The `provider_name` field appears in metrics/logging:
```python
model=f"{self.provider_name}/{self.model}"
# Example: "Together.AI/meta-llama/Llama-3.1-70B"
```

**Benefit**: Clear identification in logs and cost tracking

### Dependencies

```python
from llama_index.llms.openai import OpenAI  # Reuse OpenAI client
import tiktoken  # Token estimation
```

### Used By

- `create_llm_client()` factory
- Any pipeline with `provider: openai_compatible`
- Examples: Ollama, Together.AI, vLLM configs

### Testing Strategy

**Unit Tests (14 tests)**:
- Validation (base_url requirement)
- Provider naming
- Dummy API keys for local APIs
- Cost calculation with custom pricing
- Factory integration
- Backward compatibility

### Known Limitations

- Assumes OpenAI-compatible format (doesn't support exotic APIs)
- Token counting is approximate (cl100k_base encoding)
- Can't use provider-specific features (only OpenAI format)

### Future Improvements

- [ ] Support custom headers (for exotic auth)
- [ ] Add provider-specific tokenizers
- [ ] Support streaming responses
- [ ] Add connection pooling for performance

---

---

# Part 3: Layer 0 - Core Utilities

## 3.1 Overview

Layer 0 provides cross-cutting concerns that are used throughout the system:
- Retry logic with exponential backoff
- Rate limiting (token bucket algorithm)
- Cost tracking and accumulation
- Budget enforcement
- Structured logging
- Metrics export (Prometheus)
- Input text preprocessing

**Design Principle**: These utilities have **no dependencies on higher layers**, only on external libraries.

---

## 3.2 `utils/retry_handler.py`

### Purpose
Implements robust retry logic with exponential backoff for transient failures.

### Classes

#### `RetryableError` (Exception)

**Inheritance**: `Exception`

**Purpose**: Base class for errors that should be retried

**Design Decision**: Create custom exception hierarchy to distinguish retryable vs. non-retryable errors

```python
class RetryableError(Exception):
    """Base class for errors that should be retried."""
    pass
```

#### `RateLimitError` (Exception)

**Inheritance**: `RetryableError`

**Purpose**: Specific error for rate limit scenarios

**Usage**: Raised by LLM clients when rate limits are hit

#### `NetworkError` (Exception)

**Inheritance**: `RetryableError`

**Purpose**: Specific error for network-related failures

**Usage**: Raised when network calls fail transiently

#### `RetryHandler` (Class)

**Responsibility**: Handle retry logic with configurable exponential backoff

**Single Responsibility**: ONLY handles retry logic, nothing else

**Attributes**:
```python
max_attempts: int           # Maximum retry attempts (default: 3)
initial_delay: float        # Initial delay in seconds (default: 1.0)
max_delay: float            # Maximum delay cap (default: 60.0)
exponential_base: int       # Base for exponential calculation (default: 2)
retryable_exceptions: tuple # Which exceptions to retry
```

**Methods**:

##### `__init__(max_attempts, initial_delay, max_delay, exponential_base, retryable_exceptions)`

**Purpose**: Initialize retry configuration

**Parameters**:
- `max_attempts: int = 3` - How many times to retry
- `initial_delay: float = 1.0` - Starting delay
- `max_delay: float = 60.0` - Maximum delay cap
- `exponential_base: int = 2` - Exponent base (delay = initial * base^attempt)
- `retryable_exceptions: Optional[tuple] = None` - Which exceptions trigger retry

**Design Decision**: Make all parameters configurable for flexibility across different failure scenarios

**Default Behavior**: If `retryable_exceptions` is None, defaults to `(RetryableError, RateLimitError, NetworkError)`

##### `execute(func: Callable[[], T]) -> T`

**Purpose**: Execute a function with retry logic

**Algorithm**:
1. Create `Retrying` instance from tenacity library
2. Configure stop condition: `stop_after_attempt(max_attempts)`
3. Configure wait strategy: exponential backoff with `wait_exponential()`
4. Configure retry condition: `retry_if_exception_type(retryable_exceptions)`
5. Execute function through retryer
6. If all retries fail, reraise last exception

**Parameters**:
- `func: Callable[[], T]` - Zero-argument function to execute

**Returns**: `T` - Result from successful function execution

**Raises**: Last exception if all retries exhausted

**Example**:
```python
retry_handler = RetryHandler(max_attempts=3, initial_delay=1.0)

def risky_operation():
    # Might fail with RateLimitError
    return call_llm_api()

result = retry_handler.execute(risky_operation)
```

**Design Decision**: Use tenacity library instead of custom implementation
- **Why**: tenacity is battle-tested, handles edge cases, provides flexible configuration
- **Alternative**: Custom retry loop (simpler but less robust)
- **Trade-off**: Additional dependency vs. reliability

##### `calculate_delay(attempt: int) -> float`

**Purpose**: Calculate delay for given attempt number (for informational/testing purposes)

**Algorithm**:
```python
delay = initial_delay * (exponential_base ** (attempt - 1))
return min(delay, max_delay)
```

**Parameters**:
- `attempt: int` - Attempt number (1-based)

**Returns**: `float` - Delay in seconds

**Example**:
```python
handler = RetryHandler(initial_delay=1.0, exponential_base=2, max_delay=60.0)
handler.calculate_delay(1)  # 1.0 second
handler.calculate_delay(2)  # 2.0 seconds
handler.calculate_delay(3)  # 4.0 seconds
handler.calculate_delay(4)  # 8.0 seconds
handler.calculate_delay(10) # 60.0 seconds (capped at max_delay)
```

### Thread Safety

- **Thread-safe**: Yes (stateless operation, no shared mutable state)
- Each `execute()` call is independent
- tenacity handles thread safety internally

### Dependencies

```python
import time                  # For delays
from typing import Callable  # Type hints
from tenacity import (       # Retry library
    Retrying,
    retry_if_exception_type,
    stop_after_attempt,
    wait_exponential,
)
```

### Used By

- `LLMInvocationStage` - Retries LLM API calls
- Any component needing retry logic

### Design Patterns

1. **Strategy Pattern**: Configurable retry strategy via parameters
2. **Dependency Inversion**: Uses abstract `Callable` type, not specific implementations

### Time Complexity

- **Exponential backoff**: O(2^n) time in worst case (where n = max_attempts)
- Actual delays: 1s, 2s, 4s, 8s, 16s, ...

### Testing Considerations

- Test with mocked functions that fail predictably
- Verify exponential backoff timing
- Test max_attempts enforcement
- Test exception filtering (retryable vs. non-retryable)

### Known Limitations

- No jitter (could add randomness to prevent thundering herd)
- Fixed exponential formula (could support other backoff strategies)

### Future Improvements

- [ ] Add jitter support for distributed systems
- [ ] Support custom backoff strategies (linear, polynomial)
- [ ] Add metrics for retry count tracking

---

## 3.3 `utils/rate_limiter.py`

### Purpose
Implements token bucket algorithm for rate limiting API calls.

### Classes

#### `RateLimiter` (Class)

**Responsibility**: Control request rate using token bucket algorithm

**Algorithm**: Token Bucket
- Bucket has maximum capacity (burst_size)
- Tokens refill at constant rate (requests_per_minute / 60)
- Each request consumes tokens
- If no tokens available, wait until refilled

**Thread Safety**: **Thread-safe** via `threading.Lock`

**Attributes**:
```python
rpm: int                    # Requests per minute limit
capacity: int               # Maximum burst size
tokens: float               # Current available tokens
last_update: float          # Last refill timestamp
lock: threading.Lock        # Thread safety
refill_rate: float          # Tokens per second
```

**Methods**:

##### `__init__(requests_per_minute: int, burst_size: Optional[int] = None)`

**Purpose**: Initialize rate limiter with capacity and refill rate

**Parameters**:
- `requests_per_minute: int` - Maximum requests allowed per minute
- `burst_size: Optional[int] = None` - Maximum burst size (defaults to `requests_per_minute`)

**Design Decision**: Separate burst_size from requests_per_minute
- **Why**: Allow bursts up to capacity, then throttle to sustained rate
- **Example**: `rpm=60, burst=120` allows 120 immediate requests, then throttles to 1/second

**Initialization**:
```python
self.rpm = requests_per_minute
self.capacity = burst_size or requests_per_minute
self.tokens = float(self.capacity)  # Start with full capacity
self.last_update = time.time()
self.lock = threading.Lock()
self.refill_rate = requests_per_minute / 60.0  # Tokens per second
```

##### `acquire(tokens: int = 1, timeout: Optional[float] = None) -> bool`

**Purpose**: Acquire tokens for making requests (blocks until available)

**Algorithm**:
1. Check if requested tokens <= capacity (raise ValueError if not)
2. Loop:
   a. Acquire lock
   b. Refill tokens based on elapsed time
   c. If sufficient tokens available:
      - Deduct tokens
      - Return True
   d. Release lock
   e. Check timeout
   f. Sleep briefly (0.1s) before retry

**Parameters**:
- `tokens: int = 1` - Number of tokens to acquire
- `timeout: Optional[float] = None` - Maximum wait time (None = wait forever)

**Returns**: `bool`
- `True` if tokens acquired
- `False` if timeout reached

**Raises**: `ValueError` if `tokens > capacity`

**Thread Safety**: Lock protects token bucket state during check-and-decrement

**Example**:
```python
limiter = RateLimiter(requests_per_minute=60, burst_size=120)

# Acquire 1 token (default)
if limiter.acquire():
    make_api_call()

# Acquire with timeout
if limiter.acquire(tokens=1, timeout=5.0):
    make_api_call()
else:
    print("Timeout waiting for rate limit")

# Acquire multiple tokens
if limiter.acquire(tokens=5):
    make_batch_api_call(batch_size=5)
```

##### `_refill() -> None`

**Purpose**: Refill tokens based on elapsed time (internal method)

**Algorithm**:
```python
now = time.time()
elapsed = now - self.last_update
tokens_to_add = elapsed * self.refill_rate
self.tokens = min(self.capacity, self.tokens + tokens_to_add)
self.last_update = now
```

**Design Decision**: Continuous refill vs. discrete intervals
- **Chosen**: Continuous refill (calculate tokens based on elapsed time)
- **Why**: More accurate, smoother rate limiting
- **Alternative**: Refill in discrete chunks (simpler but less accurate)

**Thread Safety**: Called only while holding lock

##### `available_tokens` (Property)

**Purpose**: Get current available tokens (thread-safe)

**Returns**: `float` - Current token count

**Thread Safety**: Acquires lock, refills, returns tokens

**Example**:
```python
limiter = RateLimiter(requests_per_minute=60)
print(f"Available: {limiter.available_tokens}")  # e.g., 45.3
```

##### `reset() -> None`

**Purpose**: Reset rate limiter to full capacity

**Thread Safety**: Acquires lock, resets tokens and timestamp

**Use Case**: Manual reset after known idle period

### Token Bucket Algorithm

**Visual**:
```
Bucket (capacity = 100)
┌─────────────┐
│ ▓▓▓▓▓▓▓▓    │  Current: 80 tokens
│             │  Refill rate: 60/minute = 1/second
└─────────────┘

After 5 seconds:
┌─────────────┐
│ ▓▓▓▓▓▓▓▓▓▓▓ │  Current: 85 tokens (80 + 5*1)
└─────────────┘

After acquire(10):
┌─────────────┐
│ ▓▓▓▓▓▓▓▓    │  Current: 75 tokens
└─────────────┘
```

**Properties**:
- Allows bursts up to capacity
- Refills continuously at constant rate
- Smooth rate limiting (no hard boundaries)

### Dependencies

```python
import threading          # Lock for thread safety
import time              # Timestamp tracking
from typing import Optional
```

### Used By

- `LLMInvocationStage` - Rate limit API calls
- Any component needing request throttling

### Thread Safety Details

```python
with self.lock:  # Critical section
    self._refill()
    if self.tokens >= tokens:
        self.tokens -= tokens
        return True
```

**Race Condition Prevention**:
- Without lock: Two threads could both see sufficient tokens, both decrement, exceed capacity
- With lock: Only one thread can check-and-decrement at a time

### Performance

- **Time Complexity**: O(1) per acquire
- **Space Complexity**: O(1) - constant space
- **Lock Contention**: Low (critical section is very short)

### Configuration Examples

```python
# Conservative (prevent rate limits)
limiter = RateLimiter(requests_per_minute=30, burst_size=30)

# Aggressive (maximize throughput)
limiter = RateLimiter(requests_per_minute=100, burst_size=200)

# Bursty workload (allow initial burst, then throttle)
limiter = RateLimiter(requests_per_minute=60, burst_size=180)
```

### Testing

```python
def test_rate_limiter():
    limiter = RateLimiter(requests_per_minute=60)

    # Should acquire immediately
    assert limiter.acquire(timeout=0.1)

    # Deplete tokens
    for _ in range(59):
        limiter.acquire()

    # Should timeout (no tokens left)
    assert not limiter.acquire(timeout=0.1)

    # Wait for refill
    time.sleep(2.0)
    assert limiter.acquire()  # Should have ~2 tokens refilled
```

### Known Limitations

- No distributed rate limiting (single-process only)
- Fixed refill rate (can't adjust dynamically)
- Polling-based waiting (could use condition variables)

### Future Improvements

- [ ] Add distributed rate limiting (Redis-based)
- [ ] Support dynamic rate adjustment
- [ ] Use condition variables instead of polling
- [ ] Add metrics for rate limit hits

---

## 3.4 `utils/cost_tracker.py`

### Purpose
Thread-safe cost tracking for LLM API usage with detailed breakdowns.

### Classes

#### `CostEntry` (Dataclass)

**Purpose**: Single cost tracking entry

**Attributes**:
```python
tokens_in: int      # Input tokens consumed
tokens_out: int     # Output tokens generated
cost: Decimal       # Cost for this entry
model: str          # Model identifier
timestamp: float    # When request occurred
```

**Design Decision**: Use dataclass for simplicity and immutability

#### `CostTracker` (Class)

**Responsibility**: Accumulate and track costs across LLM calls

**Single Responsibility**: ONLY handles cost accounting, not enforcement (that's BudgetController's job)

**Thread Safety**: **Thread-safe** via `threading.Lock`

**Attributes**:
```python
input_cost_per_1k: Decimal       # Input token price per 1K tokens
output_cost_per_1k: Decimal      # Output token price per 1K tokens
_total_input_tokens: int         # Cumulative input tokens
_total_output_tokens: int        # Cumulative output tokens
_total_cost: Decimal             # Cumulative cost (Decimal for precision)
_entries: list[CostEntry]        # Detailed entry log
_stage_costs: Dict[str, Decimal] # Cost breakdown by stage
_lock: threading.Lock            # Thread safety
```

**Design Decision**: Use `Decimal` for cost (not `float`)
- **Why**: Avoid floating-point precision errors (`0.1 + 0.2 != 0.3`)
- **Critical**: Financial calculations must be exact
- **Trade-off**: Slightly slower than float, but necessary for accuracy

**Methods**:

##### `__init__(input_cost_per_1k, output_cost_per_1k)`

**Purpose**: Initialize cost tracker with pricing

**Parameters**:
- `input_cost_per_1k: Optional[Decimal] = None` - Price per 1K input tokens
- `output_cost_per_1k: Optional[Decimal] = None` - Price per 1K output tokens

**Default**: Both default to `Decimal("0.0")` if not provided

**Initialization**:
```python
self.input_cost_per_1k = input_cost_per_1k or Decimal("0.0")
self.output_cost_per_1k = output_cost_per_1k or Decimal("0.0")
self._total_input_tokens = 0
self._total_output_tokens = 0
self._total_cost = Decimal("0.0")
self._entries = []
self._stage_costs = {}
self._lock = threading.Lock()
```

##### `add(tokens_in, tokens_out, model, timestamp, stage) -> Decimal`

**Purpose**: Add cost entry and return cost for this operation

**Algorithm**:
1. Calculate cost: `(tokens_in/1000) * input_price + (tokens_out/1000) * output_price`
2. Acquire lock
3. Create `CostEntry` and append to `_entries`
4. Accumulate tokens and cost
5. Update stage-specific costs
6. Release lock
7. Return calculated cost

**Parameters**:
- `tokens_in: int` - Input tokens used
- `tokens_out: int` - Output tokens generated
- `model: str` - Model identifier
- `timestamp: float` - Request timestamp
- `stage: Optional[str] = None` - Stage name for breakdown

**Returns**: `Decimal` - Cost for this entry

**Thread Safety**: Entire operation protected by lock to ensure atomic update

**Example**:
```python
tracker = CostTracker(
    input_cost_per_1k=Decimal("0.00005"),  # $0.05 per 1M input tokens
    output_cost_per_1k=Decimal("0.00008"),  # $0.08 per 1M output tokens
)

cost = tracker.add(
    tokens_in=1000,
    tokens_out=500,
    model="gpt-4o-mini",
    timestamp=time.time(),
    stage="llm_invocation"
)
# cost = (1000/1000 * 0.00005) + (500/1000 * 0.00008)
#      = 0.00005 + 0.00004
#      = 0.00009 ($0.00009)
```

##### `calculate_cost(tokens_in, tokens_out) -> Decimal`

**Purpose**: Calculate cost for given token counts (without recording)

**Algorithm**:
```python
input_cost = (Decimal(tokens_in) / 1000) * self.input_cost_per_1k
output_cost = (Decimal(tokens_out) / 1000) * self.output_cost_per_1k
return input_cost + output_cost
```

**Use Case**: Estimate cost before making request

**Example**:
```python
estimated = tracker.calculate_cost(tokens_in=500, tokens_out=200)
if estimated > Decimal("0.01"):  # More than 1 cent
    print("Expensive request!")
```

##### `total_cost` (Property)

**Purpose**: Get total accumulated cost (thread-safe)

**Returns**: `Decimal` - Total cost

**Thread Safety**: Acquires lock, reads `_total_cost`, releases lock

##### `total_tokens`, `input_tokens`, `output_tokens` (Properties)

**Purpose**: Get token counts (thread-safe)

**Returns**: `int` - Token count

##### `get_estimate(rows) -> CostEstimate`

**Purpose**: Create cost estimate object from current tracking

**Returns**: `CostEstimate` model with:
- `total_cost`
- `total_tokens`
- `input_tokens`
- `output_tokens`
- `rows` - Number of rows processed
- `breakdown_by_stage` - Dict of stage costs
- `confidence="actual"` - These are actual costs, not estimates

##### `reset() -> None`

**Purpose**: Clear all tracking data (thread-safe)

**Use Case**: Reset between test runs or pipeline executions

##### `get_stage_costs() -> Dict[str, Decimal]`

**Purpose**: Get cost breakdown by stage

**Returns**: Dictionary mapping stage names to costs

**Example**:
```python
stage_costs = tracker.get_stage_costs()
# {"llm_invocation": Decimal("0.15"), "retry": Decimal("0.02")}
```

### Dependencies

```python
import threading                      # Thread safety
from dataclasses import dataclass     # CostEntry
from decimal import Decimal           # Precise financial calculations
from typing import Dict, Optional
from src.core.models import CostEstimate  # Result model
```

### Used By

- `LLMInvocationStage` - Track costs during execution
- `PipelineExecutor` - Get total costs for result
- `BudgetController` - Check against budget limits

### Thread Safety Example

```python
# Two threads executing concurrently:
# Thread 1: tracker.add(tokens_in=1000, tokens_out=500, ...)
# Thread 2: tracker.add(tokens_in=800, tokens_out=300, ...)

# Without lock: Race condition
# 1. T1 reads _total_cost: 0.00
# 2. T2 reads _total_cost: 0.00
# 3. T1 writes _total_cost: 0.10
# 4. T2 writes _total_cost: 0.08  # LOST UPDATE! T1's cost is lost
# Final: 0.08 (WRONG, should be 0.18)

# With lock: Correct
# 1. T1 acquires lock
# 2. T1 reads _total_cost: 0.00, calculates 0.10, writes 0.10
# 3. T1 releases lock
# 4. T2 acquires lock
# 5. T2 reads _total_cost: 0.10, calculates 0.08, writes 0.18
# 6. T2 releases lock
# Final: 0.18 (CORRECT)
```

### Decimal Precision Example

```python
# Why Decimal is necessary:
from decimal import Decimal

# Float (WRONG):
float_cost = 0.1 + 0.2  # 0.30000000000000004 (WRONG!)

# Decimal (CORRECT):
decimal_cost = Decimal("0.1") + Decimal("0.2")  # 0.3 (CORRECT!)

# Real-world impact:
# If processing 1,000,000 rows with $0.0001 per row:
# Float: $100.00000000014 (accumulated error)
# Decimal: $100.00 (exact)
```

### Testing

```python
def test_cost_tracker():
    tracker = CostTracker(
        input_cost_per_1k=Decimal("0.00005"),
        output_cost_per_1k=Decimal("0.00008"),
    )

    # Add cost
    cost1 = tracker.add(tokens_in=1000, tokens_out=500, model="test", timestamp=0.0)
    assert cost1 == Decimal("0.00009")

    # Check accumulation
    assert tracker.total_cost == Decimal("0.00009")
    assert tracker.total_tokens == 1500

    # Add more
    cost2 = tracker.add(tokens_in=2000, tokens_out=1000, model="test", timestamp=1.0)
    assert tracker.total_cost == Decimal("0.00027")  # 0.00009 + 0.00018
```

### Known Limitations

- Stores all entries in memory (could grow large for long runs)
- No persistence (lost on crash)
- No cost cap enforcement (that's BudgetController's job)

### Future Improvements

- [ ] Add entry pruning for long-running pipelines
- [ ] Optional persistence to disk
- [ ] Add cost forecasting based on trends
- [ ] Support tiered pricing (cost changes with volume)

---

## 3.5 `utils/budget_controller.py`

### Purpose
Enforce budget limits and provide warnings during execution.

### Classes

#### `BudgetExceededError` (Exception)

**Inheritance**: `Exception`

**Purpose**: Raised when budget limit is exceeded

**Usage**: Allows caller to catch and handle budget overruns

####

 `BudgetController` (Class)

**Responsibility**: Monitor costs and enforce budget limits

**Single Responsibility**: ONLY handles budget management (cost tracking is CostTracker's job)

**Separation of Concerns**:
- `CostTracker`: Accumulates costs
- `BudgetController`: Enforces limits

**Attributes**:
```python
max_budget: Optional[Decimal]  # Maximum allowed budget (None = no limit)
warn_at_75: bool              # Warn at 75% of budget
warn_at_90: bool              # Warn at 90% of budget
fail_on_exceed: bool          # Raise error if budget exceeded
_warned_75: bool              # Track if 75% warning already shown
_warned_90: bool              # Track if 90% warning already shown
```

**Design Decision**: Separate warning flags from budget
- **Why**: Show each warning only once
- **Alternative**: Check every time (would spam logs)

**Methods**:

##### `__init__(max_budget, warn_at_75, warn_at_90, fail_on_exceed)`

**Purpose**: Initialize budget controller with limits

**Parameters**:
- `max_budget: Optional[Decimal] = None` - Maximum budget in USD
- `warn_at_75: bool = True` - Warn at 75% usage
- `warn_at_90: bool = True` - Warn at 90% usage
- `fail_on_exceed: bool = True` - Raise error when exceeded

**Default Behavior**: Warn twice, then fail

**Flexible Configuration**:
```python
# Strict (default): Fail on exceed
controller = BudgetController(max_budget=Decimal("10.0"))

# Warn only: Don't fail
controller = BudgetController(max_budget=Decimal("10.0"), fail_on_exceed=False)

# No warnings: Just hard limit
controller = BudgetController(
    max_budget=Decimal("10.0"),
    warn_at_75=False,
    warn_at_90=False,
)
```

##### `check_budget(current_cost: Decimal) -> None`

**Purpose**: Check if cost is within budget, warn or fail as configured

**Algorithm**:
1. If `max_budget` is None, return (no limit)
2. Calculate usage ratio: `current_cost / max_budget`
3. If >= 0.75 and not warned: Log warning, set `_warned_75 = True`
4. If >= 0.90 and not warned: Log warning, set `_warned_90 = True`
5. If > `max_budget`:
   - Log error
   - If `fail_on_exceed`: raise `BudgetExceededError`

**Parameters**:
- `current_cost: Decimal` - Current accumulated cost

**Raises**: `BudgetExceededError` if cost exceeds budget and `fail_on_exceed=True`

**Example**:
```python
controller = BudgetController(max_budget=Decimal("10.0"))

# Check after each operation
controller.check_budget(Decimal("5.0"))   # OK
controller.check_budget(Decimal("7.5"))   # Logs: "75% used"
controller.check_budget(Decimal("9.0"))   # Logs: "90% used"
controller.check_budget(Decimal("10.5"))  # Raises BudgetExceededError
```

**Design Decision**: Warnings at 75% and 90%
- **Why**: Give user time to react before hitting limit
- **75%**: Early warning
- **90%**: Final warning before failure
- **Alternative**: More granular warnings (every 10%) would spam logs

##### `get_remaining(current_cost: Decimal) -> Optional[Decimal]`

**Purpose**: Calculate remaining budget

**Returns**:
- `Decimal` - Remaining budget
- `None` if no budget limit set

**Example**:
```python
controller = BudgetController(max_budget=Decimal("10.0"))
remaining = controller.get_remaining(Decimal("7.5"))
# remaining = Decimal("2.5")
```

##### `get_usage_percentage(current_cost: Decimal) -> Optional[float]`

**Purpose**: Calculate budget usage as percentage

**Returns**:
- `float` - Usage percentage (0-100+)
- `None` if no budget limit set

**Example**:
```python
controller = BudgetController(max_budget=Decimal("10.0"))
pct = controller.get_usage_percentage(Decimal("7.5"))
# pct = 75.0
```

##### `can_afford(estimated_cost: Decimal, current_cost: Decimal) -> bool`

**Purpose**: Check if estimated additional cost fits within budget

**Algorithm**:
```python
if self.max_budget is None:
    return True
return (current_cost + estimated_cost) <= self.max_budget
```

**Use Case**: Pre-flight check before expensive operation

**Example**:
```python
controller = BudgetController(max_budget=Decimal("10.0"))
current = Decimal("9.0")
estimated_next = Decimal("0.5")

if controller.can_afford(estimated_next, current):
    proceed_with_operation()
else:
    print("Would exceed budget, skipping")
```

### Dependencies

```python
from decimal import Decimal       # Precise financial calculations
from typing import Optional
import structlog                  # Structured logging
```

### Used By

- `PipelineExecutor` - Check budget during execution
- Any component needing cost enforcement

### Integration with CostTracker

```python
# Typical usage pattern:
cost_tracker = CostTracker(...)
budget_controller = BudgetController(max_budget=Decimal("10.0"))

for row in data:
    # Make LLM call
    response = llm.invoke(prompt)

    # Track cost
    cost = cost_tracker.add(tokens_in=..., tokens_out=...)

    # Check budget
    budget_controller.check_budget(cost_tracker.total_cost)
```

### Testing

```python
def test_budget_warnings():
    controller = BudgetController(max_budget=Decimal("10.0"))

    # Should not warn
    controller.check_budget(Decimal("5.0"))

    # Should warn at 75%
    controller.check_budget(Decimal("7.5"))

    # Should warn at 90%
    controller.check_budget(Decimal("9.0"))

    # Should raise error
    with pytest.raises(BudgetExceededError):
        controller.check_budget(Decimal("10.1"))
```

### Error Messages

```python
# 75% warning:
"Budget warning: 75% used ($7.50 / $10.00)"

# 90% warning:
"Budget warning: 90% used ($9.00 / $10.00)"

# Budget exceeded:
"Budget exceeded: $10.50 > $10.00"
```

### Known Limitations

- No budget rollover (each execution is independent)
- No budget sharing across pipelines
- Fixed warning thresholds (75%, 90%)

### Future Improvements

- [ ] Configurable warning thresholds
- [ ] Support budget pools (shared across pipelines)
- [ ] Budget rollover support
- [ ] Budget forecasting (estimate when limit will be hit)

---

*This is Part 1 of the Technical Reference. Continue reading for Layer 0 (remaining utils), Core Models, and all other layers...*

---

# Part 5: Layer 1 - Infrastructure Adapters (LLM Providers)

## 5.1 LLM Provider Overview

Ondine supports multiple LLM providers through the **Adapter pattern**, allowing easy switching between providers without changing core logic.

### Supported Providers

| Provider | Category | Platform | Cost | Use Case |
|----------|----------|----------|------|----------|
| **OpenAI** | Cloud API | All | $$ | Production, high quality |
| **Azure OpenAI** | Cloud API | All | $$ | Enterprise, compliance, **Managed Identity support** |
| **Anthropic** | Cloud API | All | $$$ | Claude models, long context |
| **Groq** | Cloud API | All | Free tier | Fast inference, development |
| **MLX** | Local | macOS (Apple Silicon) | Free | Privacy, offline, no costs |

### Provider Selection Guide

**Choose OpenAI if**: Production quality, mature ecosystem, GPT-4
**Choose Azure if**: Enterprise compliance, private deployments
**Choose Anthropic if**: Claude models, 100K+ context
**Choose Groq if**: Fast inference, free tier, development
**Choose MLX if**: Apple Silicon Mac, privacy, free, offline capable

---

## 5.2 OpenAI-Compatible Provider (Custom APIs)

### Purpose
Enable integration with any LLM API that implements the OpenAI chat completions format.

### Class: `OpenAICompatibleClient`

**Inheritance**: `LLMClient` (Adapter pattern)

**Platform**: All (platform-agnostic)

**Responsibility**: Connect to custom OpenAI-compatible API endpoints

**Supports**:
- **Ollama** (local LLM server)
- **vLLM** (self-hosted inference)
- **Together.AI** (cloud API)
- **Anyscale** (cloud API)
- **LocalAI** (self-hosted)
- **Any custom OpenAI-compatible API**

### Configuration

**Required Fields**:
- `provider: openai_compatible`
- `base_url`: Custom API endpoint URL
- `model`: Model identifier

**Optional Fields**:
- `provider_name`: Custom name for logging/metrics
- `api_key`: Authentication (optional for local)
- `input_cost_per_1k_tokens`: Custom pricing
- `output_cost_per_1k_tokens`: Custom pricing

### Design Decision: Reuse OpenAI Client

Instead of reimplementing HTTP, leverage llama-index's OpenAI client:

```python
self.client = OpenAI(
    model=spec.model,
    api_key=api_key or "dummy",
    api_base=spec.base_url,  # Custom URL
)
```

**Rationale**:
- DRY: Reuse well-tested code
- Reliability: Edge cases handled
- Compatibility: Ensures OpenAI format
- Maintainability: Benefit from upstream updates

### Example: Together.AI
```yaml
llm:
  provider: openai_compatible
  provider_name: "Together.AI"
  model: meta-llama/Llama-3.1-70B
  base_url: https://api.together.xyz/v1
  api_key: \${TOGETHER_API_KEY}
  input_cost_per_1k_tokens: 0.0006
```

### Validation

```python
@model_validator(mode="after")
def validate_provider_requirements(self):
    if self.provider == OPENAI_COMPATIBLE and not self.base_url:
        raise ValueError("base_url required")
    return self
```

---

## 5.3 Azure OpenAI Provider (Enterprise)

### Purpose
Enable enterprise-grade Azure OpenAI integration with support for both API key and Managed Identity authentication.

### Class: `AzureOpenAIClient`

**Inheritance**: `LLMClient` (Adapter pattern)

**Platform**: All (cloud-based)

**Responsibility**: Azure OpenAI Service integration with enterprise authentication

**Key Features**:
- Managed Identity support (keyless authentication)
- API key authentication (backward compatible)
- Pre-fetched token support (advanced scenarios)
- Multi-region deployment support
- Azure RBAC integration

### Authentication Methods

Azure OpenAI supports three authentication methods (in priority order):

#### 1. Managed Identity (Recommended for Production)

**Purpose**: Keyless authentication using Azure AD

**Requirements**:
- `pip install ondine[azure]` (installs `azure-identity>=1.15.0`)
- Managed Identity assigned to Azure resource
- RBAC role: "Cognitive Services OpenAI User"

**Configuration**:
```python
.with_llm(
    provider="azure_openai",
    model="gpt-4",
    azure_endpoint="https://your-resource.openai.azure.com/",
    azure_deployment="gpt-4-deployment",
    use_managed_identity=True  # ← Keyless!
)
```

**YAML Configuration**:
```yaml
llm:
  provider: "azure_openai"
  model: "gpt-4"
  azure_endpoint: "https://your-resource.openai.azure.com/"
  azure_deployment: "gpt-4-deployment"
  use_managed_identity: true  # No API key needed!
```

**How It Works**:
```python
from azure.identity import DefaultAzureCredential

# Ondine automatically:
credential = DefaultAzureCredential()
token = credential.get_token("https://cognitiveservices.azure.com/.default")

# Passes token to LlamaIndex
self.client = AzureOpenAI(
    azure_ad_token=token.token,  # ← Token, not API key
    azure_endpoint=spec.azure_endpoint,
    deployment_name=spec.azure_deployment,
)
```

**DefaultAzureCredential Resolution Order**:
1. Environment variables (AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_CLIENT_SECRET)
2. Managed Identity (if running on Azure VM/Container/Function)
3. Azure CLI credentials (if `az login` executed)
4. Visual Studio Code credentials
5. Azure PowerShell credentials

**Benefits**:
- ✅ No API keys in code or environment
- ✅ Automatic credential rotation (Azure AD handles it)
- ✅ Fine-grained RBAC via Azure roles
- ✅ Audit trail through Azure AD logs
- ✅ Works across dev/staging/production

**Local Development**:
```bash
# Login with Azure CLI
az login

# Run your script (uses your Azure CLI credentials)
python your_script.py
```

**Production Deployment**:
```bash
# Assign Managed Identity to Azure resource
az vm identity assign --name my-vm --resource-group my-rg

# Grant RBAC role
az role assignment create \
  --assignee <managed-identity-id> \
  --role "Cognitive Services OpenAI User" \
  --scope <openai-resource-id>

# Run your script (uses Managed Identity automatically)
python your_script.py
```

#### 2. Pre-fetched Azure AD Token (Advanced)

**Purpose**: Manual token management for custom scenarios

**Configuration**:
```python
from azure.identity import DefaultAzureCredential

# Fetch token manually
credential = DefaultAzureCredential()
token = credential.get_token("https://cognitiveservices.azure.com/.default")

.with_llm(
    provider="azure_openai",
    model="gpt-4",
    azure_endpoint="https://your-resource.openai.azure.com/",
    azure_deployment="gpt-4-deployment",
    azure_ad_token=token.token  # ← Pre-fetched token
)
```

**Use Cases**:
- Custom token lifecycle management
- Token caching strategies
- Integration with existing auth systems

#### 3. API Key (Backward Compatible)

**Purpose**: Traditional authentication method

**Configuration**:
```python
.with_llm(
    provider="azure_openai",
    model="gpt-4",
    azure_endpoint="https://your-resource.openai.azure.com/",
    azure_deployment="gpt-4-deployment",
    api_key=os.getenv("AZURE_OPENAI_API_KEY")  # ← API key
)
```

**YAML Configuration**:
```yaml
llm:
  provider: "azure_openai"
  model: "gpt-4"
  azure_endpoint: "https://your-resource.openai.azure.com/"
  azure_deployment: "gpt-4-deployment"
  api_key: ${AZURE_OPENAI_API_KEY}  # From environment
```

**Setup**:
```bash
export AZURE_OPENAI_API_KEY="your-key-from-azure-portal"  # pragma: allowlist secret
```

### Architecture

```python
class AzureOpenAIClient(LLMClient):
    def __init__(self, spec: LLMSpec):
        # Validate required fields
        if not spec.azure_endpoint:
            raise ValueError("azure_endpoint required")
        if not spec.azure_deployment:
            raise ValueError("azure_deployment required")

        # Authentication priority:
        if spec.use_managed_identity:
            # 1. Managed Identity (preferred)
            credential = DefaultAzureCredential()
            token = credential.get_token("https://cognitiveservices.azure.com/.default")
            self.client = AzureOpenAI(azure_ad_token=token.token, ...)

        elif spec.azure_ad_token:
            # 2. Pre-fetched token
            self.client = AzureOpenAI(azure_ad_token=spec.azure_ad_token, ...)

        else:
            # 3. API key (backward compatible)
            api_key = spec.api_key or os.getenv("AZURE_OPENAI_API_KEY")
            self.client = AzureOpenAI(api_key=api_key, ...)
```

### Configuration Fields

**Required**:
- `provider: "azure_openai"`
- `azure_endpoint`: Azure OpenAI resource endpoint
- `azure_deployment`: Deployment name (maps to model)

**Authentication (choose one)**:
- `use_managed_identity: bool` - Use Azure Managed Identity (recommended)
- `azure_ad_token: str` - Pre-fetched Azure AD token
- `api_key: str` - API key from Azure Portal

**Optional**:
- `api_version: str` - API version (default: "2024-02-15-preview")
- `temperature: float` - Sampling temperature
- `max_tokens: int` - Maximum output tokens

### Multi-Region Support

Azure OpenAI is available in multiple regions for data residency and latency optimization:

```python
# Region-specific endpoints
regions = {
    "eastus": "https://eastus-resource.openai.azure.com/",
    "westeurope": "https://westeurope-resource.openai.azure.com/",
    "swedencentral": "https://swedencentral-resource.openai.azure.com/",
    "japaneast": "https://japaneast-resource.openai.azure.com/",
}

# Environment-aware configuration
region = os.getenv("AZURE_REGION", "eastus")
endpoint = regions[region]

.with_llm(
    provider="azure_openai",
    azure_endpoint=endpoint,
    use_managed_identity=True
)
```

### Error Handling

**Missing Dependency**:
```
ImportError: Azure Managed Identity requires azure-identity.
Install with: pip install ondine[azure]
```

**Authentication Failure**:
```
ValueError: Failed to authenticate with Azure Managed Identity: <error>.
Ensure the resource has a Managed Identity assigned with
'Cognitive Services OpenAI User' role.
```

**Missing Configuration**:
```
ValueError: Azure OpenAI requires either:
  1. use_managed_identity=True (for keyless auth), or
  2. api_key parameter, or
  3. AZURE_OPENAI_API_KEY environment variable
```

### Dependencies

```python
from llama_index.llms.azure_openai import AzureOpenAI  # LlamaIndex wrapper
from azure.identity import DefaultAzureCredential       # Optional (for Managed Identity)
import tiktoken                                         # Token estimation
```

### Used By

- `create_llm_client()` factory
- Any pipeline with `provider: azure_openai`
- Examples: `azure_managed_identity.py`, `azure_managed_identity_config.yaml`

### Testing Strategy

**Unit Tests (11 tests)**:
- Managed Identity authentication flow
- Pre-fetched token authentication
- API key authentication (backward compatibility)
- Error handling (missing dependencies, failed authentication)
- Priority ordering (Managed Identity > Token > API Key)
- Missing configuration validation

**Integration Tests**:
- Require actual Azure credentials (optional)
- Test real Azure AD authentication
- Verify RBAC permissions

### Security Best Practices

1. **Production**: Use Managed Identity (no secrets)
2. **Development**: Use `az login` (personal credentials)
3. **CI/CD**: Use Service Principal or Federated Identity
4. **Never**: Hardcode API keys in source code

### Performance Characteristics

**Managed Identity**:
- Token fetch: ~100-200ms (first call)
- Token cached: ~1 hour (Azure AD default)
- Subsequent calls: No auth overhead

**API Key**:
- No auth overhead (key is static)
- Manual rotation required

### Known Limitations

- Managed Identity only works on Azure infrastructure
- Token refresh not automatic (pipeline must be restarted after 1 hour for long runs)
- Cannot use Managed Identity with non-Azure OpenAI providers

### Future Improvements

- [ ] Automatic token refresh for long-running pipelines
- [ ] Support for User-Assigned Managed Identity (currently uses DefaultAzureCredential)
- [ ] Token caching across pipeline instances
- [ ] Integration with Azure Key Vault for API key storage

---

## 5.9 MLX Provider (Apple Silicon) - Still Supported

### Purpose
Enable fast, free, local LLM inference on Apple Silicon using Apple's MLX framework.

### Class: `MLXClient`

**Inheritance**: `LLMClient` (Adapter pattern)

**Platform**: macOS with M1/M2/M3/M4 chips only

**Responsibility**: Local in-process LLM inference using MLX framework

**Key Features**:
- In-process (no server management)
- Model caching (load once, use many times)
- Lazy imports (only when MLX provider used)
- Dependency injection (clean testing)
- Free inference ($0 cost)

### Architecture

```python
class MLXClient(LLMClient):
    def __init__(self, spec: LLMSpec, _mlx_lm_module=None):
        # Lazy import with helpful error
        # Load model once (cached in instance)

    def invoke(self, prompt: str, **kwargs) -> LLMResponse:
        # Use cached model for inference
        # No server calls, all local
```

**Design Decision: Dependency Injection**

The `_mlx_lm_module` parameter enables clean testing:

```python
# Production usage (normal)
client = MLXClient(spec)  # Imports mlx_lm automatically

# Testing usage (mocked)
mock_mlx = MagicMock()
client = MLXClient(spec, _mlx_lm_module=mock_mlx)  # Inject mock
```

**Rationale**:
- Testable: No sys.modules hacks
- Clear: Explicit what's being injected
- Backward compatible: Parameter is optional

### Model Caching Strategy

**Problem**: Loading MLX models is expensive (~0.5-2 seconds)

**Solution**: Load once in `__init__()`, cache in instance

**Impact**:
- First invocation: ~0.7s (load) + ~0.7s (inference) = 1.4s
- Subsequent calls: ~0.7s (inference only)
- For 100 rows: Save ~70 seconds vs reload each time!

### Token Estimation

**Primary**: Use MLX tokenizer (model-specific, accurate)
```python
tokens = len(self.mlx_tokenizer.encode(text))
```

**Fallback**: Word count (if tokenizer fails)
```python
tokens = len(text.split())
```

**Rationale**: Graceful degradation, never fail on token estimation

### Error Handling

**Import Error**:
```
ImportError: MLX not installed. Install with:
  pip install ondine[mlx]
or:
  pip install mlx mlx-lm

Note: MLX only works on Apple Silicon (M1/M2/M3/M4 chips)
```

**Model Load Error**:
```
Exception: Failed to load MLX model 'model-name'.
Ensure the model exists on HuggingFace and you have access.
Error: [original error]
```

**Design**: Actionable error messages with context

### Performance Characteristics

**Tested on Apple Silicon (M2)**:
- Model: `mlx-community/Qwen3-1.7B-4bit`
- Load time: 0.74s (once per pipeline)
- Inference: 0.67s/prompt
- Throughput: 1.49 prompts/sec
- Memory: ~2GB (model in RAM)

**Comparison**:
- **vs Cloud APIs**: 10x faster (no network), free
- **vs vLLM**: Simpler (no server), Mac-compatible
- **vs Ollama**: Similar speed, more control

### Thread Safety

- **Not thread-safe**: MLX models are not thread-safe
- **Recommendation**: Use `concurrency=1` in ProcessingSpec
- **Why**: MLX optimized for Apple Neural Engine (single-threaded)

### Dependencies

```python
import mlx_lm  # Lazy imported
# mlx_lm depends on:
# - mlx>=0.29.0
# - mlx-metal (GPU acceleration)
# - transformers (HuggingFace)
```

### Used By

- `create_llm_client()` factory
- Any pipeline with `provider: mlx`
- Examples: `10_mlx_qwen3_local.py`, `10_mlx_qwen3_local.yaml`

### Testing Strategy

**Dependency Injection Pattern**:
```python
# Create mock module
mock_mlx = MagicMock()
mock_mlx.load.return_value = (mock_model, mock_tokenizer)
mock_mlx.generate.return_value = "response"

# Inject for testing
client = MLXClient(spec, _mlx_lm_module=mock_mlx)
```

**Coverage**:
- 14 unit tests
- Model loading/caching
- Token estimation with fallback
- Error handling
- Factory integration

### Known Limitations

- macOS only (MLX doesn't work on Linux/Windows)
- Single-threaded (not optimized for concurrency)
- Model download required (1-5GB per model)
- Requires HuggingFace token for some models

### Future Improvements

- [ ] Add MLX streaming support
- [ ] Support custom MLX generation parameters
- [ ] Add model warmup option
- [ ] Cache models globally (shared across pipelines)

---

## 5.4 LLM Provider Presets

### Purpose
Simplify LLM provider configuration by providing pre-configured specifications for common providers, eliminating boilerplate and configuration errors.

### Class: `LLMProviderPresets`

**Location**: `ondine/core/specifications.py` (lines 301-453)

**Responsibility**: Provide pre-validated LLMSpec instances for popular providers

**Design Pattern**: Static Registry Pattern

**Key Features**:
- Zero boilerplate for common providers (80% code reduction)
- Pre-validated configurations (correct URLs, pricing)
- No hardcoded API keys (security by design)
- IDE autocomplete support
- Pydantic validation throughout

### Available Presets

#### OpenAI Presets

```python
GPT4O_MINI = LLMSpec(
    provider=LLMProvider.OPENAI,
    model="gpt-4o-mini",
    input_cost_per_1k_tokens=Decimal("0.00015"),
    output_cost_per_1k_tokens=Decimal("0.0006"),
)

GPT4O = LLMSpec(
    provider=LLMProvider.OPENAI,
    model="gpt-4o",
    input_cost_per_1k_tokens=Decimal("0.0025"),
    output_cost_per_1k_tokens=Decimal("0.01"),
)
```

#### Together.AI Presets

```python
TOGETHER_AI_LLAMA_70B = LLMSpec(
    provider=LLMProvider.OPENAI_COMPATIBLE,
    provider_name="Together.AI",
    model="meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
    base_url="https://api.together.xyz/v1",
    input_cost_per_1k_tokens=Decimal("0.0006"),
)

TOGETHER_AI_LLAMA_8B = LLMSpec(
    provider=LLMProvider.OPENAI_COMPATIBLE,
    provider_name="Together.AI",
    model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
    base_url="https://api.together.xyz/v1",
    input_cost_per_1k_tokens=Decimal("0.0001"),
)
```

#### Ollama Local Presets (Free)

```python
OLLAMA_LLAMA_70B = LLMSpec(
    provider=LLMProvider.OPENAI_COMPATIBLE,
    provider_name="Ollama-Local",
    model="llama3.1:70b",
    base_url="http://localhost:11434/v1",
    input_cost_per_1k_tokens=Decimal("0.0"),  # FREE!
    output_cost_per_1k_tokens=Decimal("0.0"),
)
```

#### Groq & Anthropic Presets

```python
GROQ_LLAMA_70B = LLMSpec(
    provider=LLMProvider.GROQ,
    model="llama-3.1-70b-versatile",
    input_cost_per_1k_tokens=Decimal("0.00059"),
)

CLAUDE_SONNET_4 = LLMSpec(
    provider=LLMProvider.ANTHROPIC,
    model="claude-sonnet-4-20250514",
    max_tokens=8192,
    input_cost_per_1k_tokens=Decimal("0.003"),
)
```

### Factory Method: `create_custom_openai_compatible()`

**Purpose**: Simplify custom provider configuration (vLLM, LocalAI, etc.)

```python
@classmethod
def create_custom_openai_compatible(
    cls,
    provider_name: str,
    model: str,
    base_url: str,
    input_cost_per_1k: float = 0.0,
    output_cost_per_1k: float = 0.0,
    **kwargs
) -> LLMSpec:
    """Factory for custom OpenAI-compatible providers."""
    return LLMSpec(
        provider=LLMProvider.OPENAI_COMPATIBLE,
        provider_name=provider_name,
        model=model,
        base_url=base_url,
        input_cost_per_1k_tokens=Decimal(str(input_cost_per_1k)),
        output_cost_per_1k_tokens=Decimal(str(output_cost_per_1k)),
        **kwargs
    )
```

### Usage with `PipelineBuilder.with_llm_spec()`

**New Method**: `with_llm_spec(spec: LLMSpec) -> PipelineBuilder`

**Location**: `ondine/api/pipeline_builder.py` (lines 260-311)

**Purpose**: Accept pre-built LLMSpec objects instead of individual parameters

**Example Usage**:

```python
from ondine.core.specifications import LLMProviderPresets

# Simple preset usage
pipeline = (
    PipelineBuilder.create()
    .from_csv("data.csv", input_columns=["text"], output_columns=["result"])
    .with_prompt("Process: {text}")
    .with_llm_spec(LLMProviderPresets.TOGETHER_AI_LLAMA_70B)
    .build()
)

# Override preset settings
custom = LLMProviderPresets.GPT4O_MINI.model_copy(
    update={"temperature": 0.9, "max_tokens": 500}
)
pipeline.with_llm_spec(custom)

# Custom provider via factory
custom_vllm = LLMProviderPresets.create_custom_openai_compatible(
    provider_name="My vLLM Server",
    model="mistral-7b-instruct",
    base_url="http://my-server:8000/v1",
    temperature=0.7
)
pipeline.with_llm_spec(custom_vllm)
```

### Comparison: Before vs After

**Before (parameter-based)**:
```python
.with_llm(
    provider="openai_compatible",
    provider_name="Together.AI",
    model="meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
    base_url="https://api.together.xyz/v1",
    api_key="${TOGETHER_API_KEY}",
    input_cost_per_1k_tokens=0.0006,
    output_cost_per_1k_tokens=0.0006
)
```

**After (preset-based)**:
```python
.with_llm_spec(LLMProviderPresets.TOGETHER_AI_LLAMA_70B)
```

**Result**: 80% code reduction, zero configuration errors

### Design Decisions

**Why Static Class Instead of Enum?**
- Allows class methods (factory)
- Better IDE autocomplete
- Can include documentation in docstrings
- Easier to extend without breaking existing code

**Why No Hardcoded API Keys?**
- Security: API keys should never be in code
- All presets have `api_key=None` by default
- Users must provide via environment variables or `model_copy(update={"api_key": "..."})`

**Why Pydantic `model_copy()` for Overrides?**
- Immutability: Original presets unchanged
- Type safety: Validation on override
- Pythonic: Standard Pydantic pattern
- Flexible: Override any field

### Security Validation

**Test**: All presets must have `api_key=None`
```python
def test_presets_have_no_api_keys(self):
    """Security requirement: No hardcoded API keys."""
    presets = [
        LLMProviderPresets.GPT4O_MINI,
        LLMProviderPresets.TOGETHER_AI_LLAMA_70B,
        # ... all presets
    ]
    for preset in presets:
        assert preset.api_key is None
```

### Backward Compatibility

**100% backward compatible**: Existing `with_llm()` method unchanged
```python
# Old way still works
pipeline.with_llm(provider="openai", model="gpt-4o-mini")

# New way
pipeline.with_llm_spec(LLMProviderPresets.GPT4O_MINI)

# Both methods can be mixed (last call wins)
```

### Testing Coverage

**26 unit tests** (100% pass):
- Preset configuration validation
- Security checks (no API keys)
- Type safety verification
- `with_llm_spec()` method tests
- Override via `model_copy()` tests
- Factory method tests
- Backward compatibility tests

### Examples

**Complete example**: `examples/14_provider_presets.py`
- Demonstrates all preset usage patterns
- Shows customization via `model_copy()`
- Compares old vs new approach
- Lists all available presets

### Future Improvements

- [ ] Add more provider presets (Cohere, AI21, Mistral)
- [ ] Add model size variants (405B, 8B, etc.)
- [ ] Version-specific presets with deprecation warnings
- [ ] Auto-update pricing from provider APIs
- [ ] YAML preset files for user-defined presets

---

---

## 5.6 Observability Module

### Purpose
Plugin-based observability system that leverages **LlamaIndex's built-in instrumentation** for automatic tracking of all LLM calls.

### Module: `ondine/observability/`

**Location**: `ondine/observability/` (9 files, ~600 lines)

**Responsibility**: Configure LlamaIndex observability handlers via clean Ondine API

**Architecture**: Plugin-based with observer registry

**Key Features**:
- **Powered by LlamaIndex**: Leverages battle-tested LlamaIndex observability handlers
- **Automatic instrumentation**: All LLM calls tracked automatically (zero manual instrumentation)
- **Multiple observers**: OpenTelemetry, Langfuse, Logging (can use simultaneously)
- **PII sanitization**: Comprehensive regex-based redaction (custom Ondine feature)
- **Fault-tolerant**: Observer failures never crash the pipeline
- **Clean API**: One-line observer configuration via `with_observer()`

### Design Philosophy

**"Don't Reinvent the Wheel"**

Ondine delegates LLM call tracking to LlamaIndex's native handlers:
- ✅ LlamaIndex automatically instruments `llm.chat()` calls
- ✅ Ondine configures these handlers via clean API
- ✅ Minimal code (~600 lines vs. 2000+ for custom implementation)
- ✅ Production-ready, battle-tested
- ✅ Auto-updates when LlamaIndex improves

### Classes

#### `PipelineObserver` (Abstract Base Class)

**Location**: `ondine/observability/base.py`

**Responsibility**: Define observer interface for pipeline events

**Pattern**: Observer Pattern + Plugin Architecture

**Methods**:
```python
def on_pipeline_start(event) -> None:  # Optional
def on_stage_start(event) -> None:     # Optional
def on_llm_call(event) -> None:        # REQUIRED (abstract)
def on_stage_end(event) -> None:       # Optional
def on_error(event) -> None:           # Optional
def on_pipeline_end(event) -> None:    # Optional
def flush() -> None:                   # Optional
def close() -> None:                   # Optional
```

**Design Decision**: Only `on_llm_call()` is abstract
- LLM calls are the most critical events
- Other events optional (not all observers need them)

#### `ObserverRegistry` (Class)

**Location**: `ondine/observability/registry.py`

**Responsibility**: Plugin registry for observer implementations

**Pattern**: Registry Pattern with decorator-based registration

**Methods**:
```python
@classmethod
def register(name, observer_class) -> None:
    """Register an observer implementation."""

@classmethod
def get(name) -> Type[PipelineObserver]:
    """Get observer class by name."""

@classmethod
def list_observers() -> list[str]:
    """List all registered observers."""
```

**Decorator**:
```python
@observer("my_observer")
class MyObserver(PipelineObserver):
    def on_llm_call(self, event):
        print(f"LLM called: {event.model}")
```

**Auto-registration**: Observers register on import via `@observer` decorator

#### `OpenTelemetryObserver` (Class)

**Location**: `ondine/observability/observers/opentelemetry_observer.py`

**Responsibility**: Delegate to LlamaIndex's OpenTelemetry handler

**Implementation**:
```python
@observer("opentelemetry")
class OpenTelemetryObserver(PipelineObserver):
    def __init__(self, config):
        # Configure LlamaIndex OpenTelemetry handler
        LlamaIndexHandlerManager.configure_handler("opentelemetry", config)

    def on_llm_call(self, event):
        # LlamaIndex handles this automatically!
        pass
```

**What LlamaIndex Tracks** (automatic):
- ✅ LLM call spans with prompts, completions
- ✅ Token usage and latency
- ✅ Model information
- ✅ Export to Jaeger, Datadog, Grafana, etc.

**Lines of Code**: ~70 (vs. 200+ for custom implementation)

#### `LangfuseObserver` (Class)

**Location**: `ondine/observability/observers/langfuse_observer.py`

**Responsibility**: Delegate to LlamaIndex's Langfuse handler

**Configuration**:
```python
{
    "public_key": "pk-lf-...",
    "secret_key": "sk-lf-...",  # pragma: allowlist secret
    "host": "https://cloud.langfuse.com"  # optional
}
```

**What LlamaIndex Tracks** (automatic):
- ✅ Full prompts and completions
- ✅ Token usage and costs
- ✅ Latency metrics
- ✅ Model information
- ✅ Trace hierarchy
- ✅ User feedback integration (future)

**Lines of Code**: ~85 (vs. 240+ for custom implementation)

#### `LoggingObserver` (Class)

**Location**: `ondine/observability/observers/logging_observer.py`

**Responsibility**: Delegate to LlamaIndex's Simple handler

**What LlamaIndex Logs** (automatic):
- ✅ LLM calls with prompts
- ✅ Token usage
- ✅ Latency
- ✅ Console output (no external dependencies)

**Lines of Code**: ~70 (vs. 170+ for custom implementation)

#### `LlamaIndexHandlerManager` (Class)

**Location**: `ondine/observability/llamaindex_handlers.py`

**Responsibility**: Configure LlamaIndex global handlers

**Methods**:
```python
@classmethod
def configure_handler(handler_type: str, config: dict):
    """Configure a LlamaIndex global handler."""
    from llama_index.core import set_global_handler

    if handler_type == "opentelemetry":
        set_global_handler("opentelemetry", **config)
    elif handler_type == "langfuse":
        set_global_handler("langfuse",
            public_key=config["public_key"],
            secret_key=config["secret_key"])
    elif handler_type == "simple":
        set_global_handler("simple")
```

**Design Decision**: Centralized handler configuration
- Single place to manage LlamaIndex handlers
- Consistent error handling
- Future: Multi-handler support via LlamaIndex dispatcher

### PII Sanitization (Custom Ondine Feature)

**Location**: `ondine/observability/sanitizer.py`

**LlamaIndex doesn't provide this**, so we added it as a custom feature.

**Patterns**:
```python
PII_PATTERNS = {
    "email": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
    "ssn": r"\b\d{3}-\d{2}-\d{4}\b",
    "credit_card": r"\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b",
    "phone_us": r"\b\d{3}[-.]?\d{3}[-.]?\d{4}\b",
    "api_key": r"\b(?:api[_-]?key|secret|token)[:\s=]+...",
    "ip_address": r"\b(?:\d{1,3}\.){3}\d{1,3}\b",
}
```

**Functions**:
```python
def sanitize_text(text, patterns, replacement="[REDACTED]"):
    """Replace PII with redaction marker."""

def sanitize_event(event, config):
    """Sanitize LLMCallEvent fields."""
```

**Usage** (future):
```python
.with_observer("langfuse", config={
    "public_key": "...",
    "sanitize_prompts": True,  # Enable PII redaction
    "custom_patterns": {"account_id": r"ACC-\d{6}"}
})
```

### Integration with Pipeline

**PipelineBuilder API**:
```python
def with_observer(self, name: str, config: dict = None):
    """Add observability observer."""
    # Validates observer is registered
    # Stores config for later instantiation
    self._observers.append((name, config))
    return self
```

**Pipeline Initialization**:
```python
# In Pipeline.execute():
observer_configs = self.specifications.metadata.get("observers", [])
for observer_name, observer_config in observer_configs:
    observer_class = ObserverRegistry.get(observer_name)
    observer = observer_class(config=observer_config)
    # Observer.__init__() configures LlamaIndex handler
    # LlamaIndex automatically instruments all subsequent LLM calls!
```

### How It Works (Complete Flow)

```
1. User calls: .with_observer("langfuse", config={...})
   └─> Stored in PipelineBuilder._observers

2. Pipeline.execute() runs:
   └─> Instantiates LangfuseObserver(config)
       └─> LangfuseObserver.__init__() calls:
           └─> LlamaIndexHandlerManager.configure_handler("langfuse", config)
               └─> set_global_handler("langfuse", public_key=..., secret_key=...)
                   └─> LlamaIndex activates Langfuse instrumentation

3. LLM calls happen:
   └─> llm_client.invoke(prompt)
       └─> self.client.chat([message])  # LlamaIndex LLM
           └─> LlamaIndex automatically sends to Langfuse!
               - Prompt ✅
               - Completion ✅
               - Tokens ✅
               - Cost ✅
               - Latency ✅

4. Pipeline completes:
   └─> Observer.flush() and close() called
       └─> LlamaIndex flushes buffered events
```

### Benefits of Delegating to LlamaIndex

**Code Reduction**:
- Before: ~2000 lines (custom event system)
- After: ~600 lines (thin wrappers)
- **Savings**: 70% less code to maintain!

**Quality**:
- ✅ Battle-tested by LlamaIndex community
- ✅ Production-ready
- ✅ Actively maintained
- ✅ Auto-updates with LlamaIndex

**Features** (free from LlamaIndex):
- ✅ Automatic LLM call tracking
- ✅ OpenTelemetry integration
- ✅ Langfuse integration
- ✅ Multiple handler support
- ✅ Future: RAG instrumentation when we add it

### Dependencies

```python
# Core (required in pyproject.toml)
opentelemetry-api>=1.20.0
opentelemetry-sdk>=1.20.0
langfuse>=2.0.0

# Optional exporters
opentelemetry-exporter-jaeger>=1.20.0  # For Jaeger
opentelemetry-exporter-otlp>=1.20.0    # For OTLP protocol
```

### Usage Examples

**Simple logging**:
```python
pipeline = (
    PipelineBuilder.create()
    .from_csv("data.csv", ...)
    .with_observer("logging", config={})
    .build()
)
```

**Langfuse (LLM observability)**:
```python
pipeline = (
    PipelineBuilder.create()
    .from_csv("data.csv", ...)
    .with_observer("langfuse", config={
        "public_key": "pk-lf-...",
        "secret_key": "sk-lf-..."  # pragma: allowlist secret
    })
    .build()
)
```

**Multiple observers**:
```python
pipeline = (
    PipelineBuilder.create()
    .from_csv("data.csv", ...)
    .with_observer("langfuse", config={...})
    .with_observer("opentelemetry", config={...})
    .with_observer("logging", config={})
    .build()
)
```

### Thread Safety

- **Thread-safe**: Yes (LlamaIndex handlers are thread-safe)
- **Fault-tolerant**: Observer failures don't crash pipeline
- **Isolated**: Each observer runs in try/except block

### Performance

- **Overhead**: <1% (LlamaIndex handles async export)
- **LLM call tracking**: Automatic, no manual instrumentation
- **Export**: Batched, non-blocking

### Testing Coverage

**Unit tests**:
- Observer registry and @observer decorator
- Handler manager configuration
- PII sanitization patterns

**Integration tests**:
- Full pipeline with observers
- Multiple observers simultaneously
- Observer failure isolation
- LlamaIndex auto-instrumentation

### Known Limitations

- **One global handler at a time**: LlamaIndex's `set_global_handler()` is global
  - Workaround: Last observer wins
  - Future: Use LlamaIndex's lower-level dispatcher for true multi-observer
- **No pipeline-level events**: LlamaIndex tracks components, not our batch pipeline
  - Future: Add custom event handlers for pipeline start/end

### Future Enhancements

- [ ] Multi-observer support via LlamaIndex dispatcher API
- [ ] Pipeline-level event tracking (start, end, metrics)
- [ ] PII sanitization integration with handlers
- [ ] Custom observer tutorial in docs
- [ ] RAG retrieval tracking (when RAG is added)

---

---

## 5.7 Multi-Row Batching (NEW - November 2024)

### Purpose
Enable processing of N rows in a single API call to achieve 100× speedup for large datasets.

### Architecture: Strategy Pattern + New Stages

**Design Decision**: Use Strategy Pattern for batch formatting + new pipeline stages for aggregation/disaggregation

**Why This Design**:
- ✅ Follows existing Layer 2 pattern (stages are first-class)
- ✅ Strategy Pattern for extensibility (JSON, CSV, XML formats)
- ✅ Single Responsibility (each stage has one job)
- ✅ Backward compatible (only activates when batch_size > 1)

### Module: `ondine/strategies/`

**Location**: `ondine/strategies/` (4 files, ~150 lines)

**Responsibility**: Batch formatting strategies for multi-row processing

#### Classes

**`BatchFormattingStrategy` (Abstract Base Class)**:
```python
class BatchFormattingStrategy(ABC):
    @abstractmethod
    def format_batch(self, prompts: list[str], metadata: dict) -> str:
        """Format N prompts into 1 batch prompt."""
        pass
    
    @abstractmethod
    def parse_batch_response(self, response: str, expected_count: int) -> list[str]:
        """Parse 1 batch response into N individual results."""
        pass
```

**`JsonBatchStrategy` (Concrete Implementation)**:
- Formats prompts as JSON array: `[{"id": 1, "input": "..."}, ...]`
- Parses responses as JSON array: `[{"id": 1, "result": "..."}, ...]`
- Uses Pydantic models for validation (BatchItem, BatchResult)
- Handles partial failures (PartialParseError with parsed_results + failed_ids)
- Extracts JSON from markdown code blocks automatically

**`BatchItem`, `BatchResult`, `BatchMetadata` (Pydantic Models)**:
- Type-safe batch request/response structure
- Validation with Pydantic
- Sorting by ID, missing ID detection

### Stage: `BatchAggregatorStage`

**Location**: `ondine/stages/batch_aggregator_stage.py`

**Responsibility**: Aggregate N prompts into 1 mega-prompt

**Flow**:
```
Input: PromptBatch with N prompts
  ↓
Group into chunks of batch_size
  ↓
For each chunk:
  - Validate against context window
  - Use strategy.format_batch() to create mega-prompt
  - Create new PromptBatch with 1 mega-prompt
  ↓
Output: PromptBatch with 1 prompt (containing N rows)
```

**Key Features**:
- Context window validation (checks model limits)
- Strategy injection (supports JSON, CSV, etc.)
- Preserves row IDs in metadata for disaggregation
- Zero cost (aggregation is free)

### Stage: `BatchDisaggregatorStage`

**Location**: `ondine/stages/batch_disaggregator_stage.py`

**Responsibility**: Split 1 mega-response into N individual responses

**Flow**:
```
Input: ResponseBatch with 1 mega-response
  ↓
Check if batch response (metadata.is_batch)
  ↓
Use strategy.parse_batch_response()
  ↓
Handle 3 scenarios:
  1. Full success: All N results parsed
  2. Partial success: Some results parsed (PartialParseError)
  3. Complete failure: No results parsed (ValueError)
  ↓
Create ResponseBatch with N individual responses
  ↓
Output: ResponseBatch with N responses
```

**Error Handling**:
- **Full success**: Returns N results
- **Partial failure**: Parses what works, marks failed rows with `[PARSE_ERROR]`
- **Complete failure**: Marks all rows with `[BATCH_PARSE_ERROR]`

### Utility: `model_context_limits.py`

**Location**: `ondine/utils/model_context_limits.py`

**Responsibility**: Model context window registry and validation

**Registry**:
```python
MODEL_CONTEXT_LIMITS = {
    "gpt-4o": 128000,
    "gpt-4o-mini": 128000,
    "claude-sonnet-4": 200000,
    "llama-3.1-70b-versatile": 131072,
    # ... 50+ models
}
```

**Functions**:
- `get_context_limit(model)` - Get context window for model
- `validate_batch_size(model, batch_size, avg_tokens)` - Validate batch fits in context
- `suggest_optimal_batch_size(model, avg_tokens)` - Suggest optimal batch size

### Pipeline Integration

**Conditional Stage Insertion** (`ondine/api/pipeline.py`):
```python
# Stage 2: Format prompts
formatter = PromptFormatterStage(...)
batches = formatter.process(df, context)

# Stage 2.5: Aggregate (only if batch_size > 1)
if specs.prompt.batch_size > 1:
    aggregator = BatchAggregatorStage(batch_size=specs.prompt.batch_size, ...)
    batches = aggregator.process(batches, context)

# Stage 3: Invoke LLM
llm_stage = LLMInvocationStage(...)
response_batches = llm_stage.process(batches, context)

# Stage 3.5: Disaggregate (only if batch_size > 1)
if specs.prompt.batch_size > 1:
    disaggregator = BatchDisaggregatorStage(...)
    response_batches = disaggregator.process(response_batches, context)

# Stage 4: Parse responses
parser_stage = ResponseParserStage(...)
results = parser_stage.process(response_batches, context)
```

**Design Decision**: Conditional insertion (not always present)
- **Why**: Zero overhead when batch_size=1 (default)
- **How**: Check `specs.prompt.batch_size > 1` before inserting stages
- **Benefit**: Backward compatible, opt-in only

### API: PipelineBuilder

**New Methods**:
```python
def with_batch_size(self, batch_size: int) -> "PipelineBuilder":
    """Enable multi-row batching (process N rows per API call)."""
    self._prompt_spec.batch_size = batch_size
    return self

def with_batch_strategy(self, strategy: str) -> "PipelineBuilder":
    """Set batch formatting strategy ('json' or 'csv')."""
    self._prompt_spec.batch_strategy = strategy
    return self

def with_processing_batch_size(self, size: int) -> "PipelineBuilder":
    """Set internal batch size for PromptFormatterStage (internal optimization)."""
    self._processing_spec.batch_size = size
    return self
```

**Naming Clarification**:
- `with_batch_size()` - Multi-row batching (NEW, 100× speedup)
- `with_processing_batch_size()` - Internal batching (OLD, renamed)

### Specifications

**PromptSpec** (added fields):
```python
class PromptSpec(BaseModel):
    # ... existing fields ...
    batch_size: int = Field(default=1, ge=1)  # Multi-row batching
    batch_strategy: str = Field(default="json")  # "json" or "csv"
```

### Performance Characteristics

**Benchmarks** (verified with real OpenAI API):
- **Without batching** (batch_size=1): 10 rows = 10 API calls
- **With batching** (batch_size=5): 10 rows = 2 API calls (5× reduction)
- **With batching** (batch_size=100): 5M rows = 50K API calls (100× reduction!)

**Scaling to 5M Rows**:
| Batch Size | API Calls | Time | Speedup |
|------------|-----------|------|---------|
| 1 (default) | 5,000,000 | ~69 hours | 1× |
| 10 | 500,000 | ~7 hours | 10× |
| 100 | 50,000 | ~42 minutes | 100× |
| 500 | 10,000 | ~8 minutes | 500× |

**Limitations**:
- Batch size limited by model context window (auto-validated)
- Larger batches = higher risk of partial failures
- JSON parsing overhead (~200 tokens per batch)

### Testing

**Unit Tests** (24 tests):
- `test_batch_strategies.py` - 16 tests for strategies
- `test_batch_stages.py` - 8 tests for stages

**Integration Tests** (5 tests):
- `test_batch_processing_openai.py` - Real OpenAI API tests
- Verified 5× reduction with batch_size=5
- Verified backward compatibility (batch_size=1)

**Coverage**: 60% overall, 84% for batch disaggregator, 71% for batch aggregator

### Examples

**Example**: `examples/21_multi_row_batching.py`
- Example 1: Without batching (baseline)
- Example 2: With batching (5× speedup)
- Example 3: Large dataset extrapolation (5.4M rows)

### Known Limitations

- JSON strategy only (CSV planned for future)
- No automatic retry for failed rows (marks with error, user must retry)
- Batch size validation is warning-only (doesn't block)
- Requires LLM to follow JSON format instructions

### Future Improvements

- [ ] Implement CsvBatchStrategy (more compact than JSON)
- [ ] Automatic retry for failed rows within batch
- [ ] Adaptive batch sizing based on prompt length
- [ ] XML/YAML batch strategies
- [ ] Batch size optimization suggestions based on historical data

---

**Document Status**: IN PROGRESS (Layer 0 complete, Layer 1 documented with MLX, Presets, Observability, and Multi-Row Batching)

**Next Sections**:
- 3.6 `utils/logging_utils.py`
- 3.7 `utils/metrics_exporter.py`
- 3.8 `utils/input_preprocessing.py`
- Part 4: Core Models & Specifications
- Part 5: Complete Layer 1 (remaining providers)
- Part 6+: Remaining layers...
