# Plan: DSPy Optimization Ablation Benchmark (Real API Calls)

## Context

The API is already in its target state — `sequential`, `parallel_fields`, `max_val_examples`, `skip_score_threshold`, `on_progress` all exist. No `fast` toggle. The goal now is:

1. Create a real ablation benchmark (`examples/ablation_benchmark.py`) using actual API calls
2. Run it and measure wall-clock time + quality across all strategies
3. Based on results, decide whether to change defaults (e.g. flip `sequential=True` as default)

---

## Current API State

| Param | Default | Meaning |
|-------|---------|---------|
| `sequential: bool` | `False` | `False`=single-pass (1 compile); `True`=per-field (N+2 compiles) |
| `parallel_fields: bool` | `True` | Parallelise fields with ThreadPoolExecutor when `sequential=True` |
| `max_val_examples: int \| None` | `None` | Cap val examples per field eval |
| `skip_score_threshold: float \| None` | `None` | Skip fields already above this score |
| `on_progress` | `None` | Callback per phase |

`_FAST_MODE_KWARGS` (max_bootstrapped_demos=1 etc.) applied when `sequential=False`.

---

## Ablation Script Design

**File:** `examples/ablation_benchmark.py`

### Configurations

| # | Name | Params | Compiles |
|---|------|--------|----------|
| 1 | single_pass | `sequential=False` | 1 |
| 2 | sequential_serial | `sequential=True, parallel_fields=False` | N+2 |
| 3 | sequential_parallel | `sequential=True, parallel_fields=True` | N+2 (parallel) |
| 4 | sequential_max_val_3 | `sequential=True, max_val_examples=3` | N+2 (fewer evals) |
| 5 | sequential_skip_0.8 | `sequential=True, skip_score_threshold=0.8` | ≤N+2 |

### Dataset

- `Product` model with 3 fields: `name`, `price`, `category`
- 8 labelled examples, product-extraction task
- Model: `openai/gpt-4o-mini` via `OPENAI_API_KEY`
- `evaluate_fn="exact"`, `num_threads=4`

### Metrics per run

- Wall-clock time (seconds)
- `result.baseline_score`, `result.optimized_score`, improvement
- `result.estimated_cost_usd` (from OptimizationResult)

### Output

Formatted table + speedup analysis vs single-pass.

---

## Implementation

### Step 1 — Write `examples/ablation_benchmark.py`

```python
import os, time
import dspy
from dspydantic import PydanticOptimizer, Example
from pydantic import BaseModel, Field

class Product(BaseModel):
    name: str = Field(description="Product name")
    price: float = Field(description="Price in USD")
    category: str = Field(description="Product category")

EXAMPLES = [...]  # 8 product examples

CONFIGS = [
    ("single_pass",           dict(sequential=False)),
    ("sequential_serial",     dict(sequential=True, parallel_fields=False)),
    ("sequential_parallel",   dict(sequential=True, parallel_fields=True)),
    ("sequential_max_val_3",  dict(sequential=True, max_val_examples=3)),
    ("sequential_skip_0.8",   dict(sequential=True, skip_score_threshold=0.8)),
]

def run_config(name, examples, **kwargs):
    lm = dspy.LM("openai/gpt-4o-mini", api_key=os.environ["OPENAI_API_KEY"], cache=False)
    dspy.configure(lm=lm)
    t0 = time.perf_counter()
    result = PydanticOptimizer(
        model=Product, examples=examples, evaluate_fn="exact", num_threads=4, **kwargs
    ).optimize()
    return {
        "name": name, "elapsed": time.perf_counter()-t0,
        "baseline": result.baseline_score, "optimized": result.optimized_score,
        "improvement": result.optimized_score - result.baseline_score,
        "cost_usd": result.estimated_cost_usd,
    }
```

### Step 2 — Run and report

```bash
OPENAI_API_KEY=... uv run python examples/ablation_benchmark.py
```

---

## Critical Files

| File | Change |
|------|--------|
| `examples/ablation_benchmark.py` | New — real-API ablation script |

No changes to `optimizer.py` / `prompter.py` until benchmark results justify it.

---

## Verification

1. Script exits cleanly with `OPENAI_API_KEY` set
2. All 5 configs produce results
3. Table shows time, quality, and cost per strategy
