# Plan: DSPy Optimization Speed – API Redesign + Ablation Script

## Context

v0.1.3 introduced `fast=False` (default=sequential) / `fast=True` (single-pass). The user wants to:
1. Remove the `fast` toggle entirely
2. Make the **fastest** approach the default
3. Expose granular params for fine-grained control
4. Add an ablation benchmark script comparing strategies

This is a breaking change on top of v0.1.3, so we bump to **v0.1.4**.

---

## API Redesign

Remove `fast: bool` entirely. New params:

| New param | Default | Purpose |
|-----------|---------|---------|
| `sequential: bool` | `False` | Opt into sequential field-by-field mode (one compile per field) |
| `parallel_fields: bool` | `True` | Parallelize field optimization when `sequential=True` |
| `max_val_examples: int \| None` | `None` | Cap val set size per field (None = no cap) |
| `skip_score_threshold: float \| None` | `None` | Skip fields above this baseline score (None = don't skip) |

**Default behavior:** single-pass (current `fast=True`) — one DSPy compile for all fields with reduced demo budgets. `_FAST_MODE_KWARGS` applied by default unless overridden.

**Sequential mode** (`sequential=True`): one compile per field, deepest-first. With `parallel_fields=True` (default), all fields compile simultaneously in threads.

---

## Feature Details

### 1. `parallel_fields: bool = True`

Applies when in sequential mode (user opts in). Uses `concurrent.futures.ThreadPoolExecutor` to compile all fields simultaneously instead of one-after-another.

- Each field gets a snapshot of the original `current_descriptions` (no order dependency)
- Results merged after all futures complete
- `on_progress` callbacks are thread-safe (emit from each thread, order not guaranteed)
- Prompt optimization (system/instruction) remains sequential after fields

**Speed gain:** ~N× for sequential mode (e.g., 10 fields = ~10× faster wall-clock)

### 2. `max_val_examples: int | None = None`

Slice `val_examples[:max_val_examples]` before each `_optimize_single_field` and `_optimize_prompt` call.

**Speed gain:** With 20 val examples and `max_val_examples=5` → 4× fewer scoring LLM calls per field.

### 3. `skip_score_threshold: float | None = None`

Before calling `_optimize_single_field`, compute a quick baseline score for that field. If already above threshold, emit a `phase="skipped"` progress event and continue.

**Speed gain:** Varies — can save entire DSPy compiles for well-optimized fields.

---

## Ablation Benchmark Script

**File:** `examples/ablation_benchmark.py`

Design:
- Synthetic dataset with 3 fields, 10 examples (no real API needed — uses a configurable LM)
- Runs 4 configurations and records wall-clock time + final score:

```
| Config                    | Description                                  |
|---------------------------|----------------------------------------------|
| single_pass               | Default (fast=True equivalent, one compile)  |
| sequential                | Old default (one compile per field)          |
| sequential_parallel       | Sequential with parallel_fields=True         |
| sequential_max_val_5      | Sequential with max_val_examples=5           |
```

- Each config runs `optimizer.optimize()` and captures:
  - `elapsed_seconds`
  - `baseline_score`, `optimized_score`, `improvement`
  - Number of DSPy compile calls (approximated by monkey-patching or counting)
- Outputs a formatted table comparing all four

---

## Implementation Plan

### Step 1 — Replace `fast` with new params — `src/dspydantic/optimizer.py`

1. Remove `fast: bool = False` from `__init__` signature
2. Add `sequential: bool = False`, `parallel_fields: bool = True`, `max_val_examples: int | None = None`, `skip_score_threshold: float | None = None`
3. Replace `self.fast` / `self.sequential = not fast` with `self.sequential = sequential`
4. Change `_FAST_MODE_KWARGS` application: apply always (not gated on `self.fast`), still overridable via `optimizer_kwargs`
5. In `_optimize_sequential`:
   - Add `max_val_examples` slicing before `_optimize_single_field` / `_optimize_prompt` calls
   - Add `skip_score_threshold` quick-score check with `phase="skipped"` emit before calling `_optimize_single_field`
   - Add `parallel_fields` path using `ThreadPoolExecutor` (only when `sequential=True`)
6. Default path in `optimize()` remains single-pass when `sequential=False`

### Step 2 — Update `src/dspydantic/prompter.py`

Forward the 3 new params to `PydanticOptimizer`. Remove `fast` param from `Prompter.optimize()`.

### Step 3 — Update `src/dspydantic/types.py`

Add `"skipped"` as a valid phase value to `FieldOptimizationProgress` docstring/comment.

### Step 4 — Create `examples/ablation_benchmark.py`

Self-contained benchmark that can run with a real LM (uses env var `OPENAI_API_KEY`). Clear output table showing time vs quality tradeoffs.

### Step 5 — Update docs + README

- Replace `fast=True/False` references with new params
- Add migration guide section: `fast=True` → default, `fast=False` → `optimization_mode="sequential"` or similar

### Step 6 — Bump version `0.1.3 → 0.1.4`

---

## Critical Files

| File | Change |
|------|--------|
| `src/dspydantic/optimizer.py` | Remove `fast`, add 3 params, parallel field logic |
| `src/dspydantic/prompter.py` | Forward 3 new params, remove `fast` |
| `src/dspydantic/types.py` | Add `"skipped"` phase note |
| `examples/ablation_benchmark.py` | New ablation script |
| `README.md` | Update API refs |
| `docs/concepts/optimization.md` | Update mode docs |
| `docs/guides/advanced/configure-optimizations.md` | Update quick ref table |

---

## Verification

1. `uv run pytest tests/` — all existing tests pass
2. Run ablation script manually: `uv run python examples/ablation_benchmark.py`
3. Verify `parallel_fields=True` produces same descriptions as sequential (within scoring tolerance)
4. Verify `max_val_examples=3` caps val set correctly
5. Verify `skip_score_threshold=0.99` skips fields scoring above 0.99

---

## Migration Guide (for docs/CHANGELOG)

```python
# Before (v0.1.3)
PydanticOptimizer(..., fast=True)   # → now the default
PydanticOptimizer(..., fast=False)  # → use sequential=True

# After (v0.1.4)
PydanticOptimizer(...)                                        # single-pass, fastest (default)
PydanticOptimizer(..., sequential=True)                       # sequential field-by-field
PydanticOptimizer(..., sequential=True, parallel_fields=True) # sequential + parallel (default when sequential=True)
PydanticOptimizer(..., max_val_examples=5)                    # cap val set (applies to both modes)
PydanticOptimizer(..., skip_score_threshold=0.9)              # skip high-scoring fields (sequential mode)
```
