# Plan: Fast Default + Rich/tqdm Verbose Output

## Context

Real ablation results confirm:
- **Sequential + Parallel is fastest** (57.27s) vs single-pass (64.03s), same quality
- Current default `sequential=False` (single-pass) is 12% slower than the optimal
- Verbose output is plain print statements with no rich formatting
- No optimized values shown after each pass

Goals:
1. Flip default to `sequential=True, parallel_fields=True` (fastest, same quality)
2. Add `rich` + tqdm-aware verbose output showing optimized field values after each pass
3. Add `optimized_value` to `FieldOptimizationProgress` so callbacks can show what changed

---

## Changes

### 1. `pyproject.toml` — add `rich` dependency

```toml
dependencies = [
    ...
    "rich>=13.0.0",
]
```

`tqdm` is already pulled in by `dspy` — no need to add it separately.

### 2. `src/dspydantic/types.py` — extend `FieldOptimizationProgress`

Add one optional field so callbacks can display the new description text:

```python
@dataclass
class FieldOptimizationProgress:
    ...
    optimized_value: str | None = None   # new: the actual optimized description/prompt text
```

### 3. `src/dspydantic/optimizer.py` — main changes

#### a) Change default for `sequential`
```python
sequential: bool = True   # was False
```

#### b) Add rich imports at top
```python
from rich.console import Console
from rich.panel import Panel
from rich.table import Table
from rich import box
```

#### c) Replace the verbose header print block (~line 385-420, the "Starting DSPy Pydantic optimization" banner)

Replace all the plain `print(f"{'='*60}")` header with a `rich.Panel`:

```python
if self.verbose:
    console = Console()
    mode = "sequential+parallel" if self.parallel_fields else "sequential" if self.sequential else "single-pass"
    console.print(Panel.fit(
        f"[bold]Model:[/] {model.__name__}  [bold]Fields:[/] {len(field_descs)}  "
        f"[bold]Examples:[/] {len(examples)}  [bold]Mode:[/] {mode}",
        title="[bold cyan]DSPy Pydantic Optimization[/]",
    ))
```

#### d) Replace the default `on_progress` callback (lines 324-334)

Use rich formatting + show optimized value:

```python
if verbose and on_progress is None:
    _console = Console()
    def _default_progress(p: FieldOptimizationProgress):
        if p.phase == "fields":
            status = "[green]✓[/]" if p.improved else "[yellow]–[/]"
            _console.print(f"  [bold]{p.field_path}[/] {p.score_before:.0%} → {p.score_after:.0%} {status}")
            if p.optimized_value:
                _console.print(f"    [dim]→ {p.optimized_value!r}[/]")
        elif p.phase == "skipped":
            _console.print(f"  [dim]{p.field_path}: skipped ({p.score_before:.0%} ≥ threshold)[/]")
        elif p.phase in ("system_prompt", "instruction_prompt"):
            status = "[green]✓[/]" if p.improved else "[yellow]–[/]"
            _console.print(f"  [bold]{p.phase}[/] {p.score_before:.0%} → {p.score_after:.0%} {status}")
            if p.optimized_value:
                _console.print(f"    [dim]→ {p.optimized_value!r:.100}[/]")
    self.on_progress = _default_progress
```

#### e) Pass `optimized_value` in `_emit` calls

In `_optimize_sequential` around line 1212, update `_emit` signature:
```python
def _emit(phase, score_before, score_after, field_path=None, field_index=None, optimized_value=None):
    if self.on_progress is None:
        return
    try:
        self.on_progress(FieldOptimizationProgress(
            phase=phase, score_before=score_before, score_after=score_after,
            improved=score_after > score_before, total_fields=total_fields,
            field_path=field_path, field_index=field_index,
            elapsed_seconds=time.perf_counter() - _t0,
            optimized_value=optimized_value,
        ))
    except Exception:
        pass
```

At the "fields" emit sites (line 948 parallel, line 1328 sequential), pass the new description:
```python
# parallel (line ~948): new_descriptions_for_field is dict returned by _optimize_single_field
_emit("fields", score_before, new_score, field_path=field_path, field_index=idx,
      optimized_value=new_descriptions_for_field.get(field_path))

# sequential (line ~1328): same pattern
_emit("fields", score_before, new_score, field_path=field_path, field_index=idx,
      optimized_value=new_descriptions_for_field.get(field_path))
```

For prompt phases (lines 1350, 1369), pass the new prompt text:
```python
_emit("system_prompt", score_before_sys, current_score,
      optimized_value=current_system_prompt)
_emit("instruction_prompt", score_before_instr, current_score,
      optimized_value=current_instruction_prompt)
```

#### f) Replace "Optimization complete" summary with rich Table (both blocks: line ~1396 and ~1751)

```python
if self.verbose:
    _console = Console()
    table = Table(title="Optimization Complete", box=box.ROUNDED)
    table.add_column("Metric", style="bold")
    table.add_column("Value")
    table.add_row("Baseline score", f"{baseline_avg:.2%}")
    table.add_row("Final score", f"{current_score:.2%}")
    table.add_row("Improvement", f"{improvement:+.2%}" if improvement != 0 else "No change")
    if api_calls > 0:
        table.add_row("API calls", str(api_calls))
    if total_tokens > 0:
        table.add_row("Total tokens", f"{total_tokens:,}")
    _console.print(table)
```

### 4. `src/dspydantic/prompter.py` — flip default

```python
sequential: bool = True   # was False
```

### 5. `examples/ablation_benchmark.py` — update configs to include skip_threshold

Add the `sequential_skip_0.8` config that was missing from the first run.

---

## Critical Files

| File | Change |
|------|--------|
| `pyproject.toml` | Add `rich>=13.0.0` |
| `src/dspydantic/types.py` | Add `optimized_value: str \| None = None` to `FieldOptimizationProgress` |
| `src/dspydantic/optimizer.py` | Flip `sequential=True`, rich output, pass `optimized_value` in `_emit` |
| `src/dspydantic/prompter.py` | Flip `sequential=True` default |
| `examples/ablation_benchmark.py` | Add skip_threshold config |

---

## Verification

1. `uv run pytest tests/` — all pass
2. `OPENAI_API_KEY=... uv run python examples/ablation_benchmark.py` — sequential+parallel is fastest
3. Visually verify rich output shows descriptions after each field pass
