# Fix MiPROV2 Meta-Instructions + Issue #20 Phase Control

## Context

When using MiPROV2 as the optimizer, the "Proposed Instructions" log shows meta-instructions like "Given the fields `field_description`, `field_type`, produce the fields `optimized_field_description`" instead of useful optimization guidance. This happens because the DSPy module uses bare string signatures with no docstring/guidance, so MiPROV2 has no meaningful base instruction to optimize from. Additionally, issue #20 requests phase-control flags (`skip_field_description_optimization`, etc.) so users can run multi-pass optimization workflows.

## Step 1: Replace string signatures with class-based Signatures in `module.py`

**File**: `src/dspydantic/module.py`

Define three `dspy.Signature` subclasses before the `PydanticOptimizerModule` class:

```python
class OptimizeFieldDescription(dspy.Signature):
    """Rewrite the given field description to be clearer, more specific, and more effective for guiding structured data extraction. Output ONLY the improved description text."""
    field_description: str = dspy.InputField(desc="The current field description to improve")
    field_type: str = dspy.InputField(desc="The data type of the field")
    optimized_field_description: str = dspy.OutputField(desc="The improved field description")

class OptimizeSystemPrompt(dspy.Signature):
    """Rewrite the given system prompt to be clearer and more effective for guiding structured data extraction. Output ONLY the improved system prompt text."""
    system_prompt: str = dspy.InputField(desc="The current system prompt to improve")
    optimized_system_prompt: str = dspy.OutputField(desc="The improved system prompt")

class OptimizeInstructionPrompt(dspy.Signature):
    """Rewrite the given instruction prompt to be clearer and more effective for guiding structured data extraction. Preserve any template placeholders exactly as they appear. Output ONLY the improved instruction prompt text."""
    instruction_prompt: str = dspy.InputField(desc="The current instruction prompt to improve")
    optimized_instruction_prompt: str = dspy.OutputField(desc="The improved instruction prompt")
```

In `__init__`, replace:
- `dspy.ChainOfThought("field_description, field_type -> optimized_field_description")` → `dspy.ChainOfThought(OptimizeFieldDescription)`
- `dspy.ChainOfThought("system_prompt -> optimized_system_prompt")` → `dspy.ChainOfThought(OptimizeSystemPrompt)`
- `dspy.ChainOfThought("instruction_prompt -> optimized_instruction_prompt")` → `dspy.ChainOfThought(OptimizeInstructionPrompt)`

**Why this works**: MiPROV2 optimizes the instruction (docstring) of each predictor's signature. With descriptive docstrings, it starts from a meaningful base and generates better, task-specific instruction candidates instead of generic meta-instructions. The field names and `forward()` interface are identical, so no changes needed in `optimizer.py`.

## Step 2: Add phase-control flags to `optimizer.py` for issue #20

**File**: `src/dspydantic/optimizer.py`

Add three new parameters to `PydanticOptimizer.__init__` (after `skip_score_threshold`):
- `skip_field_description_optimization: bool = False`
- `skip_system_prompt_optimization: bool = False`
- `skip_instruction_prompt_optimization: bool = False`

Store as instance attributes. Apply in:

### Sequential mode (`_optimize_sequential`, line 1211):
- If `skip_field_description_optimization`: skip the field optimization loop (lines 1266-1340), keep `current_descriptions` unchanged
- If `skip_system_prompt_optimization`: skip system prompt optimization (lines 1345-1363)
- If `skip_instruction_prompt_optimization`: skip instruction prompt optimization (lines 1365-1383)

### Single-pass mode (`optimize`, line 1450):
- If `skip_field_description_optimization`: pass empty `field_descriptions={}` to `PydanticOptimizerModule`
- If `skip_system_prompt_optimization`: pass `has_system_prompt=False` to `PydanticOptimizerModule`
- If `skip_instruction_prompt_optimization`: pass `has_instruction_prompt=False` to `PydanticOptimizerModule`

## Step 3: Write tests

### `tests/unit/test_module.py` — Signature verification tests

- `test_field_optimizer_uses_class_signature`: Create module, verify field optimizers use `OptimizeFieldDescription` signature (check predictor signature has a docstring)
- `test_system_prompt_optimizer_uses_class_signature`: Same for `OptimizeSystemPrompt`
- `test_instruction_prompt_optimizer_uses_class_signature`: Same for `OptimizeInstructionPrompt`
- `test_forward_produces_descriptions_not_meta_instructions`: Mock ChainOfThought predictors to return realistic field descriptions, run `forward()`, verify output doesn't contain meta-instruction patterns ("Given the fields", "produce the fields")

### `tests/unit/test_optimizer_prompts.py` — Skip flag tests

- `test_skip_field_description_optimization_flag`: Create `PydanticOptimizer` with flag, verify stored
- `test_skip_system_prompt_optimization_flag`: Same
- `test_skip_instruction_prompt_optimization_flag`: Same

## Step 4: Export signatures (optional)

**File**: `src/dspydantic/__init__.py` — Add `OptimizeFieldDescription`, `OptimizeSystemPrompt`, `OptimizeInstructionPrompt` to exports for advanced users.

## Step 5: Integration test with real MiPROV2 optimization

**File**: `tests/integration/test_miprov2_descriptions.py`

Uses GLiNER2-inspired example datasets with real API calls. Three test scenarios:

### Scenario A: Restaurant reservation (Literal fields)
```python
class Reservation(BaseModel):
    restaurant: str = Field(description="Restaurant name")
    date: str = Field(description="Date")
    time: str = Field(description="Time")
    party_size: Literal["1","2","3","4","5","6+"] = Field(description="Number of guests")
    seating: Literal["indoor","outdoor","bar"] = Field(description="Seating preference")

examples = [
    Example(text="Reservation at Le Bernardin for 4 people on March 15th at 7:30 PM. We'd prefer outdoor seating.",
            expected_output={"restaurant": "Le Bernardin", "date": "March 15th", "time": "7:30 PM", "party_size": "4", "seating": "outdoor"}),
    Example(text="Table for 2 at Nobu on Friday at 8 PM, bar seating please.",
            expected_output={"restaurant": "Nobu", "date": "Friday", "time": "8 PM", "party_size": "2", "seating": "bar"}),
    Example(text="Booking for 6+ guests at The French Laundry, December 20th, indoor, 6:00 PM.",
            expected_output={"restaurant": "The French Laundry", "date": "December 20th", "time": "6:00 PM", "party_size": "6+", "seating": "indoor"}),
]
```

### Scenario B: Financial transaction (classification + extraction)
```python
class Transaction(BaseModel):
    broker: str = Field(description="Financial institution")
    amount: str = Field(description="Transaction amount")
    security: str = Field(description="Stock or financial instrument")
    date: str = Field(description="Transaction date")
    status: Literal["pending","completed","failed"] = Field(description="Status")
    type: Literal["equity","bond","option","future"] = Field(description="Type")
```

### Test assertions:
1. `result.optimized_descriptions` values are concrete descriptions (not meta-instructions)
2. No value in `optimized_descriptions` matches patterns: `r"Given the fields"`, `r"produce the fields"`, `r"Using the fields.*optimize"`
3. `result.optimized_score >= result.baseline_score` (no regression)
4. Each optimized description is non-empty and < 500 chars (not a paragraph of instructions)

### Test with multiple optimizers:
Parametrize with `@pytest.mark.parametrize("optimizer", ["miprov2", "miprov2zeroshot", "bootstrapfewshot"])` to validate the fix works across optimizer types.

Mark as `@pytest.mark.integration` with `OPENAI_API_KEY` skip.

## Verification

1. `uv run pytest tests/unit/test_module.py -v` — signature verification tests pass
2. `uv run pytest tests/unit/test_optimizer_prompts.py -v` — skip flag tests pass
3. `uv run pytest tests/unit/ -v` — full unit suite passes
4. `uv run pytest tests/integration/test_miprov2_descriptions.py -v` — real MiPROV2 produces descriptions, not meta-instructions
5. `uv run pytest tests/integration/ -v` — all integration tests pass

## Critical Files
- `src/dspydantic/module.py` — Core signature fix (Step 1)
- `src/dspydantic/optimizer.py` — Phase control flags (Step 2)
- `tests/unit/test_module.py` — Signature tests (Step 3)
- `tests/unit/test_optimizer_prompts.py` — Skip flag tests (Step 3)
- `tests/integration/test_miprov2_descriptions.py` — Integration tests (Step 5)
