# Phase 01.5: Evaluation Framework for GSD-lite - Research

**Researched:** 2026-01-26
**Domain:** Manual LLM Agent Evaluation & Python Data Engineering Simulation
**Confidence:** HIGH

## Summary

This research focuses on designing a robust, manual evaluation framework for GSD-lite. The core requirement is to enable human-driven iterative quality control of the GSD-lite templates without building complex automated infrastructure.

The recommended approach involves a "Simulation Sandbox" located in `tests/eval_gsd_lite` containing a minimal but representative Python data pipeline. A "Driver Manual" will provide specific copy-paste prompts to guide the agent through the GSD-lite lifecycle (Moodboard -> Whiteboard -> Execution). Results are tracked in a unified `EVAL_LOG.md`.

**Primary recommendation:** Implement a standard-library-only Python ETL pipeline as the test subject to ensure reproducibility and focus evaluation on agent behavior rather than dependency management.

## Standard Stack

### Simulation Stack (The "Test Subject")
To ensure the simulated environment is stable and focuses on agent logic:

| Library | Type | Purpose | Why |
|---------|------|---------|-----|
| `sqlite3` | Built-in | Data Storage | Zero-config SQL database, standard in Python |
| `csv` | Built-in | Data Source | Simple, human-readable input format |
| `logging` | Built-in | Observability | Standard practice for data pipelines |
| `unittest` | Built-in | Testing | Zero-dependency test runner |
| `argparse` | Built-in | CLI Interface | Standard CLI pattern for pipelines |

*Note: While production pipelines use Pandas/Airflow, the simulation should use the Standard Library to avoid "it works on my machine" issues during manual evaluation.*

### Evaluation Stack (The "Infrastructure")

| Tool | Purpose | Usage |
|------|---------|-------|
| `Markdown` | Protocol & Logging | `EVAL_LOG.md` and `DRIVER_MANUAL.md` |
| `git` | State Management | `git clean -fd && git checkout .` for reset |

## Architecture Patterns

### Recommended Folder Structure
For `./tests/eval_gsd_lite`, mimicking a real project root:

```text
tests/eval_gsd_lite/
├── data/
│   └── input.csv            # Raw data
├── src/
│   ├── __init__.py
│   ├── etl/
│   │   ├── __init__.py
│   │   ├── extract.py       # Reader logic
│   │   ├── transform.py     # Business logic (easy to break/fix)
│   │   └── load.py          # Writer logic (SQLite)
│   └── pipeline.py          # Orchestrator (CLI)
├── tests/
│   └── test_pipeline.py     # Simple assertions
├── README.md                # Project context for the Agent
└── EVAL_INSTRUCTIONS.md     # Hidden from agent (User Manual)
```

### Pattern 1: The "Broken Transformation" Scenario
**What:** The repository starts in a working state, or a slightly broken state depending on the test.
**Best Practice:** Start with a *working* pipeline, then ask the agent to add a feature that requires touching all layers (Extract, Transform, Load). This tests context understanding better than a simple bug fix.

### Pattern 2: The "Manual Driver" Loop
**What:** A structured sequence of prompts that forces the agent through the GSD-lite protocol.
**Structure:**
1.  **Setup:** "I want to add feature X. Start a moodboard." (Tests `planning` mode)
2.  **Plan:** "Here is the context. Propose a plan." (Tests `whiteboard` mode)
3.  **Execute:** "Go ahead." (Tests `execution` mode)
4.  **Verify:** "Check your work." (Tests self-correction)

## Don't Hand-Roll

| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| **Repo Reset** | Custom Python cleanup script | `git clean -fdx && git checkout .` | Git is reliable, atomic, and already there. |
| **Test Runner** | Custom logic checker | `python -m unittest` | Standard, outputs known exit codes. |
| **Logging DB** | Custom JSON/CSV logger | `EVAL_LOG.md` (Manual) | Phase requirement is manual entry; automation is premature optimization. |

## Common Pitfalls

### Pitfall 1: Complexity Creep
**What goes wrong:** The simulated repo becomes too complex (e.g., requires Docker, API keys).
**Why it happens:** Trying to make it "realistic".
**How to avoid:** Strict rule: **Standard Library Only**. If it needs `pip install`, it's too complex for this phase.

### Pitfall 2: Prompt Drift
**What goes wrong:** The evaluator (user) changes the prompts slightly each time, making results incomparable.
**Why it happens:** Natural human variance.
**How to avoid:** `DRIVER_MANUAL.md` must contain *exact code blocks* for prompts. "Copy this: ..."

### Pitfall 3: Agent Context Pollution
**What goes wrong:** The agent reads the `EVAL_LOG.md` or `DRIVER_MANUAL.md` and "cheats" or gets confused.
**How to avoid:** These files should technically live *outside* the simulated root, or be explicitly ignored in the agent's context instructions (e.g., `.gitignore` or explicit "Ignore" instruction). *Correction:* Since the agent works in the project root, these files will be visible. We must name them clearly (e.g., `_EVAL_DO_NOT_READ.md`) or accept they are part of the "meta" environment. Best approach: Keep them in `tests/eval_gsd_lite` but instruct agent to work on `src/`.

## Code Examples

### Simulated ETL Component (`transform.py`)
```python
# A simple transformation function easy to test and modify
def clean_currency(value_str):
    """
    Converts currency string '$1,200.50' to float 1200.50.
    Intentionally simplistic to allow for 'bug fix' scenarios (e.g., handling negative values).
    """
    if not value_str:
        return 0.0
    return float(value_str.replace('$', '').replace(',', ''))
```

### Evaluation Log Format (`EVAL_LOG.md`)
```markdown
| Date | Version | Scenario | Step | Exp. Behavior | Act. Behavior | Pass? | Notes |
|------|---------|----------|------|---------------|---------------|-------|-------|
| 2026-01-26 | v1.0 | Add Feature X | Moodboard | Asks scoping questions | Asked relevant questions | PASS | Good context pickup |
| 2026-01-26 | v1.0 | Add Feature X | Whiteboard | Produces checklist | Missed update to `tests/` | FAIL | Forgot TDD |
```

## Sources

### Primary (HIGH confidence)
- **Project Constraints (CONTEXT.md):** Explicitly requires manual driver, `EVAL_LOG.md`, and module-based python repo.
- **Python Standard Library Docs:** `sqlite3`, `unittest`, `csv` are stable, built-in modules perfect for dependency-free simulation.

### Secondary (MEDIUM confidence)
- **GSD-lite Protocol:** `gsd-lite/template/PROTOCOL.md` defines the states (`moodboard`, `whiteboard`, `execution`) that must be tested.

## Metadata

**Confidence breakdown:**
- Standard Stack: HIGH (Standard library is definitive for this constraint)
- Architecture: HIGH (Derived directly from GSD-lite protocol requirements)
- Pitfalls: MEDIUM (Based on general agent evaluation experience)

**Research date:** 2026-01-26
**Valid until:** Indefinite (Foundational framework)
