# Spec 001: Implementation Plan

## Wave 1: Core Library Foundation

### pyproject.toml
- Project name: `exokit`
- Python >= 3.11
- Dependencies: `typer[all]`, `rich`, `jinja2`, `pydantic>=2.0`, `pydantic-settings`, `structlog`
- Dev dependencies: `pytest`, `pytest-cov`, `ruff`, `mypy`
- Entry point: `exokit = "exokit.cli:app"`
- Ruff config: line-length 100, select = ["E", "W", "F", "I", "N", "UP", "B", "SIM"]
- mypy config: strict = true, packages = ["exokit"]

### Pydantic Models (`core/models.py`)

> **Architect review applied:** Fixed declaration order (DAG), added `default_from` for V2-ready computed defaults, split raw answers from computed context, replaced `dict[str, Any]` with typed models, typed decisions list.

```python
# --- Standalone models (no dependencies) ---

class Question(BaseModel):
    """A questionnaire question definition. JSON-serializable for V2 web forms."""
    id: str
    type: Literal["text", "select", "confirm"]
    prompt: str
    required: bool = True
    default: str | None = None              # Static default value
    default_from: str | None = None         # Key referencing a compute function (V2-ready)
    options: list[str] | None = None        # For select type
    help_text: str | None = None

class QuestionnaireAnswers(BaseModel):
    """Raw validated answers from the user. No computed fields."""
    company: str
    industry: str
    catalog: str
    warehouse_id: str
    workspace_host: str
    databricks_profile: str | None = None
    lakebase_url: str | None = None
    notification_email: str

class ScaffoldContext(QuestionnaireAnswers):
    """Answers + computed fields, ready for template rendering."""
    bundle_name: str    # derived: f"{company.lower()}-{industry}-demo"
    app_name: str       # derived: f"{company.lower()}-{industry}-app"

class PrereqResult(BaseModel):
    """Result of a single prerequisite check."""
    name: str
    passed: bool
    message: str
    version: str | None = None

class WaveDefinition(BaseModel):
    """A wave in the demo build plan."""
    id: int
    name: str
    status: Literal["pending", "in_progress", "completed", "skipped"] = "pending"
    components: list[str]
    description: str

class ManifestMeta(BaseModel):
    """Metadata about the generated demo."""
    company: str
    industry: str
    generated_at: str
    exokit_version: str

class SchemaDefinition(BaseModel):
    """Placeholder for table/schema definitions. Claude populates during build."""
    tables: dict[str, list[str]] = {}   # table_name -> column_names
    notes: str = ""

class Decision(BaseModel):
    """An architectural decision logged during the build."""
    summary: str
    rationale: str = ""
    decided_at: str = ""

# --- Composite models (depend on above) ---

class DemoManifest(BaseModel):
    """The demo_manifest.json schema."""
    meta: ManifestMeta
    waves: list[WaveDefinition]
    schemas: SchemaDefinition = SchemaDefinition()
    talk_track_hints: list[str] = []
    decisions: list[Decision] = []

class ScaffoldResult(BaseModel):
    """Result of the scaffold operation."""
    project_dir: Path
    files_created: list[str]
    warnings: list[str]
    manifest: DemoManifest
```

### Questionnaire Data (`core/questionnaire.py`)
- Define `QUESTIONS: list[Question]` as data (JSON-serializable for V2)
- Define named compute functions registry: `DEFAULT_COMPUTERS: dict[str, Callable[[dict[str, str]], str]]`
- Define `build_context(answers: QuestionnaireAnswers) -> ScaffoldContext` for derived fields
- No UI code — just question definitions, compute functions, and validation

### Prerequisites (`core/prereqs.py`)
- `check_all() -> list[PrereqResult]`
- Individual checks: `check_databricks_cli()`, `check_uv()`, `check_node()`, `check_apx()`, `check_databricks_auth(profile)`
- Each returns `PrereqResult`
- Uses `subprocess.run` with capture for version detection

### Tests
- `tests/test_models.py` — Pydantic model validation, defaults, serialization
- `tests/test_questionnaire.py` — Question definitions valid, computed defaults work
- `tests/test_prereqs.py` — Prereq checks with mocked subprocess calls

---

## Wave 2: Template Engine + Governance Templates

### Manifest Builder (`core/manifest.py`)
- `build_manifest(context: ScaffoldContext) -> DemoManifest`
- Creates the 5-wave structure from the context
- Sets all waves to "pending"
- Populates meta with company, industry, generation timestamp, version

### Scaffold Engine (`core/scaffold.py`)
- `generate(context: ScaffoldContext, output_dir: Path) -> ScaffoldResult`
- Loads templates from `core/templates/`
- Builds Jinja2 context from ScaffoldContext
- Renders all templates to output directory
- Creates directory structure (lakehouse stubs, scripts, docs)
- Returns list of files created

### Jinja2 Environment Setup
- Template loader pointing to `core/templates/`
- Custom filters: `slugify`, `snake_case`, `title_case`
- Strict undefined (error on missing variables)
- Trim blocks, lstrip blocks enabled

### Templates to Create

**Root templates (`templates/root/`):**
- `CLAUDE.md.j2` — ~400 lines, the brain of generated projects
- `pyproject.toml.j2` — Python project config for generated demo
- `.env.example.j2` — Environment variables
- `.gitignore.j2` — Standard gitignore

**Config templates (`templates/conf/`):**
- `databricks.yml.j2` — Asset Bundle config
- `demo_manifest.json.j2` — Project skeleton manifest
- `catalog_schema.json.j2` — Unity Catalog structure

**Governance templates (`templates/claude/`):**
- `agents/lakehouse-dev.md.j2` — Data gen + DLT + ML agent
- `agents/app-dev.md.j2` — FastAPI + React agent
- `agents/dashboard-dev.md.j2` — AI/BI + Genie agent
- `agents/reviewer.md.j2` — Quality checker agent
- `commands/build-next.md.j2` — Build next wave command
- `commands/deploy.md.j2` — Deploy command
- `commands/talk-track.md.j2` — Talk track generator

**Docs templates (`templates/docs/`):**
- `IMPLEMENTATION_PLAN.md.j2` — Auto-populated from manifest
- `PROGRESS.md.j2` — Wave tracker
- `supplements/ARCHITECTURE.md.j2` — 3-layer arch + lakehouse patterns
- `supplements/CODING_STANDARDS.md.j2` — Quality rules
- `supplements/WAVE_PLAYBOOK.md.j2` — Wave governance

**Lakehouse stubs (`templates/lakehouse/`):**
- `resources/jobs/data_generation_job.yml.j2`
- `resources/jobs/ml_training_job.yml.j2`
- `resources/pipelines/main_pipeline.yml.j2`
- Directory stubs: src/data_generation/structured/, src/data_generation/unstructured/, src/pipelines/bronze/, silver/, gold/, src/ml/, src/dashboards/, src/agents/
- `src/dashboards/validate_queries.py` — Utility (copied, not templated)
- `src/dashboards/inspect_schemas.py` — Utility (copied, not templated)
- `src/dashboards/deploy_dashboard_cli.py` — Utility (copied, not templated)

**Scripts templates (`templates/scripts/`):**
- `deploy.sh.j2`
- `setup-catalog.sh.j2`
- `validate.sh.j2`

### Tests
- `tests/test_scaffold.py` — Engine renders templates, creates directories, returns correct result
- `tests/test_templates/` — Each template renders with sample FSI context without errors; YAML/JSON output parses correctly

---

## Wave 3: APX Integration + Skill Bundling

### APX Bridge (`core/apx_bridge.py`)
- `scaffold_app(answers: QuestionnaireAnswers, app_dir: Path) -> None`
- Calls `apx init --name {app_name} --addons ui,lakebase,sql,claude --profile {profile} {app_dir}` via subprocess
- Handles errors (APX not installed, APX fails)
- After APX scaffold, overlays governance files (CLAUDE.md for the app, 3-layer directory structure)

### Governance Overlay
- After APX creates the base app, we:
  1. Restructure backend into 3-layer (api/, services/, repositories/, models/, schemas/)
  2. Write app-specific CLAUDE.md into the app directory
  3. Configure TanStack Query + Zustand in the frontend (modify package.json, add stores/ and hooks/ dirs)

### Skill Bundler (`core/skills.py`)
- `bundle_skills(target_dir: Path) -> list[str]` — Copies essential skills from `skills/` into `{target_dir}/.claude/skills/`
- `update_skills(project_dir: Path, source_dir: Path) -> list[str]` — Refreshes skills from source
- Skills to bundle (essential, no duplicates):
  - databricks-spark-declarative-pipelines
  - databricks-jobs
  - databricks-aibi-dashboards
  - databricks-asset-bundles
  - databricks-model-serving
  - databricks-agent-bricks
  - databricks-synthetic-data-generation
  - databricks-vector-search
  - demo-governance (custom — quality gates + wave gatekeeper)
  - demo-memory (custom — session resilience)

### MCP Configuration
- Generate `.mcp.json` pointing to ai-dev-kit installation
- Auto-detect ai-dev-kit location or prompt for it

### Tests
- `tests/test_apx_bridge.py` — Mock subprocess, verify correct flags passed, test error handling
- `tests/test_skills.py` — Bundle copies correct skills, update refreshes, no duplicates

---

## Wave 4: CLI + End-to-End Integration

### CLI (`cli.py`)
```python
app = typer.Typer(name="exokit", help="Databricks Demo Template Generator")

@app.command()
def init(output_dir: Path = typer.Argument(default=".")) -> None:
    """Scaffold a new demo project."""
    # 1. Run prereq checks
    # 2. Present questionnaire (Rich prompts)
    # 3. Call core.scaffold.generate(answers, output_dir)
    # 4. Call core.apx_bridge.scaffold_app() for the app
    # 5. Call core.skills.bundle_skills()
    # 6. Display success summary

@app.command()
def validate() -> None:
    """Check prerequisites for demo development."""
    # Run all prereq checks, display results table

@app.command()
def update_skills(project_dir: Path = typer.Argument(default=".")) -> None:
    """Refresh bundled skills from latest ai-dev-kit."""
    # Find skills source, call core.skills.update_skills()
```

### End-to-End Integration
- `core/scaffold.py:generate()` orchestrates the full flow:
  1. Create project directory
  2. Render all templates
  3. Call APX bridge for app scaffold
  4. Bundle skills
  5. Generate MCP config
  6. Write manifest
  7. Return ScaffoldResult

### Integration Tests
- `tests/test_integration.py` — Full end-to-end: mock APX, verify complete project structure
- Verify: CLAUDE.md exists and contains expected sections, databricks.yml is valid YAML, demo_manifest.json is valid JSON matching Pydantic model, .claude/skills/ has expected skills, app directory has 3-layer structure

### Tests
- `tests/test_cli.py` — CLI commands invoke correct core functions (use typer.testing.CliRunner)
- `tests/test_integration.py` — End-to-end scaffold produces valid project
