# Progress Tracker

## Spec 001: Scaffold Engine

| Wave | Name | Status | Agent(s) | Tests | Coverage | Notes |
|------|------|--------|----------|-------|----------|-------|
| Pre | Architect Review | completed | architect | — | — | APPROVED WITH CHANGES: 6 issues (3 must-fix applied to plan.md) |
| 1 | Core Library Foundation | completed | python-dev | 62 pass | 90% | All quality gates green. models 100%, questionnaire 100%, prereqs 98% |
| 2 | Template Engine + Governance | completed | python-dev + template-dev | 227 pass | 94% | scaffold.py 100%, manifest.py 100%. All 25 .j2 templates created. CLAUDE.md.j2 is 424 lines. All YAML/JSON templates parse correctly. |
| 3 | APX Integration + Skills | completed | python-dev | 262 pass | 93% | apx_bridge.py 100%, skills.py 87%. APX subprocess mocked in all tests. 3-layer governance overlay (api/, services/, repositories/, models/, schemas/). 10 essential skills bundled. MCP config auto-generates .mcp.json. |
| 4 | CLI + End-to-End | completed | python-dev | 300 pass | 89% | cli.py 66%, scaffold.py 100%. Full CLI with init/validate/update-skills. Lakehouse path mapping fixed. Duplicate manifest deduped. 3 static dashboard utilities created. Integration tests verify complete project structure. |
| Post | Final Architecture Review | completed | architect | — | — | **Grade A — Ship it.** All 7 criteria pass: library-first, no import cycles, clean Pydantic hierarchy, pure rendering, minimal API surface, thin CLI, V2-ready. |

## Spec 002: Scaffold Feedback Fixes

| Wave | Name | Status | Tests |
|------|------|--------|-------|
| 1 | Fix Broken Templates | Completed | 322 |
| 2 | New Templates | Completed | 359 |
| 3 | APX Bridge Fixes | Completed | 371 |
| 4 | Integration + Validation | Completed | 377 |

### Decisions
- **Hello-world exit criteria**: Scaffold output must pass `databricks bundle validate`
- **Deploy script split**: `deploy-all.sh`, `deploy-lakehouse.sh`, `deploy-app.sh` (was single `deploy.sh`)
- **INDUSTRY_DISPLAY_NAMES**: "fsi" → "Financial Services" to prevent "FSI FSI" naming
- **APX post-processing**: Remove `database_instances`, align profile/host/warehouse
- **Static file removal**: Deleted 3 unused dashboard stubs
- **Directory CLAUDE.md**: Added pattern docs for data_generation, pipelines, ml
- **shadcn/ui pre-install**: Pre-install 10 common components during app scaffold

---

## Decisions Log

| Date | Decision | Context |
|------|----------|---------|
| 2026-03-02 | All decisions documented in whiteboard.md | Initial planning session |
| 2026-03-02 | Removed `typer[all]` extra (not available in typer 0.24.1), use plain `typer` | Wave 1 pyproject.toml setup |
| 2026-03-02 | Removed README.md reference from pyproject.toml (no README yet) | Wave 1 build fix |
| 2026-03-02 | Scaffold engine accepts `templates_dir` override for testability | Wave 2 scaffold.py design |
| 2026-03-02 | Template path mapping: `root/` -> project root, `claude/` -> `.claude/`, others pass through | Wave 2 scaffold.py design |
| 2026-03-02 | CLAUDE.md.j2 is 424 lines with all required sections (tech stack, architecture, waves, agents, DLT patterns, quality gates, never-do list, missing context protocol) | Wave 2 templates |
| 2026-03-02 | Added pyyaml + types-pyyaml to dev deps for template rendering tests | Wave 2 test infrastructure |
| 2026-03-02 | APX bridge uses custom exception hierarchy (APXNotInstalledError, APXScaffoldError, APXTimeoutError) | Wave 3 apx_bridge.py design |
| 2026-03-02 | Governance overlay places 3-layer dirs in backend/ if it exists, otherwise at app root | Wave 3 apx_bridge.py design |
| 2026-03-02 | Skills bundler skips existing skills (no overwrite); update_skills overwrites | Wave 3 skills.py design |
| 2026-03-02 | MCP config generator auto-detects ai-dev-kit from common paths, falls back to ~/ai-dev-kit placeholder | Wave 3 skills.py design |
| 2026-03-02 | CLI uses import aliases (apx_scaffold_app, core_update_skills) to avoid name collision with Typer command functions | Wave 4 cli.py design |
| 2026-03-02 | scaffold.generate() does NOT call apx_bridge or skills — CLI calls them separately for individual skip/retry | Wave 4 integration design |
| 2026-03-02 | Fixed lakehouse path mapping: strip "lakehouse/" prefix so templates render to resources/jobs/ not lakehouse/resources/jobs/ | Wave 4 scaffold.py bugfix |
| 2026-03-02 | Fixed duplicate manifest in files_created: programmatic manifest overwrites template version, only listed once | Wave 4 scaffold.py bugfix |
| 2026-03-02 | Added static_files_dir parameter to scaffold.generate() for overriding dashboard utilities source | Wave 4 scaffold.py enhancement |
| 2026-03-02 | 3 static dashboard utility placeholders created (validate_queries.py, inspect_schemas.py, deploy_dashboard_cli.py) | Wave 4 static files |

## Lessons Learned

(Updated as we discover patterns and gotchas during implementation)

- `typer[all]` extra does not exist in typer 0.24.1 (resolved via uv). Use plain `typer` dependency instead.
- hatchling requires README.md to exist when `readme = "README.md"` is set. Use inline readme or remove the field.
- `from __future__ import annotations` is needed in all modules to support `str | None` syntax cleanly with mypy strict.
- Scaffold tests use temporary Jinja2 templates so they run independently of the real templates (parallel agent work).
- `_build_jinja_env` accepts a `templates_dir` arg, making the engine testable without the real template tree.
- Jinja2 templates need `{#- ... -#}` comment syntax (with dashes) for context variable docs to avoid blank lines in output.
- YAML templates with Jinja2 conditionals ({% if %}) must use `lstrip_blocks` and `trim_blocks` to produce valid YAML.
- All 25 .j2 templates render correctly with both full and minimal (None optionals) context variants.
- APX bridge should always mock subprocess.run — never call the real APX CLI in tests.
- `shutil.copytree` fails if the destination already exists; use `shutil.rmtree` before overwriting in `update_skills`.
- `bundle_skills` intentionally skips existing skills (idempotent first-run), while `update_skills` overwrites (refresh workflow).
- MCP config uses `uvx --from` pattern for running the databricks-mcp server from the ai-dev-kit repo.
- Typer requires function calls in argument defaults (`typer.Argument()`, `typer.Option()`), which triggers ruff B008. Suppress with `# noqa: B008`.
- CLI functions that share names with core module functions need import aliases to avoid Python name shadowing.
- `typer.testing.CliRunner` runs CLI commands in-process — ideal for testing without spawning subprocesses.
- Integration tests should mock only external binaries (APX) and user prompts, letting the real template engine run for maximum coverage.
- Lakehouse templates must strip their `lakehouse/` prefix in `_map_template_path` to land at `resources/` in the generated project.
- When both a Jinja2 template and programmatic code write the same file (demo_manifest.json), deduplicate `files_created` to avoid listing it twice.
