# Spec 002: Implementation Plan

## Architecture Principles

1. **Modular template groups** — Templates organized by concern (lakehouse, app, scripts, docs). Each group independently testable.
2. **Single Responsibility** — Each deploy script does one thing. Each directory CLAUDE.md covers one domain.
3. **Open-Closed for templates** — Adding templates requires only: create `.j2` file + add to `TEMPLATE_FILES` test list. Jinja2 loader auto-discovers.
4. **Fail-fast over silent fallback** — APX overlay raises errors instead of silently creating dirs at wrong level.
5. **Configuration hierarchy** — Root `.env` for deploy scripts. App `.env` for APX local dev. No duplication of source-of-truth.
6. **Reuse from real build** — New templates extract universal patterns from the working FSI demo, not invented from scratch.

---

## Wave 1: Fix Broken Templates

### 1.1 `templates/root/databricks.yml.j2`
Add two new variables to the `variables:` section:
```yaml
pipeline_target_schema:
  description: "Default target schema for DLT pipeline (set after Phase 0 planning)"
  default: "default"

schema:
  description: "Default schema for ML training (set after Phase 0 planning)"
  default: "default"
```

### 1.2 `templates/lakehouse/resources/jobs/data_generation_job.yml.j2`
Replace all 4 fake tasks (`generate_dimensions`, `generate_core_entities`, `generate_domain_entities`, `generate_unstructured`) with a single real task:
```yaml
tasks:
  - task_key: setup_catalog
    notebook_task:
      notebook_path: ${workspace.file_path}/src/data_generation/setup_catalog
      source: WORKSPACE
      base_parameters:
        catalog: ${var.catalog}
    environment_key: data_gen_env

  # ── PLACEHOLDER: Add entity tasks after Phase 0 planning ──
  # /build-next Step 0.6 populates these based on the approved plan.
  # Example:
  # - task_key: generate_customers
  #   notebook_task:
  #     notebook_path: ${workspace.file_path}/src/data_generation/structured/generate_customers
  #     source: WORKSPACE
  #     base_parameters:
  #       catalog: ${var.catalog}
  #   environment_key: data_gen_env
  #   depends_on:
  #     - task_key: setup_catalog
```
Keep job name, queue, environments, tags intact.

### 1.3 `templates/lakehouse/resources/jobs/ml_training_job.yml.j2`
Comment out all 3 task definitions. Remove all `${var.schema}` references. Keep the job shell valid:
```yaml
resources:
  jobs:
    {{ bundle_name | replace('-', '_') }}_ml_training:
      name: {{ bundle_name }}_ml_training

      # Tasks are populated by /build-next after Phase 0 planning.
      # Uncomment and customize after ML notebooks are created in Wave 3.
      tasks: []

      queue:
        enabled: true
      # ... environments, email_notifications, tags unchanged
```

### 1.4 `templates/lakehouse/resources/pipelines/main_pipeline.yml.j2`
Add after `catalog: ${var.catalog}`:
```yaml
# Target schema (required for UC pipelines)
target: ${var.pipeline_target_schema}
```

### 1.5 `templates/root/pyproject.toml.j2`
- Fix description: `{{ display_name }} {{ industry | upper }}` (was `{{ company }} {{ industry | upper }}`)
- Add after `[build-system]`:
```toml
[tool.hatch.build.targets.wheel]
packages = ["src"]
```
- Add after `select = [...]`:
```toml
ignore = [
    "E402",   # Module-level import not at top — Databricks notebook cells
    "F821",   # Undefined name — spark/dbutils are Databricks-injected globals
    "T201",   # print() — expected in notebooks
    "N806",   # Uppercase variable — intentional for Pandas UDF constants
    "N812",   # Lowercase imported as non-lowercase — 'functions as F' is Spark convention
    "SIM108", # Ternary operator — if/else clearer in notebook logic
]
```

### 1.6 `templates/root/requirements.txt.j2` (NEW)
```
{#- Context variables: none needed — universal deps -#}
# Python dependencies for Databricks notebooks
# Referenced by job environment specs in resources/jobs/*.yml
faker>=28.0.0
pyarrow>=14.0.0
numpy>=1.26.0
pandas>=2.1.0
python-dateutil>=2.8.0
```

### 1.7 `core/scaffold.py`
**Remove:**
- `_STATIC_FILES` list (line ~111-115)
- `_copy_static_files_from()` function (line ~200-221)
- `static_files_dir` parameter from `generate()` signature
- Static file copy step from `generate()` body
- `import shutil` (no longer needed)

**Add:**
- `INDUSTRY_DISPLAY_NAMES` dict at module level:
```python
INDUSTRY_DISPLAY_NAMES: dict[str, str] = {
    "fsi": "Financial Services",
    "telecom": "Telecommunications",
    "retail": "Retail",
    "healthcare": "Healthcare",
    "manufacturing": "Manufacturing",
}
```
- Fix display_name in `generate()`:
```python
template_context["display_name"] = (
    company_str
    if company_str
    else INDUSTRY_DISPLAY_NAMES.get(industry_str, industry_str.upper())
)
```
- Add to `_STUB_DIRECTORIES`:
  - `".claude/agent-memory/lakehouse-dev"`
  - `".claude/agent-memory/app-dev"`
  - `".claude/agent-memory/dashboard-dev"`
  - `".claude/agent-memory/reviewer"`
  - `"resources/dashboards"`

### 1.8 Delete static files
Delete entire `src/exokit/core/templates/static/` directory tree:
- `static/src/dashboards/deploy_dashboard_cli.py`
- `static/src/dashboards/inspect_schemas.py`
- `static/src/dashboards/validate_queries.py`

### 1.9 Tests
- `test_template_rendering.py`: Add `requirements.txt.j2` to `TEMPLATE_FILES`. Update data_gen job to expect 1 task. Update ML job to verify valid YAML shell (no task count). Add pipeline `target` check. Add databricks.yml variable checks.
- `test_scaffold.py`: Remove `TestStaticFileCopy`. Add `TestIndustryDisplayNames`, `TestDisplayNameFallback`, `TestStubDirectories`. Update `TestDirectoryCreation`.
- `test_integration.py`: Remove `test_static_dashboard_utilities`. Add `test_agent_memory_directories`, `test_resources_dashboards_directory`.

---

## Wave 2: New Templates

### 2.1 `templates/lakehouse/src/data_generation/setup_catalog.py.j2`
Databricks notebook format. Creates catalog, `default` schema, landing + checkpoints volumes.
```python
# Databricks notebook source
# MAGIC %md
# MAGIC # {{ display_name }} {{ industry | upper }} Demo — Unity Catalog Setup
# MAGIC Creates catalog, schemas, and volumes. Run as first task in data generation job.

# COMMAND ----------
dbutils.widgets.text("catalog", "{{ catalog }}")
catalog = dbutils.widgets.get("catalog")

# COMMAND ----------
spark.sql(f"CREATE CATALOG IF NOT EXISTS {catalog}")
schemas = ["default"]
for schema in schemas:
    spark.sql(f"CREATE SCHEMA IF NOT EXISTS {catalog}.{schema}")
    spark.sql(f"CREATE VOLUME IF NOT EXISTS {catalog}.{schema}.landing")
    spark.sql(f"CREATE VOLUME IF NOT EXISTS {catalog}.{schema}.checkpoints")
```
Maps to `src/data_generation/setup_catalog.py` via `lakehouse/` prefix strip.

### 2.2 Directory CLAUDE.md files (3 templates)

**`templates/lakehouse/src/data_generation/CLAUDE.md.j2`** — Content from FSI feedback Section 4:
- pandas_udf vs driver-side generation patterns
- Faker MUST be inside UDF body
- broadcast() doesn't work on serverless
- Vectorize np.random.choice (never loop)
- Volume path conventions
- Job DAG pattern (setup_catalog first, dimensions parallel, entities depend on parents)

**`templates/lakehouse/src/pipelines/CLAUDE.md.j2`** — DLT patterns:
- Tier naming: `{tier}_{entity}`, entity IDENTICAL across tiers
- LIVE. vs fully qualified names
- Bronze = STREAMING TABLE + read_files, Silver = STREAMING TABLE + STREAM() + constraints, Gold = LIVE TABLE
- UC pipelines require both `catalog:` AND `target:`
- .example files in tier dirs break pipeline glob includes

**`templates/lakehouse/src/ml/CLAUDE.md.j2`** — ML patterns:
- Decimal→float casting (sklearn/xgboost can't handle decimal.Decimal)
- Dynamic feature discovery from table schema
- MLflow autolog + UC model registration
- Feature name contracts (don't hardcode)

### 2.3 Deploy scripts (3 new, 1 replaced)

**`templates/scripts/deploy-all.sh.j2`:**
```bash
#!/usr/bin/env bash
# Full deployment: lakehouse + companion app
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
source "$SCRIPT_DIR/../.env"
TARGET="${1:-dev}"
"$SCRIPT_DIR/deploy-lakehouse.sh" "$TARGET"
if [ "${SKIP_APP:-}" != "true" ]; then
  "$SCRIPT_DIR/deploy-app.sh" "$TARGET"
fi
```

**`templates/scripts/deploy-lakehouse.sh.j2`:**
- Sources root `.env`
- Steps: bundle validate → bundle deploy → bundle run data_gen → bundle run pipeline → bundle run ml
- `--skip-to` flag (pipeline|ml) for iterating without re-running earlier steps
- Uses `{{ bundle_name | replace('-', '_') }}` for job names

**`templates/scripts/deploy-app.sh.j2`:**
- Sources root `.env`
- Steps: `apx build` → `bundle deploy` → `apps deploy` → get SP client ID → grant warehouse → grant UC schemas
- Derived from the working FSI `apps/main-app/deploy.sh`

**Replace `templates/scripts/deploy.sh.j2`:**
```bash
#!/usr/bin/env bash
# Backward compatibility — redirects to deploy-lakehouse.sh
exec "$(dirname "$0")/deploy-lakehouse.sh" "$@"
```

### 2.4 `templates/root/.env.example.j2`
Consolidate all vars used by deploy scripts:
```bash
DATABRICKS_CONFIG_PROFILE={{ databricks_profile or 'DEFAULT' }}
DATABRICKS_HOST={{ workspace_host }}
DATABRICKS_SQL_WAREHOUSE_ID={{ warehouse_id }}
CATALOG={{ catalog }}
APP_NAME={{ app_name }}
NOTIFICATION_EMAIL={{ notification_email }}
# Lakebase (fill in after creating instance)
# LAKEBASE_URL=postgresql://user:password@host:5432/dbname
```

### 2.5 `templates/docs/TALK_TRACK.md.j2` (NEW)
Structure stub with section headers for each wave's demo narrative. Claude fills during Wave 5.

### 2.6 `templates/docs/LESSONS_LEARNED.md.j2` (UPDATE)
Pre-populate with ≥5 universal gotchas from the FSI build:
- DLT target schema required for UC pipelines
- Faker must be inside UDF body, not module-level
- Decimal→float casting required for ML
- np.random.choice must use `size=` parameter (never loop)
- Notebook paths in job YAML must NOT have `.py` extension

### 2.7 `templates/root/CLAUDE.md.j2` (UPDATE)
- Add `src/data_generation/CLAUDE.md`, `src/pipelines/CLAUDE.md`, `src/ml/CLAUDE.md` to Key References table
- Add `resources/dashboards/`, `.claude/agent-memory/` to project structure tree
- Update deploy commands: `deploy-all.sh`, `deploy-lakehouse.sh`, `deploy-app.sh`
- Review all `{{ display_name }} {{ industry | upper }}` patterns for correctness with improved display_name

### 2.8 Tests
- Add all 9 new templates to `TEMPLATE_FILES`
- `TestSetupCatalog`: Python compile + contains catalog + contains `CREATE CATALOG`
- `TestDirectoryCLAUDEMd`: render check + domain keywords
- `TestDeployScripts`: bash shebang + references bundle_name/app_name
- `TestTalkTrack`: renders + contains wave headings

---

## Wave 3: APX Bridge Fixes

### 3.1 Fix governance overlay — fail-fast
In `apx_bridge.py`, after `_run_apx_init()`:
```python
apx_backend = app_dir / "src" / app_slug / "backend"
if not apx_backend.exists():
    msg = f"APX did not create expected backend at {apx_backend}"
    raise APXScaffoldError(msg)
```
Remove the `else: apx_backend = app_dir` fallback entirely.

### 3.2 APX databricks.yml post-processing
New function `_postprocess_apx_config(context: ScaffoldContext, app_dir: Path)`:
- Parse `app_dir / "databricks.yml"` with `yaml.safe_load()`
- Remove `resources.database_instances` if present
- Set `targets.dev.workspace.profile` to `context.databricks_profile`
- Set `targets.dev.workspace.host` to `context.workspace_host`
- Ensure `resources.apps.*.resources` has a `sql_warehouse` entry with `context.warehouse_id`
- Write back with `yaml.dump()` (preserve order)
- Called from `scaffold_app()` after `_run_apx_init()`

### 3.3 Enhanced app CLAUDE.md
Update `_APP_CLAUDE_MD` in `apx_bridge.py` to add:
- **Authentication**: SP vs OBO (ClientDependency for SQL, UserWorkspaceClientDependency for user identity)
- **SQL Warehouse**: declare as resource in databricks.yml, `valueFrom` in app.yml
- **Statement Execution API**: named `:param` syntax, StatementParameterListItem objects
- **Frontend**: use shadcn/ui for ALL components, auto-generated hooks from OpenAPI
- **Deployment**: `scripts/deploy-app.sh` from project root

### 3.4 App README post-processing
New function `_write_app_readme(context: ScaffoldContext, app_dir: Path)`:
- Overwrites APX's default README
- Sections: Overview, .env Setup, Local Development, Deployment, Architecture, Troubleshooting
- References root-level `scripts/deploy-app.sh`

### 3.5a shadcn/ui pre-install
New function `_install_shadcn_components(app_dir: Path)`:
```python
ui_dir = app_dir / "src" / app_slug / "ui"
cmd = ["npx", "shadcn@latest", "add", "button", "card", "table", "badge",
       "input", "skeleton", "dialog", "tabs", "select", "separator", "--yes"]
try:
    subprocess.run(cmd, cwd=str(ui_dir), capture_output=True, text=True, timeout=120)
except Exception:
    logger.warning("Could not pre-install shadcn components — install manually")
```
Graceful failure — log warning, don't crash.

### 3.5b App .env post-processing
New function `_postprocess_app_env(context: ScaffoldContext, app_dir: Path)`:
- Read existing `.env` from `app_dir`
- Append `DATABRICKS_SQL_WAREHOUSE_ID`, `CATALOG`, `DATABRICKS_CONFIG_PROFILE` if not present
- Don't overwrite APX-generated vars

### 3.6 `templates/claude/commands/build-next.md.j2`
Insert between Phase 0 approval and Phase 1:
```markdown
### Step 0.6: Apply Plan to Scaffold Placeholders

After the plan is approved, update scaffold files with plan-specific values:

1. **`databricks.yml`** — set variable defaults:
   - `schema` → the ML/analytics schema name from the plan
   - `pipeline_target_schema` → the DLT target schema from the plan
2. **`resources/jobs/data_generation_job.yml`** — add one task per entity with proper `depends_on` DAG
3. **`resources/pipelines/main_pipeline.yml`** — verify catalog and target are set
4. **`conf/demo_manifest.json`** — populate schemas field with planned tables
5. **`requirements.txt`** — add domain-specific packages if needed
```

### 3.7 `templates/claude/commands/deploy.md.j2`
Reference new deploy scripts. Update troubleshooting.

### 3.8 `templates/scripts/validate.sh.j2`
Add checks: `databricks bundle validate`, `setup_catalog.py` exists, `requirements.txt` exists, directory CLAUDE.md files exist.

### 3.9 `templates/docs/IMPLEMENTATION_PLAN.md.j2`
Add header: "This file is replaced by /build-next during Phase 0 planning. The structure below is a starting point."

### 3.10 Tests
- `test_apx_bridge.py`: overlay strict mode (error when backend missing), config post-processing, app README
- `test_template_rendering.py`: build-next contains "Step 0.6", deploy.md references new scripts

---

## Wave 4: Integration + Validation

### 4.1 `TestBundleValidateReady`
Generate scaffold, parse all YAML files, verify every `${var.X}` has a corresponding declaration in `databricks.yml` variables. Verify `setup_catalog` notebook path exists as a file.

### 4.2 `TestDisplayNameNoDuplication`
Generate with `company=""`, `industry="fsi"`. Read all rendered files, assert none contain "FSI FSI".

### 4.3 `TestDirectoryStructureComplete`
Verify after scaffold: `.claude/agent-memory/` (4 agent dirs), `resources/dashboards/`, `requirements.txt`, `src/data_generation/setup_catalog.py`, `src/data_generation/CLAUDE.md`, `src/pipelines/CLAUDE.md`, `src/ml/CLAUDE.md`, `scripts/deploy-all.sh`, `scripts/deploy-lakehouse.sh`, `scripts/deploy-app.sh`.

### 4.4 Generator docs
- `docs/PROGRESS.md` — Record Spec 002 wave completion
- `whiteboard.md` — Add decisions: hello-world exit criteria, deploy script split, display_name mapping, APX post-processing, static file removal

---

## Files Changed Summary

### Create (9 new templates)
| File | Wave |
|------|------|
| `templates/root/requirements.txt.j2` | 1 |
| `templates/lakehouse/src/data_generation/setup_catalog.py.j2` | 2 |
| `templates/lakehouse/src/data_generation/CLAUDE.md.j2` | 2 |
| `templates/lakehouse/src/pipelines/CLAUDE.md.j2` | 2 |
| `templates/lakehouse/src/ml/CLAUDE.md.j2` | 2 |
| `templates/scripts/deploy-all.sh.j2` | 2 |
| `templates/scripts/deploy-lakehouse.sh.j2` | 2 |
| `templates/scripts/deploy-app.sh.j2` | 2 |
| `templates/docs/TALK_TRACK.md.j2` | 2 |

### Modify (17 files)
| File | Wave |
|------|------|
| `templates/root/databricks.yml.j2` | 1 |
| `templates/lakehouse/resources/jobs/data_generation_job.yml.j2` | 1 |
| `templates/lakehouse/resources/jobs/ml_training_job.yml.j2` | 1 |
| `templates/lakehouse/resources/pipelines/main_pipeline.yml.j2` | 1 |
| `templates/root/pyproject.toml.j2` | 1 |
| `core/scaffold.py` | 1 |
| `templates/root/.env.example.j2` | 2 |
| `templates/root/CLAUDE.md.j2` | 2 |
| `templates/docs/LESSONS_LEARNED.md.j2` | 2 |
| `templates/scripts/deploy.sh.j2` | 2 |
| `core/apx_bridge.py` | 3 |
| `templates/claude/commands/build-next.md.j2` | 3 |
| `templates/claude/commands/deploy.md.j2` | 3 |
| `templates/scripts/validate.sh.j2` | 3 |
| `templates/docs/IMPLEMENTATION_PLAN.md.j2` | 3 |
| `docs/PROGRESS.md` (generator) | 4 |
| `whiteboard.md` (generator) | 4 |

### Delete (3 files)
| File | Wave |
|------|------|
| `templates/static/src/dashboards/deploy_dashboard_cli.py` | 1 |
| `templates/static/src/dashboards/inspect_schemas.py` | 1 |
| `templates/static/src/dashboards/validate_queries.py` | 1 |

### Tests (4 files, updated across waves)
| File | Waves |
|------|-------|
| `tests/test_templates/test_template_rendering.py` | 1, 2, 3 |
| `tests/test_scaffold.py` | 1 |
| `tests/test_apx_bridge.py` | 3 |
| `tests/test_integration.py` | 1, 4 |
