{#- Context variables:
   - company: str
   - industry: str
   - demo_description: str
-#}
# Lessons Learned — {{ demo_title }}

Log issues, workarounds, and patterns discovered during the build. This file helps improve both this demo and the exokit generator.

## Format

Each entry follows this structure:

```
## [Date] — [Short title]
**Context:** What you were doing
**Issue:** What went wrong
**Fix:** What solved it
**Prevention:** How to avoid it next time
```

---

## Known Gotchas (from previous builds)

### DLT pipeline requires target schema for UC pipelines
**Context:** Deploying a DLT pipeline with Unity Catalog
**Issue:** Deploy fails with "The target schema field is required for UC pipelines"
**Fix:** Add `target: {schema_name}` to pipeline YAML alongside `catalog:`
**Prevention:** Always include both `catalog` and `target` in UC pipeline configurations

### Faker must be inside UDF body on serverless
**Context:** Data generation with pandas_udf on serverless compute
**Issue:** Faker objects are not serializable — UDF fails silently
**Fix:** Instantiate Faker inside the UDF function body, not at module level
**Prevention:** For serverless, always instantiate libraries inside UDF bodies

### Decimal types break sklearn/xgboost
**Context:** ML training after reading from gold tables
**Issue:** Spark DecimalType becomes Python decimal.Decimal — sklearn can't handle it
**Fix:** Cast with `pd.to_numeric(df[col], errors='coerce')` after .toPandas()
**Prevention:** Always cast numeric columns after converting to pandas

### np.random.choice in loop is O(n squared)
**Context:** Data generation with probability weights
**Issue:** Calling np.random.choice(arr, p=weights) in a for-loop rebuilds CDF each call
**Fix:** Use size= parameter: `np.random.choice(arr, size=N, p=weights, replace=True)`
**Prevention:** Always vectorize random selections — never loop

### Notebook paths in job YAML must not have .py extension
**Context:** Referencing notebooks in job task definitions
**Issue:** bundle validate passes but job fails with "notebook not found"
**Fix:** Use path without extension: `src/data_generation/setup_catalog` (not `.py`)
**Prevention:** Databricks notebook paths never include file extensions

---

*(Additional entries will be added by Claude during the build process)*
