{#- Context variables:
   - catalog: str (e.g., "acme_fsi_demo")
   - industry: str (e.g., "fsi")
-#}
# Data Generation — Local Context

## Patterns (choose based on scale)

### Option A: pandas_udf (>1M rows, distributed)

```python
@pandas_udf("struct<id:string, name:string, email:string, score:double>")
def generate_batch(batch: pd.Series) -> pd.DataFrame:
    from faker import Faker  # MUST be inside UDF — Faker is not serializable
    fake = Faker()
    fake.seed_instance(int(batch.iloc[0]))  # Deterministic per partition
    rows = []
    for _ in range(len(batch)):
        rows.append({
            "id": fake.uuid4(),
            "name": fake.name(),
            "email": fake.email(),
            "score": round(fake.pyfloat(min_value=300, max_value=850), 2),
        })
    return pd.DataFrame(rows)

df = spark.range(100_000, numPartitions=100)
result = df.select(generate_batch("id").alias("data")).select("data.*")
result.write.mode("overwrite").parquet(VOLUME_PATH)
```

**Critical rules for pandas_udf:**
- **Faker MUST be instantiated INSIDE the UDF body**, not at module level. Faker objects are not serializable. This is the #1 gotcha — it fails silently on serverless.
- **`spark.sparkContext.broadcast()` does NOT work on serverless.** For shared lookup data, use closures or join after generation.
- Use 100-200 partitions for optimal parallelism.

### Option B: Driver-side (<1M rows, simple)

```python
from faker import Faker
import pandas as pd

fake = Faker()
fake.seed_instance(42)
N = 50_000

data = [{
    "id": fake.uuid4(),
    "name": fake.name(),
    "email": fake.email(),
} for _ in range(N)]

pdf = pd.DataFrame(data)
df = spark.createDataFrame(pdf)
df.repartition(50).write.mode("overwrite").parquet(VOLUME_PATH)
```

For datasets under 1M rows, driver-side is simpler — no serialization issues, no UDF complexity.

## Performance — np.random.choice

**NEVER call `np.random.choice(arr, p=weights)` in a loop.** Each call rebuilds the cumulative distribution — O(n) per call, O(n*M) total. One notebook took 17+ minutes for 50K rows.

```python
# BAD — O(n * M), took 17+ minutes for 50K rows with 100K-element probability array
for i in range(N):
    choice = np.random.choice(customer_ids, p=customer_probs)

# GOOD — O(M) single call, completes in seconds
choices = np.random.choice(customer_ids, size=N, p=customer_probs, replace=True)
```

**Always vectorize**: pre-draw ALL random selections at once using the `size=` parameter.

## Volume Paths

```
/Volumes/{{ catalog }}/{schema}/landing/{entity}/
/Volumes/{{ catalog }}/{schema}/checkpoints/{table}/
```

- Always `mode("overwrite")` for idempotent re-runs
- Use parquet format for structured data
- Entity directory name must match what the DLT bronze `read_files` expects

## Job DAG Pattern

```
setup_catalog (creates catalog, schemas, volumes)
    ├── generate_dimensions (parallel, no entity deps)
    ├── generate_core_entities (depend on setup_catalog only)
    │   ├── generate_dependent_entities (depend on parent core entities)
    │   │   └── generate_high_volume (depend on parents, e.g. transactions on accounts)
    └── generate_unstructured (depend on setup_catalog only)
```

- `setup_catalog` runs first — creates catalog, schemas, volumes
- Dimension/reference tables depend only on `setup_catalog` (run in parallel)
- Core entities depend on `setup_catalog`
- Domain entities depend on their parent core entities (foreign key relationships)
- High-volume tables (transactions) depend on parent entities (accounts)
- Unstructured data (PDFs) depends only on `setup_catalog`
