{#- Context variables:
   - company: str (e.g., "ACME Bank")
   - industry: str (e.g., "fsi")
   - catalog: str (e.g., "acme_fsi_demo")
   - bundle_name: str (e.g., "acme-bank-fsi-demo")
   - app_name: str (e.g., "acme-bank-fsi-app")
   - warehouse_id: str
   - workspace_host: str
   - databricks_profile: str | None
   - lakebase_url: str | None
   - notification_email: str
-#}
# Architecture -- {{ demo_title }}

## Lakehouse Architecture

### Medallion (Bronze / Silver / Gold)

```
Raw Data (Volumes)
    |
    v
+---------------------------+
| Bronze Layer              |  STREAMING TABLE + read_files
| Raw ingestion, no logic   |  Adds ingestion_timestamp
+---------------------------+
    |
    v
+---------------------------+
| Silver Layer              |  STREAMING TABLE + STREAM(LIVE.)
| Cleaned + validated       |  Quality constraints (EXPECT)
+---------------------------+
    |
    v
+---------------------------+
| Gold Layer                |  LIVE TABLE (materialized view)
| Business aggregations     |  Joins, aggregations, metrics
+---------------------------+
    |
    v
+---------------------------+
| Consumers                 |
| Dashboards, Genie, App,  |
| ML Feature Engineering    |
+---------------------------+
```

### DLT/SDP Patterns

**Bronze** -- Raw ingestion from Volumes:
```sql
CREATE OR REFRESH STREAMING TABLE {{ catalog }}.{schema}.bronze_{entity}
COMMENT "Raw {entity} data from landing zone"
TBLPROPERTIES ("quality" = "bronze")
AS SELECT
  *,
  current_timestamp() AS ingestion_timestamp
FROM STREAM read_files(
  "/Volumes/{{ catalog }}/{schema}/landing/{entity}",
  format => "parquet",
  inferColumnTypes => true,
  schemaLocation => "/Volumes/{{ catalog }}/{schema}/checkpoints/bronze_{entity}"
);
```

**Silver** -- Cleaned with quality constraints:
```sql
CREATE OR REFRESH STREAMING TABLE {{ catalog }}.{schema}.silver_{entity} (
  CONSTRAINT valid_id EXPECT ({entity}_id IS NOT NULL) ON VIOLATION DROP ROW,
  CONSTRAINT valid_required EXPECT ({required_field} IS NOT NULL) ON VIOLATION DROP ROW
)
COMMENT "Cleaned {entity} data"
TBLPROPERTIES ("quality" = "silver")
AS SELECT
  {columns},
  ingestion_timestamp,
  current_timestamp() AS processed_timestamp
FROM STREAM({{ catalog }}.{schema}.bronze_{entity});
```

**Gold** -- Business aggregations (NOT streaming):
```sql
CREATE OR REFRESH LIVE TABLE {{ catalog }}.analytics.gold_{metric}
COMMENT "Gold: {metric description}"
TBLPROPERTIES (
  "quality" = "gold",
  "pipelines.autoOptimize.zOrderCols" = "{key_columns}"
)
AS SELECT
  {dimensions},
  {aggregations},
  current_timestamp() AS _last_updated
FROM {{ catalog }}.{schema}.silver_{source}
GROUP BY {dimensions};
```

### Data Flow

```
Faker (notebooks)
  -> Parquet files -> /Volumes/{{ catalog }}/{schema}/landing/
    -> Bronze (STREAMING TABLE + read_files)
      -> Silver (STREAMING TABLE + STREAM + constraints)
        -> Gold (LIVE TABLE + aggregations)
          -> Dashboards (Lakeview, SQL Warehouse {{ warehouse_id }})
          -> Genie (natural language queries)
          -> App ({{ app_name }}, reads gold tables)
          -> ML (feature engineering from gold)
```

---

## 3-Layer Backend Architecture (Companion App)

### Layer Diagram

```
HTTP Request
    |
    v
+-------------------------------+
| Router (thin -- HTTP only)    |
| apps/{{ app_name }}/backend/app/api/   |
+-------------------------------+
    | Depends()
    v
+-------------------------------+
| Service (business logic)      |
| apps/{{ app_name }}/backend/app/services/ |
+-------------------------------+
    |
    v
+-------------------------------+
| Repository (data access)      |
| apps/{{ app_name }}/backend/app/repositories/ |
+-------------------------------+
    |
    v
[Lakebase -- Databricks-managed Postgres]
```

### Router Example
```python
@router.get("/{id}", response_model=EntityResponse)
async def get_entity(
    id: str,
    service: EntityService = Depends(get_entity_service),
) -> EntityResponse:
    return await service.get_by_id(id)
```

### Service Example
```python
class EntityService:
    def __init__(self, repo: EntityRepository) -> None:
        self.repo = repo

    async def get_by_id(self, id: str) -> EntityResponse:
        entity = await self.repo.get(id)
        if entity is None:
            raise AppException(status=404, detail=f"Entity {id} not found")
        return EntityResponse.model_validate(entity)
```

### Repository Example
```python
class EntityRepository:
    def __init__(self, db: AsyncSession) -> None:
        self.db = db

    async def get(self, id: str) -> Entity | None:
        result = await self.db.execute(select(Entity).where(Entity.id == id))
        return result.scalar_one_or_none()
```

### Pydantic Model Hierarchy
```python
class EntityBase(BaseModel):
    name: str
    description: str

class EntityCreate(EntityBase):
    pass  # No id, no timestamps

class EntityUpdate(BaseModel):
    name: str | None = None  # All optional
    description: str | None = None

class EntityResponse(EntityBase):
    model_config = ConfigDict(from_attributes=True)
    id: str
    created_at: datetime
    updated_at: datetime
```

---

## Job DAG Architecture

### Data Generation Job
```
generate_dimensions
    |
    v
generate_employees
    |
    v
generate_customers
    |
    +---> generate_sales
    +---> generate_finance
    +---> generate_marketing
    +---> generate_support
    +---> generate_operations
    +---> (domain-specific generators)
```

### ML Training Job
```
feature_engineering
    |
    +---> train_model_1 (parallel)
    +---> train_model_2 (parallel)
```

### DLT Pipeline
```
Bronze (all tables, auto-discovered via glob)
    |
    v
Silver (streaming from bronze, parallel)
    |
    v
Gold (materialized views from silver, parallel)
```
