{#- Context variables:
   - company: str (e.g., "ACME Bank")
   - industry: str (e.g., "fsi")
   - catalog: str (e.g., "acme_fsi_demo")
   - bundle_name: str (e.g., "acme-bank-fsi-demo")
   - app_name: str (e.g., "acme-bank-fsi-app")
   - warehouse_id: str
   - workspace_host: str
   - databricks_profile: str | None
   - lakebase_url: str | None
   - notification_email: str
-#}
# {{ demo_title }} — CLAUDE.md

You are an expert in Databricks, Python, SQL, DLT/SDP pipelines, AutoML, Lakeview dashboards, Genie spaces, Agent Framework, FastAPI, React, TypeScript, Tailwind CSS, shadcn/ui, TanStack Query, Zustand, and Lakebase.

---

## FIRST: Session Startup Protocol

**Every time you start a new session in this project, do these 3 things BEFORE anything else:**

1. **Read `docs/PROGRESS.md`** — tells you which waves are done, which are in progress, and what's next.
2. **Read `conf/demo_manifest.json`** — the canonical definition of this demo (waves, schemas, decisions).
3. **Check the current wave's status** — if a wave is `in_progress`, resume it. If all are `pending`, start from Wave 1.

If the user says "continue" or "pick up where we left off", follow this protocol. You have everything you need in these files.

**After completing any wave**, update `conf/demo_manifest.json` (the single source of truth):
- Set the wave `status` to `"completed"`
- Add any architectural decisions to the `decisions` array
- `docs/PROGRESS.md` is a derived view — regenerate it from the manifest, don't maintain it separately

---

## Core Principle: Everything Is Code, Everything Is Reproducible

**Every resource in this demo MUST be created via code deployed through `databricks bundle deploy`.** Nothing should exist only because an MCP tool or manual CLI command created it. The goal: anyone can clone this repo and run `databricks bundle deploy` to get a fully working demo.

This means:
- **Catalog + schemas + volumes** → SQL notebook as the first task in the data generation job
- **Data generation** → Python notebooks executed as job tasks
- **DLT pipelines** → SQL files in `src/pipelines/` referenced by pipeline YAML
- **ML models** → Python notebooks executed as job tasks
- **Dashboards** → JSON definitions deployed via dashboard utilities or bundle
- **Genie spaces** → Created via SDK in a deployment notebook
- **Agent use cases** → Defined in code, registered via notebooks

### **MCP IS FOR EXPLORATION AND DEBUGGING ONLY**

This project has a `.mcp.json` with the Databricks AI Dev Kit MCP server (70+ tools). MCP is useful for:
- **Exploring**: `execute_sql` to test queries, inspect table schemas
- **Debugging**: Check job run status, pipeline errors, table contents
- **Job Error Investigation**: `python scripts/get_job_errors.py <RUN_ID>` — fetches task errors + failed SQL queries from Query History
- **Validating**: Verify resources exist after `bundle deploy`

**NEVER use MCP to CREATE resources. If it is not in the bundle, it does not exist. No exceptions.**

---

## What This Is

**{{ demo_title }}** -- an end-to-end Databricks demo showcasing the full lakehouse platform for the {{ industry | upper }} industry vertical. This project was scaffolded by `exokit` and is designed to be built incrementally by Claude Code agents, wave by wave.

**Who runs this demo**: Databricks field engineers and solution architects presenting to {{ display_name }}.

### User's Demo Intent

> {{ demo_description }}

**This is what the user asked for.** During Phase 0 (planning), use this description to guide your research, questions, schema design, and implementation plan. The user's intent takes priority over generic templates.

**Source of truth**: `conf/demo_manifest.json` -- always read this first.

---

## Technology Stack

### Lakehouse (Databricks Platform)

| Component | Technology | Purpose |
|-----------|-----------|---------|
| **Data Ingestion** | DLT / Streaming Data Pipelines (SDP) | Medallion architecture (bronze/silver/gold) |
| **Data Generation** | Python notebooks + Faker | Synthetic data for {{ industry | upper }} domain |
| **ML Training** | AutoML + MLflow + Feature Engineering | Predictive models for {{ industry | upper }} use cases |
| **Dashboards** | Lakeview (AI/BI Dashboards) | Executive and operational dashboards |
| **Natural Language** | Genie Spaces | Ask questions about data in plain English |
| **AI Agents** | Databricks Agent Framework | Domain-specific AI assistants |
| **Orchestration** | Databricks Jobs (DAG workflows) | Multi-task job orchestration |
| **Catalog** | Unity Catalog (`{{ catalog }}`) | Governance, lineage, access control |
| **Compute** | SQL Warehouse (`{{ warehouse_id }}`) | Query execution for dashboards and Genie |
| **Deployment** | Databricks Asset Bundles (DABs) | Infrastructure-as-code deployment |

### Companion App (Databricks Apps)

| Layer | Technology | Purpose |
|-------|-----------|---------|
| **Backend** | FastAPI + Python 3.12+ | API server with 3-layer architecture |
| **Database** | Lakebase (Databricks-managed Postgres) | Operational data store |
| **Frontend** | React 19 + TypeScript 5.8 + Vite | Modern SPA |
| **Styling** | Tailwind CSS + shadcn/ui | Utility-first CSS + accessible components |
| **Server State** | TanStack Query v5 | Cache, dedup, background refetch |
| **Client State** | Zustand v5 | UI-only state (modals, sidebar, active tab) |
| **Forms** | React Hook Form + Zod | Validation + type derivation |
| **Quality** | Ruff + mypy + ESLint + Prettier | Linting and type checking |
| **Testing** | pytest + Vitest + RTL | Backend and frontend testing |

---

## Project Structure

```
{{ bundle_name }}/
+-- CLAUDE.md                           # This file -- project bible
+-- pyproject.toml                      # Python deps + tool config
+-- databricks.yml                      # Asset Bundle configuration
+-- conf/
|   +-- demo_manifest.json              # THE source of truth for this demo
|   +-- catalog_schema.json             # Unity Catalog structure
+-- .env.example                        # Environment variables template
+-- .claude/
|   +-- agents/                         # Agent definitions
|   |   +-- lakehouse-dev.md            # Data gen + DLT + ML
|   |   +-- app-dev.md                  # FastAPI + React
|   |   +-- dashboard-dev.md            # AI/BI + Genie
|   |   +-- reviewer.md                 # Quality checker
|   +-- commands/                       # Custom slash commands
|   |   +-- build-next.md               # /build-next -- build the next wave
|   |   +-- deploy.md                   # /deploy -- deploy to Databricks
|   |   +-- talk-track.md               # /talk-track -- generate talk track
|   +-- agent-memory/                   # Persistent agent memory across sessions
|       +-- lakehouse-dev/MEMORY.md
|       +-- app-dev/MEMORY.md
|       +-- dashboard-dev/MEMORY.md
|       +-- reviewer/MEMORY.md
+-- docs/
|   +-- IMPLEMENTATION_PLAN.md          # What to build, auto-populated from manifest
|   +-- PROGRESS.md                     # Wave completion tracker
|   +-- TALK_TRACK.md                   # Demo presentation narrative
|   +-- LESSONS_LEARNED.md              # Build gotchas + workarounds
|   +-- supplements/
|       +-- ARCHITECTURE.md             # 3-layer arch + lakehouse patterns
|       +-- CODING_STANDARDS.md         # Quality rules
|       +-- WAVE_PLAYBOOK.md            # Wave governance rules
+-- src/
|   +-- data_generation/
|   |   +-- CLAUDE.md                   # Data gen patterns + pitfalls
|   |   +-- setup_catalog.py            # Creates catalog, schemas, volumes
|   |   +-- structured/                 # Tabular data generators (Faker + PySpark)
|   |   +-- unstructured/              # Documents, PDFs, emails, logs
|   +-- pipelines/
|   |   +-- CLAUDE.md                   # DLT patterns + naming rules
|   |   +-- bronze/                     # STREAMING TABLE + read_files
|   |   +-- silver/                     # STREAMING TABLE + STREAM(LIVE.) + constraints
|   |   +-- gold/                       # MATERIALIZED VIEW / LIVE TABLE + aggregations
|   +-- ml/
|   |   +-- CLAUDE.md                   # ML patterns + decimal casting
|   |   # Feature engineering + model training
|   +-- dashboards/                     # Lakeview dashboard definitions + utilities
|   +-- agents/                         # Agent Framework definitions
+-- apps/
|   +-- {{ app_name }}/                 # FastAPI + React companion app
|       +-- backend/
|       |   +-- app/
|       |       +-- api/                # Routers (thin -- HTTP only)
|       |       +-- services/           # Business logic
|       |       +-- repositories/       # Data access (SQL only)
|       |       +-- schemas/            # Pydantic request/response
|       |       +-- models.py           # APX default models (extend, don't replace)
|       +-- frontend/
|           +-- src/
|               +-- pages/              # Route-level components
|               +-- components/         # Shared + feature components
|               +-- hooks/              # TanStack Query hooks
|               +-- stores/             # Zustand stores
|               +-- lib/                # Utilities (api-client, cn(), etc.)
+-- resources/
|   +-- jobs/                           # Databricks job definitions (YAML)
|   +-- pipelines/                      # DLT pipeline definitions (YAML)
|   +-- dashboards/                     # Dashboard JSON definitions
+-- scripts/
    +-- deploy-all.sh                   # Full deploy: lakehouse + app
    +-- deploy-lakehouse.sh             # Lakehouse: deploy + data gen + pipeline + ML
    +-- deploy-app.sh                   # App: build + deploy + permissions
    +-- deploy.sh                       # Backward compat → deploy-lakehouse.sh
    +-- validate.sh                     # Pre-deployment validation
    +-- ensure-mcp.sh                   # Auto-install MCP server if needed
    +-- get_job_errors.py               # Fetch errors for failed job runs
```

---

## Wave Structure

This demo is built in waves. Each wave delivers a working, demonstrable piece. Read `conf/demo_manifest.json` for the canonical wave definitions.

### Wave 1: Data Foundation
- Synthetic data generation notebooks (structured + unstructured)
- Unity Catalog setup (catalog, schemas, volumes)
- Data generation job definition
- **Deliverable**: Run the job, data lands in volumes

### Wave 2: DLT/SDP Pipelines
- Bronze layer: `STREAMING TABLE` + `read_files` from volumes
- Silver layer: `STREAMING TABLE` + `STREAM(LIVE.)` + quality constraints
- Gold layer: `MATERIALIZED VIEW` / `LIVE TABLE` + business aggregations
- Pipeline definition in `resources/pipelines/`
- **Deliverable**: Run the pipeline, medallion architecture populated

### Wave 3: ML + Analytics
- Feature engineering notebooks
- AutoML model training (domain-specific predictions)
- ML training job definition
- Lakeview dashboards (executive + operational)
- Genie space for natural-language analytics
- **Deliverable**: Trained models registered in UC, dashboards live

### Wave 4: Companion App
- FastAPI backend with 3-layer architecture
- React frontend with shadcn/ui components
- Lakebase database for operational state
- Integration with lakehouse data (read from gold tables)
- **Deliverable**: App deployed on Databricks Apps

### Wave 5: AI Agents + Polish
- Databricks Agent Framework agents for domain tasks
- Talk track generation
- End-to-end demo script
- Final polish and validation
- **Deliverable**: Complete, presentation-ready demo

---

## Agent Roles

Use the `/build-next` command to advance to the next wave. Agents are spawned as needed:

| Agent | File | When to Spawn | What It Does |
|-------|------|---------------|--------------|
| **lakehouse-dev** | `.claude/agents/lakehouse-dev.md` | Waves 1-3 | Data gen, DLT SQL, ML notebooks, job configs |
| **app-dev** | `.claude/agents/app-dev.md` | Wave 4 | FastAPI backend, React frontend, Lakebase |
| **dashboard-dev** | `.claude/agents/dashboard-dev.md` | Wave 3 | Lakeview dashboards, Genie spaces |
| **reviewer** | `.claude/agents/reviewer.md` | Every wave | Quality gates, code review, adherence check |

---

## DLT/SDP Patterns

**CRITICAL**: Follow these patterns exactly. Deviations cause pipeline failures.

### Bronze Layer (Raw Ingestion)
```sql
-- Pattern: STREAMING TABLE + read_files
CREATE OR REFRESH STREAMING TABLE {{ catalog }}.{schema}.bronze_{entity}
COMMENT "Raw {entity} data from landing zone"
TBLPROPERTIES ("quality" = "bronze")
AS SELECT
  *,
  current_timestamp() AS ingestion_timestamp
FROM STREAM read_files(
  "/Volumes/{{ catalog }}/{schema}/landing/{entity}",
  format => "parquet",
  inferColumnTypes => true,
  schemaLocation => "/Volumes/{{ catalog }}/{schema}/checkpoints/bronze_{entity}"
);
```

### Silver Layer (Cleaned + Validated)
```sql
-- Pattern: STREAMING TABLE + STREAM(LIVE.) + constraints
CREATE OR REFRESH STREAMING TABLE {{ catalog }}.{schema}.silver_{entity} (
  CONSTRAINT valid_id EXPECT ({entity}_id IS NOT NULL) ON VIOLATION DROP ROW,
  CONSTRAINT valid_required EXPECT ({required_field} IS NOT NULL) ON VIOLATION DROP ROW
)
COMMENT "Cleaned {entity} data with quality constraints"
TBLPROPERTIES ("quality" = "silver")
AS SELECT
  {selected_columns},
  ingestion_timestamp,
  current_timestamp() AS processed_timestamp
FROM STREAM({{ catalog }}.{schema}.bronze_{entity});
```

### Gold Layer (Business Aggregations)
```sql
-- Pattern: LIVE TABLE (not streaming) + aggregations
CREATE OR REFRESH LIVE TABLE {{ catalog }}.{schema}.gold_{metric}
COMMENT "Gold: {business metric description}"
TBLPROPERTIES (
  "quality" = "gold",
  "pipelines.autoOptimize.zOrderCols" = "{key_columns}"
)
AS
SELECT
  {dimensions},
  {aggregations},
  current_timestamp() AS _last_updated
FROM {{ catalog }}.{schema}.silver_{source}
{joins}
GROUP BY {dimensions};
```

### Rules
1. Bronze: Always `STREAMING TABLE` + `read_files` from Volumes
2. Silver: Always `STREAMING TABLE` + `STREAM(LIVE.)` from bronze + quality constraints
3. Gold: Always `LIVE TABLE` (materialized view) -- NOT streaming
4. Never put `.example` files in `bronze/`, `silver/`, `gold/` directories (causes pipeline failure)
5. Use JSONL format (one JSON object per line) when JSON is needed, NOT arrays
6. Each SQL file = one table. File name = `{entity}_{tier}.sql`
7. Fully qualify all table names: `{{ catalog }}.{schema}.{table}`
8. Use Volumes for data landing: `/Volumes/{{ catalog }}/{schema}/landing/{entity}`
9. Use Volumes for checkpoints: `/Volumes/{{ catalog }}/{schema}/checkpoints/{table}`

---

## 3-Layer Backend Architecture (Companion App)

```
HTTP Request
    |
    v
+-------------------------------+
| Router (thin -- HTTP only)    |  Validates input, delegates, returns response
| backend/app/api/{resource}.py |
+---------------+---------------+
                |  Depends()
                v
+-------------------------------+
| Service (business logic)      |  Rules, validation, orchestration. No HTTP, no SQL.
| backend/app/services/*.py     |
+---------------+---------------+
                |
                v
+-------------------------------+
| Repository (data access)      |  SQL queries, ORM calls. No business logic.
| backend/app/repositories/*.py |
+---------------+---------------+
                |
                v
        [Lakebase -- Databricks-managed Postgres]
```

**Rules:**
- Routers are **thin**: validate input, call service, return response. No SQL. No business logic.
- Services contain **all business logic**: validation, calculations, orchestration.
- Repositories are **data access only**: SQL queries, ORM operations.
- Dependencies flow **downward only**: router -> service -> repository. Never the reverse.
- Pydantic models: Base (shared fields), Create (no id), Update (all optional), Response (with id + timestamps).

---

## Quality Gates

### Python (Lakehouse + Backend)
```bash
uv run ruff check src/ --fix
uv run ruff format src/
uv run mypy src/
uv run pytest tests/ -v
```

### Frontend (Companion App)
```bash
cd apps/{{ app_name }}/frontend
npx tsc --noEmit
npx vitest run
npm run build
```

### Deployment
```bash
databricks bundle validate
databricks bundle deploy -t dev
```

Run quality gates after EVERY wave. No exceptions.

---

## Configuration

| Setting | Value |
|---------|-------|
| **Company** | {{ display_name }} |
| **Industry** | {{ industry | upper }} |
| **Unity Catalog** | `{{ catalog }}` |
| **SQL Warehouse** | `{{ warehouse_id }}` |
| **Workspace** | `{{ workspace_host }}` |
{% if databricks_profile %}| **CLI Profile** | `{{ databricks_profile }}` |
{% endif %}{% if lakebase_url %}| **Lakebase URL** | `{{ lakebase_url }}` |
{% endif %}| **Notifications** | `{{ notification_email }}` |
| **Bundle Name** | `{{ bundle_name }}` |
| **App Name** | `{{ app_name }}` |

---

## Development Commands

```bash
# --- Asset Bundles ---
databricks bundle validate                    # Validate bundle config
databricks bundle deploy -t dev               # Deploy to dev target
databricks bundle run -t dev {job_name}       # Run a specific job
databricks bundle destroy -t dev              # Tear down dev deployment

# --- Data Generation ---
databricks bundle run -t dev {{ bundle_name }}_data_generation

# --- DLT Pipeline ---
databricks bundle run -t dev {{ bundle_name }}_dlt_pipeline

# --- ML Training ---
databricks bundle run -t dev {{ bundle_name }}_ml_training

# --- Deploy ---
./scripts/deploy-all.sh dev                                   # Full deploy: lakehouse + app
./scripts/deploy-lakehouse.sh dev                             # Lakehouse only
./scripts/deploy-lakehouse.sh dev --skip-to=pipeline          # Skip data gen, run pipeline + ML
./scripts/deploy-app.sh dev                                   # App only
SKIP_APP=true ./scripts/deploy-all.sh dev                     # Full deploy without app

# --- Companion App (local dev) ---
cd apps/{{ app_name }}
uv run uvicorn backend.app.main:app --port 8000 --reload

# --- Debugging ---
python scripts/get_job_errors.py <RUN_ID>              # Fetch errors for a failed run
python scripts/get_job_errors.py <RUN_ID> --json       # JSON output for programmatic use

# --- Quality ---
uv run ruff check . --fix && uv run ruff format .
uv run mypy src/
uv run pytest tests/ -v --cov
```

---

## Missing Context Protocol

If you lack information needed to complete a task:

1. **Check `conf/demo_manifest.json`** -- it defines the demo structure.
2. **Check `docs/supplements/`** -- architecture, standards, playbook.
3. **Check `.claude/skills/`** -- Databricks-specific patterns and examples.
4. **Check the wave definition** -- what this wave is supposed to deliver.
5. **If still unclear, ASK the user.** Do NOT guess. Do NOT make assumptions about:
   - Table schemas (the domain expert knows the business)
   - Dashboard layouts (the user has a vision)
   - ML targets (the user knows what to predict)
   - App features (the user defines the UX)

---

## Key References

| Document | Purpose |
|----------|---------|
| `conf/demo_manifest.json` | **Single source of truth**: waves, status, schemas, decisions |
| `docs/IMPLEMENTATION_PLAN.md` | What to build in each wave (detail beyond manifest) |
| `docs/PROGRESS.md` | Human-readable view (derived from manifest — not a separate state) |
| `docs/supplements/ARCHITECTURE.md` | Detailed architecture patterns |
| `docs/supplements/CODING_STANDARDS.md` | Quality bar for all code |
| `docs/supplements/WAVE_PLAYBOOK.md` | Wave governance rules |
| `.claude/agents/*.md` | Agent role definitions |
| `.claude/commands/*.md` | Custom slash commands |
| `.claude/skills/*/skill.md` | Databricks skill references |
| `src/data_generation/CLAUDE.md` | Data gen patterns, UDF pitfalls, volume paths |
| `src/pipelines/CLAUDE.md` | DLT tier rules, naming convention, common mistakes |
| `src/ml/CLAUDE.md` | Decimal casting, feature discovery, MLflow patterns |

---

## Session Persistence

`conf/demo_manifest.json` is the **single source of truth** for all state. To survive session changes:

### After Each Wave
- Update `conf/demo_manifest.json`: set wave `status`, add `decisions`
- That's it. The manifest IS the state.

### Session Handoff
When you finish a session or the context is getting long:
1. Update manifest with current wave status + any new decisions
2. If mid-wave, add a decision entry: `{"summary": "Wave N paused: [what's done, what's remaining]"}`
3. Next session reads the manifest and knows exactly where to resume

---

## Lessons Learned

**Maintain `docs/LESSONS_LEARNED.md` throughout the build.** Every time you hit an issue, find a workaround, or discover a pattern that wasn't obvious — write it down.

### What to Log
- DLT/SDP gotchas (syntax issues, streaming vs batch, constraint failures)
- Data generation issues (Faker limitations, volume sizes, schema mismatches)
- Dashboard pitfalls (column name mismatches, queryLines formatting, Genie quirks)
- Job configuration issues (dependency ordering, environment packages, timeouts)
- App development patterns (Lakebase connection, API design choices, frontend state)
- Bundle deployment issues (validation errors, resource conflicts)

### Format
```markdown
## [Date] — [Short title]
**Context:** What you were doing
**Issue:** What went wrong
**Fix:** What solved it
**Prevention:** How to avoid it next time
```

### Why This Matters
The user may bring these lessons back to improve the exokit generator itself — fixing templates, updating skills, and making future demos better. Write them as if someone else will read them.

---

## Talk Track & Demo Narrative

After the demo is built, use `/talk-track` to generate a presentation narrative. The manifest's `talk_track_hints` field contains user-provided hints about what to emphasize.

When building each wave, think about the **demo story**:
- Wave 1 (Data): "Here's how we ingest raw {{ industry }} data at scale..."
- Wave 2 (Pipelines): "The medallion architecture ensures data quality..."
- Wave 3 (ML/Dashboards): "Now we can predict [target] and visualize insights..."
- Wave 4 (App): "This is what {{ display_name }}'s team would use day-to-day..."
- Wave 5 (Agents): "And here's AI automation for domain experts..."

Always suggest creative, impressive demo scenarios to the user. You're not just building code — you're building a story.

---

## Things You Must NEVER Do

1. Never hardcode catalog names -- use `{{ catalog }}` from config / variables.
2. Never hardcode warehouse IDs -- use `{{ warehouse_id }}` from config / variables.
3. Never commit secrets, API keys, or `.env` files.
4. Never put `.example` files in DLT `bronze/`, `silver/`, `gold/` directories.
5. Never use JSON arrays for data files -- use JSONL format.
6. Never skip quality gates between waves.
7. Never modify `conf/demo_manifest.json` structure -- only update `status` fields.
8. Never build ahead of the current wave without completing it first.
9. Never put business logic in route handlers -- use the service layer.
10. Never put SQL in the service layer -- use the repository.
11. Never use `fetch` + `useState` for server state -- use TanStack Query.
12. Never hand-roll UI components that shadcn/ui provides.
13. Never write CSS classes -- use Tailwind utilities.
14. Never deploy without running `databricks bundle validate` first.
15. Never create streaming gold tables -- gold layer uses `LIVE TABLE` (materialized view).
16. Never assume table schemas -- check `conf/demo_manifest.json` or ask the user.
17. Never skip the reviewer agent after completing a wave.
18. Never patch code without understanding the root cause first.
19. Never act silently on failure -- report errors immediately.
20. Never improvise agent prompts -- use the definitions in `.claude/agents/`.
