# Exokit: Databricks Demo Template Generator — Master Whiteboard

> **Date:** 2026-03-02
> **Status:** Planning — awaiting final go to start Phase 1
> **Repo:** `/Users/fouad.alsayadi/Documents/code/pre-demo-template-project`
> **First demo target:** FSI (Financial Services) — Fraud Detection

---

## 1. Vision

Build the best demo template generator for Databricks that:
- Produces realistic, sophisticated demos representing actual enterprise use cases
- Unifies standards across all demos
- Is easy for partner engineers to use and evolve
- Combines **deterministic scaffolding** (CLI + Jinja2) with **Claude-driven content generation**
- Makes Claude able to continue building after scaffold by reading a rich context (CLAUDE.md, manifest, skills)

### The Flow

```
User: "I need a demo about FSI fraud detection for ACME Bank"
                    │
                    ▼
    ┌───────────────────────────────────┐
    │  Claude asks 5-7 questions:       │
    │  - Specific use cases?            │
    │  - Data volume expectations?      │
    │  - App ideas?                     │
    │  - Target audience?               │
    │  - Workspace/catalog/warehouse?   │
    │  - Lakebase connection URL?       │
    └───────────────────────────────────┘
                    │
                    ▼
    ┌───────────────────────────────────┐
    │  exokit init (deterministic)     │
    │  - Scaffolds full directory       │
    │  - Generates databricks.yml       │
    │  - Creates CLAUDE.md + agents     │
    │  - Sets up app skeleton           │
    │  - Writes demo_manifest.json      │
    │  - Installs MCP server            │
    │  - Bundles skills                 │
    └───────────────────────────────────┘
                    │
                    ▼
    ┌───────────────────────────────────┐
    │  User cd's into generated project │
    │  and starts new Claude session    │
    │                                   │
    │  Claude reads CLAUDE.md and       │
    │  demo_manifest.json — knows       │
    │  exactly what to build            │
    └───────────────────────────────────┘
                    │
                    ▼
    ┌───────────────────────────────────┐
    │  Claude builds incrementally in   │
    │  waves with quality gates:        │
    │  Wave 1: Data Foundation          │
    │  Wave 2: Lakehouse Pipelines      │
    │  Wave 3: Intelligence Layer       │
    │  Wave 4: Analytics & AI           │
    │  Wave 5: Showcase Application     │
    └───────────────────────────────────┘
```

---

## 2. Decisions Locked

| Decision | Choice | Rationale |
|----------|--------|-----------|
| **Approach** | Hybrid: deterministic scaffold + Claude content | CLI generates structure/configs/governance. Claude fills business logic, schemas, ML, app pages. |
| **Tool** | Both: `exokit` CLI + Claude `/demo` command | CLI for scaffolding (distributable to partners). Command wraps CLI and continues with content gen. |
| **App stack** | **Full speckit spirit from day 1** + APX's OpenAPI client gen | 3-layer backend (Router→Service→Repo), TanStack Query, Zustand, shadcn/ui, production-grade patterns. OpenAPI-generated frontend API client. Partners get a real template, not a toy. |
| **First demo** | FSI (Financial Services) — Fraud Detection | Rich ML use cases, strong agent story, impressive dashboards. |
| **Skills distribution** | Hybrid: bundle essential only + `exokit update-skills` | No duplicates, no pollution. Essential skills only. Add more as needed over time. |
| **MCP server** | Auto-install ai-dev-kit post-scaffold | User gets 70+ MCP tools immediately when opening Claude Code. |
| **Lakebase** | Ask during init | Baked into .env.example and app.yaml. Ready to go. |
| **Scope** | Full 7 layers in waves with governance | Ambitious with wisdom. Agile incremental waves. Spec → execution plan → governance. |
| **Demo pace** | **Hybrid: fast for lakehouse, quality for app** | Waves 1-4 (data, pipelines, ML, dashboards) move fast — well-understood patterns. Wave 5 (app) gets full attention and polish since it's THE star. |
| **Manifest architecture** | **Split: env/config params vs project manifest** | Infrastructure params (workspace, lakebase URL, catalog, warehouse) go into env files / databricks.yml / app.yaml — compatible with Asset Bundles and Databricks Apps. Project skeleton (waves, schemas, decisions) stays in manifest. Claude can modify manifest **only with user approval** — changes affect project scope and deliverables, never silent. |
| **App scaffolding** | **Use APX for app foundation, layer governance on top** | `apx init --addons ui,lakebase,sql,claude` gives us FastAPI + React + Vite + OpenAPI client + Lakebase + shadcn for free. We overlay: 3-layer architecture, agents, CLAUDE.md, supplements. APX owns the app plumbing. We own the architecture + lakehouse + governance. |
| **V2 architecture** | **Design as library-first from day 1** | Scaffold engine is a Python library (`core/`), not CLI-only code. CLI is a thin wrapper (V1). Future web app with wizard UI + Claude Agent SDK is another thin wrapper (V2). Questionnaire is data, not UI — renderable as terminal prompts OR React forms. |

---

## 2b. V2 Vision: Web App with Embedded Claude

> Designed for now, built later. V1 CLI architecture makes V2 trivial.

```
V1 (Now):   exokit CLI  →  core library  →  scaffold  →  user opens Claude Code
V2 (Later): Databricks App (React wizard + Claude chat)  →  same core library  →  scaffold  →  embedded Claude session
```

**Library-first architecture:**
```
src/exokit/
├── core/                    # THE LIBRARY — reusable by CLI (V1) and web app (V2)
│   ├── scaffold.py          # Jinja2 rendering, directory creation
│   ├── manifest.py          # demo_manifest.json builder
│   ├── skills.py            # Skill bundler
│   ├── prereqs.py           # Pre-flight validation
│   ├── questionnaire.py     # Question definitions as DATA (not UI)
│   └── templates/           # Jinja2 templates
├── cli.py                   # V1 thin wrapper: Typer prompts → core.scaffold.generate(answers)
└── api.py                   # V2 thin wrapper: FastAPI routes → core.scaffold.generate(answers)
```

**Questionnaire as data (V2-ready):**
```python
# core/questionnaire.py — renderable by any UI
QUESTIONS = [
    Question(id="company", type="text", prompt="Company name", required=True),
    Question(id="industry", type="select", prompt="Industry", options=["fsi", "telecom", "retail", ...]),
    Question(id="catalog", type="text", prompt="Catalog name",
             default=lambda a: f"{a['company'].lower()}_{a['industry']}_demo"),
    # ... 7-10 questions total
]
```

---

## 3. Resources Analyzed

### What We Steal From Each

| Resource | Location | What We Steal | What We Skip |
|----------|----------|--------------|-------------|
| **speckit** | `/Users/fouad.alsayadi/Documents/code/speckit` | Governance spine (simplified): CLAUDE.md structure, 3-layer arch, spec-driven waves, missing context protocol, formatted memory, quality gates, wave gatekeeper. Spec template pattern (spec/plan/contract/execution-plan). | 15K lines of verbosity. 8 agents → 4. Sizing-specific content. Excessive duplication across agents. |
| **the_demo** | `/Users/fouad.alsayadi/Documents/demos/the_demo` | Complete lakehouse pattern: data gen → DLT → ML → dashboards. 9-department structure. Medallion architecture. Job DAGs. DLT SQL patterns. | Hardcoded to Nexus. No app. No agent use case. No unstructured data. |
| **template** | `/Users/fouad.alsayadi/Documents/demos/template` | `/demo` command orchestration. `.example` files pattern. Agent team coordination (builder/validator/reviewer/adherence). Incremental build-validate loop. Skill library (DLT, jobs, dashboards, SDK, bundles, data gen). | No real templating engine. No deterministic scaffold. No app scaffold. Full Claude runtime gen = no reproducibility. |
| **apx** | `/Users/fouad.alsayadi/Documents/code/apx` | **OpenAPI-generated frontend API client**. FastAPI+React scaffold pattern. OAuth flow. App.yaml patterns. | Too lightweight. No lakehouse integration. No governance. |
| **ai-dev-kit** | `/Users/fouad.alsayadi/Documents/code/ai-dev-kit` | **70+ MCP tools** (install post-scaffold). **26+ skills** (bundle snapshots). Pipeline/dashboard/job/UC tools. SQL execution. Layered architecture (tools-core → MCP server). | Builder app (too simple for our needs). |
| **everything-claude-code** | `/Users/fouad.alsayadi/Documents/code/everything-claude-code` | Agent definition format (YAML frontmatter). Plugin manifest schema. Hook patterns. Skill frontmatter standard. CLAUDE.md template examples. TDD workflow. Planner/architect agent patterns. | Language-specific skills we don't need. 57 skills is overkill — cherry-pick. |

---

## 4. Architecture

### Component 1: `exokit` CLI (This Repo)

```
pre-demo-template-project/          # This repo — THE GENERATOR
├── pyproject.toml                   # Installable via uv/pip
├── CLAUDE.md                        # Generator's own governance
├── src/exokit/
│   ├── __init__.py
│   ├── cli.py                       # Typer CLI: init, validate, update-skills
│   ├── questionnaire.py             # Interactive questions (Rich + Typer)
│   ├── scaffold.py                  # Jinja2 template rendering engine
│   ├── manifest.py                  # demo_manifest.json builder
│   ├── skills.py                    # Skill bundler + updater
│   ├── prereqs.py                   # Pre-flight checks
│   └── templates/                   # Jinja2 templates for everything below
│       ├── root/                    # CLAUDE.md, databricks.yml, pyproject.toml, .mcp.json
│       ├── claude/                  # .claude/agents/, skills/, commands/
│       ├── docs/                    # supplements, governance
│       ├── lakehouse/               # data_gen, pipelines, ml, dashboards, agents
│       ├── app/                     # FastAPI + React scaffold
│       ├── conf/                    # catalog_schema, data_config, demo_manifest
│       └── scripts/                 # deploy.sh, setup-catalog.sh, validate.sh
├── skills/                          # Bundled skill snapshots
│   ├── databricks-spark-declarative-pipelines/
│   ├── databricks-jobs/
│   ├── databricks-aibi-dashboards/
│   ├── databricks-asset-bundles/
│   ├── databricks-model-serving/
│   ├── databricks-agent-bricks/
│   ├── databricks-synthetic-data-generation/
│   ├── databricks-vector-search/
│   ├── databricks-app-apx/
│   ├── demo-governance/             # Custom: quality gates, wave gatekeeper
│   └── demo-memory/                 # Custom: session resilience
└── tests/                           # Generator tests
```

**CLI Commands:**
- `exokit init` — Interactive questionnaire → scaffold project
- `exokit validate` — Pre-flight checks (databricks auth, warehouse, catalog)
- `exokit update-skills` — Refresh bundled skills from latest ai-dev-kit

### Component 2: Generated Project Structure

```
{company}-{industry}-demo/
├── CLAUDE.md                          # ~400 lines — THE BRAIN
├── .claude/
│   ├── agents/
│   │   ├── lakehouse-dev.md           # Data gen + DLT + ML specialist
│   │   ├── app-dev.md                 # FastAPI + React specialist
│   │   ├── dashboard-dev.md           # AI/BI + Genie specialist
│   │   └── reviewer.md               # Quality + adherence checker
│   ├── skills/                        # Bundled from ai-dev-kit + custom
│   │   ├── databricks-spark-declarative-pipelines/
│   │   ├── databricks-jobs/
│   │   ├── databricks-aibi-dashboards/
│   │   ├── databricks-asset-bundles/
│   │   ├── databricks-model-serving/
│   │   ├── databricks-agent-bricks/
│   │   ├── databricks-synthetic-data-generation/
│   │   ├── databricks-vector-search/
│   │   ├── databricks-app-apx/
│   │   ├── demo-governance/           # Quality gates, wave gatekeeper
│   │   └── demo-memory/              # Session resilience
│   └── commands/
│       ├── build-next.md              # Build the next wave
│       ├── deploy.md                  # Deploy current state to Databricks
│       └── talk-track.md             # Generate demo narrative
├── docs/
│   ├── IMPLEMENTATION_PLAN.md         # Auto-populated from manifest
│   ├── TALK_TRACK.md                  # Demo narrative (Claude generates)
│   ├── PROGRESS.md                    # Wave-by-wave tracker
│   └── supplements/
│       ├── ARCHITECTURE.md            # Three-layer arch + lakehouse patterns
│       ├── CODING_STANDARDS.md        # Simplified from speckit
│       └── WAVE_PLAYBOOK.md           # How waves work, governance rules
├── databricks.yml                     # Asset Bundle config
├── resources/
│   ├── jobs/
│   │   ├── data_generation_job.yml
│   │   └── ml_training_job.yml
│   └── pipelines/
│       └── main_pipeline.yml
├── src/
│   ├── data_generation/
│   │   ├── structured/                # Spark + Faker scripts (Claude fills)
│   │   └── unstructured/              # PDF/doc gen for RAG (Claude fills)
│   ├── pipelines/
│   │   ├── bronze/                    # DLT SQL (Claude fills)
│   │   ├── silver/
│   │   └── gold/
│   ├── ml/                            # Feature eng + training (Claude fills)
│   ├── agents/                        # Databricks agent framework (Claude fills)
│   └── dashboards/
│       ├── deploy_dashboard_cli.py    # Utility (from template — ready to use)
│       ├── validate_queries.py        # Utility (from template — ready to use)
│       └── inspect_schemas.py         # Utility (from template — ready to use)
├── apps/
│   └── main-app/                      # THE showcase app
│       ├── CLAUDE.md                  # App-specific context
│       ├── app.yaml                   # Databricks Apps config
│       ├── pyproject.toml
│       ├── backend/
│       │   ├── app/
│       │   │   ├── main.py            # FastAPI app factory (scaffold)
│       │   │   ├── config.py          # pydantic-settings (scaffold)
│       │   │   ├── database.py        # Lakebase connection (scaffold)
│       │   │   ├── api/               # Routers (Claude fills)
│       │   │   ├── services/          # Business logic (Claude fills)
│       │   │   ├── repositories/      # Data access (Claude fills)
│       │   │   ├── models/            # SQLAlchemy models (Claude fills)
│       │   │   └── schemas/           # Pydantic schemas (Claude fills)
│       │   └── alembic/               # Migrations
│       └── frontend/
│           ├── package.json
│           ├── vite.config.ts
│           ├── tsconfig.json
│           ├── tailwind.config.ts
│           └── src/
│               ├── main.tsx
│               ├── App.tsx            # Routing + providers (scaffold)
│               ├── components/ui/     # shadcn/ui (pre-installed)
│               ├── lib/
│               │   ├── api-client.ts  # Auto-generated from OpenAPI (APX pattern)
│               │   └── utils.ts
│               ├── hooks/             # TanStack Query hooks (Claude fills)
│               ├── stores/            # Zustand stores (Claude fills)
│               ├── pages/             # Route pages (Claude fills)
│               └── types/             # TypeScript types (Claude fills)
├── conf/
│   ├── catalog_schema.json            # UC structure (from questionnaire)
│   ├── data_config.json               # Data gen params (from questionnaire)
│   └── demo_manifest.json             # MASTER MANIFEST — Claude's roadmap
├── scripts/
│   ├── deploy.sh
│   ├── setup-catalog.sh
│   └── validate.sh
├── .mcp.json                          # ai-dev-kit MCP server (auto-configured)
└── .env.example
```

### Component 3: Split Configuration Architecture

**Key insight:** Infrastructure params and project definition are two different concerns. They belong in different files, following the conventions of the systems they configure.

#### 3a. Infrastructure Config (env files + native configs)

These go where Databricks tooling expects them — NOT in a custom manifest.

**`.env` / `.env.example`** — Runtime secrets and connection strings:
```bash
# Lakebase
LAKEBASE_URL=postgresql://user:pass@ep-xxx.database.us-east-1.cloud.databricks.com:5432/demo_db

# Databricks (inherited from CLI profile or set explicitly)
DATABRICKS_HOST=https://fevm-middle-east.cloud.databricks.com
DATABRICKS_TOKEN=  # or use profile-based auth

# App
DEV_USER=fouad.alsayadi@databricks.com
```

**`databricks.yml`** — Asset Bundle config (native Databricks format):
```yaml
bundle:
  name: acme-fsi-demo

variables:
  catalog: acme_fsi_demo
  warehouse_id: "abc123"
  notification_email: fouad.alsayadi@databricks.com

targets:
  dev:
    mode: development
    workspace:
      host: https://fevm-middle-east.cloud.databricks.com
```

**`apps/main-app/app.yaml`** — Databricks Apps config:
```yaml
command: ["python", "-m", "uvicorn", "backend.app.main:app", "--host", "0.0.0.0", "--port", "8000"]
env:
  - name: LAKEBASE_URL
    valueFrom: "demo-lakebase-url"  # Databricks secret
```

#### 3b. `demo_manifest.json` — Project Skeleton (The Brain)

This is the **project definition** — what to build, not where to build it. Claude reads this to understand scope. **Claude can modify it ONLY with explicit user approval** since changes affect project scope and deliverables.

```json
{
  "meta": {
    "company": "ACME Bank",
    "industry": "fsi",
    "generated_at": "2026-03-02T...",
    "exokit_version": "0.1.0"
  },
  "waves": [
    {
      "id": 1,
      "name": "Data Foundation",
      "status": "pending",
      "components": ["structured_data", "unstructured_data", "catalog_setup"],
      "description": "Generate all raw data and set up Unity Catalog"
    },
    {
      "id": 2,
      "name": "Lakehouse Pipelines",
      "status": "pending",
      "components": ["bronze_layer", "silver_layer", "gold_layer"],
      "description": "Build SDP bronze/silver/gold medallion pipeline"
    },
    {
      "id": 3,
      "name": "Intelligence Layer",
      "status": "pending",
      "components": ["feature_engineering", "ml_models", "model_serving"],
      "description": "Feature engineering, AutoML training, model registration + serving"
    },
    {
      "id": 4,
      "name": "Analytics & AI",
      "status": "pending",
      "components": ["dashboards", "genie_spaces", "agent_use_case"],
      "description": "AI/BI dashboards, Genie spaces, agent with tools"
    },
    {
      "id": 5,
      "name": "Showcase Application",
      "status": "pending",
      "components": ["app_backend", "app_frontend", "app_deployment"],
      "description": "THE app — full-stack React + FastAPI on Databricks Apps"
    }
  ],
  "schemas": {
    "_note": "Claude populates this after planning — table definitions, relationships, volumes. Requires user approval."
  },
  "talk_track_hints": [],
  "decisions": []
}
```

#### Why the Split Matters

| Concern | Where It Lives | Who Owns It | Modifiable By |
|---------|---------------|-------------|---------------|
| Workspace URL, auth | `.env` + `databricks.yml` | User / CLI | User only |
| Lakebase connection | `.env` + `app.yaml` secrets | User / CLI | User only |
| Catalog name, warehouse | `databricks.yml` variables | CLI at init | User only |
| Wave definitions, scope | `demo_manifest.json` | CLI at init | Claude **with user approval** |
| Schema designs, decisions | `demo_manifest.json` | Claude during build | Claude **with user approval** |
| Wave status, progress | `demo_manifest.json` + `PROGRESS.md` | Claude during build | Claude (autonomous — status tracking is safe) |

---

## 5. Every Demo Must Deliver (7 Layers)

| Layer | Wave | What Gets Built | Key Technology |
|-------|------|-----------------|----------------|
| 1. Raw Data Generation | Wave 1 | Structured (Spark + Faker) + Unstructured (PDFs for RAG) | Spark, Faker, PyMuPDF |
| 2. SDP Pipelines | Wave 2 | Bronze (Auto Loader) → Silver (quality constraints) → Gold (business metrics) | Delta Live Tables / SDP |
| 3. Feature Engineering | Wave 3 | Feature tables in Feature Store from gold layer | Databricks Feature Engineering |
| 4. ML Models | Wave 3 | AutoML training, MLflow tracking, UC model registry, serving endpoint | AutoML, MLflow, Model Serving |
| 5. Dashboards + Genie | Wave 4 | AI/BI Lakeview dashboards, Genie Spaces for natural language | Lakeview, Genie API |
| 6. Agent Use Case | Wave 4 | RAG agent using unstructured data, UC functions as tools | Databricks Agent Framework, Vector Search |
| 7. Showcase Application | Wave 5 | Full-stack React + FastAPI app on Databricks Apps + Lakebase | FastAPI, React, shadcn/ui, Lakebase |

---

## 6. Governance Model (Simplified from Speckit)

### Reduction: 15K lines → ~3K lines (80% cut)

| Speckit Has | Exokit Gets | Why Simplified |
|-------------|-------------|----------------|
| 8 agents (1539 lines) | **4 agents** (~400 lines total) | lakehouse-dev, app-dev, dashboard-dev, reviewer |
| CLAUDE.md (37KB) | **CLAUDE.md (~400 lines)** | Tech stack, arch, wave workflow, critical rules only |
| 7 supplement docs | **3 supplements** | ARCHITECTURE.md, CODING_STANDARDS.md, WAVE_PLAYBOOK.md |
| 4 skills | **3 custom skills** | quality-gates, wave-gatekeeper, demo-memory |
| Spec template (4 files/spec) | **demo_manifest.json** | Single source of truth, machine-readable |
| PROGRESS.md (77KB) | **PROGRESS.md (auto-tracked)** | Wave status in manifest + lightweight markdown |

### Patterns We Keep (Non-Negotiable)

1. **Missing Context Protocol** — Every agent lists what context it NEEDS before working. Prevents silent assumptions.
2. **Wave Gatekeeper** — Quality gate between every wave. Tests pass, coverage met, end-to-end wiring verified.
3. **Session Resilience** — `demo_manifest.json` + `PROGRESS.md` + memory files = triple safety net for session recovery.
4. **Spec-before-code** — Manifest IS the spec. Claude reads it, knows what to build.
5. **Never solo-code multi-file changes** — Spawn specialist agents.
6. **Quality gates after every change** — Lint, typecheck, test, build.
7. **User approval for critical decisions** — Architecture, schema design, app UX. Autonomous for implementation within approved plan.

### Agent Definitions (4 Agents)

| Agent | Responsibility | Missing Context Protocol |
|-------|---------------|------------------------|
| **lakehouse-dev** | Data gen, DLT pipelines, ML training, feature engineering | "What tables/columns? What data volumes? What business rules for quality constraints?" |
| **app-dev** | FastAPI backend + React frontend for the showcase app | "What pages? What API endpoints? What data does each page need? What's the user flow?" |
| **dashboard-dev** | AI/BI dashboards, Genie spaces, dashboard deployment | "What KPIs? What gold tables exist? What warehouse ID? What columns are available?" |
| **reviewer** | Quality gates, adherence checking, code review | "What was the plan? What wave are we in? What are the acceptance criteria?" |

---

## 7. App Architecture (Spirit of Speckit + Simplicity of APX)

### What We Take From Speckit
- Three-layer backend: Router → Service → Repository
- pydantic-settings for configuration
- SQLAlchemy 2.0 async with Lakebase (asyncpg runtime, psycopg3 migrations)
- React 19 + TypeScript strict + Vite + Tailwind + shadcn/ui
- TanStack Query for server state, Zustand for client state
- RFC 9457 error responses
- Alembic migrations

### What We Take From APX
- **OpenAPI-generated frontend API client** (no manual typing of API calls)
- Simpler project structure (fewer layers of abstraction)
- Lighter configuration (no 20 "never-do" rules in CLAUDE.md)
- Faster scaffolding

### What We Skip
- Speckit's exhaustive testing pyramid (no 737 tests for a demo)
- Speckit's 8 specialist agents (overkill)
- Speckit's funnel architecture (domain-specific to sizing)
- Pre-commit hooks (optional for demos, not enforced)
- GitHub Actions CI (demos don't need CI)

### App Scaffold Provides (Ready to Use)
- `main.py` — FastAPI app factory with lifespan, CORS, static files
- `config.py` — pydantic-settings with Lakebase URL, workspace host
- `database.py` — Async engine + session dependency
- `App.tsx` — React routing + providers (QueryClient, ErrorBoundary)
- `api-client.ts` — Auto-generated from FastAPI's OpenAPI spec
- `components/ui/` — shadcn/ui components pre-installed
- `app.yaml` — Databricks Apps deployment config
- Alembic configured with initial migration

### App Scaffold Leaves Empty (Claude Fills)
- API routers, services, repositories
- SQLAlchemy models, Pydantic schemas
- React pages, hooks, stores
- Business logic specific to the demo industry

---

## 8. Execution Plan

### Phase 1: Foundation (Build the CLI + skeleton)

**Goal:** `exokit init` produces a working project scaffold.

| Step | What | Details |
|------|------|---------|
| 1.1 | Set up this repo | pyproject.toml, src/exokit/, Typer CLI |
| 1.2 | Pre-flight checks | `exokit validate` — check databricks CLI, auth, uv, node |
| 1.3 | Questionnaire | Interactive questions with Rich — 7-10 questions max |
| 1.4 | Jinja2 scaffold engine | Render templates → project directory |
| 1.5 | CLAUDE.md template | The generated project's brain (~400 lines) |
| 1.6 | Agent definitions | 4 agent .md files with missing context protocol |
| 1.7 | Supplement docs | ARCHITECTURE.md, CODING_STANDARDS.md, WAVE_PLAYBOOK.md |
| 1.8 | App scaffold templates | FastAPI + React skeleton with OpenAPI client gen |
| 1.9 | Lakehouse templates | databricks.yml, job configs, pipeline configs, data gen stubs |
| 1.10 | Skill bundler | Copy ai-dev-kit skills + custom skills into generated project |
| 1.11 | MCP auto-install | Post-scaffold: install ai-dev-kit, configure .mcp.json |
| 1.12 | demo_manifest.json | Manifest builder from questionnaire answers |

### Phase 2: First Demo — FSI Fraud Detection

**Goal:** Validate the generator by building a real demo end-to-end.

| Wave | What | Components |
|------|------|-----------|
| Wave 1 | Data Foundation | Generate transaction data, customer data, fraud labels, PDFs (compliance docs, account statements) |
| Wave 2 | Lakehouse Pipelines | Bronze: Auto Loader from volumes. Silver: quality constraints, dedup, enrichment. Gold: customer 360, fraud metrics, risk scores. |
| Wave 3 | Intelligence Layer | Feature engineering (transaction patterns, velocity, amount anomalies). Fraud detection model (classification). Register in UC, create serving endpoint. |
| Wave 4 | Analytics & AI | Executive fraud dashboard. Genie space for investigator queries. Compliance agent (RAG over policy docs + UC function tools). |
| Wave 5 | Showcase App | Fraud investigation workbench: transaction explorer, risk scoring, case management, agent chat interface. |

### Phase 3: Polish + Distribute

**Goal:** Make it partner-ready and prove it generalizes.

| Step | What |
|------|------|
| 3.1 | Extract all learnings from Phase 2 into generator improvements |
| 3.2 | Add `/talk-track` command for demo narrative generation |
| 3.3 | Write partner documentation |
| 3.4 | Generate a second demo (Telecom or Retail) to prove generalization |
| 3.5 | Package for distribution (pip install, GitHub release) |

---

## 9. Risk Assessment

| Risk | Likelihood | Impact | Decision | Status |
|------|-----------|--------|----------|--------|
| App scaffold too complex for partners | Medium | High | **Full speckit spirit from day 1.** Partners get production-grade patterns. The quality IS the value. | RESOLVED |
| Too many bundled skills → overwhelming | Medium | Medium | **Essential only, no duplicates.** Add more as needed over time. No pollution. | RESOLVED |
| CLI questions too many → users abandon | Low | High | 7-10 questions max. Smart defaults for everything else. | MITIGATED |
| First demo takes too long | Medium | Medium | **Hybrid pace:** Fast for lakehouse (waves 1-4), quality for app (wave 5). | RESOLVED |
| Session resilience fails across contexts | Low | High | Triple safety: demo_manifest.json + PROGRESS.md + memory files. | MITIGATED |
| Generated code doesn't run on Databricks | Medium | High | Pre-flight checks. Validate DLT syntax. Use MCP tools to test. | MITIGATED |
| Skill snapshots go stale | Low | Medium | `exokit update-skills` command. Version pinning in manifest. | MITIGATED |
| Claude silently changes project scope | Medium | High | **Manifest changes require user approval.** Infra params in env files (user-only). Scope in manifest (Claude + approval). Status tracking is autonomous (safe). | RESOLVED |

---

## 10. Key Design Principles

1. **Deterministic where possible, intelligent where necessary.** The scaffold is always the same for a given set of inputs. The content adapts to the industry.

2. **Claude-friendly output.** Every generated file should make Claude MORE effective, not confused. CLAUDE.md is the single source of truth.

3. **Wave-based execution.** Never try to build everything at once. Each wave delivers something testable. Governance between waves prevents drift.

4. **Session resilience by design.** If the session is lost, the next session reads manifest + progress + memory and picks up exactly where it left off.

5. **Partner-friendly.** A partner engineer with Databricks access and Claude Code should be able to run `exokit init`, answer questions, and have a working scaffold in 5 minutes.

6. **Quality without bureaucracy.** Keep governance lean. 4 agents, not 8. 3 supplements, not 7. Rules that matter, not rules that annoy.

7. **The app is the star.** Everything else (data, pipelines, ML, dashboards) exists to make the showcase app impressive. The app demonstrates the power of lakehouse + Lakebase + Databricks Apps.

8. **Each demo gets its own catalog.** Clean isolation. No cross-demo interference. Easy cleanup.

---

## 11. Technology Stack

### Generator (This Repo)
| Component | Technology | Why |
|-----------|-----------|-----|
| CLI framework | Typer | Clean, typed CLI with auto-help. Better than Click for modern Python. |
| Template engine | Jinja2 | Industry standard. Handles conditionals, loops, filters. |
| Interactive UI | Rich | Beautiful terminal output, progress bars, tables. |
| Package manager | uv | Fast, modern Python package management. |
| Testing | pytest | Generator tests — ensure templates render correctly. |

### Generated Project — Lakehouse
| Component | Technology | Why |
|-----------|-----------|-----|
| Data generation | Spark + Faker + Pandas UDF | Distributed, realistic, scalable. |
| Unstructured data | PyMuPDF + Faker | Generate PDFs for RAG use cases. |
| Pipelines | Spark Declarative Pipelines (DLT) | Medallion architecture, streaming, quality constraints. |
| ML | Databricks AutoML + MLflow | Fast model training, experiment tracking. |
| Feature Store | Databricks Feature Engineering | Centralized features, point-in-time lookup. |
| Model Serving | Databricks Model Serving | Low-latency inference endpoints. |
| Dashboards | Lakeview AI/BI | Native Databricks dashboards. |
| Genie | Genie Spaces API | Natural language data exploration. |
| Agents | Databricks Agent Framework | RAG + tool-calling agents. |
| Vector Search | Databricks Vector Search | Similarity search for agent knowledge. |
| Deployment | Databricks Asset Bundles | Infrastructure-as-code, multi-environment. |
| Governance | Unity Catalog | One catalog per demo, schemas per domain. |

### Generated Project — Showcase App
| Component | Technology | Why |
|-----------|-----------|-----|
| Backend framework | FastAPI | Async, OpenAPI auto-gen, Pydantic-native. |
| ORM | SQLAlchemy 2.0 (async) | asyncpg runtime, psycopg3 migrations. |
| Database | Lakebase (Postgres) | Databricks-managed, connection URL provided. |
| Migrations | Alembic | Schema versioning. |
| Validation | Pydantic v2 | Request/response models. |
| Settings | pydantic-settings | Typed env config. |
| Frontend framework | React 19 + TypeScript | Modern, typed, component-based. |
| Build tool | Vite | Fast HMR, ESM-native. |
| Styling | Tailwind CSS + shadcn/ui | Utility-first + accessible components. |
| Server state | TanStack Query v5 | Cache, dedup, background refetch. |
| Client state | Zustand v5 | Minimal API, devtools. |
| API client | Auto-generated from OpenAPI | APX pattern — no manual typing. |
| Deployment | Databricks Apps | app.yaml, OAuth proxy. |

---

## 12. Open Questions (To Resolve During Phase 1)

1. ~~**Demo manifest evolution:**~~ → RESOLVED. Split into env/config (user-only) vs manifest (Claude + user approval). See Section 4, Component 3.
2. ~~**Skill curation:**~~ → RESOLVED. Essential only, no duplicates. Finalize exact list during FSI demo build.
3. **OpenAPI client generation:** APX uses a specific approach — need to validate it works with our FastAPI scaffold. May need `openapi-typescript-codegen` or similar.
4. **Unstructured data generation:** What's the best approach for generating realistic PDFs? PyMuPDF + Faker, or use the `databricks-unstructured-pdf-generation` skill?
5. **Agent framework integration:** How to scaffold the Databricks agent code? Need to study `databricks-agent-bricks` skill more closely.
6. **Partner distribution:** pip install from GitHub? Or just `git clone` + `uv sync`?

---

## 13. Success Criteria

### Phase 1 Success
- [ ] `exokit init` runs, asks questions, scaffolds a complete project
- [ ] Generated project has valid CLAUDE.md that Claude can follow
- [ ] Generated app skeleton builds and runs locally (FastAPI + React)
- [ ] MCP server auto-installed and working
- [ ] Skills bundled and readable by Claude

### Phase 2 Success
- [ ] FSI demo Wave 1: Data generated and visible in Unity Catalog
- [ ] FSI demo Wave 2: DLT pipeline runs, bronze/silver/gold tables populated
- [ ] FSI demo Wave 3: ML model trained, registered in UC, serving endpoint live
- [ ] FSI demo Wave 4: Dashboard deployed, Genie space working, agent responding
- [ ] FSI demo Wave 5: App deployed on Databricks Apps, end-to-end functional
- [ ] Talk track document generated

### Phase 3 Success
- [ ] Second demo (different industry) generated and functional
- [ ] Partner documentation complete
- [ ] Generator distributable and installable

---

*This document is the living whiteboard. Update it as decisions evolve.*
