# Plan: Add PostgreSQL Support via DuckDB ATTACH with Query Pushdown

## Context

The n6k server serves data through DuckDB under the `db.*` schema. Clients query `db.table_name` via REST endpoints returning Arrow/Parquet. DuckDB runs with `enable_external_access = false` for security.

The goal: let app developers ATTACH a PostgreSQL database so DuckDB pushes queries (filters, aggregations) down to Postgres natively. No materialization — live data, live pushdown.

Future consideration: per-session views over postgres data (e.g., `WHERE admin = {{ current_user_id }}`). The architecture must support this via server-side template interpolation without needing ATTACH on session DBs.

## Approach: Shared DB with ATTACH + Sandboxed Sessions

`shared_db` gets `enable_external_access = true` and the postgres ATTACH (`READ_ONLY`). Session DBs stay sandboxed (`enable_external_access = false`). Postgres-backed queries route through `shared_db`.

DuckDB's `pg_experimental_filter_pushdown` defaults to `true` with ATTACH — filters push down to Postgres automatically. DuckDB views over attached postgres tables get pushdown because the optimizer inlines views before planning.

**Security model:**
- `/tables/{name}` uses `pushdown.py` — parameterized queries only, no raw client SQL. Routes to `shared_db` for postgres-backed tables.
- `/views/{view_id}` uses **session DBs** — `enable_external_access = false`. Clients can't `read_csv()`, `ATTACH`, or touch postgres.
- Developer `.view()` creates views in `db.*` as the security boundary. Nothing exposed by default.

## Developer-Facing API

`postgres()` creates a connection handle. **Nothing is exposed by default.** Only explicitly `.view()`d tables appear in `db.*`.

### File-based (in `data.py`, e.g. `demo/pg/data.py`):
```python
import os
from n6k_server.registry import postgres

pg = postgres(connection=os.environ["PG_DATABASE_URL"])

@pg.view()
def active_users():
    return "SELECT id, name FROM pgdb.public.users WHERE active = true"

@pg.view()
def recent_orders():
    return "SELECT id, total, created_at FROM pgdb.public.orders WHERE created_at > now() - interval '30 days'"

@pg.rpc()
def search_users(df, query: str):
    return df[df['name'].str.contains(query)]
```

### Programmatic (in app setup):
```python
import os
from n6k_server.app import create_app, postgres

app = create_app()
pg = postgres(app, connection=os.environ["PG_DATABASE_URL"])

@pg.view()
def active_users():
    return "SELECT id, name FROM pgdb.public.users WHERE active = true"
```

Only `.view()`d queries appear under `db.*`. The developer's SELECT is the security boundary — it defines exactly what data is exposed. Connection string lives in env vars, never in code.

## How Query Pushdown Works

```
Client: GET /tables/active_users?filters=[{"col":"name","op":"=","value":"Sean"}]
  ↓
pushdown.py builds: SELECT * FROM db."active_users" WHERE "name" = ?  [params: ["Sean"]]
  ↓
db."active_users" is a DuckDB VIEW: SELECT id, name FROM pgdb.public.users WHERE active = true
  ↓
DuckDB inlines the view, combines predicates, pushes down to Postgres:
  → SELECT id, name FROM users WHERE active = true AND name = 'Sean'
  ↓
Postgres executes, returns only matching rows → Arrow response to client
```

## Implementation

### Step 1: `src/n6k_server/postgres.py` (new)

- `create_pg_db(conn_string, alias="pgdb") -> duckdb.DuckDBPyConnection`
  - `duckdb.connect()` (in-memory)
  - `INSTALL postgres; LOAD postgres;`
  - `ATTACH '{conn_string}' AS {alias} (TYPE postgres, READ_ONLY);`
  - Fails fast if Postgres unreachable

### Step 2: `src/n6k_server/table_registry.py` (modify)

Add `PostgresTableSpec`:
```python
@dataclass
class PostgresTableSpec:
    name: str
    source_sql: str  # e.g. "SELECT * FROM pgdb.public.users"
    registered: bool = False
```

Add to `TableRegistry`:
- `self._pg_tables: dict[str, PostgresTableSpec] = {}`
- `register_postgres_table(name, source_sql)` — stores spec
- `_ensure_pg_table_loaded(name)` — creates `VIEW db."{name}" AS {source_sql}` on `shared_db` only
- `is_pg_table(name)` — returns True if name is a postgres-backed table
- Add `_pg_tables` to `all_names`, `is_known`, `ensure_loaded`

### Step 3: `src/n6k_server/routes/tables.py` (modify)

Route postgres-backed tables to `shared_db`:
```python
def _get_db(request: Request, name: str = None) -> duckdb.DuckDBPyConnection:
    registry = request.app.state.table_registry
    if name and registry.is_pg_table(name):
        return request.app.state.shared_db
    session = get_session(request)
    if session is not None:
        return session.db
    return request.app.state.shared_db
```

Pass `name` to `_get_db` in `get_table()` and `get_schema()`.

### Step 4: `src/n6k_server/app.py` (modify)

Add `PostgresHandle` class — connection only, nothing exposed by default:
```python
class PostgresHandle:
    def __init__(self, app, conn_string):
        self._app = app
        self._conn_string = conn_string
        app.state.pg_configs.append({"conn_string": conn_string})

    def view(self, name=None):
        # Same pattern as DataSource.view()
        # Registers the view SQL as a postgres-backed view in the registry
        def decorator(fn):
            resolved = name or fn.__name__
            sql = fn()
            self._app.state.table_registry.register_postgres_table(resolved, sql)
            return None
        return decorator

    def rpc(self, name=None):
        # Same pattern as DataSource.rpc()
        ...
```

Add `postgres(app, *, connection)` factory.

Modify lifespan:
- Add `app.state.pg_configs = []`
- If any pg_configs, create `shared_db` with postgres ATTACH via `create_pg_db()`
- The `shared_db` then gets the `db` schema and regular tables registered on top (existing flow)
- Close on shutdown

### Step 5: `src/n6k_server/session.py` (modify)

`create_shared_db` needs a variant that starts from an existing connection (the one with ATTACH) rather than `duckdb.connect()`. Split `_init_db` to accept an optional pre-configured connection:
```python
def _init_db(cached_tables, views=None, db=None):
    if db is None:
        db = duckdb.connect()
        db.execute("SET enable_external_access = false")
    db.execute("CREATE SCHEMA IF NOT EXISTS db")
    # ... rest unchanged
```

### Step 6: `src/n6k_server/registry.py` (modify)

Add `DeferredPostgresHandle` and `postgres()` for file-based registration:
```python
class DeferredPostgresHandle:
    def __init__(self, conn_string):
        self._conn_string = conn_string
        self._views = []   # list of (name, fn)
        self._rpcs = []    # list of (name, fn)

    def view(self, name=None):
        def decorator(fn):
            resolved = name or fn.__name__
            self._views.append((resolved, fn))
            return None
        return decorator

    def rpc(self, name=None):
        def decorator(fn):
            resolved = name or fn.__name__
            self._rpcs.append((resolved, fn))
            return fn
        return decorator

def postgres(*, connection):
    handle = DeferredPostgresHandle(connection)
    _current.postgres_sources.append(handle)
    return handle
```

Add `postgres_sources: list` to `_Registry`. Include in `collect()`.

### Step 7: `src/n6k_server/discover.py` (modify)

Handle `registered["postgres"]` in `_apply_to_app`:
- For each `DeferredPostgresHandle`, create a `PostgresHandle` on the app
- Replay `.view()` and `.rpc()` registrations

## Security Model

| Concern | Mitigation |
|---------|-----------|
| Client runs ATTACH | Session DBs have `enable_external_access = false` |
| Client calls `read_csv()` / `httpfs` | Session DBs block all external access |
| Client accesses raw Postgres tables | Only `.view()`d tables exist in `db.*`; nothing exposed by default |
| Connection string leakage | Only in `shared_db` memory; `env=` reads from os.environ |
| SQL injection via table queries | `pushdown.py` uses parameterized queries |
| Write access to Postgres | `READ_ONLY` on ATTACH |
| Client-created views touching postgres | `/views` endpoint uses session DBs (no ATTACH) |

## Files to Modify

| File | Change |
|------|--------|
| **New:** `src/n6k_server/postgres.py` | `create_pg_db()` — ATTACH logic |
| `src/n6k_server/table_registry.py` | `PostgresTableSpec`, register/load, `is_pg_table()` |
| `src/n6k_server/routes/tables.py` | Route postgres tables to `shared_db` |
| `src/n6k_server/app.py` | `PostgresHandle`, `postgres()`, lifespan changes |
| `src/n6k_server/session.py` | Accept pre-configured db in `_init_db` |
| `src/n6k_server/registry.py` | `DeferredPostgresHandle`, `postgres()`, `collect()` |
| `src/n6k_server/discover.py` | Handle postgres registrations |
| **New:** `demo/pg/data.py` | Example usage |

## Verification

1. Create `demo/pg/data.py`, ATTACH a local Postgres, expose a table with `.view()`
2. Query via `/tables/{name}` with filters — verify pushdown by checking Postgres query log
3. Verify client cannot create a view referencing postgres tables via `/views` endpoint (session db, no ATTACH)
4. Verify `pushdown.py` filters combine correctly with view predicates
5. Verify server fails fast if Postgres is unreachable at startup
