# Timescale Migration

`uforia` now supports a hybrid storage model:
- immutable Parquet remains the audit/recovery source of truth
- Postgres/Timescale becomes the query and serving layer for hot API/UI/model reads

## Runtime env

Set:

```bash
export UFORIA_DB_ENABLED=true
export UFORIA_DB_DSN=postgresql:///uforia?user=$USER
export UFORIA_DB_READ_MODE=auto
export UFORIA_DB_WRITE_MODE=dual
export UFORIA_DB_STRICT_DATASETS="$(uv run python - <<'PY'
from uforia.contracts import UI_EXPOSED_DATASETS
print(",".join(UI_EXPOSED_DATASETS))
PY
)"
```

`UFORIA_DB_READ_MODE`:
- `parquet`: ignore DB
- `db`: require DB for reads
- `auto`: prefer DB, fall back to Parquet

`UFORIA_DB_WRITE_MODE`:
- `parquet`: current behavior
- `dual`: write Parquet and DB
- `db`: DB only

`UFORIA_DB_STRICT_DATASETS`:
- comma-separated dataset names that must read from DB when DB is enabled
- the repo-managed API runner now derives this from `UI_EXPOSED_DATASETS` automatically, so all UI/API dataset reads are DB-only after full backfill

Frequent DB sync jobs are available in `ops/` and `launchd/`:
- `ops/run-db-sync-hot.sh`
- `ops/run-db-sync-raw-recent.sh`
- `launchd/com.uforia.db-sync-hot.plist`
- `launchd/com.uforia.db-sync-raw-recent.plist`

They are intended to keep DB close to parquet even if a dual-write path misses rows:
- hot serving datasets every 10 minutes
- recent raw microstructure datasets every 30 minutes

## Local bootstrap

```bash
./ops/run-db.sh
uv run uforia-admin db init
```

`ops/run-db.sh` prefers Docker Compose when Docker is available. If Docker is not installed, it falls back to Homebrew `postgresql@17` and starts a native local database instead.

For the native Postgres path, `ops/run-db.sh` also applies conservative ingest-oriented settings:
- `checkpoint_timeout = 30min`
- `checkpoint_completion_target = 0.9`
- `max_wal_size = 8GB`
- `wal_compression = on`

These settings reduce checkpoint thrash during large backfills and raw microstructure dual-write loads.

## Historical migration

```bash
./ops/run-db-backfill.sh
uv run uforia-admin db verify --dataset vvix_series
uv run uforia-admin db verify --dataset microstructure_raw_event
```

To migrate every UI/API dataset before the final cutover, use:

```bash
./ops/run-db-backfill-ui.sh
```

To stage migration in smaller passes, set a comma-separated dataset list:

```bash
export UFORIA_DB_BACKFILL_DATASETS=dvol_snapshot,funding_rate_snapshot,vvix_series,razor_series
./ops/run-db-backfill.sh
```

The backfill runner uses `nice` so it yields to the live API/UI more gracefully on the Mac Mini.

For off-hours raw microstructure loading, use the dedicated wrapper:

```bash
./ops/run-db-backfill-raw.sh
```

That defaults to:
- `microstructure_raw_event`
- `microstructure_book_state`
- `microstructure_bar`

and is intended for maintenance windows rather than daytime API/UI operation.

You can also backfill a bounded slice:

```bash
uv run uforia-admin db backfill \
  --dataset microstructure_bar \
  --start 2026-03-01T00:00:00Z \
  --end 2026-03-03T00:00:00Z \
  --filter asset=BTC
```

## Scope

The migration includes all dataset families, including:
- raw microstructure events
- book state
- bars
- feature/label/model datasets
- options/perps/VVIX/Razor/funding datasets
- ingestion and run metadata

## Operational model

- API/UI now read UI-exposed datasets from Timescale only once backfill is complete.
- Builders and collectors dual-write during migration and steady state.
- Parquet remains available for audit, replay, verification, and non-UI recovery flows.

## Verification

Minimum parity checks:
- row counts by dataset/day
- min/max timestamps
- latest-row parity on UI-serving datasets
- sample raw-event/book-state/bar parity for selected days

## Current limitation

Latest-view semantics in DB are implemented in query helpers rather than prebuilt materialized views. That keeps the contract aligned with existing dataset semantics while avoiding a large schema explosion during the first migration pass.
