# Microstructure System

## What It Is

The microstructure family is now the spec-aligned medium-horizon engine inside `uforia`.

Production direction:
- historical training on imported Binance + Bybit data from the deprecated `crypto-microstructure`
- live prediction from `uforia` native feeds
- horizons:
  - `30s`
  - `1m`
  - `2m`
  - `5m`
- output scope:
  - calibrated predictions
  - paper signals
  - engine-health / data-quality context

The old `baseline_logistic` story is retired. The active model family is:
- `tcn_boost_tabular`
- `tcn_boost_tcn`
- `tcn_boost_fusion`
- `tcn_boost_paper_signal`

## Architecture

`uforia` is now the only maintained runtime.

Responsibilities:
- Rust under [engines/microstructure](/Users/mubarisnk/dev/personal/uforia/engines/microstructure)
  - Binance + Bybit live feeds
  - book reconstruction
  - Binance REST snapshot bootstrap + WS sequence validation + reseed on gap
  - raw JSONL persistence
  - live book-state / health persistence
- Python under [src/uforia/microstructure](/Users/mubarisnk/dev/personal/uforia/src/uforia/microstructure)
  - legacy-history import
  - canonical bar / feature / label builders
  - hybrid training pipeline
  - calibration, evaluation, export
  - live inference + paper-signal generation
- UI/API
  - `/api/v1/microstructure/*`
  - `/microstructure/*`

## Model Stack

The production-aligned workflow is staged:

1. Tabular baseline gate
   - boosted-tree classifier on handcrafted features
2. Causal TCN
   - sequence model over LOB / flow tensors
3. Fusion model
   - boosted meta model on tabular features + frozen TCN embeddings
4. Calibration + paper-signal layer
   - calibrated probabilities
   - edge / tradability / blocked-state logic

Current production model version:
- `tcn_boost_fusion_v2`

Correctness changes in `v2`:
- label horizons are computed from the actual bar size, not assumed minute bars
- validation uses an embargo gap of `max(sequence_length - 1, label_horizon_steps)`
- cumulative delta is rolling and stationary (`10/30/50` bars), not a full-history cum-sum
- `iv_rv_gap` annualizes realized vol from the actual bar frequency
- paper edge / Sharpe are based on realized forward returns net of round-trip costs
- old `v1` artifacts are not considered valid for production use

Outputs include:
- `prob_up`
- `prob_flat`
- `prob_down`
- `expected_move_bps`
- `expected_edge_bps`
- `confidence`
- `confidence_bucket`
- `blocked_reason`
- `model_family`

## Canonical Datasets

The microstructure dataset family is:
- `microstructure_raw_event`
- `microstructure_book_state`
- `microstructure_bar`
- `microstructure_feature_series`
- `microstructure_label_series`
- `microstructure_model_output`
- `microstructure_regime_series`
- `microstructure_signal_series`
- `microstructure_training_run`
- `microstructure_backtest_run`

Important schema changes versus the earlier baseline path:
- raw events preserve:
  - exchange timestamp
  - receive timestamp
  - local processing timestamp
  - transaction timestamp where available
  - update-id range fields for sequence validation
- book-state rows now preserve:
  - `is_seeded`
  - `is_sequence_valid`
  - `is_gap_detected`
  - `reseed_count`
  - `book_quality`
- labels are built from fee-adjusted forward microprice movement, not the old simple direction target

## Historical Import

Legacy history from `/Users/mubaris/dev/crypto-microstructure` is imported into canonical `uforia` datasets.

Command:

```bash
uv run uforia-admin microstructure import-history \
  --root /Users/mubaris/dev/crypto-microstructure
```

What is reused from the old project:
- historical data
- feed normalization ideas
- reconstruction helpers
- feature-builder logic where it matches the new design

What is not reused:
- legacy model assumptions
- legacy signal definitions

## Feature and Label Flow

Builders are driven from canonical bars/book state and enrich with:
- book imbalance / microprice / spread state
- order-flow imbalance / rolling cumulative delta windows
- realized-vol and short-horizon flow-volatility state
- basis / funding / OI / liquidation context
- cross-venue spread state
- options context where aligned in time

Label construction is cost-aware and horizon-specific.

Important label rule:
- if the requested horizon is not divisible by the active bar size, the label row is marked `incompatible_bar_size`
- this means `30s` on minute-only historical data is intentionally rejected until sub-minute live history exists

Core commands:

```bash
uv run uforia-admin microstructure build-features \
  --asset BTC \
  --start 2026-01-01T00:00:00Z \
  --end 2026-03-09T00:00:00Z

uv run uforia-admin microstructure build-labels \
  --asset BTC \
  --start 2026-01-01T00:00:00Z \
  --end 2026-03-09T00:00:00Z \
  --horizons 30s,1m,2m,5m

uv run uforia-admin microstructure build-regime \
  --asset BTC \
  --start 2026-01-01T00:00:00Z \
  --end 2026-03-09T00:00:00Z
```

## Training and Export

Train/evaluate/export/generate is now TCN-Boost-oriented:

```bash
uv run uforia-admin microstructure train --asset BTC --horizon 1m
uv run uforia-admin microstructure evaluate --asset BTC --horizon 1m
uv run uforia-admin microstructure generate-signals --asset BTC --horizon 1m
```

Artifacts written by export:
- tabular boosted-tree model
- fusion boosted-tree model
- TCN state dict
- TCN ONNX export
- JSON metadata containing:
  - feature schema version
  - supported horizon
  - calibration curves
  - lift vs tabular
  - profitability gate result
  - feature schema version
  - label schema version
  - validation embargo bars
  - best-iteration metadata for tabular and fusion models

## Live Engine

The Rust live engine writes under:
- `data/microstructure_live/raw/market/date=YYYY-MM-DD/events.jsonl`
- `data/microstructure_live/raw/funding/date=YYYY-MM-DD/events.jsonl`
- `data/microstructure_live/raw/iv/date=YYYY-MM-DD/events.jsonl`
- `data/microstructure_live/book_state/date=YYYY-MM-DD/state.jsonl`
- `data/microstructure_live/health/latest.json`

Binance correctness rule:
- REST snapshot first
- buffer WS depth updates until seeded
- accept only sequence-bridging updates
- detect gap
- mark book degraded
- reseed from REST

Bybit remains WS-driven but is normalized into the same health contract.

## Live Inference

The Python live-inference path reads the Rust live output and writes canonical predictions/signals.

One-shot:

```bash
uv run uforia-admin microstructure infer-live-once \
  --live-root data/microstructure_live \
  --model-dir models/microstructure
```

Daemon:

```bash
uv run uforia-admin microstructure infer-live-daemon \
  --live-root data/microstructure_live \
  --model-dir models/microstructure \
  --poll-seconds 5
```

This is the bridge between:
- live Rust feeds
- trained Python artifacts
- UI/API paper signals

## UI Pages

The UI family under `/microstructure` is:
- Overview
- Predictions
- Regime
- Features
- Calibration
- Backtests
- Engine Health

Use them in this order:
1. `Engine Health`
   Confirm books are seeded and sequence-valid.
2. `Predictions`
   Check calibrated probabilities and blocked reasons by horizon.
3. `Regime`
   Confirm whether market state supports acting on the prediction.
4. `Features`
   Verify the model is reacting to interpretable pressure.
5. `Calibration`
   Confirm the active model is not overconfident.
6. `Backtests`
   Check whether the current family still clears cost-aware validation.

## Individual Signals

Current signal ids:
- `btc_microstructure_30s`
- `btc_microstructure_1m`
- `btc_microstructure_2m`
- `btc_microstructure_5m`
- `eth_microstructure_30s`
- `eth_microstructure_1m`
- `eth_microstructure_2m`
- `eth_microstructure_5m`

These are also exposed on the global `Overview` and `Signals` pages.

## Trade Usage

Use the system as a gating stack:
- prediction quality first
- cost / tradability second
- regime support third
- feed / book health always

Do not trust a strong directional score when:
- the book is unseeded
- a sequence gap was just detected
- confidence is low
- blocked reason is non-null

## Failure Modes

- Imported minute history is still weaker for true `30s` training than live sub-minute collection.
- Binance books should be treated as invalid until `is_seeded=true` and `is_sequence_valid=true`.
- TCN export is present, but the live engine still uses the exported artifact contract rather than a fully in-Rust feature-fusion path.
- The hybrid model is only as good as the feature/label alignment between historical import and live feeds.
