# Crates Workspace

**Parent**: [/CLAUDE.md](/CLAUDE.md) | **Architecture**: [/docs/ARCHITECTURE.md](/docs/ARCHITECTURE.md)

10-crate modular Rust workspace. 5 crates published to crates.io (`opendeviationbar-core`, `opendeviationbar-providers`, `opendeviationbar-streaming`, `opendeviationbar-client`, `opendeviationbar-hurst`); 5 internal-only (`publish = false`).

---

## Quick Reference

| Crate                        | Purpose         | Key Types                                                     |
| ---------------------------- | --------------- | ------------------------------------------------------------- |
| `opendeviationbar-hurst`     | Hurst exponent  | `HurstExecutor`, `HurstConfig`                                |
| `opendeviationbar-core`      | Core algorithm  | `OpenDeviationBarProcessor`, `OpenDeviationBar`, `FixedPoint` |
| `opendeviationbar-providers` | Data sources    | `HistoricalDataLoader`, `ExnessFetcher`                       |
| `opendeviationbar-io`        | I/O operations  | Polars integration                                            |
| `opendeviationbar-streaming` | Real-time       | `StreamManager`, `LiveBarEngine`, `CompletedBar`              |
| `opendeviationbar-batch`     | Batch analytics | `BatchAnalysisEngine`                                         |
| `opendeviationbar-config`    | Configuration   | `Settings`                                                    |
| `opendeviationbar-cli`       | CLI tools       | (disabled for PyPI)                                           |
| `opendeviationbar-client`    | SSE consumer    | `OdbSseClient`, `OdbBar`, `OdbSseEvent`                       |
| `opendeviationbar`           | Meta-crate      | v4.0 compatibility                                            |

---

## Dependency Graph

```
                    opendeviationbar (meta-crate)
                           │
     ┌─────────────────────┼─────────────────────┐
     │                     │                     │
opendeviationbar-cli         opendeviationbar-batch      opendeviationbar-streaming
     │                     │                     │
     └──────────┬──────────┴──────────┬──────────┘
                │                     │
          opendeviationbar-io          opendeviationbar-providers
                │                     │
                └──────────┬──────────┘
                           │
                     opendeviationbar-core
                           │
                     opendeviationbar-config

opendeviationbar-client (independent, SSE consumer via eventsource-client + reqwest)

opendeviationbar-hurst (independent library, MIT-licensed fork of GPL-3.0)
```

---

## Microstructure Features (v7.0)

**Location**: `opendeviationbar-core/src/processor.rs`

10 features computed in Rust during bar construction:

| #   | Feature                   | Formula                                     | Range        |
| --- | ------------------------- | ------------------------------------------- | ------------ |
| 1   | `duration_us`             | close_time - open_time                      | [0, +inf)    |
| 2   | `ofi`                     | (buy_vol - sell_vol) / total                | [-1, 1]      |
| 3   | `vwap_close_deviation`    | (close - vwap) / (high - low)               | ~[-1, 1]     |
| 4   | `price_impact`            | abs(close - open) / volume                  | [0, +inf)    |
| 5   | `kyle_lambda_proxy`       | ((close-open)/open) / (imbalance/total_vol) | (-inf, +inf) |
| 6   | `trade_intensity`         | trade_count / duration_sec                  | [0, +inf)    |
| 7   | `volume_per_trade`        | volume / trade_count                        | [0, +inf)    |
| 8   | `aggression_ratio`        | buy_count / sell_count                      | [0, 100]     |
| 9   | `aggregation_density_f64` | individual_count / agg_count                | [1, +inf)    |
| 10  | `turnover_imbalance`      | (buy_turn - sell_turn) / total_turnover     | [-1, 1]      |

**Edge cases**: Division by zero returns 0.0 (no information).

**Academic backing**: Kyle (1985), Amihud (2002), Easley et al. (2012).

---

## Trade ID Tracking (Issue #72, v12.4+)

**Location**: `opendeviationbar-core/src/types.rs`

Each `OpenDeviationBar` tracks aggregate trade ID range for data integrity verification:

| Field                | Type  | Description                            |
| -------------------- | ----- | -------------------------------------- |
| `first_agg_trade_id` | `i64` | First AggTrade ID that opened this bar |
| `last_agg_trade_id`  | `i64` | Last AggTrade ID processed in this bar |

**Use cases**:

- **Gap detection**: `bars[i].first_agg_trade_id == bars[i-1].last_agg_trade_id + 1`
- **Data integrity**: Verify no trades were dropped during processing
- **Checkpoint validation**: Confirm resume alignment after interruption

**Python constant**: `TRADE_ID_RANGE_COLUMNS = ("first_agg_trade_id", "last_agg_trade_id")`

**ClickHouse schema**: Columns added via migration (see `schema.sql` v12.4 migration section)

---

## Inter-Bar Features (Issue #59)

**Location**: `opendeviationbar-core/src/interbar.rs`

16 features computed from a lookback window of trades BEFORE each bar opens. All fields are `Option<T>` (None when lookback disabled or insufficient data).

### Tier 1: Core (7 features)

| Feature                    | Formula                          | Range      |
| -------------------------- | -------------------------------- | ---------- |
| `lookback_trade_count`     | COUNT(trades)                    | [0, ∞)     |
| `lookback_ofi`             | (buy_vol - sell_vol) / total     | [-1, 1]    |
| `lookback_duration_us`     | last_ts - first_ts               | [0, ∞)     |
| `lookback_intensity`       | trade_count / duration_sec       | [0, ∞)     |
| `lookback_vwap_raw`        | Σ(price×vol) / Σ(vol)            | FixedPoint |
| `lookback_vwap_position`   | (vwap - low) / (high - low)      | [0, 1]     |
| `lookback_count_imbalance` | (buy_count - sell_count) / total | [-1, 1]    |

### Tier 2: Statistical (6 features)

| Feature                     | Formula                    | Range   |
| --------------------------- | -------------------------- | ------- |
| `lookback_kyle_lambda`      | (Δp/p) / (imbalance/vol)   | (-∞, ∞) |
| `lookback_burstiness`       | (σ*τ - μ*τ) / (σ*τ + μ*τ)  | [-1, 1] |
| `lookback_volume_skew`      | E[(V-μ)³] / σ³             | (-∞, ∞) |
| `lookback_volume_kurt`      | E[(V-μ)⁴] / σ⁴ - 3         | [-2, ∞) |
| `lookback_price_range`      | (high - low) / first_price | [0, ∞)  |
| `lookback_garman_klass_vol` | OHLC volatility            | [0, ∞)  |

### Tier 3: Advanced (3 features)

| Feature                        | Formula                      | Range  | Min Trades |
| ------------------------------ | ---------------------------- | ------ | ---------- |
| `lookback_kaufman_er`          | \|net\| / Σ\|changes\|       | [0, 1] | 2          |
| `lookback_hurst`               | DFA estimator (soft-clamped) | [0, 1] | 64         |
| `lookback_permutation_entropy` | Bandt-Pompe m=3              | [0, 1] | 60         |

**Academic backing**: Kyle (1985), Goh-Barabási (2008), Garman-Klass (1980), Bandt-Pompe (2002), Peng (1994).

**Plan**: [/docs/plans/issue-59-inter-bar-features.md](/docs/plans/issue-59-inter-bar-features.md)

---

## Crate Details

### Layer 0: Foundation

#### opendeviationbar-config

- Dependencies: `config`, `serde`
- Public API: `Settings::load()`, `Settings::default()`

#### opendeviationbar-hurst

Independent MIT-licensed library for Hurst exponent estimation.

- Dependencies: `linreg`
- Purpose: DFA-based Hurst exponent computation for inter-bar feature extraction
- **License**: MIT (fork of GPL-3.0 evrom/hurst to enable PyPI distribution)

**Key types**: `HurstExecutor`, `HurstConfig`

#### opendeviationbar-core

Core algorithm with performance-first dependencies.

- Dependencies: `chrono`, `serde`, `serde_json`, `thiserror`, `foldhash` (20-40% faster than ahash on numeric keys), `mimalloc` (2-5x multithreaded), `rayon`, `smallvec`, `quick_cache`, `wide` (stable SIMD)
- Features:
  - `test-utils` - CSV loading for tests
  - `python` - PyO3 type exports
  - `api` - utoipa OpenAPI schemas
  - `simd-burstiness` (default) - 2-4x speedup for burstiness computation
  - `simd-kyle-lambda` (default) - 2-4x speedup for Kyle lambda computation

**Key types**:

- `AggTrade` - Input trade data
- `OpenDeviationBar` - Output bar with OHLCV + microstructure
- `FixedPoint` - 8-decimal fixed-point arithmetic
- `OpenDeviationBarProcessor` - Stateful processor

**Key properties**:

- Non-lookahead breach detection
- Temporal integrity (breach tick in closing bar)
- Streaming-friendly (maintains state between calls)

#### Open Deviation Bar Construction Algorithm

How an open deviation bar is built, step by step:

1. **First trade arrives** → Opens a new bar. `open = high = low = close = trade.price`. Thresholds computed: `upper = open * (1 + threshold)`, `lower = open * (1 - threshold)`.

2. **Subsequent trades arrive** → `update_with_trade()`: updates `close`, extends `high`/`low`, accumulates `volume`, `turnover`, `buy_volume`/`sell_volume`, `trade_count`, `vwap`.

3. **Breach detected** → Trade price exceeds `upper` or `lower` threshold AND timestamp differs from `open_time` (Issue #36 timestamp gate). The breaching trade is **included in the closing bar** (its price/volume updates the bar), then the bar is finalized with `compute_microstructure_features()`.

4. **After breach** → `defer_open = true`. The processor holds **no bar state**. The breaching trade does NOT open the next bar (Issue #46).

5. **Next trade after breach** → Because `defer_open == true`, this trade opens a fresh bar. `defer_open` resets to `false`.

**Key invariants**:

| Rule                                   | Detail                                                                              |
| -------------------------------------- | ----------------------------------------------------------------------------------- |
| Breaching trade belongs to closing bar | Its OHLCV contribution is in the bar that closes                                    |
| Next trade opens the new bar           | NOT the breaching trade (Issue #46, `defer_open`)                                   |
| `bar.timestamp` = `close_time`         | Downstream consumers (pandas DatetimeIndex) use close time                          |
| `bar.open_time` and `bar.close_time`   | Milliseconds UTC; close_time >= open_time always                                    |
| Timestamp gate (Issue #36)             | Bar cannot close on same timestamp as it opened                                     |
| Batch/streaming parity                 | `process_agg_trade_records()` and `process_single_trade()` produce identical output |
| Checkpoint continuity                  | `defer_open` persists in checkpoints for cross-file processing                      |
| Ouroboros reset                        | `reset_at_ouroboros()` clears bar state and resets `defer_open`                     |

### Layer 1: Data Access

#### opendeviationbar-providers

- Depends on: `opendeviationbar-core`
- Features:
  - `binance` (default) - Spot, UM futures, CM futures
  - `exness` - 10 forex pairs
  - `all-providers` - Enable all
- **TLS**: Uses `rustls-tls` (no OpenSSL dependency, enables zig cross-compilation)

**Binance types**: `HistoricalDataLoader`, `BinanceMarket`
**Exness types**: `ExnessFetcher`, `ExnessInstrument`

**Transient retry** (`send_with_rate_limit_backoff()`): All Binance REST requests retry transient errors with exponential backoff. Retries: HTTP 429 (rate limit), 5xx (server error), network errors, timeouts (30s per attempt). Backoff: 1s → 2s → 4s → 8s → 16s cap, 5 attempts max. Non-transient 4xx (except 429) returned immediately.

#### opendeviationbar-io

- Depends on: `opendeviationbar-core`
- Features:
  - `parquet` - Enables Polars
  - `all`

Export formats: CSV, Parquet, Arrow IPC

### Layer 2: Engines

#### opendeviationbar-streaming

- Depends on: `opendeviationbar-core`, `opendeviationbar-providers` (optional)
- Features: `binance-integration`, `stats`, `indicators`, `all`
- Bounded memory, circuit breaker pattern

**Types**: `StreamingProcessor`, `StreamingConfig`

**Gap detection**: `TradeIdGapDetector` emits `GapEvent` on trade-ID discontinuity. `puck_fill()` handles inline gap recovery via parallel REST fetch (sub-3s).

#### opendeviationbar-batch

- Depends on: `opendeviationbar-core`, `opendeviationbar-io`
- Rayon parallel processing

**Types**: `BatchAnalysisEngine`, `BatchConfig`, `AnalysisReport`

### Layer 3: Applications

#### opendeviationbar-cli (disabled)

6 CLI binaries preserved as source, not built for PyPI:

- `tier1-symbol-discovery`
- `opendeviationbar-analyze`
- `data-structure-validator`
- `spot-tier1-processor`
- `polars-benchmark`
- `temporal-integrity-validator`

#### opendeviationbar (meta-crate)

Re-exports for v4.0 backward compatibility.

---

## Development Commands

```bash
# Test specific crate
cargo nextest run -p opendeviationbar-core

# Test with features
cargo nextest run -p opendeviationbar-core --features test-utils

# Test all (excludes PyO3 crate)
cargo nextest run --workspace --exclude opendeviationbar-py

# Build with features
cargo build -p opendeviationbar-providers --features exness
cargo build -p opendeviationbar-core --features test-utils
```

---

## Common Patterns

### Adding a New Feature to opendeviationbar-core

1. Add field to `OpenDeviationBar` struct in `types.rs`
2. Update `compute_microstructure_features()` in `processor.rs`
3. Add tests in `processor.rs` or dedicated test file
4. Update PyO3 bindings in `/src/lib.rs`
5. Update Python types in `/python/opendeviationbar/__init__.pyi`
6. Update ClickHouse schema if caching

### Feature Flags

```toml
# Enable in Cargo.toml (use workspace inheritance for version SSoT)
[dependencies]
opendeviationbar-core = { workspace = true, features = ["test-utils"] }

# Or via CLI
cargo build -p opendeviationbar-core --features test-utils,python
```

### Adding Internal Cross-Dependencies

Internal crate cross-deps must use workspace inheritance — **never hardcode versions**:

1. Add to `[workspace.dependencies]` in root `Cargo.toml`:

   ```toml
   opendeviationbar-foo = { path = "crates/opendeviationbar-foo", version = "X.Y.Z" }
   ```

2. In the depending crate's `Cargo.toml`: `opendeviationbar-foo = { workspace = true }`
3. Add the new `version_toml` path to `pyproject.toml` semantic-release config

semantic-release bumps `workspace.package.version` and all `workspace.dependencies` versions in sync on each release (v13.6.4+).

---

## Testing

### Unit Tests

```bash
cargo nextest run -p opendeviationbar-core
```

### Property-Based Tests

Uses `proptest` for invariant checking:

```rust
proptest! {
    #[test]
    fn ofi_always_bounded(buy in 0.0f64..1e12, sell in 0.0f64..1e12) {
        let total = buy + sell;
        if total > f64::EPSILON {
            let ofi = (buy - sell) / total;
            prop_assert!(ofi >= -1.0 && ofi <= 1.0);
        }
    }
}
```

### Integration Tests

```bash
# Tests in crates/opendeviationbar/tests/
cargo nextest run -p opendeviationbar --features full
```

---

## Critical Implementation Details

### Fixed-Point Arithmetic

The opendeviationbar crate uses **8-decimal fixed-point arithmetic** to avoid floating-point errors:

```rust
// Convert opendeviationbar FixedPoint to f64 for Python
let price_f64 = bar.open.to_f64();
```

**Warning**: Do NOT round or truncate - preserve full precision.

### Timestamp Handling

**Rust internal**: `open_time` / `close_time` are i64 **microseconds**.

**Dict export** (`src/helpers.rs` → `opendeviationbar_to_dict()`):

- `"open_time_ms"` / `"close_time_ms"` — Int64 **milliseconds** (us / 1000)
- No legacy `"timestamp"` key (removed in timestamp revamp)

**Arrow export** (`arrow_export.rs`):

- `"open_time"` / `"close_time"` — Int64 **microseconds** (raw Rust values)

**ClickHouse**: `close_time_ms` (ORDER BY key) + `open_time_ms`

**DatetimeIndex**: Always `close_time` (when bar completed, actionable signal).

```python
# Dict path (conversion.py):
df["timestamp"] = pd.to_datetime(df["close_time_ms"], unit="ms", utc=True)
df = df.set_index("timestamp")

# Arrow path (processors/api.py):
df["timestamp"] = pd.to_datetime(df["close_time"], unit="us")
df = df.set_index("timestamp")
```

### Error Handling (PyO3)

Map Rust errors to Python exceptions:

```rust
let processor = OpenDeviationBarProcessor::new(threshold_decimal_bps)
    .map_err(|e| PyErr::new::<pyo3::exceptions::PyValueError, _>(
        format!("Failed to create processor: {}", e)
    ))?;
```

**Python exception types**:

- `ValueError`: Invalid input (negative threshold, missing fields)
- `RuntimeError`: Processing errors (unsorted trades, internal failures)
- `KeyError`: Missing required dictionary keys

---

## Related

- [/CLAUDE.md](/CLAUDE.md) - Project hub (architecture, principles, terminology)
- [/docs/OPEN-DEVIATION-BAR.md](/docs/OPEN-DEVIATION-BAR.md) - Full algorithm explanation: layman + bit-exact ClickHouse oracle examples (>=/<= proof, Stathera chain)
- [/docs/ARCHITECTURE.md](/docs/ARCHITECTURE.md) - System design
- [crates/opendeviationbar-providers/CLAUDE.md](opendeviationbar-providers/CLAUDE.md) - Binance Vision CSV format history (ms→μs 2024→2025), bug history
- [/src/lib.rs](/src/lib.rs) - PyO3 bindings
- [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) - Python API, caching, validation, memory guards
- [/python/opendeviationbar/clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md) - ClickHouse cache, dedup hardening
- [/scripts/CLAUDE.md](/scripts/CLAUDE.md) - Pueue ops, per-year parallelization, anti-patterns
