# opendeviationbar-providers

**Parent**: [/crates/CLAUDE.md](/crates/CLAUDE.md)

Rust crate implementing data provider adapters for Binance (crypto) and Exness (forex), normalizing into the unified `AggTrade` format.

---

## Binance Vision CSV Format History

**CRITICAL KNOWLEDGE**: Binance Vision spot aggTrades CSV format has CHANGED between 2024 and 2025.

### Format by Year

| Year Range   | Timestamp Column (field 6) | Format | Example Value      |
| ------------ | -------------------------- | ------ | ------------------ |
| 2019–2024    | 13-digit milliseconds      | ms     | `1655251200000`    |
| 2025–present | 16-digit microseconds      | μs     | `1749945600183572` |

### Column Layout (All Years, No Header)

All Binance Spot Vision ZIPs use a **headerless 8-column CSV**:

```
agg_trade_id, price, quantity, first_trade_id, last_trade_id, timestamp, is_buyer_maker, is_best_match
```

Columns 1–5, 7–8 are unchanged across all years. **Only column 6 (timestamp) changed.**

### How the Rust Code Handles Both

`crates/opendeviationbar-core/src/timestamp.rs` — `normalize_timestamp()`:

```rust
const MICROSECOND_THRESHOLD: u64 = 10_000_000_000_000;

pub fn normalize_timestamp(raw_timestamp: u64) -> i64 {
    if raw_timestamp < MICROSECOND_THRESHOLD {
        (raw_timestamp * 1_000) as i64  // ms → μs
    } else {
        raw_timestamp as i64            // already μs
    }
}
```

**Both formats are handled transparently** — no year-based branching needed.

### Python Downstream Handling

`_stream_bars_with_processor()` in `orchestration/helpers.py`:

```python
if use_native_arrow and "timestamp" in chunk.columns:
    # stream_binance_trades_arrow returns μs (after Rust normalize_timestamp).
    # process_trades_arrow() expects ms. Convert before processing.
    chunk = chunk.with_columns((pl.col("timestamp") // 1000).alias("timestamp"))
```

This `// 1000` (μs → ms) is always applied:

- **2019–2024 data**: Rust normalized 13-digit ms → 16-digit μs. Python divides back → 13-digit ms. ✓
- **2025+ data**: Rust passed 16-digit μs through unchanged. Python divides → 13-digit ms. ✓

### UM Futures vs Spot Difference

The README claims "Spot = 16-digit μs". This is accurate only for **2025+ data**. Historical spot data (2019–2024) used 13-digit ms. The `normalize_timestamp()` threshold handles the difference automatically.

| Market     | Column Layout     | Header | Timestamp (Current) | Timestamp (Pre-2025) |
| ---------- | ----------------- | ------ | ------------------- | -------------------- |
| Spot       | `a,p,q,f,l,T,m,b` | No     | 16-digit μs         | 13-digit ms          |
| UM Futures | Descriptive       | Yes    | 13-digit ms         | 13-digit ms          |

### Key Files

| File                                                          | Role                                             |
| ------------------------------------------------------------- | ------------------------------------------------ |
| `crates/opendeviationbar-core/src/timestamp.rs`               | `normalize_timestamp()` — ms/μs detection + conv |
| `crates/opendeviationbar-providers/src/binance/historical.rs` | Vision ZIP download + CSV parsing                |
| `python/opendeviationbar/orchestration/helpers.py`            | `_stream_bars_with_processor()` — μs → ms step   |

### Bug History: Vision ZIP "width 32 vs 31" (2026-03-12)

**Symptom**: `pl.concat(bar_frames)` in `_stream_bars_with_processor()` failed with "unable to append a DataFrame of width 32 with a DataFrame of width 31".

**Root cause**: When `include_incomplete=True`, the incomplete bar from `processor.get_incomplete_bar()` returns a Python dict (31 unconditional columns including `lookback_*` features), which has a different schema than the Arrow-format frames (32 columns: OHLCV + microstructure + trade IDs, NO lookback features). Mixing the two in `bar_frames` before `pl.concat` caused the mismatch.

**Fix** (`orchestration/helpers.py`): Two-part normalization before appending the incomplete bar to `bar_frames`:

1. **Width fix**: null-fill missing Arrow columns (e.g. `turnover`, `buy_trade_count`), drop dict-only columns (e.g. `lookback_*`)
2. **Type fix**: cast existing columns to match Arrow dtype (dict gives Int64 for `individual_trade_count`, `agg_record_count`, etc.; Arrow exports them as UInt32). Without this cast, ClickHouse INSERT fails with "type Int64 is incompatible with expected type UInt32".

**Confirmed**: This bug affected ALL years (not year-specific). The Vision timestamp format change (ms → μs) was a red herring; Rust's `normalize_timestamp()` already handled both correctly. Vision now writes ~100–700 bars/day at ~3 sec/day instead of REST fallback ~4 min/day.

---

## Related

- [/crates/CLAUDE.md](/crates/CLAUDE.md) - Workspace overview
- [/CLAUDE.md](/CLAUDE.md) - Project hub (Spot-Only Data Source Policy)
- `crates/opendeviationbar-core/src/timestamp.rs` — `normalize_timestamp()`
- `crates/opendeviationbar-cli/src/bin/data_structure_validator.rs` — cross-year validation binary
