---
name: full-symbol-streaming-spike
description: Spike to validate sidecar streaming all 18 symbols via combined WebSocket streams. Eliminates persistent midnight-boundary Stathera gaps caused by kintsugi/backfill join at day boundaries. Phase 1 adds SOLUSDT as proof-of-concept; Phase 2 implements combined streams in Rust; Phase 3 enables all 18 symbols.
triggers:
  - full symbol streaming
  - combined websocket
  - all symbols streaming
  - sidecar all symbols
  - midnight gap fix
  - eliminate stathera gaps
---

# Full-Symbol Streaming Spike

## Problem Statement

The sidecar only streams BTCUSDT and ETHUSDT (`deploy/sidecar.env`). The other 16 symbols
get data from kintsugi + trailing-edge backfill using independent processor instances.
At day boundaries, the two processors don't share state — creating persistent Stathera gaps
that show up in heartbeat every 5 minutes as "13 midnight-boundary gaps".

**Root cause proven 2026-03-19**: `opendeviationbar_version` tags confirm Mar 18 bars are
`+kintsugi-parquet` and Mar 19 bars are `+backfill` — two independent writers with a join gap.
BTCUSDT/ETHUSDT have **zero** midnight gaps because Puck handles them in real-time.

## Current State

### WebSocket Architecture

- **Rust `BinanceWebSocketStream`**: per-symbol connections (`wss://stream.binance.com:9443/ws/{symbol}@aggTrade`)
- **Combined streams NOT YET implemented** in code — `TODO(#279)` at `live_engine.rs:1167`
- **But architecture is validated** — the TODO says "2x9 hybrid (2 combined connections, 9 symbols each)
  delivers 4x throughput with zero errors vs 18 individual connections which suffer recv timeouts"
- Combined stream URL format: `wss://stream.binance.com:9443/stream?streams=s1@aggTrade/s2@aggTrade/...`
- **Binance limits**: 1024 streams/connection, 300 connections/5min/IP, 5 control msgs/sec, 24h max lifetime

### Capacity Concerns

1. **Ring buffer**: currently sized for 2 symbols × 4 thresholds. 18 symbols = 9x more bars
2. **Puck REST budget**: Binance REST rate limit = 2400 weight/min. 18 symbols competing for gap-fill
3. **Memory**: each symbol gets 4 processors (one per threshold). 18 × 4 = 72 processors vs current 8
4. **Batch write throughput**: ClickHouse INSERT rate with 18 symbols worth of bars
5. **Gap detector seeds**: ClickHouse queries at startup for 18 symbols' max trade IDs

## Spike Phases

### Phase 1: Quick Win — Add SOLUSDT to sidecar (per-symbol WS, no Rust changes)

**Goal**: Prove that adding one more symbol eliminates its midnight gaps.

1. Edit `deploy/sidecar.env`:

   ```
   OPENDEVIATIONBAR_STREAMING_SYMBOLS=BTCUSDT,ETHUSDT,SOLUSDT
   ```

2. Restart sidecar on bigblack:

   ```bash
   ssh bigblack 'systemctl --user restart opendeviationbar-sidecar'
   ```

3. Wait for one UTC midnight cycle (or at least several hours).

4. **Verification**: After midnight, check SOLUSDT has zero Stathera gaps at the day boundary:

   ```sql
   SELECT first_agg_trade_id - lagInFrame(last_agg_trade_id, 1) OVER w - 1 AS tid_delta
   FROM open_deviation_bars FINAL
   WHERE symbol = 'SOLUSDT'
   WINDOW w AS (PARTITION BY symbol, threshold_decimal_bps ORDER BY first_agg_trade_id)
   HAVING tid_delta > 0 AND toDate(close_time_ms/1000) >= today() - 1
   ```

5. **Success criteria**: SOLUSDT midnight gaps = 0 (same as BTCUSDT/ETHUSDT).
   Stathera gap count drops from 13 to 10 (the 3 SOLUSDT@500/750/1000 gaps disappear).

**Risk**: Adding a 3rd per-symbol WS connection is trivial. Binance allows 300 connections/5min.
This phase requires ZERO code changes — just a config change.

### Phase 2: Combined WebSocket Streams in Rust

**Goal**: Replace per-symbol WS connections with 2 combined stream connections (2×9 hybrid).

**Implementation** in `crates/opendeviationbar-providers/src/binance/websocket.rs`:

1. Add `connect_combined_stream(symbols: &[&str], trade_tx: mpsc::Sender<AggTrade>)`:
   - URL: `wss://stream.binance.com:9443/stream?streams=btcusdt@aggTrade/ethusdt@aggTrade/...`
   - Parse wrapper: `{"stream": "btcusdt@aggTrade", "data": {...}}` → extract symbol from stream name
   - Demux: set `AggTrade.symbol` from the `stream` field, send to shared `trade_tx`

2. Update `live_engine.rs` `symbol_task` → `combined_stream_task`:
   - One task handles N symbols on one WS connection
   - Demuxes trades to the correct symbol's processors
   - Gap detector per-symbol (unchanged)
   - Midnight reset per-symbol per-threshold (unchanged)

3. **Connection strategy**: 2 combined connections
   - Connection 1: 9 symbols (BTCUSDT, ETHUSDT, SOLUSDT, XRPUSDT, BNBUSDT, ADAUSDT, DOGEUSDT, AVAXUSDT, LINKUSDT)
   - Connection 2: 9 symbols (LTCUSDT, UNIUSDT, BCHUSDT, NEARUSDT, FILUSDT, WLDUSDT, SUIUSDT, WIFUSDT, AAVEUSDT)
   - Split by volume (high-volume symbols spread across connections for load balance)

4. **Reconnection**: If one combined connection drops, reconnect just that connection (9 symbols).
   The other 9 symbols continue uninterrupted. Better fault isolation than 18 individual connections.

### Phase 3: Enable All 18 Symbols

**Goal**: Deploy combined streams with all 18 symbols to bigblack.

1. Update `deploy/sidecar.env`:

   ```
   # All 18 registered symbols via 2x9 combined WS streams
   OPENDEVIATIONBAR_STREAMING_SYMBOLS=BTCUSDT,ETHUSDT,SOLUSDT,XRPUSDT,BNBUSDT,ADAUSDT,DOGEUSDT,AVAXUSDT,LINKUSDT,LTCUSDT,UNIUSDT,BCHUSDT,NEARUSDT,FILUSDT,WLDUSDT,SUIUSDT,WIFUSDT,AAVEUSDT
   ```

2. Ring buffer resize: increase from 1024 to 4096 (18 symbols × 4 thresholds × ~50 bars/flush)

3. Puck REST budget: partition REST rate limit across symbols with priority weighting
   (BTCUSDT gets more weight for gap-fill than WIFUSDT)

4. **Verification**: After midnight, ALL 18 symbols have zero Stathera gaps at day boundaries.
   Heartbeat shows "✅ bar-level continuity: clean" with zero gaps of any type.

5. **Monitoring**: Watch sidecar memory (RSS should stay under 1GB), ring buffer drops (should be 0),
   Puck fill rate (gap_trades_recovered metric).

## Anti-Patterns

- **DO NOT** use 18 individual WS connections — causes recv timeouts on low-volume symbols (#279)
- **DO NOT** increase REST ceiling beyond 3000 weight/min — Puck gap-fill needs burst capacity
- **DO NOT** remove kintsugi — it still handles historical backfill and yesterday's repairs
- **DO NOT** disable the today-guard — sidecar ownership of today is the right model

## Success Criteria (Full Spike)

- [ ] All 18 symbols streaming via 2 combined WS connections
- [ ] Zero Stathera midnight-boundary gaps after one full UTC day cycle
- [ ] Heartbeat shows "✅ bar-level continuity: clean"
- [ ] Sidecar RSS < 1GB steady state
- [ ] Ring buffer drops = 0
- [ ] Puck gap-fill working for all 18 symbols
- [ ] No Binance rate limit errors in sidecar journal

## Files to Modify

| File                                                         | Change                                              |
| ------------------------------------------------------------ | --------------------------------------------------- |
| `deploy/sidecar.env`                                         | Add symbols to `OPENDEVIATIONBAR_STREAMING_SYMBOLS` |
| `crates/opendeviationbar-providers/src/binance/websocket.rs` | Add `connect_combined_stream()`                     |
| `crates/opendeviationbar-streaming/src/live_engine.rs`       | Replace `symbol_task` with `combined_stream_task`   |
| `src/streaming_bindings.rs`                                  | Expose combined stream config to Python             |
