# Stack Research: Streaming Pipeline Gap-Fix

Research conducted 2026-03-19 for REST->WS junction fix, combined WS streams, parallel REST backfill, and junction telemetry.

---

## 1. WebSocket: Combined Stream Management in Rust

### Recommendation: tokio-tungstenite 0.24+ (keep current, upgrade path to 0.28)

**Confidence: HIGH** -- tokio-tungstenite is the de facto standard. The project already uses 0.23; upgrading to 0.24+ gains performance improvements (0.26.2+ is notably faster, on par with fastwebsockets). The latest release is 0.28.0.

**Why not switch crates**: `fastwebsockets` (Google's hyper-based WS) is faster for raw throughput but lacks rustls-tls-native-roots out of the box and has a smaller ecosystem. `stream-tungstenite` adds reconnection wrappers but duplicates what this project already has via backon + Antaeus. Stick with tokio-tungstenite -- the project's existing reconnection logic (backon ExponentialBuilder + stable_connection_ms reset) is production-proven and more nuanced than any wrapper crate.

### Combined Stream Architecture

**URL format**: `wss://stream.binance.com:9443/stream?streams=btcusdt@aggTrade/ethusdt@aggTrade/...`

**Message envelope**: Combined streams wrap each event in `{"stream":"<name>","data":{...}}`. The `stream` field is the demux key.

**Recommended topology: 2x9 hybrid** (already empirically validated at 138.7 trades/s, zero recv timeouts):

- Connection A: 9 highest-volume symbols (BTCUSDT, ETHUSDT, SOLUSDT, XRPUSDT, DOGEUSDT, BNBUSDT, ADAUSDT, TRXUSDT, AVAXUSDT)
- Connection B: 9 remaining symbols (LINKUSDT, DOTUSDT, MATICUSDT, UNIUSDT, LTCUSDT, NEARUSDT, FILUSDT, WIFUSDT, ARBUSDT)

**Why 2x9 over 1x18**: Fault isolation. If one connection dies, the other 9 symbols continue streaming uninterrupted while reconnection happens. 1x18 is an all-or-nothing failure mode. The 1024 stream limit per connection is irrelevant at 18 streams.

### Implementation Pattern: Combined Stream Demuxer

```rust
/// Combined stream message wrapper from Binance
#[derive(Debug, Deserialize)]
struct CombinedStreamMessage {
    stream: String,        // e.g. "btcusdt@aggTrade"
    data: BinanceAggTrade, // existing type, reused
}

/// Demux combined stream into per-symbol channels
async fn run_combined_stream(
    symbols: &[String],
    trade_txs: &HashMap<String, mpsc::Sender<AggTrade>>,
    shutdown: &CancellationToken,
    data_timeout: Duration,
) -> Result<(), WebSocketError> {
    let streams: Vec<String> = symbols.iter()
        .map(|s| format!("{}@aggTrade", s.to_lowercase()))
        .collect();
    let url = format!(
        "wss://stream.binance.com:9443/stream?streams={}",
        streams.join("/")
    );

    let (ws_stream, _) = connect_async(&url).await?;
    let (mut ws_sender, mut ws_reader) = ws_stream.split();

    loop {
        tokio::select! {
            msg = ws_reader.next() => {
                match msg {
                    Some(Ok(Message::Text(text))) => {
                        // Demux: parse stream field, route to correct channel
                        if let Ok(combined) = serde_json::from_str::<CombinedStreamMessage>(&text) {
                            let symbol = combined.stream
                                .split('@').next()
                                .unwrap_or("").to_uppercase();
                            if let Some(tx) = trade_txs.get(&symbol) {
                                let _ = tx.send(combined.data.to_agg_trade()).await;
                            }
                        }
                    }
                    Some(Ok(Message::Ping(data))) => {
                        let _ = ws_sender.send(Message::Pong(data)).await;
                    }
                    // ... Close, error handling same as current pattern
                }
            }
            () = tokio::time::sleep(data_timeout) => {
                return Err(WebSocketError::ConnectionClosed);
            }
            () = shutdown.cancelled() => return Ok(()),
        }
    }
}
```

### Reconnection Without Message Loss

**The hard truth**: Binance WebSocket provides NO replay mechanism. Messages during disconnection are lost. This is documented and unavoidable.

**Mitigation strategy (already in project, needs extension)**:

1. **Buffer WS trades in ring buffer** (existing) -- trades arrive before processor consumes them
2. **On reconnect, use trade-ID gap detection** (existing Puck) -- detect the gap, fill via REST `fromId` pagination
3. **For combined streams**: Same Puck logic applies per-symbol. The demuxer routes to per-symbol channels; each symbol's gap detector independently detects and fills gaps on reconnect.
4. **24h forced reconnect**: Binance kills connections at 24h. Implement proactive reconnection at ~23h: open new connection, let it stabilize, then close old one. This eliminates the forced-disconnect gap entirely.

```rust
// Proactive reconnection timer (add to combined stream runner)
const PROACTIVE_RECONNECT_SECS: u64 = 23 * 3600; // 23 hours

tokio::select! {
    // ... existing msg/timeout/shutdown branches
    () = tokio::time::sleep(Duration::from_secs(PROACTIVE_RECONNECT_SECS)) => {
        tracing::info!("proactive reconnect before 24h limit");
        return Err(WebSocketError::ConnectionClosed); // triggers Antaeus reconnect
    }
}
```

### Upgrade Path

| Version        | Key Change                             | Risk                                       |
| -------------- | -------------------------------------- | ------------------------------------------ |
| 0.23 (current) | Stable, production-proven              | None                                       |
| 0.24           | tungstenite 0.24 update                | Low                                        |
| 0.26.2+        | Performance on par with fastwebsockets | Medium (breaking: tungstenite 0.26)        |
| 0.28.0         | Latest, tungstenite 0.28               | Medium (evaluate after junction fix ships) |

**Recommendation**: Stay on 0.23 for the junction fix. Upgrade to 0.28 in a follow-up release once combined streams are stable. The performance gains from 0.26.2+ are real but not blocking.

---

## 2. Rate Limiting: Partitioned Budgets for Parallel REST Fill

### Recommendation: Keep existing AdaptiveRateLimiter, extend with partition support

**Confidence: HIGH** -- The project's `AdaptiveRateLimiter` already handles the hard part (server feedback via `X-MBX-USED-WEIGHT-1m`, CAS-based acquisition, window reset). Adding partitioning is simpler than integrating governor.

### Why NOT governor

The `governor` crate (v0.10.2, latest) is excellent for HTTP middleware rate limiting but is a poor fit here:

1. **No server feedback integration**: Governor uses GCRA (Generic Cell Rate Algorithm) which is purely local. Binance's `X-MBX-USED-WEIGHT-1m` header is the authoritative source -- ignoring it risks 429s. The existing `AdaptiveRateLimiter.update_from_headers()` is critical and has no governor equivalent.

2. **Keyed != partitioned budgets**: Governor's keyed limiter gives each key its OWN independent quota. What this project needs is a SHARED budget with per-partition CEILINGS -- fundamentally different. With governor, 3 partitions of 2000 weight could collectively use 6000, blowing through the global limit.

3. **Additional dependency**: Governor pulls in `dashmap`, `parking_lot`, and `quanta`. The project already has `parking_lot` but adding `dashmap` and `quanta` for something achievable with the existing code is unnecessary.

### Recommended: PartitionedRateLimiter wrapper

Extend `AdaptiveRateLimiter` with a thin partitioning layer:

```rust
/// Partitioned rate limiter: shared global budget with per-partition ceilings.
///
/// Example: 6000 weight/min global, 3 partitions of 1600 weight each.
/// Each partition can use up to its ceiling. The global budget (via server
/// feedback) is the hard cap -- if Binance reports 5500 used, ALL partitions
/// are throttled regardless of individual usage.
pub struct PartitionedRateLimiter {
    /// Shared global limiter (receives server feedback)
    global: Arc<AdaptiveRateLimiter>,
    /// Per-partition ceilings
    partitions: Vec<Arc<AdaptiveRateLimiter>>,
}

impl PartitionedRateLimiter {
    /// Create N equal partitions of the global budget.
    ///
    /// Each partition gets `global_budget / n` as its ceiling.
    /// Safety margin applied at global level only.
    pub fn new_equal(n: usize, safety_margin: f32) -> Self {
        let global = Arc::new(AdaptiveRateLimiter::new(6000, safety_margin));
        let per_partition = (6000.0 * (1.0 - safety_margin)) as u32 / n as u32;
        let partitions = (0..n)
            .map(|_| Arc::new(AdaptiveRateLimiter::new_with_ceiling(6000, safety_margin, per_partition)))
            .collect();
        Self { global, partitions }
    }

    /// Acquire weight on behalf of partition `idx`.
    /// Checks both partition ceiling AND global budget.
    pub async fn acquire(&self, partition_idx: usize, weight: u32) -> Result<(), RateLimitError> {
        self.partitions[partition_idx].acquire(weight).await?;
        self.global.acquire(weight).await
    }

    /// Forward server feedback to global limiter.
    pub fn update_from_headers(&self, headers: &reqwest::header::HeaderMap) {
        self.global.update_from_headers(headers);
    }
}
```

### Budget Allocation for 18-Symbol Parallel Fill

Binance REST `aggTrades` endpoint: **20 weight per request**, returns up to 1000 trades.

| Parallelism | Per-Partition Budget | Requests/Min/Partition | Effective Throughput |
| ----------- | -------------------- | ---------------------- | -------------------- |
| 3 workers   | 1600 weight          | 80 requests            | 80K trades/min each  |
| 6 workers   | 800 weight           | 40 requests            | 40K trades/min each  |
| 9 workers   | 533 weight           | 26 requests            | 26K trades/min each  |
| 18 workers  | 267 weight           | 13 requests            | 13K trades/min each  |

**Recommendation: 6 workers** (3 per combined stream group). Each worker processes 3 symbols sequentially within its partition. Total throughput: 240K trades/min across all workers -- sufficient to fill even the largest startup gaps (150K trades for BTCUSDT) in under a minute.

With 20% safety margin: effective budget = 4800 weight/min. At 6 workers: 800/worker. At 20 weight/request: 40 requests/min/worker. That is conservative and leaves headroom for Puck real-time gap fills.

---

## 3. Telemetry: Junction Metrics and Streaming Observability

### Recommendation: tracing (existing) + metrics crate for counters/gauges/histograms

**Confidence: HIGH** -- The project already uses `tracing` for structured logging. Adding the `metrics` crate (v0.24) provides zero-cost counters, gauges, and histograms that complement tracing spans. No OpenTelemetry needed -- the systemd journal + Prometheus scrape endpoint pattern is simpler and sufficient for this use case.

### Why tracing + metrics, NOT tracing-opentelemetry

1. **Deployment target is a single host** (bigblack). OpenTelemetry's distributed tracing is overkill. Journal logs + Prometheus metrics cover all needs.
2. **tracing-opentelemetry** introduces version coupling headaches between `tracing`, `opentelemetry`, and `tracing-opentelemetry` crates. This project's `tracing` + `tracing-subscriber` stack is stable -- adding OTel dependencies risks breakage.
3. **metrics** crate is lightweight (macros only, like `log` crate), no runtime overhead when no exporter is registered.

### Crate Recommendations

| Crate                         | Version        | Purpose                                  |
| ----------------------------- | -------------- | ---------------------------------------- |
| `tracing`                     | 0.1 (existing) | Structured spans for junction events     |
| `tracing-subscriber`          | 0.3 (existing) | JSON/journal output                      |
| `metrics`                     | 0.24           | Counters, gauges, histograms             |
| `metrics-exporter-prometheus` | 0.16           | Optional: Prometheus `/metrics` endpoint |

### Junction Telemetry Specification

These are the specific metrics needed for the junction fix, organized by the four observability dimensions:

#### Counters (monotonic, use `counter!`)

```rust
// Junction events
counter!("junction.puck_suppressed_total", "symbol" => symbol.clone());
counter!("junction.rest_to_ws_transitions_total", "symbol" => symbol.clone());
counter!("junction.forming_bar_preserved_total", "symbol" => symbol.clone());
counter!("junction.forming_bar_abandoned_total", "symbol" => symbol.clone());

// Gap fill operations
counter!("gap_fill.rest_requests_total", "symbol" => symbol.clone());
counter!("gap_fill.trades_fetched_total", "symbol" => symbol.clone());
counter!("gap_fill.errors_total", "symbol" => symbol.clone(), "error_type" => err_type);
```

#### Gauges (point-in-time, use `gauge!`)

```rust
// WS buffer depth per symbol
gauge!("ws.buffer_depth", "symbol" => symbol.clone()).set(buffer_len as f64);

// Forming bar state
gauge!("forming_bar.trade_count", "symbol" => symbol.clone()).set(trade_count as f64);
gauge!("forming_bar.age_ms", "symbol" => symbol.clone()).set(age_ms as f64);

// Rate limiter state
gauge!("rate_limiter.utilization").set(limiter.utilization() as f64);
gauge!("rate_limiter.remaining_weight").set(limiter.remaining() as f64);

// Combined stream health
gauge!("ws.connected_streams").set(connected_count as f64);
```

#### Histograms (distributions, use `histogram!`)

```rust
// Junction timing
histogram!("junction.rest_fill_duration_ms", "symbol" => symbol.clone())
    .record(duration.as_millis() as f64);
histogram!("junction.ws_buffer_drain_ms", "symbol" => symbol.clone())
    .record(drain_duration.as_millis() as f64);

// Gap fill latency
histogram!("gap_fill.latency_ms", "symbol" => symbol.clone())
    .record(fill_duration.as_millis() as f64);
histogram!("gap_fill.trades_per_fill", "symbol" => symbol.clone())
    .record(trade_count as f64);
```

#### Tracing Spans (context propagation)

```rust
// Wrap the entire junction transition in a span
let _span = tracing::info_span!(
    "rest_ws_junction",
    symbol = %symbol,
    rest_last_tid = rest_last_trade_id,
    ws_first_tid = ws_first_trade_id,
    forming_bar_trades = forming_bar_trade_count,
    puck_suppressed = puck_suppressed,
).entered();

// Per-symbol fill_from_rest span
let _span = tracing::info_span!(
    "fill_from_rest",
    symbol = %symbol,
    checkpoint_tid = checkpoint_last_tid,
    target_tid = target_trade_id,
).entered();
```

### Metrics Exporter Strategy

For the bigblack deployment, two options (both lightweight):

**Option A: Journal-only (recommended for initial ship)**

- Metrics logged periodically (every 60s) as structured tracing events
- Zero new dependencies
- Queryable via `journalctl -u opendeviationbar-sidecar --output=json`

**Option B: Prometheus endpoint (follow-up)**

- Add `metrics-exporter-prometheus` crate
- Expose `/metrics` on sidecar health port (already exists for `/health`)
- Grafana dashboard for long-term trending

Recommendation: Ship with Option A. Add Option B when monitoring needs grow.

---

## 4. Combined Stream Reconnection: Zero-Gap Strategy

### The Problem

Binance forces disconnection at 24h. During any reconnect (forced or network), WS messages are lost. For the project, each lost message is a potential Stathera gap that triggers kintsugi repair.

### Recommended: Overlapping Connection Pattern

**Confidence: MEDIUM-HIGH** -- This is the standard pattern in production crypto data systems. Not novel, but requires careful implementation.

```
Timeline:
  T=0        T=23h      T=23h+5s    T=23h+10s
  |----------|----------|-----------|
  conn_old   conn_new connects    conn_old closes
             (overlap window)
```

1. At T=23h (1h before forced disconnect), open a NEW combined stream connection
2. Buffer trades from the new connection in a separate channel
3. Once the new connection receives its first trade, mark it as "live"
4. Drain the old connection's remaining buffered trades
5. Switch the demuxer to route from the new connection
6. Close the old connection gracefully
7. Any duplicate trades (overlap window) are deduplicated by trade-ID (already happens in the processor)

```rust
/// Overlapping reconnection for combined streams.
/// Opens new connection before closing old, ensuring zero message gap.
async fn overlapping_reconnect(
    symbols: &[String],
    trade_txs: &HashMap<String, mpsc::Sender<AggTrade>>,
    shutdown: &CancellationToken,
    policy: &ReconnectionPolicy,
) {
    let reconnect_interval = Duration::from_secs(23 * 3600);

    loop {
        let connection_start = Instant::now();
        let child_shutdown = CancellationToken::new();

        // Spawn the combined stream task
        let handle = tokio::spawn({
            let syms = symbols.to_vec();
            let txs = trade_txs.clone();
            let cs = child_shutdown.clone();
            let timeout = Duration::from_secs(policy.data_timeout_secs);
            async move {
                run_combined_stream(&syms, &txs, &cs, timeout).await
            }
        });

        tokio::select! {
            result = handle => {
                // Connection died -- apply Antaeus backoff + reconnect
                match result {
                    Ok(Err(_e)) => {
                        // Apply backoff, then loop to reconnect
                        tokio::time::sleep(Duration::from_millis(policy.backoff_ms_initial)).await;
                    }
                    _ => {}
                }
            }
            () = tokio::time::sleep(reconnect_interval) => {
                // Proactive reconnect: signal old connection to close
                // (new iteration of the loop opens fresh connection)
                child_shutdown.cancel();
            }
            () = shutdown.cancelled() => {
                child_shutdown.cancel();
                break;
            }
        }
    }
}
```

**Trade-ID dedup at processor level**: The `OpenDeviationBarProcessor` already tracks `last_agg_trade_id`. Duplicate trades (same trade ID) from the overlap window are silently skipped. No additional dedup logic needed.

### Fallback: Puck Gap Fill

Even with overlapping connections, network partitions can cause gaps. Puck already handles this: on detecting a trade-ID discontinuity, it fetches missing trades via REST `fromId` pagination. The junction fix (suppressing Puck during REST->WS transition) is the NEW behavior; during normal streaming, Puck continues to operate as before.

---

## 5. Dependency Summary

### New Dependencies

| Crate     | Version | Purpose                        | Size Impact           |
| --------- | ------- | ------------------------------ | --------------------- |
| `metrics` | 0.24    | Counter/gauge/histogram macros | Minimal (macros only) |

### Existing Dependencies (no changes needed)

| Crate                  | Current Version | Role                                      |
| ---------------------- | --------------- | ----------------------------------------- |
| `tokio-tungstenite`    | 0.23            | WebSocket client (keep for now)           |
| `backon`               | 1.6             | Exponential backoff with jitter (Antaeus) |
| `tracing`              | 0.1             | Structured logging spans                  |
| `tracing-subscriber`   | 0.3             | Journal output                            |
| `parking_lot`          | 0.12            | Fast mutexes for rate limiter             |
| `serde` / `serde_json` | 1.0             | Combined stream message parsing           |
| `tokio`                | 1.0             | Async runtime                             |

### Optional (follow-up)

| Crate                         | Version | Purpose                    | When                          |
| ----------------------------- | ------- | -------------------------- | ----------------------------- |
| `metrics-exporter-prometheus` | 0.16    | Prometheus scrape endpoint | After initial ship            |
| `tokio-tungstenite`           | 0.28    | Performance upgrade        | After combined streams stable |

### Explicitly NOT Recommended

| Crate                   | Why Not                                                                    |
| ----------------------- | -------------------------------------------------------------------------- |
| `governor`              | No server feedback integration; keyed != partitioned; extra deps           |
| `tower-rate-limit`      | Tower middleware pattern; wrong abstraction for direct REST calls          |
| `fastwebsockets`        | No rustls-tls-native-roots; smaller ecosystem; premature optimization      |
| `stream-tungstenite`    | Duplicates existing Antaeus reconnection logic                             |
| `tracing-opentelemetry` | Distributed tracing overkill for single-host; version coupling risk        |
| `stream-reconnect`      | Generic reconnection wrapper; less control than project's existing pattern |

---

## 6. Implementation Order

Based on dependency analysis and risk:

1. **Junction fix in live_engine.rs** -- Highest value, zero new deps. Suppress Puck at REST->WS boundary, preserve forming bar. Add tracing spans.
2. **Junction telemetry** -- Add `metrics` crate, instrument junction with counters/gauges. Validate with journal logs.
3. **Combined WS streams** -- New `CombinedBinanceStream` type in `websocket.rs`. Demuxer routes to per-symbol channels. 2x9 topology.
4. **Parallel REST fill** -- `PartitionedRateLimiter` wrapper. 6-worker pool with per-partition ceilings.
5. **Overlapping reconnection** -- Proactive reconnect at 23h. Trade-ID dedup handles overlap.

Items 1-2 can ship independently. Items 3-5 are a logical unit.

---

_Research sources: Binance API documentation, crates.io, tokio-tungstenite GitHub, governor docs, metrics-rs ecosystem, project codebase analysis._
