# Feature Research: Streaming Pipeline Zero-Gap Continuity

**Date**: 2026-03-19
**Context**: REST-to-WS junction fix + full-symbol streaming expansion (see `.planning/PROJECT.md`)

---

## 1. Junction Handling (REST-to-WS Transition)

### Table Stakes

**Same-processor continuity** -- The processor that handles REST backfill trades MUST be the same instance that handles WS trades. Creating a fresh processor at the junction destroys the forming bar's accumulated state (open price, OHLCV, microstructure accumulators). This is the root cause of the current bug: 100K trades lost at the SOLUSDT junction because `backfill_gap()` used a throwaway processor.

- **ODB status**: Fixed in v13.29 -- `fill_from_rest()` preserves processors for `start()`.
- **Complexity**: Low (already implemented). The Rust side owns processor lifecycle.
- **Dependency**: None.

**Trade-ID dedup at junction** -- WS trades buffered during REST fill overlap with the tail of REST data. The junction must deduplicate by trade ID, not by timestamp (timestamps can collide across trades). Standard pattern: maintain `last_processed_trade_id` on the processor; skip any WS trade with `agg_trade_id <= last_processed_trade_id`.

- **ODB status**: Implemented. Puck's `TradeIdGapDetector` tracks `last_agg_trade_id` per symbol.
- **Complexity**: Low. The dedup is a single integer comparison per trade.
- **Dependency**: Processor must expose `last_processed_trade_id`.

**WS buffering during REST fill** -- WS must connect and buffer trades BEFORE REST fill begins, not after. If WS connects after REST finishes, there is an unbounded gap between "REST last trade" and "first WS trade." Pattern: start WS connection, begin buffering into a bounded queue, then run REST fill. At junction, drain buffer through the same processor.

- **ODB status**: Implemented. Architecture comment at sidecar.py:840 confirms "WS buffers in parallel while REST catches up."
- **Complexity**: Medium. Requires careful ordering of async task startup.
- **Dependency**: Bounded buffer (ring buffer, currently 10K capacity via Lethe).

### Differentiators

**Forming bar snapshot at junction** -- Expose the forming bar state at the exact junction moment as a telemetry event. This lets operators verify that the forming bar was NOT abandoned. Production systems that publish forming bars to consumers (SSE push) need this to prove continuity.

- **ODB status**: Forming bar watch channels exist (1Hz via `watch::Receiver<Option<FormingBar>>`), but no specific "junction snapshot" event.
- **Complexity**: Low. Emit a tracing event with forming bar trade count + trade-ID range at junction.
- **Dependency**: Tracing infrastructure (already present).

**Checkpoint-seeded processor** -- When a checkpoint file exists from a previous session, restore the processor state (forming bar, trade history ring buffer, inter-bar lookback) before connecting WS. This avoids reprocessing historical trades entirely for the "warm restart" case. Cold start still needs REST fill.

- **ODB status**: Implemented via `create_checkpoint()` / `from_checkpoint()`. Checkpoint staleness guard at sidecar.py:301-335.
- **Complexity**: Medium. Stale checkpoint detection requires ClickHouse query.
- **Dependency**: ClickHouse availability for staleness validation.

### Anti-Features

**Dual-processor merge at junction** -- Running two processors (one for REST, one for WS) and merging their output at the junction. This creates an impossible reconciliation problem: the forming bar from the REST processor contains different accumulated state than the WS processor's first bar. The merge point is undefined.

- **Why avoid**: This was the pre-v13.29 approach. It caused 665 overlaps + 169 gaps in 10 hours (Issue #284).

**Timestamp-based dedup at junction** -- Using `close_time_ms` instead of trade IDs to deduplicate at the boundary. Timestamps are not unique across aggregated trades (multiple trades can share the same millisecond). Only trade IDs provide total ordering.

---

## 2. Combined WebSocket Streams

### Table Stakes

**Single combined connection for all symbols** -- Binance supports up to 1024 streams per combined connection and 5 connections per IP. For 18 symbols (18 `@aggTrade` streams), a single connection is well within limits. This eliminates 17 TCP handshakes, 17 TLS negotiations, and 17 independent reconnection state machines.

- **ODB status**: Not yet implemented. Current: per-symbol connections. TODO at `live_engine.rs:1209`.
- **Complexity**: Medium. Requires demultiplexing combined stream messages by symbol field (`"s": "BTCUSDT"`).
- **Dependency**: `tokio-tungstenite` combined stream URL format: `wss://stream.binance.com:9443/stream?streams=btcusdt@aggTrade/ethusdt@aggTrade/...`

**Subscription management (subscribe/unsubscribe)** -- Combined streams use JSON-RPC to manage subscriptions dynamically. The engine must handle `SUBSCRIBE`/`UNSUBSCRIBE` responses and track which streams are active. If a subscription silently drops (Binance does this under load), the engine must detect via liveness monitoring and resubscribe.

- **ODB status**: Not implemented. Per-symbol connections don't need subscription management.
- **Complexity**: Medium. JSON-RPC request/response correlation + subscription state tracking.
- **Dependency**: Argus liveness monitoring (already exists per-symbol; needs adaptation to combined stream).

### Differentiators

**2xN/2 hybrid configuration** -- Two connections, each carrying half the symbols. Advantages: (1) if one connection drops, only half the symbols go dark; (2) reconnection affects fewer symbols; (3) Binance's per-connection rate limit (5 messages/sec for subscribe/unsubscribe) is split. Empirically tested at 138.7 trades/s with zero recv timeouts.

- **ODB status**: Previous spike test showed 2x9 hybrid as best configuration.
- **Complexity**: Low incremental over single combined. Just split the symbol list.
- **Dependency**: Decision on which symbols go to which connection (volume-tiered or alphabetical).

**Volume-tiered connection assignment** -- Assign high-volume symbols (BTCUSDT, ETHUSDT: ~40-60% of total trade volume) to a dedicated connection, low-volume symbols to another. This prevents a high-volume symbol's traffic from causing backpressure on low-volume symbols' message parsing.

- **ODB status**: Not implemented. Requires trade volume profiling.
- **Complexity**: Low. Static assignment based on known volume ratios. No dynamic rebalancing needed.
- **Dependency**: Volume data (readily available from existing bar statistics).

### Anti-Features

**Dynamic connection rebalancing** -- Automatically moving symbols between connections based on real-time traffic volume. This introduces connection churn (unsubscribe from one, subscribe on another), creates brief gaps during migration, and the complexity is disproportionate to the benefit for 18 symbols.

- **Why avoid**: 18 symbols is too few to benefit. Rebalancing adds a gap-prone code path. The static 2x9 split already handles the volume skew.

**Per-symbol connections with connection pooling** -- Maintaining 18 individual connections but pooling them through a connection manager. This is strictly worse than combined streams: more TCP overhead, more reconnection state, more file descriptors, and Binance counts each connection against the 5/IP limit.

- **Why avoid**: Combined streams exist specifically to solve this problem. Per-symbol connections are the legacy approach.

---

## 3. Gap Detection at Junction

### Table Stakes

**Junction grace period** -- When REST fill completes and WS trades begin flowing, the first WS trade's `agg_trade_id` may not be `last_rest_trade_id + 1`. There is always a small gap (trades that arrived on WS while REST was fetching but were already covered by REST). Puck must NOT fire on this expected gap. Pattern: set a "junction mode" flag on the gap detector for a configurable window (e.g., 5 seconds or until the first WS trade with `agg_trade_id > last_rest_trade_id` arrives).

- **ODB status**: This is the core bug. Puck fires on the expected junction gap, disrupting the forming bar. PROJECT.md lists "Suppress Puck at junction" as a pending decision.
- **Complexity**: Medium. Requires per-symbol junction state in `TradeIdGapDetector`.
- **Dependency**: `fill_from_rest()` must communicate `last_rest_trade_id` per symbol to the gap detector.

**Per-symbol junction markers** -- Each symbol finishes REST fill at a different time. The junction suppression must be per-symbol, not global. If BTCUSDT finishes REST fill while SOLUSDT is still filling, Puck should be active for BTCUSDT (normal streaming) but suppressed for SOLUSDT (still at junction).

- **ODB status**: Not implemented. Current `fill_from_rest()` is sequential across symbols.
- **Complexity**: Medium. Per-symbol state machine: `enum JunctionState { RestFilling, JunctionTransition, NormalStreaming }`.
- **Dependency**: Parallel REST fill (so symbols finish at different times).

### Differentiators

**Structured junction event** -- Emit a machine-parseable event when each symbol transitions from REST to WS. Include: `symbol`, `last_rest_trade_id`, `first_ws_trade_id`, `gap_size` (may be 0 or negative for overlap), `rest_duration_ms`, `ws_buffer_depth`. This enables post-hoc junction quality analysis without log grep.

- **ODB status**: Not implemented. Current logging is unstructured.
- **Complexity**: Low. Single tracing span with structured fields.
- **Dependency**: Tracing (already present).

**WS-ahead detection** -- At junction, if WS has already received trades BEYOND what REST fetched (i.e., `first_ws_agg_id > last_rest_agg_id + 1`), this indicates REST was too slow and WS buffer has advanced past the junction point. In this case, the gap between REST and WS is REAL (not expected) and needs Puck or immediate REST fill for the missing range.

- **ODB status**: Partially handled by trade-ID dedup. But there is no explicit detection of this "WS ran ahead" scenario.
- **Complexity**: Medium. Compare `last_rest_trade_id` with `min(ws_buffer_trade_ids)` at junction.
- **Dependency**: Access to WS buffer contents (trade IDs) before draining.

### Anti-Features

**Global Puck disable during startup** -- Disabling Puck entirely during the first N minutes after sidecar start. This is too broad: if a real WS disconnection happens during startup, Puck should still fire to recover those trades. The suppression must be junction-specific, not time-based.

- **Why avoid**: Real gaps during startup are common (Binance disconnects happen every ~30 min on bad days). Disabling Puck leaves these unrecovered until kintsugi (next day).

**Automatic REST re-fetch if junction gap detected** -- If the junction gap is larger than expected, automatically running another REST fill to close it. This creates a recursive problem: the second REST fill has its own junction, which may also have a gap. The correct approach is: suppress Puck at the expected junction, let Puck handle any real gaps after junction, let kintsugi clean up anything Puck misses.

---

## 4. Parallel REST Backfill with Rate Limiter Partitioning

### Table Stakes

**Static budget partition** -- Divide the 6000 weight/min Binance rate limit across N concurrent REST fills. For N=3 parallel fills, each gets 2000 weight/min. For N=6, each gets 1000 weight/min. The partition is set at startup and does not change during the fill. This is simple, predictable, and avoids the thundering-herd problem where all fills exhaust the budget simultaneously.

- **ODB status**: `AdaptiveRateLimiter` exists (single shared instance). Partitioning not implemented.
- **Complexity**: Medium. Either N separate limiters each with `6000/N` budget, or a single shared limiter with `acquire(weight)` blocking semantics.
- **Dependency**: Parallelism degree N must be chosen. Constraint: each `aggTrades` request is 20 weight. At 2000 weight/min, that is 100 requests/min per symbol.

**Safety margin** -- Always reserve 20% of the rate budget for Puck gap fills that may fire during REST backfill (WS is already connected and streaming). If Puck fires while all REST budget is consumed, the gap fill fails and trades are lost. Current env: `OPENDEVIATIONBAR_REST_SAFETY_MARGIN=0.20`.

- **ODB status**: Safety margin exists in config. Not enforced during parallel fill.
- **Complexity**: Low. Reduce partition from `6000/N` to `(6000 * 0.8) / N = 4800/N`.
- **Dependency**: Puck must use the same rate limiter instance (shared `Arc<AdaptiveRateLimiter>`).

**429 feedback loop** -- When Binance returns HTTP 429, the `X-MBX-USED-WEIGHT-1M` header tells you the actual consumed weight. The rate limiter must back off to zero requests until the weight decays below the limit. This is more reliable than guessing: the header is the source of truth.

- **ODB status**: `AdaptiveRateLimiter` reads this header. Exponential backoff on 429.
- **Complexity**: Already implemented. Just ensure it works with parallel fills.
- **Dependency**: None (already in place).

### Differentiators

**Shared token bucket with priority** -- Instead of static partitions, use a single shared token bucket where Puck gap fills have HIGHER priority than REST backfill requests. When the bucket is contested, Puck requests cut the queue. This ensures real-time gap recovery is never starved by historical backfill.

- **ODB status**: Not implemented. Current limiter is FIFO.
- **Complexity**: High. Priority queuing in async context requires careful fairness guarantees.
- **Dependency**: Puck and REST fill must use the same limiter instance.

**Adaptive parallelism** -- Start with N=2 parallel fills. If 429s are rare and fills are slow, increase to N=4. If 429s spike, reduce to N=1. Adjust every 60 seconds based on the 429 rate. This maximizes throughput without manual tuning.

- **ODB status**: Not implemented.
- **Complexity**: High. Feedback loop with hysteresis to prevent oscillation.
- **Dependency**: Metrics on per-fill 429 rate.

### Anti-Features

**Per-symbol rate limiters** -- Giving each symbol its own independent rate limiter. Binance rate limits are per-IP, not per-symbol. Independent limiters will collectively exceed the global limit, causing 429s for everyone.

- **Why avoid**: The rate limit is GLOBAL. Any partitioning scheme must respect the shared ceiling.

**Unlimited parallelism** -- Running all 18 symbols in parallel. At 20 weight per request, 18 symbols making requests simultaneously consume 360 weight per round. With typical pagination (100+ requests per symbol), this exhausts the 6000 weight/min budget in under 2 minutes, triggering sustained 429s and IP-level throttling.

- **Why avoid**: Binance may escalate to IP ban after sustained 429 violations.

---

## 5. Telemetry for Streaming Pipeline Health

### Table Stakes

**Buffer depth (WS ring buffer)** -- The number of completed bars waiting in the ring buffer for Python to consume. If this grows, the Python side is too slow (backpressure). If it hits capacity (10K), Lethe fires and bars are dropped. This is the single most important streaming health metric.

- **ODB status**: `max_queue_depth` metric exists. `backpressure_events` and `dropped_bars` counted.
- **Complexity**: Already implemented.
- **Dependency**: None.

**Per-symbol trade freshness** -- Time since the last trade was received for each symbol. If a symbol goes silent for >5 minutes, either the market is closed (crypto never closes) or the WS subscription dropped. This is the liveness signal.

- **ODB status**: Argus monitors per-symbol liveness. `last_trade_time` tracked per symbol.
- **Complexity**: Already implemented.
- **Dependency**: None.

**Gap frequency and size** -- Count and size distribution of Puck gap events over time. Sudden spikes indicate WS instability (Binance disconnections) or rate limiter exhaustion (Puck can't fill fast enough). Trend analysis over hours/days reveals systemic issues.

- **ODB status**: `gap_fills` and `gap_trades_recovered` metrics exist.
- **Complexity**: Already implemented.
- **Dependency**: None.

**Junction latency** -- Wall-clock time from sidecar start to "all symbols streaming normally" (all junctions complete). This is the recovery time after a restart. Target: <5 minutes for 18 symbols. Current: ~5 minutes sequential, target <2 minutes with parallel fill.

- **ODB status**: Not explicitly measured. REST fill duration is logged but not as a structured metric.
- **Complexity**: Low. Timestamp at start, timestamp when last symbol enters `NormalStreaming`.
- **Dependency**: Per-symbol junction state machine.

### Differentiators

**Forming bar age** -- Time since the current forming bar opened (i.e., time since last completed bar). If a forming bar is "too old" (e.g., >1 hour for a 250 dbps threshold on BTCUSDT), this may indicate the processor is stuck or trades are not flowing. Context-dependent: high thresholds naturally produce long-lived forming bars.

- **ODB status**: Forming bar snapshots include `open_time_ms`. Age can be derived.
- **Complexity**: Low. Compare `now_ms - forming_bar.open_time_ms` per symbol/threshold.
- **Dependency**: Threshold-aware alerting (250 dbps bars close every ~2 min; 1000 dbps bars may last >1 hour).

**Stathera continuity score** -- A running percentage: `bars_with_perfect_continuity / total_bars * 100`. Computed over a sliding window (e.g., last 1 hour). 100% means zero gaps. <99.9% means investigation needed. This is the single number that answers "is the pipeline healthy?"

- **ODB status**: Heartbeat computes this every 5 minutes via ClickHouse query. Not available as a real-time metric from the streaming engine.
- **Complexity**: Medium. Requires tracking `last_agg_trade_id` across consecutive bars in-memory.
- **Dependency**: None (all data available in the bar output).

**Rate limiter headroom** -- Percentage of REST weight budget remaining. If consistently below 20%, Puck gap fills are at risk of being throttled. Correlates with gap recovery latency.

- **ODB status**: `AdaptiveRateLimiter.remaining()` method exists. Not exposed as a metric.
- **Complexity**: Low. Periodic sampling of `remaining() / max_weight * 100`.
- **Dependency**: None.

### Anti-Features

**Per-trade latency tracking** -- Measuring the time from Binance's trade event time (`T` field) to when the bar engine processes it. This is dominated by network latency (uncontrollable) and adds overhead to every trade (18 symbols x ~8 trades/sec = 144 comparisons/sec). The latency that matters is bar-level (junction, gap fill), not trade-level.

- **Why avoid**: Noise-to-signal ratio is very high. Network jitter creates false alerts. Bar-level metrics are sufficient.

**Full trade replay logging** -- Logging every trade that flows through the engine for debugging. At 500K+ trades/day across 18 symbols, this generates ~50MB/day of logs that are never read except during incidents. Use tracing spans with `Level::TRACE` (off by default) instead.

- **Why avoid**: Storage cost, log rotation complexity, and performance overhead. Tracing with dynamic filtering is strictly better.

**Dashboard-first telemetry** -- Building a Grafana/Prometheus dashboard before the metrics are proven useful. Premature dashboarding locks in metric names and collection patterns before they are validated in production.

- **Why avoid**: Start with structured tracing events to systemd journal. Promote to Prometheus counters only after 2+ weeks of production use proves the metric is actionable.

---

## Summary Matrix

| Feature                                  | Category       | Complexity | Status                        |
| ---------------------------------------- | -------------- | ---------- | ----------------------------- |
| Same-processor continuity                | Table stakes   | Low        | Done (v13.29)                 |
| Trade-ID dedup at junction               | Table stakes   | Low        | Done                          |
| WS buffering during REST fill            | Table stakes   | Medium     | Done                          |
| Junction grace period (Puck suppression) | Table stakes   | Medium     | **Not done -- core bug**      |
| Per-symbol junction markers              | Table stakes   | Medium     | Not done                      |
| Static rate limiter partition            | Table stakes   | Medium     | Not done                      |
| Safety margin for Puck                   | Table stakes   | Low        | Partially done                |
| 429 feedback loop                        | Table stakes   | Low        | Done                          |
| Buffer depth metric                      | Table stakes   | Low        | Done                          |
| Per-symbol trade freshness               | Table stakes   | Low        | Done                          |
| Gap frequency/size metrics               | Table stakes   | Low        | Done                          |
| Junction latency metric                  | Table stakes   | Low        | Not done                      |
| Single/dual combined WS connection       | Table stakes   | Medium     | Not done                      |
| Subscription management                  | Table stakes   | Medium     | Not done                      |
| Forming bar snapshot at junction         | Differentiator | Low        | Not done                      |
| Checkpoint-seeded processor              | Differentiator | Medium     | Done                          |
| 2xN/2 hybrid WS config                   | Differentiator | Low        | Spike-tested, not implemented |
| Volume-tiered connection assignment      | Differentiator | Low        | Not done                      |
| Structured junction event                | Differentiator | Low        | Not done                      |
| WS-ahead detection                       | Differentiator | Medium     | Not done                      |
| Shared token bucket with priority        | Differentiator | High       | Not done                      |
| Adaptive parallelism                     | Differentiator | High       | Not done                      |
| Forming bar age metric                   | Differentiator | Low        | Derivable                     |
| Stathera continuity score (real-time)    | Differentiator | Medium     | Heartbeat only                |
| Rate limiter headroom metric             | Differentiator | Low        | Not exposed                   |
| Dual-processor merge                     | Anti-feature   | --         | Proven harmful                |
| Timestamp-based dedup                    | Anti-feature   | --         | Incorrect                     |
| Dynamic connection rebalancing           | Anti-feature   | --         | Over-engineered               |
| Per-symbol connections (legacy)          | Anti-feature   | --         | Current, being replaced       |
| Global Puck disable during startup       | Anti-feature   | --         | Too broad                     |
| Auto REST re-fetch on junction gap       | Anti-feature   | --         | Recursive problem             |
| Per-symbol rate limiters                 | Anti-feature   | --         | Violates global limit         |
| Unlimited parallelism                    | Anti-feature   | --         | Triggers IP ban               |
| Per-trade latency tracking               | Anti-feature   | --         | Noise                         |
| Full trade replay logging                | Anti-feature   | --         | Storage waste                 |
| Dashboard-first telemetry                | Anti-feature   | --         | Premature                     |

---

## Implementation Priority

The **critical path** for the PROJECT.md deliverables:

1. **Junction grace period** (Puck suppression) -- eliminates the root cause bug
2. **Per-symbol junction markers** -- required for parallel REST fill
3. **Combined WS streams** (2x9 hybrid) -- eliminates midnight gap class for 16 symbols
4. **Static rate limiter partition** -- enables parallel REST fill
5. **Junction latency metric** -- validates the fix works in production

Everything else is either already done, derivable from existing infrastructure, or deferrable.
