# Research Summary: Streaming Pipeline Zero-Gap Continuity

**Date**: 2026-03-19
**Sources**: STACK.md, FEATURES.md, ARCHITECTURE.md, PITFALLS.md

---

## 1. Key Findings

- **The junction bug is the root cause of all startup gaps.** Puck fires on the expected REST-to-WS transition gap, disrupting forming bars with 100K+ accumulated trades. Fix is ~15 lines in `symbol_task` before the main loop -- zero hot-path impact. Proven on SOLUSDT@500: 149,405 REST trades, 99,882-trade gap from Puck misfiring.

- **Binance delivers perfectly -- all gaps are ours.** Independent WS monitor (90K+ trades, zero gaps) proved trade-ID continuity is flawless from the source. Every Stathera gap traces to our pipeline: ring buffer overflow, processor state loss, restart transitions, or connection drops.

- **16 of 18 symbols have persistent midnight join gaps because they are not streamed.** Only BTCUSDT/ETHUSDT use live WS; the rest rely on kintsugi/backfill joins which always create day-boundary gaps. Combined WS streams (2x9 hybrid) eliminates this entire gap class.

- **The existing `AdaptiveRateLimiter` is sufficient -- governor and tower-rate-limit are wrong abstractions.** Governor's keyed limiters give per-key independent quotas (wrong: Binance enforces per-IP global budget). The project's server-feedback integration (`X-MBX-USED-WEIGHT-1m`) is critical and has no governor equivalent. Only a dynamic ceiling addition (`set_ceiling()`, 3 lines) is needed.

- **WS channel buffers overflow during sequential REST fill.** BTCUSDT at 50 trades/sec fills the 1000-capacity mpsc channel in 20 seconds. Sequential 18-symbol fill takes ~5 minutes, causing WS connections to die for early-processed symbols. Parallel fill (4 workers, ~75s total) plus increased channel capacity (50K) eliminates this.

- **Single-processor continuity is non-negotiable.** Asclepius cascade (Issue #284: 665 overlaps + 169 gaps in 10h) and the `reconstruct_today()` OOM (v13.26: 2.5GB) both proved that separate processors for gap-fill and buffering-all-today are fatal patterns. Every gap-fill must use the same processor instance.

- **Overlapping WS reconnection eliminates the 24h forced-disconnect gap.** Open new connection at 23h, let it stabilize, close old one. Trade-ID dedup at the processor level handles the overlap window with zero additional logic.

---

## 2. Recommended Stack

| Component            | Crate/Tool                       | Version        | Notes                                                         |
| -------------------- | -------------------------------- | -------------- | ------------------------------------------------------------- |
| WebSocket            | `tokio-tungstenite`              | 0.23 (current) | Stay for junction fix; upgrade to 0.28 after combined streams |
| Reconnection backoff | `backon`                         | 1.6 (current)  | ExponentialBuilder + jitter (Antaeus)                         |
| Structured logging   | `tracing` + `tracing-subscriber` | 0.1 / 0.3      | Existing; add junction spans                                  |
| Metrics              | `metrics`                        | 0.24 (new)     | Counters/gauges/histograms; macros-only, near-zero overhead   |
| Metrics export       | Journal-only (initial)           | --             | Periodic structured tracing events; Prometheus follow-up      |
| Rate limiting        | `AdaptiveRateLimiter` (extend)   | --             | Add `set_ceiling()` (AtomicU32); no new crate                 |
| Parallelism          | `tokio::task::JoinSet`           | tokio 1.0      | Bounded by Semaphore(4) for fill_from_rest                    |
| Async runtime        | `tokio`                          | 1.0 (current)  | No changes                                                    |
| Serialization        | `serde` + `serde_json`           | 1.0 (current)  | Combined stream envelope parsing                              |

**Explicitly rejected**: `governor` (no server feedback, keyed != partitioned), `fastwebsockets` (no rustls-tls-native-roots), `tracing-opentelemetry` (distributed tracing overkill for single host), `stream-tungstenite` (duplicates Antaeus).

---

## 3. Architecture Decisions

### Junction Fix

- **Location**: `symbol_task()` in `live_engine.rs`, between processor reuse and main `loop { select! {} }`
- **Approach**: Reuse `puck_fill()` directly for the junction gap (same mechanics, no new abstraction). Peek first WS trade, compute junction gap from processor's `last_agg_trade_id`, fill via REST through the SAME processor, then re-seed gap detector past the junction. ~15 lines, zero new state in the hot path.

### Combined WebSocket Streams

- **Approach**: New `CombinedWebSocketStream` struct (not a mode in existing `BinanceWebSocketStream`) -- architecturally distinct (1:N demux vs 1:1). Outputs to per-symbol `mpsc::Sender<AggTrade>` channels; `symbol_task` interface unchanged.
- **Topology**: 2x9 hybrid (volume-tiered). Connection A: 9 highest-volume symbols. Connection B: 9 remaining. Fault isolation: one connection drop affects 9 symbols, not 18.
- **Reconnection**: Overlapping connection pattern at 23h. Stagger the two connections' reconnection by 30 minutes so they never reconnect simultaneously.

### Parallel fill_from_rest

- **Approach**: Rust-side `JoinSet` bounded by `Semaphore(4)`. Processors are Rust-owned; crossing PyO3 boundary per trade is prohibitive. Each symbol's chunks processed sequentially within its task; 4 symbols fill concurrently.
- **Timing**: ~75 seconds total (vs ~5 min sequential). WS buffer window shrinks from 5 min to <90s.

### Rate Limiter

- **Approach**: Single shared `AdaptiveRateLimiter` with dynamic ceiling. During `fill_from_rest`: ceiling = 4800 (full budget, Puck inactive). After start: ceiling = 3000 (Puck + kintsugi headroom). One-line change: `ceiling: u32` to `ceiling: AtomicU32`.
- **No partitioning needed**: Parallel fills naturally compete on the shared Arc limiter. Server feedback via `X-MBX-USED-WEIGHT-1m` is the hard cap.

### Telemetry

- **Approach**: Journal-first (Option A). Metrics logged every 60s as structured tracing events. Zero new dependencies beyond `metrics` crate. Prometheus endpoint (Option B) deferred until after 2+ weeks of production use.
- **Key metrics**: Junction gap size, forming bar trade count at junction, WS buffer depth, fill duration, rate limiter utilization, Stathera continuity score.

---

## 4. Build Order

| Phase | Name                     | Effort     | Dependencies    | Rationale                                                                                           |
| ----- | ------------------------ | ---------- | --------------- | --------------------------------------------------------------------------------------------------- |
| 1     | Junction Fix             | ~15 lines  | None            | Root cause bug. Independently deployable. Eliminates forming-bar abandonment for all 18 symbols.    |
| 2     | Junction Telemetry       | ~20 lines  | Phase 1         | Validates Phase 1 in production. Provides data for Phase 3-5 tuning decisions.                      |
| 3     | Rate Limiter Ceiling     | ~10 lines  | None            | Prerequisite for Phase 4. AtomicU32 swap + `set_ceiling()` method.                                  |
| 4     | Parallel fill_from_rest  | ~40 lines  | Phase 3         | Reduces startup from ~5 min to ~75s. Eliminates WS buffer overflow (P1.2). Increase channel to 50K. |
| 5     | Combined WS Streams      | ~250 lines | None (parallel) | Eliminates midnight join gaps for 16 symbols. 2x9 hybrid with overlapping reconnection.             |
| 6     | Clean-Slate Deploy + 24h | Ops        | All above       | Backup today's data, clear checkpoints, deploy, monitor 24h. Rollback script prepared.              |

**Parallelism**: Phases 1-3 in a single session (~45 lines total). Phase 5 developed in parallel with Phases 1-4 (independent code path). Phase 4 immediately after Phase 3. Phase 6 after all code ships.

---

## 5. Top Risks

| #   | Risk                                                 | Severity | Mitigation                                                                                                                                                       |
| --- | ---------------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1   | **Puck fires at junction gap (P1.1)**                | Critical | Re-seed gap detector AFTER `fill_from_rest` with processor's `last_agg_trade_id`, BEFORE main WS loop. ~15 lines in `symbol_task`.                               |
| 2   | **Combined stream message format (P2.2)**            | High     | New `CombinedBinanceMessage` struct for `{"stream":"...","data":{...}}` envelope. Without this parser, combined streams produce zero bars. Blocking for Phase 5. |
| 3   | **Rate limit exhaustion from parallel fills (P3.1)** | High     | Bound parallelism to 4 via Semaphore. Dynamic ceiling: 4800 during fill, 3000 after. Safety margin reserves 20% for Puck. Monitor `X-MBX-USED-WEIGHT-1m`.        |
| 4   | **Combined stream reconnect kills 9 symbols (P2.1)** | Medium   | Puck fills each symbol's gap independently. Jitter Puck REST fetches (0-500ms) to prevent thundering herd. Stagger 2x9 reconnection by 30 min.                   |
| 5   | **Clean-slate deploy with no rollback (P4.1)**       | Medium   | Backup today's bars to `_backup` table before delete. Export checkpoints to `/tmp/`. Prepared rollback SQL. Deploy during UTC 00:00-02:00 low-volume window.     |

---

## 6. Open Questions

1. **WS channel capacity**: Current 1000 overflows during sequential fill. Recommended 50,000 (~10MB/symbol, ~180MB total). Acceptable? Or should we use unbounded channels with Lethe-style overflow detection?

2. **Combined stream group assignment**: Volume-tiered (recommended) or alphabetical? Need to confirm current volume ratios for the 18 symbols to determine the optimal 9/9 split.

3. **Puck grace period for combined streams**: Cross-symbol message ordering on combined streams may cause spurious gap detections on low-volume symbols (P2.3). Should the gap detector wait for 2-3 trades before confirming a gap < 10? Needs spike-testing with 2x9 config to measure false-positive rate.

4. **fill_from_rest day boundary guard**: If a symbol's last checkpoint is 3 days old, `fill_from_rest` fetches trades from kintsugi's domain (yesterday and older). Should we clamp to `max(today_start_ms, checkpoint_tid)` and let kintsugi handle older gaps? This aligns with writer ownership model (#280) but may leave a gap from yesterday's close to today's open.

5. **Kintsugi concurrent REST consumption**: Kintsugi's `HistoricalDataLoader` has no rate limiter (Issue #291). If kintsugi runs during sidecar startup, both consume from the same 6000 weight/min IP budget. Should `fill_from_rest` check if kintsugi is active and reduce its ceiling accordingly? Or is the writer ownership model (kintsugi = yesterday, sidecar = today) sufficient temporal separation?

6. **tokio-tungstenite upgrade timing**: 0.26.2+ matches fastwebsockets performance. Upgrade after combined streams are stable (Phase 5+1 release)? Or keep 0.23 indefinitely since it is production-proven?
