# Pitfalls: Streaming Pipeline Junction Fix + Full-Symbol Streaming

**Date**: 2026-03-19
**Scope**: Junction fix, combined WS streams, parallel backfill, deployment, testing
**Method**: Codebase analysis + project incident history (14 months of production data)

---

## 1. Junction Fix Pitfalls

### P1.1 Puck Fires at the Expected Junction Gap

**What happens**: After `fill_from_rest()` completes for a symbol, the WS buffer has been accumulating trades. The first WS trade's `agg_trade_id` is higher than REST's last trade. `TradeIdGapDetector.check()` (line 1378, `live_engine.rs`) detects a gap and triggers `puck_fill()`, which fetches REST trades that overlap with trades the processor already consumed during `fill_from_rest()`. The processor's forming bar (potentially holding 100K+ accumulated trades) gets disrupted.

**Warning signs**: `gap_fills` metric spikes at startup. Tracing shows "puck: filling gap via parallel REST API" immediately after "reusing pre-created processors from fill_from_rest()". Multi-hour data gaps in completed bars despite successful `fill_from_rest()`.

**Prevention**: After `fill_from_rest()` completes, set the gap detector's `last_known_tid` to the processor's `last_agg_trade_id()`. This suppresses Puck for the expected junction gap. The gap detector is seeded at line 1261-1269 from `max(checkpoint_tid, ch_seed_tid)`, but after `fill_from_rest()` processes trades, the processor's tid is higher than either seed. The gap detector must be re-seeded AFTER fill completes, BEFORE the main WS loop starts.

**Phase**: Phase 1 (Junction Fix). This is the core bug -- approximately 20-30 lines in `symbol_task`, between processor reuse (line 670) and the main `loop { select! { ... } }` (line 1285).

**Historical precedent**: Proven on 2026-03-19 with SOLUSDT@500: 149,405 trades processed by REST, forming bar with ~100K trades, Puck fired at junction, 99,882-trade gap created.

---

### P1.2 WS Buffer Overflow During REST Fill

**What happens**: The WS `mpsc::channel` has capacity 1000 (line 1242, `live_engine.rs`). For BTCUSDT at peak volume (~50 trades/sec), the buffer fills in ~20 seconds. `fill_from_rest()` processes symbols SEQUENTIALLY -- at 18 symbols, the last symbol won't start REST fill for ~5 minutes. By then, early symbols' WS buffers have been full for 4+ minutes, silently dropping trades (mpsc bounded send blocks, but the WS sender uses `send().await` which blocks the WS reader loop, causing the TCP buffer to fill and eventually the connection to be killed by Binance's 90s data timeout).

**Warning signs**: WS connections dying during `fill_from_rest()`. `data timeout -- no WebSocket frames, treating as dead` in tracing. Reconnection loop fires before engine even starts. Symbols processed last by REST have the longest WS buffer gaps.

**Prevention**: Two-pronged:

1. **Increase WS channel capacity** to at least 50,000 per symbol (50K trades \* ~200 bytes = ~10MB, acceptable). This covers ~15 minutes at peak BTCUSDT volume.
2. **Parallel `fill_from_rest()`** reduces total fill time from ~5 min to ~30 sec (with rate limiter partitioning), shrinking the buffer window by 10x.

**Phase**: Phase 2 (Parallel fill_from_rest) addresses the root cause. Phase 1 should increase channel capacity as an immediate mitigation.

**Risk if ignored**: Silent trade loss for all symbols except the one currently being REST-filled. Creates Stathera gaps that kintsugi won't heal until tomorrow (writer ownership model).

---

### P1.3 REST Returns Trades That Overlap With WS Buffer

**What happens**: After `fill_from_rest()` processes up to trade ID N, the WS buffer contains trades starting from some ID M where M < N (WS started before REST caught up). When the symbol task drains the WS buffer, trades M through N are duplicates of what the processor already consumed.

**Warning signs**: Processor errors like "trade ID X already processed" (if the processor has monotonicity guards). OR, if the processor silently re-processes: duplicate bars, incorrect microstructure features (double-counted volume), or Stathera overlaps.

**Prevention**: The current code (line 1415-1416) calls `gap_detector.update_last_tid(trade.agg_trade_id)` for every WS trade. But there's no dedup BEFORE the trade reaches the processor. The processor's `process_single_trade()` should check `trade.agg_trade_id <= self.last_agg_trade_id` and skip. Verify this guard exists in `opendeviationbar-core/src/processor.rs`. If not, add it there (Rust layer, not Python).

**Phase**: Phase 1 (Junction Fix). Must be validated before any junction changes land.

---

### P1.4 Forming Bar Trade Count Exceeds Memory Budget

**What happens**: For high thresholds on calm markets (e.g., BTCUSDT@1000 during Asia session), a single forming bar can accumulate 200K+ trades over 12+ hours without breaching. During `fill_from_rest()`, the processor holds all this state. If `fill_from_rest()` feeds 500K trades for a symbol, the processor's internal state grows unbounded.

**Warning signs**: RSS growth during `fill_from_rest()`. systemd MemoryMax kill. "fill_from_rest: ABORTING after 3 consecutive errors" in logs (OOM manifests as various errors).

**Prevention**: The current chunked REST approach (50K trades/chunk, line 459) limits per-REST-call memory but NOT per-processor state. The processor's forming bar IS the state -- it doesn't free memory until a bar completes. Monitor RSS per-symbol during fill. If a processor's forming bar exceeds a configurable trade count (e.g., 1M), log a warning but do NOT reset (resetting would create a gap).

**Phase**: Phase 2 (Parallel fill). Memory monitoring should be added in Phase 1.

**Historical precedent**: v13.30.0 reverted Rust `fill_from_rest()` because it loaded all trades into a single Vec (2M cap) causing 3G+ peak. The chunked approach (v13.30.0+) fixes the Vec issue but not the processor state issue.

---

## 2. Combined WS Stream Pitfalls

### P2.1 Reconnection Kills All Symbols on That Connection

**What happens**: With 2x9 combined streams, when one connection drops (Binance server close, TCP timeout, Cloudflare intervention), all 9 symbols on that connection lose data simultaneously. During the reconnection + backoff window (up to 16 seconds with exponential backoff), 9 symbols have zero incoming trades.

**Warning signs**: `reconnections` metric increments by 1 but 9 symbols show gaps simultaneously. Heartbeat detects 9 correlated Stathera gaps at the same timestamp.

**Prevention**:

1. **Puck fills the gap**: Each symbol's gap detector fires independently on the first WS trade after reconnect. Puck's parallel REST fetch fills the gap (sub-3s per symbol). But 9 simultaneous Puck fills = 9 \* 5 concurrent REST fetches = 45 concurrent requests against the 6000 weight/min limit.
2. **Stagger Puck fills**: Add a per-symbol delay (100ms jitter) before Puck's REST fetch to prevent thundering herd on the rate limiter.
3. **Tiered connection strategy**: Put high-volume symbols (BTCUSDT, ETHUSDT) on separate connections from low-volume ones. If the low-volume connection dies, Puck's gap is small. If the high-volume connection dies, it reconnects faster (TCP keep-alive fires sooner on active connections).

**Phase**: Phase 3 (Combined WS). Must be validated during spike testing.

**Binance-specific**: Binance enforces 300 connections/5min/IP. Rapid reconnection cycles could hit this limit with 2 connections reconnecting every 30 seconds.

---

### P2.2 Combined Stream Message Format Wrapping

**What happens**: Individual WS connections send aggTrade JSON directly: `{"e":"aggTrade","E":...,"s":"BTCUSDT",...}`. Combined streams wrap in `{"stream":"btcusdt@aggTrade","data":{"e":"aggTrade",...}}`. The current `connect_and_stream()` (line 238, `websocket.rs`) parses `serde_json::from_str::<BinanceAggTrade>(&text)` directly -- this will FAIL on combined stream messages because the outer wrapper is not `BinanceAggTrade`.

**Warning signs**: "JSON parse failed" warnings flooding the logs for every trade on every symbol. Zero trades reaching processors. System appears to be streaming but produces no bars.

**Prevention**: Add a combined stream message parser that:

1. Detects the `{"stream":"...","data":{...}}` wrapper
2. Extracts the `data` field
3. Routes to the correct symbol based on the `stream` field
4. Falls back to direct parse for backward compatibility

This is a Rust change in `opendeviationbar-providers/src/binance/websocket.rs`. Need a new `CombinedBinanceMessage` struct with `#[serde(untagged)]` enum for both formats, or a dedicated `connect_combined_stream()` method.

**Phase**: Phase 3 (Combined WS). This is blocking -- combined streams cannot work without this parser.

---

### P2.3 Message Ordering Across Combined Streams

**What happens**: Binance makes no guarantee about message ordering within a combined stream. Two trades for different symbols that occurred at the same millisecond may arrive in any order. More critically, a burst of trades for BTCUSDT may delay delivery of SOLUSDT trades on the same connection, causing the SOLUSDT gap detector to fire prematurely.

**Warning signs**: Spurious gap detections on low-volume symbols sharing a connection with high-volume symbols. Puck fires frequently for gaps of size 1-3 that don't actually exist (they're just delayed).

**Prevention**: Add a small grace period to the gap detector -- don't fire Puck immediately on a gap of size < 10. Instead, buffer the triggering trade and wait for the next 2-3 trades from that symbol before confirming the gap. This is a change to `TradeIdGapDetector` in `streaming/src/gap/detector.rs`.

**Phase**: Phase 3 (Combined WS). Grace period must be tunable per symbol (BTCUSDT needs smaller grace period than WIFUSDT).

---

### P2.4 Binance 24h Connection Lifetime Limit

**What happens**: Binance silently drops WebSocket connections after 24 hours. With per-symbol connections, this is a single-symbol reconnect. With combined streams, it's 9 symbols simultaneously (see P2.1).

**Warning signs**: Predictable daily reconnection event, always ~24h after the last full reconnect.

**Prevention**: Proactive reconnection at 23h (before Binance drops it). Stagger the two combined connections' reconnection by 30 minutes so they never reconnect simultaneously. This eliminates the worst-case scenario of all 18 symbols losing data at the same time.

**Phase**: Phase 3 (Combined WS). Add to `run_with_reconnect()`.

---

## 3. Parallel Backfill Pitfalls

### P3.1 Rate Limit Exhaustion From Simultaneous REST Retries

**What happens**: When `fill_from_rest()` runs 18 symbols in parallel, each using `fetch_aggtrades_parallel()` with 5 concurrent fetches, the peak demand is 18 _5 = 90 concurrent REST requests. Each aggTrades request costs 4 weight (corrected in Phase 0a of Issue #257). 90_ 4 = 360 weight per request cycle. At 6000 weight/min, the budget is exhausted in 16 cycles -- roughly 30 seconds with typical latency.

**Warning signs**: HTTP 429 responses from Binance. `AdaptiveRateLimiter` shows `remaining() == 0`. `fill_from_rest` chunks fail with "REST chunk failed" and `consecutive_errors` climbs to 3 (triggering the abort at line 497).

**Prevention**: **Partition the rate limiter budget.** With 18 parallel fills, each symbol gets `6000 / 18 = 333 weight/min`, supporting ~83 requests/min or ~1.4 requests/sec. Reduce per-symbol concurrency from 5 to 2 during fill_from_rest. The `AdaptiveRateLimiter` is shared (via `Arc`), so all parallel fills compete for the same budget -- this is correct but needs concurrency tuning.

**Phase**: Phase 2 (Parallel fill_from_rest).

**Historical precedent**: Issue #291 found that kintsugi has NO rate limiter at all (`HistoricalDataLoader::new()` sets `rate_limiter: None`). If kintsugi runs simultaneously with parallel fill, both consume from the same IP's 6000 weight budget. Kintsugi's today-guard prevents simultaneous writes, but its REST fetches still consume rate limit.

---

### P3.2 Thundering Herd on Reconnect

**What happens**: After a network outage (Cloudflare, ISP), all 18 symbols reconnect simultaneously. Each fires its gap detector, and Puck triggers parallel REST fills for all 18 symbols at once. Combined with normal `fill_from_rest()` traffic, the rate limiter is overwhelmed.

**Warning signs**: All 18 symbols show "puck: filling gap via parallel REST API" within the same second. Rate limit exhaustion follows within 10 seconds.

**Prevention**:

1. **Jitter Puck fills**: Random 0-500ms delay before each Puck REST fetch.
2. **Global Puck semaphore**: Limit concurrent Puck fills to 3 symbols max. Queue the rest.
3. **Backoff on 429**: The `AdaptiveRateLimiter` already reads `X-MBX-USED-WEIGHT-1m` headers. When weight > 80%, delay all new requests by 5 seconds.

**Phase**: Phase 2 (Parallel fill) for the semaphore. Phase 3 (Combined WS) for jitter (fewer connections = more correlated reconnects).

---

### P3.3 Memory Pressure From 18 Concurrent REST Fetches

**What happens**: Each `fetch_aggtrades_parallel()` loads trades into a `Vec<AggTrade>`. With 5 concurrent pages of 1000 trades each, that's 5000 trades _~200 bytes = ~1MB per symbol. With 18 symbols concurrent: ~18MB. This is fine. BUT: if a symbol has a large gap (e.g., 500K trades after a long outage), the parallel fetch loads all pages into memory before processing. At 500K trades_ 200 bytes \* 18 symbols = ~1.8GB.

**Warning signs**: RSS climbing during `fill_from_rest()`. systemd MemoryMax (currently 4GB for sidecar) breach.

**Prevention**: The current chunked approach in `fill_from_rest()` (50K trades per chunk, line 459) already bounds per-symbol memory. But if parallel, all 18 chunks are concurrent: 18 _50K_ 200 bytes = ~180MB. This is acceptable. The risk is in Puck fills, not `fill_from_rest()` -- Puck uses `fetch_aggtrades_parallel()` without chunking. For large gaps (>50K trades), Puck should fall back to chunked mode or defer to kintsugi.

**Phase**: Phase 2 (Parallel fill).

---

## 4. Deployment Pitfalls

### P4.1 Clean-Slate Deploy Deletes Today's Data With No Rollback

**What happens**: The documented deploy playbook is: Stop sidecar, Delete today's ClickHouse data, Clear checkpoints, Start sidecar. If the new version has a bug, there's no way to restore the deleted data. The sidecar will re-stream from checkpoints (which were cleared), creating a gap from the delete time to the new stream start.

**Warning signs**: N/A (this is a process risk, not a runtime symptom).

**Prevention**:

1. **Backup before delete**: `INSERT INTO open_deviation_bars_backup SELECT * FROM open_deviation_bars WHERE toDate(close_time_ms) = today()` before any delete.
2. **Checkpoint export**: Save checkpoint files to a known directory (`/tmp/odb-checkpoint-backup-YYYYMMDD/`) before clearing.
3. **Rollback script**: Prepared SQL to restore from backup table if new version fails within 1 hour.
4. **Deploy during low-volume period**: UTC 00:00-02:00 (Asian lunch, minimal crypto volume). Minimizes data loss window.

**Phase**: Phase 5 (Deploy). Must be scripted before deploy, not improvised.

**Historical precedent**: v13.26 removed `reconstruct_today()` after it OOMed at 2.5GB with 18 symbols. There is no "reconstruct from scratch" fallback. The sidecar can only stream forward.

---

### P4.2 Checkpoint Clearing Creates a No-Data Window

**What happens**: Clearing checkpoints means `fill_from_rest()` starts from ClickHouse's last known `last_agg_trade_id`. If there's a gap between ClickHouse's data and the current live stream (typically 5-60 minutes depending on flush frequency), `fill_from_rest()` must bridge this gap. During this bridge, no bars are emitted.

**Warning signs**: Heartbeat shows "stale" for all symbols for 5-10 minutes post-restart. No bars in ring buffer during `fill_from_rest()` (bars ARE emitted to the ring buffer during REST fill, but Python may not drain fast enough).

**Prevention**: The current architecture handles this correctly -- `fill_from_rest()` emits bars to the ring buffer as it processes REST trades. Python's main loop drains the ring buffer. But verify: does the Python main loop start draining BEFORE `engine.start()` returns? Currently `engine.start()` spawns tokio tasks and returns immediately. The ring buffer is drained by `engine.next_bar()` in the Python main loop, which only runs AFTER `_run_main_loop()` starts. Bars from `fill_from_rest()` sit in the ring buffer (capacity 10,000) until then.

**Phase**: Phase 5 (Deploy). Verify ring buffer doesn't overflow during the checkpoint-to-stream window.

---

### P4.3 Version Mismatch Between Rust .so and Python

**What happens**: After deploying a new wheel, the Python layer imports the new `.py` files but the `_core.abi3.so` binary may be cached or stale. Symptoms: `AttributeError` on new Rust bindings, silent behavior differences on changed bindings, segfaults on ABI mismatch.

**Warning signs**: `engine.fill_from_rest` not found (`hasattr` returns False, falling back to WS-only start). New telemetry fields missing. Crash with "invalid memory reference".

**Prevention**: Always `pip install --force-reinstall` the wheel. Add a version check at startup: `assert opendeviationbar._core.__version__ == opendeviationbar.__version__`. Or simpler: the existing `hasattr(engine, "fill_from_rest")` guard (line 850) already handles this gracefully.

**Phase**: Phase 5 (Deploy). Standard wheel deployment handles this, but worth verifying on bigblack.

---

## 5. Testing Pitfalls

### P5.1 Junction Behavior Cannot Be Tested Without Live Binance

**What happens**: The junction bug is a timing-dependent race between REST completion and WS buffer drain. Unit tests with mock data cannot reproduce the exact timing. Integration tests with recorded data lose the real-time aspect.

**Warning signs**: Tests pass, production fails. "Works on my machine" syndrome.

**Prevention**: Multi-layered testing strategy:

1. **Rust unit tests**: Test `TradeIdGapDetector` seeding and Puck suppression in isolation. Feed pre-created processors a sequence of trades with a known gap, verify gap detector doesn't fire when properly seeded. This tests the LOGIC, not timing.
2. **Replay recorded data**: Use `ws-mock` (already in `~/fork-tools`) to replay a recorded trade stream with known junction gaps. Control timing via channel backpressure.
3. **Integration with `wiremock-rs`**: Mock Binance REST API to return known trade ranges. Verify `fill_from_rest()` + WS transition produces zero Stathera gaps.
4. **Live canary**: Deploy to bigblack with 1 symbol (BTCUSDT only) first. Monitor for 1 hour. If zero gaps, expand to all 18.

**Phase**: Phase 1 (Junction Fix) for unit tests and replay. Phase 4 (Testing) for integration.

---

### P5.2 Combined Stream Testing Requires Multiple Symbols

**What happens**: Testing combined WS streams with a single symbol doesn't exercise the demultiplexing logic, cross-symbol ordering, or the `{"stream":"...","data":{...}}` wrapper format.

**Prevention**: The spike test (Phase 3) should use at minimum 3 configurations on live Binance data:

1. **1x18**: All symbols on one connection. Tests maximum demux load.
2. **2x9**: The expected production config. Tests connection isolation.
3. **3x6**: More connections, fewer symbols each. Tests scaling.

Each config runs for 30 minutes minimum. Success criteria: zero Stathera gaps, zero recv timeouts, RSS < 500MB, p99 trade latency < 100ms.

**Phase**: Phase 3 (Combined WS).

---

### P5.3 Mock Data Doesn't Exercise Rate Limiter Contention

**What happens**: Mock REST APIs respond instantly, so parallel fill tests never contend on the rate limiter. In production, 18 concurrent fills competing for 6000 weight/min create complex queuing behavior.

**Prevention**: Configure `wiremock-rs` with realistic response delays (50-200ms per request). Add a test-mode rate limiter that simulates Binance's weight budget with artificial depletion. Verify that `fill_from_rest()` completes for all 18 symbols within 10 minutes even with rate limiting.

**Phase**: Phase 4 (Testing).

---

## 6. Historical Lessons From This Project

### H6.1 Asclepius Cascade (Issue #284): Never Heal Live Gaps With a Separate Processor

**Incident**: v13.21, 665 overlaps + 169 gaps in 10 hours. Asclepius used `fill_gap()` which creates a fresh stateless processor to process gap trades. This processor doesn't know the live engine's forming bar state. Result: the fresh processor produces bars that overlap with the live engine's bars.

**Lesson**: Any gap-fill mechanism MUST use the SAME processor instance that handles WS trades. This is why `fill_from_rest()` preserves processors (line 620: `self.pre_created_processors = Some(all_processors)`) and why Puck works (it receives `&mut processors` at line 1305). Never create a new `OpenDeviationBarProcessor` for gap-fill in a live context.

**Applies to**: Junction fix (Phase 1), parallel fill (Phase 2). Both must preserve processor identity.

---

### H6.2 Cross-Midnight Retry Storm (Issue #264): Filter Before Write, Not After

**Incident**: v13.10.0 era. Rust `LiveBarEngine` lacked midnight reset. Cross-midnight bars hit the pre-write gate (`bulk_operations.py`) which raised `ValueError`. The sidecar retried, got the same cross-midnight bar, retried again -- infinite loop. Dead letters accumulated, Charon retried every 5 minutes, logs flooded.

**Lesson**: Defense-in-depth with filter-not-raise semantics. The fix changed the pre-write gate from `raise ValueError` to `filter + warn`. Invalid bars are dropped, valid bars in the same batch still written. Dead letters from permanently-invalid data are deleted, not retried.

**Applies to**: Any new bar validation added in this project (e.g., min-TID guard at line 865) must FILTER, never RAISE. A raise in the write path creates a retry storm that blocks all bar processing.

---

### H6.3 reconstruct_today() OOM (v13.26): Never Buffer All of Today's Data

**Incident**: `reconstruct_today()` loaded all trades for 18 symbols from ClickHouse into memory to rebuild today's processor state. At 18 symbols _4 thresholds_ ~500K trades = 2.5GB peak RSS. systemd MemoryMax killed it.

**Lesson**: Streaming, chunked processing is mandatory. The current `fill_from_rest()` uses 50K trade chunks (line 459), processing and freeing each chunk before fetching the next. Peak RSS is ~50MB per symbol. Any new function that processes "all of today" must follow this pattern.

**Applies to**: Parallel fill (Phase 2) must not load 18 symbols' data simultaneously. Sequential per-symbol within the chunked loop is correct. Parallel across symbols is fine only if each symbol's chunks are processed sequentially.

---

### H6.4 Writer Ownership Violation (Issue #280): Sidecar and Kintsugi Must Never Write the Same Day

**Incident**: Four fix cycles (Mar 3-16, 2026). Concurrent writers (sidecar + kintsugi + Asclepius) created overlapping bars because ReplacingMergeTree dedup only works after OPTIMIZE, not in real-time. Each writer had its own processor state, producing bars with overlapping trade-ID ranges.

**Lesson**: Clean temporal boundary: sidecar owns today (UTC), kintsugi owns yesterday and older. No exceptions. The kintsugi today-guard is ABSOLUTE (returns -1 sentinel for today's shards). No "interior gap" exception.

**Applies to**: The junction fix and parallel fill operate ONLY on today's data (they're part of the sidecar). This is safe. But verify: does parallel `fill_from_rest()` for a symbol that started 3 days ago fetch trades from 3 days ago? If yes, those trades belong to kintsugi's domain. Guard: `fill_from_rest()` should only fill from `max(today_start_ms, ch_seed_tid)` to prevent writing yesterday's data.

---

### H6.5 WS Monitor Proof (Issue #290): Binance Delivers Perfectly -- All Gaps Are Ours

**Finding**: Independent WS monitor (PID 3882596, 90K+ trades, zero gaps) proved Binance WebSocket delivers perfect trade-ID continuity. Every Stathera gap in production is caused by our pipeline: ring buffer overflow, processor state loss, restart transitions, connection drops.

**Lesson**: Never blame the data source. When investigating gaps, always look at:

1. Ring buffer `dropped_bars` metric (Lethe detection)
2. Reconnection events (Antaeus backoff)
3. Puck fill failures (REST errors)
4. Processor state loss (checkpoint issues)

**Applies to**: All phases. Especially useful during spike testing (Phase 3) -- if gaps appear, they're ours, not Binance's.

---

### H6.6 Stathera Midnight Gap Root Cause: Non-Streamed Symbols Have Join Gaps

**Finding**: 16 of 18 symbols relied on kintsugi/backfill joins at day boundaries because only BTCUSDT/ETHUSDT were streamed. Each non-streamed symbol had persistent midnight gaps proportional to its threshold. Fresh processors at day boundaries don't know the previous day's forming bar state.

**Lesson**: The ONLY way to eliminate midnight gaps is continuous streaming with a single processor that crosses midnight (with proper `reset_at_ouroboros()` calls). Backfill with fresh processors will ALWAYS create join gaps at day boundaries.

**Applies to**: Phase 3 (Combined WS) directly solves this by streaming all 18 symbols. After deployment, all 18 symbols should have zero midnight join gaps. If any persist, the root cause is the combined stream demultiplexer, not kintsugi.

---

## Summary: Phase-Pitfall Matrix

| Pitfall                           | Phase 1 (Junction) | Phase 2 (Parallel) | Phase 3 (Combined WS) | Phase 4 (Testing) | Phase 5 (Deploy) |
| --------------------------------- | :----------------: | :----------------: | :-------------------: | :---------------: | :--------------: |
| P1.1 Puck fires at junction       |      **FIX**       |                    |                       |                   |                  |
| P1.2 WS buffer overflow           |      mitigate      |      **FIX**       |                       |                   |                  |
| P1.3 REST-WS trade overlap        |      **FIX**       |                    |                       |                   |                  |
| P1.4 Forming bar memory           |      monitor       |      monitor       |                       |                   |                  |
| P2.1 Reconnect kills 9 symbols    |                    |                    |        **FIX**        |                   |                  |
| P2.2 Combined message format      |                    |                    |        **FIX**        |                   |                  |
| P2.3 Cross-symbol ordering        |                    |                    |        **FIX**        |                   |                  |
| P2.4 24h connection lifetime      |                    |                    |        **FIX**        |                   |                  |
| P3.1 Rate limit exhaustion        |                    |      **FIX**       |                       |                   |                  |
| P3.2 Thundering herd              |                    |      **FIX**       |       amplified       |                   |                  |
| P3.3 18-symbol memory pressure    |                    |      **FIX**       |                       |                   |                  |
| P4.1 Clean-slate no rollback      |                    |                    |                       |                   |     **FIX**      |
| P4.2 Checkpoint clearing window   |                    |                    |                       |                   |     **FIX**      |
| P4.3 Version mismatch             |                    |                    |                       |                   |      verify      |
| P5.1 No live junction test        |      validate      |                    |                       |      **FIX**      |                  |
| P5.2 Multi-symbol test gap        |                    |                    |                       |      **FIX**      |                  |
| P5.3 Rate limiter not tested      |                    |                    |                       |      **FIX**      |                  |
| H6.1 Separate processor = overlap |     **guard**      |     **guard**      |                       |                   |                  |
| H6.2 Raise in write path = storm  |     **guard**      |                    |                       |                   |                  |
| H6.3 Buffer all today = OOM       |                    |     **guard**      |                       |                   |                  |
| H6.4 Writer ownership violation   |                    |     **guard**      |                       |                   |                  |
| H6.5 Blame Binance (wrong)        |     debug aid      |     debug aid      |       debug aid       |                   |                  |
| H6.6 Join gaps from backfill      |                    |                    |        **FIX**        |                   |                  |

**Legend**: **FIX** = primary phase that addresses it. mitigate/monitor/guard = secondary defense. verify = confirmation step.

---

## Top 5 Highest-Risk Items

1. **P1.1 Puck fires at junction** -- The core bug. Without this fix, nothing else matters. ~20 lines of Rust.
2. **P2.2 Combined message format** -- Blocking for Phase 3. Without the wrapper parser, combined streams produce zero bars.
3. **P3.1 Rate limit exhaustion** -- 18 parallel fills will hit 429 within 30 seconds without budget partitioning.
4. **P2.1 Reconnect kills 9 symbols** -- Correlated failures on combined streams. Puck + rate limiter must handle 9 simultaneous fills.
5. **P4.1 Clean-slate no rollback** -- Irreversible data loss on failed deploy. Must have backup script BEFORE deploy.
