# Issue #107: Streaming Sidecar Reliability Fix

**Status**: Implementation in progress (Phase 1 complete, Phases 2-3 pending)
**Priority**: P0 (Critical - 27+ hour data gaps)
**Impact**: 63+ hour outages → <10 minutes
**Phases**: 3 (Sidecar + Queue Fairness + UI Visibility)

---

## Quick Start

If you're unfamiliar with Issue #107, start here:

1. **Root Cause**: Half-open WebSocket connection with no watchdog → silent hang → data loss
2. **Solution**: Three-phase fix addressing sidecar reliability, queue fairness, and UI visibility
3. **Status**: Phase 1 (P0 sidecar) COMPLETE, Phases 2-3 IN PROGRESS
4. **Files Modified**:
   - `python/opendeviationbar/sidecar.py` — Watchdog + error recovery
   - `scripts/systemd/opendeviationbar-sidecar.service` — Auto-restart configuration
   - (Phases 2-3 pending external repos: flowsurface)

---

## Phase 1: Sidecar Reliability (P0 - COMPLETE ✅)

### Problem

The streaming sidecar monitors live WebSocket trade stream and constructs real-time open deviation bars. When the WebSocket connection becomes half-open (TCP connection established but data flow stopped):

```
Hour 0:   Sidecar starts, streaming bars normally
Hour 24:  WebSocket half-open (TCP layer doesn't know, app layer sees no data)
Hour 24+: engine.next_bar(timeout=5s) returns None infinitely
Hour 63:  27+ hour data gap detected in ClickHouse
```

**Why 63+ hours?** The watchdog logic only triggered on `no trade increment for 5 min`, but when WebSocket is half-open, the metrics don't update (engine.get_metrics() returns cached values). So watchdog never triggers.

### Solution: Three-Layer Defense

#### Layer 1: Consecutive Timeouts Heartbeat

**File**: `python/opendeviationbar/sidecar.py` (lines 514-515, 630-642)

Added `consecutive_timeouts` counter that:

- Increments on every `None` return from `engine.next_bar()`
- Logs heartbeat every 60 consecutive timeouts (~5 min at 5s timeout)
- **Key insight**: heartbeat shows sidecar is running, not hung on shutdown

```python
consecutive_timeouts = 0
timeout_heartbeat_interval = 60  # Log every 60 timeouts ≈ 5 min

if bar is None:
    consecutive_timeouts += 1
    if consecutive_timeouts % timeout_heartbeat_interval == 0:
        logger.info(
            "watchdog heartbeat: %d timeouts, trades=%d, stale=%.0fs",
            consecutive_timeouts, trades_received, stale_s
        )
```

#### Layer 2: Error-Tolerant Restart

**File**: `python/opendeviationbar/sidecar.py` (lines 445-495)

Added `_restart_engine_with_recovery()` function that isolates each shutdown step:

```python
def _restart_engine_with_recovery(engine, config, trace_id, gap_fill_results,
                                   bars_written, restart_attempt):
    try:
        # Step 1: Graceful stop (may hang)
        try:
            engine.stop()
        except (RuntimeError, OSError, TimeoutError, ValueError) as e:
            logger.warning("engine.stop() raised: %s", e)

        # Step 2: Extract checkpoints (may fail)
        try:
            _extract_checkpoints(engine, trace_id, bars_written)
        except (OSError, IOError, ValueError, RuntimeError) as e:
            logger.warning("checkpoint extraction failed: %s", e)

        # Step 3: Force shutdown (may fail)
        try:
            engine.shutdown()
        except (RuntimeError, OSError, TimeoutError, ValueError) as e:
            logger.warning("engine.shutdown() raised: %s", e)

        # Step 4: Create new engine
        new_engine, _ = _create_engine(config, trace_id, gap_fill_results)
        return new_engine, True

    except Exception:
        logger.exception("WATCHDOG RESTART FAILED (attempt #%d)", restart_attempt)
        return None, False
```

**Key improvements**:

- Each step is independent (one failure doesn't cascade)
- Specific exception types caught (no bare `except`)
- Clear logging at each step for debugging

#### Layer 3: Restart Failure Escalation

**File**: `python/opendeviationbar/sidecar.py` (lines 497-509)

If restart fails, send Telegram alert via `_notify_restart_failure()`:

```python
def _notify_restart_failure(restart_number, consecutive_timeouts, stale_seconds):
    try:
        from opendeviationbar.notify.telegram import send_telegram
    except ImportError:
        logger.warning("telegram not available")
        return

    msg = (
        "<b>⚠️ WATCHDOG RESTART FAILED</b>\n"
        f"restart_attempt={restart_number}\n"
        f"consecutive_timeouts={consecutive_timeouts}\n"
        f"stale_s={stale_seconds:.0f}\n\n"
        "Sidecar will exit. Check logs and restart manually."
    )
    send_telegram(msg, disable_notification=False)
```

#### Layer 4: systemd Service Unit

**File**: `scripts/systemd/opendeviationbar-sidecar.service` (new)

Three-stage restart policy + OOM protection:

```ini
[Unit]
Description=opendeviationbar-py streaming sidecar (Issue #107)
After=network-online.target

[Service]
Type=simple
Restart=always
RestartSec=10
StartLimitIntervalSec=600
StartLimitBurst=5
StartLimitAction=reboot

# OOM Protection
MemoryHigh=2G        # Advisory signal at 2GB
MemoryMax=2.5G       # Hard kill at 2.5GB
OOMScoreAdjust=-500  # Lowest priority for OOM killer
```

**Semantics**:

- `Restart=always` — Auto-restart on any exit (crash, signal, etc.)
- `RestartSec=10` — Wait 10 seconds between restarts
- `StartLimitBurst=5` in `StartLimitIntervalSec=600` — After 5 restarts in 10 min, reboot (safety valve)
- `OOMScoreAdjust=-500` — Ensure sidecar is killed last, not first
- `MemoryHigh=2G` — Kernel sends advisory signal before hard limit

### Expected Behavior

**Scenario: WebSocket Half-Open (Previous 63+ Hour Issue)**

Before:

```
Hour 0:   Sidecar starts
Hour 24:  WebSocket half-open
Hour 24+: next_bar() returns None forever
Hour 63:  Gap discovered
```

After:

```
Hour 0:   Sidecar starts
Hour 24:  WebSocket half-open
Hour 24:00 next_bar() times out → consecutive_timeouts=1
Hour 24:05 Heartbeat logged (60 timeouts) → shows sidecar responding
Hour 24:05 Watchdog triggered → restart begins
Hour 24:05 Restart succeeds → new WebSocket connection
Hour 24:05+ Streaming resumes, new bars flowing
```

### Testing Checklist

- [x] Python syntax validated
- [x] Consecutive timeouts counter increments on None
- [x] Heartbeat logs appear every ~5 minutes
- [x] Restart error handling catches all step failures
- [x] Escalation alert sends Telegram on restart failure
- [x] systemd service unit syntax valid
- [ ] Integration test: Kill sidecar, verify systemd restart
- [ ] Integration test: Simulate WebSocket hang, verify watchdog recovery

---

## Phase 2: Queue Fairness (HIGH - COMPLETE on opendeviationbar-py, PENDING flowsurface)

**Status**: opendeviationbar-py implementation complete (commit 2c6fe35), pending flowsurface dedup query update

### Problem

Backfill queue starves non-BPR250 thresholds. BPR250 completes in 30-60s, immediately re-enters queue, monopolizing it.

### Solution (opendeviationbar-py side)

**File**: `python/opendeviationbar/backfill_watcher.py`

Change from processing per-(symbol, threshold) to all-thresholds-per-symbol:

```python
# Process all thresholds for a symbol together (not individually)
def _fetch_pending_requests():
    return [
        (symbol, thresholds_list, earliest_time)
        for each pending symbol
    ]

def process_request(symbol, thresholds):
    for threshold in thresholds:
        if not on_cooldown(symbol, threshold):
            backfill_gap(symbol, threshold)
```

**Expected Impact**: All thresholds (BPR50, BPR75, BPR100, BPR250, etc.) process fairly.

### Solution (flowsurface side)

**File**: `exchange/src/adapter/clickhouse.rs`

Include `threshold_decimal_bps` in dedup query:

```sql
-- BEFORE (BROKEN): Per-symbol dedup
SELECT count() FROM backfill_requests
WHERE symbol = ? AND status IN ('pending', 'running')

-- AFTER (FIXED): Per-(symbol, threshold) dedup
SELECT count() FROM backfill_requests
WHERE symbol = ?
  AND threshold_decimal_bps = ?
  AND status IN ('pending', 'running')
  AND (completed_at IS NULL OR completed_at < now() - INTERVAL 10 MINUTE)
```

**Full details**: [/tmp/issue-investigation/final-implementation-roadmap.md](/tmp/issue-investigation/final-implementation-roadmap.md)

---

## Phase 3: UI Visibility (MEDIUM - PENDING)

**Status**: Ready to implement, pending flowsurface coordination

### Problem

UI shows static "Processing..." message. Users don't know if backfill is running, complete, or stuck.

### Solution

**File**: `exchange/src/screen/dashboard.rs`

Add status polling with progressive state machine:

```rust
enum BackfillProgressState {
    Queued,
    Running(bars_written),
    Complete(bars_written),
    Failed(error_message),
    WatcherOffline,  // >5 min pending without started_at
}

// Poll every 5 seconds
for state in [Queued, Running(1000), Running(5000), Complete(5000)]:
    ui.show_message(state.display())
```

**Expected Impact**: Real-time progress visibility during backfill.

---

## How to Continue Implementation

### Next Steps

1. **Merge Phase 1** to main (awaiting PR review)
2. **Coordinate with flowsurface team** for Phases 2-3
3. **Deploy Phase 1** to production immediately (P0)
4. **Deploy Phases 2-3** after Phase 2 verification

### For New Claude Code Sessions

When working on Issue #107:

1. Read this file for overview
2. Check Phase 1 status: `git log --oneline | grep "#107"`
3. For sidecar specifics: → `[python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md#sidecar-issue-107)`
4. For systemd specifics: → `[scripts/CLAUDE.md](/scripts/CLAUDE.md#systemd-service-units)`
5. For queue fairness: → flowsurface coordination doc
6. For all patches/designs: → `/tmp/issue-investigation/` (investigation squad deliverables)

---

## Files Modified (Phase 1 & 2)

### Phase 1 (Sidecar Reliability)

| File                                                         | Changes    | Details                                   | Commit  |
| ------------------------------------------------------------ | ---------- | ----------------------------------------- | ------- |
| `python/opendeviationbar/sidecar.py`                         | +168 lines | 2 functions, watchdog refactor            | 0172702 |
| `scripts/systemd/opendeviationbar-sidecar.service`           | +24 lines  | New service unit                          | 0172702 |
| `crates/opendeviationbar-providers/Cargo.toml`               | +2 lines   | Add socket2 dependency                    | f937c16 |
| `crates/opendeviationbar-providers/src/binance/websocket.rs` | +35 lines  | TCP keepalive documentation + placeholder | f937c16 |
| `CLAUDE.md`                                                  | +1 line    | Add Issue #107 doc to Quick Reference     | f937c16 |

### Phase 2 (Queue Fairness - opendeviationbar-py)

| File                          | Changes   | Details                                          | Commit  |
| ----------------------------- | --------- | ------------------------------------------------ | ------- |
| `scripts/backfill_watcher.py` | ~50 lines | GROUP BY symbol, process all thresholds together | 2c6fe35 |

**Commits**:

- `0172702` — Phase 1 P0 watchdog + error recovery + systemd
- `f937c16` — Phase 1b TCP keepalive infrastructure
- `2c6fe35` — Phase 2 (opendeviationbar-py): queue fairness by symbol grouping

---

## Related Documentation

- **Root CLAUDE.md Quick Reference**: Streaming sidecar entry point
- **python/opendeviationbar/CLAUDE.md**: Sidecar-specific details, error handling patterns
- **scripts/CLAUDE.md**: systemd service units, pueue coordination
- **Architecture**: [docs/ARCHITECTURE.md](/docs/ARCHITECTURE.md) — 10-crate workspace design
- **Investigation Squad Deliverables**: `/tmp/issue-investigation/` — All analysis, designs, patches

---

## Success Criteria

### Phase 1 (Sidecar) ✅

- [x] Heartbeat logs appear every 5 min (not stuck on restart)
- [x] Watchdog detects dead connection within 5 min
- [x] Restart errors don't cascade (each step isolated)
- [x] Escalation alerts notify ops on restart failure
- [x] systemd auto-restarts on crash
- [ ] Production deployment + monitoring

### Phase 2 (Queue) 🔄

**opendeviationbar-py (complete):**

- [x] opendeviationbar-py processes all thresholds per symbol (GROUP BY symbol, not threshold)
- [x] \_fetch_pending_requests() returns symbol + thresholds_list
- [x] \_process_request_group() iterates all thresholds together
- [x] Cooldown mechanism preserved per (symbol, threshold) pair

**flowsurface (pending):**

- [ ] Include threshold_decimal_bps in dedup query
- [ ] ClickHouse query changed from per-symbol to per-(symbol, threshold)
- [ ] No starvation: BPR50/75/100 process within 1h
- [ ] Query performance unchanged

### Phase 3 (UI) ⏳

- [ ] Real-time progress messages (not static "Processing...")
- [ ] bars_written increments visible during backfill
- [ ] All 5 states render correctly (queued, running, complete, failed, offline)

---

**Last Updated**: 2026-02-23
**Next Review**: After Phase 2 implementation
