# Streaming Sidecar Package

**Parent**: [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) | **Hub**: [/CLAUDE.md](/CLAUDE.md)

Long-running sidecar that streams live trades from Binance WebSocket via Rust `StreamManager`, constructs open deviation bars in real-time, and writes them to ClickHouse. Modularized from a single `sidecar.py` (Phase 01 v1.5).

---

## Files

| File                  | Purpose                                                                                                                                                       |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `__init__.py`         | Main orchestrator: `run_sidecar()`, `_create_engine()`, `_main_loop()`, watchdog, Argus liveness, Lethe overflow detection, signal handlers                   |
| `__init__.pyi`        | Type stubs for `run_sidecar`, `SidecarConfig` re-export, and all public names                                                                                 |
| `midnight.py`         | `_get_midnight_crossover_tid()` (REST-based TID discovery), `_run_midnight_preflight()` (crossover verification + seed override), `fill_from_rest()` junction |
| `http_server.py`      | HTTP health endpoint (`/health`), SSE forming bar push, consumer API routes (`/ariadne`, `/catchup`, `/trades/gap-fill`), junction state tracking             |
| `checkpoint.py`       | Streaming checkpoint persistence: `_save_checkpoint()`, `_load_checkpoint()`, `_inject_checkpoints()`, `_extract_checkpoints()` to/from JSON files            |
| `clickhouse_write.py` | `_write_bars_batch_to_clickhouse()` with retry, Niffler dead-letter fallback on write failure, batch accumulation                                             |
| `config_helpers.py`   | Lazy-loaded config accessors: `_resilience_cfg()`, `_sidecar_cfg()`, `_backfill_cfg()` -- avoids circular imports at module level                             |
| `dead_letter.py`      | Niffler dead-letter serialization (`_niffler_dead_letter()`), immediate replay (`_niffler_immediate_replay()`), Charon periodic ferry                         |
| `notifications.py`    | Telegram alert functions: `_notify_watchdog_trigger()`, `_notify_charon_replay()`, `_notify_startup()`, `_notify_shutdown()`                                  |
| `recovery.py`         | `_restart_engine_with_recovery()` -- graceful engine shutdown with timeout guards, checkpoint extraction, fresh engine creation                               |
| `yesterday_signal.py` | `write_yesterday_signal()` / `read_yesterday_signal()` -- persistent signal file (`/var/tmp/`) for sidecar->kintsugi yesterday-repair handoff (STARTUP-02)    |

---

## Startup Sequence

The sidecar startup invariant (#294, #295) is documented in root [/CLAUDE.md](/CLAUDE.md). Here is which files implement each step:

| Step | Action                                         | Implemented In                              |
| ---- | ---------------------------------------------- | ------------------------------------------- |
| 1    | Start WS (buffered mode)                       | `__init__.py` (`_create_engine()`)          |
| 2    | Skip checkpoint injection                      | `__init__.py` (`_create_engine()`)          |
| 3    | Kill mutations + OPTIMIZE FINAL + DELETE today | `__init__.py` (`_create_engine()`)          |
| 4    | Midnight preflight (crossover TID)             | `midnight.py` (`_run_midnight_preflight()`) |
| 5    | `fill_from_rest` (fresh processors)            | `midnight.py` (`fill_from_rest()`)          |
| 6    | `sync_flush` (INSERT bars to CH)               | `__init__.py` (`_create_engine()`)          |

After startup, `_main_loop()` in `__init__.py` polls `engine.next_bar()` and writes via `clickhouse_write.py`.

## Writer Ownership

The sidecar **owns today (UTC)**. It has exclusive write access to today's bars in ClickHouse. Kintsugi owns yesterday and older -- this boundary is enforced by kintsugi's today-guard (#280). The sidecar's clean-slate DELETE on startup ensures zero conflicts: every restart recomputes today from the midnight crossover TID.

## Self-Healing Layers

| Concept  | File                 | Behavior                                                                |
| -------- | -------------------- | ----------------------------------------------------------------------- |
| Niffler  | `dead_letter.py`     | Serialize failed bars to Parquet, attempt immediate replay              |
| Charon   | `dead_letter.py`     | Periodic ferry of dead-letter files back to ClickHouse                  |
| Argus    | `__init__.py`        | Per-symbol liveness monitoring via phi-accrual failure detector         |
| Lethe    | `__init__.py`        | Ring buffer overflow detection -- alerts when bars are dropped          |
| Watchdog | `__init__.py`        | Stale trade flow detection -- triggers engine restart via `recovery.py` |
| Antaeus  | Rust `StreamManager` | Resilient WS reconnection with exponential backoff                      |

---

## Related

- [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) -- Parent: Python API layer
- [/CLAUDE.md](/CLAUDE.md) -- Hub: startup sequence invariant (#294, #295), writer ownership (#280)
- [/scripts/systemd/CLAUDE.md](/scripts/systemd/CLAUDE.md) -- systemd service config for `opendeviationbar-sidecar.service`
- [/deploy/CLAUDE.md](/deploy/CLAUDE.md) -- Monit watchdog, env files, defense-in-depth monitoring
- [/python/opendeviationbar/clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md) -- ClickHouse write operations used by `clickhouse_write.py`
