# Memory Audit #06: Kintsugi Repair Pipeline

**Scope**: `python/opendeviationbar/kintsugi.py` (2,469 lines) + called paths in
`backfill.py`, `orchestration/helpers.py`, `storage/parquet.py`, `clickhouse/bulk_operations.py`

**Production config**: MAX_CONCURRENT_REPAIRS=24 (Parquet), 4 (Vision), 2 (REST);
BATCH_DAYS=30, BATCH_WORKERS=4.

---

## End-to-End Memory Flow

```
kintsugi_pass()
  |
  |--- discover_shards()                         [A] 2x CH connections (with-block)
  |     |--- _discover_shards_stathera()          [A] OpenDeviationBarCache() per call
  |     |--- _discover_genesis_shards()           [A] OpenDeviationBarCache() per call
  |     |--- _enrich_shards()                     OK  (lightweight list mutation)
  |
  |--- _select_repair_candidates()
  |     |--- _get_dismissed_set()                 [A] OpenDeviationBarCache() per call
  |     |--- _get_rate_limited_set()              [A] OpenDeviationBarCache() per call
  |
  |--- classify: _shard_is_parquet_cached() x N   [B] TickStorage() per shard
  |--- classify: _shard_has_vision_available() x N OK  (lightweight Redis check)
  |
  |--- ThreadPoolExecutor(max_workers=24)
        |
        |--- repair_shard() per shard
              |--- _repair()
              |     |--- _kintsugi_cfg()           [C] KintsugiConfig() per _repair
              |     |--- days_to_repair loop
              |           |
              |           |--- _repair_under_lock() per day
              |                 |
              |                 |--- PATH 1: _try_parquet_day_replace()
              |                 |     |--- TickStorage()                   [B]
              |                 |     |--- storage.read_ticks() -> full DataFrame  [D]
              |                 |     |--- _select_trade_columns()         [E]
              |                 |     |--- _trade_batches() via .slice()   OK
              |                 |     |--- _stream_bars_with_processor()
              |                 |     |     |--- chunk.to_arrow()          [F]
              |                 |     |     |--- processor.process_trades_arrow()
              |                 |     |     |--- pl.from_arrow()           [F]
              |                 |     |     |--- bar_frames.append()       [G]
              |                 |     |--- pl.concat(bar_frames)           [G]
              |                 |     |--- store_bars_batch()              [H]
              |                 |
              |                 |--- PATH 2: _try_vision_day_replace()
              |                 |     |--- _stream_open_deviation_bars_binance()
              |                 |     |     |--- stream_binance_trades_arrow() [Rust I/O]
              |                 |     |     |--- _stream_bars_with_processor() [same as above]
              |                 |     |--- bar_frames list accumulation    [G]
              |                 |     |--- pl.concat(bar_frames)           [G]
              |                 |     |--- store_bars_batch()              [H]
              |                 |
              |                 |--- PATH 3: _process_gap_fallback_starttime()
              |                       |--- _fetch_aggtrades_rest() -> list[dict]  [I]
              |                       |--- split_trades_by_day() -> list[list[dict]] [J]
              |                       |--- processor.process_trades() -> list[dict]  [K]
              |                       |--- pl.DataFrame(all_bars)          [L]
              |                       |--- store_bars_batch()              [H]
              |
              |--- verify_repair()  (lightweight CH query)
              |--- _maybe_log_repair() (lightweight CH INSERT)
```

---

## Findings

### F-01: TickStorage instantiation per shard in classification [B]

- **File**: `kintsugi.py:670-689` (`_shard_is_parquet_cached`)
- **Category**: AVOID ALLOCATION
- **Priority**: P1
- **Current**: `TickStorage()` created per shard in `_shard_is_parquet_cached()`. With 100+ shards per pass, this creates 100+ TickStorage objects (each resolves `platformdirs` and sets up Path objects).
- **Also**: `_try_parquet_day_replace()` at line 1152 creates another `TickStorage()` per day.
- **Proposed**: Cache a module-level or thread-local `TickStorage` instance. `TickStorage.__init__` is lightweight (just path resolution) but it's called O(shards \* days) times.
- **Impact**: Low-medium. Each instantiation is ~microseconds but multiplied by hundreds.

### F-02: Discovery creates 4 separate OpenDeviationBarCache connections [A]

- **File**: `kintsugi.py:268-275`, `440-459`, `631-640`, `655-664`
- **Category**: AVOID ALLOCATION
- **Priority**: P2
- **Current**: `discover_shards()` path creates 4 separate `OpenDeviationBarCache()` instances via `with` blocks: `_discover_shards_stathera`, `_discover_genesis_shards`, `_get_dismissed_set`, `_get_rate_limited_set`. Each `OpenDeviationBarCache()` creates a `clickhouse_connect` client with a `urllib3.PoolManager(maxsize=20)`.
- **Proposed**: Pass a single cache instance through the discovery pipeline, or use the existing `_get_worker_cache()` thread-local pattern (currently only used in repair workers, not discovery).
- **Impact**: 4 HTTP connection pools opened/closed sequentially. ~80 fds transiently. Not a leak but unnecessary churn.

### F-03: REST fallback materializes ALL trades into `list[dict]` [I]

- **File**: `backfill.py:546` (`_process_gap_fallback_starttime`)
- **Category**: AVOID COPIES / LAZY EVALUATION
- **Priority**: P0
- **Current**: `_fetch_aggtrades_rest()` returns `list[dict]` (Rust -> Python dicts). For BTCUSDT, one day = ~2-3M aggTrades. Each dict has ~6 keys. At ~200 bytes/dict overhead, this is **~400-600 MB per day per REST repair**.
- **Then**: `split_trades_by_day(trades)` iterates the entire list creating sub-lists [J]. `processor.process_trades(day_trades)` processes each sub-list producing `list[dict]` output [K]. Finally `pl.DataFrame(all_bars)` [L] creates yet another copy.
- **With 2 concurrent REST repairs**: peak ~1.2 GB just for trade dicts alone.
- **Proposed**: The Rust side should return Arrow or Polars directly (already exists: `process_trades_arrow`). The `_process_gap_fallback_starttime` should use the streaming Arrow path like Parquet/Vision paths do, eliminating the dict intermediary entirely.
- **Impact**: Major. REST repairs are the most memory-intensive path and this is the root cause.

### F-04: Parquet path loads full day into memory then slices [D]

- **File**: `kintsugi.py:1174` (`_try_parquet_day_replace`)
- **Category**: LAZY EVALUATION
- **Priority**: P1
- **Current**: `storage.read_ticks(shard.symbol, day_start_ms, day_end_ms)` materializes the entire day's ticks into a single `pl.DataFrame`. For BTCUSDT this is ~2-3M rows x 8 columns = **~80-120 MB per day**. Then `_trade_batches()` uses `.slice()` (zero-copy view, good) but the full DF stays alive until `del day_ticks`.
- **Note**: `read_ticks_streaming()` already exists (`parquet.py:432`) and yields chunks — it's unused in kintsugi.
- **Proposed**: Replace `read_ticks()` with `read_ticks_streaming()` and feed chunks directly to `_stream_bars_with_processor()`. Eliminates the full-day materialization.
- **Impact**: With 24 concurrent Parquet repairs (max_concurrent_parquet=64 in config), peak = 24 _100MB = **2.4 GB just for tick data**. Streaming would cap at 24_ chunk_size _row_bytes = 24_ 100K \* 40B = **~96 MB**.

### F-05: `_select_trade_columns` creates a copy of the DataFrame [E]

- **File**: `kintsugi.py:1178`, `orchestration/helpers.py:209`
- **Category**: AVOID COPIES
- **Priority**: P2
- **Current**: `_select_trade_columns(day_ticks)` uses `pl.select()` which creates a new DataFrame with only the needed columns. Polars `select` on eager DataFrames creates new column references (cheap if columns are already Arrow-backed) but the intermediate DF holds both old and new until the old goes out of scope.
- **Proposed**: In the Parquet path, `read_ticks()` could accept a `columns` parameter to push column selection into Parquet reading (predicate pushdown at I/O level). This avoids ever materializing unused columns.
- **Impact**: Minor. The extra columns (if any) are typically only 1-2 beyond what's needed.

### F-06: Arrow round-trip in `_stream_bars_with_processor` [F]

- **File**: `orchestration/helpers.py:324-326`
- **Category**: AVOID COPIES
- **Priority**: P2
- **Current**: `chunk.to_arrow()` converts Polars -> Arrow, then `processor.process_trades_arrow(chunk_arrow)` processes, then `pl.from_arrow(bars_arrow)` converts Arrow -> Polars. The Polars -> Arrow conversion is zero-copy (Polars uses Arrow internally). Arrow -> Polars is also effectively zero-copy. So this is already well-optimized.
- **Assessment**: No action needed. The "copies" are actually just pointer exchanges.

### F-07: Vision path accumulates all bar_frames before concat [G]

- **File**: `kintsugi.py:1294-1331` (`_try_vision_day_replace`)
- **Category**: AVOID COPIES / LAZY EVALUATION
- **Priority**: P1
- **Current**: Both Parquet and Vision paths collect `bar_frames: list[pl.DataFrame]` then `pl.concat(bar_frames, rechunk=False)` before writing. The `rechunk=False` is correct (avoids a full copy). However, the entire day's bars must be in memory before the CH write begins.
- **For Parquet path** (`kintsugi.py:1193-1206`): bar_frames are accumulated, then concatenated, then written. A typical BTCUSDT@250 day produces ~2K bars x 32 columns x 8 bytes = ~0.5 MB. This is fine.
- **For Vision path**: same pattern, same sizes. Also fine.
- **Assessment**: The bar DataFrames themselves are small (<1 MB/day). The real memory issue is the trade data, not the bar output. Low priority.

### F-08: KintsugiConfig re-instantiated on each `_kintsugi_cfg()` call [C]

- **File**: `kintsugi.py:81-83`
- **Category**: AVOID ALLOCATION
- **Priority**: P2
- **Current**: `_kintsugi_cfg()` creates a new `KintsugiConfig()` (pydantic-settings BaseSettings) on every call. This is called:
  - Once per `_repair()` (line 998) - cached in `cfg` local var (good)
  - Once per `_emit_telemetry` for rotation check (line 120) - called dozens of times per pass
  - Multiple times in `_notify_kintsugi_failures` (lines 2084, 2091, 2092)
  - Multiple times in Telegram summary (lines 2027, 2035, 2037)
- **Proposed**: Cache at module level with lazy init, or use `functools.lru_cache`. Pydantic BaseSettings reads env vars on each instantiation.
- **Impact**: ~10-20 instantiations per pass. Each reads all env vars. Not a memory issue but CPU waste.

### F-09: `_process_gap_fallback_starttime` creates per-call OpenDeviationBarCache [H]

- **File**: `backfill.py:595`
- **Category**: AVOID ALLOCATION
- **Priority**: P1
- **Current**: REST fallback path creates `OpenDeviationBarCache()` via a `with` block per day repair. Since REST repairs are called from within `_repair_under_lock` which runs in thread pool workers, this creates a new CH connection per day instead of using the thread-local `_get_worker_cache()`.
- **Contrast**: Parquet and Vision paths correctly use `_get_worker_cache()` (lines 1215, 1349).
- **Proposed**: Pass cache instance or use `_get_worker_cache()` in backfill.py's REST path. The thread-local pattern is already proven.
- **Impact**: Medium. REST repairs are slow (minutes per day) so the per-call connection overhead is small relative to I/O, but it's inconsistent with the other paths.

### F-10: Full shard discovery before any repair [LAZY EVALUATION]

- **File**: `kintsugi.py:1698-1706`
- **Category**: LAZY EVALUATION
- **Priority**: P2
- **Current**: `discover_shards()` runs 2 CH queries per (symbol, threshold) pair to build a complete shard list before any repair starts. Classification (`_shard_is_parquet_cached`) also runs for all candidates.
- **Assessment**: Discovery is I/O-bound (ClickHouse queries) not memory-bound. The Shard objects themselves are tiny (~200 bytes each). Even 1000 shards = 200 KB. Interleaving discovery and repair would add complexity for negligible memory savings.
- **Impact**: Negligible. Discovery memory is dwarfed by repair memory.

### F-11: `split_trades_by_day` copies trades into sub-lists [J]

- **File**: `validation/day_mode/guards.py:12-37`
- **Category**: AVOID COPIES
- **Priority**: P1 (when triggered via REST path)
- **Current**: Takes `list[dict]` and produces `list[list[dict]]` where each sub-list is a separate day. The dict objects are shared (no deep copy), but the list structure is new. The assertion loop at the end iterates all trades again.
- **In kintsugi context**: This only fires in the REST fallback path (`_process_gap_fallback_starttime`), and kintsugi repairs are always single-day (`start_ms` to `end_ms` within one UTC day), so `split_trades_by_day` will produce exactly 1 group. The split is a no-op but still iterates all trades.
- **Proposed**: Since `_repair_under_lock` already passes single-day boundaries (`day_start_ms`, `day_end_ms`), the `_process_gap_fallback_starttime` call from kintsugi will never span multiple days. The split is wasted work. Could skip it when caller guarantees single-day.
- **Impact**: One extra pass over 2-3M dicts = ~10-20ms. Memory impact is near-zero (shared references).

---

## Peak Memory Estimation: Typical Repair Pass (24 Concurrent Parquet Shards)

### Scenario: 24 BTCUSDT shards, all Parquet-cached

| Component                             | Per-Worker      | x24 Workers    | Notes                      |
| ------------------------------------- | --------------- | -------------- | -------------------------- |
| Tick DataFrame (read_ticks, full day) | 80-120 MB       | **1.9-2.9 GB** | F-04: full materialization |
| Trade columns after select            | ~60 MB          | ~1.4 GB        | Shares backing data        |
| Arrow chunks (to_arrow)               | ~0 (zero-copy)  | ~0             | F-06: pointer exchange     |
| Bar output (bar_frames + concat)      | ~0.5 MB         | ~12 MB         | Small relative to trades   |
| Processor state (Rust)                | ~1 MB           | ~24 MB         | Rust heap, bounded         |
| Thread-local CH cache                 | ~2 MB           | ~48 MB         | Connection pool per thread |
| **Worker subtotal**                   | **~140-180 MB** | **3.4-4.3 GB** |                            |

| Component                                      | Total          | Notes                            |
| ---------------------------------------------- | -------------- | -------------------------------- |
| Discovery (Shard objects, dismissed set, etc.) | ~5 MB          | Negligible                       |
| ThreadPoolExecutor overhead                    | ~10 MB         | Thread stacks                    |
| Python interpreter + imports                   | ~100 MB        | Base cost                        |
| **Grand total estimate**                       | **3.5-4.4 GB** | At 24 concurrent Parquet repairs |

### Scenario: 2 Concurrent REST Shards (worst case per-worker)

| Component                      | Per-Worker      | x2 Workers     | Notes                  |
| ------------------------------ | --------------- | -------------- | ---------------------- |
| `list[dict]` from Rust         | 400-600 MB      | **0.8-1.2 GB** | F-03: main bottleneck  |
| `all_bars: list[dict]`         | ~0.5 MB         | ~1 MB          | Output bars are small  |
| `pl.DataFrame(all_bars)`       | ~0.5 MB         | ~1 MB          | Copy of above          |
| OpenDeviationBarCache per call | ~2 MB           | ~4 MB          | F-09: not thread-local |
| **Worker subtotal**            | **~400-600 MB** | **0.8-1.2 GB** |                        |

### Combined Worst Case (24 Parquet + 2 REST)

**Peak estimate: ~4.3-5.5 GB** (well within bigblack's 64 GB but relevant for MemoryMax=4G systemd guard)

---

## Priority Summary

| ID   | Priority | Category      | Summary                                                             | Est. Savings          |
| ---- | -------- | ------------- | ------------------------------------------------------------------- | --------------------- |
| F-03 | **P0**   | COPIES + LAZY | REST path materializes all trades as `list[dict]` (~500 MB/day)     | 400-600 MB/worker     |
| F-04 | **P1**   | LAZY          | Parquet path loads full day; `read_ticks_streaming` exists unused   | 60-80 MB/worker       |
| F-01 | **P1**   | ALLOCATION    | TickStorage() created per shard + per day                           | Allocation churn      |
| F-09 | **P1**   | ALLOCATION    | REST path creates OpenDeviationBarCache per call (not thread-local) | FD + connection churn |
| F-11 | **P1**   | COPIES        | `split_trades_by_day` unnecessary for single-day kintsugi repairs   | 10-20ms CPU waste     |
| F-02 | **P2**   | ALLOCATION    | Discovery creates 4 separate CH connections                         | 80 transient FDs      |
| F-05 | **P2**   | COPIES        | `_select_trade_columns` could push down to Parquet reader           | Minor                 |
| F-08 | **P2**   | ALLOCATION    | `KintsugiConfig()` re-instantiated ~20x per pass                    | CPU waste             |
| F-06 | **P2**   | COPIES        | Arrow round-trip (already zero-copy, no action needed)              | None                  |
| F-07 | **P2**   | LAZY          | Bar frame accumulation before concat (bars are tiny)                | Negligible            |
| F-10 | **P2**   | LAZY          | Full discovery before repair (shard objects are tiny)               | Negligible            |

---

## Recommended Fix Order

1. **F-03** (P0): Convert REST fallback to Arrow/streaming path. The `_fetch_aggtrades_rest` -> `list[dict]` -> `process_trades()` -> `list[dict]` -> `pl.DataFrame()` chain is the single largest memory consumer. Replace with Rust returning Arrow directly + `_stream_bars_with_processor`.

2. **F-04** (P1): Replace `read_ticks()` with `read_ticks_streaming()` in `_try_parquet_day_replace`. The streaming reader already exists; the kintsugi Parquet path just needs to feed chunks directly to `_stream_bars_with_processor` instead of loading the full day.

3. **F-01 + F-09** (P1): Thread-local TickStorage and consistent use of `_get_worker_cache()` across all three data source paths. Simple refactor with proven pattern.

4. **F-08** (P2): Cache `KintsugiConfig` with `lru_cache` or module-level lazy singleton.
