# Roadmap: opendeviationbar-py

## Milestones

- ✅ **v1.0 Junction Fix** -- Phases 1-3 (shipped 2026-03-22) -- [archive](milestones/v1.0-ROADMAP.md)
- ✅ **v1.4 Zero-Gap Zero-Overlap Guarantee** -- Phases 1-6 (shipped 2026-03-23) -- [archive](milestones/v1.4-ROADMAP.md)
- ✅ **v1.5 Sidecar Startup Architecture** -- Phases 1-3 (shipped 2026-03-23) -- [archive](milestones/v1.5-ROADMAP.md)
- ✅ **v2.0 Seeder as Sole REST Owner + Microsecond Alignment** -- Phases 1-5 (shipped 2026-03-24) -- [archive](milestones/v2.0-ROADMAP.md)
- ✅ **v3.0 Streaming Reliability + Pipeline Cleanup** -- Phases 1-5.1 (shipped 2026-03-24) -- [archive](milestones/v3.0-ROADMAP.md)
- ✅ **v4.0 Microsecond Column Rename + Deploy Gap Repair** -- Phases 1-5 (shipped 2026-03-25) -- [archive](milestones/v4.0-ROADMAP.md)
- ✅ **v5.0 Seeder Day-Boundary Completeness + Kintsugi Parquet Trust** -- Phases 1-5 (shipped 2026-03-25, v13.54.0)
- ✅ **v5.1 SEED-001 Fix + Heartbeat Unified View** -- Phases 6-7 (shipped 2026-03-25, v13.55.0)
- ✅ **v6.0 Data Integrity & Checksum Verification** -- Phases 8-12 (shipped 2026-03-26, v13.56.1)
- ✅ **v7.0 Aion Mode -- Genesis-Continuous for Crypto** -- Phases 13-19 (shipped 2026-03-27) -- [archive](milestones/v7.0-ROADMAP.md)
- ✅ **v7.1 Aion Hardening -- Footgun Elimination + Startup Fix** -- Phases 20-22 (shipped 2026-03-27) -- [archive](milestones/v7.1-ROADMAP.md)
- 🚧 **v7.2 Memory Efficiency -- Zero-OOM Sidecar** -- Phases 23-25 (in progress)

## Phases

<details>
<summary>✅ v5.0 Seeder Day-Boundary Completeness (Phases 1-5) - SHIPPED 2026-03-25</summary>

- [x] **Phase 1: Crossover TID Utility** - Extract shared crossover lookup callable by seeder, kintsugi, and heartbeat
- [x] **Phase 2: Seeder Day-Boundary Backfill** - Seeder backfills yesterday's Parquet file after midnight to crossover TID boundary
- [x] **Phase 3: Kintsugi Parquet Trust** - Kintsugi verifies Parquet completeness before using as repair source
- [x] **Phase 4: Heartbeat Coverage** - Yesterday completion check expanded to all streaming symbols
- [x] **Phase 5: Documentation** - Root cause and fix documented across CLAUDE.md network

</details>

<details>
<summary>✅ v5.1 SEED-001 Fix + Heartbeat Unified View (Phases 6-7) - SHIPPED 2026-03-25</summary>

- [x] **Phase 6: Fix SEED-001 Parquet-First Bar Processing** - Process Parquet ticks into bars instead of skipping past them
- [x] **Phase 7: Heartbeat Unified Per-Symbol View** - Merge Yesterday + Junction into exception-only display with TID deltas

</details>

<details>
<summary>✅ v6.0 Data Integrity & Checksum Verification (Phases 8-12) - SHIPPED 2026-03-26</summary>

- [x] **Phase 8: Checksum Spike** - Prove Rust SHA-256 callable from Python with correct results and edge case handling
- [x] **Phase 9: Verification Registry** - Centralized per-file verification state store as SSoT
- [x] **Phase 10: Seeder Checksum Integration** - Wire SHA-256 into seeder download pipeline with retry and audit trail
- [x] **Phase 11: Integrity Sweep** - Audit existing Parquet cache against Vision CHECKSUM files
- [x] **Phase 12: Telemetry & Heartbeat** - Heartbeat checksum health, LOUD alerts, local telemetry log

</details>

### v7.0 Aion Mode -- Genesis-Continuous for Crypto

- [x] **Phase 13: Registry + Mode Enum** - Ouroboros mode SSoT in symbols.toml with Aion variant, plumbed through registry to all consumers
- [x] **Phase 14: Rust Core -- Gap Fields + Conditional Reset** - Bar struct gap awareness fields and conditional midnight reset based on ouroboros_mode
- [x] **Phase 15: Oracle Validation** - Bit-exact oracle tests proving Aion correctness: single-day equivalence, multi-day continuity, gap flag lifecycle
- [x] **Phase 16: Sidecar Aion Startup** - Checkpoint-resume startup path for Aion symbols, skip clean-slate DELETE and midnight preflight
- [x] **Phase 17: Kintsugi Aion Repair** - Genesis-forward reconciliation with TID-based writer boundary for Aion symbols
- [x] **Phase 18: Monitor + Validation Gates** - Rolling-window freshness replaces yesterday completion, conditional day-mode gate, mode-agnostic Stathera
- [x] **Phase 19: Fleet Recompute + Telemetry** - Wipe Day-mode crypto bars, recompute as Aion on bigblack, update all ops artifacts, validate Stathera

## Phase Details

<details>
<summary>✅ v5.0 Phase Details (Phases 1-5)</summary>

### Phase 1: Crossover TID Utility

**Goal**: Any module can look up the crossover TID for a given symbol and date without importing sidecar internals
**Depends on**: Nothing (first phase)
**Requirements**: XOVER-01, XOVER-02
**Plans:** 1 plan

Plans:

- [x] 01-01-PLAN.md -- Extract crossover TID lookup into shared module, rewire sidecar and heartbeat

**Success Criteria** (what must be TRUE):

1. Seeder, kintsugi, and heartbeat can call a shared `get_crossover_tid(symbol, date)` without importing anything from `sidecar.py`
2. The utility reuses the existing `_get_midnight_crossover_tid` + `_verify_crossover_from_seed` logic (refactored out, not reimplemented)
3. Existing sidecar midnight preflight still works identically after the extraction (no behavioral change)

### Phase 2: Seeder Day-Boundary Backfill

**Goal**: Yesterday's Parquet file contains every trade from 00:00:00.000 through the last trade before midnight crossover
**Depends on**: Phase 1
**Requirements**: SEED-01, SEED-02
**Plans:** 1 plan

Plans:

- [x] 02-01-PLAN.md -- Day-boundary backfill: detect incomplete yesterday Parquet, fetch missing trailing trades via REST

**Success Criteria** (what must be TRUE):

1. After the seeder's post-midnight run, yesterday's Parquet `max(agg_trade_id)` equals `crossover_tid - 1`
2. The seeder detects an incomplete yesterday file and backfills only the missing trailing trades (the ~5 min gap)
3. A complete yesterday file (max TID already at crossover boundary) is not re-fetched or modified

### Phase 3: Kintsugi Parquet Trust

**Goal**: Kintsugi only uses Parquet as a repair source when it can prove the file covers the full day
**Depends on**: Phase 1
**Requirements**: TRUST-01, TRUST-02
**Plans:** 1 plan

Plans:

- [x] 03-01-PLAN.md -- Parquet completeness gate + orphan bar marking in Parquet/Vision repair paths

**Success Criteria** (what must be TRUE):

1. Before day-recompute, kintsugi verifies `max(agg_trade_id) >= crossover_tid - 1` on the Parquet file
2. If Parquet verification fails, kintsugi falls back to Vision/REST instead of using incomplete data
3. Orphan bars emitted from Parquet and Vision repair paths have `is_orphan=True` set correctly

### Phase 4: Heartbeat Coverage

**Goal**: Yesterday completion monitoring covers all streaming symbols, not just a hardcoded subset
**Depends on**: Phase 1
**Requirements**: HB-01
**Plans:** 1 plan

Plans:

- [x] 04-01-PLAN.md -- Dynamic symbol list from registry for yesterday completion check

**Success Criteria** (what must be TRUE):

1. Yesterday completion check runs for every symbol in the streaming config (all 17+ Binance symbols)
2. An incomplete yesterday for any symbol triggers the existing LOUD Telegram alert

### Phase 5: Documentation

**Goal**: Root cause of the day-boundary gap and the v5.0 fix are documented across the CLAUDE.md network
**Depends on**: Phase 2, Phase 3, Phase 4
**Requirements**: DOC-01

Plans:

- [x] 05-01-PLAN.md -- Update CLAUDE.md network with v5.0 day-boundary completeness docs

**Success Criteria** (what must be TRUE):

1. `scripts/CLAUDE.md` documents seeder day-boundary semantics and the post-midnight backfill mechanism
2. `python/opendeviationbar/kintsugi/CLAUDE.md` documents the Parquet trust verification gate
3. `scripts/heartbeat/CLAUDE.md` documents the expanded symbol coverage (no more hardcoded pair)
4. Root `CLAUDE.md` references the day-boundary completeness guarantee in the monitoring table

</details>

<details>
<summary>✅ v5.1 Phase Details (Phases 6-7)</summary>

### Phase 6: Fix SEED-001 Parquet-First Bar Processing

**Goal:** On sidecar restart, process Parquet ticks into bars via Rust processor instead of skipping past them. Eliminates 274K+ trade gaps caused by SEED-001 advancing the REST seed past deleted bars.
**Requirements**: SEED-FIX-01, SEED-FIX-02
**Depends on:** Phase 5 (v5.0 shipped)
**Plans:** 1 plan

Plans:

- [x] 06-01-PLAN.md -- Process Parquet ticks into bars and seed with last_completed_bar_tid

### Phase 7: Heartbeat Unified Per-Symbol View

**Goal:** Merge Yesterday + Junction sections into a single exception-only display covering all registry symbols. Yesterday expands from BTC/ETH to all enabled Binance spot symbols (from `symbols.toml`). Junction replaces raw TID dumps with delta-only (0=clean). When all symbols are clean, show one summary line. Only expand individual symbols when issues exist.
**Requirements**: HB-VIEW-01, HB-VIEW-02
**Depends on:** Phase 6 (v5.1)
**Plans:** 1 plan

Plans:

- [x] 07-01-PLAN.md -- Merge Yesterday + Junction into unified exception-only Per-Symbol Health section

</details>

<details>
<summary>✅ v6.0 Phase Details (Phases 8-12)</summary>

### Phase 8: Checksum Spike

**Goal**: Rust checksum.rs SHA-256 fetch+verify is proven callable from Python with correct results, acceptable performance, and graceful edge case handling
**Depends on**: Phase 7 (v5.1 shipped)
**Requirements**: SPIKE-01, SPIKE-02, SPIKE-03
**Success Criteria** (what must be TRUE):

1. A Python script can call the Rust SHA-256 checksum function and verify a real Vision ZIP download against its .CHECKSUM file, producing a pass/fail result
2. Checksum verification adds less than 10% wall-clock overhead to a typical Vision ZIP download-and-extract cycle
3. A 404 CHECKSUM response (old data) results in a soft-skip (not a crash), and a corrupted download results in a hard fail with a clear error message
4. Network timeout during CHECKSUM fetch results in a soft-skip with a warning, not a pipeline halt
   **Plans:** 1/1 plans complete

Plans:

- [x] 08-01-PLAN.md -- PyO3 checksum bindings + offline tests + network spike validation

### Phase 9: Verification Registry

**Goal**: A centralized store tracks per-file checksum verification state as the single source of truth, queryable by seeder, sweep, and heartbeat
**Depends on**: Phase 8
**Requirements**: TELEM-03
**Success Criteria** (what must be TRUE):

1. The registry stores per-file records (symbol, date, sha256, verified_at, source) and can be queried by symbol or date range
2. The seeder can write a verification record after each successful checksum, and the sweep can read existing records to skip already-verified files
3. The registry persists across process restarts (file-based JSON or SQLite, not in-memory)
   **Plans:** 1 plan

Plans:

- [x] 09-01: Verification registry -- data model, read/write API, persistence

### Phase 10: Seeder Checksum Integration

**Goal**: Every Vision ZIP downloaded by tick_cache_seeder is SHA-256 verified against the .CHECKSUM file before being accepted into the Parquet cache
**Depends on**: Phase 8, Phase 9
**Requirements**: SEEDER-01, SEEDER-02, SEEDER-03
**Success Criteria** (what must be TRUE):

1. Every Vision ZIP download is automatically verified against the .CHECKSUM file before the seeder processes it into Parquet
2. A checksum mismatch triggers up to 2 re-downloads; if all retries fail, a LOUD Telegram alert fires with symbol, date, and hash details
3. Each successful verification is recorded in the verification registry with symbol, date, checksum, and timestamp
4. The seeder's Telegram summary report includes checksum pass/fail/skip counts
   **Plans:** 1/1 plans complete

Plans:

- [x] 10-01: Wire checksum verification into seeder download pipeline with retry, registry write, and Telegram reporting

### Phase 11: Integrity Sweep

**Goal**: A script can audit all existing Parquet cache files against Vision CHECKSUM files, with incremental mode to avoid redundant re-verification
**Depends on**: Phase 9
**Requirements**: SWEEP-01, SWEEP-02, SWEEP-03
**Success Criteria** (what must be TRUE):

1. Running the sweep script re-downloads Vision CHECKSUM files and compares SHA-256 hashes against the original ZIPs used to create each Parquet file
2. Incremental mode reads the verification registry and skips files that are already verified, only checking new or unverified files
3. The sweep produces a summary report with pass/fail/skip counts broken down by symbol and date range
4. A failed file is flagged in the registry and optionally triggers a Telegram alert
   **Plans:** 1/1 plans complete

Plans:

- [x] 11-01: Integrity sweep script -- full and incremental modes, registry integration, summary report

### Phase 12: Telemetry & Heartbeat

**Goal**: Heartbeat reports checksum verification health per symbol with LOUD Telegram alerts on any integrity failure, and all checksum operations are logged locally for post-mortem
**Depends on**: Phase 9, Phase 10
**Requirements**: TELEM-01, TELEM-02, TELEM-04
**Success Criteria** (what must be TRUE):

1. The 5-minute heartbeat includes a checksum health section showing last verified date and any failures per symbol
2. Any checksum failure (seeder or sweep) triggers a LOUD Telegram alert with symbol, date, expected hash, and actual hash
3. All checksum operations (verify, skip, fail, retry) are written to a local telemetry log file for post-mortem analysis
4. When all symbols have clean checksums, the heartbeat shows a single summary line (exception-only pattern from Phase 7)
   **Plans:** 1/1 plans complete

Plans:

- [x] 12-01: Heartbeat checksum health check + LOUD alert + local telemetry log

</details>

### Phase 13: Registry + Mode Enum

**Goal**: Every consumer (sidecar, kintsugi, heartbeat, validation) can query any symbol's ouroboros mode from the registry, with Aion as a first-class variant alongside Day
**Depends on**: Phase 12 (v6.0 shipped)
**Requirements**: REG-01, REG-02, REG-03, REG-04, REG-05
**Plans:** 2/2 plans complete

Plans:

- [x] 13-01-PLAN.md -- OuroborosMode.AION enum + symbols.toml ouroboros_mode field + SymbolEntry wiring + crypto-only validation
- [x] 13-02-PLAN.md -- Expand all public API signatures to accept ouroboros_mode="aion"

**Success Criteria** (what must be TRUE):

1. `symbols.toml` has an `ouroboros_mode` field on every symbol entry, defaulting to `"day"` for existing symbols and set to `"aion"` for crypto symbols
2. Python code can call `get_symbol_entry("BTCUSDT").ouroboros_mode` and receive `OuroborosMode.AION`
3. `get_open_deviation_bars("BTCUSDT", ..., ouroboros_mode="aion")` accepts the mode parameter without error
4. Attempting to register `ouroboros_mode = "aion"` on a forex symbol (`asset_class != "crypto"`) raises a validation error

### Phase 14: Rust Core -- Gap Fields + Conditional Reset

**Goal**: The Rust processor produces bars with gap awareness metadata and conditionally skips midnight reset for Aion-mode symbols
**Depends on**: Phase 13
**Requirements**: CORE-01, CORE-02, CORE-03, CORE-04
**Success Criteria** (what must be TRUE):

1. An `OpenDeviationBar` constructed from trades with a TID gap has `has_gap=true`, `gap_trade_count > 0`, and `max_gap_duration_us > 0`
2. An `OpenDeviationBar` constructed from continuous trades has `has_gap=false`, `gap_trade_count=0`, `max_gap_duration_us=0`
3. When processor config specifies Aion mode, bars span UTC midnight boundaries without reset -- no orphan bar emitted at 00:00
4. When processor config specifies Day mode, midnight reset behavior is unchanged from current production (backward compatible)
5. `is_exchange_gap` is set to `true` only when REST backfill confirms the source exchange has no trades for the gap range
   **Plans:** 2/2 plans complete

Plans:

- [ ] 14-01-PLAN.md -- Gap fields on OpenDeviationBar struct + gap detection + Arrow/PyO3 export
- [ ] 14-02-PLAN.md -- Conditional midnight reset: OuroborosMode enum + Aion skip in trade_dispatch

### Phase 15: Oracle Validation

**Goal**: Bit-exact oracle tests prove Aion mode correctness before any service integration -- single-day equivalence, multi-day continuity, and gap flag lifecycle
**Depends on**: Phase 13, Phase 14
**Requirements**: VAL-01, VAL-02, VAL-03
**Success Criteria** (what must be TRUE):

1. Aion mode produces bit-exact identical bars as Day mode for trades within a single UTC day (excluding the orphan bar at midnight boundary)
2. Aion bars spanning a UTC midnight have continuous trade IDs across the boundary, no orphan bar, and correct microstructure features (VWAP, tick count)
3. A bar constructed with a TID gap has `has_gap=true`; after Puck/Kintsugi repair recomputes the bar from complete trades, the replacement bar has `has_gap=false`
   **Plans:** 0/1 plans complete

Plans:

- [ ] 15-01: TBD

### Phase 16: Sidecar Aion Startup

**Goal**: Sidecar starts Aion symbols via checkpoint-resume without wiping today's bars, while Puck inline gap fill remains fully operational
**Depends on**: Phase 14, Phase 15
**Requirements**: SIDE-01, SIDE-02, SIDE-03
**Success Criteria** (what must be TRUE):

1. Sidecar startup for an Aion symbol resumes from the last checkpoint TID without running `_run_midnight_preflight()` or `ALTER TABLE DELETE` for today
2. Sidecar startup for a Day-mode symbol still runs the existing clean-slate DELETE + midnight preflight path (backward compatible)
3. Puck detects and fills TID gaps on Aion symbols with sub-3s latency, identical to Day-mode behavior
   **Plans:** 2/2 plans complete

Plans:

- [x] 16-01-PLAN.md -- Rust-side per-symbol OuroborosMode wiring (LiveEngineConfig + symbol_task + puck + PyO3)
- [x] 16-02-PLAN.md -- Python sidecar startup branching (Aion checkpoint-resume, Day clean-slate, conditional cross-midnight filters)

### Phase 17: Kintsugi Aion Repair

**Goal**: Kintsugi repairs Aion symbol gaps by reprocessing from the gap point forward, with TID-based writer boundaries instead of UTC day boundaries
**Depends on**: Phase 14, Phase 15
**Requirements**: KINT-01, KINT-02, KINT-03
**Success Criteria** (what must be TRUE):

1. Kintsugi Aion repair reprocesses from the gap's anchor TID forward to the next known bar, not by recomputing an entire UTC day
2. Kintsugi writer boundary for Aion symbols is TID-based (sidecar owns TIDs beyond its committed_floors); the UTC-day today-guard is skipped
3. Niffler and Charon dead-letter recovery fire for Aion symbols identically to Day-mode -- no mode-specific branching in dead-letter paths
   **Plans:** 2/2 plans complete

Plans:

- [x] 17-01-PLAN.md -- Per-symbol ouroboros_mode wiring + conditional today-guard for Aion
- [x] 17-02-PLAN.md -- Aion TID-range repair strategy (\_repair_aion)

### Phase 18: Monitor + Validation Gates

**Goal**: Heartbeat, pre-write validation, and Stathera auditing work correctly for both Aion and Day-mode symbols without manual mode configuration
**Depends on**: Phase 13, Phase 14
**Requirements**: MON-01, MON-02, MON-03
**Plans:** 2/2 plans complete

**Success Criteria** (what must be TRUE):

1. Heartbeat for Aion symbols reports rolling-window freshness (last bar within N minutes) instead of yesterday completion status
2. The `bulk_operations.py` pre-write gate skips cross-midnight rejection for Aion symbols while still enforcing it for Day-mode symbols
3. Stathera continuity audit produces identical gap/overlap reports regardless of ouroboros_mode -- trade-ID continuity is mode-agnostic

Plans:

- [x] 18-01-PLAN.md -- Per-symbol pre-write gate + cross-midnight oracle Aion exclusion
- [ ] 18-02-PLAN.md -- Aion rolling-window freshness check in heartbeat yesterday module

### Phase 19: Fleet Recompute + Telemetry

**Goal**: All crypto symbols are recomputed as Aion on bigblack with zero Stathera gaps, and all operational artifacts reflect the new mode
**Depends on**: Phase 16, Phase 17, Phase 18
**Requirements**: RECOMP-01, RECOMP-02, RECOMP-03, RECOMP-04, RECOMP-05, VAL-04
**Plans:** 2/2 plans complete

**Success Criteria** (what must be TRUE):

1. All Day-mode crypto bar partitions are dropped from ClickHouse via `DROP PARTITION` per (symbol, threshold) pair, and full recompute populates Aion bars for all 18 crypto symbols
2. Heartbeat Stathera check confirms 0-gap 0-overlap for every Aion crypto symbol after recompute
3. Monit monitors, systemd unit configs, and Telegram heartbeat message formats reflect Aion mode (no stale Day-mode references for crypto)
4. Day-mode telemetry artifacts (kintsugi_log, population_checkpoints, heartbeat state for crypto) are wiped clean
5. No internal Day-mode references for crypto symbols remain in Stathera queries, validation gates, or diagnostic scripts

Plans:

- [x] 19-01-PLAN.md -- Update diagnostic scripts, deploy config, schema comments for Aion mode
- [x] 19-02-PLAN.md -- Fleet recompute script + bigblack migration execution + Stathera verification

### v7.1 Aion Hardening -- Footgun Elimination + Startup Fix

- [x] **Phase 20: Aion-Aware Sidecar Startup** - Preserve pre-midnight bars, seed from max(committed_tid, crossover_tid), zero midnight-boundary Stathera gaps (#310) (completed 2026-03-27)
- [x] **Phase 21: Universal Write Path ouroboros_mode** - Wire per-symbol mode in backfill.py, dead_letter.py, count_bounded.py; verify zero global fallback for crypto (#311) (completed 2026-03-27)
- [x] **Phase 22: Monitor Hardening + Schema** - Freshness query mode filter, sidecar restart investigation, CH DEFAULT review (#312) (completed 2026-03-27)

## Phase Details

### Phase 20: Aion-Aware Sidecar Startup

**Goal**: Sidecar startup for Aion symbols preserves pre-midnight bars and produces zero midnight-boundary Stathera gaps on restart
**Depends on**: Nothing (first phase)
**Requirements**: START-01, START-02, START-03
**Success Criteria** (what must be TRUE):

1. `fill_from_rest` for Aion symbols seeds at `max(last_committed_bar_tid, crossover_tid - 1)`, not always `crossover_tid - 1`
2. Clean-slate DELETE for Aion symbols only deletes today's bars (preserves pre-midnight bars)
3. After sidecar restart spanning midnight, Stathera audit shows zero gaps for all Aion (symbol, threshold) pairs

Plans:

- [x] 20-01-PLAN.md -- Aion max-seed (crossover TID) + targeted today-DELETE before fill_from_rest

### Phase 21: Universal Write Path ouroboros_mode

**Goal**: Every `store_bars_batch`/`store_bars_bulk` call site in the codebase passes explicit `ouroboros_mode` — zero global fallback for crypto symbols
**Depends on**: Nothing (independent)
**Requirements**: WRITE-01, WRITE-02, WRITE-03, WRITE-04
**Plans:** 1/1 plans complete

**Success Criteria** (what must be TRUE):

1. `backfill_gap()` accepts `ouroboros_mode` parameter and passes it to all 3 internal `store_bars_batch` calls
2. `replay_dead_letters()` resolves per-symbol mode from registry before writing
3. `count_bounded.py` passes `ouroboros_mode` to both `store_bars_batch` calls
4. `grep -rn "get_operational_ouroboros_mode" python/ scripts/` shows zero call sites that could produce crypto bars with wrong mode

Plans:

- [x] 21-01-PLAN.md -- Wire per-symbol ouroboros_mode in backfill.py, dead_letter.py, count_bounded.py

### Phase 22: Monitor Hardening + Schema

**Goal**: Heartbeat freshness, sidecar stability, and CH schema enforce correct Aion mode without silent fallbacks
**Depends on**: Nothing (independent)
**Plans:** 2/2 plans complete
**Requirements**: MON-04, MON-05, SCHEMA-01
**Success Criteria** (what must be TRUE):

1. `_check_aion_freshness()` query includes `AND ouroboros_mode = 'aion'` filter to prevent false-positive freshness from day-mode bars
2. Sidecar restart root cause identified and fixed (target: stable PID for >1 hour under normal operation)
3. CH `ouroboros_mode` column either has no DEFAULT (INSERT-time error) or has INSERT-time validation that warns on mismatch with registry

Plans:

- [x] 22-01-PLAN.md -- Aion freshness query filter + Python-side ouroboros_mode INSERT validation
- [x] 22-02-PLAN.md -- Sidecar restart investigation + heartbeat restart telemetry

### v7.2 Memory Efficiency -- Zero-OOM Sidecar

- [ ] **Phase 23: Arrow FFI drain_bars** - Replace dict-based drain_bars with Arrow RecordBatch FFI, zero-copy Polars slicing in sync_flush (#313)
- [ ] **Phase 24: Hot-Path Allocation Reduction** - Single CH cache instance, eliminate per-flush dict→DataFrame, cache symbol_entries at startup
- [x] **Phase 25: Lazy Evaluation + Cache Efficiency** - scan_parquet for dedup, in-place sort for Stathera diagnostic (completed 2026-03-27)

## Phase Details (v7.2)

### Phase 23: Arrow FFI drain_bars

**Goal**: drain_bars returns Arrow RecordBatch instead of list[dict], reducing peak RSS during sync_flush from ~2.6G to ~0.9G
**Depends on**: Nothing
**Requirements**: ARROW-01, ARROW-02, ARROW-03
**Success Criteria** (what must be TRUE):

1. `StreamManager.drain_bars_arrow()` returns Arrow RecordBatch with all bar fields + \_symbol/\_threshold metadata
2. `_sync_flush_rest_bars` groups RecordBatch by (symbol, threshold) using Polars zero-copy filter/slice
3. Peak sidecar RSS during 200K-bar drain stays below 2.5G (measured with memray or /proc/self/status)

Plans:

- [ ] 23-01: TBD

### Phase 24: Hot-Path Allocation Reduction

**Goal**: Eliminate redundant allocations in sidecar startup and per-flush write paths
**Depends on**: Nothing (independent)
**Requirements**: ALLOC-01, ALLOC-02, ALLOC-03
**Success Criteria** (what must be TRUE):

1. Single `OpenDeviationBarCache()` instance created in `_create_engine` and passed through all startup phases
2. `_write_bars_batch_to_clickhouse` accepts Polars DataFrame or Arrow directly (no dict→DataFrame per 10-bar batch)
3. `get_symbol_entries()` called once at sidecar startup, result passed as parameter to write functions

Plans:

- [ ] 24-01: TBD

### Phase 25: Lazy Evaluation + Cache Efficiency

**Goal**: Replace eager full-file loads with lazy scans and eliminate unnecessary copy operations
**Depends on**: Nothing (independent)
**Requirements**: LAZY-01, LAZY-02
**Success Criteria** (what must be TRUE):

1. `TickStorage.write_ticks` dedup check uses `scan_parquet` with column projection for ID-only comparison (not full file read)
2. Stathera diagnostic in `_sync_flush_rest_bars` avoids `sorted()` copy — uses in-place sort or skip if not needed

Plans:

- [x] 25-01: TBD

## Backlog

### Phase 999.1: Rust multi-threshold Arrow API for repair pipeline (BACKLOG)

**Goal:** Add `process_trades_arrow_multi()` to Rust processor -- accept multiple thresholds in one call, eliminate Python GIL round-trips. Benchmarked 2026-03-24: 48 workers hit GIL ceiling at 469% CPU (15/32 cores). Est. 2-3x speedup (3,200 -> 6,000-10,000 days/min).
**Requirements:** TBD
**Plans:** 1/1 plans complete

Plans:

- [ ] TBD (promote with /gsd:review-backlog when ready)

## Progress

**Execution Order:** Phases execute in numeric order: 13 -> 14 -> 15 -> 16 -> 17 -> 18 -> 19
Phase 13 is the registry foundation. Phase 14 is Rust core changes. Phase 15 validates correctness. Phases 16/17/18 can be parallelized after 15. Phase 19 is the final fleet recompute.

| Phase                                           | Milestone | Plans Complete | Status      | Completed  |
| ----------------------------------------------- | --------- | -------------- | ----------- | ---------- |
| 1. Crossover TID Utility                        | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 2. Seeder Day-Boundary Backfill                 | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 3. Kintsugi Parquet Trust                       | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 4. Heartbeat Coverage                           | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 5. Documentation                                | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 6. Fix SEED-001 Parquet-First Processing        | v5.1      | 1/1            | Complete    | 2026-03-25 |
| 7. Heartbeat Unified Per-Symbol View            | v5.1      | 1/1            | Complete    | 2026-03-25 |
| 8. Checksum Spike                               | v6.0      | 1/1            | Complete    | 2026-03-26 |
| 9. Verification Registry                        | v6.0      | 1/1            | Complete    | 2026-03-26 |
| 10. Seeder Checksum Integration                 | v6.0      | 1/1            | Complete    | 2026-03-26 |
| 11. Integrity Sweep                             | v6.0      | 1/1            | Complete    | 2026-03-26 |
| 12. Telemetry & Heartbeat                       | v6.0      | 1/1            | Complete    | 2026-03-26 |
| 13. Registry + Mode Enum                        | v7.0      | 2/2            | Complete    | 2026-03-27 |
| 14. Rust Core -- Gap Fields + Conditional Reset | v7.0      | 0/2            | Complete    | 2026-03-27 |
| 15. Oracle Validation                           | v7.0      | 0/1            | Complete    | 2026-03-27 |
| 16. Sidecar Aion Startup                        | v7.0      | 2/2            | Complete    | 2026-03-27 |
| 17. Kintsugi Aion Repair                        | v7.0      | 2/2            | Complete    | 2026-03-27 |
| 18. Monitor + Validation Gates                  | v7.0      | 1/2            | Complete    | 2026-03-27 |
| 19. Fleet Recompute + Telemetry                 | v7.0      | 2/2            | Complete    | 2026-03-27 |
| 20. Aion-Aware Sidecar Startup                  | v7.1      | 1/1            | Complete    | 2026-03-27 |
| 21. Universal Write Path ouroboros_mode         | v7.1      | 1/1            | Complete    | 2026-03-27 |
| 22. Monitor Hardening + Schema                  | v7.1      | 2/2            | Complete    | 2026-03-27 |
| 23. Arrow FFI drain_bars                        | v7.2      | 0/1            | Not started | -          |
| 24. Hot-Path Allocation Reduction               | v7.2      | 0/1            | Not started | -          |
| 25. Lazy Evaluation + Cache Efficiency          | v7.2      | 1/1 | Complete    | 2026-03-27 |
