# Roadmap: opendeviationbar-py

## Milestones

- ✅ **v1.0 Junction Fix** -- Phases 1-3 (shipped 2026-03-22) -- [archive](milestones/v1.0-ROADMAP.md)
- ✅ **v1.4 Zero-Gap Zero-Overlap Guarantee** -- Phases 1-6 (shipped 2026-03-23) -- [archive](milestones/v1.4-ROADMAP.md)
- ✅ **v1.5 Sidecar Startup Architecture** -- Phases 1-3 (shipped 2026-03-23) -- [archive](milestones/v1.5-ROADMAP.md)
- ✅ **v2.0 Seeder as Sole REST Owner + Microsecond Alignment** -- Phases 1-5 (shipped 2026-03-24) -- [archive](milestones/v2.0-ROADMAP.md)
- ✅ **v3.0 Streaming Reliability + Pipeline Cleanup** -- Phases 1-5.1 (shipped 2026-03-24) -- [archive](milestones/v3.0-ROADMAP.md)
- ✅ **v4.0 Microsecond Column Rename + Deploy Gap Repair** -- Phases 1-5 (shipped 2026-03-25) -- [archive](milestones/v4.0-ROADMAP.md)
- ✅ **v5.0 Seeder Day-Boundary Completeness + Kintsugi Parquet Trust** -- Phases 1-5 (shipped 2026-03-25, v13.54.0)
- ✅ **v5.1 SEED-001 Fix + Heartbeat Unified View** -- Phases 6-7 (shipped 2026-03-25, v13.55.0)
- ✅ **v6.0 Data Integrity & Checksum Verification** -- Phases 8-12 (shipped 2026-03-26, v13.56.1)
- ✅ **v7.0 Aion Mode -- Genesis-Continuous for Crypto** -- Phases 13-19 (shipped 2026-03-27) -- [archive](milestones/v7.0-ROADMAP.md)
- ✅ **v7.1 Aion Hardening -- Footgun Elimination + Startup Fix** -- Phases 20-22 (shipped 2026-03-27) -- [archive](milestones/v7.1-ROADMAP.md)
- ✅ **v7.2 Memory Efficiency -- Zero-OOM Sidecar** -- Phases 23-25 (shipped 2026-03-27) -- [archive](milestones/v7.2-ROADMAP.md)
- ✅ **v7.3 Exchange Gap Classification** -- Phases 26-28 (shipped 2026-03-28)
- 🚧 **v8.0 Aion-Native Pipeline -- Zero-Gap Restarts** -- Phases 29-33 (in progress)

## Phases

<details>
<summary>✅ v5.0 Seeder Day-Boundary Completeness (Phases 1-5) - SHIPPED 2026-03-25</summary>

- [x] **Phase 1: Crossover TID Utility** - Extract shared crossover lookup callable by seeder, kintsugi, and heartbeat
- [x] **Phase 2: Seeder Day-Boundary Backfill** - Seeder backfills yesterday's Parquet file after midnight to crossover TID boundary
- [x] **Phase 3: Kintsugi Parquet Trust** - Kintsugi verifies Parquet completeness before using as repair source
- [x] **Phase 4: Heartbeat Coverage** - Yesterday completion check expanded to all streaming symbols
- [x] **Phase 5: Documentation** - Root cause and fix documented across CLAUDE.md network

</details>

<details>
<summary>✅ v5.1 SEED-001 Fix + Heartbeat Unified View (Phases 6-7) - SHIPPED 2026-03-25</summary>

- [x] **Phase 6: Fix SEED-001 Parquet-First Bar Processing** - Process Parquet ticks into bars instead of skipping past them
- [x] **Phase 7: Heartbeat Unified Per-Symbol View** - Merge Yesterday + Junction into exception-only display with TID deltas

</details>

<details>
<summary>✅ v6.0 Data Integrity & Checksum Verification (Phases 8-12) - SHIPPED 2026-03-26</summary>

- [x] **Phase 8: Checksum Spike** - Prove Rust SHA-256 callable from Python with correct results and edge case handling
- [x] **Phase 9: Verification Registry** - Centralized per-file verification state store as SSoT
- [x] **Phase 10: Seeder Checksum Integration** - Wire SHA-256 into seeder download pipeline with retry and audit trail
- [x] **Phase 11: Integrity Sweep** - Audit existing Parquet cache against Vision CHECKSUM files
- [x] **Phase 12: Telemetry & Heartbeat** - Heartbeat checksum health, LOUD alerts, local telemetry log

</details>

<details>
<summary>✅ v7.0 Aion Mode (Phases 13-19) - SHIPPED 2026-03-27</summary>

- [x] **Phase 13: Registry + Mode Enum** - Ouroboros mode SSoT in symbols.toml with Aion variant
- [x] **Phase 14: Rust Core -- Gap Fields + Conditional Reset** - Bar struct gap awareness and conditional midnight reset
- [x] **Phase 15: Oracle Validation** - Bit-exact oracle tests proving Aion correctness
- [x] **Phase 16: Sidecar Aion Startup** - Checkpoint-resume startup path for Aion symbols
- [x] **Phase 17: Kintsugi Aion Repair** - Genesis-forward reconciliation with TID-based writer boundaries
- [x] **Phase 18: Monitor + Validation Gates** - Rolling-window freshness, conditional day-mode gate, mode-agnostic Stathera
- [x] **Phase 19: Fleet Recompute + Telemetry** - Wipe Day-mode crypto bars, recompute as Aion on bigblack

</details>

<details>
<summary>✅ v7.1 Aion Hardening (Phases 20-22) - SHIPPED 2026-03-27</summary>

- [x] **Phase 20: Aion-Aware Sidecar Startup** - Preserve pre-midnight bars, seed from max(committed_tid, crossover_tid)
- [x] **Phase 21: Universal Write Path ouroboros_mode** - Wire per-symbol mode in all write call sites
- [x] **Phase 22: Monitor Hardening + Schema** - Freshness query filter, sidecar restart investigation, CH validation

</details>

<details>
<summary>✅ v7.2 Memory Efficiency (Phases 23-25) - SHIPPED 2026-03-27</summary>

- [x] **Phase 23: Arrow FFI drain_bars** - Arrow RecordBatch FFI for zero-copy sync_flush
- [x] **Phase 24: Hot-Path Allocation Reduction** - Single CH cache, eliminate per-flush dict conversions
- [x] **Phase 25: Lazy Evaluation + Cache Efficiency** - scan_parquet for dedup, in-place sort

</details>

<details>
<summary>✅ v7.3 Exchange Gap Classification (Phases 26-28) - SHIPPED 2026-03-28</summary>

- [x] **Phase 26: Exchange Gap Classification** - Classify gaps as pipeline vs exchange
- [x] **Phase 27: Kintsugi Exchange Gap Awareness** - Skip unfillable exchange gaps in repair
- [x] **Phase 28: Stathera Exchange Gap Resolution** - Permanent Stathera resolution for exchange gaps

</details>

### v8.0 Aion-Native Pipeline -- Zero-Gap Restarts

- [x] **Phase 29: DELETE Strategy + Exchange Gap Floor** - Spike safest DELETE approach and clamp resume TID above exchange gaps (completed 2026-03-28)
- [x] **Phase 30: Parquet-First Gap Fill + Junction Telemetry** - Read local tick cache before REST API, measure handoff quality (completed 2026-03-28)
- [ ] **Phase 31: Aion-First Defaults** - Change default ouroboros mode from "day" to "aion" across ~30 call sites
- [x] **Phase 32: Historical Gap Cleanup** - Bulk reprocess 98K day-boundary gaps as continuous Aion streams (completed 2026-03-28)
- [x] **Phase 33: Dead Code Removal** - Remove day-mode code paths unreachable with Aion-only crypto (completed 2026-03-28)

## Phase Details

### Phase 29: DELETE Strategy + Exchange Gap Floor

**Goal**: Sidecar restarts without creating gaps, using the safest DELETE strategy (prefer extra bars over gaps) and never attempting to fill unfillable exchange gap ranges
**Depends on**: Phase 28 (v7.3 shipped)
**Requirements**: RESUME-04, RESUME-01
**Success Criteria** (what must be TRUE):

1. Sidecar startup DELETE phase completes in under 60 seconds (down from ~6 minutes for 42 partitions)
2. After sidecar restart, Stathera audit shows zero new pipeline gaps -- extra bars (handled by RMT dedup) are preferred over missing bars
3. Sidecar resume TID is clamped above the highest known exchange gap from kintsugi_log -- no REST/Parquet fill attempts for ranges where the exchange had no trades
4. DELETE strategy selection is documented with measured timings for each approach (skip-if-clean, partition-scoped, full-table)
   **Plans**: 2 plans

Plans:

- [x] 29-01-PLAN.md — Skip-if-clean DELETE strategy with pre-check query
- [x] 29-02-PLAN.md — Exchange gap floor clamping for resume TID

### Phase 30: Parquet-First Gap Fill + Junction Telemetry

**Goal**: Sidecar fills restart gaps from local Parquet tick cache before falling back to REST API, and logs junction quality metrics at the REST-to-WS handoff point
**Depends on**: Phase 29
**Requirements**: RESUME-02, RESUME-03
**Success Criteria** (what must be TRUE):

1. On sidecar restart, gap fill reads trades from Parquet tick cache first (3ms/file) and only falls back to REST API for ranges not covered by Parquet
2. Sidecar logs junction telemetry at REST-to-WS handoff: `junction_gap_tid_delta`, `ws_buffer_depth`, and `skipped_trades` per symbol
3. When Parquet covers the full gap range, zero REST API calls are made during gap fill (verifiable in telemetry log)
   **Plans**: 2 plans

Plans:

- [x] 30-01-PLAN.md — Parquet-first Aion gap fill (process local tick cache before REST)
- [x] 30-02-PLAN.md — Junction telemetry logging at REST-to-WS handoff

### Phase 31: Aion-First Defaults

**Goal**: Every function default, config model default, fallback lookup, and CLI default uses "aion" instead of "day" -- forex symbols still use day mode via explicit registry lookup
**Depends on**: Phase 29 (DELETE strategy must be correct before changing defaults)
**Requirements**: DEFAULT-01, DEFAULT-02, DEFAULT-03, DEFAULT-04, DEFAULT-05
**Success Criteria** (what must be TRUE):

1. `grep -rn '"day"' python/ scripts/ src/` shows zero hardcoded "day" defaults in function signatures, config models, `.get()` fallbacks, or CLI `--ouroboros-mode` defaults
2. Forex symbols (disabled in symbols.toml) still resolve to day mode via explicit `get_symbol_entry().ouroboros_mode` registry lookup -- not from any default value
3. All existing tests pass without modification (or with updated assertions reflecting the new default)
4. Public API backward compatibility: callers who previously relied on implicit `ouroboros_mode="day"` default now get "aion" -- this is intentional and documented in CHANGELOG
   **Plans**: 1 plan

Plans:

- [ ] 31-01-PLAN.md — Batch refactor all ouroboros_mode defaults from "day" to "aion"

### Phase 32: Historical Gap Cleanup

**Goal**: The 98K historical day-boundary gaps (artifacts from pre-Aion era) are eliminated by bulk reprocessing affected date ranges as continuous Aion streams from Parquet
**Depends on**: Phase 29, Phase 30 (resume/DELETE work must be correct before bulk reprocess)
**Requirements**: CLEANUP-01
**Success Criteria** (what must be TRUE):

1. `repair_direct_parquet.py` (or equivalent bulk script) reprocesses affected date ranges as continuous Aion streams, producing bars without day-boundary orphan gaps
2. Stathera audit after bulk reprocess shows fewer than 100 remaining gaps across all symbols (down from 98K), with all remaining gaps classified as exchange gaps
3. Reprocessed bars pass bit-exact Stathera continuity validation -- no new overlaps introduced
   **Plans**: 1 plan

Plans:

- [x] 32-01-PLAN.md — Bulk Aion repair orchestration script for all crypto symbols

### Phase 33: Dead Code Removal

**Goal**: Day-mode code paths unreachable with Aion-only crypto are removed, reducing maintenance surface and eliminating footgun risk
**Depends on**: Phase 31, Phase 32 (defaults changed and historical gaps cleaned before removing code)
**Requirements**: CLEANUP-02, CLEANUP-03, CLEANUP-04
**Success Criteria** (what must be TRUE):

1. Midnight preflight code path (`_run_midnight_preflight`) is guarded behind a day-mode check -- unreachable for any crypto symbol
2. Crossover TID seeding and checkpoint injection code for Aion symbols is removed or guarded behind day-mode checks
3. `grep -rn "OuroborosMode.DAY\|ouroboros_mode.*day" python/ scripts/` shows only: (a) symbols.toml registry entries for forex, (b) enum definition, (c) explicit day-mode guard conditions -- no dead fallback paths
4. All tests pass after removal -- no test depends on day-mode crypto code paths
   **Plans**: 1 plan

Plans:

- [x] 33-01-PLAN.md — Audit day-mode guards, remove unused params, add skip-logging

## Backlog

### Phase 999.1: Rust multi-threshold Arrow API for repair pipeline (BACKLOG)

**Goal:** Add `process_trades_arrow_multi()` to Rust processor -- accept multiple thresholds in one call, eliminate Python GIL round-trips. Benchmarked 2026-03-24: 48 workers hit GIL ceiling at 469% CPU (15/32 cores). Est. 2-3x speedup (3,200 -> 6,000-10,000 days/min).
**Requirements:** TBD
**Plans:** 1/1 plans complete

Plans:

- [ ] TBD (promote with /gsd:review-backlog when ready)

### Phase 999.2: Default ouroboros mode from "day" to "aion" (BACKLOG -- PROMOTED to Phase 31)

**Goal:** Promoted to Phase 31 in v8.0 milestone.

## Progress

**Execution Order:** Phases execute in numeric order: 29 -> 30 -> 31 -> 32 -> 33
Phase 29 is foundational (DELETE strategy + gap floor). Phase 30 adds Parquet-first fill. Phase 31 is the batch default refactor. Phase 32 is bulk historical cleanup. Phase 33 is final dead code removal.

| Phase                                           | Milestone | Plans Complete | Status      | Completed  |
| ----------------------------------------------- | --------- | -------------- | ----------- | ---------- |
| 1. Crossover TID Utility                        | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 2. Seeder Day-Boundary Backfill                 | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 3. Kintsugi Parquet Trust                       | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 4. Heartbeat Coverage                           | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 5. Documentation                                | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 6. Fix SEED-001 Parquet-First Processing        | v5.1      | 1/1            | Complete    | 2026-03-25 |
| 7. Heartbeat Unified Per-Symbol View            | v5.1      | 1/1            | Complete    | 2026-03-25 |
| 8. Checksum Spike                               | v6.0      | 1/1            | Complete    | 2026-03-26 |
| 9. Verification Registry                        | v6.0      | 1/1            | Complete    | 2026-03-26 |
| 10. Seeder Checksum Integration                 | v6.0      | 1/1            | Complete    | 2026-03-26 |
| 11. Integrity Sweep                             | v6.0      | 1/1            | Complete    | 2026-03-26 |
| 12. Telemetry & Heartbeat                       | v6.0      | 1/1            | Complete    | 2026-03-26 |
| 13. Registry + Mode Enum                        | v7.0      | 2/2            | Complete    | 2026-03-27 |
| 14. Rust Core -- Gap Fields + Conditional Reset | v7.0      | 2/2            | Complete    | 2026-03-27 |
| 15. Oracle Validation                           | v7.0      | 1/1            | Complete    | 2026-03-27 |
| 16. Sidecar Aion Startup                        | v7.0      | 2/2            | Complete    | 2026-03-27 |
| 17. Kintsugi Aion Repair                        | v7.0      | 2/2            | Complete    | 2026-03-27 |
| 18. Monitor + Validation Gates                  | v7.0      | 2/2            | Complete    | 2026-03-27 |
| 19. Fleet Recompute + Telemetry                 | v7.0      | 2/2            | Complete    | 2026-03-27 |
| 20. Aion-Aware Sidecar Startup                  | v7.1      | 1/1            | Complete    | 2026-03-27 |
| 21. Universal Write Path ouroboros_mode         | v7.1      | 1/1            | Complete    | 2026-03-27 |
| 22. Monitor Hardening + Schema                  | v7.1      | 2/2            | Complete    | 2026-03-27 |
| 23. Arrow FFI drain_bars                        | v7.2      | 1/1            | Complete    | 2026-03-27 |
| 24. Hot-Path Allocation Reduction               | v7.2      | 1/1            | Complete    | 2026-03-27 |
| 25. Lazy Evaluation + Cache Efficiency          | v7.2      | 1/1            | Complete    | 2026-03-27 |
| 26. Exchange Gap Classification                 | v7.3      | -/-            | Complete    | 2026-03-28 |
| 27. Kintsugi Exchange Gap Awareness             | v7.3      | -/-            | Complete    | 2026-03-28 |
| 28. Stathera Exchange Gap Resolution            | v7.3      | -/-            | Complete    | 2026-03-28 |
| 29. DELETE Strategy + Exchange Gap Floor        | v8.0      | 2/2            | Complete    | 2026-03-28 |
| 30. Parquet-First Gap Fill + Junction Telemetry | v8.0      | 2/2            | Complete    | 2026-03-28 |
| 31. Aion-First Defaults                         | v8.0      | 0/1            | Not started | -          |
| 32. Historical Gap Cleanup                      | v8.0      | 1/1            | Complete    | 2026-03-28 |
| 33. Dead Code Removal                           | v8.0      | 1/1 | Complete   | 2026-03-28 |
