# opendeviationbar-py — Project

## What This Is

Rust workspace with Python bindings for open deviation bar construction. Performance-optimized production system processing 42M+ bars across 17 symbols on bigblack, with zero-gap zero-overlap guarantees validated by continuous telemetry.

## Core Value

Zero-gap, zero-overlap bars across every restart, every boundary, every deploy — verified by continuous telemetry. Now supports both Day-mode (UTC-midnight reset) and Aion-mode (continuous genesis-forward) per symbol.

## Requirements

### Validated

- ✓ Rust-side bar dedup prevents overlaps at source (v13.33.0)
- ✓ Sidecar overlap self-heal detects+repairs today's overlaps (v13.32.0)
- ✓ Deploy hardening: monit unmonitor, mutation cleanup, OPTIMIZE FINAL (v13.32.1)
- ✓ P0-P8: v1.2 Performance milestone (all 12 requirements)
- ✓ Atomic replace in sync_flush using store_bars_replacing() (v1.3)
- ✓ Legacy pruning: dead restart-gap code removed (v1.3)
- ✓ Clean slate DELETE today before fill_from_rest (v1.3)
- ✓ drain_bars() synchronous ring buffer drain (v1.3)
- ✓ 1000 dbps removed, 100 dbps added for BTC/ETH/SOL (v13.38.0)
- ✓ committed_floors from last completed bar, not forming bar tail — v1.4 (RUST-01, RUST-02)
- ✓ Midnight crossover trade ID verification via REST API preflight — v1.4 (PREFLIGHT-01-03)
- ✓ Heartbeat unified per-symbol Stathera continuity + LOUD alerts — v1.4 (TELEM-01-04)
- ✓ WS resilience: ping/pong keepalive, ban handling, stream signals — v1.4 (Phase 4)
- ✓ Seeder REST fallback + cross-process rate limit coordination — v1.4 (SEEDER-01-04)
- ✓ Memory efficiency: Copy AggTrade, Arrow streaming, GIL release, Python optimizations — v1.4 (MEM-01-08)
- ✓ Yesterday integrity check + signal file — v1.5 (STARTUP-01-02)
- ✓ Kintsugi yesterday priority queue — v1.5 (STARTUP-03)
- ✓ Sub-minute sidecar startup (27s measured) — v1.5 (STARTUP-04)

- ✓ End-to-end SHA-256 checksum verification for Vision ZIP downloads — v6.0 (SPIKE-01..03, SEEDER-01..03)
- ✓ Centralized ChecksumRegistry with per-file verification state as SSoT — v6.0 (TELEM-03)
- ✓ Integrity sweep audits existing Parquet cache against Vision CHECKSUMs — v6.0 (SWEEP-01..03)
- ✓ Heartbeat checksum health + LOUD Telegram alerts + JSONL telemetry log — v6.0 (TELEM-01, 02, 04)

- ✓ OuroborosMode.AION enum + symbols.toml per-symbol SSoT — v7.0 (REG-01..05)
- ✓ Gap awareness on bars: `has_gap`, `gap_trade_count`, `max_gap_duration_us`, `is_exchange_gap` — v7.0 (CORE-01..04)
- ✓ Bit-exact oracle: Day==Aion single-day equivalence, midnight-spanning continuity — v7.0 (VAL-01..03)
- ✓ Sidecar Aion startup: checkpoint-resume, no clean-slate DELETE — v7.0 (SIDE-01..03)
- ✓ Kintsugi Aion repair: TID-range reprocessing, conditional today-guard — v7.0 (KINT-01..03)
- ✓ Monitor + Validation: per-symbol pre-write gate, Aion freshness, Stathera oracle — v7.0 (MON-01..03)
- ✓ Fleet recompute script + diagnostic updates for bigblack migration — v7.0 (RECOMP-01..05, VAL-04)

### Active

- [ ] Aion-native sidecar resume with exchange gap floor and Parquet-first fill
- [ ] Parquet-first gap fill before REST API in sidecar startup
- [ ] Smarter resume: scan backwards past pipeline gaps, stop at exchange gaps
- [ ] Junction telemetry: measure gap quality at REST→WS handoff
- [ ] Aion-first defaults across ~30 call sites (backlog 999.2)
- [ ] Startup speed: skip unnecessary DELETEs, parallelize or batch
- [ ] Historical gap cleanup: reprocess 98K day-boundary gaps as continuous Aion streams

## Current Milestone: v8.0 Aion-Native Pipeline — Zero-Gap Restarts

**Goal:** Make the entire pipeline Aion-native: restart without creating gaps, fill from Parquet before REST, default everything to Aion, and clean up 98K historical day-boundary gaps.

**Target features:**

- Aion-native sidecar resume: exchange gap floor, Parquet-first before REST, skip unnecessary DELETEs
- Parquet-first gap fill: sidecar reads local tick cache before REST API
- Smarter resume with exchange gap awareness: scan backwards past pipeline gaps, stop at exchange gaps
- Junction telemetry: measure and log gap quality at REST→WS handoff point
- Aion-first defaults: default ouroboros from "day" to "aion" across ~30 call sites
- Startup speed: skip DELETE for partitions where max_tid == resume_tid
- Historical gap cleanup: bulk reprocess 98K day-boundary gaps from Parquet as continuous Aion streams

## Current State

v7.3 Exchange Gap Classification shipped (2026-03-28). 12 milestones complete.
Aion-native resume deployed (resume from last committed TID, no midnight concepts).
Forex symbols disabled in symbols.toml. Sidecar now crypto-only (16 aion, 0 day).
98K historical day-boundary gaps remain from pre-Aion era — kintsugi healing at ~45 gaps/pass.
Sidecar DELETE phase takes ~6 minutes for 42 partitions — major startup bottleneck.

**Gap/Overlap Taxonomy (5 types):**

- G1: Intra-bar gap (pipeline) — `has_gap=true` on bar, transient, healed by Puck/Kintsugi
- G2: Intra-bar gap (exchange) — `has_gap=true` + `is_exchange_gap=true`, permanent/unhealable
- G3: Inter-bar gap (pipeline) — Stathera query-time only, healed by Kintsugi
- G4: Inter-bar gap (exchange) — Stathera query persists permanently
- O1: Overlap (concurrent writers) — Stathera query-time, healed by Kintsugi dedup

### Out of Scope

- OpenDeviationBar struct split (P9) — major refactor, separate milestone
- Full 18-symbol streaming — planned for later milestone
- Kintsugi two-day processor spans — preflight verification is simpler and sufficient
- Plugin Polars->Pandas — no plugins in production

## Key Decisions

| Decision                                       | Rationale                                               | Outcome                                 |
| ---------------------------------------------- | ------------------------------------------------------- | --------------------------------------- |
| Clean slate DELETE today on restart            | Proven zero-gap pattern (v1.3)                          | ✓ Good — 0/0 Stathera on restart        |
| store_bars_replacing for sync_flush            | Proven kintsugi mechanism                               | ✓ Good — atomic per-chunk               |
| REST API crossover verification                | User insight — pinpoint exact midnight trade ID         | ✓ Good — verified in production         |
| committed_floors from completed bar            | 6-agent spike root cause (Bug 1)                        | ✓ Good — junction gaps eliminated       |
| Unified Stathera in heartbeat                  | No more intra-day vs midnight classification            | ✓ Good — single per-symbol check        |
| File-based rate limit coordination             | Cross-process budget sharing without Redis              | ✓ Good — atomic JSON, stale expiry      |
| Oracle golden snapshots before refactoring     | Bit-exact regression baseline                           | ✓ Good — caught 0 regressions           |
| AggTrade Copy + Arrow zero-copy                | Eliminate clone overhead + heap allocation              | ✓ Good — verified bit-exact             |
| Aion = continuous Ouroboros variant            | CEO: crypto is 24/7, orphan bars unnecessary            | ✓ Good — shipped v7.0, 27/27 reqs       |
| Gap taxonomy: intra-bar flag, inter-bar query  | Marking individual bars for inter-bar gaps is ambiguous | ✓ Good — G1/G2 flagged, G3/G4/O1 query  |
| symbols.toml SSoT for ouroboros_mode           | Registry already controls thresholds, market, priority  | ✓ Good — per-symbol mode control        |
| has_gap is transient, healers fire immediately | Mark-and-continue, not accept-and-ignore                | ✓ Good — Puck/Kintsugi repair ASAP      |
| Per-symbol mode wiring (not global)            | Mixed Day+Aion fleet in one sidecar instance            | ✓ Good — HashMap<String, OuroborosMode> |
| Aion repair = TID-range, not day-recompute     | Continuous bars have no day boundaries                  | ✓ Good — \_repair_aion() in kintsugi    |

## Context

v7.0 shipped with 7 phases, 13 plans across 2 days (2026-03-26 → 2026-03-27). 75 commits, 89 files changed.

**Known tech debt:**

- Bigblack fleet recompute pending execution (`scripts/fleet_recompute_aion.py`)
- TELEM-03: `ws_first_tid` and `committed_floors` junction telemetry sub-fields not yet populated
- Phase 14 VERIFICATION had gaps_found (prev_close_time — fixed inline)
- Phase 18 test regression fixed inline (get_symbol_entries mock)

## Constraints

- Python 3.13 only
- No Rust API changes that break PyO3 bindings
- Spot-only data sources (Binance)
- 10 hard constraints from git history (see v1.3 debug findings)

## Evolution

This document evolves at phase transitions and milestone boundaries.

**After each phase transition** (via `/gsd:transition`):

1. Requirements invalidated? → Move to Out of Scope with reason
2. Requirements validated? → Move to Validated with phase reference
3. New requirements emerged? → Add to Active
4. Decisions to log? → Add to Key Decisions
5. "What This Is" still accurate? → Update if drifted

**After each milestone** (via `/gsd:complete-milestone`):

1. Full review of all sections
2. Core Value check — still the right priority?
3. Audit Out of Scope — reasons still valid?
4. Update Context with current state

---

_Last updated: 2026-03-28 — v8.0 Aion-Native Pipeline milestone started_
