# opendeviationbar-py — Project

## What This Is

Rust workspace with Python bindings for open deviation bar construction. Performance-optimized production system processing 42M+ bars across 17 symbols on bigblack.

## Core Value

Zero-gap, zero-overlap Ouroboros day-mode bars across every restart, every midnight boundary, every deploy — verified by continuous telemetry.

## Current Milestone: v1.4 Zero-Gap Zero-Overlap Guarantee

**Goal:** Eliminate ALL Stathera gaps and overlaps — both intra-day junction gaps (committed_floors bug) and midnight boundary gaps (multi-processor seam) — with full telemetry to validate.

**Target features:**

1. **Rust fix: committed_floors** — Initialize dedup floor from last COMPLETED bar, not forming bar tail. Eliminates intra-day junction gaps.
2. **Midnight preflight verification** — Use Binance REST API to get the exact midnight crossover trade ID. Verify yesterday's orphan bar closes at that trade. Anchor today's fill_from_rest from verified crossover ID.
3. **Sidecar anchoring** — fill_from_rest anchors from verified yesterday crossover trade ID (not CH seed). Guarantees `bar[yesterday_last].last_agg_trade_id + 1 == bar[today_first].first_agg_trade_id`.
4. **Heartbeat telemetry overhaul** — Per-symbol Stathera continuity (unified, no intra-day vs midnight split), yesterday completion status, junction telemetry, regression detector with LOUD alerts.

**Context:** 6-agent spike (2026-03-21) found two root causes:

- Bug 1: `committed_floors` in `live_engine.rs:1379` set from forming bar tail, suppresses legitimate bars at the REST→WS junction
- Bug 2: Multi-processor seam at midnight — kintsugi (yesterday) and sidecar (today) use independent processors with no trade-ID handoff. BTCUSDT/ETHUSDT (streamed continuously) have ZERO midnight gaps. Non-streamed symbols have gaps from the processor seam.

**User's key insight:** "We can always use REST API to pinpoint the crossover time of day to get a precise read of the exact crossover aggregator trade ID." The orphan bar should always close at the very last aggregated trade ID of the day. A preflight check can verify this.

## Requirements

### Validated

- ✓ Rust-side bar dedup prevents overlaps at source (v13.33.0)
- ✓ Sidecar overlap self-heal detects+repairs today's overlaps (v13.32.0)
- ✓ Deploy hardening: monit unmonitor, mutation cleanup, OPTIMIZE FINAL (v13.32.1)
- ✓ P0-P8: v1.2 Performance milestone (all 12 requirements)
- ✓ Atomic replace in sync_flush using store_bars_replacing() (v1.3)
- ✓ Legacy pruning: dead restart-gap code removed (v1.3)
- ✓ Clean slate DELETE today before fill_from_rest (v1.3)
- ✓ drain_bars() synchronous ring buffer drain (v1.3)
- ✓ 1000 dbps removed, 100 dbps added for BTC/ETH/SOL (v13.38.0)

### Active

- [ ] Fix committed_floors to use last completed bar, not forming bar tail
- [ ] Midnight crossover trade ID verification via REST API preflight
- [ ] Sidecar fill_from_rest anchors from verified crossover trade ID
- [ ] Heartbeat per-symbol unified Stathera continuity check
- [ ] Heartbeat yesterday completion status
- [ ] Heartbeat junction telemetry (REST→WS trade IDs, forming bar state)
- [ ] Heartbeat regression detector with LOUD alerts

### Out of Scope

- OpenDeviationBar struct split (P9) — major refactor, separate milestone
- Full 18-symbol streaming — planned for later milestone (GSD Phases 6-10)
- Kintsugi two-day processor spans — alternative approach, not needed if preflight verification works
- Plugin Polars→Pandas round-trip — no plugins in production

## Key Decisions

| Decision                            | Rationale                                       | Outcome                          |
| ----------------------------------- | ----------------------------------------------- | -------------------------------- |
| Clean slate DELETE today on restart | Proven zero-gap pattern (v1.3)                  | ✓ Good — 0/0 Stathera on restart |
| store_bars_replacing for sync_flush | Proven kintsugi mechanism                       | ✓ Good — atomic per-chunk        |
| REST API crossover verification     | User insight — pinpoint exact midnight trade ID | — Pending                        |
| committed_floors from completed bar | 6-agent spike root cause (Bug 1)                | — Pending                        |
| Unified Stathera in heartbeat       | No more intra-day vs midnight classification    | — Pending                        |

## Constraints

- Python 3.13 only
- No Rust API changes that break PyO3 bindings
- Spot-only data sources (Binance)
- 10 hard constraints from git history (see v1.3 debug findings)

## Evolution

This document evolves at phase transitions and milestone boundaries.

**After each phase transition** (via `/gsd:transition`):

1. Requirements invalidated? → Move to Out of Scope with reason
2. Requirements validated? → Move to Validated with phase reference
3. New requirements emerged? → Add to Active
4. Decisions to log? → Add to Key Decisions
5. "What This Is" still accurate? → Update if drifted

**After each milestone** (via `/gsd:complete-milestone`):

1. Full review of all sections
2. Core Value check — still the right priority?
3. Audit Out of Scope — reasons still valid?
4. Update Context with current state

---

_Last updated: 2026-03-21 after v1.4 milestone initialization_
