# opendeviationbar-py — Project

## What This Is

Rust workspace with Python bindings for open deviation bar construction. Performance-optimized production system processing 42M+ bars across 18 symbols on bigblack.

## Core Value

Reliable, hands-off production operation: kintsugi self-heals gaps in minutes, sidecar streams without crashes, and hot paths minimize allocations.

## Current Milestone: v1.3 Atomic Restart Recovery

**Goal:** Every sidecar restart produces a clean bar chain for today — zero gaps, zero overlaps, zero legacy fragments — using the proven kintsugi `store_bars_replacing()` pattern.

**Target features:**

- Atomic replace of today's bars during sync_flush (per trade-ID range, not whole-day DELETE)
- Prune 15+ legacy restart-gap mechanisms (dead code from #286 history)
- Scope `_heal_day_boundary_gaps()` to today only
- Remove `_gap_fill()` from watchdog restart path (let fill_from_rest handle)

**Context:** v13.35.0 deploy revealed that 3 deploys in one day create interleaved bar fragments from 3 different processor sessions. RMT dedup can't resolve them because different processor states produce bars with different `first_agg_trade_id` ORDER BY keys. The 4-agent research spike (2026-03-20) confirmed `store_bars_replacing()` is the proven correct mechanism — the same per-chunk atomic DELETE+INSERT that kintsugi uses for day-recompute.

## Requirements

### Validated

- ✓ Rust-side bar dedup prevents overlaps at source (v13.33.0)
- ✓ Sidecar overlap self-heal detects+repairs today's overlaps (v13.32.0)
- ✓ Deploy hardening: monit unmonitor, mutation cleanup, OPTIMIZE FINAL (v13.32.1)
- ✓ Git SHA logged at sidecar startup (v13.32.0)
- ✓ P0: Telegram thread leak — bounded ThreadPoolExecutor (v1.2)
- ✓ P1-P2: REST concurrency + CH connection churn — 3-tier concurrency + thread-local cache (v1.2)
- ✓ P3-P5: Rust hot path — FixedPoint::from_f64(), O(N) chunk cache, burst frame reuse (v1.2)
- ✓ P6-P8: Python efficiency — config caching, Vision TTL, copy removal (v1.2)

### Active

- [ ] Atomic replace in sync_flush using store_bars_replacing()
- [ ] Prune \_inline_startup_backfill() (dead code)
- [ ] Prune \_startup_tid_reconciliation() (redundant with fill_from_rest)
- [ ] Remove \_gap_fill() from watchdog restart path
- [ ] Scope \_heal_day_boundary_gaps() to today only
- [ ] Remove dead config gap_fill_on_startup

### Out of Scope

- OpenDeviationBar struct split (P9) — major refactor, separate milestone
- Plugin Polars→Pandas round-trip — no plugins installed in production
- Monitoring threshold adjustments — fix performance, not alerts
- Midnight-fill approach — git history shows different processor state = different bar boundaries = RMT can't resolve
- Whole-day DELETE on startup — tried 3x, reverted 3x (data vacuum)
- Delayed/timer-based heal — irreducible race during CH slowness

## Key Decisions

| Decision                                  | Rationale                                                                 | Outcome                                |
| ----------------------------------------- | ------------------------------------------------------------------------- | -------------------------------------- |
| Bounded ThreadPoolExecutor over queue     | Simpler, 2-thread cap sufficient                                          | ✓ Good — no crashes since deployment   |
| Separate REST concurrency limit           | REST repairs shouldn't dilute Parquet workers                             | ✓ Good — 3-tier split clean            |
| Thread-local CH cache pattern             | Proven in repair_direct_parquet.py                                        | ✓ Good — fd count stable               |
| FixedPoint::from_f64() over String fields | Keeps csv crate native f64 deserialization fast                           | ✓ Good — oracle-verified bit-identical |
| Cached CSV decompression (Approach B)     | ZipFile doesn't support Seek; cache bytes in memory                       | ✓ Good — O(N) achieved                 |
| store_bars_replacing() for sync_flush     | Proven kintsugi mechanism; per-chunk atomic DELETE+INSERT; no data vacuum | — Pending                              |
| Prune legacy restart mechanisms           | 15+ failed attempts created 6+ conflicting code paths                     | — Pending                              |

## Constraints

- Python 3.13 only
- No Rust API changes (PyO3 bindings stable)
- Spot-only data sources (Binance)
- 10 hard constraints from git history (see v1.3 REQUIREMENTS.md)

## Evolution

This document evolves at phase transitions and milestone boundaries.

**After each phase transition** (via `/gsd:transition`):

1. Requirements invalidated? → Move to Out of Scope with reason
2. Requirements validated? → Move to Validated with phase reference
3. New requirements emerged? → Add to Active
4. Decisions to log? → Add to Key Decisions
5. "What This Is" still accurate? → Update if drifted

**After each milestone** (via `/gsd:complete-milestone`):

1. Full review of all sections
2. Core Value check — still the right priority?
3. Audit Out of Scope — reasons still valid?
4. Update Context with current state

---

_Last updated: 2026-03-20 after v1.3 milestone initialization_
