# opendeviationbar-py — Project

## What This Is

Rust workspace with Python bindings for open deviation bar construction. Performance-optimized production system processing 42M+ bars across 18 symbols on bigblack.

## Core Value

Reliable, hands-off production operation: kintsugi self-heals gaps in minutes, sidecar streams without crashes, and hot paths minimize allocations.

## Requirements

### Validated

- ✓ Rust-side bar dedup prevents overlaps at source (v13.33.0)
- ✓ Sidecar overlap self-heal detects+repairs today's overlaps (v13.32.0)
- ✓ Deploy hardening: monit unmonitor, mutation cleanup, OPTIMIZE FINAL (v13.32.1)
- ✓ Git SHA logged at sidecar startup (v13.32.0)
- ✓ P0: Telegram thread leak — bounded ThreadPoolExecutor (v1.2)
- ✓ P1-P2: REST concurrency + CH connection churn — 3-tier concurrency + thread-local cache (v1.2)
- ✓ P3-P5: Rust hot path — FixedPoint::from_f64(), O(N) chunk cache, burst frame reuse (v1.2)
- ✓ P6-P8: Python efficiency — config caching, Vision TTL, copy removal (v1.2)

### Active

- [ ] P9: OpenDeviationBar struct — split hot/cold fields (future milestone)
- [ ] P10: Vec preallocation — conservative sizing for load_historical_range

### Out of Scope

- OpenDeviationBar struct split (P9) — major refactor, separate milestone
- Plugin Polars→Pandas round-trip — no plugins installed in production
- Monitoring threshold adjustments — fix performance, not alerts

## Current State

**Shipped v1.2 Performance** (2026-03-20): 4 phases, 5 plans, 12/12 requirements satisfied. Key wins: bounded thread pool eliminates crash risk, thread-local CH cache + 3-tier concurrency cuts kintsugi repair time, zero-alloc FixedPoint::from_f64() eliminates 4M allocs/day, O(N) chunk iteration, burst frame Vec reuse.

## Key Decisions

| Decision                                  | Rationale                                           | Outcome                                |
| ----------------------------------------- | --------------------------------------------------- | -------------------------------------- |
| Bounded ThreadPoolExecutor over queue     | Simpler, 2-thread cap sufficient                    | ✓ Good — no crashes since deployment   |
| Separate REST concurrency limit           | REST repairs shouldn't dilute Parquet workers       | ✓ Good — 3-tier split clean            |
| Thread-local CH cache pattern             | Proven in repair_direct_parquet.py                  | ✓ Good — fd count stable               |
| FixedPoint::from_f64() over String fields | Keeps csv crate native f64 deserialization fast     | ✓ Good — oracle-verified bit-identical |
| Cached CSV decompression (Approach B)     | ZipFile doesn't support Seek; cache bytes in memory | ✓ Good — O(N) achieved                 |

## Constraints

- Python 3.13 only
- No Rust API changes (PyO3 bindings stable)
- Spot-only data sources (Binance)

---

_Last updated: 2026-03-20 after v1.2 milestone_
