# Roadmap: v1.4 Zero-Gap Zero-Overlap Guarantee

## Overview

Three phases that fix the two root causes of Stathera gaps (committed_floors dedup bug, midnight processor seam) and then instrument the system to prove they stay fixed. Phase 1 fixes the Rust dedup floor so bars are never suppressed at the REST-to-WS junction. Phase 2 uses Binance REST to verify the exact midnight crossover trade ID and anchor fill_from_rest from it, closing the multi-processor seam. Phase 3 overhauls heartbeat telemetry to continuously validate zero gaps across every symbol, every day.

## Phases

**Phase Numbering:**

- Integer phases (1, 2, 3): Planned milestone work
- Decimal phases (2.1, 2.2): Urgent insertions (marked with INSERTED)

Decimal phases appear between their surrounding integers in numeric order.

- [ ] **Phase 1: Dedup Floor Fix** - Fix committed_floors to use last completed bar, eliminating intra-day junction gaps
- [ ] **Phase 2: Midnight Preflight Verification** - Query Binance REST for exact midnight crossover trade ID, verify yesterday's orphan bar, anchor today's fill_from_rest
- [ ] **Phase 3: Heartbeat Telemetry Overhaul** - Unified per-symbol Stathera check, yesterday completion, junction telemetry, LOUD regression alerts

## Phase Details

### Phase 1: Dedup Floor Fix

**Goal**: Bars at the fill_from_rest to WS junction are never silently suppressed by the dedup floor
**Depends on**: Nothing (first phase)
**Requirements**: RUST-01, RUST-02
**Success Criteria** (what must be TRUE):

1. `committed_floors` in `symbol_task()` reads from the last COMPLETED bar's `last_agg_trade_id`, not from `processor.last_agg_trade_id()` which includes the forming bar tail
2. After a fill_from_rest completes and WS trades resume, Stathera audit of today's bars shows zero intra-day gaps (`tid_delta == 0` for all consecutive bar pairs within a single UTC day)
3. No regression in existing bar processing -- cargo nextest and pytest suites pass with zero failures
   **Plans**: 2 plans

Plans:

- [x] 01-01-PLAN.md -- Add last_completed_bar_tid field to processor and checkpoint (RUST-01)
- [x] 01-02-PLAN.md -- Wire into committed_floors, add PyO3 bindings, full regression (RUST-01, RUST-02)

### Phase 2: Midnight Preflight Verification

**Goal**: Every sidecar restart anchors today's bars from a verified midnight crossover trade ID, closing the multi-processor seam between kintsugi (yesterday) and sidecar (today)
**Depends on**: Phase 1 (correct dedup floor needed to verify junction bars are not suppressed after anchoring)
**Requirements**: PREFLIGHT-01, PREFLIGHT-02, PREFLIGHT-03
**Success Criteria** (what must be TRUE):

1. On checkpoint-cleared restart, sidecar queries Binance REST for each streaming symbol's first trade at or after UTC midnight (`startTime=midnight_ms&limit=1`) and obtains the crossover trade ID
2. Sidecar verifies that yesterday's orphan bar in ClickHouse has `last_agg_trade_id == crossover_trade_id - 1`. If verification fails, sidecar logs a WARNING and defers repair to kintsugi (no data corruption, no crash)
3. `fill_from_rest` uses the verified crossover trade ID as its anchor (not ClickHouse seed), and the resulting first bar of today satisfies `yesterday_orphan.last_agg_trade_id + 1 == today_first_bar.first_agg_trade_id`
4. For symbols where yesterday has no orphan bar in ClickHouse (gap or fresh deploy), sidecar falls back to the current fill_from_rest behavior gracefully -- no crash, no hang
   **Plans**: 2 plans

Plans:

- [x] 02-01-PLAN.md -- Add crossover query, orphan verification, and ClickHouse query (PREFLIGHT-01, PREFLIGHT-02)
- [x] 02-02-PLAN.md -- Wire preflight into \_create_engine, override fill_from_rest seeds (PREFLIGHT-03)

### Phase 3: Heartbeat Telemetry Overhaul

**Goal**: Heartbeat continuously validates zero-gap zero-overlap across every symbol with unified Stathera checks and LOUD alerts on any regression
**Depends on**: Phase 2 (telemetry must validate the fixes from Phase 1 and Phase 2 are working in production)
**Requirements**: TELEM-01, TELEM-02, TELEM-03, TELEM-04
**Success Criteria** (what must be TRUE):

1. Heartbeat reports a single unified Stathera continuity result per symbol -- no separate "intra-day" vs "midnight-boundary" classification. A gap is a gap, regardless of where it occurs
2. Heartbeat reports yesterday completion status per symbol: whether yesterday's orphan bar's `last_agg_trade_id` matches `crossover_trade_id - 1` (the verified midnight boundary)
3. After a sidecar restart, heartbeat reports junction telemetry: REST last trade ID, WS first trade ID, forming bar state, and committed_floors value -- sufficient to diagnose any junction issue without SSH
4. If any new Stathera gap appears in today's bars for any symbol, heartbeat sends a LOUD Telegram alert within 5 minutes containing exact (symbol, threshold, tid_delta, trade ID range)
   **Plans**: 2 plans

Plans:

- [x] 03-01-PLAN.md -- Unify Stathera check to per-symbol results, add today-gap LOUD alert (TELEM-01, TELEM-04)
- [x] 03-02-PLAN.md -- Add yesterday completion and junction telemetry checks (TELEM-02, TELEM-03)

## Progress

**Execution Order:**
Phases execute in numeric order: 1 -> 2 -> 3

| Phase                              | Plans Complete | Status      | Completed  |
| ---------------------------------- | -------------- | ----------- | ---------- |
| 1. Dedup Floor Fix                 | 2/2            | Complete    | 2026-03-21 |
| 2. Midnight Preflight Verification | 0/2            | Not started | -          |
| 3. Heartbeat Telemetry Overhaul    | 1/2 | In Progress|  |
