# CLAUDE.md - Project Memory

**opendeviationbar-py**: Rust workspace with Python bindings via PyO3/maturin. Publishes to PyPI (`opendeviationbar`) and crates.io (`opendeviationbar-core`, `opendeviationbar-providers`, `opendeviationbar-streaming`, `opendeviationbar-client`, `opendeviationbar-hurst`).

> **Naming**: This project was renamed from `rangebar-py` → `opendeviationbar-py` in March 2026. The algorithm is NOT a classic Nicolellis range bar (which uses dynamic `High - Low` range). It uses **open-anchored deviation thresholds**: breach when `price >= open * (1 + threshold)` or `price <= open * (1 - threshold)`. This is a genuinely novel bar type — an open-anchored, bidirectional, percentage-threshold bar. The name "Open Deviation Bar" precisely describes the mechanism. The legacy `rangebar` PyPI package and `rangebar-core`/`rangebar-providers`/`rangebar-streaming` crates.io packages redirect to the new names via shim releases.

---

## Quick Reference

| Task                         | Entry Point                                                                                             | Details                                               |
| ---------------------------- | ------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
| Generate open deviation bars | `get_open_deviation_bars()`                                                                             | [Python API](#python-api)                             |
| Real-time streaming          | `run_sidecar()` / `scripts/streaming_sidecar.py`                                                        | [ADR](/docs/adr/2026-01-31-realtime-streaming-api.md) |
| Understand architecture      | [docs/ARCHITECTURE.md](/docs/ARCHITECTURE.md)                                                           | 10-crate workspace                                    |
| Work with Rust crates        | [crates/CLAUDE.md](/crates/CLAUDE.md)                                                                   | Core algorithm, microstructure, inter-bar features    |
| Work with Python layer       | [python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md)                                 | API, caching, validation, symbol registry             |
| ClickHouse cache ops         | [python/opendeviationbar/clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md)           | Schema, population, remote setup, dedup hardening     |
| Plugin system                | [python/opendeviationbar/plugins/CLAUDE.md](/python/opendeviationbar/plugins/CLAUDE.md)                 | FeatureProvider protocol, entry-point discovery       |
| Operations & scripts         | [scripts/CLAUDE.md](/scripts/CLAUDE.md)                                                                 | Pueue, cache population, per-year parallelism         |
| Release workflow             | [docs/development/RELEASE.md](/docs/development/RELEASE.md)                                             | Zig cross-compile, mise tasks                         |
| Release + Deploy             | [.claude/skills/odb-ship/SKILL.md](/.claude/skills/odb-ship/SKILL.md)                                   | Unified release-publish-deploy pipeline               |
| Performance monitoring       | [docs/development/PERFORMANCE.md](/docs/development/PERFORMANCE.md)                                     | Benchmarks, metrics                                   |
| Performance optimization     | [.claude/pgo-workflow.md](/.claude/pgo-workflow.md)                                                     | LTO, PGO, SIMD, GPU strategies (Issue #96)            |
| Streaming sidecar (P0)       | [docs/development/ISSUE-107-SIDECAR-RELIABILITY.md](/docs/development/ISSUE-107-SIDECAR-RELIABILITY.md) | Watchdog, error recovery, systemd auto-restart        |
| Project context              | [docs/CONTEXT.md](/docs/CONTEXT.md)                                                                     | Why this project exists                               |
| API reference                | [docs/api/INDEX.md](/docs/api/INDEX.md)                                                                 | Full Python API docs                                  |
| Research                     | [docs/research/INDEX.md](/docs/research/INDEX.md)                                                       | ML labeling, regime patterns, TDA                     |
| Oracle verification          | [docs/verification/](/docs/verification/)                                                               | Bit-exact cross-reference reports                     |
| Source validation            | [.claude/skills/source-validation/SKILL.md](/.claude/skills/source-validation/SKILL.md)                 | Vision→REST→WS junction proof, fromId pagination      |
| Deploy secrets (local.env)   | `deploy/opendeviationbar.local.env`                                                                     | Survives git reset, holds tokens                      |
| Heartbeat system             | `scripts/health_heartbeat.py`                                                                           | Ephemeral Telegram heartbeat via systemd              |
| Historical backfill          | `mise run kintsugi:catchup`                                                                             | Kintsugi catch-up for historical backfill             |
| Autonomous agent ops         | [.claude/skills/autonomous-agent-ops/SKILL.md](/.claude/skills/autonomous-agent-ops/SKILL.md)           | Multi-agent monitoring, pueue guardrails, watchdogs   |

---

## Critical Principles

### 1. Leverage Rust

**ALWAYS use local Rust crates for heavy lifting.** Python handles I/O only.

1. **Check Rust first**: Before writing Python code, check if `opendeviationbar-core` already provides the capability
2. **Stream to Rust**: The `OpenDeviationBarProcessor` maintains state between `process_trades()` calls
3. **Checkpoint API**: Use `create_checkpoint()` and `from_checkpoint()` for cross-session continuity
4. **No reinventing**: Don't reimplement open deviation bar logic in Python

```python
# CORRECT: Stream to Rust (maintains state automatically)
for chunk in data_stream:
    bars = processor.process_trades(chunk.to_dicts())

# WRONG: Buffer in Python (OOM risk)
all_data = []
for chunk in data_stream:
    all_data.extend(chunk)
```

### 2. Kintsugi-First Gap Filling

**Kintsugi is the sole gap-filling mechanism.** All historical backfill flows through kintsugi with recency-first ordering (newest gaps first). Direct `populate_cache_resumable()` is guarded to max 366 days.

- **Daemon**: `mise run kintsugi:daemon` or `mise run kintsugi:catchup` (identical; `Conflicts=` prevents concurrent)
- **CLI defaults**: 60s active, 300s clean, 10 repairs. Always reverse-chronological (newest gaps first).
- `populate_cache_resumable()` has a `max_days=366` guard. Kintsugi bypasses with `max_days=None`.
- **Full details**: [scripts/CLAUDE.md](/scripts/CLAUDE.md#kintsugi-sole-gap-filling-mechanism)

### 3. Symbol Registry Gate

**Every symbol must be registered before processing.** Unregistered symbols raise `SymbolNotRegisteredError`.

- **SSoT**: `symbols.toml` (repo root symlink → `python/opendeviationbar/data/symbols.toml`)
- **Adding new symbols**: Edit `symbols.toml`, run `maturin develop`, then process
- **Override** (dev only): `export OPENDEVIATIONBAR_SYMBOL_GATE=off`
- **Full details**: [python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md#symbol-registry-issue-79)

### 4. Day-Mode Invariant (#264, #263)

**Every bar belongs to exactly one UTC day.** Cross-midnight bars cause cascading Sisyphean repair cycles in kintsugi. Three batch paths fixed:

| Path                                | File          | Fix                                                                     |
| ----------------------------------- | ------------- | ----------------------------------------------------------------------- |
| `_fetch_and_process_gap()`          | `backfill.py` | Filter `>= start_ms` + `split_trades_by_day()` + `reset_at_ouroboros()` |
| `_process_gap_fallback_starttime()` | `backfill.py` | `split_trades_by_day()` + `reset_at_ouroboros()`                        |
| `_discover_shards_stathera()`       | `kintsugi.py` | All `tid_delta > 0` gaps are real; no exclusions                        |

**Pre-write gate**: `store_bars_batch()` and `store_bars_bulk()` reject cross-midnight bars before INSERT.
**Oracle**: `health_heartbeat.py` checks `cross_midnight_bars` count every 5 min.
**Validation module**: [python/opendeviationbar/validation/day_mode/CLAUDE.md](/python/opendeviationbar/validation/day_mode/CLAUDE.md)

#### Bit-Exact Day Boundary Oracle

The gold-standard check: `day_x.last_agg_trade_id + 1 == day_x+1.first_agg_trade_id`. Run after any kintsugi repair or `backfill.py`/`kintsugi.py` change. Full query + interpretation: [validation/day_mode/CLAUDE.md](/python/opendeviationbar/validation/day_mode/CLAUDE.md#bit-exact-day-boundary-oracle).

**Verified production output (bigblack, 2026-03-12, BTCUSDT@250dbps):**

```
day_x       last_tid_day_x  close_utc                day_x1      first_tid_day_x1  open_utc                 tid_delta  verdict
2026-03-11  3901480937      2026-03-11 23:59:59.739  2026-03-12  3901480938        2026-03-12 00:00:00.081  0          PERFECT ✓
2026-03-10  3900165464      2026-03-10 23:59:59.009  2026-03-11  3900165465        2026-03-11 00:00:00.109  0          PERFECT ✓
2026-03-09  3898606471      2026-03-09 23:59:59.698  2026-03-10  3898606472        2026-03-10 00:00:00.116  0          PERFECT ✓
2026-03-08  3897105526      2026-03-08 23:59:59.396  2026-03-09  3897105527        2026-03-09 00:00:00.102  0          PERFECT ✓
2026-03-07  3896167900      2026-03-07 23:59:59.989  2026-03-08  3896167901        2026-03-08 00:00:00.179  0          PERFECT ✓
```

`tid_delta = 0` means the last aggTrade of day X and the first aggTrade of day X+1 are **literally consecutive integers**. UTC midnight fell between them (e.g., 81ms gap). No missing trades, no overlap, no cross-midnight bar.

### 5. Spot-Only Data Source Policy

**All Binance data uses spot market exclusively. No futures.**

| Source          | Endpoint                                                 | Notes                               |
| --------------- | -------------------------------------------------------- | ----------------------------------- |
| REST API        | `https://api.binance.com/api/v3/aggTrades`               | Spot only. NOT `fapi.binance.com`   |
| Vision archives | `https://data.binance.vision/data/spot/daily/aggTrades/` | Spot, daily only. NOT `futures/um/` |
| WebSocket       | `wss://stream.binance.com:9443/ws/{symbol}@aggTrade`     | Spot                                |

**REST and Vision produce identical day boundaries** — verified empirically (6/6 date pairs including cross-year, all tid_gap=0, exact trade-ID match). Vision is preferred for efficiency (bulk ZIP), not correctness. Both are fully deterministic at UTC midnight splits.

---

## Python API

```python
from opendeviationbar import get_open_deviation_bars, get_n_open_deviation_bars, process_trades_polars

# Date-bounded (backtesting)
df = get_open_deviation_bars("BTCUSDT", "2024-01-01", "2024-06-30")

# Count-bounded (ML training)
df = get_n_open_deviation_bars("BTCUSDT", n_bars=10000)

# Polars users (2-3x faster)
import polars as pl
trades = pl.scan_parquet("trades.parquet")
df = process_trades_polars(trades, threshold_decimal_bps=250)

# With microstructure features
df = get_open_deviation_bars("BTCUSDT", "2024-01-01", "2024-06-30", include_microstructure=True)

# Long ranges (>30 days) — must populate cache first
from opendeviationbar import populate_cache_resumable
populate_cache_resumable("BTCUSDT", "2019-01-01", "2025-12-31")
df = get_open_deviation_bars("BTCUSDT", "2019-01-01", "2025-12-31")
```

| API                           | Use Case                   | Details                                                                                                   |
| ----------------------------- | -------------------------- | --------------------------------------------------------------------------------------------------------- |
| `get_open_deviation_bars()`   | Date range, backtesting    | [docs/api/primary-api.md](/docs/api/primary-api.md)                                                       |
| `get_n_open_deviation_bars()` | Exact N bars, ML           | [docs/api/primary-api.md](/docs/api/primary-api.md)                                                       |
| `populate_cache_resumable()`  | Single segment (≤366 days) | [docs/api/cache-api.md](/docs/api/cache-api.md) — use `mise run kintsugi:catchup` for historical backfill |
| `process_trades_polars()`     | Polars DataFrames          | [docs/api/processing-api.md](/docs/api/processing-api.md)                                                 |
| `process_trades_chunked()`    | Large datasets >10M trades | [docs/api/processing-api.md](/docs/api/processing-api.md)                                                 |

---

## Architecture

```
opendeviationbar-py/
├── crates/                    10 Rust crates (5 on crates.io, 5 internal)
│   └── opendeviationbar-core/         Core algorithm, microstructure features
├── src/                       PyO3 bindings (Issue #94: domain modules)
│   ├── lib.rs                 Thin orchestrator (~200 lines)
│   ├── helpers.rs             Conversion functions
│   ├── core_bindings.rs       PyOpenDeviationBarProcessor, PyPositionVerification
│   ├── arrow_bindings.rs      Arrow export functions
│   ├── binance_bindings.rs    Binance data fetching
│   ├── exness_bindings.rs     Exness data fetching
│   ├── streaming_bindings.rs  LiveBarEngine, streaming classes
│   └── stream_manager_bindings.rs  StreamManager (Issue #161)
├── python/opendeviationbar/           Python API layer
│   ├── clickhouse/            ClickHouse cache (bigblack)
│   ├── sidecar.py             Streaming sidecar orchestrator (v12.20+)
│   ├── sidecar_api.py         Consumer API: ariadne, gap-fill, catchup (#213, #224)
│   ├── validation/            Microstructure validation
│   └── storage/               Tier 1 cache (Parquet)
├── scripts/                   Pueue jobs, validation, cache population
├── .cargo/config.toml         Cross-compile friendly rustflags
└── pyproject.toml             Maturin config
```

**Key files**: `src/lib.rs` (PyO3 module registration), `python/opendeviationbar/__init__.py` (public API), `crates/opendeviationbar-core/src/processor.rs` (core algorithm), `python/opendeviationbar/sidecar.py` (streaming sidecar), `python/opendeviationbar/sidecar_api.py` (consumer API: ariadne + gap-fill + catchup)

**Full architecture**: [docs/ARCHITECTURE.md](/docs/ARCHITECTURE.md)

---

## Development Commands

```bash
# Setup (mise manages all tools including zig)
mise install

# Build & test
mise run build              # maturin develop
mise run test               # Rust tests (cargo nextest)
mise run test-py            # Python tests (pytest)

# Quality
mise run check-full         # fmt + lint + test + deny

# Release (see docs/development/RELEASE.md)
mise run release:full       # Full 4-phase workflow
mise run release:linux      # Zig cross-compile (~55 sec)
mise run release:pypi         # Upload to PyPI

# Benchmarks
mise run bench:run          # Full benchmarks
mise run bench:validate     # Verify 1M ticks < 100ms
```

---

## Build System

Zig cross-compilation from macOS (no remote SSH needed).

| Platform     | Strategy          | Time    | Command                        |
| ------------ | ----------------- | ------- | ------------------------------ |
| macOS ARM64  | Native maturin    | ~10 sec | `mise run release:macos-arm64` |
| Linux x86_64 | Zig cross-compile | ~55 sec | `mise run release:linux`       |

**Why rustls**: Eliminates OpenSSL dependency, enabling pure Rust cross-compilation.

**Full release workflow**: [docs/development/RELEASE.md](/docs/development/RELEASE.md)

---

## Microstructure Features

10 intra-bar features + 16 inter-bar lookback features computed in Rust during bar construction.

**Intra-bar**: `ofi`, `vwap_close_deviation`, `kyle_lambda_proxy`, `trade_intensity`, `volume_per_trade`, `aggression_ratio`, `aggregation_density`, `turnover_imbalance`, `duration_us`, `price_impact`

**Inter-bar** (lookback window): `lookback_ofi`, `lookback_intensity`, `lookback_kyle_lambda`, `lookback_burstiness`, `lookback_hurst`, `lookback_permutation_entropy`, + 10 more

**Formulas and ranges**: [crates/CLAUDE.md](/crates/CLAUDE.md#microstructure-features-v70)

**Validation**: [python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md#validation-framework-v70)

---

## ClickHouse Infrastructure

All open deviation bar data served from **bigblack** (remote GPU host). No local ClickHouse.

| Host        | Thresholds (dbps)   | ClickHouse | Total Bars     |
| ----------- | ------------------- | ---------- | -------------- |
| bigblack    | 250, 500, 750, 1000 | Native     | ~42M (growing) |
| littleblack | 100                 | Docker     | —              |

**Connection mode**: `OPENDEVIATIONBAR_MODE=remote` (set in `.mise.toml`). Preflight: `mise run db:ensure`.

**Cache operations**: [python/opendeviationbar/clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md) (schema, population, dedup hardening)

**Plugin columns**: Not in `schema.sql` — managed via `ALTER TABLE ADD COLUMN IF NOT EXISTS` by each `FeatureProvider` plugin. See [python/opendeviationbar/plugins/CLAUDE.md](/python/opendeviationbar/plugins/CLAUDE.md).

**Distributed jobs**: [scripts/CLAUDE.md](/scripts/CLAUDE.md) (pueue, per-year parallelization, autoscaler)

---

## Monitoring & Gap Prevention

Automated monitoring to prevent silent data loss in the ClickHouse cache. Born from the Feb 2026 incident where 5.5 days of BTCUSDT@500 bars were lost due to `try_cache_write()` silently swallowing connection errors.

### Root Cause (Resolved)

`try_cache_write()` in `orchestration/open_deviation_bars_cache.py` was fire-and-forget — `populate_cache_resumable()` reused it even though ClickHouse IS the destination, not an optional cache. When SSH tunnel dropped, writes failed silently for days while the checkpoint advanced.

### Defense-in-Depth (Issue #158 Phase 7)

**Write integrity**: Four layers prevent silent data loss. **Overlap prevention**: Four layers prevent duplicate bars from concurrent writers.

| Layer               | Mechanism                                                              | Details                                                             |
| ------------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------- |
| **Bounded flush**   | `OPENDEVIATIONBAR_FLUSH_THRESHOLD` (default 10K) in `_process_day()`   | Caps peak bars-in-flight during populate; prevents memory spikes    |
| **Fatal writes**    | `_fatal_cache_write()` in populate path                                | Raises on ClickHouse failure instead of warning                     |
| **Circuit breaker** | `fatal_write_breaker` (3 failures → OPEN, 60s recovery)                | Prevents cascade during ClickHouse outages                          |
| **T-1 guard**       | `populate_cache_resumable()` clamps `end_date`                         | Prevents crash on unavailable Binance Vision data                   |
| **Gap detection**   | `scripts/detect_gaps.py` + heartbeat timer                             | Every 5 min via systemd, alerts via @obdpy_bot                      |
| **L1 Redis lock**   | `cache_write_lock()` per (symbol, threshold)                           | Prevents concurrent writes; graceful degradation if Redis down      |
| **L2 Schema dedup** | ORDER BY `first_agg_trade_id` in ReplacingMergeTree                    | Native dedup on merge — same trades from different writers collapse |
| **L3 Heartbeat**    | Service liveness + Stathera audit + disk + circuit breaker every 5 min | `health_heartbeat.py` (sole Telegram telemetry checkpoint)          |

### Telegram Notifications (@obdpy_bot)

| Event                      | Behavior                                                            |
| -------------------------- | ------------------------------------------------------------------- |
| All checks pass            | Ephemeral green heartbeat, auto-deletes after 15 min (~2-3 visible) |
| Gap or regression detected | **Persistent** red alert, LOUD notification, stays forever          |
| Cache write failure        | Circuit breaker alert via `opendeviationbar.notify.telegram`        |

**Config**: `OPENDEVIATIONBAR_TELEGRAM_TOKEN` in `.mise.local.toml` (local) or `deploy/opendeviationbar.local.env` (bigblack). Chat ID in `.mise.toml`.

**Heartbeat**: `scripts/health_heartbeat.py` via systemd timer (`opendeviationbar-heartbeat.timer`, every 5 min). Checks: ClickHouse, sidecar, kintsugi, Stathera audit (gaps + overlaps), disk, circuit breaker. **Sole Telegram telemetry checkpoint** for all ODB monitoring (#238 Phase 3).

**Full details**: [scripts/CLAUDE.md](/scripts/CLAUDE.md#gap-detection--monitoring)

### Monitoring Stack (bigblack)

**Monit** (watchdog-of-watchdogs): 9 monitors in `deploy/monit/opendeviationbar.conf`, daemon cycle 120s.

| Monitor         | Type    | Check                                        | Debounce      |
| --------------- | ------- | -------------------------------------------- | ------------- |
| clickhouse      | host    | HTTP `/ping` on :8123                        | 3 cycles      |
| sidecar         | process | PID exists, mem < 2048 MB, cpu < 80%         | immediate/3/5 |
| sidecar-health  | host    | HTTP `/health` on :8081                      | 3 cycles      |
| kintsugi        | process | PID exists, mem < 5120 MB                    | immediate/3   |
| heartbeat-timer | program | `obdpy-check-heartbeat-timer.sh` exit code   | 3 cycles      |
| system ($HOST)  | system  | loadavg < 8, mem < 85%, swap < 25%           | 3/5 cycles    |
| redis           | host    | Redis protocol on :6379                      | 3 cycles      |
| rootfs          | fs      | Disk < 80%/90%, inodes < 85%                 | immediate     |
| monit-heartbeat | program | `obdpy-monit-heartbeat.sh` (every 10 cycles) | 3 cycles      |

**Alert script**: `/usr/local/bin/obdpy-monit-alert.sh` — sources `local.env` (TOKEN) and `opendeviationbar.env` (CHAT_ID), sends Telegram alert. All exec actions run `as uid "tca" and gid "tca"`.

**Permission model**: Monit runs as root. Requires systemd drop-in (`deploy/monit/monit-override.conf`) to disable `NoNewPrivileges`, add `CAP_SETUID`/`CAP_SETGID`, set `ProtectHome=no`.

**External monitoring ports** (bigblack, not ODB-managed):

| Service       | Port  | Purpose             |
| ------------- | ----- | ------------------- |
| Prometheus    | 9090  | Metrics collection  |
| node_exporter | 9100  | Host metrics        |
| Netdata       | 19999 | Real-time dashboard |
| Cockpit       | 9091  | System admin web UI |

---

## Common Errors

| Error                                          | Cause                         | Fix                                                      |
| ---------------------------------------------- | ----------------------------- | -------------------------------------------------------- |
| `OpenDeviationBarProcessor has no attribute X` | Outdated binding              | `maturin develop`                                        |
| `Invalid threshold_decimal_bps`                | Wrong units                   | Use 250 for 0.25%                                        |
| `High < Low` assertion                         | Bad input data                | Check sorting                                            |
| `target-cpu=native` cross-compile error        | RUSTFLAGS pollution           | Use `RUSTFLAGS=""` or `.cargo/config.toml`               |
| OOM with `include_microstructure=True`         | Large date range              | MEM-011 adaptive chunk size                              |
| Duplicate ticks in Parquet cache               | Pre-v12.8 bug                 | `scripts/deduplicate_parquet_cache.py`                   |
| `SymbolNotRegisteredError`                     | Symbol not in registry        | Edit `symbols.toml`, `maturin develop`                   |
| `ValueError: Date range spans N days`          | Range exceeds max_days (366)  | Use `mise run kintsugi:catchup` for historical backfill  |
| `TypeError` on gap-fill response iteration     | v12.62+ returns dict not list | Read `.trades` from response (see breaking change below) |

**Memory guards**: [python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) (MEM-001 through MEM-013)

### Breaking Change: gap_fill_trades() Response (v12.62+, #220)

`GET /trades/gap-fill` and `gap_fill_trades()` now return a dict instead of a bare list:

```json
{"trades": [...], "partial": false, "count": 1234}
```

- `trades`: list of `{a, p, q, T, m}` dicts (same format as before)
- `partial`: `true` when REST bridge failed after Parquet returned partial data
- `count`: number of trades returned

Consumers must update: `trades = response["trades"]` (was: `trades = response`).

### Breaking Change: ariadne_lookup() includes sources_tried (v12.62+, #221)

`GET /ariadne/{symbol}/{threshold}` responses now include `sources_tried` — a diagnostic list showing which sources were attempted and their outcomes. No consumer code change required (additive field).

### New: Server-Side Gap-Fill (v12.62+, #224-#228)

`GET /catchup/{symbol}/{threshold}` — single-call gap-fill from last committed CH bar. Composes `ariadne_lookup(committed_only=True)` + paginated Parquet→REST gap-fill. Replaces 3-step client flow.

**New ariadne query param**: `?committed_only=true` skips Sources 1-2 (live processor, checkpoint file), returns only committed ClickHouse data.

**New functions** in `sidecar_api.py`:

- `gap_fill_trades_paginated(symbol, from_agg_id, max_trades)` — server-side pagination, 100K hard cap
- `catchup_lookup(symbol, threshold, max_trades, engine)` — composite endpoint

```bash
# Usage
curl 'http://localhost:8081/catchup/BTCUSDT/250?max_trades=50000' | jq '{count, partial, source_breakdown}'
curl 'http://localhost:8081/ariadne/BTCUSDT/250?committed_only=true' | jq '.source'
```

---

## CLAUDE.md Network (Hub-and-Spoke)

This file is the **hub**. Each spoke CLAUDE.md is loaded automatically when working in that directory.

| Directory                                       | CLAUDE.md                                                                                     | SSoT For                                                                  |
| ----------------------------------------------- | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
| `/`                                             | This file                                                                                     | Project overview, API, architecture, principles                           |
| `/crates/`                                      | [crates/CLAUDE.md](/crates/CLAUDE.md)                                                         | Rust crates, microstructure formulas, algorithm details                   |
| `/python/opendeviationbar/`                     | [python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md)                       | Python API, caching, validation, symbol registry, memory guards           |
| `/python/opendeviationbar/clickhouse/`          | [python/opendeviationbar/clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md) | ClickHouse schema (core only), dedup hardening, cache population          |
| `/python/opendeviationbar/config/`              | [python/opendeviationbar/config/CLAUDE.md](/python/opendeviationbar/config/CLAUDE.md)         | Configuration via pydantic-settings, env var priority                     |
| `/python/opendeviationbar/plugins/`             | [python/opendeviationbar/plugins/CLAUDE.md](/python/opendeviationbar/plugins/CLAUDE.md)       | FeatureProvider protocol, plugin schema migration, entry-point discovery  |
| `/python/opendeviationbar/validation/day_mode/` | [validation/day_mode/CLAUDE.md](/python/opendeviationbar/validation/day_mode/CLAUDE.md)       | Day-mode invariant enforcement: pre-write gate, oracle, observation       |
| `/scripts/`                                     | [scripts/CLAUDE.md](/scripts/CLAUDE.md)                                                       | Kintsugi, gap detection, Stathera, validation scripts                     |
| `/scripts/systemd/`                             | [scripts/systemd/CLAUDE.md](/scripts/systemd/CLAUDE.md)                                       | 7 services, 3 timers, TZ=UTC, OOM protection, dependency chain, local.env |

---

## Terminology

| Term                    | Definition                                                                                                                                                                                                                                                                                       |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **dbps**                | 1 dbps = 0.00001 = 0.001%. Example: 250 dbps = 0.25%. **All threshold values use dbps.**                                                                                                                                                                                                         |
| **Ouroboros**           | UTC-midnight daily reset boundary. Resets processor state at each day boundary for reproducibility and cache-friendly processing. One reset per day — all 42M+ bars on bigblack are day mode.                                                                                                    |
| **Orphan Bar**          | Incomplete bar emitted when the processor resets at UTC midnight. Marked `is_orphan=True`. One per day maximum (the bar in progress at 00:00:00 UTC). Filter for ML: `df[~df.get("is_orphan", False)]`. Synonyms: "orphaned bar", "boundary bar".                                                |
| **Stathera**            | (σταθερά) Trade-ID alignment paradigm (Issue #158). Invariant: `bar[i].last_agg_trade_id + 1 == bar[i+1].first_agg_trade_id` for all consecutive bars within same (symbol, threshold). Uses aggregate trade IDs already in ClickHouse as source of truth. Replaces all time-based gap detection. |
| **Stathera Anomaly**    | A violation of the Stathera invariant. `tid_delta > 0` = gap (missing aggTrade records). `tid_delta < 0` = overlap (bars share aggTrade ranges from different processing sources).                                                                                                               |
| **Kintsugi**            | Self-healing gap reconciliation (Issue #115). Discovers Stathera gaps (shards) via trade-ID continuity check and repairs using Ariadne `fromId` pagination + Ouroboros. All `tid_delta > 0` gaps are real and trigger repairs.                                                                   |
| **Shard**               | A Stathera gap — consecutive bars with `tid_delta > 0` (missing aggTrade records). Identified by `anchor_last_tid` and `resume_first_tid`. Repaired by backfill_gap() (Vision preferred for efficiency, REST produces identical results).                                                        |
| **Circuit Breaker**     | `fatal_write_breaker` in `open_deviation_bars_cache.py`. States: CLOSED (normal), OPEN (3 consecutive failures, blocks writes), HALF_OPEN (60s recovery probe). Prevents cascade failures during ClickHouse outages.                                                                             |
| **Ariadne**             | (Ἀριάδνη) `fromId` pagination for REST API gap fill. Thread of trade IDs through the labyrinth — uses `agg_trade_id` anchors to fetch exactly the missing trades. Zero-gap by construction.                                                                                                      |
| **Antaeus**             | (Ανταίος) Resilient WebSocket reconnection. Backoff resets after stable connection (≥60s); jitter via `backon` prevents thundering herd. Named for the giant who regained strength by touching the earth.                                                                                        |
| **Lethe**               | (Λήθη) Silent data loss detection. Alerts when ring buffer drops bars during backpressure. Named for the river of forgetfulness — ensures nothing is silently forgotten.                                                                                                                         |
| **Argus**               | (Ἄργος Πανόπτης) Universal liveness watchdog. Tracks per-symbol last-bar time; alerts when any symbol goes silent for ≥30 min regardless of aggregate trade flow. The hundred-eyed, all-seeing.                                                                                                  |
| **Charon**              | (Χάρων) Dead-letter ferry. Sidecar autonomously replays failed-write Parquet files on startup and every 5 min, without requiring kintsugi. Ferryman carrying lost bars back to ClickHouse.                                                                                                       |
| **Puck**                | (Shakespeare, _A Midsummer Night's Dream_) Inline Rust gap fill. Detects trade-ID discontinuity on every WS trade via `TradeIdGapDetector`, fills in sub-3s via parallel REST fetch (`puck_fill()`). "I'll put a girdle round about the earth in forty minutes."                                 |
| **Niffler**             | (Rowling, _Fantastic Beasts_) Immediate dead-letter replay. When CH write fails and bars are dead-lettered, instantly triggers `replay_dead_letters()` instead of waiting for Charon's 5-min cycle. Retrieves lost bar-treasures within seconds.                                                 |
| **StreamManager**       | Managed streaming engine wrapping `LiveBarEngine` with adaptive rate limiting (Issue #162), trade-ID continuity tracking seeded from ClickHouse, and shadow validation diagnostics. Drop-in replacement used by sidecar since v12.49.                                                            |
| **AdaptiveRateLimiter** | Token bucket rate limiter for Binance REST API (Issue #162). Syncs with `X-MBX-USED-WEIGHT-1m` server header, 2400 weight/min budget, 20% safety margin. Paces gap-fill REST calls in `puck_fill()`.                                                                                             |
| **Shadow Validation**   | Diagnostic background loop in StreamManager. Every 5s, fetches latest trade via REST and compares with WS-tracked trade IDs. Skips when rate limiter utilization > 60%. Divergences logged, not auto-corrected.                                                                                  |

<!-- gitnexus:start -->

# GitNexus MCP

This project is indexed by GitNexus as **opendeviationbar-py** (8147 symbols, 21188 relationships, 300 execution flows).

## Always Start Here

1. **Read `gitnexus://repo/{name}/context`** — codebase overview + check index freshness
2. **Match your task to a skill below** and **read that skill file**
3. **Follow the skill's workflow and checklist**

> If step 1 warns the index is stale, run `npx gitnexus analyze` in the terminal first.

## Skills

| Task                                         | Read this skill file                                        |
| -------------------------------------------- | ----------------------------------------------------------- |
| Understand architecture / "How does X work?" | `.claude/skills/gitnexus/gitnexus-exploring/SKILL.md`       |
| Blast radius / "What breaks if I change X?"  | `.claude/skills/gitnexus/gitnexus-impact-analysis/SKILL.md` |
| Trace bugs / "Why is X failing?"             | `.claude/skills/gitnexus/gitnexus-debugging/SKILL.md`       |
| Rename / extract / split / refactor          | `.claude/skills/gitnexus/gitnexus-refactoring/SKILL.md`     |
| Tools, resources, schema reference           | `.claude/skills/gitnexus/gitnexus-guide/SKILL.md`           |
| Index, status, clean, wiki CLI commands      | `.claude/skills/gitnexus/gitnexus-cli/SKILL.md`             |

<!-- gitnexus:end -->
