# Scripts & Operations

**Parent**: [/CLAUDE.md](/CLAUDE.md) | **Skills**: `Skill(devops-tools:pueue-job-orchestration)` | `Skill(opendeviationbar-job-safety)`

Operational scripts for cache population, validation, and distributed job management on remote GPU hosts.

---

## Quick Reference

| Task                           | Command                                           | Details                                   |
| ------------------------------ | ------------------------------------------------- | ----------------------------------------- |
| Kintsugi catch-up (historical) | `mise run kintsugi:catchup`                       | [Kintsugi](#kintsugi)                     |
| Check cache status             | `mise run cache:status`                           | ClickHouse bar counts                     |
| Detect volume overflow         | `mise run cache:detect-overflow`                  | Issue #88                                 |
| Validate dedup hardening       | `mise run cache:validate-dedup`                   | Issue #90                                 |
| Validate microstructure        | `uv run python scripts/validate_microstructure_*` | [Validation Scripts](#validation-scripts) |
| Run streaming sidecar          | `mise run streaming:start`                        | v12.20+, real-time open deviation bars    |
| Kintsugi (single pass)         | `mise run kintsugi`                               | [Kintsugi](#kintsugi)                     |
| Kintsugi (daemon)              | `mise run kintsugi:daemon`                        | Long-running self-healer                  |
| Kintsugi (dry-run)             | `mise run kintsugi:check`                         | Discover shards without repairing         |

---

## Bigblack Capacity Envelope

Hardware profile: **Ryzen 9 7950X (32 cores), 61 GB RAM, NVMe SSD, 1Gbps Ethernet, RTX 4090 (unused by ODB pipeline).**

### Safe Parallelism Defaults

| Workload type             | Safe default workers          | Example                                    |
| ------------------------- | ----------------------------- | ------------------------------------------ |
| CPU-bound (Parquet→Rust)  | `min(nproc, 24)` = **24**     | `repair_direct_parquet.py`                 |
| Network-bound (Vision DL) | **12**                        | Seeder downloads, kintsugi Vision fallback |
| ClickHouse-bound          | **24** (CH soft cap ~143 qps) | Kintsugi concurrent repairs                |
| Parquet-cached repairs    | **64**                        | Kintsugi when all shards have local ticks  |

### fd Budget

| Context              | `ulimit -n` | Budget rule                                                |
| -------------------- | ----------- | ---------------------------------------------------------- |
| Ad-hoc shell scripts | 1024        | Thread-local CH cache: ~4N+50 fds for N workers. Max ~243. |
| systemd services     | 65536       | No fd constraint. Set via `LimitNOFILE=` in service units. |

### Anti-Pattern: Conservative Defaults

**Do NOT default batch scripts to 2-4 workers "to be safe".** This wastes 90%+ of bigblack's hardware. The root cause of past conservatism was fd leaks (now fixed via thread-local CH cache reuse, Issue #279).

**Rule for new scripts**: Default worker count should be `min(os.cpu_count() or 8, 24)` for CPU-bound work, or `12` for network-bound work. Never hardcode a small number without documenting why.

### Observed Utilization at Optimal Settings

| Setting                            | CPU  | RAM    | CH Active Queries | Throughput     |
| ---------------------------------- | ---- | ------ | ----------------- | -------------- |
| Kintsugi @ 24 workers              | 22%  | 4.6 GB | 143               | 44 repairs/30s |
| Seeder @ 12 concurrent downloads   | <5%  | 2.5 GB | n/a               | 48 files/min   |
| repair_direct_parquet @ 24 workers | ~20% | ~13 GB | ~100              | ~24 days/min   |

**ClickHouse is the bottleneck** at 24 workers (soft cap ~143-145 concurrent queries), not CPU, RAM, or NVMe. Increasing beyond 24 causes contention collapse (32 workers = 0 completions/30s).

---

## Kintsugi: Sole Gap-Filling Mechanism

**All gap filling flows through kintsugi.** Direct `populate_cache_resumable()` is guarded to max 366 days. Historical backfill uses kintsugi catch-up mode, which enforces recency-first ordering at every level.

### Reverse Chronological Invariant (3 Levels)

| Level                      | What                              | How                                                                                        |
| -------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------ |
| **L1: Shard selection**    | Which gap to repair first         | `_shard_sort_key()` sorts by `-gap_end_ms`                                                 |
| **L2: Segment processing** | Within a gap, which month first   | `reversed(segments)` — newest Ouroboros segment first                                      |
| **L3: Day processing**     | Within a segment, which day first | Chronological (required for processor state), but L1+L2 ensure newest _segment_ runs first |

### Polling Intervals

The CLI (`scripts/kintsugi.py`) defaults to aggressive intervals for all deployments:

| Parameter           | CLI Default | Python API Default |
| ------------------- | ----------- | ------------------ |
| `--interval-active` | 60s         | 900s               |
| `--interval-clean`  | 300s        | 1800s              |
| `--max-repairs`     | 10          | 3                  |

Both `kintsugi` and `kintsugi-catchup` systemd services use the CLI defaults (identical `ExecStart`). The two services exist for operational control (`Conflicts=` prevents concurrent runs) — not for different modes. **Reverse-chronological processing is the only mode.**

### Parallelism Tuning (bigblack)

> **SSoT for all bigblack values: [`deploy/opendeviationbar.env`](/deploy/opendeviationbar.env)**
> Edit that file — not this table. Values here are _descriptions only_; the env file is what actually runs.

| Env Var                                            | Code Default | Bigblack Value | Purpose                                                                                                                                                                                                                       |
| -------------------------------------------------- | ------------ | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `OPENDEVIATIONBAR_KINTSUGI_MAX_REPAIRS_PER_PASS`   | 3            | 500            | Cap shards repaired per pass                                                                                                                                                                                                  |
| `OPENDEVIATIONBAR_KINTSUGI_MAX_CONCURRENT_REPAIRS` | 4            | **24**         | Outer parallelism for Vision-download shards. **24 = empirical sweet spot on bigblack (2026-03-13).** Do NOT increase to 32 (cliff: 0 repairs/30s). See [kintsugi parallelism tuning](#kintsugi-parallelism-tuning-bigblack). |
| `OPENDEVIATIONBAR_KINTSUGI_MAX_CONCURRENT_PARQUET` | 64           | 64             | Outer parallelism when ALL candidates are Parquet-cached (no download, no cascade risk)                                                                                                                                       |
| `OPENDEVIATIONBAR_KINTSUGI_BATCH_WORKERS`          | 4            | 4              | Inner-level day workers per shard                                                                                                                                                                                             |
| `OPENDEVIATIONBAR_KINTSUGI_BATCH_DAYS`             | 30           | 365            | Days per shard batch (365 = one year; caps peak RSS ~3.8 GB vs 3650-day runs that hit 5.3 GB+)                                                                                                                                |
| `OPENDEVIATIONBAR_REDIS_LOCK_BLOCKING_TIMEOUT`     | 60           | 2              | Redis lock wait (s) before falling through to L2 ReplacingMergeTree dedup                                                                                                                                                     |

#### Dynamic Parallelism (implemented)

`kintsugi_pass()` selects the concurrency limit at runtime via `_shard_is_parquet_cached()`:

- ALL candidates Parquet-cached → `max_concurrent_parquet` (high, no network constraint)
- ANY candidate needs Vision download → `max_concurrent_repairs` (network-safe)

#### Kintsugi Parallelism Tuning (bigblack)

Empirical step-up test on bigblack (2026-03-13): Ryzen 9 7950X, 32 cores, 61 GB RAM, NVMe, local ClickHouse. BTCUSDT Parquet cache fully populated — zero Vision downloads in hot path.

| MAX_CONCURRENT_REPAIRS | CPU  | CH active queries | Repairs/30s | Verdict                       |
| ---------------------- | ---- | ----------------- | ----------- | ----------------------------- |
| 8                      | 8.8% | 130               | ~8          | baseline                      |
| 16                     | 19%  | 138               | 17          | linear ✓                      |
| **24**                 | 22%  | 143               | **44**      | super-linear ✓ **sweet spot** |
| 32                     | 14%  | 271               | **0**       | cliff ✗ — contention          |

**Why 24 is optimal**: Pipeline parallelism (Parquet read + Rust bar compute + CH write) overlaps across 24 concurrent repairs. ClickHouse soft cap is ~143-145 concurrent active queries — 24 saturates it cleanly.

**Why 32 fails**: CH active queries double to 271, exceeding the soft cap. CPU paradoxically drops (threads block waiting for CH scheduler). Zero completions in 4-minute observation window.

**Bottleneck at 24**: NOT CPU (22%), NOT NVMe (0% disk reads — served from page cache), NOT RAM (4.6 GB of 36 GB). The ClickHouse internal query scheduler governs throughput.

**Do NOT increase beyond 24.** This is a hard finding, not a soft guideline.

#### Redis Lock Serialization (Issue #266, fixed)

Default `lock_blocking_timeout=60s` serialized all workers to effective parallelism=1. **Fix**: set `OPENDEVIATIONBAR_REDIS_LOCK_BLOCKING_TIMEOUT=2` — workers immediately fall through to concurrent writes. L2 dedup (ReplacingMergeTree `ORDER BY first_agg_trade_id`) handles safety.

#### Two-Mode Throughput

| Shard type          | Bottleneck            | Notes                                                    |
| ------------------- | --------------------- | -------------------------------------------------------- |
| **Parquet-cached**  | CH query scheduler    | Data in `~/.cache/opendeviationbar/ticks/` — no download |
| **Vision-download** | USB ethernet ~10 MB/s | First-time dates need Binance Vision ZIP download        |

**Bigblack status (2026-03-13)**: BTCUSDT 2018-01-17 → 2026-03-12 fully seeded (531 days, 42 GB local NVMe, 99 Parquet files). All kintsugi repairs use the 64-worker Parquet path — zero Vision downloads in hot path. Seeder runs as 5-min timer to keep cache current for rolling window.

**Symbol scope**: Seeder processes all enabled symbols (`get_registered_symbols(enabled_only=True)`). Currently BTCUSDT + ETHUSDT processed sequentially. `OPENDEVIATIONBAR_SEEDER_MAX_CONCURRENT_DOWNLOADS=12` applies per-symbol (see [Bigblack Capacity Envelope](#bigblack-capacity-envelope)). Multi-threshold efficiency means all 4 thresholds reuse the same raw Parquet files per symbol.

### Focus Mode (Operational Override)

To restrict kintsugi to a specific symbol/threshold (e.g., during a large backfill of BTCUSDT@250), use a systemd drop-in:

```bash
# Create focus-mode drop-in on bigblack
mkdir -p ~/.config/systemd/user/opendeviationbar-kintsugi.service.d/
cat > ~/.config/systemd/user/opendeviationbar-kintsugi.service.d/focus-mode.conf << 'EOF'
[Service]
ExecStart=
ExecStart=/home/tca/opendeviationbar-py/.venv/bin/python3 /home/tca/opendeviationbar-py/scripts/kintsugi.py --daemon --symbol BTCUSDT --threshold 250
EOF
systemctl --user daemon-reload
systemctl --user restart opendeviationbar-kintsugi
```

Drop-ins live in `~/.config/systemd/user/opendeviationbar-kintsugi.service.d/` (NOT in git) and survive `git reset --hard`. Remove them when the focused backfill is complete:

```bash
rm ~/.config/systemd/user/opendeviationbar-kintsugi.service.d/focus-mode.conf
systemctl --user daemon-reload && systemctl --user restart opendeviationbar-kintsugi
```

Heartbeat shows `⚠️ overrides active: opendeviationbar-kintsugi` when drop-ins exist — this is informational only, not UNHEALTHY (v13.15.4+).

### Usage

```bash
# Normal steady-state healing
mise run kintsugi:daemon

# Aggressive historical backfill (start manually, stop when caught up)
mise run kintsugi:catchup

# Single-job bounded population (max 366 days)
uv run python scripts/populate_full_cache.py \
    --symbol BTCUSDT --threshold 250 \
    --start-date 2025-01-01 --end-date 2025-12-31
```

---

## Monitoring

```bash
# Cache status
mise run cache:status                 # ClickHouse bar counts
python scripts/cache_status.py        # Direct

# System resources (bigblack)
ssh bigblack 'uptime && free -h'
```

---

## Gap Detection & Monitoring

Automated gap detection to prevent silent data loss in ClickHouse. Born from the Feb 2026 BTCUSDT@500 incident (5.5-day gap caused by `try_cache_write()` silently swallowing SSH tunnel failures).

### Scripts

| Script                           | Purpose                                            | Exit Code              |
| -------------------------------- | -------------------------------------------------- | ---------------------- |
| `detect_gaps.py`                 | Find temporal/price gaps + freshness in ClickHouse | 0=clean, 1=gaps, 2=err |
| `verify_checkpoint_integrity.py` | Cross-validate checkpoint files vs ClickHouse      | 0=clean, 1=mismatch    |

### Usage

```bash
# Direct script usage (no mise task exists)
python scripts/detect_gaps.py --help
python scripts/detect_gaps.py --symbol BTCUSDT --threshold 500
python scripts/detect_gaps.py --recent-days 7 --json

# Freshness check (flag stale data >72 hours since last bar)
python scripts/detect_gaps.py --recent-days 7 --max-stale-hours 72

# Checkpoint integrity
python scripts/verify_checkpoint_integrity.py --quiet
```

### Freshness Detection

The `--max-stale-hours N` flag computes `now - latest_bar` for each (symbol, threshold) pair. If any pair hasn't produced a bar in N hours, it's flagged as stale and triggers exit code 1. The heartbeat uses `--max-stale-hours 72` (3 days).

This catches scenarios where ALL thresholds for a symbol stop simultaneously (e.g., backfill_watcher not running) — something inter-bar gap detection alone misses because it only checks gaps between consecutive bars within existing data.

### Telegram Alerts (@obdpy_bot)

Gap detection integrates with `opendeviationbar.notify.telegram`:

- **Gaps found** → persistent alert (stays forever, LOUD notification)
- **All clear** → no message (or ephemeral heartbeat via launchd)
- **`--no-notify`** flag suppresses Telegram (for dry runs)

### Heartbeat (systemd timer — bigblack only)

`scripts/health_heartbeat.py` runs every 5 minutes via `opendeviationbar-heartbeat.timer`. Self-contained (no package imports for Telegram API — avoids version mismatch with installed PyPI package).

| Check           | Method                              | Threshold   |
| --------------- | ----------------------------------- | ----------- |
| ClickHouse      | `SELECT 1` via `clickhouse_connect` | Reachable   |
| sidecar         | `systemctl --user is-active`        | Active      |
| kintsugi        | `systemctl --user is-active`        | Active      |
| Stathera audit  | `lagInFrame()` gap + overlap query  | 0 anomalies |
| Disk            | `shutil.disk_usage("/")`            | >10 GB free |
| Circuit breaker | `get_fatal_write_breaker().state`   | Not OPEN    |

**Healthy**: Ephemeral message, tracked in `~/.cache/opendeviationbar/heartbeat/pending_messages.json`, auto-deleted after 15 min (~2-3 visible at any time).

**Unhealthy**: LOUD persistent message, never auto-deleted.

**Systemd files**: `scripts/systemd/opendeviationbar-heartbeat.{service,timer}` | [systemd CLAUDE.md](/scripts/systemd/CLAUDE.md)

### Gap Detection Query (SSoT)

Uses ClickHouse `lagInFrame()` window function to compute trade-ID deltas within each `(symbol, threshold)` partition. `tid_delta > 0` = gap, `tid_delta < 0` = overlap.

---

## Writer Ownership Model (#280)

**Sidecar owns today (UTC). Kintsugi owns yesterday and older.** Clean boundary eliminates concurrent-writer races that recurred across 4 fix cycles (Mar 3→9→12→15→16).

| Writer   | Scope               | Gap Healing                       |
| -------- | ------------------- | --------------------------------- |
| Sidecar  | Today (UTC)         | Puck (inline, real-time)          |
| Kintsugi | Yesterday and older | Stathera discover → day-recompute |

- **Kintsugi today-guard is ABSOLUTE** — no interior-gap exception. All gaps touching today clamp to yesterday or defer.
- **Puck** (sidecar): inline real-time gap fill on every WS trade. Sub-3s recovery.
- **Asclepius REMOVED** (#284): both `backfill_gap()` and `fill_gap()` proved fundamentally unsafe for old gaps on a live engine — caused cascading overlaps. Only Puck (real-time) and kintsugi (tomorrow) are safe.

### Persistent Midnight-Boundary Gap Root Cause (2026-03-19 investigation)

**The 13 recurring midnight gaps in Stathera are NOT a sidecar/Puck/Rust bug.** They affect only non-streaming symbols (all symbols except BTCUSDT and ETHUSDT). Root cause: the sidecar only streams BTCUSDT and ETHUSDT (`OPENDEVIATIONBAR_STREAMING_SYMBOLS=BTCUSDT,ETHUSDT` in `deploy/sidecar.env`). The other 16 symbols are filled entirely by **kintsugi + trailing-edge backfill** — two independent writers using separate processor instances.

**Why the gap appears at day boundaries:** Kintsugi processes each Ouroboros day independently (fresh processor per day). Trailing-edge backfill starts a fresh processor from REST data. At the day boundary, the two processors don't share state — kintsugi's last bar ends at trade N, but the backfill's first bar starts at trade N+K because the REST `fromId` pagination starts at a different point. The K trades in between are unaccounted.

**Proof:** `opendeviationbar_version` tags confirm the split: March 18 bars are `+kintsugi-parquet`, March 19 bars are `+backfill`. BTCUSDT and ETHUSDT (streamed by sidecar with Puck gap fill) have **zero** midnight gaps.

**Fix (applied):** Removed `OPENDEVIATIONBAR_STREAMING_SYMBOLS` override from `deploy/sidecar.env`. `SidecarConfig` now defaults to all `enabled=true` symbols from `symbols.toml` (the SSoT). Kintsugi reads from the same registry. No separate symbol list to maintain. Combined WS streams (#286) will reduce from 18 per-symbol connections to 2×9 hybrid — tracked in `full-symbol-streaming-spike` skill.

---

## Kintsugi

Unified gap reconciler + trailing-edge fill (Issues #115, #238). Discovers interior gaps ("shards") in the ClickHouse cache and repairs them, plus fills the trailing edge (latest bar → now). All gap filling consolidated in kintsugi daemon since #238 Phase 2.

| Script/File        | Purpose                                |
| ------------------ | -------------------------------------- |
| `kintsugi.py`      | CLI entry point + daemon mode          |
| `kintsugi.toml`    | mise task definitions (`.mise/tasks/`) |
| `kintsugi.service` | systemd unit (`scripts/systemd/`)      |

### Core engine: `python/opendeviationbar/kintsugi.py`

- `discover_shards()` — Stathera trade-ID audit via direct ClickHouse query (Issue #158), day-boundary exclusion (#264)
- `repair_shard()` — All repairs via `backfill_gap()` (Vision preferred for efficiency, REST produces identical results)
- `verify_repair()` — Stathera verification via ClickHouse (checks trade-ID gap is healed)
- `kintsugi_pass()` — single discover-repair-verify cycle
- `kintsugi_daemon()` — long-running adaptive polling (60s active, 300s clean; always reverse-chronological)

### Safeguards

- Max 3 repair attempts per shard per 24h (rate-limited via `kintsugi_log`)
- Repairs capped by `--max-repairs` (default 3)
- Sidecar dead-letter replay on daemon startup
- Telegram alerts on repair failures

---

## Parquet Tick Cache Seeder (Issue #270)

Proactive BTCUSDT Parquet cache seeder. Seeds `~/.cache/opendeviationbar/ticks/BINANCE_SPOT_BTCUSDT/` from Binance Vision so kintsugi's 64-worker Parquet path is always active.

### Service Design

- **Type**: systemd oneshot + 5-min timer (`opendeviationbar-seeder.timer`)
- **Entry point**: `scripts/tick_cache_seeder.py`
- **Symbols**: All enabled symbols (`get_registered_symbols(enabled_only=True)`), processed sequentially (BTCUSDT → ETHUSDT → …)
- **Start date**: `OPENDEVIATIONBAR_SEEDER_HISTORICAL_START=2018-01-01`
- **Concurrency**: `OPENDEVIATIONBAR_SEEDER_MAX_CONCURRENT_DOWNLOADS=12` (1Gbps Ethernet handles 12 concurrent Vision downloads; verified 48 files/min)
- **Fresh process per run**: RSS resets to 0 each 5-min timer trigger — no long-running accumulation

### Memory Optimization (MEM-021)

Deployed commit c416024. Peak cgroup RSS reduced from 4.47 GB (glibc ptmalloc2) to **2.5 GB** (−44%).

| Optimization | Mechanism                                                   | File         |
| ------------ | ----------------------------------------------------------- | ------------ |
| MEM-021      | `LD_PRELOAD=libmimalloc.so.2` + `MIMALLOC_PURGE_DELAY=1000` | service file |

**Note**: MEM-024 (`malloc_trim(0)` in monthly `finally:` block) was removed during the daily partitioning migration (#277). With daily files and mimalloc, the glibc arena plateau no longer occurs.

### Vision CSV Format (Critical Gotcha)

Binance Vision aggTrades CSVs have always had **8 columns, no header row**, since the first day of trading (2017-08-17). The only breaking change was timestamp precision:

| Period                   | Columns | Header | `transact_time` precision | Digits | Example            |
| ------------------------ | ------- | ------ | ------------------------- | ------ | ------------------ |
| 2017-08-17 → 2024-12-31  | 8       | None   | milliseconds              | 13     | `1735603200213`    |
| **2025-01-01 → present** | 8       | None   | **microseconds**          | 16     | `1735689600010866` |

**Column order (all periods, positional — no header):**

| Position | Field            | Type    | Notes                                                     |
| -------- | ---------------- | ------- | --------------------------------------------------------- |
| 0        | `agg_trade_id`   | int64   |                                                           |
| 1        | `price`          | float64 |                                                           |
| 2        | `quantity`       | float64 | Binance calls it `qty`                                    |
| 3        | `first_trade_id` | int64   |                                                           |
| 4        | `last_trade_id`  | int64   |                                                           |
| 5        | `transact_time`  | int64   | ms before 2025-01-01; **µs from 2025-01-01**              |
| 6        | `is_buyer_maker` | bool    |                                                           |
| 7        | `was_best_price` | bool    | Always present; seeder drops via `new_columns` truncation |

`_VISION_COLUMNS` in `tick_cache_seeder.py` names only 7 columns — column 7 (`was_best_price`) is silently discarded by Polars. This is intentional (field unused).

**Fix deployed** (`_download_vision_day()` in `scripts/tick_cache_seeder.py`): after CSV read, any `transact_time > 1_500_000_000_000_000` is divided by 1000 to normalize µs→ms. Guard is safe — a legitimate ms timestamp cannot reach that threshold until year 47658.

**Exact boundary confirmed**: last ms day = `2024-12-31`, first µs day = `2025-01-01` (UTC). Both BTCUSDT and ETHUSDT changed simultaneously.

### Status (bigblack, 2026-03-13)

| Symbol  | Period                | Months | Status                                             |
| ------- | --------------------- | ------ | -------------------------------------------------- |
| BTCUSDT | 2018-01-17→2026-03-12 | 99     | All correct (ms format, seeded before µs change) ✓ |
| ETHUSDT | 2018–2024-12          | 86     | All correct (ms format) ✓                          |
| ETHUSDT | 2025-01–2026-03       | 15     | Had µs timestamps — deleted, re-seeding with fix   |

BTCUSDT historical backfill: 531 days, 42 GB, 99 Parquet files, seeded in 609 seconds. Seeder now maintains rolling window; kintsugi runs exclusively on Parquet-cached path.

### REST API + WebSocket Field Names

All three Binance data sources use the same underlying fields but with different encodings. **REST and WS field names have never changed** since Binance launched in 2017.

#### REST `/api/v3/aggTrades` — JSON single-letter keys

| Key | Full name        | Type   | Notes                                       |
| --- | ---------------- | ------ | ------------------------------------------- |
| `a` | `agg_trade_id`   | int64  | Monotonic per symbol                        |
| `p` | `price`          | string | Decimal string e.g. `"93576.00000000"`      |
| `q` | `quantity`       | string | Decimal string                              |
| `f` | `first_trade_id` | int64  | First individual trade in aggregate         |
| `l` | `last_trade_id`  | int64  | Last individual trade in aggregate          |
| `T` | `trade_time`     | int64  | **Always milliseconds** — never changed     |
| `m` | `is_buyer_maker` | bool   | True = sell taker, False = buy taker        |
| `M` | `is_best_match`  | bool   | Always `true` for spot; ignored in our code |

#### WebSocket `{symbol}@aggTrade` — JSON single-letter keys

> **Note (Issue #279)**: Per-symbol connections (`wss://stream.binance.com:9443/ws/{symbol}@aggTrade`) are legacy. Production should use combined streams (`wss://stream.binance.com:9443/stream?streams=s1@aggTrade/s2@aggTrade/...`) with 2x9 hybrid architecture. Combined streams wrap payload in `{"stream":"...","data":{...}}`. See root CLAUDE.md and `binance-api-schema.md` for details.

Same payload as REST plus 3 stream-envelope fields (all ignored as redundant):

| Key | Full name        | Type   | Notes                          |
| --- | ---------------- | ------ | ------------------------------ |
| `e` | `event_type`     | string | Always `"aggTrade"` (ignored)  |
| `E` | `event_time`     | int64  | Server event time ms (ignored) |
| `s` | `symbol`         | string | e.g. `"BTCUSDT"` (ignored)     |
| `a` | `agg_trade_id`   | int64  |                                |
| `p` | `price`          | string | Decimal string                 |
| `q` | `quantity`       | string | Decimal string                 |
| `f` | `first_trade_id` | int64  |                                |
| `l` | `last_trade_id`  | int64  |                                |
| `T` | `trade_time`     | int64  | **Always milliseconds**        |
| `m` | `is_buyer_maker` | bool   |                                |
| `M` | `is_best_match`  | bool   | Ignored                        |

**Critical**: REST and WS timestamps (`T`) are **always milliseconds** across all years. This is unlike Vision CSV which switched to microseconds on 2025-01-01. No normalization needed for REST/WS paths.

**Memory**: Full reference at `project-memory/binance-api-schema.md` and `vision-csv-schema.md`.

---

## Stathera (σταθερά)

Trade-ID alignment paradigm (Issue #158). Replaces all time-based gap detection with aggregate trade-ID continuity checking. The invariant: `bar[i].last_agg_trade_id + 1 == bar[i+1].first_agg_trade_id` for all consecutive bars within the same (symbol, threshold) pair.

### Core Concepts

| Concept                | Definition                                                                                              |
| ---------------------- | ------------------------------------------------------------------------------------------------------- |
| **Stathera Invariant** | Trade-ID continuity: `bar[i].last_agg_trade_id + 1 == bar[i+1].first_agg_trade_id` for consecutive bars |
| **tid_delta**          | `first_agg_trade_id - lagInFrame(last_agg_trade_id, 1) - 1` per bar. >0 = gap, =0 = clean, <0 = overlap |
| **Stathera Gap**       | A violation where `tid_delta > 0` (missing aggTrade records between consecutive bars)                   |
| **Stathera Overlap**   | A violation where `tid_delta < 0` (bars from different sources share trade-ID ranges)                   |

### Scripts

| Script                  | Purpose                                                                              |
| ----------------------- | ------------------------------------------------------------------------------------ |
| `detect_gaps.py`        | Trade-ID audit for all (symbol, threshold) pairs. Exit code 1 if gaps/overlaps found |
| `detect_gaps.py --json` | Machine-readable Stathera audit output                                               |

### How Stathera Audit Works

1. **Audit Query** (`detect_gaps.py`):
   - Window function ordered by `(first_agg_trade_id, close_time_ms)` (critical: fixes overlap detection)
   - Computes `tid_delta = first_agg_trade_id - lagInFrame(last_agg_trade_id, 1) - 1`
   - Classifies: gap (>0), clean (=0), overlap (<0), legacy (first_agg_trade_id=0)

2. **Overlap Repair** (kintsugi day-recompute):
   - Both gaps (`tid_delta > 0`) and overlaps (`tid_delta < 0`) trigger day-recompute
   - DELETE entire affected Ouroboros day(s), reprocess from scratch via Ariadne
   - Absorbed from legacy `stathera_cleanup.py` in #238 Phase 1

3. **Source Tagging** (Phase 2):
   - Each bar's `opendeviationbar_version` encoded as `{version}+{source}`:
     - `+sidecar` — single writer (cannot overlap itself)
     - `+backfill` — REST backfill (skip if sidecar already wrote)
     - `+kintsugi` — repair processing (authoritative, can replace)
     - `+populate` — cache population (warn on overlaps, investigate)

### Integration with Kintsugi

- `_discover_shards_stathera()` — direct ClickHouse Stathera audit, no subprocess
- Per-gap trade-ID anchors (`anchor_last_tid`, `resume_first_tid`) replace global high-water-mark
- All repairs via `backfill_gap()` (Vision preferred for efficiency, REST produces identical results)

### Verification

```bash
# Local
python scripts/detect_gaps.py --json               # Audit all pairs

# Bigblack
python scripts/detect_gaps.py --symbol BTCUSDT --threshold 250
```

---

## Streaming Reliability Concepts

The sidecar uses a layered self-healing architecture with named concepts from Greek mythology:

| Concept      | Layer      | Purpose                                                                                      |
| ------------ | ---------- | -------------------------------------------------------------------------------------------- |
| **Antaeus**  | Rust (WS)  | Resilient reconnection: `backon` exponential backoff with jitter, resets after stable (≥60s) |
| **Lethe**    | Python     | Ring buffer overflow alerting: detects `dropped_bars` metric increases, Telegram alerts      |
| **Argus**    | Python     | Per-symbol liveness: tracks last bar time per symbol, alerts if silent ≥30 min               |
| **Puck**     | Rust (WS)  | Inline gap fill on trade-ID discontinuity. Parallel REST fetch, sub-3s. `puck_fill()`        |
| **Niffler**  | Python     | Immediate dead-letter replay on CH write failure. Sub-5s recovery.                           |
| **Charon**   | Python     | Dead-letter ferry: auto-replays failed Parquet writes on startup + every 5 min               |
| **Ariadne**  | Python     | `fromId` REST pagination for gap-fill: trade-ID anchored, zero-gap by construction           |
| **Kintsugi** | Python     | Self-healing gap reconciliation via Stathera trade-ID audit + Ariadne repair                 |
| **Stathera** | ClickHouse | Trade-ID alignment invariant: `bar[i].last_agg_trade_id + 1 == bar[i+1].first_agg_trade_id`  |

### Additional Streaming Components

| Component               | Purpose                                                                                                                                                                             |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **StreamManager**       | Managed streaming engine wrapping `LiveBarEngine` with adaptive rate limiting (Issue #162), trade-ID continuity tracking seeded from ClickHouse, and shadow validation diagnostics. |
| **AdaptiveRateLimiter** | Token bucket rate limiter for Binance REST API (Issue #162). Syncs with `X-MBX-USED-WEIGHT-1m` server header, 2400 weight/min budget, 20% safety margin.                            |
| **Shadow Validation**   | Diagnostic background loop in StreamManager. Every 5s, fetches latest trade via REST and compares with WS-tracked trade IDs. Divergences logged, not auto-corrected.                |

### Configuration (Environment Variables)

| Variable                                 | Default | Purpose                         |
| ---------------------------------------- | ------- | ------------------------------- |
| `OPENDEVIATIONBAR_LETHE_ALERT_THRESHOLD` | `1`     | Alert on first ring buffer drop |
| `OPENDEVIATIONBAR_ARGUS_TIMEOUT_S`       | `1800`  | Per-symbol silence timeout (s)  |
| `OPENDEVIATIONBAR_CHARON_INTERVAL_S`     | `300`   | Dead-letter replay interval (s) |

---

## Validation Scripts

Portable scripts for GPU workstations without full dev environment:

| Script                                 | Purpose                             |
| -------------------------------------- | ----------------------------------- |
| `validate_n_open_deviation_bars.py`    | Count-bounded API validation        |
| `validate_microstructure_features.py`  | v7.0 feature validation             |
| `validate_microstructure_roundtrip.py` | Compute → store → read → verify     |
| `validate_backfill_preflight.py`       | Pre-population checks               |
| `validate_clickhouse.py`               | ClickHouse connectivity             |
| `validate_memory_efficiency.py`        | Memory usage during processing      |
| `deduplicate_parquet_cache.py`         | Fix pre-v12.8 duplicate ticks (#78) |
| `detect_volume_overflow.py`            | Find negative volumes (#88)         |
| `validate_dedup_hardening.py`          | Issue #90 dedup layer validation    |

---

## Repair Scripts (Issue #275)

One-off data quality repair scripts. Lessons from the #275 microstructure repair session.

| Script                     | Purpose                                                        |
| -------------------------- | -------------------------------------------------------------- |
| `repair_direct_parquet.py` | Parquet→Rust→CH bulk repair, 24 workers, thread-local CH cache |

**`repair_direct_parquet.py` is the ONLY repair script.** It uses thread-local CH cache (`_get_worker_cache()`), auto-detects worker count, and processes all 4 thresholds from one Parquet read per day. Uses `to_dicts()` path with bounded futures + allocator purge to manage RSS.

**Do NOT copy scripts to `/tmp` and run from there** — always run from the repo (`scripts/repair_direct_parquet.py`). The `/tmp` copy pattern caused the "old code still running" bug during #275. After a `git pull`, the repo script IS the latest version. No manual deployment needed for scripts (unlike the wheel for library code).

### Post-Repair Cleanup (bigblack)

After completing a repair session, remove stale temp artifacts from the `/tmp` copy anti-pattern:

```bash
ssh bigblack 'rm -f /tmp/repair-*.sh /tmp/repair_direct_parquet_fixed.py /tmp/repair-*.log /tmp/seeder-*.log /tmp/seed-and-repair.*
rm -rf /tmp/odb-repair-*
echo "temp files cleaned"'
```

### Anti-Patterns & Best Practices (#275)

**Anti-pattern: New `OpenDeviationBarCache()` per iteration.** Each instantiation creates `urllib3.PoolManager(maxsize=20)` = up to 20 fds. In a loop over N workers x 4 thresholds, this causes `[Errno 24] Too many open files` within minutes. **Fix**: thread-local cache reuse via `_get_worker_cache()` using `threading.local()` — one CH connection per worker thread. fd budget stays at ~4N+50 for N workers.

**Anti-pattern: Deleting entire days to fix microstructure-only bugs.** During #275, 1.3M bars were initially deleted when only 12,959 orphan bars had the actual defect. Full-day DELETE+reprocess is expensive and creates unnecessary Stathera gaps. **Fix**: for microstructure-only fixes, use targeted Ariadne repair — fetch trades by trade-ID range from the affected bars and reprocess just those bars. Reserve full-day recompute for actual data corruption.

**fd budget rule**: `ulimit -n` on bigblack is **1024** for ad-hoc shell sessions (systemd services have `LimitNOFILE=65536`). Thread-local cache reuse keeps fd count at ~4N+50 for N workers, safely under the 1024 limit.

---

## Performance: Verified Bottlenecks (2026-03-15)

Two root causes identified: fd exhaustion (#1) and DELETE scan timeout (#2).

| Hypothesis                | Verified?                                                               | Result                             |
| ------------------------- | ----------------------------------------------------------------------- | ---------------------------------- |
| CH insert throughput      | NO — inserts 13-23ms, 0 delayed                                         | Not a bottleneck                   |
| CH parts pressure         | NO — 1,658 parts, merges kept up                                        | Not a bottleneck for INSERT        |
| CH DELETE scan on replace | **YES** — `overlap_strategy="replace"` timed out at 300s scanning parts | **Bottleneck #2** (0 days/min)     |
| fd exhaustion             | **YES** — old code 1,777 socket fds vs 32 with fix                      | **Bottleneck #1** (9->24 days/min) |
| CPU saturation            | NO — load 2-15 on 32 cores                                              | Not a bottleneck                   |
| RAM (dict path)           | PARTIALLY — starts 3 GB, plateaus ~13 GB (arena fragmentation)          | Manageable with allocator purge    |

**Root cause #1**: `OpenDeviationBarCache()` per threshold per day per worker. Each creates `urllib3.PoolManager(maxsize=20)` + preflight + schema migration. Fix: thread-local cache reuse (`_get_worker_cache()`) = ONE cache per worker thread.

**Root cause #2**: `overlap_strategy="replace"` runs DELETE-before-INSERT. With 1,658+ active parts, the DELETE scan takes >300s and times out. Fix: use `overlap_strategy="warn"` (plain INSERT) for days with no existing bars.

**Final empirical rates**:

| Config                        | Rate                             | Why                            |
| ----------------------------- | -------------------------------- | ------------------------------ |
| 24 workers, old code, replace | 9 days/min -> 0 (timeout)        | Connection churn + DELETE scan |
| 24 workers, fixed, replace    | 0 days/min (300s DELETE timeout) | DELETE scans 1,658 parts       |
| 24 workers, fixed, warn       | 24 days/min                      | Plain INSERT, no DELETE        |
| 32 workers, fixed, warn       | 21-39 days/min                   | Optimal for dense data         |

**RSS with bounded futures + allocator purge**: Starts ~3 GB, peaks ~13 GB with periodic `_force_allocator_purge()` every 50 completions returning freed glibc/mimalloc pages to the OS. Without purge, RSS would plateau higher from arena fragmentation; with purge, it stays manageable.

**Lesson**: Always verify bottlenecks with data, not assumptions. Profile at steady state (>30 min), not cold start. The six checks:

1. `ls /proc/PID/fd | wc -l` — fd count (connection churn)
2. `awk '/VmRSS/{print $2}' /proc/PID/status` — RSS at 3 time points 30s apart (leak vs plateau)
3. `system.query_log` — CH insert latency p50/p95/p99 + DelayedInserts
4. `uptime` — load average vs `nproc`
5. `free -m` + `swapon --show` — available RAM and swap usage (cold-start RSS ≠ steady-state RSS)

---

## Related

- [/CLAUDE.md](/CLAUDE.md) - Project hub
- [/crates/CLAUDE.md](/crates/CLAUDE.md) - Rust crate details, microstructure features
- [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) - Python API, caching, validation
- [/python/opendeviationbar/clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md) - ClickHouse cache layer
- mise task definitions (cache status, kintsugi)
