---
name: odb-ship
description: >-
  Unified release-publish-deploy pipeline for opendeviationbar-py.
  Covers semantic-release versioning, Rust+Python wheel builds, PyPI and crates.io publishing,
  bigblack production deployment (atomic venv swap, systemd sync, service orchestration),
  monit alignment, and 73-point post-deploy verification.
  Use when shipping a new version, deploying hotfixes, verifying production health,
  or recovering from partial release/deploy failures.
  Replaces mise deploy tasks and odb-deploy-verification skill.
allowed-tools: Read, Bash, Grep, Glob, Agent, WebFetch, TeamCreate, SendMessage, TaskCreate, TaskUpdate, TaskList, TaskGet
---

# odb-ship: Release, Publish & Deploy

ultrathink

Unified pipeline from "code ready on main" to "production verified healthy". Claude Code is the orchestrator — dynamically adapting to failures instead of rigid bash scripts that break on SSH quoting edge cases.

## When to Use

- `/odb-ship` — Full pipeline: version bump through production verification
- `/odb-ship --agents` — Full pipeline with agent team (parallel test+build, parallel publish)
- `/odb-ship deploy` — Deploy only (version already published)
- `/odb-ship verify` — Verify production health without deploying
- `/odb-ship hotfix` — Pure Python-only deploy (no Rust rebuild needed)
- `/odb-ship rollback` — Recover from a bad deploy

---

## Architecture: Why a Skill, Not a Script

The `mise run deploy:bigblack` bash script fails on:

- SSH command quoting with complex ClickHouse queries
- Network transients (SSH timeouts, PyPI CDN delays)
- Partial failures requiring human judgment
- Service orchestration timing (monit respawn windows)

Claude Code as orchestrator **adapts**: retries with different quoting, falls back to HTTPS when SSH fails, waits for monit respawn, and explains what it's doing.

---

## Agent Team Mode (`--agents`)

When invoked with `--agents`, odb-ship creates an agent team to parallelize the two slowest segments of the pipeline. Without `--agents`, the default single-agent sequential mode is used (simpler, lower token cost).

### Why Teams Help

The pipeline has two naturally parallel segments that account for ~7 of ~11 minutes:

| Segment                                | Sequential | With Team | Savings  |
| -------------------------------------- | ---------- | --------- | -------- |
| Phase 1: Tests (~2.5m) + Build (~2.8m) | ~5.3 min   | ~2.8 min  | ~2.5 min |
| Phase 2: PyPI (~2m) + crates.io (~2m)  | ~4 min     | ~2 min    | ~2 min   |

Phases 3-5 (deploy, service orchestration, verification) are sequential SSH commands to one host — no parallelism benefit.

### Team Structure

```
Team: odb-ship-v{VERSION}
Lead (you): Phase 0 → Phase 1.1-1.2 → coordinate → Phase 3-5

Teammates:
  tester     — Phase 1.3: mise run release:test
  builder    — Phase 1.4: mise run release:build-all
  pypi-pub   — Phase 2.1: mise run release:pypi
  crates-pub — Phase 2.2: mise run release:crates
```

### Team Flow

**Step 1: Pre-flight + Version** (Lead, sequential)

```
Phase 0: Pre-flight checks
Phase 1.1: git pull --rebase
Phase 1.2: semantic-release (must complete before tests/builds)
```

**Step 2: Test + Build** (Parallel teammates)

```python
# After semantic-release succeeds, create team and spawn:
TeamCreate(team_name="odb-ship-v{VERSION}")

TaskCreate(title="Run tests", description="mise run release:test — gate on Rust + Python passing")
TaskCreate(title="Build wheels", description="mise run release:build-all — macOS ARM64 + Linux x86_64 + sdist")

# Spawn teammates
Agent(name="tester", team_name="odb-ship-v{VERSION}",
      prompt="Run: mise run release:test. Report PASS/FAIL with test counts.",
      mode="bypassPermissions")

Agent(name="builder", team_name="odb-ship-v{VERSION}",
      prompt="Run: mise run release:build-all. Report wheel filenames and sizes from dist/.",
      mode="bypassPermissions")

# Wait for both to complete — messages arrive automatically
```

**Step 3: Publish** (Parallel teammates)

```python
# After both test + build pass:
TaskCreate(title="Publish PyPI", description="mise run release:pypi")
TaskCreate(title="Publish crates.io", description="mise run release:crates")

Agent(name="pypi-pub", team_name="odb-ship-v{VERSION}",
      prompt="Run: mise run release:pypi. Then verify: curl -sf 'https://pypi.org/pypi/opendeviationbar/{VERSION}/json' | jq '.info.version'. Report the confirmed version.",
      mode="bypassPermissions")

Agent(name="crates-pub", team_name="odb-ship-v{VERSION}",
      prompt="Run: mise run release:crates. Then verify: cargo search opendeviationbar-core --limit 1. Report the confirmed version.",
      mode="bypassPermissions")

# Wait for both to complete
```

**Step 4: Deploy + Verify** (Lead, sequential)

```
Phase 2.3: Verify publications (quick confirmation)
Phase 3: Deploy to bigblack
Phase 4: Service orchestration
Phase 5: Verification
```

**Step 5: Cleanup**

```python
# Shut down teammates
SendMessage(to="tester", message={"type": "shutdown_request"})
SendMessage(to="builder", message={"type": "shutdown_request"})
SendMessage(to="pypi-pub", message={"type": "shutdown_request"})
SendMessage(to="crates-pub", message={"type": "shutdown_request"})
TeamDelete(team_name="odb-ship-v{VERSION}")
```

### Failure Handling in Team Mode

| Failure                         | Action                                                                                |
| ------------------------------- | ------------------------------------------------------------------------------------- |
| **tester fails**                | Stop builder (no point publishing broken code). Report failure.                       |
| **builder fails**               | Wait for tester. If tests pass, retry build. If tests also fail, report both.         |
| **pypi-pub fails**              | Wait for crates-pub. Retry PyPI. If still fails, fall back to SCP wheel.              |
| **crates-pub fails**            | Non-blocking for deploy. Log warning, continue to Phase 3 (PyPI wheel is sufficient). |
| **Teammate stops unexpectedly** | Lead receives idle notification. Check task status. Spawn replacement if needed.      |

### When NOT to Use `--agents`

- **`/odb-ship deploy`** — No test/build/publish, just SSH commands. Single agent is faster.
- **`/odb-ship verify`** — Just SSH checks. No parallelism benefit.
- **`/odb-ship hotfix`** — No wheel build, just rsync. Single agent is faster.
- **`/odb-ship rollback`** — Emergency path. Keep simple.

### Prerequisites

Agent teams require:

- `CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1` in settings.json or environment
- Claude Code v2.1.32+

---

## Phase Overview

| Phase | Name                                                    | Purpose                                | Skippable         |
| ----- | ------------------------------------------------------- | -------------------------------------- | ----------------- |
| 0     | [Pre-Flight](#phase-0-pre-flight)                       | Connectivity, credentials, clean state | No                |
| 1     | [Version & Build](#phase-1-version--build)              | Semantic-release + wheel builds        | Yes (deploy-only) |
| 2     | [Publish](#phase-2-publish)                             | PyPI + crates.io + GitHub release      | Yes (deploy-only) |
| 3     | [Deploy](#phase-3-deploy-to-bigblack)                   | Atomic venv swap on bigblack           | No                |
| 4     | [Service Orchestration](#phase-4-service-orchestration) | Stop/start services, monit alignment   | No                |
| 5     | [Verification](#phase-5-verification)                   | 73-point production health check       | No                |

---

## Phase 0: Pre-Flight

### 0.1 Network Connectivity (Run First)

```bash
# GitHub SSH (may fail — HTTPS fallback available)
ssh -o ConnectTimeout=5 -T git@github.com 2>&1 | head -3

# Bigblack SSH (REQUIRED — no fallback)
ssh -o ConnectTimeout=5 bigblack 'echo OK'

# GitHub API via HTTPS (works even when SSH is blocked)
gh api repos/terrylica/opendeviationbar-py --jq '.pushed_at'
```

**If GitHub SSH fails**: Swap remote to HTTPS before any git/release operations:

```bash
git remote set-url origin "https://github.com/terrylica/opendeviationbar-py.git"
# Restore after: git remote set-url origin "git@github.com-terrylica:terrylica/opendeviationbar-py.git"
```

### 0.2 Clean Working Directory

```bash
git status --porcelain
git log --oneline @{u}..HEAD  # unpushed commits
```

- Unpushed commits: `git push origin main` first
- Dirty files: commit or stash before proceeding
- Lockfile drift (uv.lock): `git checkout -- uv.lock`

### 0.3 Checksum Bindings (v6.0+)

Verify Rust checksum PyO3 bindings compiled into the wheel. If these are missing, the seeder will crash on first run after deploy.

```bash
python3 -c "from opendeviationbar._core import compute_sha256_hex, parse_checksum_file_content, verify_checksum_hex; print('✓ Checksum bindings present')"
```

**If this fails**: `maturin develop` or `maturin build --release` did not include the `data-providers` feature gate, or `src/binance_bindings.rs` changes were not compiled. Rebuild before proceeding.

### 0.4 Version State

```bash
VERSION=$(grep -A5 '\[workspace.package\]' Cargo.toml | grep '^version' | head -1 | sed 's/.*= "\(.*\)"/\1/')
echo "Current Cargo.toml version: $VERSION"
```

---

## Phase 1: Version & Build

### 1.1 Sync with Remote

```bash
git pull --rebase origin main
```

If SSH fails, use HTTPS push:

```bash
GH_TOKEN=$(gh auth token) git \
  -c "url.https://x-access-token:$(gh auth token)@github.com/.insteadOf=git@github.com-terrylica:" \
  push origin main
```

### 1.2 Semantic Release

```bash
mise run release:version
```

This runs `semantic-release --no-ci` which:

- Analyzes commits since last tag (feat: → minor, fix: → patch)
- Bumps `Cargo.toml` + creates git tag + GitHub release
- **CRITICAL**: Must use `--no-ci` flag (otherwise dry-run only)
- **CRITICAL**: Needs SSH or HTTPS connectivity to `git ls-remote`

**If no releasable commits**: semantic-release exits cleanly, creates `.release-skip` file. All downstream phases skip gracefully.

### 1.3 Run Tests

```bash
mise run release:test
```

Gates on both Rust (`cargo nextest`) and Python (`pytest`) passing.

### 1.4 Build Wheels

```bash
mise run release:build-all
```

Builds:

- macOS ARM64 wheel (native maturin)
- Linux x86_64 wheel (zig cross-compile, ~55s)
- Source distribution (sdist)

Output: `dist/opendeviationbar-{VERSION}-*.whl`

> **Team mode (`--agents`)**: After semantic-release (1.2), spawn `tester` and `builder` teammates in parallel instead of running 1.3 and 1.4 sequentially. Wait for both to report back. If `tester` fails, shut down `builder` immediately — no point publishing broken code. See [Agent Team Mode](#agent-team-mode---agents) for full flow.

---

## Phase 2: Publish

### 2.1 PyPI

```bash
mise run release:pypi
```

Uses 1Password credentials from `~/.claude/.secrets/`. After upload, PyPI CDN may take 30-60s to propagate. The deploy phase polls for availability.

### 2.2 Crates.io

```bash
mise run release:crates
```

Uses `cargo publish --workspace` (Rust 1.90+ native topological ordering). Publishes 5 crates: core, providers, streaming, client, hurst.

**Token resolution** (automatic, 3-source fallback in `release-crates.toml`):

1. `~/.claude/.secrets/crates-io-token` (local file, fastest)
2. Doppler `claude-config/prd` → `CRATES_IO_CLAUDE_CODE` (auto-caches to #1 on success)
3. Manual: `echo TOKEN > ~/.claude/.secrets/crates-io-token && chmod 600 ~/.claude/.secrets/crates-io-token`

**Known issue**: If crates.io publish races with version bump, you get "already exists" errors. Fix: wait for `Cargo.toml` to have the new version, then re-run.

> **Team mode (`--agents`)**: Spawn `pypi-pub` and `crates-pub` teammates in parallel instead of running 2.1 and 2.2 sequentially. Each teammate verifies its own publication before reporting back. If `crates-pub` fails, continue anyway — PyPI wheel is sufficient for deploy. See [Agent Team Mode](#agent-team-mode---agents) for full flow.

### 2.3 Verify Publications

```bash
# PyPI
curl -sf "https://pypi.org/pypi/opendeviationbar/$VERSION/json" | jq '.info.version'

# Crates.io (check one representative crate)
cargo search opendeviationbar-core --limit 1
```

---

## Phase 3: Deploy to Bigblack

### Critical Context

| Fact                    | Detail                                                                       |
| ----------------------- | ---------------------------------------------------------------------------- |
| **Deploy target**       | `bigblack` (remote GPU host, SSH alias)                                      |
| **Remote dir**          | `/home/tca/opendeviationbar-py`                                              |
| **Python**              | mise-managed 3.13 at `~/.local/share/mise/installs/python/3.13*/bin/python3` |
| **Package manager**     | `uv` at `~/.local/bin/uv` (NOT in venv)                                      |
| **Services managed by** | systemd `--user` (monit watches and auto-restarts)                           |
| **Secrets**             | `deploy/opendeviationbar.local.env` (Telegram TOKEN, Redis password)         |

### 3.1 Git Sync on bigblack

```bash
ssh bigblack 'cd ~/opendeviationbar-py && git fetch origin main --tags && git reset --hard origin/main && git clean -fd'
```

### 3.2 Poll PyPI CDN Readiness

```bash
ssh bigblack "timeout 3 curl -sf 'https://pypi.org/pypi/opendeviationbar/$VERSION/json' | jq -e '.info.version'"
```

Retry up to 8 times with 20s intervals (~2.5 min max). If still not available, fall back to SCP of local wheel.

### 3.3 Build Staging Venv

```bash
REMOTE_PYTHON=$(ssh bigblack "ls -d ~/.local/share/mise/installs/python/3.13*/bin/python3 2>/dev/null | tail -1")
ssh bigblack "cd ~/opendeviationbar-py && rm -rf .venv-staging && uv venv --python $REMOTE_PYTHON .venv-staging"
```

### 3.4 Install Package

**Primary (PyPI)**:

```bash
ssh bigblack "uv pip install --python ~/opendeviationbar-py/.venv-staging/bin/python3 --refresh-package opendeviationbar opendeviationbar==$VERSION"
```

**Fallback (local wheel via SCP)**:

```bash
LINUX_WHEEL=$(ls dist/opendeviationbar-${VERSION}-*manylinux*.whl | head -1)
scp "$LINUX_WHEEL" bigblack:~/opendeviationbar-py/
ssh bigblack "uv pip install --python ~/opendeviationbar-py/.venv-staging/bin/python3 ~/opendeviationbar-py/$(basename $LINUX_WHEEL)"
```

### 3.5 Verify Staging Venv

```bash
ssh bigblack "~/opendeviationbar-py/.venv-staging/bin/python3 -c '
import opendeviationbar; print(opendeviationbar.__version__)
from opendeviationbar import OpenDeviationBarProcessor
from pathlib import Path
import opendeviationbar.clickhouse.cache as ch
assert (Path(ch.__file__).parent / \"schema.sql\").exists()
import opendeviationbar.symbol_registry as sr
assert (Path(sr.__file__).parent / \"data\" / \"symbols.toml\").exists()
from opendeviationbar.config.sidecar import SidecarConfig
print(f\"ws_mode default: {SidecarConfig(symbols=[\\\"BTCUSDT\\\"], thresholds=[250]).ws_mode}\")
print(\"OK: version + .so + schema.sql + symbols.toml + ws_mode\")
'"
```

**Also verify Parquet tick cache directory** (v3.0 Parquet-first catchup reads ticks on sidecar startup):

```bash
ssh bigblack 'ls -la ~/.cache/opendeviationbar/ticks/BTCUSDT/ 2>/dev/null | tail -3 || echo "WARNING: No Parquet tick cache — sidecar will fall back to full REST fill"'
```

**Also verify checksum layer** (v6.0: seeder crashes if these imports fail):

```bash
ssh bigblack "~/opendeviationbar-py/.venv-staging/bin/python3 -c '
from opendeviationbar._core import compute_sha256_hex, verify_checksum_hex, parse_checksum_file_content
from opendeviationbar.checksum_registry import ChecksumRegistry
from opendeviationbar.checksum_telemetry import log_checksum_event
print(\"OK: checksum bindings + registry + telemetry\")
'"
```

**If this fails**: Wheel lacks v6.0 checksum modules. Seeder will crash on next timer trigger — rollback and rebuild.

### 3.6 Nuke .pth Shadowing Artifacts

After install, check for editable-install `.pth` files that shadow the wheel:

```bash
ssh bigblack "
SITE_PKG=\$(~/opendeviationbar-py/.venv-staging/bin/python3 -c 'import site; print(site.getsitepackages()[0])')
PTH=\"\$SITE_PKG/opendeviationbar.pth\"
if [ -f \"\$PTH\" ]; then
  echo 'FOUND .pth shadowing — removing...'
  rm -f \"\$PTH\"
fi
"
```

---

## Phase 4: Service Orchestration

### 4.1 Stop Services (BEFORE Venv Swap)

**CRITICAL**: Must unmonitor from monit first, otherwise monit auto-restarts within 120s:

```bash
ssh bigblack 'sudo monit unmonitor sidecar 2>/dev/null; sudo monit unmonitor kintsugi 2>/dev/null; sleep 1'
ssh bigblack 'systemctl --user stop opendeviationbar-sidecar opendeviationbar-kintsugi opendeviationbar-kintsugi-catchup 2>/dev/null; true'
sleep 2
```

### 4.1b Kill ALL ClickHouse Mutations (BEFORE Venv Swap)

**CRITICAL**: Kill ALL mutations — both pending and completed. Kintsugi's `_do_replace()` creates `DELETE IN PARTITION` mutations for historical repairs; any still-executing mutations can interfere with the sidecar's fresh start. Killing preemptively ensures a clean mutation queue.

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "KILL MUTATION WHERE database = '"'"'opendeviationbar_cache'"'"'" && echo "All mutations killed"'
# Verify zero remaining
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "SELECT count() FROM system.mutations WHERE database = '"'"'opendeviationbar_cache'"'"'"'
```

**Why kill ALL**: Executing mutations from the previous kintsugi session can race with the new sidecar's sync_flush writes. The only safe path is killing every mutation before restart.

### 4.2 OPTIMIZE TABLE FINAL (Force RMT Dedup)

**CRITICAL**: Before swapping venvs, force ReplacingMergeTree to deduplicate all parts. This resolves any pre-existing overlaps from previous deploys, mutation storms, or concurrent writer races. Much faster than ALTER TABLE DELETE (no mutations created).

```bash
ssh bigblack 'curl -s "http://localhost:8123/?wait_end_of_query=1" --data-binary "OPTIMIZE TABLE opendeviationbar_cache.open_deviation_bars FINAL" && echo "OPTIMIZE FINAL done"'
```

**Why every deploy**: Overlaps can accumulate silently between deploys (sidecar restarts, kintsugi races). OPTIMIZE FINAL resolves **same-key duplicates** (RMT deduplicates by `ORDER BY first_agg_trade_id`, keeping the latest version). It does **NOT** resolve overlapping bars with different `first_agg_trade_id` (different TID boundaries from different processor sessions) — those require kintsugi repair or Pattern 2. Cost: 5-30s depending on parts count.

### 4.2b Run Pending Schema Migrations (AFTER OPTIMIZE, BEFORE Venv Swap)

Check for and run any pending ClickHouse migration scripts. Services are already stopped, so migrations are safe.

```bash
# Check for pending migrations
MIGRATION=$(ssh bigblack 'ls ~/opendeviationbar-py/scripts/migrate_*.py 2>/dev/null | head -1')
if [ -n "$MIGRATION" ]; then
  echo "Found migration: $(basename $MIGRATION)"
  ssh bigblack "~/opendeviationbar-py/.venv/bin/python3 $MIGRATION --apply"
  echo "Migration complete"
else
  echo "No pending migrations"
fi
```

**v4.0 migration**: `scripts/migrate_column_rename_us.py --apply` renames `close_time_ms` → `close_time_us` and `open_time_ms` → `open_time_us` via `ALTER TABLE RENAME COLUMN` (metadata-only, instant on 42M+ rows). Must run BEFORE venv swap because the new code expects `_us` column names.

**Why here (after OPTIMIZE, before venv swap)**: OPTIMIZE FINAL uses the old column names. The migration renames them. Then the new venv (with `_us` code) starts against the renamed columns. If migration fails, the old venv still works with old column names — safe rollback.

### 4.3 Clear Stale Streaming Checkpoints

Checkpoints from the previous sidecar session often point beyond ClickHouse data (especially after OPTIMIZE merges parts). Stale checkpoints are harmless (rejected + preserved) but cause noisy warnings on startup. Clear them for a clean start:

```bash
ssh bigblack 'rm -f ~/.cache/opendeviationbar/checkpoints/streaming_*.json && echo "Streaming checkpoints cleared"'
```

The sidecar's `_inline_startup_backfill()` will cover any gaps from the cleared checkpoints.

### 4.3b Refresh Parquet Tick Cache

Fresh Parquet ensures the sidecar's Parquet-first catchup (SEED-001) has data to work with, avoiding full REST fill which is slow and burns rate limit. Trigger the Binance seeder manually and wait for all enabled Binance spot symbols (from `symbols.toml` registry — the SSoT) to have fresh Parquet before the sidecar starts.

```bash
# Trigger seeder oneshot and wait for completion
ssh bigblack 'systemctl --user start opendeviationbar-seeder && echo "Seeder triggered"'
# Wait for seeder to complete (oneshot exits when done)
for i in $(seq 1 60); do
  if ssh bigblack 'systemctl --user is-active opendeviationbar-seeder 2>/dev/null' | grep -q 'inactive'; then
    echo "Seeder complete"
    break
  fi
  echo "Waiting for seeder... ($i/60)"
  sleep 5
done
```

Verify Parquet files exist for today (or yesterday if just past midnight):

```bash
EXPECTED=$(ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "from opendeviationbar.symbol_registry import get_registered_symbols; print(len([s for s in get_registered_symbols(enabled_only=True) if not s.startswith(\"EXNESS\")]))"')
ACTUAL=$(ssh bigblack "find ~/.cache/opendeviationbar/ticks/ -name '$(date -u +%Y-%m-%d).parquet' | wc -l")
echo "Parquet: $ACTUAL / $EXPECTED symbols have today's file"
```

All enabled Binance spot symbols (count from `symbols.toml` registry) should have a Parquet file for the current UTC date. If count is less than expected, the sidecar will fall back to REST for missing symbols (slow but functional). The seeder uses batch rotation (6 symbols/run), so 2-3 seeder runs may be needed to cover all symbols.

### 4.4 Atomic Venv Swap

```bash
ssh bigblack 'cd ~/opendeviationbar-py && rm -rf .venv-old && mv .venv .venv-old 2>/dev/null; mv .venv-staging .venv && rm -rf .venv-old'
```

Same-filesystem `mv` is atomic. Old venv untouched until swap succeeds.

### 4.5 Sync Systemd Units

```bash
ssh bigblack '
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-sidecar.service ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-kintsugi.service ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-kintsugi-catchup.service ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-heartbeat.service ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-heartbeat.timer ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-seeder.service ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-seeder.timer ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-exness-seeder.service ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-exness-seeder.timer ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now opendeviationbar-heartbeat.timer
systemctl --user enable --now opendeviationbar-seeder.timer
systemctl --user enable --now opendeviationbar-exness-seeder.timer
echo "OK: 6 services + 3 timers synced"
'
```

### 4.6 Start Services (Order Matters)

```bash
# Sidecar first (provides health endpoint + real-time data)
ssh bigblack 'systemctl --user start opendeviationbar-sidecar'
sleep 8

# Verify sidecar health
ssh bigblack 'curl -sf http://localhost:8081/health | python3 -m json.tool'

# CRITICAL (#295): Wait for sync_flush completion before starting kintsugi.
# Kintsugi's _do_replace mutations can delete sidecar's freshly-written bars
# if started before sync_flush completes.
#
# RECOMMENDED: Use the mise task (built in #297, wraps steps below):
#   mise run deploy:verify-restart
#
# MANUAL FALLBACK (if mise task not yet available):
SYNC_FLUSH_DONE=0
for i in $(seq 1 30); do
  if ssh bigblack 'journalctl --user -u opendeviationbar-sidecar --since "5 min ago" --no-pager 2>/dev/null | grep -q "sync_flush.*drained and wrote"'; then
    echo "sync_flush complete"
    SYNC_FLUSH_DONE=1
    break
  fi
  echo "Waiting for sync_flush... ($i/30)"
  sleep 10
done
if [ "$SYNC_FLUSH_DONE" -eq 0 ]; then
  echo "ERROR: sync_flush did not complete within 5 min — DO NOT start kintsugi yet"
  echo "Check sidecar logs: ssh bigblack 'journalctl --user -u opendeviationbar-sidecar -n 50'"
  echo "Options: (a) wait longer, (b) check sidecar health endpoint, (c) restart sidecar if stuck"
  exit 1
fi

# Verify zero Stathera anomalies on today BEFORE starting kintsugi.
# Must show overlaps=0 AND gaps=0 (intra-day only — day-boundary gaps are structural).
TODAY_ANOMALIES=$(ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT countIf(tid_delta < 0) AS overlaps, countIf(tid_delta > 0 AND day_gap = 0) AS intraday_gaps
FROM (
  SELECT
    first_agg_trade_id - lagInFrame(last_agg_trade_id, 1) OVER w - 1 AS tid_delta,
    toDate(close_time_us/1000000) - toDate(lagInFrame(close_time_us, 1) OVER w / 1000) AS day_gap,
    row_number() OVER w AS rn
  FROM opendeviationbar_cache.open_deviation_bars FINAL
  WHERE toDate(close_time_us/1000000) = today()
  WINDOW w AS (PARTITION BY symbol, threshold_decimal_bps, ouroboros_mode ORDER BY first_agg_trade_id)
) WHERE rn > 1
FORMAT TSV"')
echo "Today Stathera: $TODAY_ANOMALIES  (target: overlaps=0, intraday_gaps=0)"

# If non-zero, see "Failure Handling: Stathera Anomalies Before Kintsugi Start" below.
# DO NOT start kintsugi if overlaps or intraday_gaps > 0.

# Kintsugi (gap reconciliation — only after today is verified clean)
ssh bigblack 'systemctl --user start opendeviationbar-kintsugi'
sleep 3
```

### 4.6c Clear Kintsugi Rate Limit and Wait for Yesterday Repair

Deploy kills the sidecar, creating a gap at the day boundary. Kintsugi repairs yesterday's gaps but the `kintsugi_log` rate limit from the previous session blocks immediate re-repair. Clearing the log and waiting for one pass ensures yesterday is clean before verification.

1. Clear the `kintsugi_log` table (removes rate-limit entries that prevent immediate re-repair):

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "TRUNCATE TABLE opendeviationbar_cache.kintsugi_log" && echo "kintsugi_log cleared"'
```

1. Restart kintsugi so it picks up the cleared rate limits:

```bash
ssh bigblack 'systemctl --user restart opendeviationbar-kintsugi'
```

1. Wait for kintsugi to complete at least one yesterday-repair pass:

```bash
KINTSUGI_DONE=0
for i in $(seq 1 60); do
  if ssh bigblack 'journalctl --user -u opendeviationbar-kintsugi --since "5 min ago" --no-pager 2>/dev/null | grep -q "repair.*complete\|no.*gaps\|0 shards"'; then
    echo "Kintsugi yesterday-repair pass complete"
    KINTSUGI_DONE=1
    break
  fi
  echo "Waiting for kintsugi repair pass... ($i/60)"
  sleep 10
done
if [ "$KINTSUGI_DONE" -eq 0 ]; then
  echo "WARNING: Kintsugi repair pass did not complete within 10 min"
  echo "Check logs: ssh bigblack 'journalctl --user -u opendeviationbar-kintsugi -n 50'"
fi
```

See also: [Kintsugi Log Table Rate-Limit Reset](#kintsugi-log-table-rate-limit-reset) in Lessons Learned.

### 4.6b Cross-Midnight Gap Note

**Context**: Killing the old sidecar loses its forming orphan bars at the day boundary. The new sidecar starts from midnight crossover TID, leaving a gap between yesterday's last committed bar and today's first bar.

**Resolution**: Kintsugi repairs these gaps automatically when they become "yesterday or older" at the next UTC midnight. This is the **only safe path**.

**Anti-pattern (BANNED): Running `backfill_gap()` while sidecar is streaming.** Tested v13.53.0 deploy — `backfill_gap(force=True)` while sidecar was running caused 13 overlaps. Different processor instances produce bars with different TID boundaries for the same trade range. `overlap_strategy="replace"` does NOT prevent this because the overlapping bars have different `first_agg_trade_id` values (RMT can't dedup them). Required Pattern 2 (stop sidecar → delete today → restart) to clean up.

**If cross-midnight gaps are unacceptable (rare)**: Run `backfill_gap` BEFORE starting the sidecar, as part of Pattern 2 from the restart recovery playbook. Never while the sidecar is running.

### 4.6a Failure Handling: Stathera Anomalies Before Kintsugi Start

If the Stathera check above shows `overlaps > 0` or `intraday_gaps > 0` before kintsugi starts, **do not start kintsugi**. The overlap/gap must be resolved first.

**Decision tree:**

```
overlaps > 0?
├── Run OPTIMIZE TABLE FINAL (fast RMT dedup, no mutations):
│     ssh bigblack 'curl -s "http://localhost:8123/?wait_end_of_query=1" \
│       --data-binary "OPTIMIZE TABLE opendeviationbar_cache.open_deviation_bars FINAL"'
│   Re-run Stathera check. If still > 0 → stop sidecar, run Pattern 2 from
│   sidecar-restart-zero-overlap-zero-gap-recovery-playbook.
│
intraday_gaps > 0 (no overlaps)?
├── Small gap (<1000 trades): Puck should fill on next WS trade — wait 60s and re-check.
├── Large gap (>1000 trades): stop sidecar, Pattern 2 from the restart playbook.
│
Both overlaps AND gaps?
└── Pattern 2 is the only safe fix. Do not start kintsugi until both are 0.
```

**Pattern 2 reference**: Stop sidecar → DELETE today's bars (`mutations_sync=1`) → `backfill_gap(force=True)` → start sidecar → wait for sync_flush → re-verify → THEN start kintsugi.

After any recovery loop, run the sync_flush gate (step 4.6) again before proceeding to 4.7.

### 4.7 Re-Monitor in Monit

```bash
ssh bigblack 'sudo monit monitor sidecar 2>/dev/null; sudo monit monitor kintsugi 2>/dev/null'
```

### 4.8 Post-Start Stathera Audit

**MANDATORY**: After services start, verify zero overlaps on today's data. If overlaps exist, run OPTIMIZE FINAL again (the restart may have created transient overlaps before Rust dedup initialized).

```bash
# Check today's overlaps
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "SELECT countIf(tid_delta < 0) AS overlaps FROM (SELECT first_agg_trade_id - lagInFrame(last_agg_trade_id, 1) OVER w - 1 AS tid_delta, row_number() OVER w AS rn FROM opendeviationbar_cache.open_deviation_bars FINAL WHERE first_agg_trade_id > 0 AND last_agg_trade_id > 0 AND toDate(close_time_us/1000000) >= today() WINDOW w AS (PARTITION BY symbol, threshold_decimal_bps, ouroboros_mode ORDER BY first_agg_trade_id, close_time_us)) WHERE rn > 1 AND tid_delta < 0"'
```

**If overlaps > 0**: Stop sidecar, run OPTIMIZE FINAL again, clear checkpoints, restart sidecar, then repeat the sync_flush gate (4.6) and Stathera check before starting kintsugi. This loop converges in 1-2 iterations because OPTIMIZE FINAL is deterministic.

### 4.9 Verify Services Active

```bash
ssh bigblack '
for svc in opendeviationbar-sidecar opendeviationbar-kintsugi opendeviationbar-heartbeat.timer opendeviationbar-seeder.timer; do
  svc_state=$(systemctl --user is-active $svc 2>/dev/null || echo "inactive")
  echo "$svc: $svc_state"
done
'
```

---

## Phase 5: Verification

Full 73-point checklist in [references/verification-checklist.md](./references/verification-checklist.md). Below are the 5 critical quick-path checks.

### Quick Path (5 Commands)

**1. Version alignment** — all sources agree:

```bash
ssh bigblack '
VER=$(~/opendeviationbar-py/.venv/bin/python3 -c "import opendeviationbar; print(opendeviationbar.__version__)")
cd ~/opendeviationbar-py
TAG=$(git describe --tags --abbrev=0 2>/dev/null | sed "s/^v//")
CARGO=$(grep "^version" Cargo.toml | head -1 | sed "s/.*\"\(.*\)\"/\1/")
echo "Package=$VER Git=$TAG Cargo=$CARGO"
[ "$VER" = "$CARGO" ] && echo "ALIGNED" || echo "MISALIGNED"
'
```

**2. ClickHouse + Stathera gaps**:

```bash
ssh bigblack 'curl -sf "http://localhost:8123/ping" && echo " ClickHouse OK" || echo "ClickHouse DOWN"'
```

**2b. Yesterday-to-Today Stathera Boundary Check** -- verify zero gaps at the day boundary for all symbol+threshold combinations:

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT
  symbol,
  threshold_decimal_bps,
  yesterday_last_tid,
  today_first_tid,
  today_first_tid - yesterday_last_tid - 1 AS tid_delta
FROM (
  SELECT
    symbol,
    threshold_decimal_bps,
    argMaxIf(last_agg_trade_id, close_time_us, toDate(close_time_us/1000000) = yesterday()) AS yesterday_last_tid,
    argMinIf(first_agg_trade_id, close_time_us, toDate(close_time_us/1000000) = today()) AS today_first_tid
  FROM opendeviationbar_cache.open_deviation_bars FINAL
  WHERE toDate(close_time_us/1000000) >= yesterday()
  GROUP BY symbol, threshold_decimal_bps
  HAVING yesterday_last_tid > 0 AND today_first_tid > 0
)
WHERE tid_delta != 0
FORMAT PrettyCompact"'
# Empty result = PASS (0 boundary gaps)
# Any rows = FAIL (gaps or overlaps at day boundary)
```

Empty result means zero Stathera gaps at the yesterday/today boundary. Any rows indicate gaps (positive `tid_delta`) or overlaps (negative `tid_delta`) that need investigation. If gaps exist, verify kintsugi completed its yesterday-repair pass (step 4.6c). If overlaps exist, run OPTIMIZE TABLE FINAL.

**3. Sidecar health + WS mode** (v3.0: /health exposes ws_mode and expected_ws_connections):

```bash
ssh bigblack 'curl -sf http://localhost:8081/health | python3 -m json.tool'
# Verify: ws_mode should be "combined" (default) or "per_symbol" (rollback)
# expected_ws_connections: 2 for combined, N for per_symbol
```

**3b. Junction telemetry** (v3.0: /status exposes REST→WS junction state):

```bash
ssh bigblack 'curl -sf http://localhost:8081/status | python3 -m json.tool'
# Verify per-symbol: rest_first_tid, rest_last_tid, ws_first_tid present
# Stathera check: ws_first_tid == rest_last_tid + 1 for each symbol
```

**4. Monit summary**:

```bash
ssh bigblack 'sudo monit summary 2>/dev/null'
```

**5. Checksum layer health** (v6.0+):

```bash
ssh bigblack 'cat ~/.cache/opendeviationbar/checksum-telemetry.jsonl 2>/dev/null | tail -3 || echo "No telemetry yet (expected on first deploy)"'
# Expected after first seeder run: recent verify_pass/verify_skip events
```

**6. Native extension + constants**:

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar._core import PyOpenDeviationBarProcessor
from opendeviationbar.constants import DEFAULT_THRESHOLD
from opendeviationbar.symbol_registry import get_thresholds_for_symbol
assert DEFAULT_THRESHOLD == 250
assert get_thresholds_for_symbol(\"BTCUSDT\") == (100, 250, 500, 750)
print(\"All imports + constants OK\")
"'
```

### Post-Deploy: Batch Repair Note

If running `repair_direct_parquet.py` after deploy, use `--workers 24` (or omit for auto-detected default). Thread-local CH cache (Issue #279) makes this fd-safe — ~4N+50 fds for N workers, well under `ulimit -n 1024`. See [scripts/CLAUDE.md Bigblack Capacity Envelope](/scripts/CLAUDE.md#bigblack-capacity-envelope) for full hardware budget.

### Verdict Table

After running all sections, summarize:

| Section                  | Status | Notes                                                                  |
| ------------------------ | ------ | ---------------------------------------------------------------------- |
| Network & Connectivity   |        |                                                                        |
| Package & Version        |        |                                                                        |
| Native Extension         |        |                                                                        |
| Process Management       |        |                                                                        |
| ClickHouse Health        |        |                                                                        |
| Stathera Audit           |        |                                                                        |
| Yesterday-Today Boundary |        | Step 2b: cross-day Stathera boundary check (0 gaps required)           |
| Sidecar /health          |        | v3.0: ws_mode + expected_ws_connections                                |
| Sidecar /status          |        | v3.0: junction telemetry (rest_first_tid, rest_last_tid, ws_first_tid) |
| Checksum Layer           |        | v6.0: bindings + registry + telemetry imports OK, JSONL events present |
| Infrastructure           |        |                                                                        |
| Monitoring               |        |                                                                        |
| Configuration            |        |                                                                        |
| Release Pipeline         |        |                                                                        |

Mark each PASS / WARN / FAIL / SKIP.

---

## Hotfix Path (Python-Only Changes)

When the change is pure Python (no Rust modifications), skip the full build pipeline:

1. Push changes to `origin/main`
2. Git sync on bigblack: `ssh bigblack 'cd ~/opendeviationbar-py && git fetch origin main --tags && git reset --hard origin/main'`
3. Rsync Python files to site-packages (excludes Rust .so):

   ```bash
   ssh bigblack '
   SITE_PKG=$(~/opendeviationbar-py/.venv/bin/python3 -c "import site; print(site.getsitepackages()[0])")
   rsync -av --exclude="__pycache__" --exclude="*.pyc" --exclude="_core*.so" \
     ~/opendeviationbar-py/python/opendeviationbar/ \
     $SITE_PKG/opendeviationbar/
   find $SITE_PKG -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null; true
   '
   ```

4. Restart **ALL** services (sidecar first, then kintsugi — not just sidecar):

   ```bash
   ssh bigblack 'sudo monit unmonitor sidecar 2>/dev/null; sudo monit unmonitor kintsugi 2>/dev/null'
   ssh bigblack 'systemctl --user stop opendeviationbar-sidecar opendeviationbar-kintsugi opendeviationbar-kintsugi-catchup 2>/dev/null; true'
   ssh bigblack 'systemctl --user start opendeviationbar-sidecar'
   sleep 8
   ```

   Running processes have the OLD code in memory; rsync only updates files on disk.

5. **Wait for sync_flush before starting kintsugi** (same as Phase 4.6 — applies to hotfixes too):

   ```bash
   # Use mise run deploy:verify-restart OR manual poll:
   for i in $(seq 1 30); do
     if ssh bigblack 'journalctl --user -u opendeviationbar-sidecar --since "5 min ago" --no-pager 2>/dev/null | grep -q "sync_flush.*drained and wrote"'; then
       echo "sync_flush complete"; break
     fi
     echo "Waiting... ($i/30)"; sleep 10
   done
   ssh bigblack 'systemctl --user start opendeviationbar-kintsugi'
   ssh bigblack 'sudo monit monitor sidecar 2>/dev/null; sudo monit monitor kintsugi 2>/dev/null'
   ```

6. Quick-path verification (Phase 5)

**NEVER** run `maturin develop` or `uv pip install -e .` on bigblack — it creates `.pth` files that shadow the wheel install and cause version drift on next deploy.

---

## Rollback

If a deploy goes wrong:

### Option A: Restore Previous Venv (if .venv-old exists)

```bash
ssh bigblack '
systemctl --user stop opendeviationbar-sidecar opendeviationbar-kintsugi 2>/dev/null; true
cd ~/opendeviationbar-py
if [ -d .venv-old ]; then
  mv .venv .venv-bad && mv .venv-old .venv
  echo "Restored .venv-old"
fi
systemctl --user start opendeviationbar-sidecar opendeviationbar-kintsugi
'
```

### Option B: Install Specific Version

```bash
ssh bigblack "uv pip install --python ~/opendeviationbar-py/.venv/bin/python3 --refresh-package opendeviationbar opendeviationbar==13.13.0"
```

Then restart services.

---

## Auto-Heal Procedures

When checks fail, execute these in order (dependencies flow top-to-bottom):

### Priority 1: Unpushed Commits

```bash
UNPUSHED=$(git log origin/main..HEAD --oneline 2>/dev/null)
if [ -n "$UNPUSHED" ]; then
  git push origin main || GH_TOKEN=$(gh auth token) git \
    -c "url.https://x-access-token:$(gh auth token)@github.com/.insteadOf=git@github.com-terrylica:" \
    push origin main
fi
```

### Priority 2: Git SHA Diverged on bigblack

```bash
ssh bigblack 'cd ~/opendeviationbar-py && git fetch origin main && git reset --hard origin/main'
```

### Priority 3: Version Mismatch

```bash
EXPECTED=$(grep '^version' Cargo.toml | head -1 | sed 's/.*"\(.*\)"/\1/')
ssh bigblack "uv pip install --python ~/opendeviationbar-py/.venv/bin/python3 --upgrade --refresh-package opendeviationbar opendeviationbar==$EXPECTED"
```

### Priority 4: Stale .so in Running Processes

```bash
ssh bigblack '
SO_EPOCH=$(stat -c %Y ~/opendeviationbar-py/.venv/lib/python*/site-packages/opendeviationbar/_core*.so 2>/dev/null | head -1)
for proc in "streaming_sidecar" "scripts/kintsugi"; do
  PID=$(pgrep -f "$proc" | head -1)
  if [ -n "$PID" ]; then
    START_EPOCH=$(date -d "$(ps -o lstart= -p $PID)" +%s 2>/dev/null)
    if [ -n "$START_EPOCH" ] && [ -n "$SO_EPOCH" ] && [ "$START_EPOCH" -lt "$SO_EPOCH" ]; then
      echo "STALE: $proc PID $PID — killing for monit respawn"
      sudo kill $PID
    fi
  fi
done
echo "Waiting 10s for monit respawn..."
sleep 10
pgrep -fa "streaming_sidecar|kintsugi"
'
```

### Priority 5: Dead Letters

```bash
ssh bigblack '
DL_COUNT=$(ls /tmp/opendeviationbar-dead-letter/ 2>/dev/null | wc -l)
if [ "$DL_COUNT" -gt 0 ]; then
  echo "$DL_COUNT dead letters — Charon replays every 5 min"
fi
'
```

---

## Lessons Learned (Hard-Won)

Full list in [references/lessons-learned.md](./references/lessons-learned.md). Critical ones:

### SSH Quoting with ClickHouse Queries

Complex SQL with single quotes (`'UTC'`) breaks when nested in SSH commands. **Solution**: Write query to temp file, SCP, execute remotely:

```bash
cat > /tmp/ch_query.sql << 'SQL'
SELECT count() FROM opendeviationbar_cache.open_deviation_bars FINAL
WHERE toDate(open_time_ms / 1000, 'UTC') != toDate(close_time_ms / 1000, 'UTC')
SQL
scp /tmp/ch_query.sql bigblack:/tmp/
ssh bigblack 'curl -s "http://localhost:8123/" -d @/tmp/ch_query.sql'
```

### Two Git Checkouts on bigblack

bigblack has `~/opendeviationbar-py` (where services run) and `~/eon/opendeviationbar-py` (another checkout). **Always** deploy to `~/opendeviationbar-py`.

### Venv vs Git Checkout Divergence

After `git pull`, the `.venv/lib/python3.13/site-packages/opendeviationbar/` still has the OLD wheel code. Services import from site-packages, not the git checkout. For Python-only hotfixes, rsync to site-packages. For Rust changes, full wheel install.

### Monit Must Be Unmonitored Before Service Stop

```bash
sudo monit unmonitor sidecar  # THEN systemctl stop
```

Otherwise monit restarts the service within 120s (daemon cycle).

### Heartbeat False Alarms During Restart

When kintsugi restarts (brief 0m-uptime window), heartbeat may report UNHEALTHY. This is timing — next cycle (5 min) shows healthy. Check `sudo monit summary` to confirm.

### zsh `status` Variable is Read-Only

On bigblack (zsh shell), never use `status` as a variable name in SSH commands. Use `svc_state` or similar.

### **pycache** Must Be Cleared After rsync

After rsyncing Python files, stale `.pyc` bytecode can cause import issues:

```bash
find $SITE_PKG -type d -name __pycache__ -exec rm -rf {} +
```

### Kintsugi Log Table Rate-Limit Reset

After deploying fixes that change shard discovery logic, the `kintsugi_log` table may contain stale rate-limit entries that prevent immediate re-repair. Clear the table:

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "TRUNCATE TABLE opendeviationbar_cache.kintsugi_log"'
```

### Partition Migration Checklist (learned 2026-03-23)

If the PARTITION BY key changes (requires table recreation), follow this **exact** order:

1. **Stop ALL services** (sidecar, kintsugi, heartbeat)
2. **Create new table** with new partition key
3. **Copy data**: `INSERT INTO new SELECT * FROM old SETTINGS max_partitions_per_insert_block = 0`
4. **Verify counts**: `SELECT count() FROM old` vs `SELECT count() FROM new`
5. **Atomic rename**: `RENAME TABLE old TO backup, new TO production`
6. **OPTIMIZE TABLE FINAL** — **MANDATORY, wait for completion** (10-30 min for 19M+ bars). Without this, the table has 50K+ unmerged parts and every mutation times out at 300s.
7. **Verify parts**: `SELECT count() FROM system.parts WHERE active AND table = 'open_deviation_bars'` — target <10K
8. **Only then** restart services

**Lesson**: Skipping step 6 caused 5+ hours of cascading failures — kintsugi mutations timing out, phantom Stathera gaps, stuck repair queues. The parts count is the root cause, not the partition count.

### ClickHouse DELETE Strategy (#295 — 11 Iterations, Resolved)

The sidecar uses **pre-fill clean-slate `DELETE IN PARTITION`** (v13.49.0). Delete ALL today's bars per (symbol, threshold) partition BEFORE `fill_from_rest`. Scans only the target partition.

| Context          | Strategy              | Syntax                                              | Speed   |
| ---------------- | --------------------- | --------------------------------------------------- | ------- |
| Bulk cleanup     | `DROP PARTITION`      | `ALTER TABLE t DROP PARTITION (sym, thr)`           | Instant |
| Sidecar pre-fill | `DELETE IN PARTITION` | `ALTER TABLE t DELETE IN PARTITION (...) WHERE ...` | Fast    |
| Kintsugi replace | `DELETE IN PARTITION` | Same, with `mutations_sync=1`                       | Fast    |
| Manual recovery  | `DELETE IN PARTITION` | Same, with `mutations_sync=1`                       | Fast    |
| Fallback only    | `ALTER TABLE DELETE`  | Full table scan                                     | Slow    |

**Anti-pattern (BANNED):** `DELETE FROM` (lightweight DELETE) — creates `_row_exists=0` ghost rows causing false Stathera anomalies. Banned since v13.49.0.

**Current approach**: KILL mutations → OPTIMIZE FINAL → `DELETE IN PARTITION` today → midnight preflight → fill_from_rest → sync_flush.

### Start Kintsugi AFTER Sidecar sync_flush Completes (#295)

Kintsugi's `_do_replace()` creates DELETE mutations that scan all parts. If kintsugi starts before the sidecar's sync_flush writes today's bars, kintsugi mutations can delete freshly-written bars as collateral damage.

**Rule**: After starting sidecar, poll logs for `sync_flush: drained and wrote`, verify zero Stathera anomalies on today, THEN start kintsugi.

### Sidecar Startup: Skip Checkpoints + Skip CH Seeding (#295)

Stale checkpoints and CH seeds both override midnight preflight via `max(existing, override)`. The sidecar now:

1. Skips checkpoint injection entirely when `fill_from_rest` is available
2. Skips `_seed_trade_ids_from_clickhouse` — midnight preflight is the sole seed source
3. Uses fresh processors that start from the midnight crossover TID

---

## Monit Configuration Alignment

### 11 Monitors (SSoT: `deploy/monit/opendeviationbar.conf`)

| Monitor             | Type       | Check                                                             | Alert       |
| ------------------- | ---------- | ----------------------------------------------------------------- | ----------- |
| clickhouse          | host:8123  | HTTP /ping                                                        | 3 cycles    |
| sidecar             | process    | PID, mem<38.5GB, cpu<80%                                          | immediate   |
| sidecar-health      | host:8081  | HTTP /health                                                      | 3 cycles    |
| kintsugi            | process    | PID, mem<42.7GB                                                   | immediate   |
| heartbeat-timer     | program    | exit code check                                                   | 3×10 cycles |
| system              | system     | load<8, mem<70%, swap<25%                                         | 3/5 cycles  |
| redis               | host:6379  | Redis protocol                                                    | 3 cycles    |
| rootfs              | filesystem | space<80%/90%, inodes<85%                                         | immediate   |
| monit-heartbeat     | program    | self-report every 10 cycles                                       | 3 cycles    |
| seeder-state        | file       | `/var/tmp/opendeviationbar-seeder-state.txt` mtime <15 min        | 3 cycles    |
| exness-seeder-state | file       | `/var/tmp/opendeviationbar-exness-seeder-state.txt` mtime <30 min | 3 cycles    |

### Monit Files on bigblack

| Local Source                                  | Remote Target                                       |
| --------------------------------------------- | --------------------------------------------------- |
| `deploy/monit/opendeviationbar.conf`          | `/etc/monit/conf.d/opendeviationbar.conf`           |
| `deploy/monit/monit-override.conf`            | `/etc/systemd/system/monit.service.d/override.conf` |
| `deploy/monit/obdpy-monit-alert.sh`           | `/usr/local/bin/obdpy-monit-alert.sh`               |
| `deploy/monit/obdpy-monit-heartbeat.sh`       | `/usr/local/bin/obdpy-monit-heartbeat.sh`           |
| `deploy/monit/obdpy-check-heartbeat-timer.sh` | `/usr/local/bin/obdpy-check-heartbeat-timer.sh`     |

### Syncing Monit Config (Requires sudo)

```bash
scp deploy/monit/opendeviationbar.conf bigblack:/tmp/
ssh bigblack 'sudo cp /tmp/opendeviationbar.conf /etc/monit/conf.d/ && sudo monit reload'
```

### Monit Drop-in Override (for uid/gid exec + namespace isolation)

`deploy/monit/monit-override.conf` fixes systemd hardening that blocks monit's `as uid "tca"`:

- `NoNewPrivileges=false`
- `CapabilityBoundingSet` includes `CAP_SETUID`, `CAP_SETGID`, `CAP_SYS_PTRACE`
- `ProtectHome=no`
- `PrivateTmp=no` — **critical for seeder-state monitor**: monit upstream ships with `PrivateTmp=true`, giving it a private `/var/tmp/` mount namespace. Files written by services to the real `/var/tmp/` are invisible to monit's `check file` monitor. Without `PrivateTmp=no`, `seeder-state` permanently shows "Does not exist" even though the file exists. Diagnosed via `sudo nsenter -t $(systemctl show monit --property=MainPID --value) -m -- ls /var/tmp/`.

---

## Environment Files (SSoT)

| File                                | Tracked   | Purpose                | Deployed To                     |
| ----------------------------------- | --------- | ---------------------- | ------------------------------- |
| `deploy/opendeviationbar.env`       | git       | Non-secret config      | `~/opendeviationbar-py/deploy/` |
| `deploy/opendeviationbar.local.env` | gitignore | Secrets (TOKEN, Redis) | `~/opendeviationbar-py/deploy/` |
| `deploy/sidecar.env`                | git       | Streaming vars         | `~/opendeviationbar-py/deploy/` |

Services load both via `EnvironmentFile=` directives. `local.env` uses `EnvironmentFile=-` (optional, no error if missing).

### v3.0 Environment Variables

| Variable                             | Default    | Purpose                                                                                                           | Set In               |
| ------------------------------------ | ---------- | ----------------------------------------------------------------------------------------------------------------- | -------------------- |
| `OPENDEVIATIONBAR_STREAMING_WS_MODE` | `combined` | WS connection mode: `combined` (2 connections, 2x13 split) or `per_symbol` (rollback to N per-symbol connections) | `deploy/sidecar.env` |

**Rollback**: Set `OPENDEVIATIONBAR_STREAMING_WS_MODE=per_symbol` in `deploy/sidecar.env` and restart sidecar to revert to pre-v3.0 per-symbol WS connections. No code change needed.

**Parquet tick cache**: Sidecar startup (v3.0) reads `~/.cache/opendeviationbar/ticks/{SYMBOL}/YYYY-MM-DD.parquet` to minimize REST API calls. Seeder timer (5 min) populates this cache. If cache is missing/stale, sidecar falls back to full REST fill (no regression).

---

## Systemd Service Reference

| Service          | Type    | MemoryMax | OOMScore | Purpose                                                                |
| ---------------- | ------- | --------- | -------- | ---------------------------------------------------------------------- |
| sidecar          | simple  | 4G        | -500     | Real-time WS streaming (v3.0: combined 2x13 or per-symbol via WS_MODE) |
| kintsugi         | notify  | 20G       | -200     | Gap reconciliation + trailing-edge                                     |
| kintsugi-catchup | notify  | 20G       | -200     | Historical backfill (Conflicts= with kintsugi)                         |
| heartbeat        | oneshot | 256M      | 0        | Health telemetry every 5 min                                           |
| seeder           | oneshot | 6G        | 0        | Proactive Binance Parquet tick cache seeder                            |
| exness-seeder    | oneshot | 6G        | 0        | Proactive Exness forex tick cache seeder                               |

Timers: `heartbeat.timer` (every 5 min), `seeder.timer` (every 5 min), `exness-seeder.timer` (every 15 min).

All services: `TZ=UTC`, `MemorySwapMax=0`, `LD_PRELOAD=libmimalloc.so.2` (sidecar, kintsugi, seeder — all Polars DataFrame processors).

---

## References

| File                                                                | Content                                                   |
| ------------------------------------------------------------------- | --------------------------------------------------------- |
| [verification-checklist.md](./references/verification-checklist.md) | Full 73-point checklist (11 sections)                     |
| [lessons-learned.md](./references/lessons-learned.md)               | All hard-won deployment lessons from production incidents |

## Related Skills

| Skill                                                      | Use When                                                              |
| ---------------------------------------------------------- | --------------------------------------------------------------------- |
| `/sidecar-restart-zero-overlap-zero-gap-recovery-playbook` | Restart recovery patterns, anti-patterns, today's data repair (SSoT)  |
| `/odb-telemetry`                                           | Reading production logs, ClickHouse queries, heartbeat interpretation |
| `/ops-monitoring`                                          | On-demand monitoring scripts, system resource checks                  |
| `/mise:run-full-release`                                   | Generic mise release pipeline (odb-ship is the ODB-specific superset) |
