# Lessons Learned: Release & Deploy

**Skill**: [odb-ship](../SKILL.md)

Hard-won operational knowledge from production incidents. Each lesson comes from a real failure.

---

## SSH & Networking

### SSH Quoting with ClickHouse Queries

**Incident**: Complex SQL with single quotes (`'UTC'`, `'opendeviationbar_cache'`) breaks when nested in double-quoted SSH commands. Triple-quoting (`'"'"'`) is fragile and unreadable.

**Solution**: Write query to temp file, SCP, execute remotely:

```bash
cat > /tmp/ch_query.sql << 'SQL'
SELECT count() FROM opendeviationbar_cache.open_deviation_bars FINAL
WHERE toDate(open_time_ms / 1000, 'UTC') != toDate(close_time_ms / 1000, 'UTC')
SQL
scp /tmp/ch_query.sql bigblack:/tmp/
ssh bigblack 'curl -s "http://localhost:8123/" -d @/tmp/ch_query.sql'
```

Alternatively, use URL-encoded queries for simple ones:

```bash
ssh bigblack 'curl -s "http://localhost:8123/?query=SELECT+count()+FROM+opendeviationbar_cache.open_deviation_bars"'
```

### zsh Variable Name `status` is Read-Only

**Incident**: SSH commands to bigblack (zsh shell) failed with cryptic errors when using `status` as a variable name.

**Solution**: Always use `svc_state` or similar alternative names in SSH command scripts.

### SSH Command Expansion in Zsh

**Incident**: `SSH_CMD="ssh -o ConnectTimeout=30 ..."` then `$SSH_CMD "$REMOTE_HOST" "command"` fails in zsh — tries to find a command named `ssh -o ConnectTimeout=30 ...`.

**Solution**: Use direct `ssh bigblack` commands. Don't store SSH with options in a variable.

### GitHub SSH Timeouts

**Incident**: ISP-level port 22/443 blocks prevent all SSH git operations, including `semantic-release --no-ci` (which calls `git ls-remote` internally).

**Solution**: Temporarily swap remote to HTTPS, restore after release:

```bash
git remote set-url origin "https://github.com/terrylica/opendeviationbar-py.git"
GH_TOKEN=$(gh auth token) semantic-release --no-ci
# After: git remote set-url origin "git@github.com-terrylica:terrylica/opendeviationbar-py.git"
```

---

## Deploy Infrastructure

### Two Git Checkouts on bigblack

**Incident**: bigblack has `~/opendeviationbar-py` (where systemd services run) AND `~/eon/opendeviationbar-py` (another checkout). `git pull` was run on the wrong one.

**Solution**: Always verify which directory services reference:

```bash
systemctl --user cat opendeviationbar-sidecar | grep ExecStart
```

Deploy target is always `~/opendeviationbar-py`.

### Venv vs Git Checkout Divergence

**Incident**: After `git pull`, `.venv/lib/python3.13/site-packages/opendeviationbar/` still has OLD wheel code. Services import from site-packages, not the git checkout.

**Solution for Python-only hotfixes**: rsync to site-packages (exclude `_core*.so`):

```bash
rsync -av --exclude="__pycache__" --exclude="*.pyc" --exclude="_core*.so" \
  ~/opendeviationbar-py/python/opendeviationbar/ \
  $SITE_PKG/opendeviationbar/
```

**Solution for Rust changes**: Full wheel install via `uv pip install`.

### .pth Shadowing from Editable Installs

**Incident**: `maturin develop` or `uv pip install -e .` creates a `.pth` file in site-packages that adds the git checkout to `sys.path`. This shadows the PyPI wheel — services load stale code from disk even after `uv pip install opendeviationbar==X.Y.Z`.

**Solution**: After wheel install, nuke `.pth` files:

```bash
SITE_PKG=$(.venv/bin/python3 -c 'import site; print(site.getsitepackages()[0])')
rm -f "$SITE_PKG/opendeviationbar.pth"
```

**Prevention**: NEVER run `maturin develop` or `uv pip install -e .` on bigblack.

### **pycache** Stale Bytecode

**Incident**: After rsyncing Python files, stale `.pyc` bytecode causes import errors or uses old code.

**Solution**: Always clear after rsync:

```bash
find $SITE_PKG -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null; true
```

---

## Service Management

### Monit Must Be Unmonitored Before Service Stop

**Incident**: Stopped kintsugi via `systemctl --user stop`, but monit restarted it within 120s (daemon cycle).

**Solution**: Unmonitor first, then stop:

```bash
sudo monit unmonitor sidecar
sudo monit unmonitor kintsugi
systemctl --user stop opendeviationbar-sidecar opendeviationbar-kintsugi
```

### Service Restart Order

**Correct order**:

1. Stop all: sidecar, kintsugi, kintsugi-catchup
2. Swap venv / install package
3. Sync systemd units + daemon-reload
4. Start sidecar first (provides health endpoint)
5. Wait ~5s, verify sidecar health
6. Start kintsugi (depends on sidecar for gap-fill)
7. Re-monitor in monit

### Heartbeat False Alarms During Restart

**Incident**: Heartbeat ran during kintsugi restart window (0m uptime), reported UNHEALTHY with "kintsugi is not healing".

**Solution**: This is a timing artifact. Check `sudo monit summary` to confirm recovery. Next heartbeat cycle (5 min) shows healthy.

### Background Processes via SSH Die on Disconnect

**Incident**: Running `kintsugi.py &` via SSH — process dies when SSH disconnects.

**Solution**: Always use systemd for persistent processes. Never use SSH backgrounding.

### Systemd Drop-in Override for Temporary Config

To temporarily focus kintsugi on a single threshold:

```bash
mkdir -p ~/.config/systemd/user/opendeviationbar-kintsugi.service.d
cat > ~/.config/systemd/user/opendeviationbar-kintsugi.service.d/threshold-250.conf << EOF
[Service]
ExecStart=
ExecStart=/path/to/python3 /path/to/kintsugi.py --daemon --threshold 250
EOF
systemctl --user daemon-reload && systemctl --user restart opendeviationbar-kintsugi
```

First `ExecStart=` (empty) clears the base service's ExecStart.

---

## Release Pipeline

### semantic-release Dry-Run Trap

**Incident**: Ran `semantic-release` outside CI without `--no-ci`. Output showed analysis but no tags/versions were created — all in dry-run mode.

**Solution**: Always use `semantic-release --no-ci` for local releases.

### Partial Semantic-Release Recovery

**Incident**: semantic-release bumped version files but failed before creating tag/release.

**Recovery**:

1. Commit version artifacts: `git add Cargo.toml CHANGELOG.md && git commit -m "chore(release): vX.Y.Z"`
2. Push: `git push origin main`
3. Create tag: `git tag -a vX.Y.Z -m "vX.Y.Z"`
4. Push tag: `git push origin vX.Y.Z`
5. Create GitHub release: `gh release create vX.Y.Z`
6. Publish: `mise run release:pypi && mise run release:crates`

**NEVER** re-run `semantic-release --no-ci` after partial failure — it bumps version AGAIN.

### Crates.io Race Condition

**Incident**: `release:crates` ran before `release:version` finished bumping Cargo.toml, tried to publish old version ("already exists").

**Solution**: Verify `Cargo.toml` has new version before running `mise run release:crates`.

### PyPI CDN Propagation

**Incident**: `uv pip install opendeviationbar==X.Y.Z` failed immediately after PyPI upload.

**Solution**: CDN takes 30-60s. Poll with JSON API:

```bash
curl -sf "https://pypi.org/pypi/opendeviationbar/$VERSION/json" | jq -e '.info.version'
```

### uv Index Cache vs PyPI CDN

**Incident**: Even after PyPI CDN returned the version, `uv pip install` failed because uv's own resolver cache was stale.

**Solution**: Use `--refresh-package opendeviationbar` flag:

```bash
uv pip install --refresh-package opendeviationbar opendeviationbar==$VERSION
```

### Local **version** Lags After Release

**Incident**: After semantic-release bumps Cargo.toml, local `python -c "import opendeviationbar; print(opendeviationbar.__version__)"` shows old version.

**Explanation**: The `.so` bakes in version at compile time. `maturin develop` not yet re-run.

**Not a real problem**: bigblack always has correct version after `uv pip install` (wheel has correct version baked in). The local lag is cosmetic.

### Flaky Integration Tests Block Deploy

**Incident**: `mise run deploy:bigblack` depends on `release:build-all` which runs tests. Binance API rate-limiter tests are flaky (network-dependent).

**Solution**: Deploy should not re-run integration tests if they passed during the release phase. Or: retry flaky tests once before failing.

### deploy:bigblack Script Hangs

**Incident**: The `mise run deploy:bigblack` bash script hung at "Verifying package data integrity..." due to SSH command quoting issues in the TOML heredoc.

**Solution**: Claude Code as orchestrator (this skill) replaces the fragile bash script. Each command is run individually and adapted on failure.

---

## Data Integrity

### Kintsugi Log Table Rate-Limit Stale Entries

**Incident**: After deploying new shard discovery logic (e.g., `day_boundary_exclusion()`), kintsugi couldn't repair newly-discovered shards because old rate-limit entries blocked them.

**Solution**: Clear the table to allow immediate repairs:

```bash
curl -s "http://localhost:8123/" --data-binary "TRUNCATE TABLE opendeviationbar_cache.kintsugi_log"
```

### Cross-Midnight Bar Cascade

**Incident**: Rust LiveBarEngine doesn't reset at UTC midnight. Cross-midnight bars → pre-write gate rejection → retry storm → dead-letter cascade → Charon noise.

**Solution (v13.10.0+)**: L1 sidecar filter (skip cross-midnight bars), L2 pre-write gate (filter+warn instead of raise), L3 Charon cleanup (delete permanently-unplayable dead letters).

### Dead Letters in /tmp

Dead letters live in `/tmp/opendeviationbar-dead-letter/` (NOT `~/opendeviationbar-py/dead_letters/`). Charon replays every 5 min. Stale dead letters that will never replay should be manually deleted.

---

## Sidecar Startup & Ownership (#280, #286)

> **SSoT**: Full patterns, anti-patterns, and decision tree in [sidecar-restart-zero-overlap-zero-gap-recovery-playbook](/.claude/skills/sidecar-restart-zero-overlap-zero-gap-recovery-playbook/SKILL.md). Below are the individual incident write-ups.

### Asclepius (Intra-Day Self-Heal) Is Fundamentally Unsafe

**Incident**: Asclepius attempted to fill old gaps on a live sidecar engine. `backfill_gap()` uses a stateless processor (produces malformed 0-5 dbps bars). `fill_gap()` injects old trades into an advanced processor (produces cascading overlaps — 665 overlaps + 169 gaps in 10 hours).

**Solution**: Asclepius permanently removed (#284). Only two safe gap-fill mechanisms exist: Puck (inline real-time, sub-3s) and kintsugi (full day-recompute, runs tomorrow).

### Never DELETE Today's Bars on Sidecar Restart

**Incident**: Clean-slate DELETE on restart created a gap from midnight to first WS trade. This gap exceeded Puck's 100K trade limit (since removed), and `_reconstruct_today()` OOMed at 2.5G MemoryMax when processing 18 symbols.

**Solution**: No DELETE, no fresh processor. Checkpoint-restored processor + buffered WS start + unlimited Puck. ReplacingMergeTree dedup handles any transient overlaps from checkpoint restoration (same `first_agg_trade_id` key → deduplicated on merge).

### Stale Checkpoints Must Be Auto-Rejected

**Incident**: After manual DELETE of today's bars, checkpoint still pointed past the CH data. Sidecar restored from stale checkpoint, creating a massive invisible gap.

**Solution**: Stale checkpoint guard in `_inject_checkpoints()` — if `cp_first_tid > ch_last_tid + 1`, reject and auto-delete the checkpoint file. This is autonomous (in the codebase, not manual intervention).

### Individual Bar Deletion Cascades

**Incident**: Attempted to delete individual overlapping bars. Each deletion created new gaps adjacent to the deleted bars, making the situation worse.

**Solution**: Never delete individual bars. Only kintsugi's full day-recompute (DELETE entire day + reprocess from scratch) can fix complex corruption. For transient overlaps, rely on ReplacingMergeTree background merge.

### Buffered WS Start Prevents Startup Trade Loss

**Incident**: Between checkpoint load and WS connection, trades were missed. The gap between `_gap_fill()` end and first WS trade caused recurring Stathera anomalies.

**Solution**: `start_ws()` called before checkpoint loading. Trades buffer in mpsc channels (capacity 1000) while initialization completes. Zero trade loss during startup transition.

### Unified Rate Limiting Required

**Incident**: Sidecar Puck, kintsugi, and populate paths each had independent rate limiters. Combined usage exceeded Binance's 6000 weight/min limit.

**Solution**: OnceLock singleton `AdaptiveRateLimiter` in PyO3 bindings — one rate limiter per process. `OPENDEVIATIONBAR_REST_CEILING` env var caps per-process budget (3000 for sidecar, 2400 for kintsugi). Official global limit is 6000 weight/min.

### Sidecar MemoryMax Raised to 4G (#286)

**Incident**: 18-symbol streaming with checkpoint restoration and Puck gap fill exceeded 2.5G MemoryMax, causing OOM kills (~15 restarts/24h).

**Solution**: MemoryHigh 2G→3G, MemoryMax 2.5G→4G, MEMORY_MAX_BYTES 2.7G→4.3G. Observed steady-state ~2.5G with headroom for bursts.

---

## Version Verification Pattern

After ANY deploy, verify with:

```bash
# Package version
.venv/bin/python3 -c 'import opendeviationbar; print(opendeviationbar.__version__)'
# Rust extension loads
.venv/bin/python3 -c 'from opendeviationbar import OpenDeviationBarProcessor; print("OK")'
# Service health
curl -sf http://localhost:8081/health | python3 -m json.tool
# Systemd status
systemctl --user is-active opendeviationbar-sidecar opendeviationbar-kintsugi
```

---

## ClickHouse Mutations

### overlap_strategy="replace" Causes Mutation Storms on Hot Path (v13.32.0)

**Incident**: Changed sidecar's `overlap_strategy` from `"off"` to `"replace"` for every batch write. Each write created an `ALTER TABLE DELETE` mutation scanning 156+ parts. With 72 streaming pairs (~50 bars/min), mutations queued faster than ClickHouse could process them.

**Symptoms**:

- 34+ pending mutations accumulating
- Write lock TTL=120s exceeded (DELETE scan >120s)
- kintsugi reads timing out at 300s
- Sidecar's overlap self-heal detecting overlaps, triggering repairs, creating MORE mutations (feedback loop)

**Root cause**: `ALTER TABLE DELETE` in ClickHouse is a heavyweight mutation, not a lightweight operation. It scans ALL parts to find matching rows. This was already documented in scripts/CLAUDE.md as "Bottleneck #2" from the #275 repair session — but missed during planning.

**Fix**:

1. Reverted `overlap_strategy` to `"off"` for sidecar's hot write path
2. `KILL MUTATION` to clear the backlog (must loop — new mutations created by in-flight requests)
3. L5 ReplacingMergeTree `ORDER BY first_agg_trade_id` handles dedup natively on merge

**Rule**: `overlap_strategy="replace"` is ONLY safe for occasional repairs (kintsugi, self-heal). NEVER for the sidecar's hot write path. Detection + targeted repair is the correct architecture.

### Must Kill Pending Mutations During Deploy

**Incident**: After reverting `overlap_strategy`, the sidecar's overlap self-heal still found overlaps from the mutation storm and triggered `_repair_under_lock()`, which also creates mutations via `store_bars_replacing()`. The self-heal became a mutation amplifier.

**Fix**: Added Step 4.1b to odb-ship: kill ALL pending mutations after stopping services, before venv swap. Loop until count=0 because `KILL MUTATION` is async.

```bash
ssh bigblack 'while true; do
  COUNT=$(curl -s "http://localhost:8123/" --data-binary "SELECT count() FROM system.mutations WHERE database = '"'"'opendeviationbar_cache'"'"' AND is_done = 0")
  if [ "$COUNT" = "0" ]; then break; fi
  curl -s "http://localhost:8123/" --data-binary "KILL MUTATION WHERE database = '"'"'opendeviationbar_cache'"'"' AND is_done = 0" > /dev/null
  sleep 2
done
```
