---
name: odb-ship
description: >-
  Unified release-publish-deploy pipeline for opendeviationbar-py.
  Covers semantic-release versioning, Rust+Python wheel builds, PyPI and crates.io publishing,
  bigblack production deployment (atomic venv swap, systemd sync, service orchestration),
  monit alignment, and 73-point post-deploy verification.
  Use when shipping a new version, deploying hotfixes, verifying production health,
  or recovering from partial release/deploy failures.
  Replaces mise deploy tasks and odb-deploy-verification skill.
allowed-tools: Read, Bash, Grep, Glob, Agent, WebFetch
---

# odb-ship: Release, Publish & Deploy

ultrathink

Unified pipeline from "code ready on main" to "production verified healthy". Claude Code is the orchestrator — dynamically adapting to failures instead of rigid bash scripts that break on SSH quoting edge cases.

## When to Use

- `/odb-ship` — Full pipeline: version bump through production verification
- `/odb-ship deploy` — Deploy only (version already published)
- `/odb-ship verify` — Verify production health without deploying
- `/odb-ship hotfix` — Pure Python-only deploy (no Rust rebuild needed)
- `/odb-ship rollback` — Recover from a bad deploy

---

## Architecture: Why a Skill, Not a Script

The `mise run deploy:bigblack` bash script fails on:

- SSH command quoting with complex ClickHouse queries
- Network transients (SSH timeouts, PyPI CDN delays)
- Partial failures requiring human judgment
- Service orchestration timing (monit respawn windows)

Claude Code as orchestrator **adapts**: retries with different quoting, falls back to HTTPS when SSH fails, waits for monit respawn, and explains what it's doing.

---

## Phase Overview

| Phase | Name                                                    | Purpose                                | Skippable         |
| ----- | ------------------------------------------------------- | -------------------------------------- | ----------------- |
| 0     | [Pre-Flight](#phase-0-pre-flight)                       | Connectivity, credentials, clean state | No                |
| 1     | [Version & Build](#phase-1-version--build)              | Semantic-release + wheel builds        | Yes (deploy-only) |
| 2     | [Publish](#phase-2-publish)                             | PyPI + crates.io + GitHub release      | Yes (deploy-only) |
| 3     | [Deploy](#phase-3-deploy-to-bigblack)                   | Atomic venv swap on bigblack           | No                |
| 4     | [Service Orchestration](#phase-4-service-orchestration) | Stop/start services, monit alignment   | No                |
| 5     | [Verification](#phase-5-verification)                   | 73-point production health check       | No                |

---

## Phase 0: Pre-Flight

### 0.1 Network Connectivity (Run First)

```bash
# GitHub SSH (may fail — HTTPS fallback available)
ssh -o ConnectTimeout=5 -T git@github.com 2>&1 | head -3

# Bigblack SSH (REQUIRED — no fallback)
ssh -o ConnectTimeout=5 bigblack 'echo OK'

# GitHub API via HTTPS (works even when SSH is blocked)
gh api repos/terrylica/opendeviationbar-py --jq '.pushed_at'
```

**If GitHub SSH fails**: Swap remote to HTTPS before any git/release operations:

```bash
git remote set-url origin "https://github.com/terrylica/opendeviationbar-py.git"
# Restore after: git remote set-url origin "git@github.com-terrylica:terrylica/opendeviationbar-py.git"
```

### 0.2 Clean Working Directory

```bash
git status --porcelain
git log --oneline @{u}..HEAD  # unpushed commits
```

- Unpushed commits: `git push origin main` first
- Dirty files: commit or stash before proceeding
- Lockfile drift (uv.lock): `git checkout -- uv.lock`

### 0.3 Version State

```bash
VERSION=$(grep -A5 '\[workspace.package\]' Cargo.toml | grep '^version' | head -1 | sed 's/.*= "\(.*\)"/\1/')
echo "Current Cargo.toml version: $VERSION"
```

---

## Phase 1: Version & Build

### 1.1 Sync with Remote

```bash
git pull --rebase origin main
```

If SSH fails, use HTTPS push:

```bash
GH_TOKEN=$(gh auth token) git \
  -c "url.https://x-access-token:$(gh auth token)@github.com/.insteadOf=git@github.com-terrylica:" \
  push origin main
```

### 1.2 Semantic Release

```bash
mise run release:version
```

This runs `semantic-release --no-ci` which:

- Analyzes commits since last tag (feat: → minor, fix: → patch)
- Bumps `Cargo.toml` + creates git tag + GitHub release
- **CRITICAL**: Must use `--no-ci` flag (otherwise dry-run only)
- **CRITICAL**: Needs SSH or HTTPS connectivity to `git ls-remote`

**If no releasable commits**: semantic-release exits cleanly, creates `.release-skip` file. All downstream phases skip gracefully.

### 1.3 Run Tests

```bash
mise run release:test
```

Gates on both Rust (`cargo nextest`) and Python (`pytest`) passing.

### 1.4 Build Wheels

```bash
mise run release:build-all
```

Builds:

- macOS ARM64 wheel (native maturin)
- Linux x86_64 wheel (zig cross-compile, ~55s)
- Source distribution (sdist)

Output: `dist/opendeviationbar-{VERSION}-*.whl`

---

## Phase 2: Publish

### 2.1 PyPI

```bash
mise run release:pypi
```

Uses 1Password credentials from `~/.claude/.secrets/`. After upload, PyPI CDN may take 30-60s to propagate. The deploy phase polls for availability.

### 2.2 Crates.io

```bash
mise run release:crates
```

Uses `cargo publish --workspace` (Rust 1.90+ native topological ordering). Publishes 5 crates: core, providers, streaming, client, hurst.

**Known issue**: If crates.io publish races with version bump, you get "already exists" errors. Fix: wait for `Cargo.toml` to have the new version, then re-run.

### 2.3 Verify Publications

```bash
# PyPI
curl -sf "https://pypi.org/pypi/opendeviationbar/$VERSION/json" | jq '.info.version'

# Crates.io (check one representative crate)
cargo search opendeviationbar-core --limit 1
```

---

## Phase 3: Deploy to Bigblack

### Critical Context

| Fact                    | Detail                                                                       |
| ----------------------- | ---------------------------------------------------------------------------- |
| **Deploy target**       | `bigblack` (remote GPU host, SSH alias)                                      |
| **Remote dir**          | `/home/tca/opendeviationbar-py`                                              |
| **Python**              | mise-managed 3.13 at `~/.local/share/mise/installs/python/3.13*/bin/python3` |
| **Package manager**     | `uv` at `~/.local/bin/uv` (NOT in venv)                                      |
| **Services managed by** | systemd `--user` (monit watches and auto-restarts)                           |
| **Secrets**             | `deploy/opendeviationbar.local.env` (Telegram TOKEN, Redis password)         |

### 3.1 Git Sync on bigblack

```bash
ssh bigblack 'cd ~/opendeviationbar-py && git fetch origin main --tags && git reset --hard origin/main && git clean -fd'
```

### 3.2 Poll PyPI CDN Readiness

```bash
ssh bigblack "timeout 3 curl -sf 'https://pypi.org/pypi/opendeviationbar/$VERSION/json' | jq -e '.info.version'"
```

Retry up to 8 times with 20s intervals (~2.5 min max). If still not available, fall back to SCP of local wheel.

### 3.3 Build Staging Venv

```bash
REMOTE_PYTHON=$(ssh bigblack "ls -d ~/.local/share/mise/installs/python/3.13*/bin/python3 2>/dev/null | tail -1")
ssh bigblack "cd ~/opendeviationbar-py && rm -rf .venv-staging && uv venv --python $REMOTE_PYTHON .venv-staging"
```

### 3.4 Install Package

**Primary (PyPI)**:

```bash
ssh bigblack "uv pip install --python ~/opendeviationbar-py/.venv-staging/bin/python3 --refresh-package opendeviationbar opendeviationbar==$VERSION"
```

**Fallback (local wheel via SCP)**:

```bash
LINUX_WHEEL=$(ls dist/opendeviationbar-${VERSION}-*manylinux*.whl | head -1)
scp "$LINUX_WHEEL" bigblack:~/opendeviationbar-py/
ssh bigblack "uv pip install --python ~/opendeviationbar-py/.venv-staging/bin/python3 ~/opendeviationbar-py/$(basename $LINUX_WHEEL)"
```

### 3.5 Verify Staging Venv

```bash
ssh bigblack "~/opendeviationbar-py/.venv-staging/bin/python3 -c '
import opendeviationbar; print(opendeviationbar.__version__)
from opendeviationbar import OpenDeviationBarProcessor
from pathlib import Path
import opendeviationbar.clickhouse.cache as ch
assert (Path(ch.__file__).parent / \"schema.sql\").exists()
import opendeviationbar.symbol_registry as sr
assert (Path(sr.__file__).parent / \"data\" / \"symbols.toml\").exists()
print(\"OK: version + .so + schema.sql + symbols.toml\")
'"
```

### 3.6 Nuke .pth Shadowing Artifacts

After install, check for editable-install `.pth` files that shadow the wheel:

```bash
ssh bigblack "
SITE_PKG=\$(~/opendeviationbar-py/.venv-staging/bin/python3 -c 'import site; print(site.getsitepackages()[0])')
PTH=\"\$SITE_PKG/opendeviationbar.pth\"
if [ -f \"\$PTH\" ]; then
  echo 'FOUND .pth shadowing — removing...'
  rm -f \"\$PTH\"
fi
"
```

---

## Phase 4: Service Orchestration

### 4.1 Stop Services (BEFORE Venv Swap)

**CRITICAL**: Must unmonitor from monit first, otherwise monit auto-restarts within 120s:

```bash
ssh bigblack 'sudo monit unmonitor sidecar 2>/dev/null; sudo monit unmonitor kintsugi 2>/dev/null; sleep 1'
ssh bigblack 'systemctl --user stop opendeviationbar-sidecar opendeviationbar-kintsugi opendeviationbar-kintsugi-catchup 2>/dev/null; true'
sleep 2
```

### 4.2 Atomic Venv Swap

```bash
ssh bigblack 'cd ~/opendeviationbar-py && rm -rf .venv-old && mv .venv .venv-old 2>/dev/null; mv .venv-staging .venv && rm -rf .venv-old'
```

Same-filesystem `mv` is atomic. Old venv untouched until swap succeeds.

### 4.3 Sync Systemd Units

```bash
ssh bigblack '
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-sidecar.service ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-kintsugi.service ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-kintsugi-catchup.service ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-heartbeat.service ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-heartbeat.timer ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-seeder.service ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-seeder.timer ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now opendeviationbar-heartbeat.timer
systemctl --user enable --now opendeviationbar-seeder.timer
echo "OK: 5 services + 2 timers synced"
'
```

### 4.4 Verify mimalloc

Services use `LD_PRELOAD=libmimalloc.so.2` to prevent RSS creep:

```bash
ssh bigblack 'test -f /usr/lib/x86_64-linux-gnu/libmimalloc.so.2 && echo "OK: mimalloc" || echo "MISSING: install libmimalloc2.0"'
```

### 4.5 Start Services (Order Matters)

```bash
# Sidecar first (provides health endpoint + real-time data)
ssh bigblack 'systemctl --user start opendeviationbar-sidecar'
sleep 5

# Verify sidecar health before starting kintsugi
ssh bigblack 'curl -sf http://localhost:8081/health | python3 -c "import sys,json; d=json.load(sys.stdin); print(f\"sidecar: {d.get(\"status\",\"?\")}, version: {d.get(\"version\",\"?\")}\")"'

# Kintsugi (gap reconciliation — depends on sidecar for gap-fill coordination)
ssh bigblack 'systemctl --user start opendeviationbar-kintsugi'
sleep 3
```

**Sidecar startup behavior (#286)**: On restart, the sidecar uses buffered WS start + checkpoint restore + unlimited Puck + min-TID overlap guard (v13.27.0). Stale checkpoints are rejected but **preserved** (not deleted). If repairs are needed, follow the **Stop → Delete → Backfill → Start** sequence. Full playbook: [sidecar-restart-zero-overlap-zero-gap-recovery-playbook](/.claude/skills/sidecar-restart-zero-overlap-zero-gap-recovery-playbook/SKILL.md).

### 4.6 Re-Monitor in Monit

```bash
ssh bigblack 'sudo monit monitor sidecar 2>/dev/null; sudo monit monitor kintsugi 2>/dev/null'
```

### 4.7 Verify Services Active

```bash
ssh bigblack '
for svc in opendeviationbar-sidecar opendeviationbar-kintsugi opendeviationbar-heartbeat.timer opendeviationbar-seeder.timer; do
  state=$(systemctl --user is-active $svc 2>/dev/null || echo "inactive")
  echo "$svc: $state"
done
'
```

---

## Phase 5: Verification

Full 73-point checklist in [references/verification-checklist.md](./references/verification-checklist.md). Below are the 5 critical quick-path checks.

### Quick Path (5 Commands)

**1. Version alignment** — all sources agree:

```bash
ssh bigblack '
VER=$(~/opendeviationbar-py/.venv/bin/python3 -c "import opendeviationbar; print(opendeviationbar.__version__)")
cd ~/opendeviationbar-py
TAG=$(git describe --tags --abbrev=0 2>/dev/null | sed "s/^v//")
CARGO=$(grep "^version" Cargo.toml | head -1 | sed "s/.*\"\(.*\)\"/\1/")
echo "Package=$VER Git=$TAG Cargo=$CARGO"
[ "$VER" = "$CARGO" ] && echo "ALIGNED" || echo "MISALIGNED"
'
```

**2. ClickHouse + Stathera gaps**:

```bash
ssh bigblack 'curl -sf "http://localhost:8123/ping" && echo " ClickHouse OK" || echo "ClickHouse DOWN"'
```

**3. Sidecar health**:

```bash
ssh bigblack 'curl -sf http://localhost:8081/health | python3 -m json.tool'
```

**4. Monit summary**:

```bash
ssh bigblack 'sudo monit summary 2>/dev/null'
```

**5. Native extension + constants**:

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar._core import PyOpenDeviationBarProcessor
from opendeviationbar.constants import DEFAULT_THRESHOLD
from opendeviationbar.symbol_registry import get_thresholds_for_symbol
assert DEFAULT_THRESHOLD == 250
assert get_thresholds_for_symbol(\"BTCUSDT\") == (250, 500, 750, 1000)
print(\"All imports + constants OK\")
"'
```

### Post-Deploy: Batch Repair Note

If running `repair_direct_parquet.py` after deploy, use `--workers 24` (or omit for auto-detected default). Thread-local CH cache (Issue #279) makes this fd-safe — ~4N+50 fds for N workers, well under `ulimit -n 1024`. See [scripts/CLAUDE.md Bigblack Capacity Envelope](/scripts/CLAUDE.md#bigblack-capacity-envelope) for full hardware budget.

### Verdict Table

After running all sections, summarize:

| Section                | Status | Notes |
| ---------------------- | ------ | ----- |
| Network & Connectivity |        |       |
| Package & Version      |        |       |
| Native Extension       |        |       |
| Process Management     |        |       |
| ClickHouse Health      |        |       |
| Stathera Audit         |        |       |
| Sidecar Endpoints      |        |       |
| Infrastructure         |        |       |
| Monitoring             |        |       |
| Configuration          |        |       |
| Release Pipeline       |        |       |

Mark each PASS / WARN / FAIL / SKIP.

---

## Hotfix Path (Python-Only Changes)

When the change is pure Python (no Rust modifications), skip the full build pipeline:

1. Push changes to `origin/main`
2. Git sync on bigblack: `ssh bigblack 'cd ~/opendeviationbar-py && git fetch origin main --tags && git reset --hard origin/main'`
3. Rsync Python files to site-packages (excludes Rust .so):

   ```bash
   ssh bigblack '
   SITE_PKG=$(~/opendeviationbar-py/.venv/bin/python3 -c "import site; print(site.getsitepackages()[0])")
   rsync -av --exclude="__pycache__" --exclude="*.pyc" --exclude="_core*.so" \
     ~/opendeviationbar-py/python/opendeviationbar/ \
     $SITE_PKG/opendeviationbar/
   find $SITE_PKG -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null; true
   '
   ```

4. Restart **ALL** services (sidecar + kintsugi + kintsugi-catchup), not just sidecar.
   Running processes have the OLD code in memory; rsync only updates files on disk.
5. Quick-path verification (Phase 5)

**NEVER** run `maturin develop` or `uv pip install -e .` on bigblack — it creates `.pth` files that shadow the wheel install and cause version drift on next deploy.

---

## Rollback

If a deploy goes wrong:

### Option A: Restore Previous Venv (if .venv-old exists)

```bash
ssh bigblack '
systemctl --user stop opendeviationbar-sidecar opendeviationbar-kintsugi 2>/dev/null; true
cd ~/opendeviationbar-py
if [ -d .venv-old ]; then
  mv .venv .venv-bad && mv .venv-old .venv
  echo "Restored .venv-old"
fi
systemctl --user start opendeviationbar-sidecar opendeviationbar-kintsugi
'
```

### Option B: Install Specific Version

```bash
ssh bigblack "uv pip install --python ~/opendeviationbar-py/.venv/bin/python3 --refresh-package opendeviationbar opendeviationbar==13.13.0"
```

Then restart services.

---

## Auto-Heal Procedures

When checks fail, execute these in order (dependencies flow top-to-bottom):

### Priority 1: Unpushed Commits

```bash
UNPUSHED=$(git log origin/main..HEAD --oneline 2>/dev/null)
if [ -n "$UNPUSHED" ]; then
  git push origin main || GH_TOKEN=$(gh auth token) git \
    -c "url.https://x-access-token:$(gh auth token)@github.com/.insteadOf=git@github.com-terrylica:" \
    push origin main
fi
```

### Priority 2: Git SHA Diverged on bigblack

```bash
ssh bigblack 'cd ~/opendeviationbar-py && git fetch origin main && git reset --hard origin/main'
```

### Priority 3: Version Mismatch

```bash
EXPECTED=$(grep '^version' Cargo.toml | head -1 | sed 's/.*"\(.*\)"/\1/')
ssh bigblack "uv pip install --python ~/opendeviationbar-py/.venv/bin/python3 --upgrade --refresh-package opendeviationbar opendeviationbar==$EXPECTED"
```

### Priority 4: Stale .so in Running Processes

```bash
ssh bigblack '
SO_EPOCH=$(stat -c %Y ~/opendeviationbar-py/.venv/lib/python*/site-packages/opendeviationbar/_core*.so 2>/dev/null | head -1)
for proc in "streaming_sidecar" "scripts/kintsugi"; do
  PID=$(pgrep -f "$proc" | head -1)
  if [ -n "$PID" ]; then
    START_EPOCH=$(date -d "$(ps -o lstart= -p $PID)" +%s 2>/dev/null)
    if [ -n "$START_EPOCH" ] && [ -n "$SO_EPOCH" ] && [ "$START_EPOCH" -lt "$SO_EPOCH" ]; then
      echo "STALE: $proc PID $PID — killing for monit respawn"
      sudo kill $PID
    fi
  fi
done
echo "Waiting 10s for monit respawn..."
sleep 10
pgrep -fa "streaming_sidecar|kintsugi"
'
```

### Priority 5: Dead Letters

```bash
ssh bigblack '
DL_COUNT=$(ls /tmp/opendeviationbar-dead-letter/ 2>/dev/null | wc -l)
if [ "$DL_COUNT" -gt 0 ]; then
  echo "$DL_COUNT dead letters — Charon replays every 5 min"
fi
'
```

---

## Lessons Learned (Hard-Won)

Full list in [references/lessons-learned.md](./references/lessons-learned.md). Critical ones:

### SSH Quoting with ClickHouse Queries

Complex SQL with single quotes (`'UTC'`) breaks when nested in SSH commands. **Solution**: Write query to temp file, SCP, execute remotely:

```bash
cat > /tmp/ch_query.sql << 'SQL'
SELECT count() FROM opendeviationbar_cache.open_deviation_bars FINAL
WHERE toDate(open_time_ms / 1000, 'UTC') != toDate(close_time_ms / 1000, 'UTC')
SQL
scp /tmp/ch_query.sql bigblack:/tmp/
ssh bigblack 'curl -s "http://localhost:8123/" -d @/tmp/ch_query.sql'
```

### Two Git Checkouts on bigblack

bigblack has `~/opendeviationbar-py` (where services run) and `~/eon/opendeviationbar-py` (another checkout). **Always** deploy to `~/opendeviationbar-py`.

### Venv vs Git Checkout Divergence

After `git pull`, the `.venv/lib/python3.13/site-packages/opendeviationbar/` still has the OLD wheel code. Services import from site-packages, not the git checkout. For Python-only hotfixes, rsync to site-packages. For Rust changes, full wheel install.

### Monit Must Be Unmonitored Before Service Stop

```bash
sudo monit unmonitor sidecar  # THEN systemctl stop
```

Otherwise monit restarts the service within 120s (daemon cycle).

### Heartbeat False Alarms During Restart

When kintsugi restarts (brief 0m-uptime window), heartbeat may report UNHEALTHY. This is timing — next cycle (5 min) shows healthy. Check `sudo monit summary` to confirm.

### zsh `status` Variable is Read-Only

On bigblack (zsh shell), never use `status` as a variable name in SSH commands. Use `svc_state` or similar.

### **pycache** Must Be Cleared After rsync

After rsyncing Python files, stale `.pyc` bytecode can cause import issues:

```bash
find $SITE_PKG -type d -name __pycache__ -exec rm -rf {} +
```

### Kintsugi Log Table Rate-Limit Reset

After deploying fixes that change shard discovery logic, the `kintsugi_log` table may contain stale rate-limit entries that prevent immediate re-repair. Clear the table:

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "TRUNCATE TABLE opendeviationbar_cache.kintsugi_log"'
```

---

## Monit Configuration Alignment

### 10 Monitors (SSoT: `deploy/monit/opendeviationbar.conf`)

| Monitor         | Type       | Check                                                      | Alert       |
| --------------- | ---------- | ---------------------------------------------------------- | ----------- |
| clickhouse      | host:8123  | HTTP /ping                                                 | 3 cycles    |
| sidecar         | process    | PID, mem<38.5GB, cpu<80%                                   | immediate   |
| sidecar-health  | host:8081  | HTTP /health                                               | 3 cycles    |
| kintsugi        | process    | PID, mem<42.7GB                                            | immediate   |
| heartbeat-timer | program    | exit code check                                            | 3×10 cycles |
| system          | system     | load<8, mem<70%, swap<25%                                  | 3/5 cycles  |
| redis           | host:6379  | Redis protocol                                             | 3 cycles    |
| rootfs          | filesystem | space<80%/90%, inodes<85%                                  | immediate   |
| monit-heartbeat | program    | self-report every 10 cycles                                | 3 cycles    |
| seeder-state    | file       | `/var/tmp/opendeviationbar-seeder-state.txt` mtime <15 min | 3 cycles    |

### Monit Files on bigblack

| Local Source                                  | Remote Target                                       |
| --------------------------------------------- | --------------------------------------------------- |
| `deploy/monit/opendeviationbar.conf`          | `/etc/monit/conf.d/opendeviationbar.conf`           |
| `deploy/monit/monit-override.conf`            | `/etc/systemd/system/monit.service.d/override.conf` |
| `deploy/monit/obdpy-monit-alert.sh`           | `/usr/local/bin/obdpy-monit-alert.sh`               |
| `deploy/monit/obdpy-monit-heartbeat.sh`       | `/usr/local/bin/obdpy-monit-heartbeat.sh`           |
| `deploy/monit/obdpy-check-heartbeat-timer.sh` | `/usr/local/bin/obdpy-check-heartbeat-timer.sh`     |

### Syncing Monit Config (Requires sudo)

```bash
scp deploy/monit/opendeviationbar.conf bigblack:/tmp/
ssh bigblack 'sudo cp /tmp/opendeviationbar.conf /etc/monit/conf.d/ && sudo monit reload'
```

### Monit Drop-in Override (for uid/gid exec + namespace isolation)

`deploy/monit/monit-override.conf` fixes systemd hardening that blocks monit's `as uid "tca"`:

- `NoNewPrivileges=false`
- `CapabilityBoundingSet` includes `CAP_SETUID`, `CAP_SETGID`, `CAP_SYS_PTRACE`
- `ProtectHome=no`
- `PrivateTmp=no` — **critical for seeder-state monitor**: monit upstream ships with `PrivateTmp=true`, giving it a private `/var/tmp/` mount namespace. Files written by services to the real `/var/tmp/` are invisible to monit's `check file` monitor. Without `PrivateTmp=no`, `seeder-state` permanently shows "Does not exist" even though the file exists. Diagnosed via `sudo nsenter -t $(systemctl show monit --property=MainPID --value) -m -- ls /var/tmp/`.

---

## Environment Files (SSoT)

| File                                | Tracked   | Purpose                | Deployed To                     |
| ----------------------------------- | --------- | ---------------------- | ------------------------------- |
| `deploy/opendeviationbar.env`       | git       | Non-secret config      | `~/opendeviationbar-py/deploy/` |
| `deploy/opendeviationbar.local.env` | gitignore | Secrets (TOKEN, Redis) | `~/opendeviationbar-py/deploy/` |
| `deploy/sidecar.env`                | git       | Streaming vars         | `~/opendeviationbar-py/deploy/` |

Services load both via `EnvironmentFile=` directives. `local.env` uses `EnvironmentFile=-` (optional, no error if missing).

---

## Systemd Service Reference

| Service          | Type    | MemoryMax | OOMScore | Purpose                                        |
| ---------------- | ------- | --------- | -------- | ---------------------------------------------- |
| sidecar          | simple  | 4G        | -500     | Real-time WebSocket streaming                  |
| kintsugi         | notify  | 20G       | -200     | Gap reconciliation + trailing-edge             |
| kintsugi-catchup | notify  | 20G       | -200     | Historical backfill (Conflicts= with kintsugi) |
| heartbeat        | oneshot | 256M      | 0        | Health telemetry every 5 min                   |
| seeder           | oneshot | 6G        | 0        | Proactive BTCUSDT Parquet tick cache seeder    |

Timers: `heartbeat.timer` (every 5 min), `seeder.timer` (every 5 min).

All services: `TZ=UTC`, `MemorySwapMax=0`, `LD_PRELOAD=libmimalloc.so.2` (sidecar, kintsugi, seeder — all Polars DataFrame processors).

---

## References

| File                                                                | Content                                                   |
| ------------------------------------------------------------------- | --------------------------------------------------------- |
| [verification-checklist.md](./references/verification-checklist.md) | Full 73-point checklist (11 sections)                     |
| [lessons-learned.md](./references/lessons-learned.md)               | All hard-won deployment lessons from production incidents |

## Related Skills

| Skill                                                      | Use When                                                              |
| ---------------------------------------------------------- | --------------------------------------------------------------------- |
| `/sidecar-restart-zero-overlap-zero-gap-recovery-playbook` | Restart recovery patterns, anti-patterns, today's data repair (SSoT)  |
| `/odb-telemetry`                                           | Reading production logs, ClickHouse queries, heartbeat interpretation |
| `/ops-monitoring`                                          | On-demand monitoring scripts, system resource checks                  |
| `/mise:run-full-release`                                   | Generic mise release pipeline (odb-ship is the ODB-specific superset) |
