---
name: odb-deploy-verification
description: >-
  Comprehensive post-deploy verification for opendeviationbar on bigblack.
  Run after every release, deploy, or package update to verify version alignment,
  native extension health, ClickHouse integrity, sidecar endpoints, and infrastructure.
  Use whenever deploying, updating packages, or checking production health after a release.
disable-model-invocation: true
allowed-tools: Read, Bash, Grep, Glob, Agent
---

# ODB Deploy Verification

ultrathink

73-point checklist across 11 sections for verifying opendeviationbar production deployments, with automatic remediation for common failures. Run after every release, deploy, or package update.

## Prerequisites

- SSH access to target host (default: `bigblack`)
- The deploy has already been executed (`mise run deploy:bigblack` or `uv pip install --upgrade opendeviationbar`)

## Execution Strategy

Run checks in parallel where possible. Group SSH commands to minimize round-trips. Use a single SSH session with multiple commands chained by `&&` or `;`.

For local-only checks (version comparison, wheel verification), run locally. For remote checks (ClickHouse, sidecar, process health), run via SSH.

---

## Section Overview

Detailed bash commands for each section are in the reference files. Read them before executing.

| Section                   | Checks | Reference File                                                              | Focus                                      |
| ------------------------- | ------ | --------------------------------------------------------------------------- | ------------------------------------------ |
| 0. Network & Connectivity | 5      | [version-and-release-checks.md](./references/version-and-release-checks.md) | GitHub SSH, bigblack SSH, HTTPS fallback   |
| 1. Package & Version      | 9      | [version-and-release-checks.md](./references/version-and-release-checks.md) | Version alignment across all sources       |
| 2. Native Extension       | 6      | [production-health-checks.md](./references/production-health-checks.md)     | .so import, stale detection, DRY constants |
| 3. Process Management     | 6      | [production-health-checks.md](./references/production-health-checks.md)     | Systemd/monit status, memory, restarts     |
| 4. ClickHouse Health      | 8      | [production-health-checks.md](./references/production-health-checks.md)     | Ping, schema, bar counts, mutations        |
| 5. Stathera Audit         | 4      | [production-health-checks.md](./references/production-health-checks.md)     | Trade-ID continuity gaps and overlaps      |
| 6. Sidecar Endpoints      | 4      | [production-health-checks.md](./references/production-health-checks.md)     | /health, ariadne, catchup, forming bars    |
| 7. Infrastructure         | 6      | [production-health-checks.md](./references/production-health-checks.md)     | Disk, memory, load, Redis, monit, Charon   |
| 8. Monitoring             | 3      | [production-health-checks.md](./references/production-health-checks.md)     | Heartbeat timer, circuit breaker           |
| 9. Configuration          | 5      | [production-health-checks.md](./references/production-health-checks.md)     | Mode, ouroboros, symbols, thresholds       |
| 10. Release Pitfalls      | 7      | [version-and-release-checks.md](./references/version-and-release-checks.md) | Crates.io race, PyPI CDN, dry-run trap     |

---

## Critical Checks (Quick Path)

For a fast post-deploy sanity check, run these 5 commands. They cover the highest-value checks from each section. If all pass, the deploy is likely healthy. Run the full checklist for complete verification.

### 1. Version alignment (Section 1.8)

The single most important check — verifies package version, Cargo.toml, git tag, and running process .so timestamps all agree:

```bash
ssh bigblack '
EXPECTED=$(~/opendeviationbar-py/.venv/bin/python3 -c "import opendeviationbar; print(opendeviationbar.__version__)")
cd ~/opendeviationbar-py
GIT_TAG=$(git describe --tags --abbrev=0 2>/dev/null | sed "s/^v//")
CARGO_VER=$(grep "^version" Cargo.toml | head -1 | sed "s/.*\"\(.*\)\"/\1/")
echo "Package: $EXPECTED | Git tag: $GIT_TAG | Cargo.toml: $CARGO_VER"
# Use svc_state (not status) to avoid zsh read-only variable conflict on bigblack
svc_state="ALIGNED"
[ "$EXPECTED" != "$CARGO_VER" ] && echo "MISMATCH: package != Cargo.toml" && svc_state="MISALIGNED"
[ -n "$GIT_TAG" ] && [ "$EXPECTED" != "$GIT_TAG" ] && echo "MISMATCH: package != git tag" && svc_state="MISALIGNED"

# Check for stale .so in running processes
SO_EPOCH=$(stat -c %Y ~/opendeviationbar-py/.venv/lib/python*/site-packages/opendeviationbar/_core*.so 2>/dev/null | head -1)
for proc in "streaming_sidecar" "kintsugi"; do
  PID=$(pgrep -f "$proc" | head -1)
  if [ -n "$PID" ]; then
    START_EPOCH=$(date -d "$(ps -o lstart= -p $PID)" +%s 2>/dev/null)
    if [ -n "$START_EPOCH" ] && [ -n "$SO_EPOCH" ] && [ "$START_EPOCH" -lt "$SO_EPOCH" ]; then
      echo "STALE .so: $proc (PID $PID) started before package install"
      svc_state="MISALIGNED"
    fi
  fi
done
[ "$svc_state" = "ALIGNED" ] && echo "All version sources ALIGNED at $EXPECTED"
'
```

### 2. ClickHouse + Stathera (Sections 4.1, 5.1)

```bash
ssh bigblack '
echo "--- ClickHouse ping ---"
curl -s "http://localhost:8123/ping"
echo ""
echo "--- Bar count ---"
curl -s "http://localhost:8123/?query=SELECT+count()+FROM+opendeviationbar_cache.open_deviation_bars"
echo ""
echo "--- Stathera gaps (intra-day only) ---"
curl -s "http://localhost:8123/" --data-binary "
WITH bars AS (
    SELECT symbol, threshold_decimal_bps, first_agg_trade_id, last_agg_trade_id,
           lagInFrame(last_agg_trade_id, 1) OVER (PARTITION BY symbol, threshold_decimal_bps ORDER BY first_agg_trade_id) AS prev_last_tid,
           close_time_ms, lagInFrame(close_time_ms, 1) OVER (PARTITION BY symbol, threshold_decimal_bps ORDER BY first_agg_trade_id) AS prev_close_ms
    FROM opendeviationbar_cache.open_deviation_bars FINAL
)
SELECT symbol, threshold_decimal_bps, count() AS gap_count
FROM bars WHERE prev_last_tid > 0 AND first_agg_trade_id - prev_last_tid > 1
  AND toDate(fromUnixTimestamp64Milli(close_time_ms)) - toDate(fromUnixTimestamp64Milli(prev_close_ms)) <= 0
GROUP BY symbol, threshold_decimal_bps ORDER BY gap_count DESC
FORMAT PrettyCompact
" 2>/dev/null
'
```

### 3. Sidecar health (Section 6.1)

```bash
ssh bigblack 'curl -s http://localhost:8081/health | python3 -m json.tool'
```

### 4. Monit status (Section 7.5)

```bash
ssh bigblack 'sudo monit summary 2>/dev/null || echo "Monit not available"'
```

### 5. Native extension import (Section 2.1)

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar._core import PyOpenDeviationBarProcessor
p = PyOpenDeviationBarProcessor(250)
print(f\"Processor OK: threshold={p.threshold_decimal_bps}\")
from opendeviationbar.clickhouse.constants import CLICKHOUSE_TRANSIENT_ERRORS
from opendeviationbar.constants import DEFAULT_THRESHOLD, PRODUCTION_THRESHOLDS
assert DEFAULT_THRESHOLD == 250
assert PRODUCTION_THRESHOLDS == (250, 500, 750, 1000)
print(\"All imports + constants OK\")
"'
```

---

## Auto-Heal

When a check fails, execute these remediation steps AUTOMATICALLY without prompting. Log each action taken in the verdict table's Notes column.

### Auto-heal: Version mismatch on bigblack

If the installed package version on bigblack does not match the expected version (from `Cargo.toml` or latest git tag):

```bash
EXPECTED=$(grep '^version' Cargo.toml | head -1 | sed 's/.*"\(.*\)"/\1/')
ssh bigblack "cd ~/opendeviationbar-py && ~/opendeviationbar-py/.venv/bin/uv pip install --upgrade opendeviationbar==${EXPECTED}"
```

After install, re-run Section 1.8 to confirm alignment.

### Auto-heal: Stale .so in running processes

If Section 1.8 detects processes started before the .so mtime, kill them and let monit respawn:

```bash
ssh bigblack '
SO_EPOCH=$(stat -c %Y ~/opendeviationbar-py/.venv/lib/python*/site-packages/opendeviationbar/_core*.so 2>/dev/null | head -1)
STALE_PIDS=""
for proc in "streaming_sidecar" "scripts/kintsugi"; do
  PID=$(pgrep -f "$proc" | head -1)
  if [ -n "$PID" ]; then
    START_EPOCH=$(date -d "$(ps -o lstart= -p $PID)" +%s 2>/dev/null)
    if [ -n "$START_EPOCH" ] && [ -n "$SO_EPOCH" ] && [ "$START_EPOCH" -lt "$SO_EPOCH" ]; then
      STALE_PIDS="$STALE_PIDS $PID"
    fi
  fi
done
if [ -n "$STALE_PIDS" ]; then
  echo "Killing stale PIDs:$STALE_PIDS"
  sudo kill $STALE_PIDS
  sleep 10
  echo "Waiting for monit respawn..."
  pgrep -fa "streaming_sidecar|kintsugi" || echo "WARN: processes not yet respawned"
else
  echo "No stale processes found"
fi
'
```

### Auto-heal: Unpushed commits (with HTTPS fallback)

If local commits exist that are not on `origin/main`, push them. If SSH fails, automatically fall back to HTTPS:

```bash
# Check for unpushed commits
UNPUSHED=$(git log origin/main..HEAD --oneline 2>/dev/null)
if [ -n "$UNPUSHED" ]; then
  echo "Unpushed commits found, pushing..."
  if ! git push origin main 2>/dev/null; then
    echo "SSH push failed, falling back to HTTPS..."
    GH_TOKEN=$(gh auth token) git \
      -c "url.https://x-access-token:$(gh auth token)@github.com/.insteadOf=git@github.com-terrylica:" \
      push origin main
  fi
fi
```

### Auto-heal: Git SHA diverged on bigblack

If Section 1.6 shows local and remote SHA differ, pull latest on bigblack:

```bash
ssh bigblack 'cd ~/opendeviationbar-py && git fetch origin main && git reset --hard origin/main'
```

### Auto-heal: Dead letters found

If Section 7.6 finds dead-letter files, trigger an immediate Charon replay rather than waiting for the 5-minute cycle:

```bash
ssh bigblack '
DL_COUNT=$(ls ~/opendeviationbar-py/dead_letters/ 2>/dev/null | wc -l)
if [ "$DL_COUNT" -gt 0 ]; then
  echo "Found $DL_COUNT dead letters, triggering immediate replay..."
  curl -s -X POST http://localhost:8081/replay-dead-letters 2>/dev/null \
    || echo "Replay endpoint not available; Charon will handle on next 5-min cycle"
fi
'
```

### Auto-heal execution order

When multiple issues are detected, heal in this order (dependencies flow top-to-bottom):

1. Unpushed commits (ensures bigblack can pull the right code)
2. Git SHA diverged (gets bigblack to the right commit)
3. Version mismatch (installs the right package)
4. Stale .so (restarts processes with the new package)
5. Dead letters (replays after services are healthy)

---

## Key Operational Notes

### Service management on bigblack

Services are managed by **monit**, not systemd units. `sudo systemctl restart opendeviationbar-*` will fail with "Unit not found".

```bash
# Preferred: monit stop+start (graceful)
ssh bigblack 'sudo monit stop sidecar && sudo monit stop kintsugi && sleep 2 && sudo monit start sidecar && sudo monit start kintsugi'

# If monit restart doesn't change PIDs, kill directly (monit auto-respawns):
ssh bigblack 'sudo kill $(pgrep -f streaming_sidecar) $(pgrep -f "scripts/kintsugi")'
```

### Stale .so after upgrade

PyO3 native extensions are loaded once into process memory. After `pip install --upgrade`, running processes still use the old `.so`. The version alignment check (Section 1.8) detects this automatically. If detected, restart services via monit.

### GitHub SSH timeouts

If `ssh -o ConnectTimeout=5 -T git@github.com` fails, all SSH-based git operations are blocked. Use the HTTPS fallback procedure in Section 0.3 of the [version-and-release-checks](./references/version-and-release-checks.md).

---

## Verdict

After running all sections, summarize:

| Section                   | Status | Notes |
| ------------------------- | ------ | ----- |
| 0. Network & Connectivity |        |       |
| 1. Package & Version      |        |       |
| 2. Native Extension       |        |       |
| 3. Process Management     |        |       |
| 4. ClickHouse Health      |        |       |
| 5. Stathera Audit         |        |       |
| 6. Sidecar Endpoints      |        |       |
| 7. Infrastructure         |        |       |
| 8. Monitoring             |        |       |
| 9. Configuration          |        |       |
| 10. Release Pipeline      |        |       |

Mark each as PASS / WARN / FAIL / SKIP. Any FAIL requires immediate action. WARN items should be tracked. SKIP for sections not applicable (e.g., Section 10 only matters during release, not routine health checks).

---

## Common Remediation

| Issue                        | Fix                                                                                                       |
| ---------------------------- | --------------------------------------------------------------------------------------------------------- |
| Version mismatch             | `ssh bigblack 'cd ~/opendeviationbar-py && uv pip install --upgrade opendeviationbar==X.Y.Z'`             |
| Stale .so in running process | Kill PIDs and let monit respawn: `sudo kill $(pgrep -f streaming_sidecar) $(pgrep -f "scripts/kintsugi")` |
| Git SHA diverged             | `ssh bigblack 'cd ~/opendeviationbar-py && git pull origin main'`                                         |
| Stathera gaps                | Kintsugi auto-heals. Check `sudo monit status kintsugi`                                                   |
| ClickHouse down              | `ssh bigblack 'sudo systemctl restart clickhouse-server'`                                                 |
| High memory                  | Check for runaway kintsugi backfill. Consider restart.                                                    |
| Dead letters present         | Charon replays automatically every 5 min. Monitor.                                                        |
| Circuit breaker OPEN         | ClickHouse likely down. Fix CH first, breaker auto-recovers in 60s.                                       |
| GitHub SSH timeout           | Swap remote to HTTPS (see Section 0.3 in reference)                                                       |
| semantic-release dry-run     | Add `--no-ci` flag when running outside CI                                                                |
| Crates.io "already exists"   | Version bump didn't finish before publish. Run `mise run release:crates` after version bump               |
| PyPI version not visible     | CDN propagation delay (30-60s). Check `https://pypi.org/project/opendeviationbar/X.Y.Z/`                  |
| Local Python version stale   | Run `maturin develop` to rebuild the editable install with new version                                    |
