---
name: odb-deploy-verification
description: >-
  Comprehensive post-deploy verification for opendeviationbar on bigblack or any host.
  Run after every release, deploy, or update to verify package version, native extension,
  systemd services, ClickHouse health, Stathera data integrity, sidecar endpoints,
  infrastructure resources, monitoring, and configuration consistency.
  Use this skill whenever the user deploys, updates, or wants to verify production alignment.
  TRIGGERS - deploy verification, post-deploy check, bigblack alignment, production health,
  update verification, odb verify, post-release check, comprehensive check, bigblack check.
allowed-tools: Read, Bash, Grep, Glob, Agent
---

# ODB Deploy Verification

ultrathink

Comprehensive 58-point checklist for verifying opendeviationbar production deployments. Covers 9 perspectives from package integrity to configuration consistency. Run this after every release, deploy, or package update.

## Prerequisites

- SSH access to target host (default: `bigblack`)
- The deploy has already been executed (`mise run deploy:bigblack` or `uv pip install --upgrade opendeviationbar`)

## Execution Strategy

Run checks in parallel where possible. Group SSH commands to minimize round-trips. Use a single SSH session with multiple commands chained by `&&` or `;`.

For local-only checks (version comparison, wheel verification), run locally. For remote checks (systemd, ClickHouse, sidecar), run via SSH.

---

## Section 1: Package & Version Integrity

Verify the deployed version matches what was released.

### 1.1 Python package version

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "import opendeviationbar; print(opendeviationbar.__version__)"'
```

Compare against the expected version (from `git describe --tags --abbrev=0` locally or latest PyPI).

### 1.2 Cargo.toml workspace version

```bash
grep '^version' Cargo.toml | head -1
```

Must match the Python `__version__` (without `v` prefix). If they diverge, semantic-release may have partially failed.

### 1.3 PyPI published version

```bash
uv pip index versions opendeviationbar 2>/dev/null | head -5
```

The latest PyPI version should match the deployed version.

### 1.4 Crates.io published versions

Check the 5 published crates:

```bash
for crate in opendeviationbar-core opendeviationbar-providers opendeviationbar-streaming opendeviationbar-client opendeviationbar-hurst; do
  echo "$crate: $(cargo search $crate --limit 1 2>/dev/null | head -1)"
done
```

### 1.5 Git state

```bash
# Local
git log -1 --oneline
git describe --tags --abbrev=0
git status --porcelain

# Remote (bigblack)
ssh bigblack 'cd ~/opendeviationbar-py && git log -1 --oneline && git describe --tags --abbrev=0 && git status --porcelain'
```

Both should show the same commit SHA and tag. No dirty files.

### 1.6 Local vs remote git SHA

```bash
LOCAL_SHA=$(git rev-parse HEAD)
REMOTE_SHA=$(ssh bigblack 'cd ~/opendeviationbar-py && git rev-parse HEAD')
echo "Local:  $LOCAL_SHA"
echo "Remote: $REMOTE_SHA"
[ "$LOCAL_SHA" = "$REMOTE_SHA" ] && echo "MATCH" || echo "DIVERGED"
```

### 1.7 Wheel architecture

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
import opendeviationbar._core as c
import pathlib
so = pathlib.Path(c.__file__)
print(f\"Extension: {so.name}\")
print(f\"Size: {so.stat().st_size / 1024 / 1024:.1f} MB\")
"'
```

The `.so` should be `_core.abi3.so` (stable ABI). Size should be >1 MB (not a stub).

---

## Section 2: Native Extension & Process Health

Verify the native Rust extension is loaded correctly and running processes use the current version.

### 2.1 Native .so import

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar._core import PyOpenDeviationBarProcessor
p = PyOpenDeviationBarProcessor(250)
print(f\"Processor created: threshold={p.threshold_decimal_bps}\")
"'
```

### 2.2 Stale .so detection

Running processes hold old `.so` in memory after `pip install --upgrade`. Check process start time vs package install time:

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
import os, pathlib, datetime
so = pathlib.Path(__import__(\"opendeviationbar._core\", fromlist=[\"_core\"]).__file__)
install_time = datetime.datetime.fromtimestamp(so.stat().st_mtime)
print(f\"Package .so mtime: {install_time}\")
"'

# Compare with service start times
ssh bigblack 'for svc in opendeviationbar-sidecar opendeviationbar-kintsugi; do
  echo "$svc: $(systemctl show $svc --property=ActiveEnterTimestamp --value)"
done'
```

If services started BEFORE the package install time, they are using a stale `.so`. Restart them:

```bash
ssh bigblack 'sudo systemctl restart opendeviationbar-sidecar opendeviationbar-kintsugi'
```

### 2.3 Circular import test

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar import get_open_deviation_bars, process_trades_to_dataframe
from opendeviationbar.clickhouse import OpenDeviationBarCache, CLICKHOUSE_TRANSIENT_ERRORS
from opendeviationbar.clickhouse.constants import CLICKHOUSE_TRANSIENT_ERRORS, FINAL_READ_SETTINGS
print(\"All imports OK\")
"'
```

### 2.4 DRY constants importable

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar.clickhouse.constants import CLICKHOUSE_TRANSIENT_ERRORS
from opendeviationbar.constants import DEFAULT_THRESHOLD, PRODUCTION_THRESHOLDS
print(f\"CLICKHOUSE_TRANSIENT_ERRORS: {CLICKHOUSE_TRANSIENT_ERRORS}\")
print(f\"DEFAULT_THRESHOLD: {DEFAULT_THRESHOLD}\")
print(f\"PRODUCTION_THRESHOLDS: {PRODUCTION_THRESHOLDS}\")
assert DEFAULT_THRESHOLD == 250
assert PRODUCTION_THRESHOLDS == (250, 500, 750, 1000)
assert ConnectionError in CLICKHOUSE_TRANSIENT_ERRORS
print(\"All constants verified\")
"'
```

### 2.5 Config defaults using correct constants

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar.config.sidecar import SidecarConfig
from opendeviationbar.constants import PRODUCTION_THRESHOLDS
cfg = SidecarConfig()
assert cfg.thresholds == list(PRODUCTION_THRESHOLDS), f\"Expected {list(PRODUCTION_THRESHOLDS)}, got {cfg.thresholds}\"
print(f\"SidecarConfig.thresholds default: {cfg.thresholds} (matches PRODUCTION_THRESHOLDS)\")
"'
```

### 2.6 Code verification via inspect.getsource()

Verify new/modified functions exist in the deployed code:

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
import inspect
from opendeviationbar.sidecar import StreamingSidecar
src = inspect.getsource(StreamingSidecar._parse_symbol_threshold)
assert \"normalize_symbol\" in src, \"_parse_symbol_threshold should use normalize_symbol\"
print(\"_parse_symbol_threshold verified (uses normalize_symbol)\")
"'
```

---

## Section 3: Systemd Services & Process Management

### 3.1 Service status overview

```bash
ssh bigblack 'systemctl status opendeviationbar-sidecar opendeviationbar-kintsugi opendeviationbar-heartbeat.timer --no-pager -l 2>/dev/null | grep -E "Active:|Main PID:|Memory:|Tasks:"'
```

All three should show `active (running)` or `active (waiting)` for the timer.

### 3.2 Service memory vs MemoryMax

| Service   | MemoryMax |
| --------- | --------- |
| sidecar   | 2.5G      |
| kintsugi  | 20G       |
| heartbeat | 256M      |

```bash
ssh bigblack 'for svc in opendeviationbar-sidecar opendeviationbar-kintsugi opendeviationbar-heartbeat; do
  mem=$(systemctl show $svc --property=MemoryCurrent --value 2>/dev/null)
  max=$(systemctl show $svc --property=MemoryMax --value 2>/dev/null)
  echo "$svc: current=${mem:-N/A} max=${max:-N/A}"
done'
```

### 3.3 Service restart count

```bash
ssh bigblack 'for svc in opendeviationbar-sidecar opendeviationbar-kintsugi; do
  restarts=$(systemctl show $svc --property=NRestarts --value 2>/dev/null)
  echo "$svc restarts: ${restarts:-0}"
done'
```

Any non-zero restart count warrants investigation (check journal for crash reasons).

### 3.4 Service uptime

```bash
ssh bigblack 'for svc in opendeviationbar-sidecar opendeviationbar-kintsugi; do
  echo "$svc: $(systemctl show $svc --property=ActiveEnterTimestamp --value)"
done'
```

### 3.5 Journal error scan (last 1h)

```bash
ssh bigblack 'journalctl -u "opendeviationbar-*" --since "1 hour ago" --priority=err --no-pager -q 2>/dev/null | tail -20'
```

### 3.6 PID verification

```bash
ssh bigblack 'for svc in opendeviationbar-sidecar opendeviationbar-kintsugi; do
  pid=$(systemctl show $svc --property=MainPID --value)
  if [ "$pid" != "0" ] && [ -d "/proc/$pid" ]; then
    echo "$svc: PID $pid alive"
  else
    echo "$svc: PID $pid NOT FOUND"
  fi
done'
```

---

## Section 4: ClickHouse Health

### 4.1 ClickHouse ping

```bash
ssh bigblack 'curl -s "http://localhost:8123/ping"'
```

Expected: `Ok.`

### 4.2 ClickHouse version

```bash
ssh bigblack 'curl -s "http://localhost:8123/?query=SELECT+version()"'
```

### 4.3 Database and table exist

```bash
ssh bigblack 'curl -s "http://localhost:8123/?query=SELECT+count()+FROM+system.tables+WHERE+database%3D%27opendeviationbar_cache%27+AND+name%3D%27open_deviation_bars%27"'
```

Expected: `1`

### 4.4 Total bar count

```bash
ssh bigblack 'curl -s "http://localhost:8123/?query=SELECT+count()+FROM+opendeviationbar_cache.open_deviation_bars"'
```

Expected: >40M (currently ~42M and growing).

### 4.5 Per-symbol per-threshold bar counts

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT symbol, threshold_decimal_bps, count() as bars
FROM opendeviationbar_cache.open_deviation_bars
GROUP BY symbol, threshold_decimal_bps
ORDER BY symbol, threshold_decimal_bps
FORMAT PrettyCompact
"'
```

### 4.6 Schema column verification

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT name, type FROM system.columns
WHERE database = '\''opendeviationbar_cache'\'' AND table = '\''open_deviation_bars'\''
ORDER BY position
FORMAT PrettyCompact
"'
```

Verify microstructure columns exist (ofi, vwap_close_deviation, kyle_lambda_proxy, etc.).

### 4.7 Pending mutations

```bash
ssh bigblack 'curl -s "http://localhost:8123/?query=SELECT+count()+FROM+system.mutations+WHERE+database%3D%27opendeviationbar_cache%27+AND+is_done%3D0"'
```

Expected: `0` (no pending mutations).

### 4.8 Active merges

```bash
ssh bigblack 'curl -s "http://localhost:8123/?query=SELECT+count()+FROM+system.merges+WHERE+database%3D%27opendeviationbar_cache%27"'
```

Low count is normal. High count may indicate OPTIMIZE in progress.

---

## Section 5: Stathera Audit (Data Integrity)

The Stathera invariant is the core data integrity check: `bar[i].last_agg_trade_id + 1 == bar[i+1].first_agg_trade_id`.

### 5.1 Gap scan

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
WITH bars AS (
    SELECT symbol, threshold_decimal_bps,
           first_agg_trade_id,
           last_agg_trade_id,
           lagInFrame(last_agg_trade_id, 1) OVER (
               PARTITION BY symbol, threshold_decimal_bps
               ORDER BY first_agg_trade_id
           ) AS prev_last_tid,
           close_time_ms,
           lagInFrame(close_time_ms, 1) OVER (
               PARTITION BY symbol, threshold_decimal_bps
               ORDER BY first_agg_trade_id
           ) AS prev_close_ms
    FROM opendeviationbar_cache.open_deviation_bars FINAL
)
SELECT symbol, threshold_decimal_bps,
       count() AS gap_count,
       min(first_agg_trade_id - prev_last_tid - 1) AS min_gap,
       max(first_agg_trade_id - prev_last_tid - 1) AS max_gap
FROM bars
WHERE prev_last_tid > 0
  AND first_agg_trade_id - prev_last_tid > 1
  AND toDate(fromUnixTimestamp64Milli(close_time_ms)) - toDate(fromUnixTimestamp64Milli(prev_close_ms)) <= 0
GROUP BY symbol, threshold_decimal_bps
ORDER BY gap_count DESC
FORMAT PrettyCompact
" 2>/dev/null'
```

Gaps with `day_boundary = 0` are real Stathera violations (not structural day-resets).

### 5.2 Overlap scan

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
WITH bars AS (
    SELECT symbol, threshold_decimal_bps,
           first_agg_trade_id,
           lagInFrame(last_agg_trade_id, 1) OVER (
               PARTITION BY symbol, threshold_decimal_bps
               ORDER BY first_agg_trade_id
           ) AS prev_last_tid
    FROM opendeviationbar_cache.open_deviation_bars FINAL
)
SELECT symbol, threshold_decimal_bps,
       count() AS overlap_count
FROM bars
WHERE prev_last_tid > 0
  AND first_agg_trade_id <= prev_last_tid
GROUP BY symbol, threshold_decimal_bps
HAVING overlap_count > 0
ORDER BY overlap_count DESC
FORMAT PrettyCompact
" 2>/dev/null'
```

Any overlaps should be zero. If present, kintsugi should be actively healing.

### 5.3 Kintsugi healing progress

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT
    status,
    count() AS cnt,
    max(repaired_at) AS last_repair
FROM opendeviationbar_cache.kintsugi_log
WHERE repaired_at > now() - INTERVAL 24 HOUR
GROUP BY status
FORMAT PrettyCompact
" 2>/dev/null'
```

### 5.4 Kintsugi log recent entries

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT repaired_at, symbol, threshold_decimal_bps, status, gap_trades
FROM opendeviationbar_cache.kintsugi_log
ORDER BY repaired_at DESC
LIMIT 10
FORMAT PrettyCompact
" 2>/dev/null'
```

---

## Section 6: Sidecar HTTP Endpoints

### 6.1 Health endpoint

```bash
ssh bigblack 'curl -s http://localhost:8081/health | python3 -m json.tool'
```

Should return `{"status": "ok", ...}` with version, uptime, and symbol list.

### 6.2 Ariadne lookup

```bash
ssh bigblack 'curl -s "http://localhost:8081/ariadne/BTCUSDT/250" | python3 -c "
import sys, json
d = json.load(sys.stdin)
print(f\"Source: {d.get(\"source\", \"?\")}\")
print(f\"Bars: {len(d.get(\"bars\", []))}\")
print(f\"Sources tried: {d.get(\"sources_tried\", [])}\")
"'
```

### 6.3 Catchup endpoint

```bash
ssh bigblack 'curl -s "http://localhost:8081/catchup/BTCUSDT/250?max_trades=1000" | python3 -c "
import sys, json
d = json.load(sys.stdin)
print(f\"Count: {d.get(\"count\", 0)}\")
print(f\"Partial: {d.get(\"partial\", False)}\")
"'
```

### 6.4 Forming bar status

```bash
ssh bigblack 'curl -s http://localhost:8081/health | python3 -c "
import sys, json
d = json.load(sys.stdin)
engines = d.get(\"engines\", {})
for sym, info in engines.items():
    print(f\"{sym}: forming={info.get(\"has_forming_bar\", \"?\")}\")
" 2>/dev/null'
```

---

## Section 7: Infrastructure & Resources

### 7.1 Disk usage

```bash
ssh bigblack 'df -h / /home 2>/dev/null | tail -n +2'
```

Root should be <80%. Data partition <90%.

### 7.2 Memory and swap

```bash
ssh bigblack 'free -h'
```

Swap usage should be <25% of total swap.

### 7.3 Load average

```bash
ssh bigblack 'uptime'
```

Load average should be <8 (bigblack has multiple cores).

### 7.4 Redis connectivity

```bash
ssh bigblack 'redis-cli ping 2>/dev/null || echo "Redis not available (graceful degradation OK)"'
```

### 7.5 Monit status

```bash
ssh bigblack 'sudo monit summary 2>/dev/null || echo "Monit not available"'
```

All 9 monitors should show `OK` or `Running`. See CLAUDE.md for the monitor list.

### 7.6 Dead-letter directory (Charon)

```bash
ssh bigblack 'ls -la ~/opendeviationbar-py/dead_letters/ 2>/dev/null && echo "Dead letters found - Charon should replay" || echo "No dead letters (clean)"'
```

---

## Section 8: Monitoring & Alerting

### 8.1 Heartbeat timer

```bash
ssh bigblack 'systemctl status opendeviationbar-heartbeat.timer --no-pager 2>/dev/null | grep -E "Active:|Trigger:"'
```

Timer should be `active (waiting)` with next trigger within 5 minutes.

### 8.2 Circuit breaker state

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar.orchestration.open_deviation_bars_cache import fatal_write_breaker
print(f\"Circuit breaker state: {fatal_write_breaker.state.name}\")
" 2>/dev/null || echo "Cannot check circuit breaker (import error?)"'
```

Expected: `CLOSED`.

### 8.3 Monit heartbeat last run

```bash
ssh bigblack 'sudo monit status monit-heartbeat 2>/dev/null | grep -E "last|status" || echo "N/A"'
```

---

## Section 9: Configuration Consistency

### 9.1 Connection mode

```bash
ssh bigblack 'cd ~/opendeviationbar-py && grep OPENDEVIATIONBAR_MODE .mise.toml 2>/dev/null'
```

Expected: `remote`.

### 9.2 Ouroboros mode

```bash
ssh bigblack 'cd ~/opendeviationbar-py && grep OUROBOROS .mise.toml 2>/dev/null'
```

Expected: `day`.

### 9.3 Flush threshold

```bash
ssh bigblack 'cd ~/opendeviationbar-py && grep FLUSH_THRESHOLD .mise.toml 2>/dev/null'
```

### 9.4 Symbol registry

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar.symbol_registry import get_registered_symbols
syms = get_registered_symbols()
print(f\"Registered symbols: {len(syms)}\")
for s in sorted(syms)[:5]:
    print(f\"  {s}\")
print(f\"  ... and {len(syms)-5} more\" if len(syms) > 5 else \"\")
"'
```

### 9.5 Threshold configuration

```bash
ssh bigblack 'cd ~/opendeviationbar-py && grep -E "CRYPTO_MIN_THRESHOLD|thresholds" .mise.toml 2>/dev/null'
```

---

## Verdict

After running all sections, summarize:

| Section              | Status | Notes |
| -------------------- | ------ | ----- |
| 1. Package & Version |        |       |
| 2. Native Extension  |        |       |
| 3. Systemd Services  |        |       |
| 4. ClickHouse Health |        |       |
| 5. Stathera Audit    |        |       |
| 6. Sidecar Endpoints |        |       |
| 7. Infrastructure    |        |       |
| 8. Monitoring        |        |       |
| 9. Configuration     |        |       |

Mark each as PASS / WARN / FAIL. Any FAIL requires immediate action. WARN items should be tracked.

---

## Common Remediation

| Issue                        | Fix                                                                                           |
| ---------------------------- | --------------------------------------------------------------------------------------------- |
| Version mismatch             | `ssh bigblack 'cd ~/opendeviationbar-py && uv pip install --upgrade opendeviationbar==X.Y.Z'` |
| Stale .so in running process | `ssh bigblack 'sudo systemctl restart opendeviationbar-sidecar opendeviationbar-kintsugi'`    |
| Git SHA diverged             | `ssh bigblack 'cd ~/opendeviationbar-py && git pull origin main'`                             |
| Stathera gaps                | Kintsugi should auto-heal. Check `systemctl status opendeviationbar-kintsugi`                 |
| ClickHouse down              | `ssh bigblack 'sudo systemctl restart clickhouse-server'`                                     |
| High memory                  | Check for runaway kintsugi backfill. Consider restart.                                        |
| Dead letters present         | Charon replays automatically every 5 min. Monitor.                                            |
| Circuit breaker OPEN         | ClickHouse likely down. Fix CH first, breaker auto-recovers in 60s.                           |
