# Production Health Checks (Sections 2-9)

**Skill**: [ODB Deploy Verification](../SKILL.md)

---

## Section 2: Native Extension & Process Health

Verify the native Rust extension is loaded correctly and running processes use the current version.

### 2.1 Native .so import

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar._core import PyOpenDeviationBarProcessor
p = PyOpenDeviationBarProcessor(250)
print(f\"Processor created: threshold={p.threshold_decimal_bps}\")
"'
```

### 2.2 Stale .so detection and auto-restart

Running processes hold old `.so` in memory after `pip install --upgrade`. Section 1.8 detects this with automatic timestamp comparison and STALE warnings.

When stale .so is detected, this check **automatically restarts** affected services. On bigblack, services are managed by **monit** (not systemd units):

```bash
ssh bigblack '
SO_EPOCH=$(stat -c %Y ~/opendeviationbar-py/.venv/lib/python*/site-packages/opendeviationbar/_core*.so 2>/dev/null | head -1)
STALE_PIDS=""
for proc in "streaming_sidecar" "scripts/kintsugi"; do
  PID=$(pgrep -f "$proc" | head -1)
  if [ -n "$PID" ]; then
    START_EPOCH=$(date -d "$(ps -o lstart= -p $PID)" +%s 2>/dev/null)
    if [ -n "$START_EPOCH" ] && [ -n "$SO_EPOCH" ] && [ "$START_EPOCH" -lt "$SO_EPOCH" ]; then
      echo "STALE: $proc (PID $PID) started before .so install"
      STALE_PIDS="$STALE_PIDS $PID"
    else
      echo "OK: $proc (PID $PID) is current"
    fi
  fi
done

if [ -n "$STALE_PIDS" ]; then
  echo "Auto-restarting stale processes:$STALE_PIDS"
  sudo kill $STALE_PIDS
  echo "Waiting 10s for monit respawn..."
  sleep 10
  # Verify new PIDs
  for proc in "streaming_sidecar" "scripts/kintsugi"; do
    NEW_PID=$(pgrep -f "$proc" | head -1)
    if [ -n "$NEW_PID" ]; then
      echo "Respawned: $proc PID $NEW_PID"
    else
      echo "WARN: $proc not yet respawned -- check monit"
    fi
  done
else
  echo "No stale processes -- all current"
fi
'
```

**Do NOT use** `sudo systemctl restart opendeviationbar-*` on bigblack -- the systemd units are not installed. Services are managed by monit.

### 2.3 Circular import test

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar import get_open_deviation_bars, process_trades_to_dataframe
from opendeviationbar.clickhouse import OpenDeviationBarCache, CLICKHOUSE_TRANSIENT_ERRORS
from opendeviationbar.clickhouse.constants import CLICKHOUSE_TRANSIENT_ERRORS, FINAL_READ_SETTINGS
print(\"All imports OK\")
"'
```

### 2.4 DRY constants importable

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar.clickhouse.constants import CLICKHOUSE_TRANSIENT_ERRORS
from opendeviationbar.constants import DEFAULT_THRESHOLD, PRODUCTION_THRESHOLDS
print(f\"CLICKHOUSE_TRANSIENT_ERRORS: {CLICKHOUSE_TRANSIENT_ERRORS}\")
print(f\"DEFAULT_THRESHOLD: {DEFAULT_THRESHOLD}\")
print(f\"PRODUCTION_THRESHOLDS: {PRODUCTION_THRESHOLDS}\")
assert DEFAULT_THRESHOLD == 250
assert PRODUCTION_THRESHOLDS == (250, 500, 750, 1000)
assert ConnectionError in CLICKHOUSE_TRANSIENT_ERRORS
print(\"All constants verified\")
"'
```

### 2.5 Config defaults using correct constants

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar.config.sidecar import SidecarConfig
from opendeviationbar.constants import PRODUCTION_THRESHOLDS
cfg = SidecarConfig()
assert cfg.thresholds == list(PRODUCTION_THRESHOLDS), f\"Expected {list(PRODUCTION_THRESHOLDS)}, got {cfg.thresholds}\"
print(f\"SidecarConfig.thresholds default: {cfg.thresholds} (matches PRODUCTION_THRESHOLDS)\")
"'
```

### 2.6 Code verification via inspect.getsource()

Verify new/modified functions exist in the deployed code. Use module-level function inspection rather than class attribute access, since class names may change across versions:

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
import inspect, importlib
# Verify sidecar module loads and has key functions
sidecar_mod = importlib.import_module(\"opendeviationbar.sidecar\")
members = [name for name, _ in inspect.getmembers(sidecar_mod, inspect.isfunction)]
print(f\"sidecar functions: {members[:10]}\")

# Verify run_sidecar exists (the main entry point)
assert hasattr(sidecar_mod, \"run_sidecar\"), \"run_sidecar not found in sidecar module\"
print(\"run_sidecar entry point: OK\")

# Verify sidecar_api module loads
api_mod = importlib.import_module(\"opendeviationbar.sidecar_api\")
api_members = [name for name, _ in inspect.getmembers(api_mod, inspect.isfunction)]
print(f\"sidecar_api functions: {api_members[:10]}\")
print(\"Code verification OK\")
"'
```

---

## Section 3: Systemd Services & Process Management

### 3.1 Service status overview

Note: On bigblack with zsh, the variable name `status` is read-only. All scripts here use `svc_state` instead.

```bash
ssh bigblack '
for svc in opendeviationbar-sidecar opendeviationbar-kintsugi opendeviationbar-heartbeat.timer; do
  svc_state=$(systemctl is-active "$svc" 2>/dev/null || echo "unknown")
  echo "$svc: $svc_state"
done
'
```

All three should show `active`. For detailed info:

```bash
ssh bigblack 'systemctl status opendeviationbar-sidecar opendeviationbar-kintsugi opendeviationbar-heartbeat.timer --no-pager -l 2>/dev/null | grep -E "Active:|Main PID:|Memory:|Tasks:"'
```

### 3.2 Service memory vs MemoryMax

| Service   | MemoryMax |
| --------- | --------- |
| sidecar   | 2.5G      |
| kintsugi  | 20G       |
| heartbeat | 256M      |

```bash
ssh bigblack 'for svc in opendeviationbar-sidecar opendeviationbar-kintsugi opendeviationbar-heartbeat; do
  mem=$(systemctl show $svc --property=MemoryCurrent --value 2>/dev/null)
  max=$(systemctl show $svc --property=MemoryMax --value 2>/dev/null)
  echo "$svc: current=${mem:-N/A} max=${max:-N/A}"
done'
```

### 3.3 Service restart count

```bash
ssh bigblack 'for svc in opendeviationbar-sidecar opendeviationbar-kintsugi; do
  restarts=$(systemctl show $svc --property=NRestarts --value 2>/dev/null)
  echo "$svc restarts: ${restarts:-0}"
done'
```

Any non-zero restart count warrants investigation (check journal for crash reasons).

### 3.4 Service uptime

```bash
ssh bigblack 'for svc in opendeviationbar-sidecar opendeviationbar-kintsugi; do
  echo "$svc: $(systemctl show $svc --property=ActiveEnterTimestamp --value)"
done'
```

### 3.5 Journal error scan (last 1h)

```bash
ssh bigblack 'journalctl -u "opendeviationbar-*" --since "1 hour ago" --priority=err --no-pager -q 2>/dev/null | tail -20'
```

### 3.6 PID verification

```bash
ssh bigblack 'for svc in opendeviationbar-sidecar opendeviationbar-kintsugi; do
  pid=$(systemctl show $svc --property=MainPID --value)
  if [ "$pid" != "0" ] && [ -d "/proc/$pid" ]; then
    echo "$svc: PID $pid alive"
  else
    echo "$svc: PID $pid NOT FOUND"
  fi
done'
```

---

## Section 4: ClickHouse Health

### 4.1 ClickHouse ping

```bash
ssh bigblack 'curl -s "http://localhost:8123/ping"'
```

Expected: `Ok.`

### 4.2 ClickHouse version

```bash
ssh bigblack 'curl -s "http://localhost:8123/?query=SELECT+version()"'
```

### 4.3 Database and table exist

```bash
ssh bigblack 'curl -s "http://localhost:8123/?query=SELECT+count()+FROM+system.tables+WHERE+database%3D%27opendeviationbar_cache%27+AND+name%3D%27open_deviation_bars%27"'
```

Expected: `1`

### 4.4 Total bar count

```bash
ssh bigblack 'curl -s "http://localhost:8123/?query=SELECT+count()+FROM+opendeviationbar_cache.open_deviation_bars"'
```

Expected: >40M (currently ~42M and growing).

### 4.5 Per-symbol per-threshold bar counts

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT symbol, threshold_decimal_bps, count() as bars
FROM opendeviationbar_cache.open_deviation_bars
GROUP BY symbol, threshold_decimal_bps
ORDER BY symbol, threshold_decimal_bps
FORMAT PrettyCompact
"'
```

### 4.6 Schema column verification

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT name, type FROM system.columns
WHERE database = '\''opendeviationbar_cache'\'' AND table = '\''open_deviation_bars'\''
ORDER BY position
FORMAT PrettyCompact
"'
```

Verify microstructure columns exist (ofi, vwap_close_deviation, kyle_lambda_proxy, etc.).

### 4.7 Pending mutations

```bash
ssh bigblack 'curl -s "http://localhost:8123/?query=SELECT+count()+FROM+system.mutations+WHERE+database%3D%27opendeviationbar_cache%27+AND+is_done%3D0"'
```

Expected: `0` (no pending mutations).

### 4.8 Active merges

```bash
ssh bigblack 'curl -s "http://localhost:8123/?query=SELECT+count()+FROM+system.merges+WHERE+database%3D%27opendeviationbar_cache%27"'
```

Low count is normal. High count may indicate OPTIMIZE in progress.

---

## Section 5: Stathera Audit (Data Integrity)

The Stathera invariant is the core data integrity check: `bar[i].last_agg_trade_id + 1 == bar[i+1].first_agg_trade_id`.

### 5.1 Gap scan

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
WITH bars AS (
    SELECT symbol, threshold_decimal_bps,
           first_agg_trade_id,
           last_agg_trade_id,
           lagInFrame(last_agg_trade_id, 1) OVER (
               PARTITION BY symbol, threshold_decimal_bps
               ORDER BY first_agg_trade_id
           ) AS prev_last_tid,
           close_time_ms,
           lagInFrame(close_time_ms, 1) OVER (
               PARTITION BY symbol, threshold_decimal_bps
               ORDER BY first_agg_trade_id
           ) AS prev_close_ms
    FROM opendeviationbar_cache.open_deviation_bars FINAL
)
SELECT symbol, threshold_decimal_bps,
       count() AS gap_count,
       min(first_agg_trade_id - prev_last_tid - 1) AS min_gap,
       max(first_agg_trade_id - prev_last_tid - 1) AS max_gap
FROM bars
WHERE prev_last_tid > 0
  AND first_agg_trade_id - prev_last_tid > 1
  AND toDate(fromUnixTimestamp64Milli(close_time_ms)) - toDate(fromUnixTimestamp64Milli(prev_close_ms)) <= 0
GROUP BY symbol, threshold_decimal_bps
ORDER BY gap_count DESC
FORMAT PrettyCompact
" 2>/dev/null'
```

Gaps with `day_boundary = 0` are real Stathera violations (not structural day-resets).

### 5.2 Overlap scan

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
WITH bars AS (
    SELECT symbol, threshold_decimal_bps,
           first_agg_trade_id,
           lagInFrame(last_agg_trade_id, 1) OVER (
               PARTITION BY symbol, threshold_decimal_bps
               ORDER BY first_agg_trade_id
           ) AS prev_last_tid
    FROM opendeviationbar_cache.open_deviation_bars FINAL
)
SELECT symbol, threshold_decimal_bps,
       count() AS overlap_count
FROM bars
WHERE prev_last_tid > 0
  AND first_agg_trade_id <= prev_last_tid
GROUP BY symbol, threshold_decimal_bps
HAVING overlap_count > 0
ORDER BY overlap_count DESC
FORMAT PrettyCompact
" 2>/dev/null'
```

Any overlaps should be zero. If present, kintsugi should be actively healing.

### 5.3 Kintsugi healing progress

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT
    result,
    count() AS cnt,
    max(timestamp) AS last_repair
FROM opendeviationbar_cache.kintsugi_log
WHERE timestamp > now() - INTERVAL 24 HOUR
GROUP BY result
FORMAT PrettyCompact
" 2>/dev/null'
```

### 5.4 Kintsugi log recent entries

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT timestamp, symbol, threshold_decimal_bps, result, bars_written
FROM opendeviationbar_cache.kintsugi_log
ORDER BY timestamp DESC
LIMIT 10
FORMAT PrettyCompact
" 2>/dev/null'
```

---

## Section 6: Sidecar HTTP Endpoints

### 6.1 Health endpoint

```bash
ssh bigblack 'curl -s http://localhost:8081/health | python3 -m json.tool'
```

Should return `{"status": "ok", ...}` with version, uptime, and symbol list.

### 6.2 Ariadne lookup

```bash
ssh bigblack 'curl -s "http://localhost:8081/ariadne/BTCUSDT/250" | python3 -c "
import sys, json
d = json.load(sys.stdin)
print(f\"Source: {d.get(\"source\", \"?\")}\")
print(f\"Bars: {len(d.get(\"bars\", []))}\")
print(f\"Sources tried: {d.get(\"sources_tried\", [])}\")
"'
```

### 6.3 Catchup endpoint

```bash
ssh bigblack 'curl -s "http://localhost:8081/catchup/BTCUSDT/250?max_trades=1000" | python3 -c "
import sys, json
d = json.load(sys.stdin)
print(f\"Count: {d.get(\"count\", 0)}\")
print(f\"Partial: {d.get(\"partial\", False)}\")
"'
```

### 6.4 Forming bar status

```bash
ssh bigblack 'curl -s http://localhost:8081/health | python3 -c "
import sys, json
d = json.load(sys.stdin)
engines = d.get(\"engines\", {})
for sym, info in engines.items():
    print(f\"{sym}: forming={info.get(\"has_forming_bar\", \"?\")}\")
" 2>/dev/null'
```

---

## Section 7: Infrastructure & Resources

### 7.1 Disk usage

```bash
ssh bigblack 'df -h / /home 2>/dev/null | tail -n +2'
```

Root should be <80%. Data partition <90%.

### 7.2 Memory and swap

```bash
ssh bigblack 'free -h'
```

Swap usage should be <25% of total swap.

### 7.3 Load average

```bash
ssh bigblack 'uptime'
```

Load average should be <8 (bigblack has multiple cores).

### 7.4 Redis connectivity

```bash
ssh bigblack 'redis-cli ping 2>/dev/null || echo "Redis not available (graceful degradation OK)"'
```

### 7.5 Monit status

```bash
ssh bigblack 'sudo monit summary 2>/dev/null || echo "Monit not available"'
```

All 9 monitors should show `OK` or `Running`. See CLAUDE.md for the monitor list.

### 7.6 Dead-letter directory (Charon)

```bash
ssh bigblack 'ls -la ~/opendeviationbar-py/dead_letters/ 2>/dev/null && echo "Dead letters found - Charon should replay" || echo "No dead letters (clean)"'
```

---

## Section 8: Monitoring & Alerting

### 8.1 Heartbeat timer

```bash
ssh bigblack 'systemctl status opendeviationbar-heartbeat.timer --no-pager 2>/dev/null | grep -E "Active:|Trigger:"'
```

Timer should be `active (waiting)` with next trigger within 5 minutes.

### 8.2 Circuit breaker state

Use the `/health` endpoint as the primary check (reliable, no import path issues). Fall back to Python import if sidecar is not running:

```bash
ssh bigblack '
# Primary: check via sidecar /health endpoint (most reliable)
HEALTH=$(curl -s http://localhost:8081/health 2>/dev/null)
if [ -n "$HEALTH" ]; then
  echo "Sidecar health response:"
  echo "$HEALTH" | python3 -m json.tool 2>/dev/null || echo "$HEALTH"
  # Circuit breaker state is embedded in health response if available
  echo "$HEALTH" | python3 -c "
import sys, json
d = json.load(sys.stdin)
cb = d.get(\"circuit_breaker\", d.get(\"circuit_breaker_state\", \"not reported\"))
print(f\"Circuit breaker: {cb}\")
" 2>/dev/null
else
  echo "Sidecar not responding, falling back to Python import..."
  ~/opendeviationbar-py/.venv/bin/python3 -c "
try:
    from opendeviationbar.orchestration.open_deviation_bars_cache import fatal_write_breaker
    print(f\"Circuit breaker state: {fatal_write_breaker.state.name}\")
except ImportError as e:
    print(f\"Import failed: {e}\")
    print(\"Try: from opendeviationbar.clickhouse.cache import fatal_write_breaker\")
" 2>/dev/null
fi
'
```

Expected: `CLOSED`.

### 8.3 Monit heartbeat last run

```bash
ssh bigblack 'sudo monit status monit-heartbeat 2>/dev/null | grep -E "last|status" || echo "N/A"'
```

---

## Section 9: Configuration Consistency

### 9.1 Connection mode

```bash
ssh bigblack 'cd ~/opendeviationbar-py && grep OPENDEVIATIONBAR_MODE .mise.toml 2>/dev/null'
```

Expected: `remote`.

### 9.2 Ouroboros mode

```bash
ssh bigblack 'cd ~/opendeviationbar-py && grep OUROBOROS .mise.toml 2>/dev/null'
```

Expected: `day`.

### 9.3 Flush threshold

```bash
ssh bigblack 'cd ~/opendeviationbar-py && grep FLUSH_THRESHOLD .mise.toml 2>/dev/null'
```

### 9.4 Symbol registry

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar.symbol_registry import get_registered_symbols
syms = get_registered_symbols()
print(f\"Registered symbols: {len(syms)}\")
for s in sorted(syms)[:5]:
    print(f\"  {s}\")
print(f\"  ... and {len(syms)-5} more\" if len(syms) > 5 else \"\")
"'
```

### 9.5 Threshold configuration

```bash
ssh bigblack 'cd ~/opendeviationbar-py && grep -E "CRYPTO_MIN_THRESHOLD|thresholds" .mise.toml 2>/dev/null'
```
