# Version & Release Checks (Sections 0, 1, 10)

**Skill**: [ODB Deploy Verification](../SKILL.md)

---

## Section 0: Network & Connectivity Pre-Check

Run this FIRST. GitHub SSH timeouts and bigblack connectivity failures have derailed entire release+deploy workflows. Detect problems early so you can apply workarounds before wasting time on downstream steps.

### 0.1 GitHub SSH connectivity

```bash
ssh -o ConnectTimeout=5 -T git@github.com 2>&1 | head -3
```

If this times out, GitHub SSH is down or blocked. All `git push`, `semantic-release`, and `cargo publish` steps that use SSH will fail. Apply the HTTPS fallback (see 0.3).

### 0.2 Bigblack SSH connectivity

```bash
ssh -o ConnectTimeout=5 bigblack 'echo OK' 2>&1
```

If this fails, all remote verification sections (3-8) are blocked. Check VPN/ZeroTier status.

### 0.3 GitHub HTTPS fallback (automatic)

When GitHub SSH times out (common with ISP-level port 22/443 blocks), this step detects the failure and automatically swaps to HTTPS. No manual intervention needed.

```bash
# Auto-detect and swap if SSH is down
if ! ssh -o ConnectTimeout=5 -T git@github.com 2>&1 | grep -q "successfully authenticated"; then
  CURRENT_URL=$(git remote get-url origin)
  if echo "$CURRENT_URL" | grep -q "git@github.com"; then
    echo "SSH failed, swapping remote to HTTPS..."
    git remote set-url origin "https://github.com/terrylica/opendeviationbar-py.git"
    echo "Swapped to: $(git remote get-url origin)"
    echo "IMPORTANT: Restore SSH remote after release with:"
    echo "  git remote set-url origin \"git@github.com-terrylica:terrylica/opendeviationbar-py.git\""
  fi
fi

# Verify gh auth is valid (needed for HTTPS push)
gh auth status
```

For pushing via HTTPS, use the `gh auth token` inline override (works even with SSH remote URL configured):

```bash
GH_TOKEN=$(gh auth token) git \
  -c "url.https://x-access-token:$(gh auth token)@github.com/.insteadOf=git@github.com-terrylica:" \
  push origin main
```

**After the release is complete**, restore the SSH remote:

```bash
git remote set-url origin "git@github.com-terrylica:terrylica/opendeviationbar-py.git"
```

### 0.4 GitHub API connectivity (HTTPS)

```bash
gh api repos/terrylica/opendeviationbar-py --jq '.pushed_at' 2>&1 | head -1
```

This uses HTTPS and works even when SSH is blocked. If both SSH and HTTPS fail, you have a genuine network problem.

### 0.5 semantic-release SSH dependency

semantic-release internally runs `git ls-remote --heads <remote>` using whatever URL is configured. If the remote is SSH and SSH is down, semantic-release will fail even with `--no-ci`. You MUST swap the remote URL to HTTPS before running semantic-release (see 0.3). The release script does NOT do this automatically.

Additionally, semantic-release defaults to dry-run when not running in CI. Always use `--no-ci` flag:

```bash
GH_TOKEN=$(gh auth token) semantic-release --no-ci
```

---

## Section 1: Package & Version Integrity

Verify the deployed version matches what was released.

### 1.1 Python package version

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "import opendeviationbar; print(opendeviationbar.__version__)"'
```

Compare against the expected version (from `git describe --tags --abbrev=0` locally or latest PyPI).

### 1.2 Cargo.toml workspace version

```bash
grep '^version' Cargo.toml | head -1
```

Must match the Python `__version__` (without `v` prefix). If they diverge, semantic-release may have partially failed.

**Local dev note**: After a version bump, the local Python `__version__` will still show the old version until you run `maturin develop`. This is expected — the editable install caches the old `.so`. Run `maturin develop` to sync.

### 1.3 PyPI published version

```bash
uv pip index versions opendeviationbar 2>/dev/null | head -5
```

The latest PyPI version should match the deployed version. PyPI CDN propagation can take 30-60 seconds after upload — if the version isn't showing yet, wait and retry rather than treating it as a failure.

### 1.4 Crates.io published versions

Check the 5 published crates:

```bash
for crate in opendeviationbar-core opendeviationbar-providers opendeviationbar-streaming opendeviationbar-client opendeviationbar-hurst; do
  echo "$crate: $(cargo search $crate --limit 1 2>/dev/null | head -1)"
done
```

### 1.5 Git state

```bash
# Local
git log -1 --oneline
git describe --tags --abbrev=0
git status --porcelain

# Remote (bigblack)
ssh bigblack 'cd ~/opendeviationbar-py && git log -1 --oneline && git describe --tags --abbrev=0 && git status --porcelain'
```

Both should show the same commit SHA and tag. No dirty files.

### 1.6 Local vs remote git SHA

```bash
LOCAL_SHA=$(git rev-parse HEAD)
REMOTE_SHA=$(ssh bigblack 'cd ~/opendeviationbar-py && git rev-parse HEAD')
echo "Local:  $LOCAL_SHA"
echo "Remote: $REMOTE_SHA"
[ "$LOCAL_SHA" = "$REMOTE_SHA" ] && echo "MATCH" || echo "DIVERGED"
```

### 1.7 Wheel architecture

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
import opendeviationbar._core as c
import pathlib
so = pathlib.Path(c.__file__)
print(f\"Extension: {so.name}\")
print(f\"Size: {so.stat().st_size / 1024 / 1024:.1f} MB\")
"'
```

The `.so` should be `_core.abi3.so` (stable ABI). Size should be >1 MB (not a stub).

### 1.8 Version alignment (cross-check all sources)

This is the most critical version check. All of these must agree:

```bash
ssh bigblack '
EXPECTED=$(~/opendeviationbar-py/.venv/bin/python3 -c "import opendeviationbar; print(opendeviationbar.__version__)")
echo "Package version: $EXPECTED"

# Git tag (may lag if tags not fetched)
cd ~/opendeviationbar-py
GIT_TAG=$(git describe --tags --abbrev=0 2>/dev/null | sed "s/^v//")
echo "Git tag version:  $GIT_TAG"

# Cargo.toml
CARGO_VER=$(grep "^version" Cargo.toml | head -1 | sed "s/.*\"\(.*\)\"/\1/")
echo "Cargo.toml:       $CARGO_VER"

# Running sidecar process version (query via /proc to read its loaded module)
SIDECAR_PID=$(pgrep -f "streaming_sidecar" | head -1)
if [ -n "$SIDECAR_PID" ]; then
  SIDECAR_START=$(ps -o lstart= -p $SIDECAR_PID 2>/dev/null)
  SO_MTIME=$(stat -c %y ~/opendeviationbar-py/.venv/lib/python*/site-packages/opendeviationbar/_core*.so 2>/dev/null | head -1)
  echo "Sidecar PID:      $SIDECAR_PID (started: $SIDECAR_START)"
  echo ".so mtime:        $SO_MTIME"

  # Parse timestamps and compare
  SIDECAR_EPOCH=$(date -d "$SIDECAR_START" +%s 2>/dev/null)
  SO_EPOCH=$(stat -c %Y ~/opendeviationbar-py/.venv/lib/python*/site-packages/opendeviationbar/_core*.so 2>/dev/null | head -1)
  if [ -n "$SIDECAR_EPOCH" ] && [ -n "$SO_EPOCH" ] && [ "$SIDECAR_EPOCH" -lt "$SO_EPOCH" ]; then
    echo "  *** STALE .so: sidecar started BEFORE package install — RESTART NEEDED ***"
  else
    echo "  Sidecar .so is current"
  fi
else
  echo "Sidecar:          NOT RUNNING"
fi

# Running kintsugi process
KINTSUGI_PID=$(pgrep -f "kintsugi" | head -1)
if [ -n "$KINTSUGI_PID" ]; then
  KINTSUGI_START=$(ps -o lstart= -p $KINTSUGI_PID 2>/dev/null)
  echo "Kintsugi PID:     $KINTSUGI_PID (started: $KINTSUGI_START)"

  KINTSUGI_EPOCH=$(date -d "$KINTSUGI_START" +%s 2>/dev/null)
  if [ -n "$KINTSUGI_EPOCH" ] && [ -n "$SO_EPOCH" ] && [ "$KINTSUGI_EPOCH" -lt "$SO_EPOCH" ]; then
    echo "  *** STALE .so: kintsugi started BEFORE package install — RESTART NEEDED ***"
  else
    echo "  Kintsugi .so is current"
  fi
else
  echo "Kintsugi:         NOT RUNNING"
fi

# Summary — use svc_state (not status) to avoid zsh read-only variable conflict
svc_state="ALIGNED"
[ "$EXPECTED" != "$CARGO_VER" ] && echo "MISMATCH: package ($EXPECTED) != Cargo.toml ($CARGO_VER)" && svc_state="MISALIGNED"
[ -n "$GIT_TAG" ] && [ "$EXPECTED" != "$GIT_TAG" ] && echo "MISMATCH: package ($EXPECTED) != git tag ($GIT_TAG)" && svc_state="MISALIGNED"
[ "$svc_state" = "ALIGNED" ] && echo "All version sources ALIGNED at $EXPECTED"
'
```

If any source is misaligned, investigate:

- **Package vs Cargo.toml**: semantic-release bumped Cargo.toml but `uv pip install` installed a different version
- **Package vs git tag**: Tags not fetched — run `git fetch --tags`
- **Stale .so**: Restart services via monit (`sudo monit restart sidecar kintsugi`) or kill PIDs directly and let monit respawn

### 1.9 Auto-heal version mismatch

If bigblack's installed version does not match the expected version (from local `Cargo.toml` or latest git tag), auto-install the correct version. This runs as part of the auto-heal flow -- no manual intervention needed.

```bash
# Determine expected version from local Cargo.toml
EXPECTED=$(grep '^version' Cargo.toml | head -1 | sed 's/.*"\(.*\)"/\1/')
REMOTE_VER=$(ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "import opendeviationbar; print(opendeviationbar.__version__)" 2>/dev/null')

if [ "$EXPECTED" != "$REMOTE_VER" ]; then
  echo "Version mismatch: local=$EXPECTED bigblack=$REMOTE_VER"
  echo "Auto-installing opendeviationbar==$EXPECTED on bigblack..."
  ssh bigblack "cd ~/opendeviationbar-py && ~/opendeviationbar-py/.venv/bin/uv pip install --upgrade opendeviationbar==${EXPECTED}"

  # Verify install succeeded
  NEW_VER=$(ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "import opendeviationbar; print(opendeviationbar.__version__)" 2>/dev/null')
  if [ "$EXPECTED" = "$NEW_VER" ]; then
    echo "Auto-heal SUCCESS: bigblack now at $NEW_VER"
    echo "NOTE: Running processes still use old .so -- restart needed (see Auto-heal: Stale .so)"
  else
    echo "Auto-heal FAILED: expected $EXPECTED but got $NEW_VER"
  fi
else
  echo "Versions aligned at $EXPECTED"
fi
```

**Important**: After a successful version install, running sidecar/kintsugi processes still hold the old `.so` in memory. The stale .so auto-heal step (SKILL.md) handles restarting them.

---

## Section 10: Release Pipeline Pitfalls

Learned from real failures. Check these when a release didn't go cleanly.

### 10.1 Version file consistency after release

After semantic-release runs, verify ALL version sources agree:

```bash
# All of these should show the same version
echo "Cargo.toml: $(grep '^version' Cargo.toml | head -1)"
echo "Git tag:    $(git describe --tags --abbrev=0)"
echo "Python:     $(python3 -c 'import opendeviationbar; print(opendeviationbar.__version__)')"
```

If Python `__version__` is stale (common after version bump), run `maturin develop` to rebuild the local editable install. Running processes on bigblack need service restart to pick up the new `.so`.

### 10.2 Crates.io "already exists" error

`release:full` may run `release:crates` before `release:version` (semantic-release) finishes bumping version files. This causes crates.io to try publishing the OLD version, which already exists.

**Detection**: Error message `crate opendeviationbar-X@OLD_VERSION already exists on crates.io index`

**Fix**: Run `mise run release:crates` manually AFTER confirming `Cargo.toml` has the new version.

### 10.3 semantic-release SSH dependency

semantic-release uses `git ls-remote --heads <remote>` internally. If the remote URL uses SSH and SSH is blocked, semantic-release fails before analyzing any commits.

**Detection**: `Connection timed out during banner exchange` in semantic-release output

**Fix**: Temporarily swap to HTTPS (Section 0.3), run `GH_TOKEN=$(gh auth token) semantic-release --no-ci`, then restore SSH URL.

### 10.4 semantic-release dry-run outside CI

semantic-release defaults to dry-run when not in a CI environment (no `CI=true` env var). Without `--no-ci`, it analyzes commits and reports what it WOULD do but doesn't actually create tags or bump versions.

**Detection**: Output shows `Skip step "prepare" ... in dry-run mode` and `Skip vX.Y.Z tag creation in dry-run mode`

**Fix**: Always use `semantic-release --no-ci` when running locally.

### 10.5 PyPI CDN propagation delay

After `twine upload` succeeds, the version may not be immediately available via `uv pip index versions` or `pip install`. CDN propagation typically takes 30-60 seconds but can take up to 5 minutes.

**Detection**: `uv pip index versions opendeviationbar` doesn't show the new version immediately after upload

**Fix**: Wait 60 seconds, then retry. The package page at `https://pypi.org/project/opendeviationbar/X.Y.Z/` is usually available before the CDN updates. Check there first.

### 10.6 Git push via HTTPS with gh token

When SSH is blocked, push using `gh auth token` as the credential:

```bash
GH_TOKEN=$(gh auth token) git \
  -c "url.https://x-access-token:$(gh auth token)@github.com/.insteadOf=git@github.com-terrylica:" \
  push origin main
```

This works even with the SSH remote URL configured — the `-c` flag overrides per-invocation without changing the persistent config.

### 10.7 Version string in .so lags behind semantic-release bump (known, cosmetic)

After semantic-release bumps `Cargo.toml` and creates a git tag, the `__version__` string compiled into the local `.so` still shows the OLD version. This is because `maturin develop` has not been re-run yet. The `.so` bakes in the version at compile time.

**This is cosmetic and expected.** The local `python -c "import opendeviationbar; print(opendeviationbar.__version__)"` will show the pre-bump version until you run `maturin develop`.

On bigblack, `uv pip install --upgrade opendeviationbar==X.Y.Z` installs the wheel built by CI, which has the correct version baked in. So the bigblack version is always correct after install.

**Detection**: Local `__version__` != `Cargo.toml` version after a release. This is NOT a real mismatch.

**Fix**: Run `maturin develop` locally. Or simply ignore -- the version alignment check on bigblack is what matters.
