# IMMEDIATE ACTIONS - Production Hang 2.5.18

**Status:** URGENT - Production indexing frozen
**ETA to Fix:** 3 hours (hotfix development + testing + release)

---

## TL;DR - What Happened

**Bug:** `time.sleep()` blocking call in async event loop → infinite deadlock
**Version:** 2.5.18 (memory_monitor.py line 175)
**Impact:** All indexing jobs hang when memory limit exceeded
**Fix:** Change `time.sleep()` → `asyncio.sleep()` (3 line change)

---

## 🚨 IMMEDIATE ACTIONS (Next 15 minutes)

### 1. Kill Hung Process on AWS

```bash
# SSH using EC2 Instance Connect
aws ec2-instance-connect ssh --instance-id i-02d8caca52b2e209b --region us-east-1

# Kill the process
sudo kill -9 20032

# Verify it's dead
ps aux | grep 20032
```

### 2. Downgrade to Working Version

```bash
# On AWS instance (or any affected server)
pip uninstall mcp-vector-search -y
pip install mcp-vector-search==2.2.40

# Verify version
python -c "import mcp_vector_search; print(mcp_vector_search.__version__)"
# Expected output: 2.2.40
```

### 3. Re-run Indexing

```bash
# Run indexing with working version
mcp-vector-search index /path/to/your/project

# Monitor the first 5 minutes to ensure it doesn't hang
tail -f /tmp/reindex-*.log
```

---

## 🔧 HOTFIX DEVELOPMENT (Next 2 hours)

### Step 1: Apply Code Fix (10 minutes)

**File:** `src/mcp_vector_search/core/memory_monitor.py`

```python
# Line 3: Add asyncio import
import asyncio
import os
from collections.abc import Callable

# Line 154: Make function async
async def wait_for_memory_available(
    self, target_pct: float = 0.8, poll_interval_sec: float = 1.0
) -> None:
    """Block until memory usage drops below target threshold."""
    # Line 165: Remove time import (delete this line)
    # import time

    usage_pct = self.get_memory_usage_pct()
    if usage_pct < target_pct:
        return

    logger.info(
        f"Memory usage at {usage_pct * 100:.1f}%, waiting for it to drop below {target_pct * 100:.0f}%..."
    )

    # Line 175: Replace time.sleep with asyncio.sleep
    while usage_pct >= target_pct:
        await asyncio.sleep(poll_interval_sec)  # ← CRITICAL FIX
        usage_pct = self.get_memory_usage_pct()

    logger.info(
        f"Memory usage dropped to {usage_pct * 100:.1f}%, resuming processing"
    )
```

**File:** `src/mcp_vector_search/core/indexer.py`

```python
# Line 496: Add await
await self.memory_monitor.wait_for_memory_available(
    target_pct=self.memory_monitor.warn_threshold
)

# Line 625: Add await
await self.memory_monitor.wait_for_memory_available(
    target_pct=self.memory_monitor.warn_threshold
)
```

**Or apply patch:**

```bash
cd /Users/masa/Projects/mcp-vector-search
git apply HOTFIX_2.5.19.patch
```

### Step 2: Run Tests (30 minutes)

```bash
# Install test dependencies
pip install -e ".[dev]"

# Run new integration test
pytest tests/integration/test_memory_backpressure.py -v -s

# Run existing tests
pytest tests/ -v

# Expected: All tests pass
```

### Step 3: Manual Testing (30 minutes)

```bash
# Test 1: Low memory cap (triggers backpressure)
export MCP_VECTOR_SEARCH_MAX_MEMORY_GB=1
mcp-vector-search index tests/fixtures/sample_project

# Expected:
# - Indexing completes (no hang)
# - Logs show "Memory usage at X%, waiting..."
# - Logs show "Memory usage dropped to Y%, resuming processing"

# Test 2: Normal memory cap (no backpressure)
export MCP_VECTOR_SEARCH_MAX_MEMORY_GB=25
mcp-vector-search index tests/fixtures/sample_project

# Expected:
# - Indexing completes normally
# - No memory warnings

# Test 3: Large project (stress test)
mcp-vector-search index /path/to/large/project

# Expected:
# - Indexing completes
# - Pipeline parallelism works
# - Memory monitoring works without hangs
```

### Step 4: Build and Release (30 minutes)

```bash
# Update version
# File: src/mcp_vector_search/__init__.py
__version__ = "2.5.19"

# Update changelog
# File: CHANGELOG.md
## [2.5.19] - 2026-02-18

### Fixed
- **CRITICAL:** Fixed event loop hang when memory limit exceeded during indexing
  - Changed memory_monitor.wait_for_memory_available() from blocking to async
  - Replaced time.sleep() with asyncio.sleep() to prevent event loop deadlock
  - Affects all users running large indexing jobs with 2.5.18
  - Downgrade to 2.2.40 if experiencing hangs with 2.5.18

# Commit
git add .
git commit -m "fix: replace blocking time.sleep with asyncio.sleep in memory monitor

CRITICAL FIX for 2.5.18 hang bug:
- Make wait_for_memory_available() async
- Replace time.sleep() with asyncio.sleep()
- Add integration tests for memory backpressure
- Prevents event loop deadlock when memory limit exceeded

Fixes: Production hang on AWS (i-02d8caca52b2e209b)
Root cause: Blocking call in async context
Impact: All indexing jobs hang when memory cap hit

Users affected by 2.5.18: Downgrade to 2.2.40 until 2.5.19 released

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>"

# Tag
git tag -a v2.5.19 -m "Hotfix: Fix memory monitor event loop hang"

# Push
git push origin main --tags

# Build
python -m build

# Upload to PyPI
python -m twine upload dist/mcp_vector_search-2.5.19*
```

---

## 🧪 VERIFICATION (30 minutes)

### Deploy to AWS GPU Instance

```bash
# SSH to instance
aws ec2-instance-connect ssh --instance-id i-02d8caca52b2e209b --region us-east-1

# Upgrade to 2.5.19
pip install --upgrade mcp-vector-search==2.5.19

# Verify version
python -c "import mcp_vector_search; print(mcp_vector_search.__version__)"
# Expected: 2.5.19

# Run full reindex with default memory cap
mcp-vector-search index /path/to/project --log-file /tmp/reindex-2.5.19-verification.log

# Monitor for first 10 minutes
tail -f /tmp/reindex-2.5.19-verification.log

# Expected behavior:
# ✅ File discovery completes (31,699 files)
# ✅ Chunking starts immediately (no hang)
# ✅ Embedding phase starts
# ✅ Progress updates every few seconds
# ✅ No infinite hangs
```

### Stress Test with Low Memory Cap

```bash
# Test backpressure mechanism
export MCP_VECTOR_SEARCH_MAX_MEMORY_GB=5
mcp-vector-search index /path/to/project --log-file /tmp/reindex-2.5.19-stress.log

# Monitor logs
tail -f /tmp/reindex-2.5.19-stress.log

# Expected behavior:
# ✅ "Memory usage high: X GB / 5.0GB"
# ✅ "Memory usage at X%, waiting for it to drop below Y%..."
# ✅ "Memory usage dropped to Z%, resuming processing"
# ✅ Indexing completes (may be slower, but no hang)
```

---

## 📊 SUCCESS CRITERIA

### Must Pass:
- ✅ Integration test `test_memory_backpressure.py` passes
- ✅ All existing tests pass
- ✅ Manual test with low memory cap (1GB) completes
- ✅ AWS GPU instance can complete full reindex (31k files)
- ✅ No hangs observed in 30-minute monitoring window

### Performance:
- ✅ Indexing speed matches 2.2.40 baseline
- ✅ Pipeline parallelism efficiency >30%
- ✅ Memory backpressure activates at 80% threshold
- ✅ Event loop remains responsive during memory wait

---

## 📢 COMMUNICATION

### Internal Status Update (Every 30 minutes)

**Update 1 (T+0):**
> 🚨 Production hang root cause identified: blocking time.sleep() in async event loop
> - Affected version: 2.5.18
> - Fix: 3-line change (time.sleep → asyncio.sleep)
> - Immediate action: Downgrading to 2.2.40
> - ETA for hotfix: 3 hours

**Update 2 (T+30m):**
> ✅ AWS process killed, downgraded to 2.2.40
> 🔧 Hotfix code complete and tested locally
> 📦 Building release 2.5.19
> ⏱️ ETA: 2.5 hours

**Update 3 (T+2h):**
> ✅ 2.5.19 released to PyPI
> 🧪 Deploying to AWS for verification
> ⏱️ ETA: 1 hour

**Update 4 (T+3h):**
> ✅ AWS verification complete - no hangs
> ✅ Stress test with low memory cap passed
> 📣 Announcing fix to users
> ✅ INCIDENT RESOLVED

### User Announcement (After verification)

**Subject:** mcp-vector-search 2.5.19 Released - Fixes Critical Hang Bug

**Body:**
> We've released version 2.5.19 which fixes a critical bug in 2.5.18 that caused indexing to hang indefinitely.
>
> **What Happened:**
> Version 2.5.18 introduced a memory monitoring feature that used a blocking call (`time.sleep()`) in an async context, causing the event loop to freeze when memory limits were exceeded.
>
> **Who Was Affected:**
> Users running large indexing jobs (>10k files) with version 2.5.18, especially on servers with <25GB RAM.
>
> **How to Fix:**
> ```bash
> pip install --upgrade mcp-vector-search==2.5.19
> ```
>
> **If You're Currently Stuck:**
> 1. Kill the hung process: `pkill -9 mcp-vector-search`
> 2. Upgrade to 2.5.19: `pip install --upgrade mcp-vector-search==2.5.19`
> 3. Re-run indexing: `mcp-vector-search index /path/to/project`
>
> **Technical Details:**
> See: INCIDENT_REPORT_2.5.18_HANG.md
>
> We apologize for the disruption and have added integration tests to prevent similar issues in the future.

---

## 🔮 PREVENTION (Next Release)

### Add to CI/CD Pipeline

```yaml
# .github/workflows/test.yml
- name: Test async event loop safety
  run: |
    pytest tests/integration/test_memory_backpressure.py -v
    pytest tests/ --cov=mcp_vector_search --cov-report=term-missing
```

### Add Lint Rules

```toml
# pyproject.toml
[tool.ruff.lint]
select = ["ASYNC"]  # Detect blocking calls in async functions

[tool.ruff.lint.flake8-async]
block-on-timeout = true
```

### Add Pre-commit Hook

```yaml
# .pre-commit-config.yaml
- repo: https://github.com/astral-sh/ruff-pre-commit
  rev: v0.1.9
  hooks:
    - id: ruff
      args: [--select, ASYNC]
```

---

## 📋 CHECKLIST

### Immediate (15 minutes)
- [ ] Kill hung process on AWS (PID 20032)
- [ ] Downgrade to 2.2.40 on AWS
- [ ] Verify indexing works with 2.2.40
- [ ] Notify team of root cause

### Hotfix Development (2 hours)
- [ ] Apply code fix (memory_monitor.py + indexer.py)
- [ ] Add integration test (test_memory_backpressure.py)
- [ ] Run all tests locally
- [ ] Manual testing with low memory cap
- [ ] Update version to 2.5.19
- [ ] Update CHANGELOG.md
- [ ] Commit and tag

### Release (30 minutes)
- [ ] Build distribution: `python -m build`
- [ ] Upload to PyPI: `python -m twine upload dist/*`
- [ ] Verify PyPI shows 2.5.19
- [ ] Test install: `pip install mcp-vector-search==2.5.19`

### Verification (30 minutes)
- [ ] Deploy to AWS GPU instance
- [ ] Run full reindex (31k files)
- [ ] Monitor for 10 minutes (no hangs)
- [ ] Stress test with low memory cap
- [ ] Verify pipeline parallelism works

### Communication (15 minutes)
- [ ] Post internal status update
- [ ] Draft user announcement
- [ ] Update GitHub issues
- [ ] Post to mcp-vector-search users channel
- [ ] Mark incident as resolved

### Prevention (30 minutes)
- [ ] Add async lint rules to ruff config
- [ ] Add integration test to CI/CD
- [ ] Add pre-commit hook for async safety
- [ ] Document async safety guidelines

---

## 🆘 ROLLBACK PLAN

If 2.5.19 has issues:

1. **Immediate Rollback:**
   ```bash
   pip install mcp-vector-search==2.2.40
   ```

2. **Yank 2.5.19 from PyPI:**
   ```bash
   python -m twine yank mcp-vector-search 2.5.19 -m "Hotfix had issues, use 2.2.40"
   ```

3. **Notify Users:**
   > 2.5.19 has been yanked. Please use 2.2.40 until we release a new hotfix.

---

## 📞 CONTACTS

- **Primary:** Bob Matsuoka (on-call)
- **Backup:** Claude Sonnet 4.5 (AI assistant)
- **AWS Access:** EC2 Instance Connect (i-02d8caca52b2e209b)
- **PyPI:** twine credentials in ~/.pypirc

---

**Last Updated:** 2026-02-18 00:15:00 UTC
**Incident Status:** IN PROGRESS - Hotfix Development
**Next Update:** T+30 minutes
