# Memory Cap Fix - Critical Production Issue

## Problem Summary
Memory cap not enforced - process reached 114GB when configured for 25GB cap (4.5x over limit).

## Root Causes Identified

### Bug #1: Infrequent Memory Checks During Chunking
**Location:** `src/mcp_vector_search/core/indexer.py:490`
```python
if idx > 0 and idx % 100 == 0:  # Only checks every 100 files!
    is_ok, usage_pct, status = self.memory_monitor.check_memory_limit()
```

**Impact:**
- Large files can accumulate 5-10GB before first check
- With 100 files @ 50MB each = 5GB unmonitored growth
- No protection during the critical accumulation phase

**Fix:** Check memory EVERY file, not every 100 files:
```python
# Check memory before processing each file
is_ok, usage_pct, status = self.memory_monitor.check_memory_limit()
if not is_ok:
    await self.memory_monitor.wait_for_memory_available(
        target_pct=self.memory_monitor.warn_threshold
    )
```

---

### Bug #2: Memory Check Before Data Loading (Wrong Order)
**Location:** `src/mcp_vector_search/core/indexer.py:618-641`
```python
# WRONG: Check first, THEN load data
is_ok, usage_pct, status = self.memory_monitor.check_memory_limit()
if not is_ok:
    await self.memory_monitor.wait_for_memory_available(...)

# Load batch AFTER check (too late!)
pending = await self.chunks_backend.get_pending_chunks(batch_size)
```

**Impact:**
- Check passes at 20GB
- Immediately fetch 2GB batch
- Now at 22GB without another check
- Next iteration: check at 22GB, fetch 2GB more → 24GB
- Race condition allows gradual creep past limit

**Fix:** Check AFTER loading data, when memory is at peak:
```python
# Load batch first
pending = await self.chunks_backend.get_pending_chunks(batch_size)
if not pending:
    break

# NOW check memory after loading
is_ok, usage_pct, status = self.memory_monitor.check_memory_limit()
if not is_ok:
    # Memory exceeded - wait before embedding
    await self.memory_monitor.wait_for_memory_available(
        target_pct=self.memory_monitor.warn_threshold
    )
```

---

### Bug #3: No Memory Check During Embedding Generation
**Location:** `src/mcp_vector_search/core/indexer.py:652-659`
```python
contents = [c["content"] for c in pending]
# HUGE memory spike here - no check!
vectors = self.database._embedding_function(contents)
```

**Impact:**
- Embedding 1000 chunks can spike memory by 5-10GB
- No monitoring during the largest memory allocation
- This is likely the main cause of 114GB explosion

**Fix:** Add memory check + reduced batch retry:
```python
try:
    contents = [c["content"] for c in pending]

    # Check memory before expensive embedding operation
    is_ok, usage_pct, status = self.memory_monitor.check_memory_limit()
    if not is_ok:
        # Too much memory - reduce batch and retry
        logger.warning(
            f"Memory limit exceeded before embedding "
            f"({usage_pct * 100:.1f}%), reducing batch and waiting..."
        )

        # Return chunks to pending state
        await self.chunks_backend.mark_chunks_pending(chunk_ids)

        # Wait for memory to free up
        await self.memory_monitor.wait_for_memory_available(
            target_pct=self.memory_monitor.warn_threshold
        )

        # Reduce batch size dramatically
        batch_size = max(100, batch_size // 4)
        continue  # Retry with smaller batch

    # Generate embeddings
    vectors = self.database._embedding_function(contents)

except Exception as e:
    # Handle embedding errors...
```

---

## Additional Issues

### Issue #4: Batch Size Reduction Not Aggressive Enough
**Location:** `src/mcp_vector_search/core/memory_monitor.py:196-209`

Current reduction:
- At 90% memory: reduce to 25% of batch (too late)
- At 80% memory: reduce to 50% of batch (too late)

**Fix:** More aggressive reduction at lower thresholds:
```python
def get_adjusted_batch_size(
    self, current_batch_size: int, min_batch_size: int = 100
) -> int:
    usage_pct = self.get_memory_usage_pct()

    if usage_pct >= 1.0:
        # At limit - minimum batch
        return min_batch_size
    elif usage_pct >= 0.85:  # Changed from 0.9
        # Very high - reduce to 20%
        return max(min_batch_size, current_batch_size // 5)
    elif usage_pct >= 0.70:  # Changed from 0.8
        # High - reduce to 50%
        return max(min_batch_size, current_batch_size // 2)
    else:
        return current_batch_size
```

---

### Issue #5: No Memory Monitoring During LanceDB Operations
**Location:** `src/mcp_vector_search/core/lancedb_backend.py`

LanceDB operations (add_vectors, search) can spike memory significantly with no monitoring.

**Fix:** Add memory monitor to LanceDB backend:
```python
class LanceDBBackend:
    def __init__(self, ..., memory_monitor: MemoryMonitor | None = None):
        self.memory_monitor = memory_monitor

    async def add_vectors(self, chunks, vectors, metadata_list):
        # Check memory before large write
        if self.memory_monitor:
            is_ok, _, _ = self.memory_monitor.check_memory_limit()
            if not is_ok:
                # Split into smaller batches
                batch_size = len(chunks) // 2
                ...
```

---

## Implementation Priority

### CRITICAL (Fix Immediately):
1. **Bug #3**: Add memory check before embedding generation
2. **Bug #2**: Move memory check to AFTER data loading
3. **Bug #1**: Check memory every file, not every 100 files

### HIGH (Fix Soon):
4. **Issue #4**: More aggressive batch size reduction
5. **Issue #5**: Add memory monitoring to LanceDB operations

### MEDIUM (Improve Later):
6. Add memory profiling to identify other hotspots
7. Add memory usage to status endpoint
8. Add memory alerts to monitoring dashboard

---

## Immediate Patch Code

```python
# PATCH 1: indexer.py line 488-498
# BEFORE:
for idx, (file_path, rel_path, file_hash) in enumerate(files_to_process):
    if idx > 0 and idx % 100 == 0:
        is_ok, usage_pct, status = self.memory_monitor.check_memory_limit()
        if not is_ok:
            await self.memory_monitor.wait_for_memory_available(...)

# AFTER:
for idx, (file_path, rel_path, file_hash) in enumerate(files_to_process):
    # Check memory EVERY file (not every 100)
    is_ok, usage_pct, status = self.memory_monitor.check_memory_limit()
    if not is_ok:
        logger.warning(
            f"Memory limit exceeded during chunking "
            f"({usage_pct * 100:.1f}%), waiting..."
        )
        await self.memory_monitor.wait_for_memory_available(
            target_pct=self.memory_monitor.warn_threshold
        )
```

```python
# PATCH 2: indexer.py line 616-640
# BEFORE:
while True:
    is_ok, usage_pct, status = self.memory_monitor.check_memory_limit()
    if not is_ok:
        await self.memory_monitor.wait_for_memory_available(...)

    pending = await self.chunks_backend.get_pending_chunks(batch_size)

# AFTER:
while True:
    # Fetch batch FIRST
    pending = await self.chunks_backend.get_pending_chunks(batch_size)
    if not pending:
        logger.info("No more pending chunks to embed")
        break

    # THEN check memory after loading
    is_ok, usage_pct, status = self.memory_monitor.check_memory_limit()
    if not is_ok:
        logger.warning(
            f"Memory limit exceeded after loading batch "
            f"({usage_pct * 100:.1f}%), waiting..."
        )
        await self.memory_monitor.wait_for_memory_available(
            target_pct=self.memory_monitor.warn_threshold
        )
```

```python
# PATCH 3: indexer.py line 650-660
# BEFORE:
contents = [c["content"] for c in pending]
vectors = self.database._embedding_function(contents)

# AFTER:
contents = [c["content"] for c in pending]

# Critical: Check before expensive embedding operation
is_ok, usage_pct, status = self.memory_monitor.check_memory_limit()
if not is_ok:
    logger.error(
        f"Memory limit exceeded before embedding "
        f"({usage_pct * 100:.1f}% of {self.memory_monitor.max_memory_gb}GB). "
        f"Reducing batch size and retrying..."
    )

    # Return chunks to pending state
    await self.chunks_backend.mark_chunks_pending(chunk_ids)

    # Wait for memory to free
    await self.memory_monitor.wait_for_memory_available(
        target_pct=self.memory_monitor.warn_threshold
    )

    # Dramatically reduce batch size
    batch_size = max(100, batch_size // 4)
    logger.info(f"Reduced batch size to {batch_size} due to memory pressure")
    continue  # Skip to next iteration with smaller batch

# Now safe to generate embeddings
vectors = self.database._embedding_function(contents)
```

---

## Testing & Verification

### 1. Set Memory Cap
```bash
export MCP_VECTOR_SEARCH_MAX_MEMORY_GB=25
```

### 2. Monitor During Indexing
```bash
# In separate terminal, watch memory usage
watch -n 1 'ps aux | grep mcp-vector-search | grep -v grep | awk "{print \$6/1024/1024 \" GB\"}"'
```

### 3. Check Logs
Look for these log messages:
- "Memory monitor initialized: 25.0GB cap"
- "Memory limit exceeded during chunking"
- "Memory limit exceeded before embedding"
- "Reduced batch size to X due to memory pressure"

### 4. Expected Behavior
- Memory should NEVER exceed 25GB
- Should see frequent "waiting for memory" messages if approaching limit
- Batch size should reduce under pressure
- Indexing will be slower but stable

---

## Version Update

After applying fixes, bump version to **2.5.20** with changelog:
```
2.5.20 (2025-02-18)
- CRITICAL FIX: Memory cap enforcement now works correctly
- Fix: Check memory every file, not every 100 files during chunking
- Fix: Check memory after loading data, not before
- Fix: Add memory check before expensive embedding operation
- Fix: More aggressive batch size reduction under memory pressure
```

---

## Long-Term Improvements

1. **Add memory profiling mode**: Track exactly where memory spikes occur
2. **Add memory regression tests**: Automated tests that verify cap is enforced
3. **Add memory dashboard**: Real-time memory usage visualization
4. **Add memory alerts**: Trigger alerts when approaching 80% of cap
5. **Memory optimization**: Investigate why embeddings use so much memory
6. **Streaming embeddings**: Process embeddings in smaller micro-batches
