# Force Reindex LanceDB Error Fix

## Problem
AWS instance was hitting LanceDB errors during force reindex (`mcp-vector-search index -f`):
```
lance error: Not found: data/repos/.mcp-vector-search/code_search.lance/data/...
```

The v2.3.51 "skip delete" optimization wasn't preventing all delete calls, AND there was a code path still calling delete that crashed when files didn't exist.

## Root Causes

### 1. Incomplete skip_delete Propagation
The `_atomic_rebuild_active` flag was being set correctly, but there were **multiple code paths** that still called delete operations without checking the flag:

- `index_file()` line 1367: called `delete_by_file()` without checking
- `_process_file_batch()` line 1189: called `delete_by_file()` without checking
- `_parse_and_prepare_file()` line 1122: called `delete_by_file()` without checking
- `_phase1_chunk_files()` line 446: called `delete_file_chunks()` without checking
- `index_files_with_progress()` line 1847: called `delete_file_chunks()` without checking
- `_process_file_batch()` line 1276: called `delete_file_chunks()` without checking

### 2. Delete Operations Not Graceful
Even when delete was called, it should not crash when files don't exist. The delete methods in three backends were crashing with "Not found" errors:

- `chunks_backend.delete_file_chunks()` - no error handling for missing files
- `lancedb_backend.delete_by_file()` - no error handling for missing files
- `database.delete_by_file()` (ChromaDB) - no error handling for missing files

## Solution

### Part 1: Comprehensive skip_delete Checking
Added `self._atomic_rebuild_active` checks to ALL code paths that call delete operations:

**Files modified:**
- `src/mcp_vector_search/core/indexer.py` (7 locations)

**Pattern applied:**
```python
# Before
await self.database.delete_by_file(file_path)

# After
if not self._atomic_rebuild_active:
    await self.database.delete_by_file(file_path)
```

### Part 2: Graceful Delete Error Handling
Added "Not found" error handling to all three delete methods:

**Files modified:**
1. `src/mcp_vector_search/core/chunks_backend.py` - `delete_file_chunks()`
2. `src/mcp_vector_search/core/lancedb_backend.py` - `delete_by_file()`
3. `src/mcp_vector_search/core/database.py` - `delete_by_file()` (ChromaDB)

**Pattern applied:**
```python
except Exception as e:
    # Handle LanceDB "Not found" errors gracefully (file not in index)
    error_msg = str(e).lower()
    if "not found" in error_msg:
        logger.debug(f"No chunks to delete for {file_path} (not in index)")
        return 0
    logger.error(f"Failed to delete chunks for {file_path}: {e}")
    raise DatabaseError(f"Failed to delete chunks: {e}") from e
```

## Changes Summary

### Modified Files (4 total):
1. **`src/mcp_vector_search/core/indexer.py`** - 7 edits
   - Added `self._atomic_rebuild_active` checks to all delete call sites

2. **`src/mcp_vector_search/core/chunks_backend.py`** - 1 edit
   - Added graceful "Not found" error handling to `delete_file_chunks()`

3. **`src/mcp_vector_search/core/lancedb_backend.py`** - 1 edit
   - Added graceful "Not found" error handling to `delete_by_file()`

4. **`src/mcp_vector_search/core/database.py`** - 1 edit
   - Added graceful "Not found" error handling to `delete_by_file()`

## Testing

### Test Script
Created `test_force_reindex.sh` to verify the fix:
```bash
./test_force_reindex.sh
```

This script:
1. Creates a test directory with sample Python files
2. Runs initial index
3. Runs force reindex (should not crash)
4. Checks output for LanceDB errors
5. Reports success/failure

### Manual Testing
```bash
# Install the updated version
cd /Users/masa/Projects/mcp-vector-search
pip install -e .

# Test on a real codebase
cd /path/to/your/project
mcp-vector-search index -f --limit 50

# Should complete without LanceDB "Not found" errors
```

## Expected Behavior

### Before Fix
```
❌ Force reindex would crash with:
lance error: Not found: data/repos/.mcp-vector-search/code_search.lance/data/...
Error: Failed to delete chunks for file.py
```

### After Fix
```
✅ Force reindex completes successfully:
🔄 Atomic rebuild: preparing new database directories...
✓ New database directories prepared
📄 Phase 1: Chunking 1000 files...
✓ Phase 1 complete: 1000 files processed, 15000 chunks created
🧠 Phase 2: Embedding pending chunks...
✓ Phase 2 complete: 15000 chunks embedded
✓ Atomic rebuild complete: databases switched successfully
```

Delete operations that encounter missing files now log debug messages instead of crashing:
```
DEBUG: No chunks to delete for src/file.py (not in index)
```

## Deployment

After these changes are merged:
1. Bump version to **v2.3.52**
2. AWS instance should be able to run `mcp-vector-search index -f` without errors
3. All delete operations will gracefully handle missing files

## Related Issues
- Fixes LanceDB "Not found" errors during force reindex
- Improves robustness of atomic rebuild feature (v2.3.51)
- Prevents database corruption from failed delete operations

## Verification Checklist
- [x] Added `_atomic_rebuild_active` checks to all delete call sites
- [x] Added graceful error handling to `chunks_backend.delete_file_chunks()`
- [x] Added graceful error handling to `lancedb_backend.delete_by_file()`
- [x] Added graceful error handling to `database.delete_by_file()`
- [x] Created test script to verify fix
- [x] Documented all changes and expected behavior
