# Optimization: Skip Delete Operations on Force Reindex

## Problem
When running `mcp-vector-search index -f` (force reindex), the indexer builds to a **FRESH** temp database (`.new` suffix) but still calls `delete_by_file()` for every file. This is wasteful overhead:

- **CTO codebase**: 25,182 files × delete operation = ~25k unnecessary database queries
- Each delete queries the database to find existing chunks (there are none in fresh DB)
- This adds significant latency, especially for large codebases

## Solution
Skip `delete_by_file()` calls when building into a fresh database during force reindex.

### Implementation Details

#### 1. Added Instance Variable to Track Atomic Rebuild State
```python
# src/mcp_vector_search/core/indexer.py:232
self._atomic_rebuild_active: bool = False
```

#### 2. Set Flag During Atomic Rebuild
```python
# src/mcp_vector_search/core/indexer.py:721
async def _atomic_rebuild_databases(self, force: bool = False) -> bool:
    # ... setup new databases with .new suffix ...
    logger.info("✓ New database directories prepared")
    # Set flag to skip delete operations (building into fresh database)
    self._atomic_rebuild_active = True
    return True
```

#### 3. Reset Flag After Finalization
```python
# src/mcp_vector_search/core/indexer.py:797, 817
async def _finalize_atomic_rebuild(self) -> None:
    # ... atomic switch from .new to final databases ...
    logger.info("✓ Atomic rebuild complete: databases switched successfully")
    # Reset flag after successful finalization
    self._atomic_rebuild_active = False
```

#### 4. Added skip_delete Parameter to Methods
Three methods updated to accept `skip_delete` parameter:
- `_parse_and_prepare_file(file_path, force_reindex, skip_delete=False)`
- `_process_file_batch(file_paths, force_reindex, skip_delete=False)`
- `index_file(file_path, force_reindex, skip_delete=False)`

#### 5. Conditional Delete Operations
Wrapped `delete_by_file()` calls in conditional checks:

**Location 1**: `_parse_and_prepare_file()` (line ~1121)
```python
# Skip delete on force reindex with fresh database - no existing chunks to delete
# This optimization eliminates ~25k unnecessary database queries for large codebases
if not skip_delete:
    await self.database.delete_by_file(file_path)
```

**Location 2**: `_process_file_batch()` (line ~1188)
```python
# Only schedule deletion if not building fresh database
if not skip_delete:
    delete_task = asyncio.create_task(self.database.delete_by_file(file_path))
    delete_tasks.append(delete_task)
```

**Location 3**: `index_file()` (line ~1365)
```python
# Skip delete on force reindex with fresh database - no existing chunks to delete
if not skip_delete:
    await self.database.delete_by_file(file_path)
```

#### 6. Pass Flag in Batch Processing
```python
# src/mcp_vector_search/core/indexer.py:1773
task = asyncio.create_task(
    self._parse_and_prepare_file(
        file_path, force_reindex, skip_delete=self._atomic_rebuild_active
    )
)
```

## Flow Diagram

### Before Optimization
```
index -f → _atomic_rebuild_databases() → creates .new database
    ↓
for each file:
    ↓
    delete_by_file(file)  ← WASTEFUL: queries empty database
    ↓
    parse_file()
    ↓
    insert_chunks()
```

### After Optimization
```
index -f → _atomic_rebuild_databases() → creates .new database
           ↓
           set _atomic_rebuild_active = True
    ↓
for each file:
    ↓
    if not skip_delete:  ← SKIPPED when building fresh
        delete_by_file(file)
    ↓
    parse_file()
    ↓
    insert_chunks()
```

## Expected Impact

### Before (Force Reindex with 25,182 files)
- 25,182 `delete_by_file()` calls to FRESH database
- Each call queries database (which is empty)
- Significant overhead from unnecessary queries

### After (Force Reindex with 25,182 files)
- **0** `delete_by_file()` calls (skip_delete=True)
- Database is fresh, no existing chunks to delete
- Eliminates ~25k unnecessary database operations

### Performance Improvement
- **Reduced per-file overhead**: No database query for empty delete
- **Faster force reindex**: Especially noticeable with large codebases (10k+ files)
- **No functional change**: Deletes only matter for incremental updates

## Testing

### Manual Testing
```bash
cd /Users/masa/Projects/mcp-vector-search

# Test force reindex (should skip deletes)
mcp-vector-search index -f --limit 100

# Test incremental index (should NOT skip deletes)
# Modify a file and re-index
mcp-vector-search index
```

### Expected Behavior
1. **Force reindex** (`index -f`):
   - `_atomic_rebuild_active = True`
   - All `delete_by_file()` calls skipped
   - Faster indexing with reduced overhead

2. **Incremental reindex** (default):
   - `_atomic_rebuild_active = False`
   - `delete_by_file()` calls still execute
   - Ensures consistency for incremental updates

### Verification Script
Run the included verification script:
```bash
python test_skip_delete_optimization.py
```

## Backward Compatibility
✅ **Fully backward compatible**:
- `skip_delete` parameter defaults to `False`
- Existing code (watcher, tests) continues to work unchanged
- Only force reindex benefits from optimization
- Incremental updates still delete old chunks as before

## Code Changes Summary
1. Added `_atomic_rebuild_active` instance variable
2. Set flag in `_atomic_rebuild_databases()` (1 location)
3. Reset flag in `_finalize_atomic_rebuild()` (3 locations: success, rollback, exception)
4. Added `skip_delete` parameter to 3 methods
5. Wrapped 3 `delete_by_file()` calls in conditionals
6. Passed flag in `index_files_with_progress()`

**Total lines changed**: ~20 lines
**Files modified**: 1 (`src/mcp_vector_search/core/indexer.py`)

## Related Issues
- Addresses force reindex performance for large codebases
- Optimizes atomic rebuild workflow (`.new` database strategy)
- No impact on incremental indexing or watcher functionality
