# Critical Search Bug Fix - Schema Mismatch Causing Index Deletion

## Problem Summary

The search commands were incorrectly opening `vectors.lance` using `LanceVectorDatabase` with the chunks schema, causing a schema mismatch that triggered overly-aggressive corruption recovery logic, which deleted the vectors table.

## Root Cause

### Issue 1: Wrong Collection Name in Search Commands

**Files affected:**
- `src/mcp_vector_search/cli/commands/search.py` (3 occurrences)
- `src/mcp_vector_search/cli/commands/status.py` (2 occurrences)

**The bug:**
```python
database = create_database(
    persist_directory=config.index_path / "lance",
    embedding_function=embedding_function,
    collection_name="vectors",  # BUG: Opens vectors.lance with chunks schema
)
```

**Why this failed:**
1. `create_database()` with `collection_name="vectors"` creates a `LanceVectorDatabase` instance
2. `LanceVectorDatabase` expects the **chunks schema** (defined in `LANCEDB_SCHEMA`)
3. `vectors.lance` uses the **vectors schema** (defined in `VECTORS_SCHEMA` in vectors_backend.py)
4. Schema mismatch triggers "not found" errors when accessing fields
5. Overly-aggressive `_is_corruption_error()` check treats this as corruption
6. Corruption handler deletes `vectors.lance` table

**The fix:**
```python
database = create_database(
    persist_directory=config.index_path / "lance",
    embedding_function=embedding_function,
    collection_name="chunks",  # FIXED: Opens chunks.lance (correct schema)
)
```

**Why this works:**
- Search engine (`SemanticSearchEngine`) has built-in two-phase architecture detection
- In `search.py` lines 127-129, it checks for `VectorsBackend` availability
- If `vectors.lance` exists, it uses `VectorsBackend` for search (two-phase)
- Otherwise, it falls back to `LanceVectorDatabase` on chunks.lance (single-phase)
- By opening `chunks.lance` as the base database, we let the search engine decide the architecture

### Issue 2: Overly-Aggressive Corruption Detection

**Files affected:**
- `src/mcp_vector_search/core/lancedb_backend.py` (line 231-242)
- `src/mcp_vector_search/core/vectors_backend.py` (line 112-123)

**The bug:**
```python
def _is_corruption_error(self, error: Exception) -> bool:
    error_msg = str(error).lower()
    # BUG: Too broad - catches schema mismatches, not just corruption
    return "not found" in error_msg and ".lance" in error_msg
```

**Why this failed:**
- This check triggers on **any** "not found" error mentioning `.lance`
- Schema mismatches generate "field not found" errors
- These are NOT corruption - they're configuration errors
- The handler incorrectly deleted the table

**The fix:**
```python
def _is_corruption_error(self, error: Exception) -> bool:
    error_msg = str(error).lower()

    # Check for genuine corruption: missing data fragment files
    is_fragment_error = (
        ("not found" in error_msg or "no such file" in error_msg)
        and ("fragment" in error_msg or "data/" in error_msg)
    )

    # Schema errors are NOT corruption
    is_schema_error = (
        "schema" in error_msg
        or "field" in error_msg
        or "column" in error_msg
        or "type mismatch" in error_msg
    )

    # Only treat as corruption if it's a fragment error AND NOT a schema error
    return is_fragment_error and not is_schema_error
```

**Why this works:**
- Distinguishes between **actual corruption** (missing data fragments) and **schema mismatches**
- Data fragments are named like `data/abc123.lance` and referenced in "fragment not found" errors
- Schema errors mention "field", "column", "schema", "type mismatch"
- Only deletes tables for genuine data corruption, not operational errors

## Architecture Notes

### Two-Phase Architecture (New)

**Phase 1: Chunks Storage** (chunks.lance)
- Stores parsed code chunks with metadata
- Schema: `LANCEDB_SCHEMA` (includes all chunk fields, no vectors)
- Managed by: `LanceVectorDatabase` with `collection_name="chunks"`

**Phase 2: Vectors Storage** (vectors.lance)
- Stores embedded vectors with denormalized metadata for fast search
- Schema: `VECTORS_SCHEMA` (includes vectors + minimal metadata)
- Managed by: `VectorsBackend` (separate class)

**Search Flow:**
1. `SemanticSearchEngine` checks if `vectors.lance` exists (`_check_vectors_backend()`)
2. If yes: Use `VectorsBackend` for two-phase search (faster, no JOINs)
3. If no: Use `LanceVectorDatabase` on chunks.lance (legacy single-phase)

### Single-Phase Architecture (Legacy)

**Single Table: chunks.lance**
- Stores both chunks AND vectors in one table
- Schema: `LANCEDB_SCHEMA` (includes vectors field)
- Managed by: `LanceVectorDatabase` with `collection_name="chunks"`

**This is still supported for backward compatibility.**

## Files Changed

1. **src/mcp_vector_search/cli/commands/search.py**
   - Changed `collection_name="vectors"` → `collection_name="chunks"` (3 locations)
   - Lines: 394, 740, 850

2. **src/mcp_vector_search/cli/commands/status.py**
   - Changed `collection_name="vectors"` → `collection_name="chunks"` (2 locations)
   - Lines: 235, 526

3. **src/mcp_vector_search/core/lancedb_backend.py**
   - Improved `_is_corruption_error()` to distinguish corruption from schema errors
   - Line: 231-256

4. **src/mcp_vector_search/core/vectors_backend.py**
   - Improved `_is_corruption_error()` to distinguish corruption from schema errors
   - Line: 112-143

## Impact

### Before Fix
- ❌ Search commands deleted `vectors.lance` on first use
- ❌ Users lost embedded vectors and had to reindex
- ❌ No error message indicating the real problem
- ❌ Schema mismatches treated as corruption

### After Fix
- ✅ Search commands correctly open `chunks.lance`
- ✅ Two-phase architecture automatically detected and used when available
- ✅ Schema mismatches no longer trigger corruption recovery
- ✅ Corruption recovery only deletes tables with genuine data loss
- ✅ Clear separation between operational errors and corruption

## Testing Recommendations

1. **Test single-phase search** (chunks.lance only):
   ```bash
   mcp-vector-search search "query" --project /path/to/project
   ```

2. **Test two-phase search** (both chunks.lance and vectors.lance):
   ```bash
   # Ensure vectors.lance exists from a recent index
   mcp-vector-search search "query" --project /path/to/project
   ```

3. **Test status command**:
   ```bash
   mcp-vector-search status --project /path/to/project
   ```

4. **Verify no vectors.lance deletion**:
   ```bash
   # Before search
   ls -lah .mcp-vector-search/lance/

   # Run search
   mcp-vector-search search "test query"

   # After search - vectors.lance should still exist
   ls -lah .mcp-vector-search/lance/
   ```

## Recovery for Users Affected by Bug

If users already lost their `vectors.lance` table:

1. The chunks.lance table should still be intact (contains parsed code)
2. Re-running `mcp-vector-search index` will:
   - Read chunks from chunks.lance
   - Regenerate embeddings
   - Rebuild vectors.lance table
3. No data loss - just need to re-embed (which is faster than re-parsing)

## Preventive Measures

1. **Code review**: Ensure all `create_database()` calls use correct `collection_name`
2. **Testing**: Add integration tests for search CLI commands
3. **Monitoring**: Log when corruption recovery is triggered (currently done)
4. **Documentation**: Update docs to clarify two-phase architecture

## Related Issues

- User reported search commands hanging or deleting index
- Likely caused by this schema mismatch bug
- Should be resolved by this fix
