# Deduplication Fix Summary

## Problem
The indexer was creating massive duplication in `vectors.lance`:
- **Before fix**: 30,379 total rows, 5,262 unique chunk_ids (5.8x duplication)
- Root cause: `add_vectors()` used append mode without checking for existing chunk_ids
- Every incremental index or reindex blindly re-added vectors

## Solution
Implemented deduplication in `vectors_backend.py`:

### Key Changes
1. **Added `_delete_chunk_ids()` method** (lines 177-224):
   - Deletes existing vectors with same chunk_ids before appending
   - Batched deletes (500 chunks/batch) to avoid SQL length limits
   - Proper escaping for single quotes in chunk_ids
   - Non-fatal (logs warnings but continues on error)

2. **Updated `add_vectors()` method** (lines 300-484):
   - Extract chunk_ids before appending
   - Call `_delete_chunk_ids()` to remove duplicates
   - Then append new vectors
   - Applied to both code paths (table open and table append)

### Code Flow
```python
# Before appending vectors:
chunk_ids_to_add = [row["chunk_id"] for row in normalized_vectors]

# Delete existing vectors with same chunk_ids
self._delete_chunk_ids(chunk_ids_to_add)

# Now safe to append without creating duplicates
self._table.add(pa_table, mode="append")
```

## Testing

### Unit Tests (test_dedup_fix.py)
✅ **Test 1**: Adding same chunk_id twice → Only 1 row in table
✅ **Test 2**: Batch deduplication → 10 chunks added twice = 10 rows

### Integration Test
- Incremental index of 17 new chunks
- **Result**: 17 rows added (no duplicates created)
- **Before**: Would have created 17 duplicates (30,179 → 30,196 with same 5,300 unique)
- **After**: 30,179 → 30,196 with 5,317 unique (17 new unique chunks)

## Impact

### Immediate Benefits
- ✅ Prevents future duplication in incremental indexes
- ✅ Prevents duplication in `--force` reindexes
- ✅ No performance impact (delete is fast with indexed chunk_id)

### Historical Duplicates
- Still have 5,081 chunks with duplicates (created before fix)
- These are non-critical (search still works, just with redundancy)
- Can be cleaned up with: `mcp-vector-search index --force`

### No Version Bump
- Internal fix, no API changes
- No need to publish or bump version
- Just commit and deploy

## Files Changed
- `src/mcp_vector_search/core/vectors_backend.py` (+62 lines)
  - Added `_delete_chunk_ids()` method
  - Updated `add_vectors()` to call dedup before append
  - Proper escaping and batching

## Commit
```
fix: prevent duplicate vectors in vectors.lance during indexing

Problem: 5.8x duplication (30,379 rows, 5,262 unique chunk_ids)
Solution: Delete existing chunk_ids before appending new vectors
Testing: Unit tests + integration test confirmed dedup working
Impact: Prevents future duplication, historical requires --force reindex
```

## Verification
```bash
# Check current state
uv run python -c "
import lancedb
db = lancedb.connect('.mcp-vector-search/lance/')
t = db.open_table('vectors')
df = t.to_pandas()
print(f'Total: {len(df)}, Unique: {df[\"chunk_id\"].nunique()}, Factor: {len(df)/df[\"chunk_id\"].nunique():.2f}x')
"

# Run incremental index
uv run mcp-vector-search index

# Verify no new duplicates created
# Total should increase by number of new unique chunks only
```

## Notes
- LanceDB `delete()` is non-fatal and fast
- Batched deletes prevent SQL length limits
- Proper quote escaping prevents SQL injection
- Debug logging for dedup operations (not visible in normal output)
