# Index Metadata Architecture

## Overview

This document clarifies the architecture of metadata storage in mcp-vector-search, addressing common misconceptions about what `index_metadata.json` contains and how statistics are computed.

## Metadata Storage Architecture

### 1. `index_metadata.json` - File Modification Tracking

**Purpose**: Enable incremental indexing by tracking file modification times.

**Location**: `.mcp-vector-search/index_metadata.json`

**Structure**:
```json
{
  "index_version": "2.5.49",
  "indexed_at": "2026-02-20T02:51:23.367557+00:00",
  "file_mtimes": {
    "/path/to/file1.py": 1770347648.900154,
    "/path/to/file2.ts": 1770347650.123456,
    ...
  }
}
```

**Fields**:
- `index_version`: Version of mcp-vector-search that created the index
- `indexed_at`: ISO timestamp of last indexing run
- `file_mtimes`: Dictionary mapping absolute file paths to Unix timestamps (modification times)

**Usage**:
- Check if files need reindexing (compare mtime with stored value)
- Version compatibility checks (recommend reindex on major/minor version changes)
- Clean up stale entries for deleted files

**NOT STORED HERE**:
- ❌ Total file count
- ❌ Total chunk count
- ❌ Language distribution
- ❌ Indexing statistics

### 2. Lance Databases - Source of Truth for Statistics

All indexing statistics are computed **on-demand** by querying the Lance databases directly.

#### `chunks.lance` - Phase 1 Storage

**Purpose**: Store parsed code chunks before embedding.

**Statistics Method**: `chunks_backend.get_stats()`

**Returns**:
```python
{
    "total": 65440,              # Total chunks
    "pending": 0,                # Awaiting embedding
    "processing": 0,             # Currently embedding
    "complete": 65440,           # Embedded
    "error": 0,                  # Failed
    "files": 684,                # Unique file count
    "languages": {               # Language distribution
        "python": 30000,
        "typescript": 20000,
        "javascript": 15440
    }
}
```

**Implementation**:
```python
# Queries chunks.lance directly - no file parsing needed
scanner = self._table.to_lance().scanner(
    columns=["embedding_status", "file_path", "language"]
)
result = scanner.to_table()
df = result.to_pandas()

# Compute stats in-memory from database rows
status_counts = df["embedding_status"].value_counts().to_dict()
file_count = df["file_path"].nunique()
language_counts = df["language"].value_counts().to_dict()
```

#### `vectors.lance` - Phase 2 Storage

**Purpose**: Store embedded vectors for semantic search.

**Statistics Method**: `vectors_backend.get_stats()`

**Returns**:
```python
{
    "total": 65440,              # Total vectors
    "files": 684,                # Unique file count
    "chunk_types": {             # Distribution by type
        "function": 25000,
        "class": 15000,
        "method": 20000,
        "module": 5440
    }
}
```

### 3. `progress.json` - Real-Time Indexing Progress

**Purpose**: Track progress during active indexing operations.

**Location**: `.mcp-vector-search/progress.json`

**Structure**:
```json
{
  "phase": "chunking",
  "chunking": {
    "total_files": 727,
    "processed_files": 512,
    "total_chunks": 2022
  },
  "embedding": {
    "total_chunks": 2022,
    "embedded_chunks": 2022
  },
  "kg_build": {
    "total_chunks": 0,
    "processed_chunks": 0,
    "entities": 0,
    "relations": 0
  },
  "started_at": 1771555873.389448,
  "updated_at": 1771555969.47298
}
```

**Usage**:
- Show progress bars during indexing
- Resume interrupted indexing operations
- **NOT USED** by search or status commands

## How Commands Get Statistics

### `mcp-vector-search status`

**Flow**:
1. Load config from `.mcp-vector-search/config.json`
2. Initialize `ChunksBackend` with database path
3. Call `chunks_backend.get_stats()` → queries `chunks.lance` directly
4. Display statistics from database query results

**Code Path**:
```python
# status.py
index_stats = await indexer.get_indexing_stats(db_stats=db_stats)

# indexer.py
chunks_stats = await self.chunks_backend.get_stats()
return {
    "total_files": chunks_stats["files"],
    "total_chunks": chunks_stats["total"],
    "languages": chunks_stats["languages"],
    ...
}
```

**Does NOT read**: `index_metadata.json` (except for version info)

### `mcp-vector-search search`

**Flow**:
1. Load config
2. Initialize `VectorsBackend` with database path
3. Generate query embedding
4. Call `vectors_backend.search(query_vector)` → queries `vectors.lance` directly
5. Return results with metadata from database rows

**Code Path**:
```python
# search.py
results = await self.database.similarity_search(query, limit, filters)

# vectors_backend.py
results = self._table.search(query_vector).limit(limit).to_list()
```

**Does NOT read**: `index_metadata.json` at all

## Why This Architecture?

### Benefits

1. **Single Source of Truth**: Lance databases are the authoritative source
2. **No Sync Issues**: Statistics are always accurate (computed on-demand)
3. **Crash Recovery**: Databases persist state even if indexing crashes
4. **Incremental Updates**: `index_metadata.json` only tracks what changed

### Trade-offs

1. **Compute Cost**: Stats require full table scan (acceptable for reporting)
2. **No Caching**: Every `status` call queries database (fast for Lance)
3. **Large DB Risk**: Very large databases (>500MB) may hang on stats queries

## Common Misconceptions

### ❌ "index_metadata.json should have total_files/total_chunks"
**Reality**: Those stats are computed from Lance databases, not stored in JSON.

### ❌ "Search fails because index_metadata.json is missing stats"
**Reality**: Search queries `vectors.lance` directly and never reads `index_metadata.json`.

### ❌ "status command reads from index_metadata.json"
**Reality**: Status queries `chunks.lance` for stats. Only reads `index_metadata.json` for version info.

### ❌ "progress.json should be updated after indexing completes"
**Reality**: `progress.json` is for real-time progress tracking during indexing, not post-indexing stats.

## Troubleshooting

### "Status shows 0 files but databases have data"

**Possible causes**:
1. Corrupted Lance database (missing data fragments)
2. Database not initialized properly
3. Wrong database path in config

**Debug steps**:
```bash
# Check database files exist
ls -lh .mcp-vector-search/lance/

# Check chunks.lance size
du -sh .mcp-vector-search/lance/chunks.lance/

# Check table schema (validates database structure)
python -c "
import lancedb
db = lancedb.connect('.mcp-vector-search/lance')
table = db.open_table('chunks')
print(table.schema)
print(f'Total rows: {table.count_rows()}')
"
```

### "Search returns no results but vectors exist"

**Possible causes**:
1. Query embedding model mismatch
2. Similarity threshold too high
3. Corrupted vector index
4. Wrong database path

**Debug steps**:
```bash
# Check vectors.lance exists and has data
du -sh .mcp-vector-search/lance/vectors.lance/

# Check vector count
python -c "
import lancedb
db = lancedb.connect('.mcp-vector-search/lance')
table = db.open_table('vectors')
print(f'Total vectors: {table.count_rows()}')
"

# Test with very low threshold
mcp-vector-search search "test query" --threshold 0.1
```

## Implementation Reference

### IndexMetadata Class
**File**: `src/mcp_vector_search/core/index_metadata.py`

**Key Methods**:
- `load()` - Load file_mtimes dictionary
- `save(metadata)` - Save file_mtimes dictionary
- `needs_reindexing(file_path, metadata)` - Check if file changed
- `get_index_version()` - Get version from metadata
- `needs_reindex_for_version()` - Check if version upgrade requires reindex

### ChunksBackend.get_stats()
**File**: `src/mcp_vector_search/core/chunks_backend.py`

**Query Strategy**:
```python
# Efficient column projection (only read needed columns)
scanner = self._table.to_lance().scanner(
    columns=["embedding_status", "file_path", "language"]
)
result = scanner.to_table()
df = result.to_pandas()

# In-memory aggregation
status_counts = df["embedding_status"].value_counts().to_dict()
file_count = df["file_path"].nunique()
language_counts = df["language"].value_counts().to_dict()
```

**Performance**:
- Fast for small-medium databases (<100K chunks)
- May hang on very large databases (>500K chunks)
- Use `--skip-stats` flag for large databases

### VectorsBackend.search()
**File**: `src/mcp_vector_search/core/vectors_backend.py`

**Query Strategy**:
```python
# Vector similarity search with IVF_PQ index
results = (
    self._table
    .search(query_vector)
    .metric("cosine")
    .limit(limit)
    .to_list()
)
```

**Performance**:
- O(log n) with IVF_PQ index (after indexing)
- O(n) without index (linear scan)
- Index rebuilt automatically after batch inserts

## Summary

- **`index_metadata.json`**: File modification times only (incremental indexing)
- **`chunks.lance`**: Source of truth for chunk statistics (Phase 1)
- **`vectors.lance`**: Source of truth for vector statistics (Phase 2)
- **`progress.json`**: Real-time progress tracking during indexing
- **Statistics**: Always computed on-demand from Lance databases
- **Search**: Queries `vectors.lance` directly, never reads metadata JSON

**Key Principle**: Lance databases are the single source of truth. Metadata files are auxiliary.
