# Bug Investigation Report: index_metadata.json

**Date**: 2026-02-19
**Investigator**: Python Engineer (Claude Opus 4.6)
**Issue**: Claimed bug where `index_metadata.json` not updating with statistics

## Summary

**FINDING**: There is **NO BUG**. The reported issue is based on architectural misunderstanding.

`index_metadata.json` was **never designed** to store aggregate statistics like `total_files`, `total_chunks`, or `languages`. These statistics are computed on-demand from the Lance databases.

## Investigation Details

### 1. What `index_metadata.json` Actually Contains

**Current Structure** (verified in codebase):
```json
{
  "index_version": "2.5.49",
  "indexed_at": "2026-02-20T02:51:23.367557+00:00",
  "file_mtimes": {
    "/Users/masa/Projects/mcp-vector-search/CHANGELOG.md": 1770347648.900154,
    // ... 727 files tracked
  }
}
```

**Source**: `src/mcp_vector_search/core/index_metadata.py` lines 92-100

```python
def save(self, metadata: dict[str, float]) -> None:
    """Save file modification times to metadata file."""
    data = {
        "index_version": __version__,
        "indexed_at": datetime.now(UTC).isoformat(),
        "file_mtimes": metadata,  # <-- Only file paths -> timestamps
    }
    with open(self._metadata_file, "w") as f:
        json.dump(data, f, indent=2)
```

**Verification**:
```bash
$ python3 -c "import json; data = json.load(open('.mcp-vector-search/index_metadata.json')); print('Keys:', list(data.keys())); print('file_mtimes count:', len(data.get('file_mtimes', {})))"
Keys: ['index_version', 'indexed_at', 'file_mtimes']
file_mtimes count: 727
```

**Conclusion**: File is working correctly. It stores file modification times as designed.

### 2. Where Statistics Actually Come From

#### Status Command Flow

**Code Path**: `src/mcp_vector_search/cli/commands/status.py` lines 266-296

```python
# status.py line 266
index_stats = await indexer.get_indexing_stats(db_stats=db_stats)

# ↓ Calls indexer.py line 2426
chunks_stats = await self.chunks_backend.get_stats()

# ↓ Calls chunks_backend.py line 843-866
scanner = self._table.to_lance().scanner(
    columns=["embedding_status", "file_path", "language"]
)
result = scanner.to_table()
df = result.to_pandas()

# Compute stats in-memory from database rows
status_counts = df["embedding_status"].value_counts().to_dict()
file_count = df["file_path"].nunique()
language_counts = df["language"].value_counts"].to_dict()

return {
    "total": len(df),                              # Total chunks
    "pending": status_counts.get("pending", 0),
    "complete": status_counts.get("complete", 0),
    "files": file_count,                           # Unique files
    "languages": language_counts,                  # Language distribution
}
```

**Key Finding**: Statistics are computed on-demand by querying `chunks.lance` directly.

**Does NOT read** `index_metadata.json` for statistics (except version info).

#### Search Command Flow

**Code Path**: `src/mcp_vector_search/core/search.py` → `vectors_backend.py`

```python
# Search queries vectors.lance directly
results = self._table.search(query_vector).limit(limit).to_list()
```

**Key Finding**: Search never reads `index_metadata.json` at all.

### 3. Purpose of `index_metadata.json`

**Design Intent** (from `index_metadata.py` docstrings):

```python
class IndexMetadata:
    """Manages index metadata including file modification times and version tracking.

    This class encapsulates all logic related to tracking which files have been
    indexed, when they were modified, and what version of the indexer created them.
    """
```

**Use Cases**:
1. **Incremental Indexing**: Skip unchanged files by comparing mtimes
2. **Version Tracking**: Recommend reindex on major/minor version changes
3. **Cleanup**: Remove entries for deleted files

**NOT USED FOR**:
- ❌ Storing aggregate statistics
- ❌ Search operations
- ❌ Status reporting (except version)

### 4. Code Verification

#### When `index_metadata.json` is Written

**Search Results**: `grep -r "\.save\(" src/mcp_vector_search/core/indexer.py`

**Lines where metadata is saved**:
- Line 445: After force reindex (clears all metadata)
- Line 886: After successful file indexing batch
- Line 1630: After incremental indexing
- Line 1698: After updating file mtimes
- Line 2076: After batch processing
- Line 2228: After single file index
- Line 2879: After embedding batch completes

**Verification**: Metadata is being saved correctly after indexing operations.

#### What Gets Saved

**Every save call**:
```python
self.metadata.save(metadata_dict)  # metadata_dict = {file_path: mtime, ...}
```

**Never saved**:
- Total file counts
- Total chunk counts
- Language distributions

### 5. Alternative Explanation

The user might be confusing `index_metadata.json` with `progress.json`:

**`progress.json` Structure** (verified):
```json
{
  "phase": "chunking",
  "chunking": {
    "total_files": 727,
    "processed_files": 512,
    "total_chunks": 2022
  },
  "embedding": {
    "total_chunks": 2022,
    "embedded_chunks": 2022
  }
}
```

**Purpose**: Real-time progress tracking during indexing (NOT used by status/search).

## Architectural Design Analysis

### Why Statistics Are Computed On-Demand

**Benefits**:
1. **Single Source of Truth**: Lance databases are authoritative
2. **No Sync Issues**: Stats always reflect actual database state
3. **Crash Recovery**: Database state persists even if indexing crashes
4. **Simpler Logic**: No need to maintain dual state (JSON + DB)

**Trade-offs**:
1. **Compute Cost**: Stats require table scan (acceptable for <100K chunks)
2. **No Caching**: Each status call re-queries database (but Lance is fast)

### Potential Real Issues (If User Is Seeing 0 Stats)

If the user genuinely sees 0 files/chunks despite databases having data:

**Possible Root Causes**:
1. **Database Not Initialized**: `chunks_backend._db is None`
2. **Corrupted Lance Files**: Missing data fragments
3. **Wrong Database Path**: Config points to wrong directory
4. **Exception in get_stats()**: Returns default empty dict

**Debug Steps** (add to troubleshooting guide):
```python
# Check if table is accessible
import lancedb
db = lancedb.connect('.mcp-vector-search/lance')
table = db.open_table('chunks')
print(f'Chunks table rows: {table.count_rows()}')
print(f'Schema: {table.schema}')

# Check if get_stats() works
import asyncio
from pathlib import Path
from mcp_vector_search.core.chunks_backend import ChunksBackend

async def test_stats():
    backend = ChunksBackend(Path('.mcp-vector-search/lance'))
    await backend.initialize()
    stats = await backend.get_stats()
    print(f'Stats: {stats}')

asyncio.run(test_stats())
```

## Recommendations

### 1. Update Documentation

Create clear documentation explaining:
- What `index_metadata.json` contains and why
- How statistics are computed (on-demand from databases)
- Difference between `index_metadata.json` and `progress.json`

**Action**: Created `INDEX_METADATA_ARCHITECTURE.md` (see attached)

### 2. Add Validation to Status Command

If databases exist but return 0 stats, log a warning:

```python
# In status.py after getting stats
if db_stats.total_chunks == 0 and chunks_db_path.exists():
    logger.warning(
        "Database files exist but no chunks found. "
        "This may indicate database corruption. "
        "Run 'mcp-vector-search index --force' to rebuild."
    )
```

### 3. Improve Error Messages

If `chunks_backend.get_stats()` fails, provide actionable error:

```python
# In chunks_backend.py get_stats() exception handler
except Exception as e:
    logger.error(
        f"Failed to read chunk statistics: {e}\n"
        f"Database path: {self.db_path}\n"
        f"Table exists: {self._table is not None}\n"
        f"Run 'mcp-vector-search status --health-check' for diagnostics"
    )
```

### 4. Add Health Check for Database Integrity

```python
# In status.py perform_health_check()
try:
    # Check chunks.lance exists and is readable
    chunks_path = config.index_path / "lance" / "chunks.lance"
    if chunks_path.exists():
        chunks_backend = ChunksBackend(config.index_path / "lance")
        await chunks_backend.initialize()
        stats = await chunks_backend.get_stats()

        if stats["total"] == 0:
            health_status["components"]["chunks_database"] = "warning"
            health_status["issues"].append(
                "Chunks database exists but is empty. Run 'mcp-vector-search index'."
            )
        else:
            health_status["components"]["chunks_database"] = "ok"
    else:
        health_status["components"]["chunks_database"] = "missing"
        health_status["issues"].append("Chunks database not found. Run 'mcp-vector-search index'.")
except Exception as e:
    health_status["components"]["chunks_database"] = "error"
    health_status["issues"].append(f"Chunks database error: {e}")
```

## Conclusion

**PRIMARY FINDING**: No bug exists. The architecture is working as designed.

**USER MISCONCEPTION**: `index_metadata.json` was never meant to store statistics.

**STATISTICS SOURCE**: Always computed from Lance databases on-demand.

**SEARCH BEHAVIOR**: Queries `vectors.lance` directly, independent of `index_metadata.json`.

**RECOMMENDED ACTIONS**:
1. ✅ Created comprehensive architecture documentation
2. ⚠️ Consider adding database health checks to `status --health-check`
3. ⚠️ Consider improving error messages when stats return 0 despite databases existing
4. ✅ No code changes needed to "fix" metadata writing (it's already working correctly)

## Files Created

1. **INDEX_METADATA_ARCHITECTURE.md** - Comprehensive architecture documentation
2. **BUG_INVESTIGATION_REPORT.md** - This report

## Test Plan (If User Still Sees Issues)

If the user genuinely experiences 0 stats despite databases having data:

```bash
# 1. Verify databases exist and have data
du -sh .mcp-vector-search/lance/*.lance/

# 2. Check database contents directly
python3 << 'EOF'
import lancedb
db = lancedb.connect('.mcp-vector-search/lance')

# Check chunks table
try:
    chunks = db.open_table('chunks')
    print(f'Chunks table: {chunks.count_rows()} rows')
    print(f'Schema: {chunks.schema}')
    sample = chunks.to_pandas().head(3)
    print(f'Sample:\n{sample}')
except Exception as e:
    print(f'Chunks error: {e}')

# Check vectors table
try:
    vectors = db.open_table('vectors')
    print(f'Vectors table: {vectors.count_rows()} rows')
except Exception as e:
    print(f'Vectors error: {e}')
EOF

# 3. Run status with verbose logging
mcp-vector-search status --verbose --health-check

# 4. Test search
mcp-vector-search search "test" --limit 5 --threshold 0.1
```

If all of the above show data but status returns 0, then there IS a real bug in the query path.

## Signature

**Investigation Complete**: 2026-02-19
**Status**: No bug found - architecture working as designed
**Confidence**: 95% (based on code inspection and verification)
**Next Steps**: Wait for user feedback on whether they're still experiencing issues
