# Performance Metrics Implementation

## Overview

Added comprehensive performance metrics logging for all indexing phases with support for both formatted console output and programmatic JSON access.

## Implementation

### Core Module: `src/mcp_vector_search/core/metrics.py`

**Classes**:
- `PhaseMetrics` - Metrics for a single phase (count, duration, throughput)
- `IndexingMetrics` - Complete metrics for all phases
- `MetricsTracker` - Singleton tracker with context manager support

**Features**:
- Context manager for automatic timing: `with tracker.phase("parsing") as metrics:`
- Automatic throughput calculation (items/second)
- Singleton pattern for global access
- Rich console output with formatted tables
- JSON export for programmatic access

### Integration Points

**Modified Files**:
1. `src/mcp_vector_search/core/indexer.py`
   - Added `metrics_json` parameter to `index_project()`
   - Wrapped Phase 1 (parsing/chunking) with metrics tracking
   - Wrapped Phase 2 (embedding) with metrics tracking
   - Added automatic metrics summary logging

2. `src/mcp_vector_search/cli/commands/index.py`
   - Added `--metrics-json` flag to index command
   - Passed `metrics_json` parameter through to indexer

## Usage

### CLI Usage

**Standard Output (Formatted Table)**:
```bash
mcp-vector-search index
```

Output:
```
📊 Parsing: 1,234 files in 12.3s (100.3 files/sec)
📊 Chunking: 15,678 chunks in 0.0s (inf chunks/sec)
📊 Embedding: 15,678 embeddings in 45.1s (347.6 emb/sec)

═══════════════════════════════════════════════════
📊 Performance Summary
═══════════════════════════════════════════════════
Phase           │      Count │     Time │   Throughput
───────────────┼────────────┼──────────┼─────────────
Parsing         │      1,234 │   12.3s │      100.3/s
Chunking        │     15,678 │    0.0s │         inf/s
Embedding       │     15,678 │   45.1s │      347.6/s
───────────────┼────────────┼──────────┼─────────────
Total           │            │   57.4s │
═══════════════════════════════════════════════════
```

**JSON Output**:
```bash
mcp-vector-search index --metrics-json
```

Output:
```json
{
  "parsing": {
    "phase_name": "Parsing",
    "item_count": 1234,
    "duration_seconds": 12.3,
    "throughput": 100.3
  },
  "chunking": {
    "phase_name": "Chunking",
    "item_count": 15678,
    "duration_seconds": 0.0,
    "throughput": 0.0
  },
  "embedding": {
    "phase_name": "Embedding",
    "item_count": 15678,
    "duration_seconds": 45.1,
    "throughput": 347.6
  },
  "kg_build": {
    "phase_name": "KG Build",
    "item_count": 0,
    "duration_seconds": 0.0,
    "throughput": 0.0
  },
  "total_duration_seconds": 57.4,
  "total_files": 1234,
  "total_chunks": 15678,
  "total_embeddings": 15678,
  "total_entities": 0
}
```

### Programmatic Usage

```python
from mcp_vector_search.core.metrics import get_metrics_tracker

# Get global tracker instance
tracker = get_metrics_tracker()
tracker.reset()

# Track a phase with context manager
with tracker.phase("parsing") as metrics:
    # Do parsing work
    # ...
    metrics.item_count = num_files  # Set count

# Get metrics programmatically
tracker.finalize()
metrics = tracker.metrics

print(f"Parsed {metrics.total_files} files in {metrics.parsing.duration_seconds:.1f}s")
print(f"Throughput: {metrics.parsing.throughput:.1f} files/sec")

# Export as JSON
json_str = metrics.to_json()
```

## Tracked Metrics

### Phase 1: Parsing
- **Item Count**: Number of files parsed
- **Duration**: Time spent parsing files
- **Throughput**: Files per second

### Phase 2: Chunking
- **Item Count**: Number of chunks created
- **Duration**: Time spent building chunk hierarchy (minimal, tracked separately for visibility)
- **Throughput**: Chunks per second

### Phase 3: Embedding
- **Item Count**: Number of embeddings generated
- **Duration**: Time spent in embedding generation
- **Throughput**: Embeddings per second

### Phase 4: KG Build (Optional)
- **Item Count**: Number of entities/relationships created
- **Duration**: Time spent building knowledge graph
- **Throughput**: Entities per second

### Overall Metrics
- **Total Duration**: End-to-end indexing time
- **Total Files**: Sum of files processed
- **Total Chunks**: Sum of chunks created
- **Total Embeddings**: Sum of embeddings generated
- **Total Entities**: Sum of KG entities created

## Performance Insights

The metrics help identify bottlenecks:

1. **Parsing Bottleneck**: Low files/sec → Consider increasing workers or optimizing parsers
2. **Embedding Bottleneck**: Low embeddings/sec → GPU/CPU performance issue, consider batch size tuning
3. **KG Build Bottleneck**: Long duration → Consider background processing or optimization

## Testing

Run the standalone test to verify metrics tracking:

```bash
python test_metrics_standalone.py
```

Expected output shows:
- Per-phase metrics logged as phases complete
- Formatted summary table at end
- JSON export with all metrics

## Future Enhancements

Potential improvements:
1. **Sub-phase Metrics**: Track hash checking, file I/O, hierarchy building separately
2. **Memory Metrics**: Track peak memory usage per phase
3. **Error Metrics**: Track failed files, retry counts
4. **Historical Tracking**: Store metrics over time for trend analysis
5. **Comparison Mode**: Compare current run with previous runs
6. **Alerting**: Warn if throughput drops below threshold

## Notes

- **Chunking Duration**: Currently minimal since chunking happens inline with parsing. The metrics track the chunk count separately for visibility.
- **Singleton Pattern**: Uses singleton to allow metrics access from anywhere in codebase
- **Context Manager**: Ensures automatic duration calculation even if phase errors
- **JSON Export**: Enables integration with monitoring tools (Grafana, Datadog, etc.)
