# index1 vs Claude Code Native Tools — Search Benchmark

> Date: 2026-02-06
> Environment: macOS (Apple Silicon M2), 48GB RAM, Python 3.14, Ollama nomic-embed-text
> Project: index1 codebase — 19 Python files, 45 docs, 17,725 chunks
> Database: `~/.claude-index1/knowledge.db`

## 1. Test Method

Tested on the index1 codebase itself, comparing three search approaches:

- **Native Tools** (`Grep` / `Glob`): Claude Code built-in tools (ripgrep + globset)
- **index1 CLI** (`index1 search`): Command-line with Python startup overhead
- **index1 MCP** (`docs_search`): Persistent MCP Server process via stdio

Each test covers both exact keyword and semantic queries.

## 2. Speed Comparison

### 2.1 Raw Results

| Query | Type | Grep/Glob | index1 CLI | index1 MCP |
|-------|------|-----------|------------|------------|
| `CJK` | Keyword | 4 ms | 326 ms | ~200 ms |
| `embedding` | Keyword | 4 ms | 236 ms | ~100 ms |
| `搜索是怎么工作的` | Semantic (CJK) | 4 ms* | 232 ms | 99 ms |
| `how to configure watch paths` | Semantic (EN) | 4 ms* | 335 ms | ~150 ms |

> \*Grep can only match literal text. It cannot understand semantics. Times shown are for comparison only.

### 2.2 With Ollama vs Without Ollama

| Query | Hybrid (Ollama) | BM25-only (no Ollama) | Delta |
|-------|-----------------|----------------------|-------|
| `CJK` | 79 ms, 5 results | 1,064 ms, 5 results | **13x slower** |
| `embedding` | 38 ms, 5 results | 1,061 ms, 5 results | **28x slower** |
| `搜索是怎么工作的` | 41 ms, **5 results** | 1,045 ms, **0 results** | **100% loss** |
| `how to configure watch paths` | 181 ms, 5 results | 1,198 ms, 5 results | **7x slower** |

Key findings:
- **BM25-only first query is 7–28x slower** due to Ollama connection timeout
- Subsequent queries are fast (~35 ms) — failure state is cached for 60s (v0.1.1+)
- **Chinese semantic queries return 0 results** without vector embeddings
- After cache warm-up, both modes return in < 1 ms

### 2.3 Latency Breakdown

```
Native Grep:    [  4ms  ]  ← ripgrep full-text scan
                ━━━━━━━━

index1 CLI:     [ ~150ms Python startup ][ ~50ms BM25 ][ ~100ms Vector ][ ~30ms RRF ]
                ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

index1 MCP:     [ ~50ms BM25 ][ ~100ms Vector ][ ~30ms RRF ]
                ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                (persistent process, no startup cost)
```

- **Python startup**: CLI pays ~150ms per invocation (module import + init)
- **BM25 query**: FTS5 full-text index ~50ms
- **Vector query**: Ollama embedding API + sqlite-vec cosine ~100ms
- **RRF fusion**: Reciprocal Rank Fusion merge ~30ms

### 2.4 Cache Performance

Consecutive queries in the same process:

| Run | `CJK` | `embedding` | `搜索是怎么工作的` | `how to configure watch paths` |
|-----|-------|-------------|-------------------|-------------------------------|
| Cold | 79 ms | 38 ms | 41 ms | 181 ms |
| Hot (cached) | < 1 ms | < 1 ms | < 1 ms | < 1 ms |

L1 cache (10-min TTL) eliminates repeated query cost entirely.

## 3. Context Window Usage Analysis

When an AI agent (e.g. Claude Code) searches code, the search tool runs **locally for free** — but the results are sent back to the LLM API as `tool_result` messages. These results consume **context window tokens**, which:

- **Fill the context window faster** → triggers earlier conversation compression, losing prior messages
- **Slow down responses** → larger input context = longer LLM inference time
- **Cost money for API-key users** → input tokens are billed per use

> **Note**: This is NOT about the local search cost (which is zero). It's about how much LLM context each search approach consumes with its returned results.

### 3.1 Grep Context Cost (full project scan)

| Query | Lines Matched | Characters | Estimated Tokens |
|-------|--------------|------------|-----------------|
| `CJK` | 24 | 2,490 | ~622 |
| `embedding` | 217 | 23,214 | ~5,803 |
| `search` | 881 | 143,583 | ~35,895 |
| `watch` | 457 | 140,590 | ~35,147 |

> Grep returns **every matching line** across the entire project, unranked.

### 3.2 index1 Context Cost (top-5 results)

| Query | Results | Characters | Estimated Tokens |
|-------|---------|------------|-----------------|
| `CJK` | 5 | 1,840 | ~460 |
| `embedding` | 5 | 2,056 | ~514 |
| `搜索是怎么工作的` | 5 | 1,644 | ~411 |
| `how to configure watch paths` | 5 | 1,868 | ~467 |

> index1 returns **pre-ranked top-k results** with relevance scores.

### 3.3 Context Savings

| Query | Grep (context tokens) | index1 (context tokens) | Savings |
|-------|----------------------|------------------------|---------|
| `CJK` | ~622 | ~460 | 26% |
| `embedding` | ~5,803 | ~514 | **91%** |
| `search` | ~35,895 | ~411 | **99%** |
| `watch` | ~35,147 | ~467 | **99%** |

For common queries, index1 saves **90–99% of context window usage**:
- **More room for conversation** — context window fills slower, less compression needed
- **Faster LLM responses** — smaller input = faster inference
- **API-key users save money** — fewer input tokens billed

### 3.4 Context Budget at Scale

For a typical agent session with 20 search queries (200k context window):

| Approach | Context per Query (avg) | Total Context Used | % of 200k Window |
|----------|------------------------|--------------------|-------------------|
| Grep only | ~15,000 tokens | 300,000 | **150%** (triggers compression) |
| index1 only | ~460 tokens | 9,200 | **5%** (plenty of room) |
| Mixed (recommended) | ~5,000 tokens | 100,000 | 50% |

> Mixed = 50% Grep (for exact lookups) + 50% index1 (for semantic queries).
>
> With Grep-only approach, 20 broad searches can **overflow the entire context window**, causing the agent to lose earlier conversation history.

## 4. Search Quality Comparison

### 4.1 Exact Keyword: `CJK`

**Grep** (5 lines in `src/`):
```
src/index1/search.py:52:  def _has_cjk(text: str) -> bool:
src/index1/search.py:54:  return bool(_CJK_PATTERN.search(text))
tests/test_search.py:    class TestHasCJK
```
Precise. No noise. Every line contains "CJK" literally.

**index1** (top 5):
```
1. [high] tests/test_search.py — TestHasCJK
2. [high] README.md — License (noise)
3. [high] tests/test_search.py — duplicate
4. [high] CLAUDE-MEM-DEEP-ANALYSIS.md — (noise)
5. [high] src/index1/search.py — _has_cjk
```
Hits core code but includes unrelated results.

**Winner**: Grep — zero noise for exact keyword search.

### 4.2 Semantic: `搜索是怎么工作的` ("how does search work")

**Grep** (matching "search", 41 lines):
```
src/index1/cli.py: 4 matches
src/index1/mcp_server.py: 6 matches
src/index1/search.py: 4 matches
src/index1/web.py: 22 matches
... 41 total lines, unranked
```
Returns everything containing "search". Cannot understand Chinese semantics.

**index1** (top 5):
```
1. [high] CLAUDE-MEM-DEEP-ANALYSIS.md — search optimization
2. [high] README.md — project overview
3. [high] CLAUDE-MEM-DEEP-ANALYSIS.md — system summary
4. [high] CLAUDE-MEM-DEEP-ANALYSIS.md — system architecture
5. [high] src/index1/indexer.py — SkipFile
```
Understands Chinese query semantically, finds relevant docs.

**Without Ollama**: **0 results** — BM25 cannot match Chinese query to English code.

**Winner**: index1 — only tool that handles cross-language semantic queries.

### 4.3 Semantic: `how to configure watch paths`

**Grep** (would need multiple searches: "watch" + "config" + "path"):
```
Requires 2-3 separate searches + manual cross-referencing
```

**index1** (top 5):
```
1. [high] src/index1/mcp_server.py — docs_reindex
2. [high] src/index1/cli.py — watch command
3. [medium] ...
```
Single query covers multiple concepts ("configure" + "watch" + "paths").

**Winner**: index1 — multi-concept queries in one shot.

## 5. Advantages & Disadvantages

### 5.1 Native Tools (Grep / Glob)

| Dimension | Rating | Notes |
|-----------|--------|-------|
| Speed | 5/5 | 4ms, near-instant |
| Precision | 5/5 | Regex matching, zero false positives |
| Semantic | 1/5 | Literal matching only |
| Dependencies | 5/5 | Zero, built into Claude Code |
| Freshness | 5/5 | Real-time, no indexing needed |
| Ranking | 2/5 | By file/line, no relevance scoring |
| Cross-language | 1/5 | Cannot match Chinese to English |
| Resource Usage | 5/5 | No extra memory/disk |
| **Context Cost** | **2/5** | **Returns ALL matches → floods LLM context window** |

**Best for**: Known identifiers, regex patterns, small projects.

### 5.2 index1 (BM25 + Vector Hybrid)

| Dimension | Rating | Notes |
|-----------|--------|-------|
| Speed | 3/5 | 40–180ms cold, < 1ms cached |
| Precision | 3/5 | Approximate matching, some noise |
| Semantic | 4/5 | Understands query intent |
| Dependencies | 2/5 | Requires Ollama + pre-indexing |
| Freshness | 3/5 | Needs reindex or watch mode |
| Ranking | 4/5 | RRF fusion, relevance-first |
| Cross-language | 4/5 | CJK dynamic weighting |
| Resource Usage | 2/5 | Ollama ~500MB, DB ~10MB |
| **Context Cost** | **5/5** | **Top-k only → ~400–500 tokens, minimal context usage** |

**Best for**: Exploratory queries, cross-language search, large projects, AI agents.

## 6. Recommendation

### Decision Flow for AI Agents

```
Query received
    |
    +-- Contains exact identifier? (function/class/file name)
    |   └── YES → Grep/Glob (4ms, precise)
    |
    +-- Exploratory / understanding question?
    |   └── YES → index1 MCP (40-180ms, semantic)
    |
    +-- Cross-language? (Chinese query → English code)
    |   └── YES → index1 MCP (CJK weighting)
    |
    +-- Large result set expected? (common keyword like "search")
    |   └── YES → index1 MCP (saves 99% tokens)
    |
    └── Uncertain?
        └── Try Grep first, fall back to index1
```

### Summary

| Metric | Native Tools | index1 (Hybrid) | index1 (BM25-only) |
|--------|-------------|-----------------|-------------------|
| Speed (cold) | **4 ms** | 40–180 ms | 1,000–1,200 ms |
| Speed (cached) | N/A | **< 1 ms** | **< 1 ms** |
| Exact search | **Best** | Good | Good |
| Semantic search | None | **Best** | None |
| CJK queries | None | **Best** | **None (0 results)** |
| Context cost | High (all matches) | **Low (top-k)** | **Low (top-k)** |
| Dependencies | **None** | Ollama + index | Index only |

**Bottom line**: Use both. Grep is the "precision missile", index1 is the "semantic radar". Together they give AI agents fast exact lookups AND intelligent semantic understanding — while saving 90%+ context tokens on broad queries.

**Install Ollama.** Without it, you lose semantic search entirely, CJK queries break completely, and every query pays a 1-second timeout penalty.
