# Milestone v2.0: Chunk Foundation

**Status:** ✅ SHIPPED 2026-02-24
**Phases:** 17–18
**Total Plans:** 4

## Overview

Established the chunk data layer and IDE-clickable CLI display. Every indexed chunk now carries exact line boundaries (`start_line`, `end_line`), construct name (`chunk_name`), and full text (`chunk_text`) in LanceDB. `corpus search` output switched from a 3-line snippet format to grep/compiler-error format — IDE-clickable in VSCode and IntelliJ.

## Phases

### Phase 17: Schema v4 — Chunk Data Layer

**Goal**: LanceDB stores `start_line`, `end_line`, `chunk_name`, and `chunk_text` per chunk; existing data migrates cleanly without re-indexing; all chunkers populate these fields
**Depends on**: Phase 16 (v1.5 TypeScript AST Chunking complete)
**Requirements**: CHUNK-01
**Plans**: 2 plans

Plans:

- [x] 17-01-PLAN.md — TDD RED: write failing tests for `ChunkRecord` v4 fields and `ensure_schema_v4()` migration
- [x] 17-02-PLAN.md — GREEN: implement `ensure_schema_v4()`, update `ChunkRecord`, wire all chunkers to populate new fields

**Details:**
- `ensure_schema_v4()` idempotent migration: adds `chunk_name`, `chunk_text`, `start_line`, `end_line` columns to existing LanceDB tables; re-running on an already-migrated table is a no-op
- `ChunkRecord` dataclass gains four new fields with empty-string / zero defaults for existing rows
- All three chunkers emit v4 fields: Markdown uses heading text as `chunk_name`, Python uses `node.name`, TypeScript uses construct name
- `_enforce_char_limit` carries `chunk_name`/`chunk_text` through all four sub-chunk split paths
- `CorpusIndex.open()` calls `ensure_schema_v4(table)`; migration chain: v2→v3→v4
- Parametrised round-trip test across `.md`, `.py`, `.ts` fixtures verifies zero-hallucination line-range contract
- 330 tests green after plan 17-02

---

### Phase 18: CLI Chunk Display

**Goal**: `corpus search` output switches to grep/compiler-error format showing exact line ranges — IDE-clickable and human-readable
**Depends on**: Phase 17 (chunk fields in schema)
**Requirements**: CHUNK-02
**Plans**: 2 plans

Plans:

- [x] 18-01-PLAN.md — TDD RED: write failing tests for new result formatter (grep-style output, truncation, path format)
- [x] 18-02-PLAN.md — GREEN: implement `format_result()`, wire into CLI search command

**Details:**
- `format_result(result, cwd) -> tuple[str, str | None]` — pure function; `Console.print()` owns rendering
- Primary line: `path/to/file.md:42-67 [skill] score:0.021` — relative path via `Path.relative_to(cwd)` with absolute fallback
- Line range omitted when both `start_line == 0` and `end_line == 0` (legacy row rule)
- Score always 3 decimal places in `[cyan]score:X.XXX[/cyan]`
- Preview: first line of `chunk_text`, 4-space indent, truncated at 200 chars with `...`
- `rich.markup.escape()` applied to path, construct_type, and preview to prevent MarkupError
- `search_command` render loop replaced — `extract_snippet` removed from cli.py import (still exported from module)
- 340 tests green after plan 18-02

---

## Milestone Summary

**Requirements Completed:**
- CHUNK-01: Schema v4 chunk data layer — ✅ Phase 17
- CHUNK-02: CLI grep-style chunk display — ✅ Phase 18

**Requirements Deferred to v2.1+:**
- CHUNK-03: MCP chunk response fields → v2.1
- SUB-01–SUB-04: Method sub-chunking (Python + TypeScript) → v2.1
- NAME-01–NAME-03: `chunk_name` filtering (CLI + MCP) → v2.1
- SORT-01, NORM-01: MCP sort_by + score normalisation → v2.1/v2.2
- JSON-01: `--output json` → v2.1/v2.2
- GRAPH-01: `corpus_graph` MCP tool → v2.2

**Key Decisions:**

- `format_result` returns `(primary, preview | None)` tuple — pure function; Console.print() owns rendering
- `chunk_text` read via `chunk.get("chunk_text", "")` in indexer — safe because summary prepend mutates `chunk["text"]` only
- `ensure_schema_v2 → ensure_schema_v3 → ensure_schema_v4` call order preserved in `CorpusIndex.open()`
- Deferred import pattern in RED tests for `ensure_schema_v4` keeps pre-existing tests collectable while new tests are RED
- `extract_snippet` removed from cli.py import (F401 — unused after render loop replacement); still exported from module

**Issues Resolved:**
- LanceDB direct-insert test fixtures broke after ChunkRecord schema added new required fields — auto-fixed with `chunk_name=""`, `chunk_text=""` across 9 row dicts in test_engine.py and test_indexer.py
- Two E501 docstring violations in RED-phase tests fixed inline (Rule 1)

**Technical Debt:**
- Existing rows in user LanceDB tables have `start_line=0`, `end_line=0`, `chunk_name=""`, `chunk_text=""` until re-indexed
- `--min-score` help text still references raw RRF range (0.009–0.033) — will need updating after NORM-01 (Phase 23) ships score normalisation

---

_For current project status, see .planning/PROJECT.md_
