---
phase: 17-schema-v4-chunk-data-layer
plan: "02"
type: tdd
wave: 2
depends_on:
  - "17-01"
files_modified:
  - src/corpus_analyzer/store/schema.py
  - src/corpus_analyzer/ingest/chunker.py
  - src/corpus_analyzer/ingest/ts_chunker.py
  - src/corpus_analyzer/ingest/indexer.py
autonomous: true
requirements:
  - CHUNK-01

must_haves:
  truths:
    - "All RED tests from Plan 17-01 pass GREEN"
    - "ChunkRecord.model_fields includes chunk_name (str, default '') and chunk_text (str, default '')"
    - "ensure_schema_v4(table) adds chunk_name and chunk_text columns; calling twice raises no error"
    - "chunk_markdown() returns dicts with chunk_name = heading line verbatim and chunk_text = section text"
    - "chunk_python() returns dicts with chunk_name = function/class name and chunk_text = raw lines slice"
    - "chunk_typescript() returns dicts with chunk_name (existing) and chunk_text = raw source slice"
    - "_enforce_char_limit carries chunk_name and chunk_text through to sub-chunks unchanged"
    - "Indexer stores chunk_name and chunk_text in LanceDB for all three file types"
    - "Round-trip test: start_line, end_line, chunk_name, chunk_text all correct in stored rows"
    - "All 320 pre-existing tests remain green"
    - "ruff check . exits 0 and mypy src/ exits 0"
  artifacts:
    - path: "src/corpus_analyzer/store/schema.py"
      provides: "ensure_schema_v4() function + ChunkRecord chunk_name and chunk_text fields"
      contains: "ensure_schema_v4"
    - path: "src/corpus_analyzer/ingest/chunker.py"
      provides: "chunk_markdown and chunk_python emit chunk_name + chunk_text; _enforce_char_limit carries them"
      contains: "chunk_name.*chunk_text"
    - path: "src/corpus_analyzer/ingest/ts_chunker.py"
      provides: "chunk_typescript emits chunk_text (chunk_name already present)"
      contains: "chunk_text"
    - path: "src/corpus_analyzer/ingest/indexer.py"
      provides: "CorpusIndex calls ensure_schema_v4; chunk_dict includes chunk_name and chunk_text"
      contains: "ensure_schema_v4"
  key_links:
    - from: "src/corpus_analyzer/ingest/indexer.py"
      to: "src/corpus_analyzer/store/schema.py"
      via: "ensure_schema_v4 import and call in CorpusIndex.open()"
      pattern: "ensure_schema_v4"
    - from: "src/corpus_analyzer/ingest/indexer.py"
      to: "src/corpus_analyzer/ingest/chunker.py"
      via: "chunk.get('chunk_name', '') and chunk.get('chunk_text', '') in chunk_dict"
      pattern: "chunk\\.get\\(['\"]chunk_name"
    - from: "src/corpus_analyzer/ingest/indexer.py"
      to: "src/corpus_analyzer/ingest/ts_chunker.py"
      via: "same chunk.get pattern for TypeScript chunks"
      pattern: "chunk\\.get\\(['\"]chunk_text"
---

<objective>
Implement ChunkRecord v4 fields (`chunk_name`, `chunk_text`), `ensure_schema_v4()` migration, wire all three chunkers to emit the new fields, update `_enforce_char_limit` to carry them through, and update the indexer to pass them to LanceDB. All RED tests from Plan 17-01 must turn GREEN.

Purpose: Establish the zero-hallucination line-range contract (CHUNK-01) — every indexed chunk carries its exact source text, name, and line boundaries.
Output: Production code across 4 files; all 320+ tests green; ruff+mypy clean.
</objective>

<execution_context>
@/home/ollie/.claude/get-shit-done/workflows/execute-plan.md
@/home/ollie/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/milestones/v2-ROADMAP.md
@.planning/STATE.md
@.planning/phases/17-schema-v4-chunk-data-layer/17-CONTEXT.md
@.planning/phases/17-schema-v4-chunk-data-layer/17-RESEARCH.md
@.planning/phases/17-schema-v4-chunk-data-layer/17-01-SUMMARY.md
@src/corpus_analyzer/store/schema.py
@src/corpus_analyzer/ingest/chunker.py
@src/corpus_analyzer/ingest/ts_chunker.py
@src/corpus_analyzer/ingest/indexer.py
@tests/store/test_schema.py
@tests/ingest/test_chunker.py
@tests/ingest/test_round_trip.py
</context>

<feature>
  <name>ChunkRecord v4 implementation — schema + chunkers + indexer</name>
  <files>
    src/corpus_analyzer/store/schema.py,
    src/corpus_analyzer/ingest/chunker.py,
    src/corpus_analyzer/ingest/ts_chunker.py,
    src/corpus_analyzer/ingest/indexer.py
  </files>
  <behavior>
    GREEN state — all RED tests from 17-01 passing.
    Follow the RESEARCH.md patterns exactly. Implement in order: schema → chunkers → indexer.
  </behavior>
  <implementation>
    See tasks below. Commit after GREEN: `feat(17-02): implement ChunkRecord v4 fields, ensure_schema_v4, wire all chunkers`
  </implementation>
</feature>

<tasks>

<task type="auto">
  <name>Task 1: GREEN — Add ensure_schema_v4() and ChunkRecord v4 fields to schema.py</name>
  <files>src/corpus_analyzer/store/schema.py</files>
  <action>
    Two additions to store/schema.py:

    ADDITION 1 — ensure_schema_v4() function after ensure_schema_v3():

    ```python
    def ensure_schema_v4(table: lancedb.table.Table) -> None:
        """Add Phase 17 v4 columns to an existing chunks table if they don't exist.

        Adds chunk_name and chunk_text with empty-string defaults.
        start_line and end_line already exist (created with ChunkRecord schema).
        Idempotent: re-running on an already-migrated table is a no-op.

        Args:
            table: The LanceDB chunks table to upgrade.
        """
        existing_cols = {field.name for field in table.schema}
        if "chunk_name" not in existing_cols:
            table.add_columns({"chunk_name": "cast('' as string)"})
        if "chunk_text" not in existing_cols:
            table.add_columns({"chunk_text": "cast('' as string)"})
    ```

    ADDITION 2 — Two new fields at the end of ChunkRecord class after the Phase 6 fields block:

    ```python
    # --- Phase 17 fields (non-nullable, empty-string defaults for existing rows) ---
    chunk_name: str = ""
    """Heading text (Markdown), function/class name (Python), or construct name (TypeScript).

    Empty string for chunks indexed before Phase 17.
    """

    chunk_text: str = ""
    """Raw file slice from start_line to end_line (1-indexed, inclusive).

    Exact source text — comments, docstrings, and formatting intact. No size cap.
    Empty string for chunks indexed before Phase 17.
    """
    ```

    Also update the module docstring's "Schema lock-in warning" note to mention Phase 17 fields.

    CRITICAL: Do NOT change start_line or end_line — they already exist as required int fields and must NOT be altered.
  </action>
  <verify>
    <automated>cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer && uv run pytest tests/store/test_schema.py -v --tb=short 2>&1 | tail -20</automated>
    <manual>All TestChunkRecordInstantiation tests pass. TestEnsureSchemaV4 tests pass. test_all_required_fields_present passes.</manual>
  </verify>
  <done>All tests in tests/store/test_schema.py GREEN; ensure_schema_v4 importable and idempotent; chunk_name and chunk_text in ChunkRecord.model_fields with str type and "" default</done>
</task>

<task type="auto">
  <name>Task 2: GREEN — Emit chunk_name and chunk_text from all chunkers; update indexer pass-through</name>
  <files>
    src/corpus_analyzer/ingest/chunker.py
    src/corpus_analyzer/ingest/ts_chunker.py
    src/corpus_analyzer/ingest/indexer.py
  </files>
  <action>
    Four targeted changes across three files:

    CHANGE 1 — chunker.py: chunk_markdown — add chunk_name and chunk_text to every appended chunk dict.

    Read chunk_markdown carefully. There are two append sites: the normal per-section append and the preamble-merge path that modifies chunks[0]. For each section chunk:
    - chunk_name = lines[start_idx] (verbatim heading line text, e.g. "# Overview"; this is 0-indexed into the lines list)
    - chunk_text = section_text (the full section text already computed; same value as "text" — set it BEFORE any other mutation)

    For the preamble-merge path (where preamble content is prepended to the first chunk): chunk_name remains the first heading's text; chunk_text is the merged text at that point. Set chunk_text to the merged section_text value.

    Also handle the "no headings" fallback path — chunk_name = "" (no heading to use), chunk_text = section_text.

    CHANGE 2 — chunker.py: chunk_python — add chunk_name and chunk_text to every appended chunk dict.

    In chunk_python, for each node (FunctionDef, AsyncFunctionDef, ClassDef):
    - chunk_name = node.name (the AST .name attribute — same for functions and classes)
    - chunk_text = chunk_text (the local variable already computed as "\n".join(lines[start_line-1:end_line]).rstrip() — same value as "text")

    CHANGE 3 — chunker.py: _enforce_char_limit — carry chunk_name and chunk_text through to sub-chunks.

    When _enforce_char_limit creates sub-chunks (splits an oversized chunk), each new dict must include:
    - "chunk_name": chunk.get("chunk_name", "")  — parent's name carried unchanged
    - "chunk_text": chunk.get("chunk_text", "")  — full parent text carried unchanged (not the sub-chunk slice)

    There are multiple dict-append sites in _enforce_char_limit (the normal split path, the extremely-long-line path, and the flush-at-end path). Add chunk_name and chunk_text to ALL of them. Pass-through from parent chunk dict using .get() with "" default.

    CHANGE 4 — ts_chunker.py: chunk_typescript — add chunk_text key.

    chunk_typescript already emits chunk_name. Read the file and find every chunks.append() call. Add "chunk_text": chunk_text to each. The local variable chunk_text (the raw slice) is already computed — use it verbatim. Also add chunk_text to the export-default branch (around line 151–159 based on grep output). The export-default branch builds chunk_text via "\n".join(...); add "chunk_text": chunk_text to that dict too.

    CHANGE 5 — indexer.py: update imports + chunk_dict + ensure_schema_v4 call.

    In the import from corpus_analyzer.store.schema, add ensure_schema_v4 to the import list (currently imports ensure_schema_v2 and ensure_schema_v3).

    In CorpusIndex.open() or wherever ensure_schema_v2 and ensure_schema_v3 are called (around line 150), add a call to ensure_schema_v4(table) immediately after ensure_schema_v3(table).

    In the chunk_dict construction (around line 436–454), add two new keys after "summary":
    ```python
    "chunk_name": chunk.get("chunk_name", ""),
    "chunk_text": chunk.get("chunk_text", ""),
    ```

    CRITICAL PITFALL — chunk_text vs summary prepend:
    The indexer modifies chunks[0]["text"] with a summary prefix: chunks[0]["text"] = f"{summary_text}\n\n{chunks[0]['text']}". chunk_text must NOT be derived from chunk["text"] after this mutation. chunk.get("chunk_text", "") reads from the chunker-set value which is immutable by this path — safe.

    CRITICAL PITFALL — do NOT apply _enforce_char_limit to chunk_text. The existing _enforce_char_limit is called on chunks before embedding and the chunk_text should flow through as-is from each chunker-set dict, carried by _enforce_char_limit itself (Change 3 above).

    After all changes, run ruff check src/ and fix any violations (line length, unused imports, etc.) before running tests.
  </action>
  <verify>
    <automated>cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer && uv run pytest tests/ -v --tb=short 2>&1 | tail -40</automated>
    <manual>
      1. All new RED tests from 17-01 now GREEN.
      2. All 320 pre-existing tests still GREEN.
      3. Run: uv run ruff check . — must exit 0 (zero violations).
      4. Run: uv run mypy src/ — must exit 0 (zero errors).
      5. Check round-trip test specifically: uv run pytest tests/ingest/test_round_trip.py -v
    </manual>
  </verify>
  <done>
    Full test suite GREEN (320 pre-existing + all new v4 tests).
    ruff check . exits 0.
    mypy src/ exits 0.
    chunk_name and chunk_text populated correctly in LanceDB for .md, .py, .ts fixtures in round-trip test.
    commit `feat(17-02): implement ChunkRecord v4 fields, ensure_schema_v4, wire all chunkers` created.
  </done>
</task>

</tasks>

<verification>
Final quality gate after GREEN:

```bash
cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer
uv run ruff check .
uv run mypy src/
uv run pytest --tb=no -q 2>&1 | tail -5
```

All three commands must exit 0 / report zero failures.

Round-trip verification — confirm zero-hallucination contract:
```bash
uv run pytest tests/ingest/test_round_trip.py -v
```
All three parametrised variants (md, py, ts) must PASS.
</verification>

<success_criteria>
- All tests from Plan 17-01 RED state are now GREEN
- ChunkRecord has chunk_name: str = "" and chunk_text: str = "" fields
- ensure_schema_v4() adds chunk_name and chunk_text columns; idempotent (double-call safe)
- chunk_markdown returns chunk_name (heading line) and chunk_text (section text incl. heading)
- chunk_python returns chunk_name (AST node.name) and chunk_text (raw lines slice)
- chunk_typescript returns chunk_text (same as text before summary prepend) in addition to existing chunk_name
- _enforce_char_limit carries chunk_name and chunk_text through to all sub-chunks
- Indexer stores chunk_name and chunk_text via chunk.get() pass-through
- Indexer calls ensure_schema_v4(table) on open
- round-trip test: .md chunk_name="# Section One", start_line=1, chunk_text starts with "# Section One"
- round-trip test: .py chunk_name="greet", start_line=1, chunk_text starts with "def greet"
- round-trip test: .ts chunk_name="hello", start_line=1, chunk_text starts with "export function hello"
- `uv run ruff check .` exits 0
- `uv run mypy src/` exits 0
- Total test count: 320 + new tests, all passing
</success_criteria>

<output>
After completion, create `.planning/phases/17-schema-v4-chunk-data-layer/17-02-SUMMARY.md`
</output>
