---
phase: 17-schema-v4-chunk-data-layer
plan: "01"
type: tdd
wave: 1
depends_on: []
files_modified:
  - tests/store/test_schema.py
  - tests/ingest/test_chunker.py
  - tests/ingest/test_round_trip.py
autonomous: true
requirements:
  - CHUNK-01

must_haves:
  truths:
    - "tests/store/test_schema.py::test_all_required_fields_present FAILS (chunk_name, chunk_text not in ChunkRecord yet)"
    - "tests/store/test_schema.py::TestEnsureSchemaV4 tests FAIL (ensure_schema_v4 does not exist yet)"
    - "tests/ingest/test_chunker.py chunk_name/chunk_text assertions FAIL for Markdown and Python chunkers"
    - "tests/ingest/test_round_trip.py round-trip assertions FAIL (fields not stored in LanceDB yet)"
    - "All 320 pre-existing tests remain green — RED state is localised to the new tests only"
  artifacts:
    - path: "tests/store/test_schema.py"
      provides: "Updated test_all_required_fields_present + new TestEnsureSchemaV4 class"
      contains: "chunk_name.*chunk_text"
    - path: "tests/ingest/test_chunker.py"
      provides: "Assertions that chunk_markdown and chunk_python return chunk_name and chunk_text keys"
      contains: "chunk_name.*chunk_text"
    - path: "tests/ingest/test_round_trip.py"
      provides: "Parametrised zero-hallucination round-trip test for Markdown, Python, TypeScript"
      contains: "test_round_trip"
  key_links:
    - from: "tests/store/test_schema.py"
      to: "corpus_analyzer.store.schema.ChunkRecord"
      via: "ChunkRecord.model_fields check"
      pattern: "chunk_name.*chunk_text.*model_fields"
    - from: "tests/ingest/test_round_trip.py"
      to: "corpus_analyzer.ingest.indexer.CorpusIndex"
      via: "index_source then direct LanceDB table query"
      pattern: "index\\.table\\.search"
---

<objective>
Write failing tests (RED state) for ChunkRecord v4 fields (`chunk_name`, `chunk_text`) and `ensure_schema_v4()` migration function, covering all three chunkers via a round-trip integration test.

Purpose: Establish the RED state that drives Plan 17-02 implementation. All new tests must FAIL before any implementation; all 320 existing tests must stay GREEN.
Output: Three test files with failing assertions for the v4 chunk data layer.
</objective>

<execution_context>
@/home/ollie/.claude/get-shit-done/workflows/execute-plan.md
@/home/ollie/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/milestones/v2-ROADMAP.md
@.planning/milestones/v2-REQUIREMENTS.md
@.planning/STATE.md
@.planning/phases/17-schema-v4-chunk-data-layer/17-CONTEXT.md
@.planning/phases/17-schema-v4-chunk-data-layer/17-RESEARCH.md
@src/corpus_analyzer/store/schema.py
@src/corpus_analyzer/ingest/chunker.py
@src/corpus_analyzer/ingest/ts_chunker.py
@src/corpus_analyzer/ingest/indexer.py
@tests/store/test_schema.py
@tests/ingest/test_chunker.py
</context>

<feature>
  <name>ChunkRecord v4 schema fields + ensure_schema_v4() + chunker v4 field emission</name>
  <files>
    tests/store/test_schema.py,
    tests/ingest/test_chunker.py,
    tests/ingest/test_round_trip.py
  </files>
  <behavior>
    After Phase 17 GREEN:
    - ChunkRecord has two new fields: chunk_name: str = "" and chunk_text: str = ""
    - ensure_schema_v4(table) adds "chunk_name" and "chunk_text" string columns to an existing LanceDB table; calling it twice on the same table is a no-op
    - chunk_markdown() returns dicts with "chunk_name" (heading line verbatim) and "chunk_text" (full section text including heading)
    - chunk_python() returns dicts with "chunk_name" (function/class name from AST) and "chunk_text" (raw lines slice)
    - chunk_typescript() already returns "chunk_name"; it must also return "chunk_text" (same as "text" before any summary prepend)
    - Round-trip: index a synthetic file, query LanceDB directly, assert start_line/end_line/chunk_name/chunk_text match expected values

    RED test cases to write:

    1. tests/store/test_schema.py — update test_all_required_fields_present:
       Add "chunk_name" and "chunk_text" to expected_fields set.
       This test WILL FAIL immediately (field not in ChunkRecord yet).

    2. tests/store/test_schema.py — new TestEnsureSchemaV4 class:
       - test_ensure_schema_v4_adds_chunk_name_column: open a real LanceDB table with v3 schema (using lancedb.connect in tmp_path), call ensure_schema_v4, assert "chunk_name" in table schema field names
       - test_ensure_schema_v4_adds_chunk_text_column: same, assert "chunk_text" in table schema field names
       - test_ensure_schema_v4_is_idempotent: call ensure_schema_v4 twice on same table, assert no RuntimeError is raised and both columns present
       These WILL FAIL (ensure_schema_v4 does not exist yet — ImportError at collection or NameError).

    3. tests/ingest/test_chunker.py — new assertions in TestChunkMarkdown and TestChunkPython:
       - TestChunkMarkdown: test_chunk_markdown_emits_chunk_name — given a file with headings, first chunk dict has "chunk_name" key equal to the heading line verbatim (e.g. "# Title")
       - TestChunkMarkdown: test_chunk_markdown_emits_chunk_text — chunk dict has "chunk_text" key equal to section text including heading
       - TestChunkPython: test_chunk_python_emits_chunk_name — given a .py file with a function def, chunk dict has "chunk_name" equal to the function name string (e.g. "my_function")
       - TestChunkPython: test_chunk_python_emits_chunk_text — chunk dict has "chunk_text" key containing the raw source lines slice
       These WILL FAIL (chunkers do not emit these keys yet).

    4. tests/ingest/test_round_trip.py — new module (create __init__.py if missing):
       Parametrised test hitting Markdown, Python, TypeScript fixtures:

       @pytest.mark.parametrize("fixture", ["md", "py", "ts"])
       def test_round_trip_chunk_fields(tmp_path, fixture):
         Write a known synthetic file (see fixture specs below), index via CorpusIndex using MockEmbedder (copy pattern from tests/ingest/test_indexer.py), query table directly, assert fields.

       Markdown fixture: "# Section One\n\nSome body text here.\n\n## Section Two\n\nMore content.\n"
         Expected for first chunk: start_line=1, end_line=3, chunk_name="# Section One", chunk_text starts with "# Section One"

       Python fixture: "def greet(name: str) -> str:\n    \"\"\"Greet.\"\"\"\n    return f'Hello {name}'\n\n\nclass Foo:\n    pass\n"
         Expected for "greet" chunk: chunk_name="greet", start_line=1, chunk_text starts with "def greet"

       TypeScript fixture: "export function hello(name: string): string {\n    return `Hello ${name}`;\n}\n"
         Expected for "hello" chunk: chunk_name="hello", start_line=1, chunk_text starts with "export function hello"

       These WILL FAIL (fields not stored in LanceDB yet — KeyError or assertion failure).

       IMPORTANT: MockEmbedder class should be imported from or pattern-matched against tests/ingest/test_indexer.py — do not duplicate independently; check if it can be imported from conftest or test_indexer. If not importable, define locally in test_round_trip.py with same contract (embed_batch returns list of [0.0]*768 vectors, model="test-model").
  </behavior>
  <implementation>
    RED phase only — do NOT implement any production code changes.
    Only modify test files to add failing assertions.

    Commit after RED confirmed: `test(17-01): add failing tests for ChunkRecord v4 fields and ensure_schema_v4`
  </implementation>
</feature>

<tasks>

<task type="auto">
  <name>Task 1: RED — Update test_schema.py and write TestEnsureSchemaV4</name>
  <files>tests/store/test_schema.py</files>
  <action>
    Make two changes to tests/store/test_schema.py:

    CHANGE 1 — Update test_all_required_fields_present (line ~121):
    Add "chunk_name" and "chunk_text" to the expected_fields set:
    ```python
    expected_fields = {
        "chunk_id", "file_path", "source_name", "text", "vector",
        "start_line", "end_line", "file_type", "content_hash",
        "embedding_model", "indexed_at", "construct_type", "summary",
        "classification_source", "classification_confidence",
        "chunk_name",   # Phase 17
        "chunk_text",   # Phase 17
    }
    ```
    This immediately turns the test RED (chunk_name and chunk_text not in ChunkRecord yet).

    CHANGE 2 — Add new TestEnsureSchemaV4 class after TestChunkRecordInstantiation:
    Import ensure_schema_v4 from corpus_analyzer.store.schema (add to existing import line).
    Import lancedb and create a minimal v3-schema table in tmp_path for testing ensure_schema_v4.

    The class needs 3 tests:
    - test_ensure_schema_v4_adds_chunk_name_column: connect lancedb in tmp_path, create table with a minimal v3 schema dict (use pa.schema with chunk_id:string, chunk_name NOT included), call ensure_schema_v4(table), assert "chunk_name" in {f.name for f in table.schema}
    - test_ensure_schema_v4_adds_chunk_text_column: same setup, assert "chunk_text" present
    - test_ensure_schema_v4_is_idempotent: call ensure_schema_v4(table) twice, assert no exception raised, both columns present

    For the minimal table creation in tmp_path, use pyarrow schema:
    ```python
    import pyarrow as pa
    import lancedb

    def _make_minimal_table(tmp_path):
        db = lancedb.connect(str(tmp_path / "test.lancedb"))
        schema = pa.schema([
            pa.field("chunk_id", pa.string()),
            pa.field("text", pa.string()),
        ])
        table = db.create_table("chunks", schema=schema)
        return table
    ```

    Each test method receives `tmp_path` fixture from pytest.

    The import of ensure_schema_v4 will cause ImportError/AttributeError at collection time if the function does not exist — this is the correct RED state for the ensure_schema_v4 tests.
  </action>
  <verify>
    <automated>cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer && uv run pytest tests/store/test_schema.py -v --tb=short 2>&1 | tail -30</automated>
    <manual>Confirm test_all_required_fields_present FAILS (missing chunk_name, chunk_text). Confirm TestEnsureSchemaV4 tests FAIL (ImportError or AttributeError for ensure_schema_v4). Confirm all other tests in test_schema.py still PASS.</manual>
  </verify>
  <done>test_all_required_fields_present is RED; TestEnsureSchemaV4 tests are RED; TestMakeChunkId and TestChunkRecordInstantiation remain GREEN</done>
</task>

<task type="auto">
  <name>Task 2: RED — Add chunk_name/chunk_text assertions to test_chunker.py and create test_round_trip.py</name>
  <files>
    tests/ingest/test_chunker.py
    tests/ingest/test_round_trip.py
  </files>
  <action>
    PART A — tests/ingest/test_chunker.py:

    Read the existing file first to understand TestChunkMarkdown and TestChunkPython class structure.

    Add to TestChunkMarkdown class (use tmp_path fixture):
    - test_chunk_markdown_emits_chunk_name: write a tmp_path file with "# My Heading\n\nBody text.\n", call chunk_markdown(path), assert chunks[0]["chunk_name"] == "# My Heading"
    - test_chunk_markdown_emits_chunk_text: same file, assert "chunk_text" in chunks[0] and chunks[0]["chunk_text"].startswith("# My Heading")

    Add to TestChunkPython class (use tmp_path fixture):
    - test_chunk_python_emits_chunk_name: write a tmp_path .py file with "def my_func(x: int) -> int:\n    return x + 1\n", call chunk_python(path), assert chunks[0]["chunk_name"] == "my_func"
    - test_chunk_python_emits_chunk_text: same, assert "chunk_text" in chunks[0] and "def my_func" in chunks[0]["chunk_text"]

    These 4 tests will FAIL because chunker.py does not yet emit these keys — KeyError on dict access.

    PART B — create tests/ingest/test_round_trip.py:

    Check if tests/ingest/__init__.py exists (it should — project standard). Create test_round_trip.py.

    Import pattern: look at tests/ingest/test_indexer.py for how MockEmbedder and CorpusIndex are imported and used. Import or reproduce MockEmbedder inline (a class with embed_batch returning [[0.0]*768]*len(texts) and model="test-model").

    Write parametrised test:
    ```python
    import pytest
    import lancedb

    from corpus_analyzer.ingest.indexer import CorpusIndex, SourceConfig

    class MockEmbedder:
        model = "test-model"
        def embed_batch(self, texts):
            return [[0.0] * 768 for _ in texts]

    @pytest.mark.parametrize("fixture_type,content,expected_chunk_name,expected_start_line,expected_chunk_text_prefix", [
        (
            ".md",
            "# Section One\n\nSome body text here.\n\n## Section Two\n\nMore content.\n",
            "# Section One",
            1,
            "# Section One",
        ),
        (
            ".py",
            "def greet(name: str) -> str:\n    \"\"\"Greet.\"\"\"\n    return f'Hello {name}'\n",
            "greet",
            1,
            "def greet",
        ),
        (
            ".ts",
            "export function hello(name: string): string {\n    return `Hello ${name}`;\n}\n",
            "hello",
            1,
            "export function hello",
        ),
    ])
    def test_round_trip_chunk_fields(
        tmp_path, fixture_type, content, expected_chunk_name, expected_start_line, expected_chunk_text_prefix
    ):
        # Write synthetic file
        source_dir = tmp_path / "source"
        source_dir.mkdir()
        test_file = source_dir / f"test{fixture_type}"
        test_file.write_text(content, encoding="utf-8")

        # Index via CorpusIndex
        index = CorpusIndex.open(tmp_path / "index", MockEmbedder())
        source = SourceConfig(name="test-source", path=str(source_dir))
        index.index_source(source)

        # Query LanceDB directly
        rows = index._table.search().limit(20).to_list()
        # Find the chunk for our file
        matching = [r for r in rows if test_file.name in r["file_path"]]
        assert matching, f"No chunks found for {fixture_type} file"

        # Find the specific chunk by expected chunk_name
        target = next((r for r in matching if r.get("chunk_name") == expected_chunk_name), None)
        assert target is not None, f"No chunk with chunk_name={expected_chunk_name!r}; found: {[r.get('chunk_name') for r in matching]}"
        assert target["start_line"] == expected_start_line
        assert target["chunk_text"].startswith(expected_chunk_text_prefix)
    ```

    This test WILL FAIL: "chunk_name" key missing from rows or chunk_text is empty string (default "" from schema default) — the fields are not populated yet.

    Note: Use `index._table` to access the underlying LanceDB table (check indexer.py for the actual attribute name — it may be `self._table` or `self.table`; grep for it).
  </action>
  <verify>
    <automated>cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer && uv run pytest tests/ingest/test_chunker.py tests/ingest/test_round_trip.py -v --tb=short 2>&1 | tail -40</automated>
    <manual>Confirm 4 new chunker tests FAIL (KeyError on chunk_name/chunk_text). Confirm round-trip tests FAIL. Confirm all 320 pre-existing tests still pass: uv run pytest --tb=no -q 2>&1 | tail -5</manual>
  </verify>
  <done>6+ new tests RED; existing 320 tests GREEN; full test suite count increases by the number of new failing tests</done>
</task>

</tasks>

<verification>
Run full suite after RED state confirmed:

```bash
cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer
uv run pytest --tb=no -q 2>&1 | tail -10
```

Expected: 320 passed + N failed (where N = new RED tests only). Zero pre-existing test regressions.

Commit: `test(17-01): add failing tests for ChunkRecord v4 fields and ensure_schema_v4`
</verification>

<success_criteria>
- test_all_required_fields_present in test_schema.py FAILS (chunk_name, chunk_text missing from ChunkRecord)
- TestEnsureSchemaV4 tests FAIL (ensure_schema_v4 not importable yet)
- 4 new TestChunkMarkdown/TestChunkPython tests FAIL (chunk_name/chunk_text keys absent from returned dicts)
- test_round_trip_chunk_fields[md], [py], [ts] all FAIL (fields not in LanceDB yet)
- All 320 pre-existing tests remain GREEN
- commit `test(17-01): add failing tests for ChunkRecord v4 fields and ensure_schema_v4` created
</success_criteria>

<output>
After completion, create `.planning/phases/17-schema-v4-chunk-data-layer/17-01-SUMMARY.md`
</output>
