---
phase: 01-foundation
plan: 03
type: tdd
wave: 1
depends_on: []
files_modified:
  - src/corpus_analyzer/ingest/__init__.py
  - src/corpus_analyzer/ingest/chunker.py
  - src/corpus_analyzer/ingest/scanner.py
  - tests/ingest/__init__.py
  - tests/ingest/test_chunker.py
  - tests/ingest/test_scanner.py
autonomous: true
requirements:
  - INGEST-02
  - INGEST-03
  - INGEST-04
  - INGEST-05

must_haves:
  truths:
    - "chunk_markdown() splits a markdown file on ATX headings and returns dicts with text/start_line/end_line"
    - "chunk_python() returns one chunk per top-level function/class using AST walk, with correct line ranges"
    - "chunk_lines() splits any file into N-line chunks with overlap, returning dicts with text/start_line/end_line"
    - "chunk_file() dispatches to the correct chunker by file extension"
    - "Large heading sections exceeding the token limit are sub-split with chunk_lines() fallback"
    - "file_content_hash() returns a stable sha256 hex string for a given file"
    - "needs_reindex() returns False when mtime is unchanged, True when mtime changed AND hash changed"
    - "walk_source() yields only files matching include patterns and not matching exclude patterns"
  artifacts:
    - path: "src/corpus_analyzer/ingest/chunker.py"
      provides: "chunk_markdown, chunk_python, chunk_lines, chunk_file"
      exports: ["chunk_markdown", "chunk_python", "chunk_lines", "chunk_file"]
    - path: "src/corpus_analyzer/ingest/scanner.py"
      provides: "walk_source, file_content_hash, needs_reindex"
      exports: ["walk_source", "file_content_hash", "needs_reindex"]
    - path: "tests/ingest/test_chunker.py"
      provides: "Chunker correctness tests with inline fixture strings"
    - path: "tests/ingest/test_scanner.py"
      provides: "Scanner, hash, and change-detection tests using tmp_path"
  key_links:
    - from: "src/corpus_analyzer/ingest/chunker.py"
      to: "ast module"
      via: "ast.parse() + ast.walk()"
      pattern: "ast\\.walk"
    - from: "src/corpus_analyzer/ingest/chunker.py"
      to: "chunk_lines"
      via: "fallback when heading chunk exceeds max_words"
      pattern: "chunk_lines"
    - from: "src/corpus_analyzer/ingest/scanner.py"
      to: "hashlib"
      via: "hashlib.sha256()"
      pattern: "hashlib\\.sha256"
---

<objective>
Implement and TDD-verify the chunking pipeline (chunker.py) and file scanner/change-detection (scanner.py).

Purpose: These are pure functions with defined inputs and outputs — ideal TDD candidates. The chunker is the heart of INGEST-03; scanner.py implements INGEST-05's mtime+sha256 change detection. Both are independent of LanceDB and Ollama, so they can be developed in parallel with Plan 01 and Plan 02.

Output: src/corpus_analyzer/ingest/chunker.py and ingest/scanner.py, verified by passing tests.
</objective>

<execution_context>
@/home/ollie/.claude/get-shit-done/workflows/execute-plan.md
@/home/ollie/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/01-foundation/01-RESEARCH.md
@.planning/phases/01-foundation/01-CONTEXT.md
</context>

<feature>
  <name>File chunker and scanner with change detection</name>
  <files>
    src/corpus_analyzer/ingest/__init__.py
    src/corpus_analyzer/ingest/chunker.py
    src/corpus_analyzer/ingest/scanner.py
    tests/ingest/__init__.py
    tests/ingest/test_chunker.py
    tests/ingest/test_scanner.py
  </files>
  <behavior>
    chunker.py exports:
      chunk_markdown(path: Path, max_words: int = 512) -> list[dict[str, Any]]:
        Split on ATX headings (^#{1,6}\s). Each chunk = {"text": str, "start_line": int, "end_line": int}.
        If a heading section exceeds max_words words, sub-split using chunk_lines() with 50-line windows.
        First chunk includes content before the first heading (if any).

      chunk_python(path: Path) -> list[dict[str, Any]]:
        Parse with ast.parse(). Walk with ast.walk(). Yield one chunk per top-level
        FunctionDef, AsyncFunctionDef, or ClassDef. Each chunk = {"text": str, "start_line": int, "end_line": int}.
        Nested classes/functions inside a class are NOT yielded separately (walk yields them, but only include
        nodes whose parent is the module, not another class/function — use a direct child check).
        If no top-level defs found, fall back to chunk_lines().

      chunk_lines(path: Path, window: int = 50, overlap: int = 10) -> list[dict[str, Any]]:
        Split into overlapping windows of `window` lines with `overlap` lines of overlap.
        Each chunk = {"text": str, "start_line": int, "end_line": int}.

      chunk_file(path: Path) -> list[dict[str, Any]]:
        Dispatch: .md -> chunk_markdown, .py -> chunk_python, else -> chunk_lines.
        Supported extensions: .md, .py, .ts, .js, .json, .yaml, .yml

    scanner.py exports:
      file_content_hash(path: Path) -> str:
        Read file in binary mode, sha256, return hex digest.
        Reads in 65536-byte chunks to handle large files.

      needs_reindex(path: Path, stored_mtime: float, stored_hash: str) -> bool:
        Return False if os.stat(path).st_mtime == stored_mtime (fast path).
        Return True if mtime changed AND file_content_hash(path) != stored_hash.
        Return False if mtime changed but hash is the same (touch without content change).

      walk_source(source_path: Path, include: list[str], exclude: list[str]) -> Iterator[Path]:
        Recursively yield files under source_path that match any include pattern
        and do not match any exclude pattern. Use pathlib.Path.match() for patterns.
        Default include ["**/*"] yields all files. Default exclude [] skips nothing.

    Cases (chunker):
      chunk_markdown: file with 3 headings -> 3+ chunks, each with correct line ranges
      chunk_markdown: file with no headings -> 1 chunk containing all lines
      chunk_markdown: heading section > max_words -> multiple sub-chunks from that section
      chunk_python: file with 2 top-level functions -> 2 chunks
      chunk_python: file with 0 top-level defs -> falls back to chunk_lines output
      chunk_python: nested function inside class -> NOT a separate chunk
      chunk_lines: 120-line file with window=50, overlap=10 -> chunks at 1-50, 41-90, 81-120 (approx)
      chunk_file(".md") -> calls chunk_markdown; chunk_file(".py") -> calls chunk_python
      chunk_file(".ts") -> calls chunk_lines; chunk_file(".json") -> calls chunk_lines

    Cases (scanner):
      file_content_hash: same file content -> same hash; different content -> different hash
      needs_reindex: same mtime -> False (no stat call on hash)
      needs_reindex: different mtime, same content -> False
      needs_reindex: different mtime, different content -> True
      walk_source: include=["*.md"] only yields .md files
      walk_source: exclude=["**/node_modules/**"] skips node_modules directory
  </behavior>
  <implementation>
    1. Create src/corpus_analyzer/ingest/__init__.py (empty)
    2. Create src/corpus_analyzer/ingest/chunker.py — implement chunk_markdown, chunk_python, chunk_lines, chunk_file
       - chunk_python: to get only top-level nodes, iterate ast.parse(source).body directly (not ast.walk),
         filtering for FunctionDef, AsyncFunctionDef, ClassDef — this is simpler and correct
       - All functions: full type annotations, PEP 257 docstrings, return type list[dict[str, Any]]
    3. Create src/corpus_analyzer/ingest/scanner.py — implement file_content_hash, needs_reindex, walk_source
       - Use typing.Iterator[Path] return type for walk_source
    4. Create tests/ingest/__init__.py
    5. Create tests/ingest/test_chunker.py — use inline strings as fixtures (no temp files needed for chunker)
       - For chunk_python tests: use ast.parse on inline source strings; write to tmp_path for Path-based API
       - For chunk_markdown: write fixture .md content to tmp_path
    6. Create tests/ingest/test_scanner.py — all file operations use pytest tmp_path fixture

    Follow RED-GREEN-REFACTOR cycle.
    Note: chunk_python should iterate source.body (top-level nodes), NOT ast.walk() — the research example uses
    ast.walk() which includes nested nodes; the plan requires top-level only (better chunking semantics).
  </implementation>
</feature>

<verification>
uv run pytest tests/ingest/ -v
uv run mypy src/corpus_analyzer/ingest/chunker.py src/corpus_analyzer/ingest/scanner.py
uv run ruff check src/corpus_analyzer/ingest/
</verification>

<success_criteria>
All tests in tests/ingest/ pass. mypy reports no errors. chunk_file() dispatches correctly to all supported extensions. needs_reindex() fast-path (mtime unchanged) skips hash computation.
</success_criteria>

<output>
After completion, create .planning/phases/01-foundation/01-03-SUMMARY.md
</output>
