---
phase: 05-extension-filtering
plan: 01
type: tdd
wave: 1
depends_on: []
files_modified:
  - src/corpus_analyzer/config/schema.py
  - src/corpus_analyzer/ingest/scanner.py
  - tests/config/test_schema.py
  - tests/ingest/test_scanner.py
autonomous: true
requirements:
  - CONF-06
  - CONF-08

must_haves:
  truths:
    - "SourceConfig accepts extensions = [\".md\", \".py\"] and stores it as a list"
    - "SourceConfig with no extensions field gets the default allowlist (.md, .py, .ts, .tsx, .js, .jsx, .yaml, .yml, .txt)"
    - "walk_source filters files to only those whose suffix is in the extensions list"
    - "An empty extensions list [] is treated gracefully (no files yielded)"
    - "Extension strings are normalized to lowercase with a leading dot"
  artifacts:
    - path: "src/corpus_analyzer/config/schema.py"
      provides: "SourceConfig.extensions field with default allowlist"
      contains: "extensions"
    - path: "src/corpus_analyzer/ingest/scanner.py"
      provides: "walk_source accepts and applies extensions filter"
      contains: "extensions"
    - path: "tests/config/test_schema.py"
      provides: "Tests for SourceConfig.extensions field"
      contains: "test_extensions"
    - path: "tests/ingest/test_scanner.py"
      provides: "Tests for walk_source extension filtering"
      contains: "test_extension"
  key_links:
    - from: "src/corpus_analyzer/config/schema.py"
      to: "src/corpus_analyzer/ingest/scanner.py"
      via: "SourceConfig.extensions passed to walk_source"
      pattern: "extensions.*walk_source|walk_source.*extensions"
---

<objective>
Add per-source extension allowlist to SourceConfig and wire it into walk_source() via TDD.

Purpose: Establish the data model and core filtering function that the indexer integration (Plan 02) will consume. Testing this boundary first ensures the filtering contract is correct before integrating with the indexer.
Output: SourceConfig.extensions field with sensible defaults + walk_source() that honors the allowlist + tests proving both work.
</objective>

<execution_context>
@/home/ollie/.claude/get-shit-done/workflows/execute-plan.md
@/home/ollie/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/05-extension-filtering/05-CONTEXT.md
@src/corpus_analyzer/config/schema.py
@src/corpus_analyzer/ingest/scanner.py
@tests/config/test_schema.py
@tests/ingest/test_scanner.py
</context>

<feature>
  <name>Extension allowlist on SourceConfig + walk_source filtering</name>
  <files>
    src/corpus_analyzer/config/schema.py,
    src/corpus_analyzer/ingest/scanner.py,
    tests/config/test_schema.py,
    tests/ingest/test_scanner.py
  </files>
  <behavior>
    SourceConfig.extensions:
    - When not provided: defaults to [".md", ".py", ".ts", ".tsx", ".js", ".jsx", ".yaml", ".yml", ".txt"]
    - When provided as [".MD", ".py"]: normalizes to [".md", ".py"] (lowercase, leading dot enforced)
    - When provided as ["md", "py"]: normalizes to [".md", ".py"] (adds leading dot if missing)
    - When provided as []: stores as [] (empty list — user explicitly opts out of all extensions)

    walk_source(base_path, include, exclude, extensions):
    - extensions=None or not passed: no extension filtering (backward-compatible, yields all include-matched files)
    - extensions=[".md", ".py"]: only yields files whose suffix is in the list (case-insensitive comparison)
    - extensions=[]: yields nothing (empty allowlist)
    - Extension filtering applied AFTER glob include/exclude checks

    Cases (for tests):
    - SourceConfig() with no extensions → extensions == [".md", ".py", ".ts", ".tsx", ".js", ".jsx", ".yaml", ".yml", ".txt"]
    - SourceConfig(extensions=[".md", ".py"]) → extensions == [".md", ".py"]
    - SourceConfig(extensions=["MD", "py"]) → extensions == [".md", ".py"]  (normalization)
    - SourceConfig(extensions=["md", ".PY"]) → extensions == [".md", ".py"]  (normalization)
    - SourceConfig(extensions=[]) → extensions == []
    - walk_source(dir, extensions=[".md"]) with .md and .py files → yields only .md
    - walk_source(dir, extensions=[".md", ".py"]) → yields .md and .py but not .sh
    - walk_source(dir, extensions=[]) → yields nothing
    - walk_source(dir, extensions=None) → yields all (no extension filter)
  </behavior>
  <implementation>
    1. In schema.py — add `extensions` field to SourceConfig:
       ```python
       DEFAULT_EXTENSIONS: list[str] = [
           ".md", ".py", ".ts", ".tsx", ".js", ".jsx", ".yaml", ".yml", ".txt"
       ]

       extensions: list[str] = Field(default_factory=lambda: list(DEFAULT_EXTENSIONS))
       """File extensions to index for this source. Defaults to common doc and code types."""
       ```
       Add a Pydantic field_validator that normalizes each entry: strip whitespace, lowercase,
       add leading dot if missing. Use `@field_validator("extensions", mode="before")` from pydantic.

    2. In scanner.py — add optional `extensions` parameter to walk_source():
       ```python
       def walk_source(
           base_path: Path,
           include: list[str] | None = None,
           exclude: list[str] | None = None,
           extensions: list[str] | None = None,
       ) -> Iterator[Path]:
       ```
       After the existing include/exclude check, add:
       ```python
       # Extension filter (applied after include/exclude)
       if extensions is not None and path.suffix.lower() not in extensions:
           continue
       ```

    Normalization validator on SourceConfig.extensions:
    ```python
    @field_validator("extensions", mode="before")
    @classmethod
    def normalize_extensions(cls, v: object) -> list[str]:
        if not isinstance(v, list):
            raise ValueError("extensions must be a list")
        result = []
        for ext in v:
            ext = str(ext).strip().lower()
            if not ext.startswith("."):
                ext = f".{ext}"
            result.append(ext)
        return result
    ```

    No new dependencies. All changes follow existing patterns (pydantic field_validator, walk_source signature extension).
  </implementation>
</feature>

<tasks>

<task type="tdd">
  <name>Task 1: RED — Write failing tests for SourceConfig.extensions and walk_source filtering</name>
  <files>tests/config/test_schema.py, tests/ingest/test_scanner.py</files>
  <action>
    Add failing tests BEFORE changing any source code. Run tests to confirm they fail (RED).

    In tests/config/test_schema.py, add to TestSourceConfig class:
    - test_extensions_default: SourceConfig(name="x", path="/x").extensions equals [".md", ".py", ".ts", ".tsx", ".js", ".jsx", ".yaml", ".yml", ".txt"]
    - test_extensions_custom_explicit: SourceConfig(name="x", path="/x", extensions=[".md", ".py"]).extensions == [".md", ".py"]
    - test_extensions_normalizes_uppercase: SourceConfig(name="x", path="/x", extensions=[".MD", ".PY"]).extensions == [".md", ".py"]
    - test_extensions_normalizes_missing_dot: SourceConfig(name="x", path="/x", extensions=["md", "py"]).extensions == [".md", ".py"]
    - test_extensions_empty_list_allowed: SourceConfig(name="x", path="/x", extensions=[]).extensions == []
    - test_extensions_mixed_normalization: SourceConfig(name="x", path="/x", extensions=["MD", ".py", "ts"]).extensions == [".md", ".py", ".ts"]

    In tests/ingest/test_scanner.py, add to TestWalkSource class:
    - test_extensions_filter_single: dir with readme.md, script.py, run.sh; walk_source(dir, extensions=[".md"]) → [readme.md]
    - test_extensions_filter_multiple: dir with .md, .py, .sh files; walk_source(dir, extensions=[".md", ".py"]) → [.md, .py]
    - test_extensions_empty_yields_nothing: walk_source(dir, extensions=[]) → []
    - test_extensions_none_no_filter: walk_source(dir, extensions=None) → all files (backward compat)
    - test_extensions_case_insensitive: file named FILE.MD; walk_source(dir, extensions=[".md"]) → yields the file

    Commit: `test(05-01): add failing tests for SourceConfig.extensions and walk_source filtering`
  </action>
  <verify>
    <automated>cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer && uv run pytest tests/config/test_schema.py::TestSourceConfig::test_extensions_default tests/ingest/test_scanner.py::TestWalkSource::test_extensions_filter_single -x 2>&1 | tail -20</automated>
    <manual>Tests must FAIL (RED) — SourceConfig has no extensions field yet, walk_source has no extensions param</manual>
    <sampling_rate>run immediately after writing tests, before implementing</sampling_rate>
  </verify>
  <done>New tests exist and fail with AttributeError or similar — confirming RED state</done>
</task>

<task type="tdd">
  <name>Task 2: GREEN — Implement SourceConfig.extensions field and walk_source filtering</name>
  <files>src/corpus_analyzer/config/schema.py, src/corpus_analyzer/ingest/scanner.py</files>
  <action>
    Implement minimal code to make all tests pass.

    In src/corpus_analyzer/config/schema.py:
    1. Add module-level constant at top of file (after imports):
       ```python
       DEFAULT_EXTENSIONS: list[str] = [
           ".md", ".py", ".ts", ".tsx", ".js", ".jsx", ".yaml", ".yml", ".txt"
       ]
       ```
    2. Add import for field_validator: `from pydantic import BaseModel, Field, field_validator`
    3. Add to SourceConfig class after existing fields:
       ```python
       extensions: list[str] = Field(default_factory=lambda: list(DEFAULT_EXTENSIONS))
       """File extensions to index for this source (e.g. [".md", ".py"]).

       Defaults to common documentation and code types. Set to an empty list to skip
       all files in this source. The leading dot is required but normalized automatically.
       """

       @field_validator("extensions", mode="before")
       @classmethod
       def normalize_extensions(cls, v: object) -> list[str]:
           """Normalize extension strings: lowercase, ensure leading dot."""
           if not isinstance(v, list):
               raise ValueError("extensions must be a list of strings")
           result: list[str] = []
           for raw in v:
               ext = str(raw).strip().lower()
               if ext and not ext.startswith("."):
                   ext = f".{ext}"
               result.append(ext)
           return result
       ```

    In src/corpus_analyzer/ingest/scanner.py:
    1. Add `extensions` parameter to `walk_source()`:
       ```python
       def walk_source(
           base_path: Path,
           include: list[str] | None = None,
           exclude: list[str] | None = None,
           extensions: list[str] | None = None,
       ) -> Iterator[Path]:
       ```
    2. Update docstring to document the new parameter.
    3. After the existing include check (`if any(_match_glob...)`), add:
       ```python
       # Extension filter: applied after include/exclude
       if extensions is not None and path.suffix.lower() not in extensions:
           continue

       yield path
       ```
       (Remove or replace the existing bare `yield path` at the end of the if-block.)

    Run all tests. Fix mypy/ruff issues.
    Commit: `feat(05-01): add SourceConfig.extensions field and walk_source extension filtering`
  </action>
  <verify>
    <automated>cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer && uv run pytest tests/config/test_schema.py tests/ingest/test_scanner.py -x -v 2>&1 | tail -30</automated>
    <manual>All new extension tests pass (GREEN). Existing tests still pass — no regressions.</manual>
    <sampling_rate>run after implementation, before refactor step</sampling_rate>
  </verify>
  <done>All tests in test_schema.py and test_scanner.py pass. uv run pytest tests/config/ tests/ingest/test_scanner.py exits 0.</done>
</task>

<task type="tdd">
  <name>Task 3: REFACTOR + full suite — Clean up, type check, confirm no regressions</name>
  <files>src/corpus_analyzer/config/schema.py, src/corpus_analyzer/ingest/scanner.py</files>
  <action>
    Refactor for quality, then run the full test suite plus type/lint checks.

    Checks:
    1. Docstrings complete on new field and new validator — both have PEP 257 docstrings
    2. DEFAULT_EXTENSIONS constant has a module-level docstring comment explaining its purpose
    3. walk_source docstring updated to document `extensions` parameter in its Args section
    4. Type annotations correct: `extensions: list[str] | None = None` in walk_source signature
    5. No bare `str(raw)` without handling None — the validator casts via `str(raw)` which is acceptable
    6. Run ruff: `uv run ruff check src/corpus_analyzer/config/schema.py src/corpus_analyzer/ingest/scanner.py`
    7. Run mypy: `uv run mypy src/corpus_analyzer/config/schema.py src/corpus_analyzer/ingest/scanner.py`
    8. Run full test suite: `uv run pytest -x`

    Fix any issues found. Then commit:
    `refactor(05-01): clean up extensions field docs and type annotations`
  </action>
  <verify>
    <automated>cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer && uv run ruff check src/corpus_analyzer/config/schema.py src/corpus_analyzer/ingest/scanner.py && uv run mypy src/corpus_analyzer/config/schema.py src/corpus_analyzer/ingest/scanner.py && uv run pytest -x --tb=short 2>&1 | tail -20</automated>
    <manual>No ruff errors, no mypy errors, full test suite passes</manual>
    <sampling_rate>run after refactor, before closing plan</sampling_rate>
  </verify>
  <done>ruff exits 0, mypy exits 0, all tests pass. Plan 01 artifacts are committed.</done>
</task>

</tasks>

<verification>
Full verification:
- uv run pytest tests/config/test_schema.py -v (SourceConfig.extensions tests pass)
- uv run pytest tests/ingest/test_scanner.py -v (walk_source extension filter tests pass)
- uv run ruff check src/corpus_analyzer/config/schema.py src/corpus_analyzer/ingest/scanner.py
- uv run mypy src/corpus_analyzer/config/schema.py src/corpus_analyzer/ingest/scanner.py
- uv run pytest -x (full suite, no regressions)
</verification>

<success_criteria>
- SourceConfig.extensions field exists with DEFAULT_EXTENSIONS default
- Extension strings are normalized (lowercase, leading dot)
- walk_source filters by suffix when extensions is not None
- extensions=None preserves existing behavior (no filter)
- extensions=[] yields nothing
- All new tests pass, no existing tests broken
- ruff and mypy clean
</success_criteria>

<output>
After completion, create .planning/phases/05-extension-filtering/05-01-SUMMARY.md
</output>
