---
phase: 05-extension-filtering
plan: 02
type: execute
wave: 2
depends_on:
  - "05-01"
files_modified:
  - src/corpus_analyzer/ingest/indexer.py
  - src/corpus_analyzer/cli.py
  - tests/ingest/test_indexer.py
autonomous: true
requirements:
  - CONF-07
  - CONF-08

must_haves:
  truths:
    - "corpus index skips files not in the extensions allowlist for each source, silently"
    - "Files previously indexed but now excluded by extension config are removed from the index"
    - "A removal count message is printed: 'Removed N files no longer in extension allowlist.' (only when N > 0)"
    - "On first index of a source (or when default allowlist is active), the active extensions list is displayed so user knows what defaults are in effect"
    - "Calling corpus index twice with no config change does not trigger spurious removals"
  artifacts:
    - path: "src/corpus_analyzer/ingest/indexer.py"
      provides: "index_source passes SourceConfig.extensions to walk_source and reconciles stale extension-excluded chunks"
      contains: "extensions"
    - path: "src/corpus_analyzer/cli.py"
      provides: "index_command prints active extensions on first run and removal count when files removed"
      contains: "extensions"
    - path: "tests/ingest/test_indexer.py"
      provides: "Tests verifying extension-excluded files are skipped and stale chunks removed"
      contains: "test_extension"
  key_links:
    - from: "src/corpus_analyzer/ingest/indexer.py"
      to: "src/corpus_analyzer/ingest/scanner.py"
      via: "walk_source called with source.extensions"
      pattern: "walk_source.*extensions|extensions.*walk_source"
    - from: "src/corpus_analyzer/ingest/indexer.py"
      to: "LanceDB table"
      via: "_delete_stale_chunks removes extension-excluded files"
      pattern: "_delete_stale_chunks|delete.*extension"
    - from: "src/corpus_analyzer/cli.py"
      to: "src/corpus_analyzer/ingest/indexer.py"
      via: "IndexResult exposes files_removed count"
      pattern: "files_removed"
---

<objective>
Wire SourceConfig.extensions into the indexer pipeline and add reconciliation + CLI output for extension changes.

Purpose: Make CONF-07 and CONF-08 observable — skipping non-allowlisted files and removing stale indexed chunks when config changes. Adds user-visible feedback when defaults are active and when files are removed.
Output: indexer passes extensions to walk_source; stale extension-excluded chunks deleted on each run; CLI prints active extensions on first index and removal counts when non-zero.
</objective>

<execution_context>
@/home/ollie/.claude/get-shit-done/workflows/execute-plan.md
@/home/ollie/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/05-extension-filtering/05-CONTEXT.md
@.planning/phases/05-extension-filtering/05-01-SUMMARY.md
@src/corpus_analyzer/ingest/indexer.py
@src/corpus_analyzer/cli.py
@src/corpus_analyzer/config/schema.py
@tests/ingest/test_indexer.py
</context>

<tasks>

<task type="auto">
  <name>Task 1: Wire extensions into indexer — filter + reconcile stale extension-excluded chunks</name>
  <files>src/corpus_analyzer/ingest/indexer.py, tests/ingest/test_indexer.py</files>
  <action>
    Make two changes to indexer.py and add tests for both.

    **Change 1: Pass source.extensions to walk_source in index_source()**

    In `index_source()`, find the `walk_source(source_path, source.include, source.exclude)` call and
    add the extensions parameter:
    ```python
    for file_path in walk_source(
        source_path,
        source.include,
        source.exclude,
        extensions=source.extensions if source.extensions else None,
    ):
    ```
    Note: If source.extensions is [] (empty list), pass [] not None — empty list correctly yields nothing.
    Revise to:
    ```python
    for file_path in walk_source(
        source_path,
        source.include,
        source.exclude,
        extensions=source.extensions,  # always pass — empty list means skip all
    ):
    ```
    This is correct because walk_source(extensions=None) = no filter (backward compat), but
    SourceConfig.extensions always has a value (either default or user-specified), so we always pass it.

    Also apply the same fix to the `total = sum(1 for _ in walk_source(...))` call in cli.py —
    but that's in the CLI task (Task 2). For indexer, the key call is in index_source().

    **Change 2: Add files_removed to IndexResult and reconcile extension-excluded chunks**

    Update `IndexResult` dataclass to include:
    ```python
    files_removed: int = 0
    """Number of files removed from index because they are no longer in the extension allowlist."""
    ```

    In `index_source()`, after the walk loop and before `_delete_stale_chunks()`, track which files
    in `existing_files` are now excluded by the extension filter. These are files that exist on disk
    but whose extension is no longer allowed. They won't appear in `current_file_hashes` (because
    walk_source skipped them), but they're still in the index under this source.

    The existing `_delete_stale_chunks()` already deletes files not in `current_file_hashes` — this
    naturally handles extension-excluded files because walk_source won't yield them, so they won't
    be in current_file_hashes, and _delete_stale_chunks will remove their chunks.

    Count the removals: compare `existing_files` keys against `current_file_hashes` keys. Files
    in existing_files but not in current_file_hashes will be deleted. Capture this count.

    ```python
    # Count files that will be removed (stale or extension-excluded)
    files_to_remove = set(existing_files.keys()) - set(current_file_hashes.keys())
    files_removed = len(files_to_remove)
    ```

    Add `files_removed=files_removed` to the IndexResult returned at the end.

    **Tests to add in tests/ingest/test_indexer.py:**

    Look at the existing test structure (mock-based or real LanceDB). Add tests that:
    - test_extension_filter_skips_excluded: mock walk_source to confirm it's called with extensions=source.extensions
    - OR create a real source with .md and .sh files, index with extensions=[".md"], confirm .sh not indexed (check chunks_written)
    - test_files_removed_counted: index a source, then change extensions to exclude previously-indexed type, re-index, confirm files_removed > 0 in result

    Use the existing test patterns (mock or real). If the existing tests mock walk_source or use real tmp_path + mock embedder, follow that pattern exactly. Do not introduce new test infrastructure.

    Run: `uv run pytest tests/ingest/test_indexer.py -x -v` after changes.
  </action>
  <verify>
    <automated>cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer && uv run pytest tests/ingest/test_indexer.py -x -v 2>&1 | tail -30</automated>
    <manual>New extension-related tests pass. IndexResult has files_removed field.</manual>
    <sampling_rate>run after this task completes</sampling_rate>
  </verify>
  <done>walk_source in index_source() called with source.extensions. IndexResult.files_removed populated. All indexer tests pass.</done>
</task>

<task type="auto">
  <name>Task 2: CLI output — active extensions on first run, removal count when non-zero</name>
  <files>src/corpus_analyzer/cli.py</files>
  <action>
    Update `index_command()` in cli.py with two output additions per the locked decisions in 05-CONTEXT.md.

    **Change 1: Fix total file count to use source.extensions**

    The `total = sum(1 for _ in walk_source(source_path, source.include, source.exclude))` count
    does not yet pass extensions. Update to:
    ```python
    total = sum(
        1 for _ in walk_source(source_path, source.include, source.exclude, extensions=source.extensions)
    )
    ```
    This ensures the progress bar total matches the files that will actually be processed.

    **Change 2: Display active extensions on first index of a source**

    Determine "first index" by checking if the source has any existing chunks before indexing.
    Use the existing `index._get_existing_files(source.name)` (or equivalent) to check. If the
    result is empty (no existing files for this source), this is the first run.

    After validating the source path exists and before the progress bar, add:
    ```python
    # Show active extensions on first index so user can adjust corpus.toml if needed
    is_first_run = len(index._get_existing_files(source.name)) == 0
    if is_first_run:
        ext_str = ", ".join(source.extensions) if source.extensions else "(none — all files skipped)"
        console.print(
            f"[dim]Active extensions for {source.name}: {ext_str}[/dim]\n"
            f"[dim]Add 'extensions = [...]' to this source in corpus.toml to customize.[/dim]"
        )
    ```

    **Change 3: Print removal count after indexing (only when > 0)**

    After the progress bar completes, the `result = index.index_source(...)` call returns IndexResult.
    After the existing summary print, add:
    ```python
    if result.files_removed > 0:
        console.print(
            f"[yellow]Removed {result.files_removed} file(s) no longer in extension allowlist.[/]"
        )
    ```

    Place this immediately after the existing `console.print(f"[green]✔[/] Indexed ...")` line.

    **Edge case handling (Claude's Discretion per 05-CONTEXT.md):**
    - Empty extensions list []: walk_source yields nothing, total=0, the existing
      "No files found" warning fires — no special handling needed.
    - Extension without leading dot: already normalized by SourceConfig validator (Plan 01).
    - Case sensitivity: already handled by walk_source `path.suffix.lower()` comparison (Plan 01).

    Run ruff and mypy on cli.py after changes.
  </action>
  <verify>
    <automated>cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer && uv run ruff check src/corpus_analyzer/cli.py && uv run mypy src/corpus_analyzer/cli.py 2>&1 | tail -20</automated>
    <manual>
      1. Check that `walk_source(..., extensions=source.extensions)` appears in both the total count and the indexer call site in cli.py
      2. Check that first-run extension display and removal count messages are present in index_command
    </manual>
    <sampling_rate>run after this task completes</sampling_rate>
  </verify>
  <done>cli.py passes ruff and mypy. index_command uses source.extensions in walk_source total count. First-run extension display and removal count messages added.</done>
</task>

<task type="auto">
  <name>Task 3: Full suite verification — no regressions across all tests</name>
  <files></files>
  <action>
    Run the complete test suite and type/lint checks to confirm no regressions.

    ```bash
    cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer
    uv run ruff check src/
    uv run mypy src/
    uv run pytest -x --tb=short
    ```

    Fix any issues found. Common things to check:
    - IndexResult consumers (api/public.py, any other callers) may need updating if they access IndexResult fields by position — check with grep
    - If IndexResult is a dataclass with a new field `files_removed=0`, it has a default, so existing call sites that don't pass it are fine
    - The api/public.py `index()` function returns a dict — update it to include `files_removed` if present in IndexResult

    Check api/public.py index() function:
    - The returned dict currently has: source_name, files_indexed, chunks_written, files_skipped, elapsed
    - Add files_removed to the dict to expose it via the public API

    After all checks pass, commit the plan 02 changes.
  </action>
  <verify>
    <automated>cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer && uv run ruff check src/ && uv run mypy src/ && uv run pytest -x --tb=short 2>&1 | tail -30</automated>
    <manual>All checks green. No regressions.</manual>
    <sampling_rate>run as final verification for this plan</sampling_rate>
  </verify>
  <done>ruff exits 0, mypy exits 0, full pytest suite passes. Phase 05 implementation is complete and correct.</done>
</task>

</tasks>

<verification>
End-to-end verification checklist:
1. uv run pytest tests/config/test_schema.py -v — SourceConfig.extensions field tests pass
2. uv run pytest tests/ingest/test_scanner.py -v — walk_source extension filter tests pass
3. uv run pytest tests/ingest/test_indexer.py -v — indexer extension filter + removal tests pass
4. uv run ruff check src/ — no lint errors
5. uv run mypy src/ — no type errors
6. uv run pytest -x — full suite, no regressions

Manual smoke test (if Ollama available):
- Add extensions = [".md"] to a source in corpus.toml
- Run corpus index — only .md files appear in indexed count
- Remove extensions key — re-run corpus index — default allowlist shown, .md/.py/.ts etc indexed
- Grep for ".sh" files in index results — none returned
</verification>

<success_criteria>
- CONF-06: SourceConfig accepts extensions field and stores it validated
- CONF-07: corpus index skips files not in extensions list per source
- CONF-08: Default allowlist of .md/.py/.ts/.tsx/.js/.jsx/.yaml/.yml/.txt applies when extensions not set
- Reconciliation: files removed from index when extension config narrows — count printed
- First-run display: active extensions printed on first index of a source
- All tests pass, ruff and mypy clean
</success_criteria>

<output>
After completion, create .planning/phases/05-extension-filtering/05-02-SUMMARY.md
</output>
