---
phase: 04-hardening
plan: 02
type: execute
wave: 1
depends_on: []
files_modified:
  - src/corpus_analyzer/config/schema.py
  - src/corpus_analyzer/config/__init__.py
  - src/corpus_analyzer/cli.py
  - src/corpus_analyzer/api/public.py
  - src/corpus_analyzer/mcp/server.py
  - src/corpus_analyzer/ingest/indexer.py
  - src/corpus_analyzer/ingest/scanner.py
  - src/corpus_analyzer/search/engine.py
  - tests/config/test_schema.py
  - tests/ingest/test_scanner.py
  - .planning/phases/01-foundation/01-VERIFICATION.md
  - .planning/phases/02-search-core/02-VERIFICATION.md
  - .planning/phases/03-agent-interfaces/03-VERIFICATION.md
autonomous: true
requirements:
  - CLI-02
  - INGEST-07
  - MCP-02

must_haves:
  truths:
    - "`DATA_DIR` and `CONFIG_PATH` are defined once in `config/schema.py` — not in `cli.py` or `api/public.py`"
    - "`SourceConfig` has `use_llm_classification: bool = False` field and indexer uses it (not the hardcoded parameter default)"
    - "`needs_reindex()` function is removed from `ingest/scanner.py` — no callers exist"
    - "`CorpusSearch.__init__()` no longer calls `create_fts_index` — FTS is only built by `index_source()`"
    - "Retroactive VERIFICATION.md files exist for phases 1, 2, and 3 with correct frontmatter"
  artifacts:
    - path: "src/corpus_analyzer/config/schema.py"
      provides: "DATA_DIR, CONFIG_PATH constants + use_llm_classification field on SourceConfig"
      contains: "DATA_DIR"
    - path: "src/corpus_analyzer/ingest/scanner.py"
      provides: "File without needs_reindex() function"
    - path: "src/corpus_analyzer/search/engine.py"
      provides: "CorpusSearch.__init__ without create_fts_index call"
    - path: ".planning/phases/01-foundation/01-VERIFICATION.md"
      provides: "Retroactive requirements verification for phase 1"
    - path: ".planning/phases/02-search-core/02-VERIFICATION.md"
      provides: "Retroactive requirements verification for phase 2"
    - path: ".planning/phases/03-agent-interfaces/03-VERIFICATION.md"
      provides: "Retroactive requirements verification for phase 3"
  key_links:
    - from: "src/corpus_analyzer/cli.py"
      to: "src/corpus_analyzer/config/schema.py"
      via: "from corpus_analyzer.config.schema import CONFIG_PATH, DATA_DIR"
      pattern: "from corpus_analyzer\\.config\\.schema import.*DATA_DIR"
    - from: "src/corpus_analyzer/ingest/indexer.py"
      to: "source.use_llm_classification"
      via: "classify_file(..., use_llm=source.use_llm_classification)"
      pattern: "use_llm=source\\.use_llm_classification"
---

<objective>
Remove accumulated tech debt before the v1.0 milestone is archived: consolidate the DATA_DIR/CONFIG_PATH dual-definition, expose use_llm_classification as a config field, remove the needs_reindex() dead code, eliminate the redundant FTS rebuild in CorpusSearch.__init__(), and write retroactive VERIFICATION.md audit trail documents for phases 1-3.

Purpose: These items were deferred during earlier phases. They do not close safety gaps (those are in plan 04-01) but they eliminate confusion, dead code, and the silent "default-on LLM classification" footgun before the codebase is archived.
Output: Cleaner config module (single source of truth for paths), simpler scanner and search engine, a working use_llm_classification config knob, and three retroactive VERIFICATION.md files.
</objective>

<execution_context>
@/home/ollie/.claude/get-shit-done/workflows/execute-plan.md
@/home/ollie/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/04-hardening/04-CONTEXT.md
@.planning/phases/04-hardening/04-RESEARCH.md
</context>

<tasks>

<task type="auto">
  <name>Task 1: Consolidate DATA_DIR/CONFIG_PATH and expose use_llm_classification config field</name>
  <files>src/corpus_analyzer/config/schema.py
src/corpus_analyzer/config/__init__.py
src/corpus_analyzer/cli.py
src/corpus_analyzer/api/public.py
src/corpus_analyzer/mcp/server.py
src/corpus_analyzer/ingest/indexer.py
tests/config/test_schema.py</files>
  <action>
    **Step 1 — Move path constants to config/schema.py**

    Add to `config/schema.py` (after the existing imports, before class definitions):
    ```python
    from pathlib import Path
    from platformdirs import user_config_dir, user_data_dir

    CONFIG_PATH = Path(user_config_dir("corpus")) / "corpus.toml"
    DATA_DIR = Path(user_data_dir("corpus"))
    ```
    Note: `config/schema.py` currently only imports from `pydantic`. Adding `platformdirs` is a new import — the package is already in `pyproject.toml` (used in `cli.py`). No circular dependency risk: `config/schema.py` only imports stdlib + pydantic + platformdirs, none of which import from corpus_analyzer.

    **Step 2 — Update config/__init__.py exports**

    Add `CONFIG_PATH` and `DATA_DIR` to the `__all__` list in `config/__init__.py`:
    ```python
    from corpus_analyzer.config.schema import CorpusConfig, EmbeddingConfig, SourceConfig, CONFIG_PATH, DATA_DIR
    __all__ = [
        "CONFIG_PATH",
        "CorpusConfig",
        "DATA_DIR",
        "EmbeddingConfig",
        "SourceConfig",
        "load_config",
        "save_config",
        "settings",
    ]
    ```
    Export both together or neither — do not export only one.

    **Step 3 — Update cli.py**

    Remove the local definitions at cli.py lines ~35-36:
    ```python
    # REMOVE these two lines:
    CONFIG_PATH = Path(user_config_dir("corpus")) / "corpus.toml"
    DATA_DIR = Path(user_data_dir("corpus"))
    ```
    Add import from config.schema:
    ```python
    from corpus_analyzer.config.schema import CONFIG_PATH, DATA_DIR
    ```
    Also remove `user_config_dir` and `user_data_dir` from cli.py's imports if they are no longer used elsewhere in cli.py. Check: `user_config_dir` is used only for CONFIG_PATH; `user_data_dir` is used only for DATA_DIR. Both can be removed from cli.py imports after the move.

    **Step 4 — Update api/public.py**

    Find the inline `data_dir = Path(user_data_dir("corpus"))` at approximately line 68. Replace with:
    ```python
    from corpus_analyzer.config.schema import DATA_DIR
    # ... then use DATA_DIR directly instead of data_dir local variable
    index = CorpusIndex.open(DATA_DIR, embedder)
    ```
    Remove the `from platformdirs import user_data_dir` import if it is no longer used elsewhere in api/public.py.

    **Step 5 — Update mcp/server.py**

    Change line 18:
    ```python
    # Before:
    from corpus_analyzer.cli import CONFIG_PATH, DATA_DIR
    # After:
    from corpus_analyzer.config.schema import CONFIG_PATH, DATA_DIR
    ```

    **Step 6 — Add use_llm_classification to SourceConfig**

    In `config/schema.py`, add the new field to `SourceConfig` after the `summarize` field (following the exact same pattern):
    ```python
    use_llm_classification: bool = False
    """Whether to use LLM-based classification for construct type detection.

    When True, Ollama is called to classify each changed file during indexing.
    When False (default), rule-based classification is used.
    Set to True per source for richer classification when LLM cost is acceptable.
    """
    ```

    **Step 7 — Wire use_llm_classification through indexer.py**

    In `indexer.py` `index_source()`:
    1. Update the parameter default from `True` to `False`: `use_llm_classification: bool = False`
    2. Update the classify_file call to use the source config value:
       ```python
       construct_type = classify_file(
           file_path,
           full_text,
           model=self._embedder.model,
           use_llm=source.use_llm_classification,
       )
       ```
    The `use_llm_classification` parameter on `index_source()` becomes redundant once the source config value is used. Remove the parameter from the method signature to make `SourceConfig` the single control point (per locked decision). Verify no callers in cli.py or api/public.py pass this parameter explicitly — if they do, remove those call-site arguments too.

    **Step 8 — Add test to tests/config/test_schema.py**

    Add `test_source_config_use_llm_classification_defaults_false`:
    - Create a `SourceConfig(name="x", path="/tmp/x")` and assert `sc.use_llm_classification is False`.
    - Create `SourceConfig(name="x", path="/tmp/x", use_llm_classification=True)` and assert it is `True`.

    **Pitfall:** Avoid circular import. Confirm with quick mental trace: `config/schema.py` → platformdirs (not corpus_analyzer). Safe.
    **Pitfall:** `use_llm_classification` default mismatch — the parameter default in `index_source()` was `True`; the new config field default is `False`. After wiring `source.use_llm_classification`, the parameter can be removed entirely. If kept for backward compat, update its default to `False` to match.
  </action>
  <verify>
    <automated>uv run pytest tests/config/test_schema.py -x -q && uv run mypy src/corpus_analyzer/config/schema.py src/corpus_analyzer/cli.py src/corpus_analyzer/api/public.py src/corpus_analyzer/mcp/server.py src/corpus_analyzer/ingest/indexer.py && uv run ruff check src/corpus_analyzer/config/ src/corpus_analyzer/cli.py src/corpus_analyzer/api/public.py src/corpus_analyzer/mcp/server.py src/corpus_analyzer/ingest/indexer.py</automated>
    <manual>Grep confirms DATA_DIR is no longer defined in cli.py: `grep -n "DATA_DIR = " src/corpus_analyzer/cli.py` returns nothing</manual>
    <sampling_rate>run after this task commits, before next task begins</sampling_rate>
  </verify>
  <done>
    - `DATA_DIR` and `CONFIG_PATH` defined only in `config/schema.py`; all importers use `from corpus_analyzer.config.schema import ...`
    - `SourceConfig.use_llm_classification` field exists, defaults to `False`, is wired to `classify_file(..., use_llm=source.use_llm_classification)` in indexer
    - New config test passes; mypy and ruff clean on all five modified source files
    - Full test suite passes (no regressions from import chain changes)
  </done>
</task>

<task type="auto">
  <name>Task 2: Remove needs_reindex() dead code and redundant FTS rebuild</name>
  <files>src/corpus_analyzer/ingest/scanner.py
src/corpus_analyzer/search/engine.py
tests/ingest/test_scanner.py</files>
  <action>
    **Step 1 — Verify needs_reindex() is truly dead code**

    Before deletion, confirm no callers exist:
    ```bash
    grep -r "needs_reindex" src/corpus_analyzer/
    grep -r "needs_reindex" tests/
    ```
    Expected: only found in `ingest/scanner.py` (its own definition). If found in tests, those tests must also be deleted.

    **Step 2 — Remove needs_reindex() from scanner.py**

    Delete the function definition and its docstring (lines 33-56 approximately):
    ```python
    def needs_reindex(path: Path, stored_mtime: float, stored_hash: str) -> bool:
        ...
    ```
    The remaining file will contain: module docstring, imports, `file_content_hash()`, `walk_source()`, `_match_glob()`. Verify the file still parses cleanly.

    **Step 3 — Check and update tests/ingest/test_scanner.py**

    Open `tests/ingest/test_scanner.py`. If any test imports or directly tests `needs_reindex`, delete that test. It tests dead code. If no such test exists, no change needed.

    **Step 4 — Remove FTS rebuild from CorpusSearch.__init__()**

    In `search/engine.py`, remove the `create_fts_index` call from `__init__`:
    ```python
    # REMOVE this entire try/except block from __init__:
    try:
        self._table.create_fts_index("text", replace=True)
    except Exception as e:  # pragma: no cover
        raise RuntimeError(
            "Failed to create FTS index on 'text'. "
            "Hybrid search requires an FTS index for BM25."
        ) from e
    ```
    After removal, `__init__` should simply set `self._table = table` and `self._embedder = embedder` with no FTS call.

    Rationale: `create_fts_index` is already called at the end of every `index_source()` call (added in Phase 2, plan 02-05). Calling it again in `CorpusSearch.__init__()` is idempotent but wasteful — it rebuilds the FTS index every time a search engine is opened, including on every `corpus search` CLI invocation.

    Risk assessment: A freshly-opened table with zero records has nothing to FTS-search regardless. The existing test `test_index_rebuilds_fts_index` in `test_indexer.py` patches the table method and will remain valid — it only calls `index_source()`, not `CorpusSearch()`.

    Check `tests/search/test_engine.py` for any test that:
    - Patches or asserts `create_fts_index` on the table
    - Constructs `CorpusSearch` on an empty table expecting FTS to work
    If found, update those tests to match the new behaviour (FTS built only by indexer).

    **Step 5 — Run full suite**

    ```bash
    uv run pytest -x -q
    ```
    Confirm no regressions from either removal.
  </action>
  <verify>
    <automated>uv run pytest tests/ingest/test_scanner.py tests/search/ -x -q && uv run pytest -x -q && uv run mypy src/corpus_analyzer/ingest/scanner.py src/corpus_analyzer/search/engine.py && uv run ruff check src/corpus_analyzer/ingest/scanner.py src/corpus_analyzer/search/engine.py</automated>
    <manual>Confirm `grep -n "needs_reindex" src/corpus_analyzer/ingest/scanner.py` returns nothing; confirm `grep -n "create_fts_index" src/corpus_analyzer/search/engine.py` returns nothing</manual>
    <sampling_rate>run after this task commits, before next task begins</sampling_rate>
  </verify>
  <done>
    - `needs_reindex()` function absent from scanner.py; any test for it is also removed
    - `CorpusSearch.__init__()` contains no `create_fts_index` call
    - Full test suite passes with no regressions
    - mypy and ruff clean on both modified files
  </done>
</task>

<task type="auto">
  <name>Task 3: Write retroactive VERIFICATION.md files for phases 1, 2, and 3</name>
  <files>.planning/phases/01-foundation/01-VERIFICATION.md
.planning/phases/02-search-core/02-VERIFICATION.md
.planning/phases/03-agent-interfaces/03-VERIFICATION.md</files>
  <action>
    Create three retroactive VERIFICATION.md files, one per completed phase. Each file captures requirement-level verification status from the audit trail. Mark all files as retroactive.

    **Source material:**
    - Phase 1 requirements: CONF-01 through CONF-05, INGEST-01 through INGEST-07 (from ROADMAP.md Phase 1)
    - Phase 2 requirements: SEARCH-01 through SEARCH-05, CLASS-01 through CLASS-03, SUMM-01 through SUMM-03, CLI-01 through CLI-05 (from ROADMAP.md Phase 2)
    - Phase 3 requirements: MCP-01 through MCP-06, API-01 through API-03 (from ROADMAP.md Phase 3)
    - Evidence: from SUMMARY.md files for each phase (they confirm which requirements were satisfied during execution)
    - Hardening status: CLI-02, INGEST-07, MCP-02 are marked "SATISFIED (hardened in Phase 4)" — these were functionally implemented in earlier phases but had audit-identified gaps closed in Phase 4

    **Structure for each file (use this exact frontmatter and format):**

    ```markdown
    ---
    phase: 0N-name
    verification: retroactive
    captured: 2026-02-23
    audit_source: .planning/v1.0-MILESTONE-AUDIT.md
    ---

    # Phase 0N: [Name] — Verification (Retroactive)

    **Note:** This VERIFICATION.md was captured retroactively as part of Phase 4 (Defensive Hardening).
    Verification was performed inline during phase execution and documented in individual SUMMARY.md files.
    The audit source is `.planning/v1.0-MILESTONE-AUDIT.md`, which cross-references all evidence.

    ## Requirements Status

    | Requirement ID | Description | Status | Evidence |
    |---------------|-------------|--------|----------|
    | ... | ... | SATISFIED | [SUMMARY file reference] |
    ```

    **Phase 1 requirements evidence** — all 12 requirements SATISFIED:
    - CONF-01 through CONF-05: `01-02-SUMMARY.md` (CorpusConfig, SourceConfig, load_config, save_config, corpus add)
    - INGEST-01 through INGEST-06: `01-04-SUMMARY.md` (CorpusIndex, index_source, chunkers, embedder)
    - INGEST-07: `01-04-SUMMARY.md` (_delete_stale_chunks implemented) — note: "SATISFIED (silent failure hardened in Phase 4)"

    **Phase 2 requirements evidence** — all 15 requirements SATISFIED:
    - SEARCH-01 through SEARCH-05: `02-02-SUMMARY.md` (CorpusSearch hybrid_search, filters, result format)
    - CLASS-01 through CLASS-03: `02-03-SUMMARY.md` (ConstructClassifier, construct_type stored, filterable)
    - SUMM-01 through SUMM-03: `02-04-SUMMARY.md` (generate_summary, summary embedded, summarize flag)
    - CLI-01 through CLI-05: `02-05-SUMMARY.md` (corpus index, corpus search, corpus status, corpus add)
    - CLI-02: note "SATISFIED (KeyError hardened in Phase 4)"

    **Phase 3 requirements evidence** — all 9 requirements SATISFIED:
    - MCP-01 through MCP-06: `03-01-SUMMARY.md` and `03-03-SUMMARY.md` (FastMCP server, corpus_search tool, stderr logging, pre-warm)
    - API-01 through API-03: `03-02-SUMMARY.md` (SearchResult, search(), index() functions)
    - MCP-02: note "SATISFIED (silent OSError hardened in Phase 4)"

    Write all three files to disk. Use Write tool for each file. These are planning documents, not source code.
  </action>
  <verify>
    <automated>ls .planning/phases/01-foundation/01-VERIFICATION.md .planning/phases/02-search-core/02-VERIFICATION.md .planning/phases/03-agent-interfaces/03-VERIFICATION.md && grep -l "verification: retroactive" .planning/phases/01-foundation/01-VERIFICATION.md .planning/phases/02-search-core/02-VERIFICATION.md .planning/phases/03-agent-interfaces/03-VERIFICATION.md | wc -l</automated>
    <manual>Open each file and verify frontmatter has `verification: retroactive` and the requirements table covers all IDs for that phase</manual>
    <sampling_rate>run after this task</sampling_rate>
  </verify>
  <done>
    - Three VERIFICATION.md files exist with correct retroactive frontmatter
    - Phase 1 file covers 12 requirements (CONF-01..05, INGEST-01..07)
    - Phase 2 file covers 15 requirements (SEARCH-01..05, CLASS-01..03, SUMM-01..03, CLI-01..05)
    - Phase 3 file covers 9 requirements (MCP-01..06, API-01..03)
    - CLI-02, INGEST-07, MCP-02 have hardening notes
  </done>
</task>

</tasks>

<verification>
Run full test suite plus type-check and lint:

```bash
uv run pytest -x -q
uv run mypy src/corpus_analyzer/
uv run ruff check src/corpus_analyzer/
```

Spot-checks:
```bash
# DATA_DIR only defined in config/schema.py:
grep -rn "DATA_DIR = " src/corpus_analyzer/ | grep -v "config/schema.py"
# Expected: empty output

# needs_reindex gone from scanner:
grep -n "needs_reindex" src/corpus_analyzer/ingest/scanner.py
# Expected: empty output

# FTS rebuild gone from engine __init__:
grep -n "create_fts_index" src/corpus_analyzer/search/engine.py
# Expected: empty output

# All three VERIFICATION.md exist:
ls .planning/phases/0*/0*-VERIFICATION.md
```
</verification>

<success_criteria>
1. `DATA_DIR` and `CONFIG_PATH` are defined only in `config/schema.py`; `cli.py` and `api/public.py` import from there; `mcp/server.py` imports from config.schema (not from cli)
2. `SourceConfig.use_llm_classification: bool = False` field exists; `index_source()` passes `source.use_llm_classification` to `classify_file(..., use_llm=...)`
3. `needs_reindex()` is absent from `ingest/scanner.py` and from all test files
4. `CorpusSearch.__init__()` contains no `create_fts_index` call; FTS is built only in `index_source()`
5. Three retroactive VERIFICATION.md files created with correct frontmatter, requirements tables, and hardening notes
6. Full test suite passes; mypy and ruff clean on all modified source files
</success_criteria>

<output>
After completion, create `.planning/phases/04-hardening/04-02-SUMMARY.md` using the summary template.
</output>
