---
phase: 02-search-core
plan: 04
type: tdd
wave: 3
depends_on:
  - 02-01
  - 02-03
files_modified:
  - src/corpus_analyzer/search/summarizer.py
  - src/corpus_analyzer/ingest/indexer.py
  - tests/search/test_summarizer.py
  - tests/ingest/test_indexer.py
autonomous: true
requirements:
  - SUMM-01
  - SUMM-02
  - SUMM-03
  - CLASS-01
  - CLASS-02
  - CLI-01

must_haves:
  truths:
    - "generate_summary() calls ollama.generate() with a focused prompt and returns a trimmed string"
    - "generate_summary() returns None (not raises) when source.summarize is False"
    - "CorpusIndex.index_source() populates construct_type on new/changed chunks via classify_file()"
    - "CorpusIndex.index_source() generates and stores a summary for new/changed files"
    - "CorpusIndex.index_source() skips summary generation for files whose content_hash is unchanged"
    - "Summary text is prepended to the first chunk's text before embedding (SUMM-02)"
  artifacts:
    - path: "src/corpus_analyzer/search/summarizer.py"
      provides: "generate_summary() function"
      exports: ["generate_summary"]
      min_lines: 35
    - path: "src/corpus_analyzer/ingest/indexer.py"
      provides: "index_source() extended with classify_file() + generate_summary() per file"
      contains: "classify_file"
  key_links:
    - from: "src/corpus_analyzer/ingest/indexer.py"
      to: "src/corpus_analyzer/search/classifier.py"
      via: "classify_file(file_path, full_text, model) called per changed file"
      pattern: "classify_file"
    - from: "src/corpus_analyzer/ingest/indexer.py"
      to: "src/corpus_analyzer/search/summarizer.py"
      via: "generate_summary() called per changed file when source.summarize=True"
      pattern: "generate_summary"
    - from: "src/corpus_analyzer/ingest/indexer.py"
      to: "src/corpus_analyzer/store/schema.py"
      via: "ensure_schema_v2(table) called in CorpusIndex.open() before any indexing"
      pattern: "ensure_schema_v2"
---

<objective>
TDD implementation of the Summarizer and extension of CorpusIndex.index_source() to classify and summarize files during indexing.

Purpose: This plan wires classification and summary generation into the existing ingestion pipeline. After this plan, `corpus index` will populate construct_type and summary for every new/changed file.

Output: Working generate_summary() and extended index_source() that produce construct_type and summary on indexed chunks.
</objective>

<execution_context>
@/home/ollie/.claude/get-shit-done/workflows/execute-plan.md
@/home/ollie/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/phases/02-search-core/02-CONTEXT.md
@.planning/phases/02-search-core/02-RESEARCH.md
@.planning/phases/02-search-core/02-01-SUMMARY.md
@.planning/phases/02-search-core/02-03-SUMMARY.md
@src/corpus_analyzer/ingest/indexer.py
@src/corpus_analyzer/store/schema.py
@src/corpus_analyzer/config/schema.py
@tests/search/test_summarizer.py
@tests/ingest/test_indexer.py
</context>

<feature>
  <name>Summarizer and indexer extension for classification + summary</name>
  <files>
    src/corpus_analyzer/search/summarizer.py
    src/corpus_analyzer/ingest/indexer.py
    tests/search/test_summarizer.py
    tests/ingest/test_indexer.py
  </files>
  <behavior>
    **generate_summary(filename: str, content: str, model: str, host: str) -> str**

    Calls ollama.generate() using SUMMARY_PROMPT_TEMPLATE (from RESEARCH.md Pattern 5).
    Returns response.response.strip().
    Truncates content to 4000 chars before sending to Ollama.
    If ollama.generate() raises, returns "" (empty string, not crash — indexer treats empty as missing).

    **generate_summary when source.summarize=False:**
    The caller (indexer) is responsible for checking source.summarize. generate_summary itself
    always generates when called. The Summarizer module also exposes a convenience:
    `should_summarize(source_summarize: bool, stored_summary: str | None, content_hash_changed: bool) -> bool`
    Returns True only when: source_summarize=True AND (content_hash_changed OR stored_summary is None).

    **CorpusIndex.open() extension:**
    After table open/creation, call ensure_schema_v2(table) unconditionally.
    Import ensure_schema_v2 from corpus_analyzer.store.schema.

    **CorpusIndex.index_source() extension:**
    Add two new parameters:
    - `use_llm_classification: bool = True` — passed through to classify_file()
    For each changed/new file:
    1. Read full file text (already chunked — use the joined text of all chunks as full_text, or re-read file directly)
    2. Call classify_file(file_path, full_text, model=self._embedder.model, use_llm=use_llm_classification) → construct_type
    3. Check should_summarize(source.summarize, stored_summary_for_file, content_hash_changed)
    4. If should_summarize → call generate_summary(file_path.name, full_text, model, host) → summary_text
    5. For summary embedding (SUMM-02): prepend summary to first chunk's text before embedding:
       chunks[0]["text"] = f"{summary_text}\n\n{chunks[0]['text']}" if summary_text else chunks[0]["text"]
    6. Store construct_type and summary in every chunk dict for this file

    Retrieve stored summary per file: extend _get_existing_files() to also return stored summaries.
    Or add a separate _get_stored_summaries(source_name) -> dict[str, str | None] method.

    Test cases:
    - generate_summary("file.md", "content", "model", "host") with mocked ollama → returns stripped string
    - generate_summary with ollama raising ConnectionError → returns "" without crash
    - should_summarize(True, None, True) → True
    - should_summarize(True, "existing", False) → False (unchanged + already has summary)
    - should_summarize(False, None, True) → False (source opted out)
    - index_source() on new file → chunk dicts have non-None construct_type
    - index_source() on unchanged file (second run) → ollama.generate not called again
  </behavior>
  <implementation>
    **src/corpus_analyzer/search/summarizer.py:**

    Implement generate_summary() and should_summarize() per the behavior spec above.
    Use the SUMMARY_PROMPT_TEMPLATE from research Pattern 5.
    Import: `import ollama  # type: ignore[import-untyped]`

    **src/corpus_analyzer/ingest/indexer.py:**

    1. Add imports at top of file:
       ```python
       from corpus_analyzer.search.classifier import classify_file
       from corpus_analyzer.search.summarizer import generate_summary, should_summarize
       from corpus_analyzer.store.schema import ensure_schema_v2
       ```

    2. In CorpusIndex.open(), add after the try/except block (table is now open):
       ```python
       ensure_schema_v2(table)
       ```

    3. In index_source(), add use_llm_classification parameter.

    4. Before the file-walk loop, add per-source stored-summary lookup:
       ```python
       stored_summaries = self._get_stored_summaries(source.name)
       ```

    5. In the "file changed or new" branch, after chunk_file():
       - Read full text: `full_text = file_path.read_text(errors="replace")`
       - Classify: `construct_type = classify_file(file_path, full_text, self._embedder.model, use_llm=use_llm_classification)`
       - Summarize: check should_summarize, call generate_summary if needed
       - Prepend summary to chunks[0]["text"] if summary non-empty
       - Add construct_type and summary to every chunk_dict in the loop

    6. Add _get_stored_summaries(source_name: str) -> dict[str, str | None] method.

    **Test mocking:** In test_summarizer.py, use `unittest.mock.patch("ollama.generate")`.
    In test_indexer.py Phase 2 tests, additionally patch `classify_file` and `generate_summary`
    to avoid hitting Ollama during indexer tests.

    IMPORTANT: The existing indexer tests must continue to pass. Do NOT change the existing
    index_source() behavior for files that don't require Phase 2 changes. Only extend — never remove.
  </implementation>
</feature>

<verification>
After GREEN phase:
`uv run pytest tests/search/test_summarizer.py tests/ingest/test_indexer.py -v`

All new tests must show PASSED. All pre-existing indexer tests must still PASS.

Run full suite:
`uv run pytest tests/search/ tests/ingest/ -v --no-header 2>&1 | tail -30`

Run lint and type check:
`uv run ruff check src/corpus_analyzer/search/summarizer.py src/corpus_analyzer/ingest/indexer.py && uv run mypy src/corpus_analyzer/search/summarizer.py src/corpus_analyzer/ingest/indexer.py`
</verification>

<success_criteria>
- All 4 test_summarizer.py tests pass
- Both new test_indexer.py Phase 2 tests pass
- All pre-existing indexer tests continue to pass
- CorpusIndex.open() calls ensure_schema_v2()
- index_source() populates construct_type on chunk dicts
- index_source() skips summary generation for unchanged files
- Summary prepended to first chunk text before embedding
- mypy and ruff pass on both files
</success_criteria>

<output>
After completion, create `.planning/phases/02-search-core/02-04-SUMMARY.md` with:
- summarizer and indexer extension import paths
- Confirmation that ensure_schema_v2 is wired into CorpusIndex.open()
- How stored summaries are looked up (method signature)
- Any deviations from plan
</output>
