---
phase: 01-foundation
plan: 04
type: execute
wave: 3
depends_on:
  - 01-01
  - 01-03
files_modified:
  - src/corpus_analyzer/ingest/embedder.py
  - src/corpus_analyzer/ingest/indexer.py
  - tests/ingest/test_embedder.py
  - tests/ingest/test_indexer.py
autonomous: true
requirements:
  - INGEST-01
  - INGEST-04
  - INGEST-05
  - INGEST-06
  - INGEST-07

must_haves:
  truths:
    - "OllamaEmbedder.embed_batch() returns a list of float vectors matching input chunk count"
    - "OllamaEmbedder fails fast with a clear error message when Ollama is not reachable"
    - "OllamaEmbedder fails fast with a clear error message when the model is not pulled"
    - "CorpusIndex.open() creates the LanceDB chunks table if it does not exist"
    - "CorpusIndex.open() detects model mismatch and raises a clear error"
    - "CorpusIndex.index_source() inserts new chunks, updates changed chunks, deletes stale chunks"
    - "CorpusIndex.index_source() skips re-embedding files whose content hash is unchanged"
    - "All metadata fields (file_path, source_name, file_type, content_hash, embedding_model, indexed_at, start_line, end_line) are populated on every chunk"
  artifacts:
    - path: "src/corpus_analyzer/ingest/embedder.py"
      provides: "OllamaEmbedder with embed_batch(), validate_connection(), get_model_dims()"
      exports: ["OllamaEmbedder"]
    - path: "src/corpus_analyzer/ingest/indexer.py"
      provides: "CorpusIndex with open(), index_source(), close(), table property"
      exports: ["CorpusIndex"]
    - path: "tests/ingest/test_embedder.py"
      provides: "Embedder tests with mocked ollama client"
    - path: "tests/ingest/test_indexer.py"
      provides: "Indexer tests using real LanceDB in tmp_path"
  key_links:
    - from: "src/corpus_analyzer/ingest/embedder.py"
      to: "ollama.embed()"
      via: "ollama client"
      pattern: "ollama\\.embed"
    - from: "src/corpus_analyzer/ingest/indexer.py"
      to: "src/corpus_analyzer/store/schema.py"
      via: "ChunkRecord + make_chunk_id"
      pattern: "ChunkRecord|make_chunk_id"
    - from: "src/corpus_analyzer/ingest/indexer.py"
      to: "src/corpus_analyzer/ingest/chunker.py"
      via: "chunk_file()"
      pattern: "chunk_file"
    - from: "src/corpus_analyzer/ingest/indexer.py"
      to: "src/corpus_analyzer/ingest/scanner.py"
      via: "needs_reindex() + file_content_hash()"
      pattern: "needs_reindex|file_content_hash"
    - from: "src/corpus_analyzer/ingest/indexer.py"
      to: "lancedb table"
      via: "merge_insert + per-source delete"
      pattern: "merge_insert"
---

<objective>
Implement the Ollama embedder and the core indexer engine (CorpusIndex) that ties together the schema, chunker, scanner, and LanceDB writes.

Purpose: This is the integration core of Phase 1 — it wires all prior components into a working pipeline. The embedder wraps Ollama with connection validation and batch embedding; the indexer orchestrates chunking, change detection, embedding, and LanceDB upsert+delete.

Output: OllamaEmbedder and CorpusIndex classes, tested with mocked Ollama and real LanceDB (in tmp_path).
</objective>

<execution_context>
@/home/ollie/.claude/get-shit-done/workflows/execute-plan.md
@/home/ollie/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/01-foundation/01-RESEARCH.md
@.planning/phases/01-foundation/01-CONTEXT.md
@.planning/phases/01-foundation/01-01-SUMMARY.md
@.planning/phases/01-foundation/01-03-SUMMARY.md
</context>

<tasks>

<task type="auto">
  <name>Task 1: OllamaEmbedder with connection validation and batch embedding</name>
  <files>
    src/corpus_analyzer/ingest/embedder.py
    tests/ingest/test_embedder.py
  </files>
  <action>
    Create src/corpus_analyzer/ingest/embedder.py with class OllamaEmbedder:

    __init__(self, model: str, host: str = "http://localhost:11434") -> None:
      Store model and host. Do NOT connect at init time.

    validate_connection(self) -> None:
      Call ollama.list() (with the configured host). On ConnectionRefusedError or httpx.ConnectError:
      raise RuntimeError(f"Cannot connect to Ollama at {self.host}. Start Ollama with: ollama serve")

    get_model_dims(self) -> int:
      Embed a single test string ("test") to get the actual vector dimension.
      On ollama.ResponseError where the message indicates model not found:
      raise RuntimeError(f"Model '{self.model}' not found. Pull it with: ollama pull {self.model}")
      Return len(response.embeddings[0]).

    embed_batch(self, texts: list[str]) -> list[list[float]]:
      Call ollama.embed(model=self.model, input=texts).
      Return response.embeddings (list of list[float]).
      Do NOT call one at a time — pass full list to a single ollama.embed() call.
      Guard: if texts is empty, return [].

    Create tests/ingest/test_embedder.py:
      Use unittest.mock.patch to mock ollama.embed and ollama.list.
      Test embed_batch returns correct shape (len(texts) vectors).
      Test embed_batch with empty list returns [].
      Test validate_connection raises RuntimeError when ollama.list raises ConnectionRefusedError.
      Test get_model_dims returns the length of the first embedding vector.

    All type annotations, PEP 257 docstrings. Import ollama at top of file.
  </action>
  <verify>
    <automated>uv run pytest tests/ingest/test_embedder.py -v && uv run mypy src/corpus_analyzer/ingest/embedder.py</automated>
    <manual>Verify no live Ollama calls are made in tests (all mocked)</manual>
  </verify>
  <done>All embedder tests pass. OllamaEmbedder is importable. No real network calls in tests.</done>
</task>

<task type="auto">
  <name>Task 2: CorpusIndex — LanceDB open, index_source, and stale chunk deletion</name>
  <files>
    src/corpus_analyzer/ingest/indexer.py
    tests/ingest/test_indexer.py
  </files>
  <action>
    Create src/corpus_analyzer/ingest/indexer.py with class CorpusIndex:

    open(cls, data_dir: Path, embedder: OllamaEmbedder) -> "CorpusIndex" (classmethod):
      Connect to lancedb at data_dir / "index".
      If "chunks" table does NOT exist: create it with ChunkRecord schema.
      If "chunks" table DOES exist: open it and verify the embedding_model matches embedder.model.
        Check by querying: table.to_pandas()["embedding_model"].iloc[0] (or handle empty table).
        If mismatch: raise RuntimeError with message including stored model vs configured model,
        and instruction to run 'corpus reindex --force' to rebuild.
      Return CorpusIndex instance with self._table and self._embedder.

    index_source(self, source: SourceConfig, progress_callback: Callable[[int], None] | None = None) -> IndexResult:
      Walk source files using walk_source(Path(source.path).expanduser(), source.include, source.exclude).
      For each file:
        1. file_path = str(file.resolve())
        2. current_hash = file_content_hash(file)
        3. Look up stored_mtime and stored_hash from self._file_index (dict built at start of method
           by querying table for distinct file_path + content_hash for this source_name).
        4. If needs_reindex(file, stored_mtime, stored_hash) is False: increment skipped count; skip.
        5. Else: chunk_file(file) -> chunks; embed_batch([c["text"] for c in chunks]) -> vectors.
        6. Build list[ChunkRecord] with all fields populated:
           - chunk_id = make_chunk_id(source.name, file_path, chunk["start_line"], chunk["text"])
           - file_type = file.suffix
           - content_hash = current_hash
           - embedding_model = self._embedder.model
           - indexed_at = datetime.utcnow().isoformat()
        7. Append to new_chunk_dicts list.
        If progress_callback: call it with 1 (advance by one file).
      After all files:
        Upsert new_chunk_dicts with: table.merge_insert("chunk_id").when_matched_update_all().when_not_matched_insert_all().execute(new_chunk_dicts)
        Delete stale chunks: collect all chunk_ids in new_chunk_dicts for this source_name;
          table.delete(f"source_name = '{source.name}' AND chunk_id NOT IN ({id_list})")
          Guard: if new_chunk_dicts is empty AND files_found == 0: skip delete (no files = source dir empty, not deleted)
          Guard: if id_list would be empty (all files skipped): query existing IDs for source to build id_list
        Call table.optimize() after full source sync.
      Return IndexResult(source_name=source.name, files_indexed=N, chunks_written=M, files_skipped=K, elapsed=float)

    IndexResult is a dataclass with fields: source_name, files_indexed, chunks_written, files_skipped, elapsed.

    Create tests/ingest/test_indexer.py using real LanceDB + real files in pytest tmp_path
    (mock only OllamaEmbedder.embed_batch to return fake 768-dim vectors of [0.1]*768):
      - Test open() creates "chunks" table when it does not exist
      - Test open() raises RuntimeError on model mismatch (create table with model="x", open with model="y")
      - Test index_source() indexes files and returns correct counts
      - Test index_source() skips unchanged files on second run (files_skipped == total files)
      - Test index_source() removes stale chunks when a file is deleted from disk
      - Test index_source() does NOT delete chunks from OTHER sources (isolation)

    Note: Use mock.patch on OllamaEmbedder.embed_batch to return [[0.1]*768]*N (N = number of input texts).
    Note: Guard the per-source delete against empty id_list edge case (empty string would produce invalid SQL).
    Note: The stored_mtime tracking: on first run there are no stored records, so needs_reindex should be True
      when stored_mtime=0.0 and stored_hash="" (use sentinel defaults). Build file_index dict at start of
      index_source() by querying: table.search().where(f"source_name = '{source.name}'").to_list() — returns
      list of dicts from which {row["file_path"]: (row["content_hash"],)} can be built. mtime is NOT stored
      in LanceDB (it's transient); use os.stat on the fly.
    Note: The mtime fast-path (needs_reindex returns False when mtime unchanged) requires storing mtime
      somewhere. Options: (a) store mtime in LanceDB as a float column, or (b) always do hash comparison
      (skip mtime fast-path). Use option (b) for simplicity in Phase 1 — compare content_hash only, not mtime.
      This means needs_reindex in the indexer is simplified to: hash_changed = (current_hash != stored_hash).
      The needs_reindex() function in scanner.py is still correct and tested; the indexer uses the simpler
      hash-only check because mtime is not persisted. Document this in a comment.
  </action>
  <verify>
    <automated>uv run pytest tests/ingest/test_indexer.py -v && uv run mypy src/corpus_analyzer/ingest/indexer.py</automated>
    <manual>Verify stale chunk test: delete a file from tmp_path between two index_source() calls and confirm chunk count drops</manual>
  </verify>
  <done>All indexer tests pass including stale chunk removal and cross-source isolation. mypy clean.</done>
</task>

</tasks>

<verification>
uv run pytest tests/ingest/ -v
uv run mypy src/corpus_analyzer/ingest/
uv run ruff check src/corpus_analyzer/ingest/
</verification>

<success_criteria>
All ingest tests pass. CorpusIndex.index_source() correctly inserts, skips, and removes chunks. Cross-source isolation verified. No mypy errors.
</success_criteria>

<output>
After completion, create .planning/phases/01-foundation/01-04-SUMMARY.md
</output>
