---
phase: 01-core-hardening
plan: 03
type: execute
wave: 1
depends_on: []
files_modified:
  - src/ztlctl/services/check.py
  - src/ztlctl/services/reweave.py
  - src/ztlctl/services/graph.py
  - tests/services/test_graph.py
autonomous: true
requirements: [HARD-06]

must_haves:
  truths:
    - "check --rebuild reads files in parallel using ThreadPoolExecutor, then writes to DB sequentially"
    - "reweave candidate scoring uses a single FTS5 MATCH query instead of N individual queries"
    - "betweenness centrality uses k-approximation for graphs with more than 500 nodes"
  artifacts:
    - path: "src/ztlctl/services/check.py"
      provides: "ThreadPoolExecutor-based parallel file reading in rebuild()"
      contains: "ThreadPoolExecutor"
    - path: "src/ztlctl/services/reweave.py"
      provides: "Batch FTS5 BM25 scoring"
      contains: "bm25"
    - path: "src/ztlctl/services/graph.py"
      provides: "betweenness_centrality with k parameter"
      contains: "k="
  key_links:
    - from: "src/ztlctl/services/check.py"
      to: "concurrent.futures"
      via: "ThreadPoolExecutor for parallel file reads"
      pattern: "from concurrent\\.futures import"
---

<objective>
Fix the three confirmed performance bottlenecks: rebuild I/O parallelization, FTS5 batch BM25 scoring, and betweenness centrality approximation.

Purpose: HARD-06 requires measurable performance improvements on performance-critical paths.
Output: Three surgical optimizations in check.py, reweave.py, and graph.py.
</objective>

<execution_context>
@/Users/shparki/.claude/get-shit-done/workflows/execute-plan.md
@/Users/shparki/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/codebase/CONCERNS.md
@.planning/phases/01-core-hardening/01-RESEARCH.md

<interfaces>
<!-- Key types and contracts the executor needs. -->

From src/ztlctl/services/check.py rebuild() (line 139):
```python
def rebuild(self) -> ServiceResult:
    # Sequential loop reads each file from disk
    # files = list(vault.filesystem.find_content())
    # for f in files: f.read_text() -> parse frontmatter -> DB write
```

From src/ztlctl/services/reweave.py _score_candidates():
```python
# Currently: per-candidate FTS5 query
# for candidate in candidates:
#     score = conn.execute(FTS5_MATCH_for_one_candidate)
```

From src/ztlctl/services/graph.py materialize_metrics() (line 587):
```python
betweenness = nx.betweenness_centrality(g)  # no k parameter = exact, O(V*E)
```

PITFALL (from RESEARCH.md): ThreadPoolExecutor for file READS only.
DB writes MUST remain sequential inside vault.transaction().
SQLite serializes writes; parallel writes cause "database is locked".

PITFALL (from RESEARCH.md): FTS5 special characters.
Sanitize terms before FTS5 MATCH: wrap each term in double quotes,
escape internal double quotes: '"' + term.replace('"', '""') + '"'
</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Parallelize rebuild file reads + batch FTS5 scoring + betweenness k-approximation</name>
  <files>src/ztlctl/services/check.py, src/ztlctl/services/reweave.py, src/ztlctl/services/graph.py, tests/services/test_graph.py</files>
  <read_first>
    - src/ztlctl/services/check.py (rebuild method — full method body starting at line 139)
    - src/ztlctl/services/reweave.py (find _score_candidates method — understand the per-candidate FTS5 query pattern)
    - src/ztlctl/services/graph.py (materialize_metrics method — find the betweenness_centrality call around line 587)
    - src/ztlctl/domain/content.py (parse_frontmatter function signature)
    - tests/services/test_graph.py (existing test patterns for graph service)
  </read_first>
  <action>
    **1. Parallel file reads in check.py rebuild():**

    Add to module-level imports:
    ```python
    from concurrent.futures import ThreadPoolExecutor, as_completed
    ```

    Add a module-level helper function (outside the class, before or after it):
    ```python
    def _read_file(path: Path) -> tuple[Path, str]:
        """Read a single content file — designed for ThreadPoolExecutor."""
        return path, path.read_text(encoding="utf-8")
    ```

    In `rebuild()`, replace the sequential file read loop with:
    - Phase 1 (parallel): Use ThreadPoolExecutor(max_workers=8) to read all files in parallel. Collect results as `list[tuple[Path, str]]`.
    - Phase 2 (sequential): Parse frontmatter from each raw string and perform DB operations inside `self._vault.transaction()`.

    The key structural change:
    ```python
    content_files = list(...)  # existing file discovery
    raw_contents: list[tuple[Path, str]] = []

    with ThreadPoolExecutor(max_workers=8) as executor:
        futures = {executor.submit(_read_file, f): f for f in content_files}
        for future in as_completed(futures):
            raw_contents.append(future.result())

    # Sequential DB write phase (existing code, just using pre-read content)
    with self._vault.transaction() as txn:
        for path, raw in raw_contents:
            fm, body = parse_frontmatter(raw)
            # ... rest of existing per-file processing ...
    ```

    CRITICAL: Only parallelize the file READ step. The DB write step MUST remain sequential inside `vault.transaction()` to avoid SQLite "database is locked" errors.

    **2. Batch FTS5 scoring in reweave.py:**

    Find the `_score_candidates()` method (or the method that computes BM25 scores per-candidate). Replace the N individual FTS5 queries with a single batch query:

    - Extract the target note's significant terms (title words, tags) — these are the query terms.
    - Sanitize each term for FTS5: wrap in double quotes, escape internal double quotes:
      ```python
      def _fts5_escape(term: str) -> str:
          """Escape a term for safe FTS5 MATCH usage."""
          return '"' + term.replace('"', '""') + '"'
      ```
    - Build a single FTS5 MATCH query joining terms with OR:
      ```python
      fts_query = " OR ".join(_fts5_escape(t) for t in terms if t.strip())
      ```
    - Execute one ranked query:
      ```python
      ranked = conn.execute(
          text(
              "SELECT id, bm25(nodes_fts) AS score "
              "FROM nodes_fts WHERE nodes_fts MATCH :q "
              "ORDER BY bm25(nodes_fts) LIMIT :limit"
          ),
          {"q": fts_query, "limit": candidate_count * 2},
      ).fetchall()
      bm25_scores = {row.id: abs(row.score) for row in ranked}
      ```
    - Use bm25_scores dict for lookup instead of per-candidate queries.
    - Handle empty terms gracefully (skip FTS5 query, return empty scores dict).

    **3. betweenness centrality k-approximation in graph.py:**

    Find the `nx.betweenness_centrality(g)` call in `materialize_metrics()` (around line 587). Replace with:
    ```python
    node_count = g.number_of_nodes()
    k_param = None if node_count <= 500 else min(500, node_count)
    betweenness = nx.betweenness_centrality(g, k=k_param, seed=42)
    ```

    This keeps exact computation for small graphs (<=500 nodes) and uses k-sample approximation for larger graphs. `seed=42` ensures deterministic results.

    **4. Add test for betweenness approximation in tests/services/test_graph.py:**

    Add a test that verifies `k` parameter is used for large graphs. Since creating a 500+ node test graph is impractical, test via mocking:
    ```python
    def test_betweenness_uses_k_approximation_for_large_graphs():
        # Verify the k parameter logic exists in materialize_metrics
        # by checking the code path handles node_count > 500
    ```
    Or add a simpler behavioral test that verifies materialize_metrics completes without error on a small graph (regression test).
  </action>
  <verify>
    <automated>uv run pytest tests/ -x -q && uv run ruff check src/ztlctl/services/check.py src/ztlctl/services/reweave.py src/ztlctl/services/graph.py && uv run mypy src/ztlctl/services/check.py src/ztlctl/services/reweave.py src/ztlctl/services/graph.py --strict</automated>
  </verify>
  <acceptance_criteria>
    - src/ztlctl/services/check.py contains `from concurrent.futures import ThreadPoolExecutor`
    - src/ztlctl/services/check.py contains `ThreadPoolExecutor(max_workers=` in rebuild method
    - src/ztlctl/services/check.py contains `def _read_file(` as module-level function
    - src/ztlctl/services/reweave.py contains `bm25(nodes_fts)` in a single batch query (not inside a per-candidate loop)
    - src/ztlctl/services/graph.py contains `k=k_param` or `k=` in the betweenness_centrality call
    - src/ztlctl/services/graph.py contains `seed=42` in the betweenness_centrality call
    - src/ztlctl/services/graph.py contains comparison against 500 (e.g., `node_count <= 500` or `node_count > 500`)
    - `uv run pytest tests/ -x -q` exits 0
    - `uv run ruff check src/ztlctl/services/check.py src/ztlctl/services/reweave.py src/ztlctl/services/graph.py` exits 0
  </acceptance_criteria>
  <done>Three performance fixes applied: rebuild uses ThreadPoolExecutor for parallel file reads, reweave uses single FTS5 batch query, betweenness uses k-approximation. All tests pass, mypy strict, ruff clean.</done>
</task>

</tasks>

<verification>
- `uv run pytest tests/ -x -q` — full suite passes
- `uv run ruff check .` — no lint issues
- `uv run mypy src/ --strict` — no type errors
- Grep confirms: ThreadPoolExecutor in check.py, batch bm25 in reweave.py, k= in graph.py betweenness call
</verification>

<success_criteria>
- rebuild() reads files in parallel via ThreadPoolExecutor, writes to DB sequentially
- reweave scoring uses single FTS5 MATCH query instead of N per-candidate queries
- betweenness_centrality uses k=500 approximation for graphs > 500 nodes
- No functional regressions in existing tests
</success_criteria>

<output>
After completion, create `.planning/phases/01-core-hardening/01-03-SUMMARY.md`
</output>
