---
phase: 23-score-normalisation-mcp-sort
plan: 01
type: tdd
wave: 1
depends_on: []
files_modified:
  - src/corpus_analyzer/search/engine.py
  - src/corpus_analyzer/cli.py
  - tests/search/test_engine.py
autonomous: true
requirements:
  - NORM-01

must_haves:
  truths:
    - "hybrid_search() returns results with _relevance_score values in 0–1 range"
    - "CLI --min-score help text references 0–1 range, not raw RRF 0.009–0.033"
    - "Python API SearchResult.score values are in 0–1 range (inherited from engine)"
    - "MCP corpus_search score field is in 0–1 range (inherited from engine)"
    - "Normalisation constant RRF_SCORE_CEILING is co-located with the engine constant K=60"
  artifacts:
    - path: "src/corpus_analyzer/search/engine.py"
      provides: "RRF_SCORE_CEILING constant + normalisation in hybrid_search()"
      contains: "RRF_SCORE_CEILING"
    - path: "tests/search/test_engine.py"
      provides: "NORM-01 tests asserting scores are in [0, 1]"
      contains: "test_scores_normalised"
  key_links:
    - from: "src/corpus_analyzer/search/engine.py"
      to: "hybrid_search() return value"
      via: "score = min(raw / RRF_SCORE_CEILING, 1.0)"
      pattern: "RRF_SCORE_CEILING"
---

<objective>
Normalise raw RRF scores to 0–1 in the search engine layer.

Purpose: All consumers (CLI, Python API, MCP) automatically receive consistent normalised
scores without surface-specific changes. Eliminates raw RRF 0.009–0.033 range leaking
to callers.

Output: Modified engine.py with RRF_SCORE_CEILING constant and in-place score
normalisation; updated min_score docstring and CLI help text; updated test_engine.py
tests that previously relied on raw RRF score range assumptions.
</objective>

<execution_context>
@/home/ollie/.claude/get-shit-done/workflows/execute-plan.md
@/home/ollie/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/23-score-normalisation-mcp-sort/23-CONTEXT.md
@src/corpus_analyzer/search/engine.py
@src/corpus_analyzer/cli.py
@tests/search/test_engine.py
</context>

<feature>
  <name>RRF score normalisation (NORM-01)</name>
  <files>
    src/corpus_analyzer/search/engine.py
    src/corpus_analyzer/cli.py
    tests/search/test_engine.py
  </files>
  <behavior>
    After hybrid_search() runs and results are filtered, each result's _relevance_score
    is divided by RRF_SCORE_CEILING and clamped to [0.0, 1.0].

    RRF_SCORE_CEILING = 1/60 + 1/60  (two-list fusion with K=60; approx 0.03333...)

    Normalisation is the LAST step before returning results — applied after all filtering
    and sorting. The _relevance_score key in the raw dict is mutated in-place.

    Cases:
      raw=0.0333...  → normalised=1.0 (at ceiling)
      raw=0.0167...  → normalised≈0.5 (midpoint)
      raw=0.009      → normalised≈0.27
      raw=0.0        → normalised=0.0
      raw=0.04       → normalised=1.0 (clamped; edge case if k changes)

    The min_score parameter continues to work: callers that used raw RRF thresholds
    (e.g. 0.01) will need to update to 0–1 equivalents. The --min-score help text is
    updated to reflect the 0–1 range.
  </behavior>
  <implementation>
    1. Add module-level constants to engine.py below the existing imports:
       ```python
       _RRF_K = 60
       RRF_SCORE_CEILING = 1.0 / _RRF_K + 1.0 / _RRF_K  # two-list RRF ceiling
       ```
    2. At the end of hybrid_search(), after sorting and before return, add a
       normalisation pass:
       ```python
       for r in results:
           raw = float(r.get("_relevance_score", 0.0))
           r["_relevance_score"] = min(raw / RRF_SCORE_CEILING, 1.0)
       ```
       Note: empty_query scan path produces rows without _relevance_score (table scan
       returns no score field). Guard the division: if raw == 0.0, result stays 0.0.
    3. Update the hybrid_search() docstring for min_score parameter:
       - Remove "RRF scores range approximately 0.009–0.033 for K=60."
       - Replace with "Scores are normalised to 0–1 range; 0.5 roughly corresponds to a
         mid-quality match."
    4. Update cli.py --min-score help text:
       - Remove "RRF scores range approximately 0.009–0.033 (K=60);"
       - Replace with "Scores are in 0–1 range (normalised); 0.0 keeps all results (default)."
    5. In tests/search/test_engine.py:
       - Import RRF_SCORE_CEILING from corpus_analyzer.search.engine
       - test_min_score_above_max_returns_empty: currently uses min_score=99.0 — still
         valid (99 > 1.0), no change needed
       - test_min_score_filters_below_threshold: uses max_score from actual results,
         compares with >=. This test is score-agnostic and still passes.
       - Add new test class TestScoreNormalisation with:
         - test_scores_normalised_to_unit_range: call hybrid_search("search"), assert
           all float(r["_relevance_score"]) are in [0.0, 1.0]
         - test_score_ceiling_constant_matches_rrf_k: assert RRF_SCORE_CEILING ==
           pytest.approx(1/60 + 1/60)
         - test_empty_query_scan_scores_zero: empty query scan (empty-query path) returns
           results with _relevance_score == 0.0 (no raw score from table scan)
  </implementation>
</feature>

<tasks>

<task type="tdd">
  <name>RED: Write failing NORM-01 normalisation tests</name>
  <files>tests/search/test_engine.py</files>
  <action>
    Add import of RRF_SCORE_CEILING from corpus_analyzer.search.engine at the top of
    the existing import block (will cause ImportError until constant is added — that IS
    the RED signal for the constant tests).

    Add class TestScoreNormalisation after existing test classes:

    ```python
    from corpus_analyzer.search.engine import CONSTRUCT_PRIORITY, CorpusSearch, RRF_SCORE_CEILING

    class TestScoreNormalisation:
        """NORM-01: Scores returned by hybrid_search() must be in [0.0, 1.0] range."""

        def test_scores_normalised_to_unit_range(self, search: CorpusSearch) -> None:
            """All _relevance_score values in results are in [0.0, 1.0]."""
            results = search.hybrid_search("search")
            assert results, "Need results to test normalisation"
            for r in results:
                score = float(r["_relevance_score"])
                assert 0.0 <= score <= 1.0, f"Score {score} is outside [0, 1]"

        def test_score_ceiling_constant_matches_rrf_k(self) -> None:
            """RRF_SCORE_CEILING equals 1/K + 1/K for the two-list fusion with K=60."""
            assert RRF_SCORE_CEILING == pytest.approx(1.0 / 60 + 1.0 / 60)

        def test_empty_query_scan_scores_zero(self, named_table) -> None:
            """Empty-query scan path (no RRF) returns _relevance_score 0.0 per row."""
            search_obj = CorpusSearch(named_table, MockEmbedder())
            results = search_obj.hybrid_search("")
            for r in results:
                assert float(r.get("_relevance_score", 0.0)) == pytest.approx(0.0)
    ```

    Run tests — expect RED (ImportError on RRF_SCORE_CEILING import).
    Commit: `test(23-01): add failing NORM-01 normalisation tests`
  </action>
  <verify>
    <automated>cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer && uv run pytest tests/search/test_engine.py::TestScoreNormalisation -x 2>&1 | tail -20</automated>
    <manual>Confirm test output shows ImportError or AttributeError for RRF_SCORE_CEILING — not a test logic error</manual>
  </verify>
  <done>Tests in TestScoreNormalisation are collected and fail RED (ImportError or AttributeError on RRF_SCORE_CEILING)</done>
</task>

<task type="tdd">
  <name>GREEN: Implement normalisation constant and engine mutation</name>
  <files>
    src/corpus_analyzer/search/engine.py
    src/corpus_analyzer/cli.py
  </files>
  <action>
    In src/corpus_analyzer/search/engine.py:

    1. After the existing module-level constants (_VALID_SORT_VALUES), add:
       ```python
       _RRF_K: int = 60
       RRF_SCORE_CEILING: float = 1.0 / _RRF_K + 1.0 / _RRF_K  # two-list fusion ceiling
       ```

    2. In hybrid_search(), after the sort block and before `return results`, add:
       ```python
       # Normalise _relevance_score to [0.0, 1.0] range (NORM-01).
       # Empty-query scan rows have no _relevance_score; they stay at 0.0.
       for r in results:
           raw = float(r.get("_relevance_score", 0.0))
           r["_relevance_score"] = min(raw / RRF_SCORE_CEILING, 1.0) if raw > 0.0 else 0.0
       ```

    3. Update the min_score parameter docstring in hybrid_search():
       - Remove: "RRF scores range approximately 0.009–0.033 for K=60. Negative values
         behave the same as 0.0."
       - Add: "Scores are normalised to 0–1 range; 0.5 is a rough mid-quality threshold.
         Negative values behave the same as 0.0."

    In src/corpus_analyzer/cli.py:
    4. Locate the --min-score typer.Option help string (around line 340):
       - Remove: "RRF scores range approximately 0.009–0.033 (K=60); "
       - The new help string should read:
         "Exclude results below this score threshold. "
         "Scores are normalised to 0–1 range; 0.0 keeps all results (default)."

    Run full test suite to confirm GREEN.
    Commit: `feat(23-01): normalise RRF scores to 0–1 range (NORM-01)`
  </action>
  <verify>
    <automated>cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer && uv run pytest tests/search/test_engine.py -x -q 2>&1 | tail -20</automated>
    <manual>Check that test_min_score_above_max_returns_empty still passes (min_score=99.0 still exceeds 1.0)</manual>
  </verify>
  <done>All tests/search/test_engine.py tests pass GREEN including TestScoreNormalisation; no regressions</done>
</task>

</tasks>

<verification>
Full test suite must pass after implementation:

```bash
cd /home/ollie/Development/Tools/vibe_coding/corpus-analyzer && uv run pytest tests/search/test_engine.py tests/mcp/test_server.py tests/api/ -v -q 2>&1 | tail -30
```

Check that:
- All TestScoreNormalisation tests pass
- Existing FILT-01 tests pass (min_score=99.0 still works since 99 > 1.0)
- MCP server tests pass (score field still populated from _relevance_score)
- API tests pass (SearchResult.score inherits normalised value)
- Linting passes: `uv run ruff check src/corpus_analyzer/search/engine.py src/corpus_analyzer/cli.py`
</verification>

<success_criteria>
- RRF_SCORE_CEILING module-level constant exists in engine.py, documented as two-list fusion ceiling
- All hybrid_search() results have _relevance_score in [0.0, 1.0]
- CLI --min-score help text references 0–1 range only (no "0.009–0.033" substring)
- engine.py min_score docstring references 0–1 range only
- uv run pytest tests/search/test_engine.py passes with zero failures
</success_criteria>

<output>
After completion, create `.planning/phases/23-score-normalisation-mcp-sort/23-score-normalisation-mcp-sort-01-SUMMARY.md`
</output>
