# Benchmarking Reranked Quality

**Goal**: Quantify the retrieval quality improvement (Recall@K / MRR) and performance impact provided by the new Cross-Encoder Reranker compared to the baseline (Vector + Graph only).

## Proposed Changes

### [clew-engine](../../)

#### [MODIFY] [tests/quality/measure_mrr.py](../../tests/quality/measure_mrr.py)
Update the measurement script to support A/B testing:
-   Add `--rerank` / `--no-rerank` CLI arguments (default: `--rerank`).
-   Pass the `rerank` boolean in the POST request to `/v1/search/`.
-   Measure and report per-query latency (p50/p95/p99).
-   Include "Reranking: ON/OFF" in the output summary.
-   Return JSON output with detailed per-query metrics for downstream analysis.

#### [NEW] [tests/quality/benchmark_reranker.py](../../tests/quality/benchmark_reranker.py)
Create an orchestration script that:
1.  Runs `measure_mrr.py --no-rerank` **without** reranking (Baseline).
2.  Runs `measure_mrr.py --rerank` **with** reranking (Experiment).
3.  Calculates lift percentages for MRR, Recall@3, and Recall@5.
4.  Computes statistical significance using bootstrap sampling (1000 iterations).
5.  Analyzes per-category performance to identify where reranking helps most.
6.  Compares latency metrics (p50/p95/p99) between baseline and reranked.
7.  Generates a comprehensive Markdown report `RERANKER_REPORT.md` with:
    - Side-by-side metric comparison
    - Statistical confidence intervals (95% CI)
    - Per-category lift breakdown
    - Latency impact analysis
    - Example queries with dramatic improvements
    - Recommendations for production deployment

#### [NEW] [tests/quality/test_reranker_integration.py](../../tests/quality/test_reranker_integration.py)
Add automated integration tests:
-   Verify `--rerank` flag correctly passes parameter to API
-   Ensure reranking changes result ordering (not a no-op)
-   Test edge cases: empty results, single result, no vector matches
-   Verify no crashes on malformed inputs

## Verification Plan

### Automated Testing
1.  Run `pytest tests/quality/test_reranker_integration.py` to verify API integration
2.  Ensure all edge cases pass without errors

### Manual Verification
1.  Start the Clew development server with reranker enabled
2.  Run `python tests/quality/benchmark_reranker.py`
3.  Verify `RERANKER_REPORT.md` is generated with complete metrics
4.  Inspect sample queries to confirm reranking improves relevance
5.  Check historical baseline against `BASELINE_REPORT.md` for regression detection

### Success Criteria

#### Quality Metrics (Primary)
-   **MRR Lift**: > 5% improvement (with 95% CI not crossing 0%)
-   **Recall@3 Lift**: > 5% improvement (with 95% CI not crossing 0%)
-   **Recall@5 Lift**: > 3% improvement

#### Performance Metrics (Secondary)
-   **p95 Latency**: < 500ms total query time
-   **Latency Overhead**: Reranking adds < 300ms at p95

#### Quality Assurance
-   **No Regressions**: Baseline metrics match historical `BASELINE_REPORT.md` (±2% tolerance)
-   **No Negative Lift**: Zero queries should show worse ranking with reranker ON
-   **Category Coverage**: At least 3 query categories show measurable improvement

#### Edge Case Handling
-   All integration tests pass
-   No crashes on empty/single/malformed results

### Contingency Plans

If targets not met:
-   **0-5% lift**: Review reranker configuration, try different cross-encoder models
-   **Negative lift**: Disable reranker, file bug report, investigate ranking algorithm
-   **Excessive latency**: Implement async reranking or batch processing optimizations

### Sample Size Validation
-   **Current gold set**: 44 queries across multiple categories
-   **Minimum required**: 30 queries for statistical validity
-   **Recommendation**: Expand to 60+ queries if results are borderline (3-7% lift)
