# Sanity Check: The Reranker, The Stack, and The Timing

**Date**: 2026-02-07
**From**: Claude (QA Auditor)
**Type**: Strategic Review — Final Assessment

---

## 1. Churn Audit

This discussion has undergone **one full architecture pivot in a single day**:

| Step | Artifact | What It Proposed |
|:---|:---|:---|
| 1 | `CHECKPOINT_HANDOFF.md` | Fine-tune Qwen to generate "Telegraphic Stories" from ASTs. 20x compression. |
| 2 | `CLAUDE_FEEDBACK.md` | Called the 20x claim unsupported. Recommended benchmarking signatures-only first. |
| 3 | `CTO_WEIGHS_IN.md` | Agreed on compression limits. Proposed "Attention Filter" reframing. |
| 4 | `CLAUDE_RESPONSE_TO_CTO.md` | Endorsed the reranker pivot. Recommended classification over generation. |
| 5 | `CTO_PIVOT.md` | Officially pivoted to cross-encoder reranker. Issued builder seed prompt. |
| 6 | `CLAUDE_RESPONSE_TO_PIVOT.md` | Flagged 5 technical issues in the seed prompt (cross-encoder vs bi-encoder, decoder-only scoring, etc). |
| 7 | Updated `README.md` | Rewrote for reranker architecture. Addressed most feedback. |

**The good**: The pivot was directionally correct. Generation -> Classification is a sound downscoping for a 1.5B model. The discussion converged rather than oscillated.

**The concern**: The original `specs/narrative_schema.md` and `agent/rules.md` are still in the repo. The directory listing in the new README doesn't include them, but unless they've been deleted, they'll confuse anyone who opens the project. The repo is telling two conflicting stories about what it wants to be.

**Verdict on churn**: Acceptable. One deliberate pivot based on reviewed feedback is not churn — it's the process working. But clean up the old artifacts.

---

## 2. Does the Reranker Plan Make Sense?

**In isolation, yes.** Cross-encoder reranking is a well-understood problem. The architecture (logit extraction from a decoder-only model, hard negative mining, LoRA fine-tuning) is technically coherent. The README now describes a buildable project.

**In context, there's a prerequisite problem.**

The reranker's job is to improve retrieval quality. But the current stack has **no measurement of retrieval quality**. You can't improve what you haven't measured.

Here's what I found when I examined the actual codebase:

| Component | State | Implication for Reranker |
|:---|:---|:---|
| Search API | Working, but quality unknown | No baseline MRR/NDCG to beat |
| Graph traversal | Shallow — 1 hop only | The differentiator is under-implemented |
| Embeddings | OpenAI or hash-based fallback | Fallback produces random vectors — not a real degradation path |
| Tests | Placeholder only (6 lines in engine, zero in platform) | No way to verify reranker integration doesn't break anything |
| Benchmarks | None | No dataset of queries with known-good results |

The reranker is an optimization layer for a retrieval system that hasn't been evaluated yet. You're building a turbocharger before you've dyno'd the engine.

---

## 3. Does It Align With the Stack?

**The vision** (`vision.md`) says: *"Simplicity of Operations (Docker Compose) over Scale at all costs."*

**The monetization strategy** (`monetization-strategy.md`) says: *"The Dual-DB architecture (Neo4j + Qdrant) is your moat."*

**The market analysis** (`market-analysis.md`) says: *"Be the Postgres of Context — the default building block people spin up when they need code search."*

A custom-trained reranker model introduces:
- A GPU or Apple Silicon requirement for inference (contradicts "simple Docker Compose")
- A model artifact to version, store, and ship (contradicts "single binary" simplicity)
- A training pipeline to maintain (contradicts "hackability" positioning)
- A dependency on a specific model family (Qwen) that will need updating as models evolve

**This is an enterprise feature being built at an indie stage.** The engine roadmap (`ROADMAP.md`) still has unchecked Phase 1 items. The platform architecture spec (`architecture.md`) describes Phase 3-4 features (billing, metering, BYOK encryption) that haven't been started. The stack is pre-alpha by its own admission.

The reranker aligns with the *long-term* vision of Clew as a sophisticated context platform. It does not align with the *current* priority of getting a working, testable, shippable product into users' hands.

---

## 4. Is It Time to Do This NOW?

**No.**

Here's what's between you and alpha, ordered by impact:

| Priority | What | Why It Matters More Than a Reranker |
|:---|:---|:---|
| **1** | Integration tests for engine + platform | You can't ship what you can't verify. Zero test coverage is a blocker. |
| **2** | Deep graph traversal (>1 hop) | This is your stated differentiator over vector-only tools. It's shallow. |
| **3** | Retrieval quality baseline | Run 50 queries against a real repo. Measure what comes back. Find out if your *existing* search is the bottleneck or if it's the graph depth. |
| **4** | Error handling and degradation paths | The embedding fallback produces random vectors. That's not graceful degradation — it's silent corruption. |
| **5** | End-to-end demo on a real repo | Can a user actually `docker-compose up`, point at a GitHub repo, and get useful results from the MCP tools? |

A reranker is item **6 or 7** on this list. It's a quality-of-retrieval optimization that presupposes retrieval is working well enough to measure and worth optimizing.

---

## 5. How Big of a Win Would This Be?

**Marginal, at this stage.**

The reranker improves the *ordering* of results that the search API returns. But the current limiting factors are:

1. **Graph depth** — you're only traversing 1 hop, so "code + neighbors" means "code + immediate imports/callers." The real value of Graph RAG is multi-hop reasoning ("this function calls X, which depends on Y, which was changed in PR Z"). That's not implemented.

2. **Embedding quality** — you're using `text-embedding-3-small`, which is a general-purpose model. A code-specific embedding model (like `voyage-code-3` or fine-tuned `nomic-embed-code`) would improve retrieval quality more than a reranker, with zero training infrastructure.

3. **Result formatting** — the MCP strategy doc describes returning "code + how it's used," but the actual API returns raw search results. The presentation layer matters as much as ranking.

A reranker on top of shallow traversal with general-purpose embeddings is polishing a link in the middle of a chain with weak links on both sides.

**Realistic impact estimate**: If current retrieval quality is at 60% MRR, a reranker might bring it to 70-75%. Fixing graph depth could bring it to 80%. Better embeddings could bring it to 85%. The reranker is the smallest individual lever available.

---

## 6. Should It Block Alpha Release?

**Absolutely not.**

The alpha should ship with:
- Working search + graph (current state, hardened)
- Signature + edges retrieval mode (the "Phase 1 Baseline" everyone agreed on — no model needed)
- Tests proving the above works
- A real demo on a real repo

The reranker is a post-alpha optimization. If alpha users report "search returns too much noise," *that's* the signal to build the reranker — and you'll have real user queries to train on instead of synthetic data.

---

## 7. Open Source or Enterprise Layer?

**Enterprise, clearly.**

| Factor | Open Source | Enterprise |
|:---|:---|:---|
| Deployment complexity | Adds GPU requirement to Docker Compose | Runs as managed service, you handle infra |
| Maintenance burden | Community must retrain when model/schema changes | You control the training pipeline |
| Competitive moat | Gives away your optimization layer | Differentiates paid tier |
| User expectation | OSS users expect "it just works" without GPUs | Enterprise users expect ML-powered features |
| Alignment with monetization | Contradicts "simple self-hosted" pitch | Fits "Managed Cloud" tier perfectly |

The open-source Clew should ship with the "Signature + Edges" baseline as its default retrieval mode. The reranker becomes a feature of the managed/enterprise tier — "Clew Cloud includes AI-powered result ranking." This is exactly the GitLab CE/EE split described in your monetization strategy.

---

## 8. Summary Recommendation

```
NOW:    Tests → Deep Traversal → Quality Baseline → Alpha
LATER:  Better Embeddings → Reranker (Enterprise) → Narrative Compression (Research)
NEVER:  Ship a reranker before you have test coverage
```

The model trainer project should be **paused and shelved** (not deleted — the specs and README are good reference material). Rename it to `clew-model-trainer-research` or move it to a `research/` directory. The engineering hours it would consume are better spent on the 5 items that stand between you and a shippable alpha.

The reranker is a real product feature. It's just not the next thing to build.
