# ADR-006: Clew-Bench Arena for Empirical Validation

**Date:** 2026-02-12
**Status:** Accepted

## Context

After completing Phase 1 (Indie V1) and nearly completing Phase 2 (Team Edition), we have a working system with:
- Multi-repo indexing with gitignore-aware discovery
- Graph + Vector hybrid search
- Quality baseline: MRR 0.54, Recall@3 0.64 (77-80% of target)
- Token efficiency estimates: 15x compression vs filesystem search

However, all efficiency claims are **theoretical calculations** based on:
- Manual token counting of hypothetical queries
- Estimated comparison to grep-based approaches
- No head-to-head agent comparison with controlled workload

Before investing in Phase 3 (Enterprise Core: SSO, audit logging, reranker), we need **empirical validation** that:
1. Clew's graph+vector approach provides measurable token compression
2. Quality improvements (hallucination reduction, refactor success) are real
3. The system is ready for production agent workloads

**The Risk:** Without validation, Phase 3 investment could be premature if:
- Token compression is overstated
- Quality issues emerge at scale
- Standard RAG proves "good enough"

## Decision

We will implement **Clew-Bench**, a comparative agent arena that quantitatively measures Clew's performance against standard RAG on realistic feature development tasks.

### Components

**1. Test Tracks (3 starter repositories)**
- Track A: Python/FastAPI microservice (minimal main.py + Dockerfile)
- Track B: React/Next.js frontend (create-next-app boilerplate)
- Track C: TypeScript/Node.js utility library (package.json + index.ts)

**2. Workload Simulation**
- Structured as `epics.json` with 10-15 tickets across 3 epics
- Simulates real JIRA workflow where context accumulates
- Includes "trap" tickets (e.g., refactoring) that test graph capabilities

**3. Test Harness**
- Side-by-side agent execution (Clew vs Standard RAG)
- Context engine abstraction (swappable backends)
- Judge LLM (Gemini 1.5 Pro) for quality scoring
- Metrics collection: tokens, quality, refactor success

**4. Key Metrics**
- **Context Bloat Curve:** Input tokens vs ticket number (expect flat for Clew, exponential for RAG)
- **Hallucination Rate:** Import errors, non-existent function calls
- **Refactor Success:** Complete vs partial coverage of code changes

### Timeline & Placement

**Roadmap Position:** Phase 2.5 (Validation)
- After Phase 2 completion (PR/Issue linking, async queue)
- Before Phase 3 investment (enterprise features)

**Estimated Effort:** 1-2 weeks
- Starter repos: 2-3 days
- Workload design: 1-2 days
- Test harness: 3-4 days
- Trial runs + analysis: 2-3 days

**Success Criteria:**
- Prove 10x+ token compression (validate current 15x estimate)
- Demonstrate <5% hallucination rate (vs >30% for standard RAG)
- Show 100% refactor success (vs <70% for standard RAG)

## Consequences

### Positive

**De-risks Phase 3 Investment**
- Quantitative proof before building enterprise features
- Identifies optimization opportunities early
- Provides marketing material for enterprise sales

**Creates Reusable Assets**
- Benchmark suite for regression testing
- Public credibility (published results)
- Feedback loop for optimization priorities

**Informs Architectural Decisions**
- Validates current quality (0.54 MRR) is "good enough"
- Identifies if reranker is needed (Phase 3 priority question)
- Reveals edge cases for improvement

**Marketing & Positioning**
- Concrete numbers for pitch deck
- "SWE-bench for feature development" positioning
- Public benchmark others can replicate

### Negative

**Delays Phase 3 Start**
- 1-2 week delay before enterprise features
- Opportunity cost if enterprise customer is waiting
- Team momentum shift from building to validation

**May Reveal Gaps**
- Could show token compression is lower than estimated
- Quality issues might require fixes before proceeding
- Might need to backtrack to Phase 2 optimizations

**Implementation Complexity**
- Requires integrating multiple LLMs (coding agent + judge)
- Need clean comparison framework
- Statistical analysis adds overhead

### Mitigations

**For Delays:**
- Run in parallel with Phase 2 completion (Jules agents working)
- Results in 1-2 weeks provide clear Phase 3 direction
- Can still do Phase 3 planning during benchmark runs

**For Revealed Gaps:**
- Better to find issues now than after enterprise investment
- Benchmark becomes regression test for fixes
- Iterative improvement with measurable progress

**For Complexity:**
- Start with single track (Python/FastAPI) to prove concept
- Expand to 3 tracks only if initial results are promising
- Reuse existing quality measurement infrastructure

## Alternatives Considered

### Alternative 1: Skip Validation, Proceed to Phase 3
**Rationale:** Current quality metrics (MRR 0.54) are acceptable, team is ready

**Rejected Because:**
- Token compression is only theoretical
- No head-to-head comparison exists
- Risk of building enterprise features nobody needs
- Harder to iterate after SSO/audit logging are built

### Alternative 2: Simple A/B Test (No Arena)
**Rationale:** Just run Clew vs RAG on one task, compare tokens

**Rejected Because:**
- Single task doesn't show accumulation effect
- No refactoring test (key differentiator)
- Lacks statistical significance
- Not reusable or publishable

### Alternative 3: User Study (Beta Testers)
**Rationale:** Get real developers to try both systems

**Rejected Because:**
- Slow (weeks to recruit, schedule, analyze)
- Subjective feedback, not quantitative
- Can't control for user skill/bias
- Expensive and hard to replicate

### Alternative 4: Defer to Phase 3+ (Research Track)
**Rationale:** Build enterprise features first, validate later

**Rejected Because:**
- Validation should inform investment, not follow it
- Phase 3 features are expensive (SSO, audit logging)
- Better to pivot early than late
- Benchmark data helps with enterprise sales

## Implementation Plan

### Phase 1: Starter Repositories (2-3 days)
```bash
# Create three minimal repos
mkdir clew-bench && cd clew-bench
./scripts/create-track-a.sh  # FastAPI
./scripts/create-track-b.sh  # Next.js
./scripts/create-track-c.sh  # TypeScript
```

### Phase 2: Workload Design (1-2 days)
```json
# epics.json - 3 epics, 10-15 tickets
{
  "epic_id": "E-101",
  "title": "User Authentication",
  "tickets": [
    {"id": "T-1", "instruction": "Create User model..."},
    {"id": "T-2", "instruction": "Implement POST /register..."},
    {"id": "T-3", "instruction": "Refactor password hashing..."}  // TRAP
  ]
}
```

### Phase 3: Test Harness (3-4 days)
```python
# arena_harness.py
class ArenaHarness:
    def run_experiment(self):
        for ticket in workload:
            # Run control (standard RAG)
            # Run test (Clew)
            # Judge both results
            # Collect metrics
```

### Phase 4: Analysis & Publication (2-3 days)
- Run 100+ trials for statistical significance
- Generate plots (context bloat, quality scores)
- Write blog post with results
- Publish benchmark suite to GitHub

## Related Documents

- **Proposal:** `clew-research/comparative-agent-arena.md`
- **Planning:** `docs/planning/clew-bench-arena.md` (to be created)
- **Roadmap:** Phase 2.5 section in `ROADMAP.md`

## Review & Approval

**Proposed By:** Claude (Session Feb 12, 2026)
**Reviewed By:** [Pending]
**Approved By:** [Pending]

**Decision Date:** 2026-02-12
**Implementation Start:** After Phase 2 completion (~Feb 13-14)
**Expected Completion:** ~Feb 26-28 (1-2 weeks)
