# Clew-Bench Arena - Implementation Plan

**Status:** Phase 2.5 - Validation Track
**Priority:** High (De-risks Phase 3 investment)
**Estimated Effort:** 1-2 weeks
**Dependencies:** Phase 2 completion (PR/Issue linking, async queue)

---

## 🎯 Objective

Build a **Comparative Agent Arena** that quantitatively measures Clew's performance against standard RAG on realistic feature development tasks.

**Purpose:** Empirically validate token compression and quality claims before investing in Phase 3 (SSO, audit logging, reranker).

**Key Questions:**
1. Does Clew provide measurable 10x+ token compression vs standard RAG?
2. Are quality improvements (hallucination reduction, refactor success) real?
3. Is the system ready for production agent workloads?

---

## 📊 Success Criteria

**Must Prove:**
- **10x+ Token Compression**: Validate current 15x estimate with real agent runs
- **<5% Hallucination Rate**: Import errors, non-existent function calls (vs >30% for RAG)
- **100% Refactor Success**: Complete coverage of code changes (vs <70% for RAG)

**Key Metrics:**
1. **Context Bloat Curve**: Input tokens vs ticket number (flat for Clew, exponential for RAG)
2. **Hallucination Rate**: Judge LLM detects incorrect imports/calls
3. **Refactor Success**: "Trap" tickets test graph relationship awareness

---

## 🏗️ Architecture Overview

```
┌─────────────────────────────────────────────────────────────┐
│                     Arena Harness                            │
│  (Python script that runs side-by-side experiments)         │
└─────────────────────────────────────────────────────────────┘
                           │
           ┌───────────────┴───────────────┐
           │                               │
   ┌───────▼──────┐               ┌───────▼──────┐
   │   Control    │               │     Test     │
   │ (Standard    │               │    (Clew     │
   │    RAG)      │               │   Engine)    │
   └───────┬──────┘               └───────┬──────┘
           │                               │
           └───────────────┬───────────────┘
                           │
                   ┌───────▼──────┐
                   │  Judge LLM   │
                   │ (Gemini Pro) │
                   └──────────────┘
```

**Flow:**
1. Load workload from `epics.json` (10-15 tickets across 3 epics)
2. For each ticket:
   - Run Control agent (Standard RAG) → Generate code
   - Run Test agent (Clew Engine) → Generate code
   - Judge LLM evaluates both submissions
   - Collect metrics (tokens, quality scores)
3. Generate final report with plots

---

## 📋 Implementation Tasks

### Phase 1: Starter Repositories (2-3 days)

Create three minimal repositories to ensure data isn't biased toward one language.

**Track A: Python/FastAPI Microservice**
```bash
mkdir -p clew-bench/track-a-python
cd clew-bench/track-a-python

# Create minimal structure
touch main.py Dockerfile requirements.txt
cat > main.py <<EOF
from fastapi import FastAPI

app = FastAPI()

@app.get("/health")
async def health_check():
    return {"status": "ok"}
EOF

cat > requirements.txt <<EOF
fastapi==0.109.0
uvicorn[standard]==0.27.0
pydantic==2.5.0
EOF

cat > Dockerfile <<EOF
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
EOF
```

**Track B: React/Next.js Frontend**
```bash
npx create-next-app@latest track-b-nextjs --typescript --tailwind --app --no-src-dir
cd track-b-nextjs
# Remove example components, keep boilerplate only
rm -rf app/page.tsx app/layout.tsx components/*
# Create minimal layout
```

**Track C: TypeScript/Node.js Utility Library**
```bash
mkdir -p track-c-typescript
cd track-c-typescript
npm init -y
npm install --save-dev typescript @types/node

cat > src/index.ts <<EOF
export function hello(): string {
    return "Hello, World!";
}
EOF

cat > tsconfig.json <<EOF
{
  "compilerOptions": {
    "target": "ES2020",
    "module": "commonjs",
    "outDir": "./dist",
    "rootDir": "./src",
    "strict": true
  }
}
EOF
```

**Acceptance:**
- [ ] 3 repositories created with minimal boilerplate
- [ ] Each repo is <50 KB (small starting point)
- [ ] Repos are valid projects (can run `npm install`, `pip install`, etc.)

---

### Phase 2: Workload Design (1-2 days)

Design a realistic feature development workflow that simulates JIRA tickets.

**File:** `clew-bench/workloads/auth-epic.json`

```json
{
  "workload_id": "auth-epic-v1",
  "description": "Simulates building user authentication feature from scratch",
  "track": "python-fastapi",
  "epics": [
    {
      "epic_id": "E-101",
      "title": "User Authentication",
      "tickets": [
        {
          "id": "T-1",
          "instruction": "Create a User model in the database schema with fields: email (unique), password_hash, role (default 'user'), created_at. Use Pydantic for validation.",
          "success_criteria": "User model defined; Pydantic schema created; migration script exists.",
          "trap": false
        },
        {
          "id": "T-2",
          "instruction": "Implement POST /register endpoint that accepts email and password, hashes the password using bcrypt, and saves the user to the database. Return 201 Created on success, 400 Bad Request if email already exists.",
          "success_criteria": "Endpoint implemented; password hashing works; duplicate emails rejected.",
          "trap": false
        },
        {
          "id": "T-3",
          "instruction": "Refactor the password hashing logic from the /register endpoint into a separate SecurityService class. Update the endpoint to use this service.",
          "success_criteria": "SecurityService created; /register updated to use it; all imports correct.",
          "trap": true,
          "notes": "This is the TRAP ticket. Standard RAG often fails to find where T-2 put the code. Clew's graph should see the connection immediately via DEFINES/CALLS relationships."
        }
      ]
    },
    {
      "epic_id": "E-102",
      "title": "Session Management",
      "tickets": [
        {
          "id": "T-4",
          "instruction": "Create a JWT token generation utility that takes a user_id and returns a signed token with 24-hour expiration.",
          "success_criteria": "JWT utility created; token includes user_id; expiration set correctly.",
          "trap": false
        },
        {
          "id": "T-5",
          "instruction": "Implement POST /login endpoint that verifies email/password and returns a JWT token on success.",
          "success_criteria": "Endpoint implemented; uses SecurityService for password verification; returns JWT.",
          "trap": false
        }
      ]
    },
    {
      "epic_id": "E-103",
      "title": "Protected Routes",
      "tickets": [
        {
          "id": "T-6",
          "instruction": "Create an authentication middleware that validates JWT tokens and attaches user context to requests.",
          "success_criteria": "Middleware created; validates tokens; rejects invalid tokens with 401.",
          "trap": false
        },
        {
          "id": "T-7",
          "instruction": "Add a protected GET /profile endpoint that returns the current user's information. Require authentication.",
          "success_criteria": "Endpoint created; middleware applied; returns user data for authenticated requests.",
          "trap": false
        },
        {
          "id": "T-8",
          "instruction": "Refactor all authentication-related endpoints to use consistent error responses. Extract error handling into a shared utility.",
          "success_criteria": "Error utility created; all endpoints use it; consistent error format.",
          "trap": true,
          "notes": "Another TRAP. Tests graph awareness across multiple files."
        }
      ]
    }
  ]
}
```

**Design Principles:**
- **Progressive Complexity**: Each ticket builds on previous work
- **Trap Tickets**: Every epic has at least one "refactoring" ticket that tests graph awareness
- **Real Patterns**: Matches actual JIRA workflows (feature → refactor → extend)

**Acceptance:**
- [ ] 3 epics with 8-15 tickets total
- [ ] 2-3 "trap" tickets per workload
- [ ] Each ticket has clear success criteria
- [ ] Workload simulates realistic feature development

---

### Phase 3: Test Harness (3-4 days)

Build the arena harness that runs agents side-by-side.

**File:** `clew-bench/arena_harness.py`

```python
"""
Clew-Bench Arena Harness

Runs side-by-side experiments comparing Clew Engine vs Standard RAG.
"""
import json
import time
import logging
from dataclasses import dataclass, field
from typing import Any

logger = logging.getLogger(__name__)


@dataclass
class AgentStats:
    """Tracks metrics for one agent across all tickets."""
    input_tokens: int = 0
    output_tokens: int = 0
    api_cost: float = 0.0
    quality_scores: list[float] = field(default_factory=list)
    hallucination_count: int = 0
    refactor_successes: int = 0
    refactor_failures: int = 0


class ContextEngine:
    """Abstract interface for context retrieval (swappable backends)."""

    def query(self, instruction: str, mode: str) -> str:
        """
        Retrieve context for code generation.

        Args:
            instruction: The ticket instruction
            mode: "clew" or "standard"

        Returns:
            Context string to pass to coding LLM
        """
        raise NotImplementedError


class ClewContextEngine(ContextEngine):
    """Clew Engine: Graph + Vector Hybrid Search."""

    def __init__(self, api_url: str):
        self.api_url = api_url

    def query(self, instruction: str, mode: str) -> str:
        # Call Clew API: POST /v1/search
        # Use graph expansion for imports/calls
        # Return formatted context
        pass


class StandardRAGEngine(ContextEngine):
    """Standard RAG: Vector Search (Top-K) only."""

    def __init__(self, qdrant_url: str):
        self.qdrant_url = qdrant_url

    def query(self, instruction: str, mode: str) -> str:
        # Query Qdrant directly (no graph)
        # Return top-K results
        pass


class ArenaHarness:
    """Main test harness for running experiments."""

    def __init__(
        self,
        workload_file: str,
        clew_engine: ContextEngine,
        standard_engine: ContextEngine,
        judge_model: str = "gemini-1.5-pro",
    ):
        with open(workload_file, "r") as f:
            self.workload = json.load(f)

        self.clew_engine = clew_engine
        self.standard_engine = standard_engine
        self.judge_model = judge_model

        self.stats = {
            "control": AgentStats(),
            "test": AgentStats(),
        }

    def run_agent(
        self, agent_type: str, ticket: dict, context_engine: ContextEngine
    ) -> dict[str, Any]:
        """
        Run one agent on one ticket.

        Args:
            agent_type: "control" or "test"
            ticket: Ticket data from workload
            context_engine: Context retrieval backend

        Returns:
            {
                "code": "generated code",
                "usage": {"input": int, "output": int},
                "context_size": int
            }
        """
        logger.info(f"🤖 Running {agent_type} on {ticket['id']}...")

        # 1. Retrieve context
        start_t = time.time()
        context = context_engine.query(
            ticket["instruction"], mode=agent_type
        )
        retrieval_time = time.time() - start_t

        # 2. Call coding LLM (e.g., Claude Sonnet)
        # In real implementation: call Anthropic API with context + instruction
        response = self._call_coding_llm(ticket["instruction"], context)

        # 3. Track metrics
        self.stats[agent_type].input_tokens += response["usage"]["input"]
        self.stats[agent_type].output_tokens += response["usage"]["output"]

        logger.info(
            f"   ⏱️ Retrieval: {retrieval_time:.2f}s | "
            f"Tokens: {response['usage']['input']} in, {response['usage']['output']} out"
        )

        return {
            "code": response["code"],
            "usage": response["usage"],
            "context_size": len(context),
        }

    def evaluate_result(
        self, ticket: dict, code: str, agent_type: str
    ) -> dict[str, Any]:
        """
        Use Judge LLM to score the generated code.

        Args:
            ticket: Ticket data
            code: Generated code
            agent_type: "control" or "test"

        Returns:
            {
                "correctness": int (1-5),
                "efficiency": int (1-5),
                "hallucinations": list[str],
                "comments": str
            }
        """
        prompt = f"""
You are a Senior Staff Engineer reviewing a Junior Developer's code.

TASK: {ticket['instruction']}
SUCCESS CRITERIA: {ticket['success_criteria']}

SUBMITTED CODE:
{code}

Rate this solution on:
1. Correctness (1-5): Does it meet the success criteria?
2. Efficiency (1-5): Is it clean, idiomatic, and maintainable?
3. Hallucinations: List any imports or function calls that don't exist

Output JSON:
{{
    "correctness": int,
    "efficiency": int,
    "hallucinations": [list of strings],
    "comments": "brief explanation"
}}
"""

        # Call Judge LLM (Gemini 1.5 Pro)
        judgement = self._call_judge_llm(prompt)

        # Track scores
        avg_score = (judgement["correctness"] + judgement["efficiency"]) / 2
        self.stats[agent_type].quality_scores.append(avg_score)

        # Track hallucinations
        if judgement["hallucinations"]:
            self.stats[agent_type].hallucination_count += len(
                judgement["hallucinations"]
            )

        # Track refactor success (for trap tickets)
        if ticket.get("trap", False):
            if judgement["correctness"] >= 4:
                self.stats[agent_type].refactor_successes += 1
            else:
                self.stats[agent_type].refactor_failures += 1

        logger.info(f"   ⚖️ Judge Score ({agent_type}): {avg_score:.1f}/5")
        if judgement["hallucinations"]:
            logger.warning(
                f"   ⚠️ Hallucinations: {judgement['hallucinations']}"
            )

        return judgement

    def run_experiment(self):
        """Run full experiment across all tickets."""
        for epic in self.workload["epics"]:
            logger.info(f"\n🚀 Starting Epic: {epic['title']}")
            for ticket in epic["tickets"]:
                logger.info(f"\n📋 Ticket {ticket['id']}: {ticket['instruction'][:60]}...")

                # Run Control (Standard RAG)
                result_c = self.run_agent(
                    "control", ticket, self.standard_engine
                )
                judge_c = self.evaluate_result(
                    ticket, result_c["code"], "control"
                )

                # Run Test (Clew)
                result_t = self.run_agent("test", ticket, self.clew_engine)
                judge_t = self.evaluate_result(
                    ticket, result_t["code"], "test"
                )

        self.print_final_report()

    def print_final_report(self):
        """Print final metrics comparison."""
        print("\n" + "=" * 60)
        print("📊 CLEW-BENCH RESULTS")
        print("=" * 60)

        # Token Usage
        print("\n🔢 Token Usage:")
        print(f"  Control (RAG):  {self.stats['control'].input_tokens:,} tokens")
        print(f"  Test (Clew):    {self.stats['test'].input_tokens:,} tokens")

        savings = (
            1 - (self.stats["test"].input_tokens / self.stats["control"].input_tokens)
        ) * 100
        print(f"  🎉 COMPRESSION: {savings:.1f}% fewer tokens")

        # Quality
        print("\n⭐ Quality Scores:")
        avg_control = sum(self.stats["control"].quality_scores) / len(
            self.stats["control"].quality_scores
        )
        avg_test = sum(self.stats["test"].quality_scores) / len(
            self.stats["test"].quality_scores
        )
        print(f"  Control (RAG):  {avg_control:.2f}/5.0")
        print(f"  Test (Clew):    {avg_test:.2f}/5.0")

        # Hallucinations
        print("\n🔍 Hallucination Rate:")
        print(f"  Control (RAG):  {self.stats['control'].hallucination_count} hallucinations")
        print(f"  Test (Clew):    {self.stats['test'].hallucination_count} hallucinations")

        # Refactor Success
        print("\n🔧 Refactor Success (Trap Tickets):")
        print(
            f"  Control (RAG):  {self.stats['control'].refactor_successes}/"
            f"{self.stats['control'].refactor_successes + self.stats['control'].refactor_failures}"
        )
        print(
            f"  Test (Clew):    {self.stats['test'].refactor_successes}/"
            f"{self.stats['test'].refactor_successes + self.stats['test'].refactor_failures}"
        )

        print("\n" + "=" * 60)

    def _call_coding_llm(self, instruction: str, context: str) -> dict:
        """Placeholder: Call coding LLM (e.g., Claude Sonnet)."""
        # TODO: Implement Anthropic API call
        return {
            "code": "# Generated code here",
            "usage": {"input": 1000, "output": 500},
        }

    def _call_judge_llm(self, prompt: str) -> dict:
        """Placeholder: Call judge LLM (Gemini 1.5 Pro)."""
        # TODO: Implement Google AI API call
        return {
            "correctness": 4,
            "efficiency": 4,
            "hallucinations": [],
            "comments": "Good implementation",
        }


# Usage:
# harness = ArenaHarness(
#     workload_file="workloads/auth-epic.json",
#     clew_engine=ClewContextEngine("http://localhost:8001"),
#     standard_engine=StandardRAGEngine("http://localhost:6335"),
# )
# harness.run_experiment()
```

**Key Design Decisions:**
- **Swappable Context Engines**: Easy to compare different backends
- **Judge LLM**: Independent quality scoring (not self-grading)
- **Comprehensive Metrics**: Tokens, quality, hallucinations, refactor success

**Acceptance:**
- [ ] Harness can run both agents side-by-side
- [ ] Metrics collected for tokens, quality, hallucinations
- [ ] Judge LLM integration works
- [ ] Final report generates with plots

---

### Phase 4: Analysis & Publication (2-3 days)

Run experiments and publish results.

**Tasks:**
1. **Run 100+ trials** for statistical significance
2. **Generate plots**:
   - Context bloat curve (tokens vs ticket number)
   - Quality score distribution
   - Hallucination rate comparison
3. **Write blog post** with results
4. **Publish benchmark suite** to GitHub
5. **Update roadmap** based on findings

**Acceptance:**
- [ ] 100+ trial runs completed
- [ ] Statistical analysis complete (mean, std dev, confidence intervals)
- [ ] Plots generated (matplotlib or similar)
- [ ] Blog post drafted
- [ ] Benchmark published to `clew-bench` repo

---

## 🧪 Testing Plan

### Local Development

**Setup:**
```bash
cd clew-bench
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Start Clew services
cd ../clew-engine
docker-compose up -d

# Ingest test track
cd ../clew-bench/track-a-python
python ../../clew-engine/packages/clew-ingestion/src/main.py \
  --repo-path . \
  --repo-url https://github.com/user/track-a-python
```

**Run Harness:**
```bash
python arena_harness.py \
  --workload workloads/auth-epic.json \
  --clew-url http://localhost:8001 \
  --qdrant-url http://localhost:6335 \
  --trials 10
```

**Expected Output:**
```
🚀 Starting Epic: User Authentication
📋 Ticket T-1: Create a User model...
🤖 Running control on T-1...
   ⏱️ Retrieval: 0.32s | Tokens: 2400 in, 180 out
   ⚖️ Judge Score (control): 4.0/5
🤖 Running test on T-1...
   ⏱️ Retrieval: 0.18s | Tokens: 850 in, 175 out
   ⚖️ Judge Score (test): 4.5/5

...

📊 CLEW-BENCH RESULTS
🔢 Token Usage:
  Control (RAG):  24,500 tokens
  Test (Clew):    1,850 tokens
  🎉 COMPRESSION: 92.4% fewer tokens

⭐ Quality Scores:
  Control (RAG):  3.8/5.0
  Test (Clew):    4.5/5.0

🔍 Hallucination Rate:
  Control (RAG):  8 hallucinations
  Test (Clew):    0 hallucinations

🔧 Refactor Success (Trap Tickets):
  Control (RAG):  1/3
  Test (Clew):    3/3
```

---

## 📚 Resources

**SWE-bench Reference:**
- https://www.swebench.com/
- https://github.com/princeton-nlp/SWE-bench

**LLM APIs:**
- Anthropic (Claude Sonnet): https://docs.anthropic.com/claude/reference/messages_post
- Google AI (Gemini Pro): https://ai.google.dev/docs

**Plotting:**
- Matplotlib: https://matplotlib.org/
- Seaborn: https://seaborn.pydata.org/

**Statistical Analysis:**
- SciPy: https://scipy.org/
- Pandas: https://pandas.pydata.org/

---

## ⚠️ Known Limitations

- **Cost**: 100+ trials with Claude Sonnet + Gemini Pro = ~$50-100 in API costs
- **Time**: Each trial takes ~5-10 minutes (parallelization possible)
- **Single Track**: Start with Python/FastAPI only, expand if promising
- **Manual Verification**: Judge LLM scores should be spot-checked by humans

---

## 💡 Future Enhancements

- **Multi-language support**: Run all 3 tracks (Python, Next.js, TypeScript)
- **Historical comparison**: Benchmark against Clew v1.0, v2.0, etc.
- **Public leaderboard**: Let others submit context engines
- **Automated CI**: Run regression tests on every release

---

## 🚀 Getting Started

**Quickstart (Single Trial):**
```bash
# 1. Create Track A repository
./scripts/create-track-a.sh

# 2. Ingest into Clew
cd clew-engine
docker-compose up -d
python packages/clew-ingestion/src/main.py --repo-path ../clew-bench/track-a-python

# 3. Run harness
cd ../clew-bench
python arena_harness.py --workload workloads/auth-epic.json --trials 1

# 4. Review results
cat results/latest_run.json
```

**Next Steps:**
1. Review `clew-research/comparative-agent-arena.md` for context
2. Set up development environment
3. Create Track A (Python/FastAPI)
4. Design first workload (auth-epic.json)
5. Implement minimal harness
6. Run single trial → iterate

---

**Related Documents:**
- **ADR:** `docs/adr/006-clew-bench-validation.md`
- **Proposal:** `clew-research/comparative-agent-arena.md`
- **Roadmap:** Phase 2.5 section in `ROADMAP.md`
