# DevBench Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Practical benchmark measuring mcp-memory-service retrieval quality using real developer workflow memories from the project itself.

**Architecture:** Single JSON dataset file with 30 memories + 20 queries, one benchmark script reusing `locomo_evaluator.py` for metrics, tag-based evidence matching.

**Tech Stack:** Python 3.11, asyncio, SQLite-Vec, ONNX embeddings (all-MiniLM-L6-v2)

**Spec:** `docs/superpowers/specs/2026-03-31-devbench-design.md`

---

### Task 1: Create the Dataset

**Files:**
- Create: `scripts/benchmarks/devbench_dataset.json`

- [ ] **Step 1: Create the dataset JSON file**

The dataset contains 30 memories (6 each of: decision, learning, bug-fix, architecture, config) and 20 queries (5 each of: exact, semantic, cross-type, negative). All content derived from the mcp-memory-service project.

Each memory has: content, tags (including unique `bench:<type>-<N>` tag), memory_type, created_at_iso.
Each query has: question, expected_tags (list of bench:* tags), category.

See the spec for the complete JSON structure. The full dataset content is provided in the implementation step of Task 2 since the implementer will create both files together.

- [ ] **Step 2: Validate JSON is well-formed**

Run: `.venv/bin/python -c "import json; d=json.load(open('scripts/benchmarks/devbench_dataset.json')); print(f'Memories: {len(d[\"memories\"])}, Queries: {len(d[\"queries\"])}')"`
Expected: `Memories: 30, Queries: 20`

- [ ] **Step 3: Commit**

```bash
git add scripts/benchmarks/devbench_dataset.json
git commit -m "feat(benchmark): add DevBench dataset -- 30 memories, 20 queries from project"
```

---

### Task 2: Benchmark Script and Tests

**Files:**
- Create: `scripts/benchmarks/benchmark_devbench.py`
- Create: `tests/benchmarks/test_benchmark_devbench.py`

- [ ] **Step 1: Write failing tests**

```python
# tests/benchmarks/test_benchmark_devbench.py
"""Tests for the DevBench practical benchmark."""
import asyncio
import json
import os
import shutil
import sys

import pytest

sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "src"))
from benchmark_devbench import (
    load_devbench_dataset,
    ingest_memories,
    evaluate_retrieval,
    match_by_tags,
    create_isolated_storage,
)


class TestLoadDataset:
    def test_loads_memories_and_queries(self):
        dataset = load_devbench_dataset()
        assert len(dataset["memories"]) == 30
        assert len(dataset["queries"]) == 20

    def test_all_memories_have_bench_tag(self):
        dataset = load_devbench_dataset()
        for mem in dataset["memories"]:
            bench_tags = [t for t in mem["tags"] if t.startswith("bench:")]
            assert len(bench_tags) >= 1, f"Missing bench tag: {mem['tags']}"

    def test_query_categories(self):
        dataset = load_devbench_dataset()
        categories = {q["category"] for q in dataset["queries"]}
        assert categories == {"exact", "semantic", "cross-type", "negative"}


class TestMatchByTags:
    def test_match_found(self):
        class FakeMemory:
            tags = ["devbench", "bench:decision-1", "storage"]
        class FakeResult:
            memory = FakeMemory()
        retrieved = [FakeResult()]
        labels, relevant = match_by_tags(retrieved, ["bench:decision-1"])
        assert "bench:decision-1" in labels
        assert relevant == {"bench:decision-1"}

    def test_no_match(self):
        class FakeMemory:
            tags = ["devbench", "bench:config-1"]
        class FakeResult:
            memory = FakeMemory()
        retrieved = [FakeResult()]
        labels, relevant = match_by_tags(retrieved, ["bench:decision-1"])
        assert "bench:decision-1" not in labels
        assert relevant == {"bench:decision-1"}

    def test_negative_query(self):
        class FakeMemory:
            tags = ["devbench", "bench:config-1"]
        class FakeResult:
            memory = FakeMemory()
        retrieved = [FakeResult()]
        labels, relevant = match_by_tags(retrieved, [])
        assert relevant == set()


class TestIngestAndRetrieve:
    def test_ingest_all_memories(self):
        dataset = load_devbench_dataset()
        storage, tmp_dir = create_isolated_storage()
        try:
            count = asyncio.run(self._run(storage, dataset))
            assert count == 30
        finally:
            shutil.rmtree(tmp_dir, ignore_errors=True)

    async def _run(self, storage, dataset):
        await storage.initialize()
        return await ingest_memories(storage, dataset["memories"])

    def test_retrieve_exact_query(self):
        dataset = load_devbench_dataset()
        storage, tmp_dir = create_isolated_storage()
        try:
            results = asyncio.run(self._run_retrieve(storage, dataset))
            recalls = [r["recall_at_5"] for r in results if r["category"] == "exact"]
            assert any(r > 0 for r in recalls), "No exact queries found relevant memories"
        finally:
            shutil.rmtree(tmp_dir, ignore_errors=True)

    async def _run_retrieve(self, storage, dataset):
        await storage.initialize()
        await ingest_memories(storage, dataset["memories"])
        return await evaluate_retrieval(storage, dataset["queries"], top_k=[5])
```

- [ ] **Step 2: Run tests to verify they fail**

Run: `.venv/bin/pytest tests/benchmarks/test_benchmark_devbench.py -v`
Expected: FAIL -- `ModuleNotFoundError`

- [ ] **Step 3: Implement the benchmark script**

Create `scripts/benchmarks/benchmark_devbench.py` with these components:

**Imports and constants:**
```python
#!/usr/bin/env python3
"""DevBench -- Practical memory retrieval benchmark.

Usage:
    python scripts/benchmarks/benchmark_devbench.py
    python scripts/benchmarks/benchmark_devbench.py --top-k 3 5 10
    python scripts/benchmarks/benchmark_devbench.py --ablation
    python scripts/benchmarks/benchmark_devbench.py --markdown
"""
import argparse
import asyncio
import hashlib
import json
import logging
import os
import shutil
import sys
import tempfile
from collections import defaultdict
from datetime import datetime
from typing import Any, Dict, List, Optional, Tuple

sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "src"))

from mcp_memory_service.models.memory import Memory
from mcp_memory_service.storage.sqlite_vec import SqliteVecMemoryStorage
from locomo_evaluator import (
    BenchmarkResult, aggregate_results, mrr, precision_at_k, recall_at_k,
)

logger = logging.getLogger(__name__)
DATASET_PATH = os.path.join(os.path.dirname(__file__), "devbench_dataset.json")
```

**Core functions:**
```python
def load_devbench_dataset(path: Optional[str] = None) -> Dict[str, Any]:
    with open(path or DATASET_PATH, "r", encoding="utf-8") as f:
        return json.load(f)

def create_isolated_storage() -> Tuple[SqliteVecMemoryStorage, str]:
    tmp_dir = tempfile.mkdtemp(prefix="devbench-")
    return SqliteVecMemoryStorage(os.path.join(tmp_dir, "memories.db")), tmp_dir

async def ingest_memories(storage, memories: List[Dict]) -> int:
    stored = 0
    for mem in memories:
        content_hash = hashlib.sha256(mem["content"].encode("utf-8")).hexdigest()
        created_at = None
        if "created_at_iso" in mem:
            try:
                dt = datetime.fromisoformat(mem["created_at_iso"].replace("Z", "+00:00"))
                created_at = int(dt.timestamp())
            except ValueError:
                pass
        memory = Memory(
            content=mem["content"], content_hash=content_hash,
            tags=mem.get("tags", []), memory_type=mem.get("memory_type", ""),
            created_at=created_at,
        )
        success, _ = await storage.store(memory, skip_semantic_dedup=True)
        if success:
            stored += 1
    return stored

def match_by_tags(retrieved_results, expected_tags: List[str]) -> Tuple[List[str], set]:
    relevant_labels = set(expected_tags)
    retrieved_labels = []
    for result in retrieved_results:
        tags = result.memory.tags or []
        matched = None
        for exp_tag in expected_tags:
            if exp_tag in tags:
                matched = exp_tag
                break
        retrieved_labels.append(matched if matched else f"irrelevant_{len(retrieved_labels)}")
    return retrieved_labels, relevant_labels

def false_positive_rate(retrieved_results, k: int) -> float:
    top_k = retrieved_results[:k]
    if not top_k:
        return 0.0
    fp = sum(1 for r in top_k if any(t.startswith("bench:") for t in (r.memory.tags or [])))
    return fp / len(top_k)
```

**Evaluation functions:**
```python
async def evaluate_retrieval(storage, queries: List[Dict], top_k=None) -> List[Dict]:
    if top_k is None:
        top_k = [5]
    max_k = max(top_k)
    per_question = []
    for q in queries:
        results = await storage.retrieve(q["question"], n_results=max_k)
        expected = q.get("expected_tags", [])
        category = q.get("category", "unknown")
        if category == "negative":
            metrics: Dict[str, Any] = {"category": category}
            for k in top_k:
                metrics[f"false_positive_rate_at_{k}"] = false_positive_rate(results, k)
            per_question.append(metrics)
        else:
            retrieved_labels, relevant_labels = match_by_tags(results, expected)
            metrics = {"category": category}
            for k in top_k:
                metrics[f"recall_at_{k}"] = recall_at_k(retrieved_labels, relevant_labels, k)
                metrics[f"precision_at_{k}"] = precision_at_k(retrieved_labels, relevant_labels, k)
            metrics["mrr"] = mrr(retrieved_labels, relevant_labels)
            per_question.append(metrics)
    return per_question

async def evaluate_ablation(storage, queries: List[Dict], top_k=None) -> List[Dict]:
    if top_k is None:
        top_k = [5]
    max_k = max(top_k)
    configs = [("baseline", "retrieve", {}), ("+quality_boost", "retrieve_with_quality_boost", {})]
    all_results = []
    for config_name, method_name, kwargs in configs:
        retrieve_fn = getattr(storage, method_name)
        for q in queries:
            if q.get("category") == "negative":
                continue
            results = await retrieve_fn(q["question"], n_results=max_k, **kwargs)
            retrieved_labels, relevant_labels = match_by_tags(results, q.get("expected_tags", []))
            metrics: Dict[str, Any] = {"config_name": config_name, "category": q["category"]}
            for k in top_k:
                metrics[f"recall_at_{k}"] = recall_at_k(retrieved_labels, relevant_labels, k)
                metrics[f"precision_at_{k}"] = precision_at_k(retrieved_labels, relevant_labels, k)
            metrics["mrr"] = mrr(retrieved_labels, relevant_labels)
            all_results.append(metrics)
    return all_results
```

**Output formatting (same pattern as benchmark_locomo.py):**
```python
def format_results_table(result: BenchmarkResult) -> str:
    lines = [f"\n{'='*60}", f"  DevBench -- {result.mode.upper()}", f"{'='*60}"]
    lines.append(f"\nOverall:")
    for key, val in sorted(result.overall.items()):
        lines.append(f"  {key:<25} {val:>10.4f}")
    if result.by_category:
        lines.append(f"\nBy Category:")
        all_keys = sorted({k for m in result.by_category.values() for k in m})
        lines.append(f"  {'Category':<15}" + "".join(f"  {k:<18}" for k in all_keys))
        lines.append("  " + "-" * (15 + 20 * len(all_keys)))
        for cat, metrics in sorted(result.by_category.items()):
            lines.append(f"  {cat:<15}" + "".join(f"  {metrics.get(k,0):<18.4f}" for k in all_keys))
    lines.append(f"\n{'='*60}\n")
    return "\n".join(lines)

def format_results_markdown(result: BenchmarkResult) -> str:
    if not result.overall:
        return "No results."
    keys = sorted(result.overall.keys())
    lines = [f"## DevBench -- {result.mode.upper()}", "",
             "| Category | " + " | ".join(keys) + " |",
             "|" + "---|" * (len(keys) + 1)]
    lines.append(f"| **Overall** | " + " | ".join(f"{result.overall[k]:.4f}" for k in keys) + " |")
    for cat in sorted(result.by_category.keys()):
        lines.append(f"| {cat} | " + " | ".join(f"{result.by_category[cat].get(k,0):.4f}" for k in keys) + " |")
    return "\n".join(lines)
```

**Main pipeline and CLI:**
```python
async def run_benchmark(args):
    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s: %(message)s")
    dataset = load_devbench_dataset()
    logger.info("Loaded DevBench: %d memories, %d queries", len(dataset["memories"]), len(dataset["queries"]))
    top_k = args.top_k or [5]
    storage, tmp_dir = create_isolated_storage()
    try:
        await storage.initialize()
        count = await ingest_memories(storage, dataset["memories"])
        logger.info("Ingested %d memories", count)
        if args.ablation:
            per_q = await evaluate_ablation(storage, dataset["queries"], top_k=top_k)
            by_config = defaultdict(list)
            for q in per_q:
                by_config[q["config_name"]].append(q)
            return [aggregate_results(qs, "devbench", f"ablation:{cn}", {"top_k": top_k})
                    for cn, qs in by_config.items()]
        else:
            per_q = await evaluate_retrieval(storage, dataset["queries"], top_k=top_k)
            normal = [q for q in per_q if q["category"] != "negative"]
            negative = [q for q in per_q if q["category"] == "negative"]
            result = aggregate_results(normal, "devbench", "retrieval", {"top_k": top_k})
            if negative:
                neg_metrics = {}
                for key in negative[0]:
                    if key != "category":
                        vals = [q[key] for q in negative if key in q]
                        neg_metrics[key] = sum(vals) / len(vals) if vals else 0.0
                result.by_category["negative"] = neg_metrics
            return result
    finally:
        shutil.rmtree(tmp_dir, ignore_errors=True)

def parse_args(argv=None):
    p = argparse.ArgumentParser(description="DevBench -- Practical memory retrieval benchmark")
    p.add_argument("--top-k", type=int, nargs="+", default=[5])
    p.add_argument("--ablation", action="store_true")
    p.add_argument("--markdown", action="store_true")
    p.add_argument("--output-dir", type=str, default=None)
    return p.parse_args(argv)

def main():
    args = parse_args()
    result = asyncio.run(run_benchmark(args))
    results = result if isinstance(result, list) else [result]
    for r in results:
        print(format_results_markdown(r) if args.markdown else format_results_table(r))

if __name__ == "__main__":
    main()
```

- [ ] **Step 4: Run tests to verify they pass**

Run: `.venv/bin/pytest tests/benchmarks/test_benchmark_devbench.py -v`
Expected: All 6 tests PASS

- [ ] **Step 5: Run the benchmark**

Run: `.venv/bin/python scripts/benchmarks/benchmark_devbench.py --top-k 3 5 10`
Expected: Results table with exact, semantic, cross-type, negative categories

- [ ] **Step 6: Run full test suite**

Run: `.venv/bin/pytest tests/benchmarks/ -v`
Expected: All tests PASS (locomo + devbench)

- [ ] **Step 7: Commit**

```bash
git add scripts/benchmarks/benchmark_devbench.py scripts/benchmarks/devbench_dataset.json tests/benchmarks/test_benchmark_devbench.py
git commit -m "feat(benchmark): add DevBench practical benchmark -- 30 memories, 20 queries, 4 categories"
```

---

### Task Summary

| Task | Description | Files | Tests |
|------|-------------|-------|-------|
| 1 | Dataset (30 memories + 20 queries) | `devbench_dataset.json` | -- |
| 2 | Benchmark script + tests | `benchmark_devbench.py`, `test_benchmark_devbench.py` | 6 |
