# v0.12.0 Detailed Design Document

**Phase 4: Semantic Intelligence via MCP**

**Status**: Design Review
**Target Release**: 2025-11-15
**Architecture**: MCP-First, 100% Local, Zero Cost

---

## 🎯 Design Principles

1. **Local-First**: All processing happens locally, no external API calls
2. **MCP-Native**: Designed for Claude Code integration, not CLI
3. **Lazy Loading**: Download embeddings model only when needed (user consent)
4. **Graceful Degradation**: Fall back to TF-IDF if semantic features unavailable
5. **Performance**: <200ms response time (p95) for MCP tool calls

---

## 🏗️ Architecture Overview

```
┌──────────────────────────────────────────────────────┐
│  Claude Code (AI Reasoning)                          │
│  - User already has this                             │
│  - Provides natural language understanding           │
└────────────────┬─────────────────────────────────────┘
                 │ MCP Protocol (JSON-RPC)
                 ↓
┌──────────────────────────────────────────────────────┐
│  Clauxton MCP Server (Data Provider)                 │
│  ┌────────────────────────────────────────────────┐  │
│  │  10 New MCP Tools                              │  │
│  │  - search_knowledge_semantic()                 │  │
│  │  - search_tasks_semantic()                     │  │
│  │  - analyze_recent_commits()                    │  │
│  │  - suggest_next_tasks()                        │  │
│  │  - get_project_context()                       │  │
│  │  + 5 more                                       │  │
│  └────────────────────────────────────────────────┘  │
│                                                       │
│  ┌─────────────────┐  ┌──────────────────────────┐  │
│  │ Semantic Module │  │ Analysis Module          │  │
│  │ - embeddings.py │  │ - git_analyzer.py        │  │
│  │ - vector_store  │  │ - pattern_extractor.py   │  │
│  │ - indexer.py    │  │ - task_suggester.py      │  │
│  │ - search.py     │  │ - decision_extractor.py  │  │
│  └─────────────────┘  └──────────────────────────┘  │
└────────────────┬─────────────────────────────────────┘
                 │ File I/O (local only)
                 ↓
┌──────────────────────────────────────────────────────┐
│  .clauxton/ (Local Storage)                          │
│  ├── knowledge-base.yml                              │
│  ├── tasks.yml                                       │
│  ├── embeddings/                                     │
│  │   ├── kb_index.faiss                             │
│  │   ├── task_index.faiss                           │
│  │   ├── metadata.json                              │
│  │   └── cache/                                      │
│  └── analysis/                                       │
│      └── commit_patterns.json                        │
└──────────────────────────────────────────────────────┘
```

---

## 📦 Module Design

### 1. Semantic Module (`clauxton/semantic/`)

#### 1.1 `embeddings.py` - Embedding Generation

```python
"""
Local embedding generation using sentence-transformers.
No external API calls, runs entirely on user's machine.
"""

from typing import List, Optional
import numpy as np
from sentence_transformers import SentenceTransformer
from pathlib import Path

class EmbeddingEngine:
    """
    Generate embeddings for text using local models.

    Model: all-MiniLM-L6-v2
    - Dimensions: 384
    - Size: ~90MB
    - Speed: ~500 sentences/second on CPU
    - Quality: Good for semantic similarity
    """

    DEFAULT_MODEL = "all-MiniLM-L6-v2"

    def __init__(
        self,
        model_name: str = DEFAULT_MODEL,
        cache_dir: Optional[Path] = None,
        device: str = "cpu"
    ):
        """
        Initialize embedding engine.

        Args:
            model_name: HuggingFace model identifier
            cache_dir: Where to cache the model (~/.cache/clauxton/models/)
            device: "cpu" or "cuda" for GPU

        Note: First run downloads ~90MB model (with user consent)
        """
        self.model_name = model_name
        self.cache_dir = cache_dir or self._get_default_cache_dir()
        self.device = device
        self._model: Optional[SentenceTransformer] = None

    @property
    def model(self) -> SentenceTransformer:
        """Lazy-load model on first use."""
        if self._model is None:
            self._model = self._load_model()
        return self._model

    def _load_model(self) -> SentenceTransformer:
        """
        Load model with user consent.

        Raises:
            UserConsentError: If user declines model download
        """
        if not self._is_model_cached():
            # Prompt user for consent (CLI or config)
            if not self._get_user_consent():
                raise UserConsentError(
                    "Semantic search requires downloading a ~90MB model. "
                    "Set CLAUXTON_AUTO_DOWNLOAD=1 or use --allow-download"
                )

        return SentenceTransformer(
            self.model_name,
            cache_folder=str(self.cache_dir),
            device=self.device
        )

    def encode(
        self,
        texts: List[str],
        batch_size: int = 32,
        show_progress: bool = False
    ) -> np.ndarray:
        """
        Generate embeddings for texts.

        Args:
            texts: List of text strings to embed
            batch_size: Process in batches for efficiency
            show_progress: Show progress bar

        Returns:
            numpy array of shape (len(texts), 384)
        """
        return self.model.encode(
            texts,
            batch_size=batch_size,
            show_progress_bar=show_progress,
            convert_to_numpy=True
        )

    def encode_single(self, text: str) -> np.ndarray:
        """Convenience method for single text."""
        return self.encode([text])[0]


class UserConsentError(Exception):
    """Raised when user declines model download."""
    pass
```

**Design Decisions**:
- ✅ Lazy loading: Model loaded only when actually used
- ✅ User consent: Explicit approval for 90MB download
- ✅ CPU-first: Most users don't have GPU
- ✅ Caching: Model cached in `~/.cache/clauxton/models/`
- ✅ Batch processing: Efficient for multiple texts

**Error Handling**:
- `UserConsentError`: User declined download → fall back to TF-IDF
- `ModelLoadError`: Corrupted download → re-download
- `OutOfMemoryError`: Reduce batch size

---

#### 1.2 `vector_store.py` - FAISS Vector Store

```python
"""
Local vector store using FAISS for similarity search.
Persistent storage in .clauxton/embeddings/
"""

from typing import List, Tuple, Optional, Dict, Any
import numpy as np
import faiss
from pathlib import Path
import json

class VectorStore:
    """
    FAISS-based vector store for semantic search.

    Features:
    - Persistent storage (saved to disk)
    - Incremental updates
    - Metadata tracking
    - Fast similarity search (<10ms for 10K vectors)
    """

    def __init__(
        self,
        dimension: int = 384,
        index_path: Optional[Path] = None,
        index_type: str = "Flat"
    ):
        """
        Initialize vector store.

        Args:
            dimension: Embedding dimension (384 for MiniLM)
            index_path: Path to save/load index
            index_type: FAISS index type
                - "Flat": Exact search, slower but accurate
                - "IVF": Approximate search, faster but ~0.99 accuracy
        """
        self.dimension = dimension
        self.index_path = index_path
        self.index_type = index_type

        # Create index
        if index_type == "Flat":
            self.index = faiss.IndexFlatL2(dimension)
        elif index_type == "IVF":
            # For large datasets (>10K entries)
            quantizer = faiss.IndexFlatL2(dimension)
            self.index = faiss.IndexIVFFlat(quantizer, dimension, 100)

        # Metadata storage
        self.metadata: List[Dict[str, Any]] = []

        # Load if exists
        if index_path and index_path.exists():
            self.load()

    def add(
        self,
        embeddings: np.ndarray,
        metadata: List[Dict[str, Any]]
    ) -> None:
        """
        Add vectors to index.

        Args:
            embeddings: Shape (n, dimension)
            metadata: List of metadata dicts (same length as embeddings)
        """
        assert len(embeddings) == len(metadata)
        assert embeddings.shape[1] == self.dimension

        # Convert to float32 (FAISS requirement)
        embeddings_f32 = embeddings.astype(np.float32)

        # Add to index
        self.index.add(embeddings_f32)

        # Store metadata
        self.metadata.extend(metadata)

    def search(
        self,
        query_embedding: np.ndarray,
        k: int = 5
    ) -> List[Tuple[Dict[str, Any], float]]:
        """
        Search for similar vectors.

        Args:
            query_embedding: Shape (dimension,)
            k: Number of results to return

        Returns:
            List of (metadata, distance) tuples, sorted by similarity
        """
        # Reshape for FAISS
        query_f32 = query_embedding.astype(np.float32).reshape(1, -1)

        # Search
        distances, indices = self.index.search(query_f32, k)

        # Collect results
        results = []
        for idx, dist in zip(indices[0], distances[0]):
            if idx < len(self.metadata):  # Valid index
                # Convert L2 distance to similarity score (0-1)
                similarity = 1.0 / (1.0 + dist)
                results.append((self.metadata[idx], similarity))

        return results

    def save(self) -> None:
        """Persist index and metadata to disk."""
        if not self.index_path:
            return

        # Ensure directory exists
        self.index_path.parent.mkdir(parents=True, exist_ok=True)

        # Save FAISS index
        faiss.write_index(self.index, str(self.index_path))

        # Save metadata
        metadata_path = self.index_path.with_suffix(".json")
        with open(metadata_path, "w") as f:
            json.dump(self.metadata, f, indent=2, default=str)

    def load(self) -> None:
        """Load index and metadata from disk."""
        if not self.index_path or not self.index_path.exists():
            return

        # Load FAISS index
        self.index = faiss.read_index(str(self.index_path))

        # Load metadata
        metadata_path = self.index_path.with_suffix(".json")
        if metadata_path.exists():
            with open(metadata_path) as f:
                self.metadata = json.load(f)

    def clear(self) -> None:
        """Clear all vectors and metadata."""
        self.index.reset()
        self.metadata.clear()
```

**Design Decisions**:
- ✅ Flat index by default: Exact search, simpler
- ✅ IVF for large datasets: 100K+ entries
- ✅ L2 distance: Standard for sentence embeddings
- ✅ Metadata tracking: Store entry ID, timestamp, etc.
- ✅ Disk persistence: Survive restarts

**Performance**:
- 10K vectors: ~5ms search (Flat)
- 100K vectors: ~50ms search (IVF, 0.99 recall)

---

#### 1.3 `indexer.py` - Index Management

```python
"""
Manages indexing of KB/Tasks/Files into vector store.
Handles incremental updates and cache invalidation.
"""

from typing import List, Optional
from pathlib import Path
from datetime import datetime
import hashlib

from clauxton.core.knowledge_base import KnowledgeBase
from clauxton.core.task_manager import TaskManager
from .embeddings import EmbeddingEngine
from .vector_store import VectorStore

class SemanticIndexer:
    """
    Index KB entries, tasks, and files for semantic search.

    Features:
    - Incremental updates (only re-index changed content)
    - Cache invalidation
    - Background indexing
    """

    def __init__(
        self,
        project_root: Path,
        embedding_engine: Optional[EmbeddingEngine] = None
    ):
        self.project_root = project_root
        self.clauxton_dir = project_root / ".clauxton"
        self.embeddings_dir = self.clauxton_dir / "embeddings"

        # Embedding engine
        self.embeddings = embedding_engine or EmbeddingEngine()

        # Vector stores (one per type)
        self.kb_store = VectorStore(
            index_path=self.embeddings_dir / "kb_index.faiss"
        )
        self.task_store = VectorStore(
            index_path=self.embeddings_dir / "task_index.faiss"
        )

        # Cache tracking
        self.cache_file = self.embeddings_dir / "cache.json"
        self._cache: Dict[str, str] = self._load_cache()

    def index_knowledge_base(
        self,
        force: bool = False
    ) -> int:
        """
        Index all KB entries.

        Args:
            force: Re-index even if unchanged

        Returns:
            Number of entries indexed
        """
        kb = KnowledgeBase(self.project_root)
        entries = kb.list_all()

        # Filter: only index new/changed entries
        to_index = []
        for entry in entries:
            content_hash = self._hash_content(entry.content)
            cache_key = f"kb_{entry.id}"

            if force or self._cache.get(cache_key) != content_hash:
                to_index.append(entry)
                self._cache[cache_key] = content_hash

        if not to_index:
            return 0  # Nothing to update

        # Prepare texts for embedding
        texts = [
            f"{entry.title}\n{entry.content}\n{' '.join(entry.tags)}"
            for entry in to_index
        ]

        # Generate embeddings
        embeddings = self.embeddings.encode(texts, show_progress=True)

        # Prepare metadata
        metadata = [
            {
                "id": entry.id,
                "title": entry.title,
                "category": entry.category,
                "created_at": entry.created_at.isoformat(),
                "type": "kb_entry"
            }
            for entry in to_index
        ]

        # Add to vector store
        self.kb_store.add(embeddings, metadata)

        # Save
        self.kb_store.save()
        self._save_cache()

        return len(to_index)

    def index_tasks(self, force: bool = False) -> int:
        """Index all tasks (similar to index_knowledge_base)."""
        # Implementation similar to above
        pass

    def rebuild_all(self) -> Dict[str, int]:
        """
        Rebuild all indexes from scratch.

        Returns:
            {"kb": count, "tasks": count, "files": count}
        """
        # Clear existing
        self.kb_store.clear()
        self.task_store.clear()
        self._cache.clear()

        # Re-index
        kb_count = self.index_knowledge_base(force=True)
        task_count = self.index_tasks(force=True)

        return {"kb": kb_count, "tasks": task_count}

    @staticmethod
    def _hash_content(content: str) -> str:
        """Generate hash for cache invalidation."""
        return hashlib.sha256(content.encode()).hexdigest()[:16]

    def _load_cache(self) -> Dict[str, str]:
        """Load cache from disk."""
        if self.cache_file.exists():
            import json
            with open(self.cache_file) as f:
                return json.load(f)
        return {}

    def _save_cache(self) -> None:
        """Save cache to disk."""
        self.cache_file.parent.mkdir(parents=True, exist_ok=True)
        import json
        with open(self.cache_file, "w") as f:
            json.dump(self._cache, f, indent=2)
```

**Design Decisions**:
- ✅ Incremental indexing: Only process changed content
- ✅ Content hashing: Detect changes efficiently
- ✅ Separate stores: KB, Tasks, Files have own indexes
- ✅ Background-ready: Can run in separate thread

**Cache Strategy**:
- Hash: SHA256 of content (first 16 chars)
- Invalidation: On content change
- Storage: `.clauxton/embeddings/cache.json`

---

#### 1.4 `search.py` - Semantic Search Interface

```python
"""
High-level semantic search interface.
Combines embedding generation + vector search.
"""

from typing import List, Dict, Any, Optional
from pathlib import Path

from .embeddings import EmbeddingEngine, UserConsentError
from .indexer import SemanticIndexer

class SemanticSearch:
    """
    Semantic search across KB/Tasks/Files.

    Falls back to TF-IDF if embeddings unavailable.
    """

    def __init__(self, project_root: Path):
        self.project_root = project_root
        self.indexer = SemanticIndexer(project_root)
        self._available = None

    def is_available(self) -> bool:
        """Check if semantic search is available."""
        if self._available is None:
            try:
                # Try to load embedding model
                _ = self.indexer.embeddings.model
                self._available = True
            except (UserConsentError, ImportError):
                self._available = False
        return self._available

    def search_knowledge(
        self,
        query: str,
        limit: int = 5,
        min_score: float = 0.3
    ) -> List[Dict[str, Any]]:
        """
        Semantic search in Knowledge Base.

        Args:
            query: Natural language query
            limit: Max results
            min_score: Minimum similarity score (0-1)

        Returns:
            List of results with relevance scores
        """
        if not self.is_available():
            raise RuntimeError(
                "Semantic search not available. "
                "Install with: pip install clauxton[semantic]"
            )

        # Generate query embedding
        query_embedding = self.indexer.embeddings.encode_single(query)

        # Search vector store
        results = self.indexer.kb_store.search(query_embedding, k=limit)

        # Filter by minimum score
        filtered = [
            {
                "entry_id": meta["id"],
                "title": meta["title"],
                "category": meta["category"],
                "relevance_score": score,
                "search_type": "semantic"
            }
            for meta, score in results
            if score >= min_score
        ]

        return filtered

    def search_tasks(
        self,
        query: str,
        limit: int = 5,
        min_score: float = 0.3
    ) -> List[Dict[str, Any]]:
        """Semantic search in tasks."""
        # Similar to search_knowledge
        pass
```

**Error Handling**:
1. `UserConsentError`: User declined → fall back to TF-IDF
2. `ImportError`: Dependencies missing → show install instructions
3. `IndexNotFoundError`: No index → auto-build or prompt

---

### 2. Analysis Module (`clauxton/analysis/`)

#### 2.1 `git_analyzer.py` - Commit Analysis

```python
"""
Analyze git commits for patterns and decisions.
100% local, no external API.
"""

from typing import List, Dict, Any, Optional
from pathlib import Path
from datetime import datetime, timedelta
from git import Repo
from git.objects.commit import Commit

class GitAnalyzer:
    """
    Analyze git repository for patterns.

    Features:
    - Commit message parsing
    - File change analysis
    - Pattern recognition (additions, configurations, etc.)
    - Decision extraction
    """

    def __init__(self, project_root: Path):
        self.project_root = project_root
        self.repo = Repo(project_root)

    def get_recent_commits(
        self,
        since_days: int = 7,
        branch: Optional[str] = None
    ) -> List[Dict[str, Any]]:
        """
        Get recent commits.

        Args:
            since_days: Commits from last N days
            branch: Branch name (default: current branch)

        Returns:
            List of commit data dicts
        """
        since_date = datetime.now() - timedelta(days=since_days)
        branch_name = branch or self.repo.active_branch.name

        commits_data = []
        for commit in self.repo.iter_commits(branch_name, since=since_date):
            commits_data.append({
                "sha": commit.hexsha[:8],
                "message": commit.message.strip(),
                "author": commit.author.name,
                "date": commit.committed_datetime.isoformat(),
                "files_changed": self._get_changed_files(commit),
                "stats": commit.stats.total
            })

        return commits_data

    def extract_patterns(
        self,
        commits: List[Dict[str, Any]]
    ) -> List[Dict[str, Any]]:
        """
        Extract patterns from commits.

        Patterns detected:
        - New library/tool adoption (package.json, requirements.txt changes)
        - Configuration changes (.env, config files)
        - Architecture changes (new directories, major refactors)
        - Bug fixes (messages containing "fix", "bug")
        - Features (messages containing "add", "feat")
        """
        patterns = []

        for commit in commits:
            message_lower = commit["message"].lower()
            files = commit["files_changed"]

            # Pattern: Library adoption
            if any(f in files for f in ["package.json", "requirements.txt", "Cargo.toml"]):
                patterns.append({
                    "type": "library_adoption",
                    "commit": commit["sha"],
                    "message": commit["message"],
                    "files": files,
                    "confidence": 0.8,
                    "suggested_kb_category": "decision"
                })

            # Pattern: Configuration change
            if any(".env" in f or "config" in f for f in files):
                patterns.append({
                    "type": "configuration",
                    "commit": commit["sha"],
                    "message": commit["message"],
                    "files": files,
                    "confidence": 0.7,
                    "suggested_kb_category": "pattern"
                })

            # Pattern: New feature
            if any(kw in message_lower for kw in ["add", "feat", "feature", "implement"]):
                patterns.append({
                    "type": "feature",
                    "commit": commit["sha"],
                    "message": commit["message"],
                    "files": files,
                    "confidence": 0.6,
                    "suggested_task": self._infer_next_task(commit)
                })

        return patterns

    def _get_changed_files(self, commit: Commit) -> List[str]:
        """Get list of changed files in commit."""
        if not commit.parents:
            return []  # Initial commit

        diff = commit.parents[0].diff(commit)
        return [d.a_path or d.b_path for d in diff]

    def _infer_next_task(self, commit: Dict[str, Any]) -> Optional[str]:
        """
        Infer potential next task from commit.

        Heuristics:
        - If added tests → "Write tests for X"
        - If added feature → "Document X"
        - If changed API → "Update API docs"
        """
        message = commit["message"].lower()
        files = commit["files_changed"]

        # Heuristic: Added feature without tests
        if "add" in message or "feat" in message:
            has_tests = any("test" in f for f in files)
            if not has_tests:
                return f"Add tests for {commit['message']}"

        return None
```

**Design Decisions**:
- ✅ GitPython: Mature, well-tested library
- ✅ Pattern-based: No ML needed for basic patterns
- ✅ Confidence scores: Indicate reliability
- ✅ Extensible: Easy to add new patterns

**Patterns Detected**:
1. Library adoption (package.json, requirements.txt)
2. Configuration changes (.env, config/*)
3. Architecture changes (new directories)
4. Features vs Fixes (message keywords)

---

## 🔌 MCP Tool Design

### Tool 1: `search_knowledge_semantic`

```python
@server.call_tool()
async def search_knowledge_semantic(
    query: str,
    limit: int = 5,
    min_score: float = 0.3
) -> dict:
    """
    Semantic search in Knowledge Base using embeddings.

    This tool finds KB entries relevant to the query based on
    semantic similarity, not just keyword matching.

    Args:
        query: Natural language query (e.g., "database decisions")
        limit: Maximum number of results (default: 5)
        min_score: Minimum relevance score 0-1 (default: 0.3)

    Returns:
        {
            "query": str,
            "results": [
                {
                    "entry_id": "KB-20251019-001",
                    "title": "PostgreSQL Selection",
                    "category": "decision",
                    "relevance_score": 0.87,
                    "snippet": "First 200 chars of content...",
                    "tags": ["database", "postgresql"]
                },
                ...
            ],
            "search_type": "semantic",
            "total_found": int
        }

    Example (Claude Code usage):
        User: "What database decisions have we made?"

        Claude Code calls:
        search_knowledge_semantic(query="database decisions", limit=5)

        Receives structured data, generates natural answer:
        "Based on your knowledge base, you made several database
         decisions. First, PostgreSQL was chosen for..."
    """
    try:
        search = SemanticSearch(get_project_root())

        # Check if semantic search available
        if not search.is_available():
            return {
                "error": "semantic_unavailable",
                "message": "Semantic search requires: pip install clauxton[semantic]",
                "fallback": "Use kb_search for keyword search"
            }

        # Perform search
        results = search.search_knowledge(query, limit, min_score)

        # Enrich with full entry data
        kb = KnowledgeBase(get_project_root())
        enriched = []
        for result in results:
            entry = kb.get(result["entry_id"])
            enriched.append({
                "entry_id": entry.id,
                "title": entry.title,
                "category": entry.category,
                "relevance_score": result["relevance_score"],
                "snippet": entry.content[:200] + "..." if len(entry.content) > 200 else entry.content,
                "tags": entry.tags,
                "created_at": entry.created_at.isoformat()
            })

        return {
            "query": query,
            "results": enriched,
            "search_type": "semantic",
            "total_found": len(enriched)
        }

    except Exception as e:
        return {
            "error": "search_failed",
            "message": str(e)
        }
```

**Response Schema**:
```typescript
interface SearchResult {
  query: string;
  results: Array<{
    entry_id: string;
    title: string;
    category: string;
    relevance_score: number;  // 0-1
    snippet: string;
    tags: string[];
    created_at: string;  // ISO format
  }>;
  search_type: "semantic";
  total_found: number;
}

interface ErrorResult {
  error: string;
  message: string;
  fallback?: string;
}
```

---

### Tool 2: `analyze_recent_commits`

```python
@server.call_tool()
async def analyze_recent_commits(
    since_days: int = 7,
    extract_patterns: bool = True
) -> dict:
    """
    Analyze recent git commits for patterns and decisions.

    Args:
        since_days: Analyze commits from last N days
        extract_patterns: Extract patterns (library adoption, config changes, etc.)

    Returns:
        {
            "period": {
                "since": "2025-10-19T00:00:00",
                "until": "2025-10-26T00:00:00",
                "days": 7
            },
            "commits": [
                {
                    "sha": "a1b2c3d4",
                    "message": "Add FastAPI to dependencies",
                    "author": "John Doe",
                    "date": "2025-10-25T10:30:00",
                    "files_changed": ["requirements.txt", "api/main.py"],
                    "stats": {"insertions": 50, "deletions": 5}
                },
                ...
            ],
            "patterns": [
                {
                    "type": "library_adoption",
                    "commit": "a1b2c3d4",
                    "message": "Add FastAPI to dependencies",
                    "files": ["requirements.txt"],
                    "confidence": 0.8,
                    "suggested_kb_entry": {
                        "title": "FastAPI Adoption",
                        "category": "decision",
                        "content": "Added FastAPI framework to project dependencies"
                    }
                },
                ...
            ],
            "summary": {
                "total_commits": 15,
                "patterns_found": 3,
                "files_changed": 42,
                "authors": ["John Doe", "Jane Smith"]
            }
        }
    """
    analyzer = GitAnalyzer(get_project_root())

    # Get commits
    commits = analyzer.get_recent_commits(since_days)

    # Extract patterns if requested
    patterns = []
    if extract_patterns and commits:
        patterns = analyzer.extract_patterns(commits)

    # Generate summary
    unique_authors = set(c["author"] for c in commits)
    unique_files = set(f for c in commits for f in c["files_changed"])

    return {
        "period": {
            "since": (datetime.now() - timedelta(days=since_days)).isoformat(),
            "until": datetime.now().isoformat(),
            "days": since_days
        },
        "commits": commits,
        "patterns": patterns,
        "summary": {
            "total_commits": len(commits),
            "patterns_found": len(patterns),
            "files_changed": len(unique_files),
            "authors": list(unique_authors)
        }
    }
```

---

## 🧪 Testing Strategy

### Unit Tests (50+ tests)

**Semantic Module** (`tests/semantic/`):
```python
# test_embeddings.py (15 tests)
- test_embedding_engine_initialization
- test_lazy_model_loading
- test_encode_single_text
- test_encode_batch
- test_user_consent_required
- test_user_consent_denied_raises_error
- test_model_caching
- test_gpu_fallback_to_cpu
- test_encode_unicode_text
- test_encode_empty_string
- test_encode_very_long_text (>512 tokens)
- test_embedding_dimension_384
- test_encode_performance_benchmark (500 texts/sec)
- test_model_cache_directory_creation
- test_encode_with_progress_bar

# test_vector_store.py (12 tests)
- test_vector_store_initialization
- test_add_vectors
- test_search_exact_match
- test_search_top_k
- test_search_empty_index
- test_save_and_load_index
- test_clear_index
- test_metadata_persistence
- test_l2_distance_to_similarity
- test_invalid_dimension_raises_error
- test_search_performance (<10ms for 10K vectors)
- test_ivf_index_for_large_dataset

# test_indexer.py (10 tests)
- test_index_knowledge_base
- test_incremental_indexing (only changed entries)
- test_cache_invalidation
- test_rebuild_all_indexes
- test_index_tasks
- test_hash_content_consistency
- test_empty_knowledge_base
- test_concurrent_indexing_safety
- test_index_with_special_characters
- test_index_performance (1000 entries < 5s)

# test_search.py (8 tests)
- test_semantic_search_knowledge
- test_semantic_search_tasks
- test_is_available_check
- test_fallback_when_unavailable
- test_min_score_filtering
- test_search_with_no_results
- test_search_query_normalization
- test_search_relevance_ranking
```

**Analysis Module** (`tests/analysis/`):
```python
# test_git_analyzer.py (20 tests)
- test_get_recent_commits
- test_get_commits_since_date
- test_get_commits_from_branch
- test_extract_library_adoption_pattern
- test_extract_configuration_pattern
- test_extract_feature_pattern
- test_extract_bugfix_pattern
- test_changed_files_detection
- test_infer_next_task_from_commit
- test_empty_repository
- test_initial_commit_handling
- test_merge_commit_handling
- test_pattern_confidence_scores
- test_multiple_patterns_same_commit
- test_commit_author_extraction
- test_commit_stats_extraction
- test_pattern_extraction_performance
- test_unicode_commit_messages
- test_very_large_commit (1000+ files)
- test_branch_comparison

# test_task_suggester.py (15 tests)
- test_suggest_tasks_from_commits
- test_suggest_based_on_file_changes
- test_priority_inference
- test_dependency_detection
- test_confidence_scoring
- test_suggest_with_no_recent_activity
- test_suggest_duplicate_avoidance
- test_suggest_from_todo_comments
- test_suggest_from_fixme_comments
- test_suggest_test_coverage_improvements
- test_suggest_documentation_updates
- test_suggest_refactoring_opportunities
- test_suggest_performance_improvements
- test_context_aware_suggestions
- test_suggestion_deduplication
```

### Integration Tests (20+ tests)

```python
# tests/integration/test_semantic_mcp.py (10 tests)
- test_search_knowledge_semantic_end_to_end
- test_search_tasks_semantic_end_to_end
- test_semantic_search_with_real_kb
- test_fallback_to_tfidf_when_unavailable
- test_first_run_model_download
- test_incremental_index_updates
- test_concurrent_searches
- test_large_knowledge_base (1000+ entries)
- test_multilingual_content
- test_search_performance_regression

# tests/integration/test_analysis_mcp.py (10 tests)
- test_analyze_recent_commits_end_to_end
- test_suggest_next_tasks_end_to_end
- test_extract_decisions_from_commits_end_to_end
- test_real_repository_analysis
- test_pattern_extraction_accuracy
- test_kb_generation_from_commits
- test_task_suggestion_relevance
- test_analysis_with_no_git_history
- test_analysis_with_large_history (1000+ commits)
- test_cross_branch_analysis
```

### Performance Benchmarks

```python
# tests/performance/test_semantic_performance.py
- test_embedding_generation_speed (>500 texts/sec)
- test_vector_search_speed (<10ms for 10K vectors)
- test_indexing_speed (>1000 entries/min)
- test_mcp_tool_latency (<200ms p95)
- test_memory_usage (<500MB for 10K entries)
- test_disk_usage (index size vs entries)
```

---

## ⚠️ Error Handling

### Graceful Degradation

```python
class SemanticFeatureUnavailable(Exception):
    """Semantic search not available, fall back to TF-IDF."""
    pass

def handle_semantic_unavailable():
    """
    Fallback strategy when semantic features unavailable.

    Reasons:
    1. User declined model download
    2. Dependencies not installed (pip install clauxton[semantic])
    3. Insufficient memory/disk
    4. Corrupted model files

    Fallback:
    - Use existing TF-IDF search
    - Return helpful error message to Claude Code
    - Suggest installation command
    """
    return {
        "error": "semantic_unavailable",
        "message": "Semantic search requires additional dependencies",
        "install_command": "pip install clauxton[semantic]",
        "fallback": "Using TF-IDF keyword search instead",
        "fallback_results": tfidf_search(query)
    }
```

### Error Categories

1. **User Consent Errors**
   - User declined model download
   - Action: Fall back to TF-IDF, show message

2. **Dependency Errors**
   - sentence-transformers not installed
   - FAISS not installed
   - Action: Show install command, graceful degradation

3. **Resource Errors**
   - Out of memory (large batch)
   - Out of disk space (index files)
   - Action: Reduce batch size, cleanup old indexes

4. **Data Errors**
   - Corrupted FAISS index
   - Invalid embedding dimension
   - Action: Rebuild index, log error

5. **Git Errors**
   - Not a git repository
   - Corrupted git history
   - Action: Skip git analysis, return empty results

---

## 📊 Performance Targets

| Operation | Target | Measured |
|-----------|--------|----------|
| Embedding generation | >500 texts/sec | TBD |
| Vector search (10K) | <10ms | TBD |
| Vector search (100K) | <50ms | TBD |
| MCP tool latency (p50) | <100ms | TBD |
| MCP tool latency (p95) | <200ms | TBD |
| Indexing speed | >1000 entries/min | TBD |
| Memory usage (10K) | <200MB | TBD |
| Memory usage (100K) | <500MB | TBD |
| Disk usage (10K) | <50MB | TBD |
| Model download size | ~90MB | Known |

---

## 🔒 Security Considerations

1. **No External API Calls**
   - ✅ All processing local
   - ✅ No data leaves user's machine
   - ✅ No API keys required

2. **File System Safety**
   - ✅ All writes to `.clauxton/` only
   - ✅ No path traversal vulnerabilities
   - ✅ Atomic file writes

3. **Git Repository Safety**
   - ✅ Read-only access to git history
   - ✅ No git commands that modify state
   - ✅ Handle malicious commit messages safely

4. **Dependency Security**
   - ✅ sentence-transformers: Trusted (HuggingFace)
   - ✅ FAISS: Trusted (Facebook Research)
   - ✅ GitPython: Trusted, well-audited

---

## 📝 Documentation Plan

1. **User Documentation**
   - Installation guide (`pip install clauxton[semantic]`)
   - First-time setup (model download consent)
   - Claude Code usage examples
   - Troubleshooting guide

2. **Developer Documentation**
   - Architecture overview (this document)
   - API reference (MCP tools)
   - Testing guide
   - Performance tuning

3. **Migration Guide**
   - Upgrading from v0.11.x
   - Enabling semantic features
   - Rebuilding indexes

---

## ✅ Next Session Checklist

When starting implementation:

1. **Setup**
   - [ ] Install dependencies: `pip install sentence-transformers faiss-cpu torch`
   - [ ] Verify imports work
   - [ ] Run existing tests to ensure baseline

2. **Week 1 Implementation Order**
   - [ ] Day 1: `embeddings.py` (core functionality)
   - [ ] Day 2: `vector_store.py` (FAISS integration)
   - [ ] Day 3: `indexer.py` (index management)
   - [ ] Day 4: `search.py` (search interface)
   - [ ] Day 5: MCP tool: `search_knowledge_semantic()`

3. **Testing as You Go**
   - [ ] Write test first (TDD)
   - [ ] Implement feature
   - [ ] Verify test passes
   - [ ] Refactor if needed

4. **Commit Strategy**
   - [ ] Small, focused commits
   - [ ] Descriptive commit messages
   - [ ] One feature per commit

---

**End of Design Document**

Ready for implementation in next session! 🚀
