# Assessment Pipeline

Detailed specification of the assessment pipeline for complexity calculation and pattern matching.

## Table of Contents

1. [Pipeline Overview](#pipeline-overview)
2. [Phase 1: Probe](#phase-1-probe)
3. [Phase 2: Pattern Matching](#phase-2-pattern-matching)
4. [Phase 3: Complexity Calculation](#phase-3-complexity-calculation)
5. [Phase 4: Decision](#phase-4-decision)
6. [Algorithm Details](#algorithm-details)

## Pipeline Overview

The assessment pipeline transforms a natural language directive into a structured assessment with a splitting decision.

```
Directive (text)
    │
    ▼
┌─────────────────┐
│  1. Probe       │ → Codebase context, relevant files
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  2. Pattern     │ → Pattern name, confidence score
│     Matching    │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  3. Complexity  │ → Systems count, modifiers, score
│     Calculation │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  4. Decision    │ → Execute | Cleave
└────────┬────────┘
         │
         ▼
Assessment Result
```

## Phase 1: Probe

**Purpose**: Scan codebase to generate context-aware questions.

**Input**: Directive text

**Output**:
- Detected stack (language, framework, database, etc.)
- Relevant files (matching directive keywords)
- Context-aware questions (conditionally triggered)

### Algorithm

```python
def probe(directive: str, codebase_path: str) -> ProbeResult:
    """
    Probe codebase for context.

    Args:
        directive: User directive text
        codebase_path: Path to codebase root

    Returns:
        ProbeResult with stack detection, files, questions
    """
    # 1. Extract keywords from directive
    keywords = extract_keywords(directive)

    # 2. Detect stack
    stack = detect_stack(codebase_path)

    # 3. Find relevant files
    files = find_files_by_keywords(codebase_path, keywords)

    # 4. Match pattern (for question generation)
    pattern_match = match_pattern(directive)

    # 5. Generate questions based on pattern + codebase state
    questions = generate_questions(
        pattern=pattern_match.pattern,
        stack=stack,
        files=files,
        directive=directive
    )

    return ProbeResult(
        stack=stack,
        files=files,
        pattern=pattern_match.pattern,
        confidence=pattern_match.confidence,
        questions=questions
    )
```

### Stack Detection

Detect language, framework, database from codebase:

```python
def detect_stack(codebase_path: str) -> Stack:
    """Detect technology stack from codebase."""
    stack = Stack()

    # Language detection (by file extensions)
    if any_files_exist(codebase_path, "*.py"):
        stack.language = "python"
    elif any_files_exist(codebase_path, "*.ts", "*.tsx"):
        stack.language = "typescript"
    # ... more languages

    # Framework detection (by config/imports)
    if file_exists(codebase_path, "manage.py"):
        stack.framework = "django"
    elif file_contains(codebase_path, "pyproject.toml", "fastapi"):
        stack.framework = "fastapi"
    # ... more frameworks

    # Database detection
    if file_contains(codebase_path, "pyproject.toml", "psycopg"):
        stack.database = "postgres"
    # ... more databases

    return stack
```

### File Discovery

Find files relevant to directive keywords:

```python
def find_files_by_keywords(
    codebase_path: str,
    keywords: List[str]
) -> List[str]:
    """Find files matching directive keywords."""
    patterns = {
        "auth": ["**/auth*.py", "**/login*.py", "**/security*.py"],
        "payment": ["**/payment*.py", "**/stripe*.py", "**/billing*.py"],
        # ... more patterns
    }

    files = []
    for keyword in keywords:
        if keyword in patterns:
            for pattern in patterns[keyword]:
                files.extend(glob(codebase_path, pattern))

    return deduplicate(files)
```

### Question Generation

Generate questions based on pattern + codebase state:

```python
def generate_questions(
    pattern: str,
    stack: Stack,
    files: List[str],
    directive: str
) -> List[Question]:
    """Generate context-aware questions."""
    pattern_def = PATTERNS[pattern]
    questions = []

    for question_def in pattern_def.questions:
        # Check trigger condition
        if question_def.trigger == "file_exists":
            if any_match(files, question_def.glob):
                questions.append(question_def)
        elif question_def.trigger == "file_missing":
            if not any_match(files, question_def.glob):
                questions.append(question_def)
        elif question_def.trigger == "keyword_detected":
            if question_def.keyword in directive.lower():
                questions.append(question_def)
        elif question_def.trigger == "always":
            questions.append(question_def)

    return questions
```

## Phase 2: Pattern Matching

**Purpose**: Match directive against known patterns for fast-path assessment.

**Input**: Directive text

**Output**: Pattern name, confidence score

### Algorithm

```python
def match_pattern(directive: str) -> PatternMatch:
    """
    Match directive against known patterns.

    Args:
        directive: User directive text

    Returns:
        PatternMatch with pattern name and confidence
    """
    directive_lower = directive.lower()
    tokens = tokenize(directive_lower)

    best_match = None
    best_confidence = 0.0

    for pattern in PATTERNS:
        # Count keyword matches
        keyword_matches = count_matches(tokens, pattern.keywords)
        total_keywords = len(pattern.keywords)

        # Calculate confidence
        confidence = keyword_matches / total_keywords

        # Apply confidence threshold
        if confidence >= pattern.confidence_threshold:
            if confidence > best_confidence:
                best_match = pattern
                best_confidence = confidence

    if best_match:
        return PatternMatch(
            pattern=best_match.name,
            confidence=best_confidence,
            systems_base=best_match.systems_base,
            modifiers=best_match.modifiers
        )
    else:
        return PatternMatch(
            pattern="Unknown",
            confidence=0.0,
            systems_base=1,
            modifiers={}
        )
```

### Pattern Definitions

Patterns are defined in `src/cleave/core/assessment.py`:

```python
PATTERNS = [
    {
        "name": "Full-Stack CRUD",
        "keywords": ["crud", "create", "read", "update", "delete", "form", "table", "list"],
        "systems_base": 3,  # UI, API, DB
        "modifiers": {
            "state_coordination": False,
            "error_handling": False,
        },
        "confidence_threshold": 0.70,
    },
    {
        "name": "Authentication System",
        "keywords": ["auth", "authentication", "login", "jwt", "token", "session", "oauth"],
        "systems_base": 3,  # Auth Service, API, DB
        "modifiers": {
            "security": True,
            "state_coordination": True,
        },
        "confidence_threshold": 0.80,
    },
    # ... more patterns
]
```

### Keyword Matching

```python
def count_matches(tokens: List[str], keywords: List[str]) -> int:
    """Count how many keywords appear in tokens."""
    matches = 0
    for keyword in keywords:
        if keyword in tokens:
            matches += 1
    return matches
```

## Phase 3: Complexity Calculation

**Purpose**: Calculate complexity score from systems count and modifiers.

**Input**: Directive text, pattern match (optional)

**Output**: Systems count, modifiers dict, complexity score

### Formula

```
complexity = (1 + systems) × (1 + 0.5 × modifiers)
```

- **Systems**: Number of distinct architectural boundaries (UI, API, DB, external services)
- **Modifiers**: Complexity factors (state, security, migration, etc.)

### Algorithm

```python
def calculate_complexity(directive: str, pattern_match: PatternMatch = None) -> Assessment:
    """
    Calculate complexity score.

    Args:
        directive: User directive text
        pattern_match: Optional pattern match for baseline

    Returns:
        Assessment with systems, modifiers, complexity
    """
    # Start with pattern baseline (if matched)
    if pattern_match:
        systems = pattern_match.systems_base
        modifiers_dict = pattern_match.modifiers.copy()
    else:
        systems = 1
        modifiers_dict = {}

    # Detect additional systems from directive
    systems += count_systems(directive)

    # Detect modifiers from directive
    modifiers_dict.update(detect_modifiers(directive))

    # Count active modifiers
    modifiers_count = sum(1 for v in modifiers_dict.values() if v)

    # Calculate complexity
    complexity = (1 + systems) * (1 + 0.5 * modifiers_count)

    return Assessment(
        systems=systems,
        modifiers=modifiers_dict,
        modifiers_count=modifiers_count,
        complexity=complexity
    )
```

### System Detection

```python
def count_systems(directive: str) -> int:
    """Count additional systems mentioned in directive."""
    directive_lower = directive.lower()
    systems = 0

    # UI indicators
    if any(word in directive_lower for word in ["ui", "frontend", "react", "vue", "page", "form"]):
        systems += 1

    # API indicators
    if any(word in directive_lower for word in ["api", "endpoint", "route", "handler"]):
        systems += 1

    # Database indicators
    if any(word in directive_lower for word in ["database", "db", "sql", "schema", "migration"]):
        systems += 1

    # External service indicators
    if any(word in directive_lower for word in ["stripe", "aws", "twilio", "sendgrid", "external"]):
        systems += 1

    # WebSocket/real-time indicators
    if any(word in directive_lower for word in ["websocket", "realtime", "live", "streaming"]):
        systems += 1

    return systems
```

### Modifier Detection

```python
def detect_modifiers(directive: str) -> Dict[str, bool]:
    """Detect complexity modifiers in directive."""
    directive_lower = directive.lower()
    modifiers = {}

    # State coordination
    if any(word in directive_lower for word in ["sync", "coordinate", "consistency", "transaction"]):
        modifiers["state_coordination"] = True

    # Error handling
    if any(word in directive_lower for word in ["error", "retry", "fallback", "recovery"]):
        modifiers["error_handling"] = True

    # Concurrency
    if any(word in directive_lower for word in ["concurrent", "parallel", "async", "race"]):
        modifiers["concurrency"] = True

    # Security
    if any(word in directive_lower for word in ["security", "auth", "encrypt", "token", "permission"]):
        modifiers["security"] = True

    # Breaking changes
    if any(word in directive_lower for word in ["breaking", "backwards", "compatibility", "migration"]):
        modifiers["breaking_changes"] = True

    # Data migration
    if any(word in directive_lower for word in ["migration", "backfill", "schema"]):
        modifiers["data_migration"] = True

    # External API
    if any(word in directive_lower for word in ["api", "integration", "webhook", "third-party"]):
        modifiers["external_api"] = True

    # Performance SLA
    if any(word in directive_lower for word in ["performance", "latency", "throughput", "scale"]):
        modifiers["performance_sla"] = True

    return modifiers
```

## Phase 4: Decision

**Purpose**: Decide whether to execute directly or cleave.

**Input**: Complexity score, threshold

**Output**: Decision (execute | cleave)

### Algorithm

```python
def decide(complexity: float, threshold: float = 5.0) -> str:
    """
    Decide whether to execute or cleave.

    Args:
        complexity: Complexity score
        threshold: Cleaving threshold (default: 5.0)

    Returns:
        "execute" if complexity <= threshold, else "cleave"
    """
    if complexity <= threshold:
        return "execute"
    else:
        return "cleave"
```

### Effective Complexity

Account for pattern confidence:

```python
def effective_complexity(
    complexity: float,
    confidence: float
) -> float:
    """
    Calculate effective complexity with confidence weighting.

    Args:
        complexity: Raw complexity score
        confidence: Pattern match confidence (0.0-1.0)

    Returns:
        Effective complexity (weighted by confidence)
    """
    if confidence >= 0.80:
        # High confidence: trust the assessment
        return complexity
    elif confidence >= 0.60:
        # Medium confidence: slight increase for uncertainty
        return complexity * 1.1
    else:
        # Low confidence: significant increase for uncertainty
        return complexity * 1.25
```

## Algorithm Details

### Complexity Examples

**Example 1: Simple CRUD**

Directive: "Add user profile page with name and bio fields"

```
Systems: 3 (UI, API, DB)
Modifiers: 0 (straightforward)
Complexity: (1 + 3) × (1 + 0.5 × 0) = 4.0
Decision: Execute (4.0 <= 5.0)
```

**Example 2: JWT Authentication**

Directive: "Add JWT authentication with refresh tokens and token blacklist"

```
Systems: 3 (Auth Service, API, DB)
Modifiers: 2 (security, state_coordination)
Complexity: (1 + 3) × (1 + 0.5 × 2) = 8.0
Decision: Cleave (8.0 > 5.0)
```

**Example 3: WebSocket Collaboration**

Directive: "Add real-time collaboration with WebSocket, conflict resolution, offline sync"

```
Systems: 4 (UI, WebSocket, Conflict Resolution, Sync Service)
Modifiers: 3 (state_coordination, concurrency, error_handling)
Complexity: (1 + 4) × (1 + 0.5 × 3) = 12.5
Decision: Cleave (12.5 > 5.0)
```

### Pattern Confidence Impact

**Example: Medium Confidence Match**

Directive: "Add payment processing"

```
Pattern: External Integration (confidence 0.65)
Systems: 3 (baseline from pattern)
Modifiers: 2 (baseline from pattern)
Complexity: (1 + 3) × (1 + 0.5 × 2) = 8.0

Effective Complexity: 8.0 × 1.1 = 8.8 (medium confidence penalty)
Decision: Cleave (8.8 > 5.0)
```

### Threshold Tuning

Adjust threshold per project:

```python
# Conservative (cleave more often)
threshold = 4.0

# Balanced (default)
threshold = 5.0

# Aggressive (execute more often)
threshold = 6.0
```

Configure via settings:

```yaml
# ~/.cleave/settings.yaml
execution:
  threshold: 5.0
```

## Calibration

### Recording Predictions

When `cleave init` runs, record the assessment:

```yaml
# .cleave/metrics.yaml
assessment:
  predicted_complexity: 8.0
  predicted_effort: "medium"
  pattern: "Authentication System"
  confidence: 0.85
```

### Capturing Outcomes

After completion, infer actual effort from task result:

```yaml
outcome:
  actual_effort: "high"  # Inferred from result summary
  accuracy: 0.67  # Predicted medium, actual high
```

### Calculating Accuracy

```python
def calculate_accuracy(predicted_effort: str, actual_effort: str) -> float:
    """Calculate assessment accuracy."""
    effort_ranges = {
        "low": (0.0, 2.5),
        "medium": (2.0, 5.0),
        "high": (4.0, 10.0),
    }

    predicted_range = effort_ranges[predicted_effort]
    actual_range = effort_ranges[actual_effort]

    # Calculate overlap
    overlap = range_overlap(predicted_range, actual_range)
    predicted_size = predicted_range[1] - predicted_range[0]

    return overlap / predicted_size
```

### Calibration Recommendations

```python
def recommend_calibration(metrics: List[Assessment]) -> List[str]:
    """Generate calibration recommendations."""
    recommendations = []

    # Analyze pattern accuracy
    pattern_accuracy = {}
    for metric in metrics:
        pattern = metric.pattern
        if pattern not in pattern_accuracy:
            pattern_accuracy[pattern] = []
        pattern_accuracy[pattern].append(metric.accuracy)

    # Recommend adjustments
    for pattern, accuracies in pattern_accuracy.items():
        avg_accuracy = sum(accuracies) / len(accuracies)
        if avg_accuracy < 0.70:
            recommendations.append(
                f"Pattern '{pattern}' has low accuracy ({avg_accuracy:.2f}). "
                f"Review systems_base or modifiers."
            )

    return recommendations
```

## Extension Points

### Custom Patterns

Add patterns to `PATTERNS` list:

```python
PATTERNS.append({
    "name": "Custom Pattern",
    "keywords": ["custom", "specific", "keywords"],
    "systems_base": 3,
    "modifiers": {"modifier_name": True},
    "confidence_threshold": 0.75,
})
```

### Custom Modifiers

Add modifier detection logic:

```python
def detect_custom_modifiers(directive: str) -> Dict[str, bool]:
    """Detect custom project-specific modifiers."""
    modifiers = {}

    if "custom_requirement" in directive.lower():
        modifiers["custom_modifier"] = True

    return modifiers
```

### Custom Complexity Formula

Override the formula for project-specific needs:

```python
def calculate_complexity_custom(systems: int, modifiers: Dict) -> float:
    """Custom complexity calculation."""
    # Example: Weight security higher
    security_weight = 1.0 if modifiers.get("security") else 0.0
    other_weight = sum(0.5 for k, v in modifiers.items() if v and k != "security")

    return (1 + systems) * (1 + security_weight + other_weight)
```

## Next Steps

- **Overview**: [System architecture](overview.md)
- **Backend Interface**: [Backend specification](backend-interface.md)
- **Contributing**: [Adding patterns](../guides/contributing.md)
