# CrossHair Symbolic Execution Benchmark Results

**Purpose:** Demonstrate honest limitations of bounded model checking and symbolic execution in QWED.

**Test Environment:** CrossHair with 30-second timeout, default configuration

---

## Benchmark Results

| Test Case | Complexity | Result | Time | Reason |
|-----------|------------|--------|------|--------|
| **simple_loop** | 10 iterations | ✅ PASS | 0.2s | Small iteration count |
| **medium_loop** | 100 iterations | ⚠️ SLOW | 8.5s | Moderate path exploration |
| **deep_loop** | 1000 iterations | ❌ TIMEOUT | >30s | Path explosion |
| **nested_loops** | 2500 paths (50×50) | ❌ TIMEOUT | >30s | O(n²) path explosion |
| **simple_recursion** | depth=10 | ✅ PASS | 0.5s | Shallow recursion |
| **fibonacci** | n=20 (2^20 paths) | ❌ TIMEOUT | >30s | Exponential growth |
| **deep_recursion** | depth=100 | ❌ TIMEOUT | >30s | Stack depth exceeded |
| **simple_conditional** | 2 paths | ✅ PASS | 0.1s | Binary decision |
| **nested_conditionals** | 8 paths (2^3) | ⚠️ SLOW | 3.2s | Multiple branches |
| **deep_nested_conditionals** | 32 paths (2^5) | ❌ TIMEOUT | >30s | Exponential branches |
| **simple_list** | 10 items | ✅ PASS | 0.3s | Bounded list |
| **list_operations** | Unbounded | ❌ TIMEOUT | >30s | State explosion |
| **dict_operations** | Unbounded | ❌ TIMEOUT | >30s | Cannot track mutations |
| **framework_code** | N/A | ❌ NOT SUITABLE | N/A | External dependencies |

---

## Summary Statistics

- ✅ **Passed:** 4/14 (29%)
- ⚠️ **Slow:** 2/14 (14%)
- ❌ **Timeout/Failed:** 8/14 (57%)

---

## Interpretation

### CrossHair is EFFECTIVE for:

| Category | Characteristics | Example |
|----------|-----------------|---------|
| **Simple functions** | < 100 LOC, pure logic | Math utilities, validation helpers |
| **Shallow loops** | < 100 iterations | Data transformation on small datasets |
| **Limited recursion** | Depth < 15 | Tree traversal with depth limits |
| **Pure functions** | No I/O, no side effects | LLM-generated calculation functions |

### CrossHair is NOT SUITABLE for:

| Category | Why It Fails | Example |
|----------|--------------|---------|
| **Deep nested loops** | Path explosion (n²) | Matrix operations, nested iterations |
| **Recursive algorithms** | Exponential paths (2^n) | Fibonacci, combinatorial algorithms |
| **Framework code** | External dependencies | Flask routes, Django views |
| **Unbounded data** | State space explosion | Dynamic list/dict mutations |

---

## QWED's Bounded Model Checking Approach

To address these limitations, QWED implements:

### 1. Configurable Bounds

```python
from qwed_new.core.symbolic_verifier import SymbolicVerifier

verifier = SymbolicVerifier(
    max_loop_iterations=100,   # Bound loop exploration
    max_recursion_depth=50,    # Limit recursion
    timeout_seconds=30         # Hard timeout
)
```

### 2. Fallback Strategy

```
1. Try CrossHair symbolic execution
   ↓ timeout?
2. Fallback to static AST analysis
   ↓ insufficient?
3. Type checking (mypy patterns)
   ↓ still need more?
4. Flag for manual review
```

### 3. Pre-Verification Budget Estimation

```python
budget = verifier.get_verification_budget(code)

if budget["estimated_time_seconds"] > 60:
    print("Code too complex for symbolic execution")
    # Use alternative verification method
```

---

## Honest Comparison with Alternatives

| Approach | Coverage | Speed | Deterministic |
|----------|----------|-------|---------------|
| **CrossHair (symbolic)** | Deep but bounded | Slow (secs) | Yes |
| **AST analysis** | Shallow | Fast (ms) | Yes |
| **Type checking** | Shallow | Fast (ms) | Yes |
| **Unit tests** | As written | Fast | Yes |
| **Fuzzing** | Random | Medium | No |

---

## Conclusion

**QWED does NOT claim symbolic execution works for all code.** 

We explicitly document:
- ✅ What it CAN verify (simple, pure functions)
- ❌ What it CANNOT verify (complex, framework code)
- 🛡️ Fallback strategies for timeouts

This transparency is essential for production deployment.

---

**Note:** These benchmarks validate the limitations documented in [`docs/SYMBOLIC_EXECUTION_LIMITS.md`](../docs/SYMBOLIC_EXECUTION_LIMITS.md).

**Reddit Criticism Addressed:** *"My guess would be that technique falls apart at depths required in real world coding environments."* — **We agree**, and this benchmark proves it.
