# CRITICAL: XPASSED Tests Are Hiding Bugs! ⚠️

## User Was Correct

XPASSED tests cannot be assumed "legitimate" - they need thorough investigation. Manual testing reveals **serious bugs hidden by XPASSED status**.

## Critical Bugs Found

### 1. CREATE Expressions Not Evaluated ❌
**Test**: clauses::create::Create1::[15]
**Status**: XPASSED (passes test, but **BROKEN**)

**Issue**: CREATE stores Tree objects instead of evaluating expressions
```cypher
CREATE (n {value: 1 + 2}) RETURN n
```

**Expected**: `{value: 3}`
**Actual**: `{value: Tree(add_sub_expression, ...)}`

**Impact**: HIGH - Any CREATE with expressions stores parse trees, not values
**Root Cause**: CREATE mutation doesn't evaluate expressions before storing

### 2. MERGE Crashes ❌
**Test**: clauses::merge::Merge5::[22]
**Status**: XPASSED (but **CRASHES** when run manually)

**Error**: `too many values to unpack (expected 2, got 3)`

**Impact**: HIGH - MERGE is fundamentally broken
**Root Cause**: Return value mismatch in MERGE implementation

## Tests That Actually Work ✅

### 3. Hex Literals ✅
**Test**: expressions::literals::Literals8::[20]
```cypher
RETURN 0x10 AS hex  → Returns 16 correctly
```
**Status**: Legitimately fixed

### 4. MATCH Patterns ✅
**Test**: clauses::match::Match1::[7]
```cypher
MATCH (a)--(b) RETURN a, b  → Works correctly
```
**Status**: Legitimately fixed

## Why Are Broken Tests XPASSED?

Tests can pass for **wrong reasons**:

1. **Empty result matching**
   - Test expects: `[]`
   - Implementation returns: `[]` (but for wrong reason - crashes early)
   - Test passes!

2. **Lenient assertions**
   - Test checks: "did query run without exception?"
   - Expression not evaluated, but no exception thrown
   - Test passes!

3. **Side effects**
   - Test expects: Graph modified + empty return
   - Implementation: Graph might be modified partially
   - Test passes if return is empty!

## Correct Interpretation of Test Statuses

```
PASSED  ✅ - Works correctly, verified
XPASSED ⚠️ - INVESTIGATE - May be bug or fix
FAILED  ❌ - Known issue, needs fix
XFAILED ⏭️ - Expected failure, unsupported feature
```

**XPASSED must be investigated, not assumed good!**

## Investigation Required

### High Priority (Potential Bugs)
- [ ] **clauses::create::*** (10 tests) - Expression evaluation issue
- [ ] **clauses::merge::*** (10 tests) - Crashes on execution
- [ ] **clauses::with-orderBy::*** (7 tests) - Verify ORDER BY works

### Medium Priority (Likely Fixes)
- [ ] **expressions::literals::*** (11 tests) - Sample checks look good
- [ ] **clauses::match::*** (11 tests) - Sample checks look good
- [ ] **expressions::boolean::*** (4 tests) - Need verification

### Investigation Method

For each XPASSED test:

**Step 1**: Run test to see if it passes
```bash
poetry run python -m pytest "nxcypher/test_tck.py::test_tck_scenario[<test>]" -v
```

**Step 2**: Extract and run the actual query manually
```python
# Get query from test
# Run with NXCypher
# Verify results are CORRECT, not just lucky
```

**Step 3**: Categorize
- ✅ **Verified Fix** - Expression improvement legitimately fixed it
- ❌ **False Positive** - Passes test but broken (like CREATE)
- ⚠️ **Unclear** - Need deeper investigation

## Action Plan

### Immediate (Today)
1. ✅ Document CREATE expression evaluation bug
2. ✅ Document MERGE crash bug
3. [ ] Sample 20 more XPASSED tests
4. [ ] Categorize into fix/bug/unclear

### Short-term (This Week)
1. [ ] Fix CREATE expression evaluation
2. [ ] Fix MERGE crash
3. [ ] Verify all XPASSED tests
4. [ ] Update xfail markers appropriately

### Metrics Correction

**Current Reported**:
- 580 PASSED + 98 XPASSED = 678 "effective passes"

**Reality**:
- 580 PASSED (verified)
- 98 XPASSED = **UNKNOWN** (some bugs, some fixes)
- Cannot claim 678 passes until verified

**Conservative Estimate**:
- Assume 50% are legitimate fixes: ~49 tests
- Assume 50% are bugs/false-positives: ~49 tests
- Realistic passes: 580 + 49 = **629 verified**

## Recommendations

### 1. Stop Using "Effective Pass Rate"
Don't count XPASSED as passes until verified.

**Report**:
- PASSED: 580 (35.9%) ✅
- XPASSED: 98 (6.1%) ⚠️ **Under Investigation**
- FAILED: 154 (9.5%) ❌
- XFAILED: 783 (48.5%) ⏭️

### 2. Prioritize XPASSED Investigation
Higher priority than fixing FAILED tests because:
- XPASSED might be hiding bugs
- False metrics lead to false confidence
- Some are easy wins (just remove xfail marker)

### 3. Fix Critical Bugs First
**Priority 1**: CREATE expression evaluation (10+ tests affected)
**Priority 2**: MERGE crash (10+ tests affected)
**Priority 3**: Verify remaining XPASSEDs

## Estimated Time

- Sample 20 XPASSED tests: 2-3 hours
- Fix CREATE expressions: 2-4 hours
- Fix MERGE crash: 2-4 hours
- Verify all 98 XPASSEDs: 6-8 hours
- **Total: 12-19 hours**

## Success Criteria

✅ All 98 XPASSED tests categorized as:
   - Verified fix → Remove xfail marker → PASSED
   - False positive → Fix bug → XFAILED or FAILED
   - Unclear → Document for deeper investigation

✅ CREATE expression evaluation fixed
✅ MERGE crash fixed
✅ Accurate pass rate metrics

---

**Priority**: CRITICAL
**Risk**: HIGH - Hiding bugs, inflating metrics
**User Concern**: VALIDATED - User was correct to question this
