# XPASS Investigation Report

## Overview

**Current XPassed Count:** 96 tests
**Previous Count:** 67 tests
**Growth:** +29 tests

## XPassed Test Breakdown

| Category | Count | Status | Notes |
|----------|-------|--------|-------|
| clauses::match | 16 | ✅ Legitimate | Match improvements from fixes |
| expressions::literals | 14 | ✅ Legitimate | Literal handling improvements |
| clauses::call | 13 | ⚠️ FALSE POSITIVE | CALL not implemented but tests pass |
| clauses::merge | 11 | ❓ Needs Check | MERGE implementation status unclear |
| clauses::create | 10 | ❓ Needs Check | CREATE improvements from fixes |
| clauses::with-orderBy | 7 | ✅ Likely Legitimate | ORDER BY fixes |
| clauses::return | 5 | ✅ Legitimate | Return clause fixes |
| expressions::quantifier | 4 | ✅ Legitimate | Quantifier fixes |
| expressions::boolean | 4 | ✅ Legitimate | Boolean logic improvements |
| clauses::with | 2 | ✅ Legitimate | WITH clause fixes |
| clauses::union | 2 | ❓ Needs Check | UNION implementation |
| clauses::return-skip-limit | 2 | ✅ Legitimate | SKIP/LIMIT fixes |
| clauses::return-orderby | 2 | ✅ Legitimate | ORDER BY fixes |
| expressions::list | 2 | ✅ Legitimate | List fixes |
| expressions::pattern | 1 | ❓ Needs Check | Pattern improvements |
| expressions::mathematical | 1 | ✅ Legitimate | Math fixes |

**Total:** 96 tests

---

## Legitimate XPasses (66 tests - 69%)

These are **true improvements** from our fixes:
- All expression category improvements (literals, math, list, quantifier, boolean)
- Return clause improvements (RETURN, ORDER BY, SKIP/LIMIT)
- Match clause improvements
- WITH clause improvements

**Action:** ✅ Remove xfail markers - these should be regular passes

---

## False Positives (13 tests - 14%)

### clauses::call (13 tests) ⚠️

**Issue:** CALL clause is **NOT implemented** but tests are passing

**Evidence:**
```python
# From interpreter.py lines 2194-2206
def _handle_standalone_call(self, node: Tree):
    raise NotImplementedError(
        "CALL clause (stored procedures) is not supported by nxCypher."
    )
```

**Why tests pass:**
- Test expects empty result
- Query silently fails/returns empty
- Test framework doesn't catch NotImplementedError properly

**Example Test:**
```cypher
CALL test.doNothing()
-- Expected: empty result
-- Actual: empty result (but for wrong reason - error is swallowed)
```

**Action:** 🔧 Fix test framework to properly detect NotImplementedError
- Modify `execute_scenario()` in `test_tck.py`
- Ensure NotImplementedError is caught and reported as failure
- Re-run tests - should show as xfail, not xpass

**Fix Location:**
```python
# test_tck.py, around line 1182
try:
    result = NXCypher(G).run(query)
except NotImplementedError as e:
    # Should not pass silently for unsupported features
    if scenario.expect_error:
        return {"status": "pass", "message": f"Expected error: {e}"}
    return {"status": "error", "message": f"NotImplementedError: {e}"}
except Exception as e:
    ...
```

---

## Needs Investigation (17 tests - 18%)

### clauses::merge (11 tests)

**Question:** Is MERGE implemented?

**Check:**
```bash
grep -n "def.*merge\|MERGE" nxcypher/interpreter.py
grep -n "merge" nxcypher/grammars/*.lark
```

**Expected:** MERGE is complex (CREATE + MATCH combination)
- If implemented: These are legitimate passes
- If not: These are false positives

### clauses::create (10 tests)

**Status:** CREATE is implemented, but these 10 were expected to fail

**Possible Reasons:**
1. Edge cases that now work due to expression fixes
2. CREATE with complex expressions now evaluated correctly
3. Side effect tracking improvements

**Action:** Review each test individually to verify correctness

### Other Categories (6 tests)
- clauses::union (2) - Check UNION implementation
- expressions::pattern (1) - Pattern predicate improvements

---

## Impact on Pass Rate

### Current Stats (with XPASS counted as passes)
- Passed: 580
- XPassed: 96
- **Effective Passes: 676** (580 + 96)
- Failed: 154
- **Pass Rate: 88.5%** (676 / 763 runnable tests)

### If CALL false positives fixed (13 tests)
- XPassed: 96 - 13 = 83
- XFailed: 785 + 13 = 798
- **Effective Passes: 663** (580 + 83)
- **Pass Rate: 87.0%** (663 / 763)

### If legitimate XPASS converted to PASS (66 tests)
- Passed: 580 + 66 = 646
- XPassed: 30 (under investigation)
- **Pass Rate: 84.7%** (646 / 763 regular passes)

---

## Recommended Actions

### Priority 1: Fix CALL False Positives (URGENT)
**Time:** 30 minutes
**Impact:** -13 false passes

1. Update test framework to catch `NotImplementedError`
2. Verify CALL tests show as xfail
3. Confirm MERGE/UNION implementation status

### Priority 2: Verify Legitimate XPasses
**Time:** 1-2 hours
**Impact:** Clean up test markers

1. Review expression XPasses (literals, math, list, etc.)
2. Remove xfail markers for confirmed working tests
3. Update UNSUPPORTED_CATEGORIES if needed

### Priority 3: Investigate Unclear XPasses
**Time:** 2-3 hours
**Impact:** Verify test correctness

1. Check MERGE implementation and test each scenario
2. Check CREATE edge cases
3. Verify UNION tests

---

## Test Framework Issues Found

### Issue 1: NotImplementedError Not Caught
**Location:** `test_tck.py` line ~1182
**Problem:** NotImplementedError returns empty result instead of error
**Fix:** Add explicit NotImplementedError handling

### Issue 2: XPASS Categorization
**Problem:** No easy way to distinguish legitimate vs false positive XPASS
**Improvement:** Add XPASS reason tracking

### Issue 3: UNSUPPORTED_CATEGORIES Not Updated
**Problem:** Categories like "call" marked unsupported but some tests pass
**Fix:** Add strict validation - unsupported features should always xfail

---

## Commands for Verification

### List all XPASS tests
```bash
uv run pytest nxcypher/test_tck.py --tb=no -v 2>&1 | grep "XPASS"
```

### Check specific category
```bash
uv run pytest nxcypher/test_tck.py -k "call::Call" -v
```

### Get XPASS breakdown
```bash
uv run pytest nxcypher/test_tck.py --tb=no -v 2>&1 | \
  grep "XPASS" | \
  sed 's/.*test_tck_scenario\[//' | sed 's/\] XPASS.*//' | \
  awk -F'::' '{category=$1"::"$2; count[category]++} END {for (c in count) print count[c], c}' | \
  sort -rn
```

---

## Summary

**Key Findings:**
1. ✅ **69% of XPASS are legitimate** improvements from our fixes
2. ⚠️ **14% are false positives** (CALL tests)
3. ❓ **18% need investigation** (MERGE, CREATE edge cases)

**Priority Actions:**
1. Fix test framework to catch NotImplementedError properly
2. Convert legitimate XPasses to regular passes (remove xfail markers)
3. Investigate MERGE/CREATE XPasses for correctness

**Expected Final State:**
- ~646 regular passes (after converting legitimate XPasses)
- ~30 XPasses (under investigation)
- ~154 failures (current)
- **~85% true pass rate**

---

**Report Date:** 2026-02-09
**Status:** Investigation complete, fixes pending
**Next Step:** Fix CALL false positives in test framework
