# XPASS Test Categorization - Phase 2 Results

## Summary

After completing Phase 1 (fixing CREATE expression evaluation and MERGE crash bugs), we have **88 XPASS tests** remaining.

**Test Results**:
- Total: 1,615 tests
- PASSED: 619 (38.3%)
- FAILED: 161 (10.0%)
- XFAILED: 747 (46.3%)
- **XPASSED: 88 ( 5.4%)**

## XPASS Categorization Strategy

XPASS tests occur when a test marked as "expected to fail" (`xfail`) unexpectedly passes. Tests are marked `xfail` for three reasons (from nxcypher/test_tck.py:1203-1244):

### 1. UNSUPPORTED_CATEGORIES (Pre-marked)
```python
UNSUPPORTED_CATEGORIES = {
    "call": "Stored procedures (CALL clause) not supported by NetworkX",
    "temporal": "Temporal types (Date/Time/Duration) not supported by NetworkX",
}
```

### 2. SyntaxError Expected (Lenient Parser)
```python
if scenario.expect_error == "SyntaxError":
    return True, "Parser validation not as strict as Neo4j"
```

Tests that expect `SyntaxError` are marked xfail because nxCypher's parser is intentionally more lenient than Neo4j.

### 3. Runtime Errors (Dynamic xfail)
```python
if result["status"] == "error":
    pytest.xfail(f"NOT IMPLEMENTED: {result['message']}")
```

Tests that throw runtime errors are marked xfail during execution.

## Breakdown of 88 XPASS Tests by Category

### Category Distribution

| Category | Count | Status |
|----------|-------|--------|
| **clauses::call** | 15 | ⚠️ FALSE POSITIVES |
| **clauses::match** | 16 | 🔍 INVESTIGATION NEEDED |
| **expressions::literals** | 14 | 🔍 INVESTIGATION NEEDED |
| **clauses::create** | 10 | ⚠️ SYNTAX ERROR TESTS |
| **clauses::with-orderBy** | 7 | 🔍 INVESTIGATION NEEDED |
| **clauses::return** | 5 | 🔍 INVESTIGATION NEEDED |
| **expressions::boolean** | 4 | 🔍 INVESTIGATION NEEDED |
| **expressions::quantifier** | 4 | 🔍 INVESTIGATION NEEDED |
| **clauses::return-orderby** | 2 | 🔍 INVESTIGATION NEEDED |
| **clauses::return-skip-limit** | 2 | 🔍 INVESTIGATION NEEDED |
| **clauses::union** | 2 | 🔍 INVESTIGATION NEEDED |
| **clauses::with** | 2 | 🔍 INVESTIGATION NEEDED |
| **expressions::list** | 2 | 🔍 INVESTIGATION NEEDED |
| **clauses::merge** | 1 | ⚠️ SYNTAX ERROR TEST |
| **expressions::mathematical** | 1 | 🔍 INVESTIGATION NEEDED |
| **expressions::pattern** | 1 | 🔍 INVESTIGATION NEEDED |

## Detailed Analysis

### 1. clauses::call (15 tests) - FALSE POSITIVES ⚠️

**Status**: Known false positives
**Root Cause**: CALL clause not implemented (methods just `pass`)
**Why XPASS**: Returns empty `{}` which matches expected empty results

**Tests**:
- Call1::[7,8,9,10,12,13,14,15,16] (9 tests)
- Call2::[4,5,6] (3 tests)
- Call5::[5,6,7] (3 tests)

**Action Taken**: Added `NotImplementedError` to CALL methods (nxcypher/interpreter.py:2194-2206)
**Current Status**: Should now be XFAIL (need to verify with rerun)

### 2. clauses::create (10 tests) - SYNTAX ERROR TESTS ⚠️

**Status**: Expect SyntaxError (VariableAlreadyBound)
**Why XPASS**: nxCypher parser is lenient, accepts invalid syntax

**Example**: Create1::[15]
```cypher
CREATE (n:Foo)-[:T1]->(),
       (n:Bar)-[:T2]->()
```
**Expected**: SyntaxError (variable `n` used with different labels)
**Actual**: Parser accepts it, test passes

**Tests**:
- Create1::[15,16,17,18,19] (5 tests) - VariableAlreadyBound scenarios
- Create2::[18,21,22,23,24] (5 tests) - More VariableAlreadyBound

**Action**: These XPASS are INTENTIONAL - nxCypher's lenient parser design choice
**Recommendation**: DOCUMENT and ACCEPT as expected behavior

### 3. clauses::match (16 tests) - MIXED 🔍

**Status**: Need investigation
**Likely**: Mix of syntax error tests and legitimate improvements

**Sample**: Match1::[7]
- Scenario Outline with multiple examples
- Expects SyntaxError: VariableTypeConflict
- Tests like `MATCH ()-[r]-() MATCH (r) RETURN r` (using relationship variable as node)

**Tests**: Match1::[7,10,11,...] + Match2, Match3 tests

**Action**: Sample 3-5 tests to determine if:
- Syntax error tests (lenient parser - OK)
- Legitimate fixes (should remove xfail)

### 4. expressions::literals (14 tests) - LIKELY LEGITIMATE ✅

**Status**: Likely legitimate improvements
**Reason**: Expression text changes improved literal parsing

**Example**: Literals8::[20] - Hex literals
```cypher
RETURN 0x10 AS hex  → Returns 16 correctly
```

**Tests**: Literals2, Literals3, Literals5, Literals6, Literals7, Literals8

**Action**: Verify 3-4 tests manually, likely should remove xfail markers

### 5. clauses::with-orderBy (7 tests) - INVESTIGATION NEEDED 🔍

**Status**: Unknown
**Tests**: WithOrderBy1::[46], WithOrderBy2::[25], WithOrderBy3::[8], WithOrderBy4::[3,4]

**Action**: Sample 2-3 tests to verify ORDER BY works correctly

### 6. Other Categories (26 tests) - LOW PRIORITY 🔍

**Categories**:
- clauses::return (5) - Return aggregation expressions
- expressions::boolean (4) - Boolean operations
- expressions::quantifier (4) - ALL/ANY/NONE/SINGLE predicates
- clauses::return-orderby (2) - Return with ORDER BY
- clauses::return-skip-limit (2) - SKIP/LIMIT
- clauses::union (2) - UNION queries
- clauses::with (2) - WITH clause
- expressions::list (2) - List operations
- clauses::merge (1) - Merge5::[27] - Syntax error test
- expressions::mathematical (1) - Math operations
- expressions::pattern (1) - Pattern expressions

**Action**: Lower priority, sample 5-10 tests across these categories

## Recommended Action Plan

### ✅ Immediate Actions (Completed)

1. **Fixed CALL false positives** - Added NotImplementedError
2. **Fixed CREATE expression bug** - Evaluate expressions in CREATE
3. **Fixed MERGE crash** - Correct unpacking of return values

### 🔄 Phase 2A: Quick Verification (2-3 hours)

1. **Rerun tests** - Verify CALL tests now XFAIL (not XPASS)
2. **Sample 5-10 tests** from key categories:
   - clauses::match (3 tests)
   - expressions::literals (3 tests)
   - clauses::with-orderBy (2 tests)
   - clauses::return (2 tests)

3. **Categorize each sample**:
   - ✅ Syntax error test (lenient parser - OK)
   - ✅ Legitimate fix (remove xfail marker)
   - ❌ False positive (fix implementation)

### 📊 Phase 2B: Full Analysis (4-6 hours)

1. Check ALL 73 remaining XPASS tests (after CALL fixes)
2. For each test:
   - Read feature file scenario
   - Determine if expects error
   - If no error expected, manually verify correctness
3. Update xfail markers appropriately
4. Document findings

### 📝 Phase 2C: Documentation (1-2 hours)

1. Document lenient parser behavior
2. Update TCK compliance report
3. Create list of intentional XPASS tests
4. Update pass rate metrics

## Expected Outcomes

### Scenario A: Most are lenient parser (GOOD)
- ~50-60 tests expect SyntaxError
- These XPASS are intentional
- Document as expected behavior
- **Result**: ~20-30 tests need attention

### Scenario B: Many are legitimate fixes (GOOD)
- ~30-40 tests legitimately improved
- Remove xfail markers
- **Result**: Pass rate increases to ~42-45%

### Scenario C: Some are false positives (BAD)
- ~10-20 tests passing incorrectly
- Fix underlying issues
- **Result**: More work needed, but metrics accurate

## Metrics Implications

**Current Metrics**:
- PASSED: 619 (38.3%)
- XPASSED: 88 (5.4%)

**Possible Future States**:

**Best Case** (most XPASS are lenient parser + legitimate):
- PASSED: 650+ (40%+) - after removing legitimate xfails
- XFAILED: 780+ (48%) - including lenient parser tests
- **Pass rate: 40-42%**

**Worst Case** (many false positives):
- PASSED: 619 (38.3%) - unchanged
- XFAILED: 800+ (49.5%) - after fixing false positives
- **Pass rate: 38.3%**

**Most Likely** (mixed):
- PASSED: 635 (39.3%) - some legitimate fixes
- XFAILED: 770 (47.7%) - lenient parser + false positives fixed
- **Pass rate: 39-40%**

## Conclusion

The 88 XPASS tests fall into distinct categories:

1. **Known False Positives** (15 CALL tests) - FIXED
2. **Lenient Parser** (~30-40 tests) - INTENTIONAL, DOCUMENT
3. **Legitimate Improvements** (~20-30 tests) - VERIFY and REMOVE xfail
4. **Need Investigation** (~10-20 tests) - MANUAL VERIFICATION

**Next Step**: Run quick verification of 5-10 samples to confirm categorization, then decide on full analysis approach.

---

**Date**: 2026-02-09
**Status**: Phase 2 in progress - categorization complete, verification pending
**Priority**: Medium (after critical bugs fixed in Phase 1)
