# XPASSED Tests: Detailed Explanation

## You're Absolutely Right! ⚠️

**XPASSED tests ARE a problem that needs fixing.** They indicate unexpected behavior.

## What is XPASSED?

```
PASSED  ✅ - Expected to pass → Passes      = CORRECT
FAILED  ❌ - Expected to pass → Fails       = BUG (needs fix)
XFAILED ⏭️ - Expected to fail → Fails       = CORRECT (unsupported)
XPASSED ⚠️ - Expected to fail → Passes      = UNEXPECTED (needs investigation!)
```

## Why Are 98 Tests XPASSED?

Tests get marked "expected to fail" (`xfail`) in TWO ways:

### Method 1: Pre-marked in UNSUPPORTED_CATEGORIES
```python
UNSUPPORTED_CATEGORIES = {
    "call": "CALL clause not supported",
    "temporal": "Date/Time types not supported"
}
```
Tests in these categories are marked xfail BEFORE running.

### Method 2: Runtime xfail
```python
if result["status"] == "error":
    pytest.xfail(f"NOT IMPLEMENTED: {result['message']}")
```
Tests that throw errors are marked xfail DURING execution.

**XPASS only happens with Method 1** - pre-marked tests that unexpectedly pass.

## Our XPASSED Tests Breakdown

**Total: 98 XPASSED tests**

### ❌ Confirmed False Positives (13 tests)
**Category**: clauses::call::* (CALL clause)
- Status: CALL not implemented (methods just `pass`)
- Why XPASSED: Returns empty `{}`, matches expected empty result
- **Action**: FIXED - added `NotImplementedError`
- **Result**: Should now be XFAILED (correctly)

### ⚠️ Need Investigation (85+ tests)
**Categories**:
- clauses::merge::* (10 tests)
- expressions::literals::* (11 tests)
- clauses::create::* (10 tests)
- clauses::match::* (11 tests)
- clauses::with-orderBy::* (7 tests)
- And 36+ more

**Question**: WHY were these marked as expected to fail?

**Possible Reasons**:
1. **Previously broken, now fixed** (GOOD) → Should remove xfail marker
2. **False positive** (BAD) → Passing for wrong reasons
3. **Marked wrong category** (MISTAKE) → Never should have been xfail

## Investigation Strategy

For each XPASSED test category, we need to:

### Step 1: Check if it's in UNSUPPORTED_CATEGORIES
```bash
# Check test_tck.py UNSUPPORTED_CATEGORIES
# If yes → investigate why it's passing
# If no → find why it was marked xfail
```

### Step 2: Run test manually and verify
```bash
poetry run python -m pytest "nxcypher/test_tck.py::test_tck_scenario[<test>]" -vvs
# Check:
# - Does it actually pass correctly?
# - What query is it running?
# - Are results correct or just "lucky"?
```

### Step 3: Categorize
- ✅ **Legit fix** → Remove xfail marker, document what was fixed
- ❌ **False positive** → Keep xfail, fix the code
- ⚠️ **Unclear** → Needs deeper investigation

## Current Status of Investigation

### ✅ Completed
- **CALL tests**: Confirmed false positives, FIXED

### 🔄 In Progress
- Need to sample and verify remaining 85 XPASSED tests

### ⏳ Todo
1. Sample 10-20 tests from major XPASSED categories
2. Run each manually with verbose output
3. Verify they're passing for correct reasons
4. Create action plan:
   - Remove xfail if legitimately fixed
   - Fix code if false positive
   - Document findings

## Why This Happened

Our expression text change (`EXPR_*` → `"expression text"`) likely fixed issues that made tests pass. But we need to verify each one is passing CORRECTLY, not accidentally.

## Action Plan

### Phase 1: Quick Sampling (1-2 hours)
Sample 2-3 tests from each major XPASSED category:
```bash
# Merge
poetry run python -m pytest "nxcypher/test_tck.py::test_tck_scenario[clauses::merge::Merge5::[22]]" -vvs

# Literals
poetry run python -m pytest "nxcypher/test_tck.py::test_tck_scenario[expressions::literals::Literals8::[20]]" -vvs

# Create
poetry run python -m pytest "nxcypher/test_tck.py::test_tck_scenario[clauses::create::Create1::[15]]" -vvs

# Match
poetry run python -m pytest "nxcypher/test_tck.py::test_tck_scenario[clauses::match::Match1::[7]]" -vvs
```

For each:
- [ ] Verify test passes correctly
- [ ] Check query and expected results
- [ ] Determine if fix is legitimate

### Phase 2: Systematic Review (2-4 hours)
For each category:
- [ ] Check 5-10 samples
- [ ] Categorize as legit/false-positive/unclear
- [ ] Document findings

### Phase 3: Fix or Update (1-2 hours)
- [ ] Remove xfail markers for legitimate fixes
- [ ] Fix code for false positives
- [ ] Document unclear cases for deeper investigation

## Expected Outcome

After investigation, we should have:

**Scenario A: Most are legitimate fixes** (GOOD)
- Remove xfail markers
- Update documentation
- Tests become PASSED (not XPASSED)
- Improves our pass rate

**Scenario B: Many are false positives** (BAD)
- Keep xfail markers
- Fix underlying code issues
- Tests become XFAILED (correctly)
- Prevents misleading metrics

**Scenario C: Mixed** (LIKELY)
- Some legitimate, some false positives
- Handle each category appropriately
- Get accurate picture of TCK compliance

## Key Insight

The user is correct: **XPASSED = Something unexpected happened**

We cannot just assume they're "legitimate improvements" without verification. Each one needs investigation to ensure we're not hiding bugs or creating false confidence in our TCK compliance.

## Next Immediate Action

**Run Phase 1 sampling NOW** to get initial sense of whether XPASSEDs are mostly good or mostly problematic.

---

**Priority**: HIGH
**Estimated Time**: 4-7 hours total
**Status**: Investigation needed
**Risk**: High (could be hiding bugs or inflating compliance metrics)
