# TCK Failure Analysis

**Date**: 2026-02-09
**Test Framework**: Official Cucumber/behave with openCypher TCK

## Current Results

| Metric | Count | Percentage |
|--------|-------|------------|
| **Passed** | 1178 | 30.2% |
| **Failed** | 2683 | 68.9% |
| **Error** | 36 | 0.9% |
| **Total** | 3897 | 100% |

## Failure Categories (by count)

### Category 1: Result Comparison Failures (1404 tests)
**Pattern**: "Expected row not found in actual results"

These are actual implementation failures where nxCypher returns wrong results.
Need to investigate specific patterns.

### Category 2: Internal Errors (198+ tests)
**Pattern**: "too many values to unpack"

| Error | Count |
|-------|-------|
| too many values to unpack (expected 2, got 3) | 150 |
| too many values to unpack (expected 2, got 4) | 48 |

**Root Cause**: Tuple unpacking issue in interpreter, likely in path handling.

### Category 3: Missing Validation (460+ tests)
**Pattern**: "Expected {ErrorType} but query succeeded"

| Error Type | Count | Priority |
|------------|-------|----------|
| InvalidArgumentType | 146 | HIGH |
| VariableAlreadyBound | 81 | HIGH |
| VariableTypeConflict | 79 | HIGH |
| UndefinedVariable | 28 | MEDIUM |
| InvalidAggregation | 26 | MEDIUM |
| UnexpectedSyntax | 12 | MEDIUM |

### Category 4: Entity Reference Errors (80+ tests)
**Pattern**: "Unknown entity name: X"

| Entity | Count |
|--------|-------|
| inputList | 30 |
| a | 23 |
| r | 11 |
| null | 10 |
| numOfValues | 9 |

**Root Cause**: Variables not resolved in expression evaluation.

### Category 5: Type Errors Expected (58+ tests)
**Pattern**: "Expected TypeError but query succeeded"

| Error | Count |
|-------|-------|
| InvalidArgumentValue | 28 |
| InvalidArgumentType | 17 |
| TypeError (*) wildcard | 11 |
| ArgumentError (InvalidArgumentType) | 13 |

### Category 6: Tree Comparison Bug (12 tests)
**Pattern**: "'<' not supported between instances of 'Tree' and 'Tree'"

Unevaluated syntax tree nodes being compared.

### Category 7: Side Effect Tracking (30+ tests)
**Pattern**: "Side effect mismatch for +/-labels/properties"

| Issue | Count |
|-------|-------|
| +labels | 11 |
| -properties | 11 |
| -labels | 7 |

### Category 8: Other Runtime Errors
| Pattern | Count |
|---------|-------|
| 'NoneType' object has no attribute 'edge_path' | 12 |
| cannot use 'tuple' as dict key | 10 |
| SET requires entity.property format | 8 |

## Recommended Priority Order

### Phase 1: Fix Critical Bugs (HIGH IMPACT - ~200 tests)
**Target**: Fix "too many values to unpack" errors

1. Fix tuple unpacking in interpreter (likely path handling)
2. **File**: `nxcypher/interpreter.py`
3. **Expected Impact**: +198 scenarios

### Phase 2: Fix Entity Reference (HIGH IMPACT - ~80 tests)
**Target**: Fix "Unknown entity name" errors

1. Ensure variable binding works for all contexts
2. Fix expression evaluation to resolve variables
3. **Expected Impact**: +80 scenarios

### Phase 3: Add Validation (MEDIUM IMPACT - ~460 tests)
**Target**: Raise proper validation errors

Priority order:
1. InvalidArgumentType (146 tests)
2. VariableAlreadyBound (81 tests) - partially done
3. VariableTypeConflict (79 tests)
4. UndefinedVariable (28 tests)
5. InvalidAggregation (26 tests)
6. **Expected Impact**: +300 scenarios (with partial implementations)

### Phase 4: Fix Result Comparison Issues (~100-200 tests)
**Target**: Better node/relationship comparison

1. Handle path comparisons
2. Handle complex nested structures
3. **Expected Impact**: +100-200 scenarios

### Phase 5: Type Error Validation (~58 tests)
**Target**: Raise TypeError for invalid operations

1. InvalidArgumentValue
2. InvalidArgumentType
3. **Expected Impact**: +58 scenarios

## Target Milestones

| Target | Passed | Rate | Delta | Notes |
|--------|--------|------|-------|-------|
| Current | 1178 | 30.2% | - | Baseline |
| Phase 1 | 1380 | 35% | +200 | Fix tuple unpacking |
| Phase 2 | 1460 | 37% | +80 | Fix entity reference |
| Phase 3 | 1760 | 45% | +300 | Add validation |
| Phase 4 | 1960 | 50% | +200 | Fix comparisons |
| Phase 5 | 2020 | 52% | +60 | Type errors |

## Quick Fix Checklist

- [ ] Fix "too many values to unpack" in interpreter
- [ ] Fix "Unknown entity name" for variable binding
- [ ] Add InvalidArgumentType validation
- [ ] Add VariableTypeConflict validation
- [ ] Fix Tree comparison bug
- [ ] Fix SET entity.property handling

## Files to Modify

| File | Changes Needed |
|------|---------------|
| `nxcypher/interpreter.py` | Tuple unpacking, validation |
| `nxcypher/__init__.py` | Exception classes |
| `features/steps/cypher_steps.py` | Side effect tracking |
