# Integrating the Complete Grammar into nxcypher

## Current State

- **Our grammar**: `lark_grammars/cypher_from_grammars_v4.lark` - 518 rules, 97.5% TCK pass rate
- **nxcypher grammar**: Embedded in `nxcypher/__init__.py` lines 48-415 - ~81 rules, integrated with interpreter

## Integration Options

### Option A: External Grammar File (Recommended)

Load the grammar from a separate file instead of embedding it.

```python
# In nxcypher/__init__.py

from pathlib import Path

# Load grammar from file
_grammar_path = Path(__file__).parent / 'grammars' / 'cypher.lark'
_NXCypherGrammar = Lark(_grammar_path.read_text(), start="start")
```

**Benefits:**
- Grammar can be updated independently
- Easier to test and validate
- Clear separation of concerns

**Steps:**
1. Copy `grammars/lark_grammars/cypher_from_grammars_v4.lark` to `nxcypher/grammars/cypher.lark`
2. Modify the Lark initialization to load from file
3. Update the Transformer class to handle new rule names

### Option B: Grammar Validation Layer

Use the complete grammar for validation, then use nxcypher's grammar for interpretation.

```python
# Add validation step
_FullCypherGrammar = Lark(Path('grammars/cypher.lark').read_text(), start="start")

def validate_cypher(query: str) -> bool:
    """Validate a Cypher query against the full grammar."""
    try:
        _FullCypherGrammar.parse(query)
        return True
    except Exception:
        return False

def run_cypher(query: str, graph: nx.Graph):
    """Run a Cypher query after validation."""
    if not validate_cypher(query):
        raise ValueError("Invalid Cypher query")
    # Continue with nxcypher's interpreter
    return NXCypher(query).run(graph)
```

**Benefits:**
- No changes to interpreter
- Better error messages from complete grammar
- Gradual migration path

### Option C: Incremental Enhancement

Add missing features to nxcypher's grammar using ours as reference.

**Priority features to add:**

1. **EXPLAIN/PROFILE prefixes**
   ```lark
   start: query_modifier? query (union_clause query)*
   query_modifier: "PROFILE"i | "EXPLAIN"i
   ```

2. **Schema commands**
   ```lark
   schema_command: create_index | drop_index | create_constraint | drop_constraint
   create_index: "CREATE"i "INDEX"i "ON"i ":" TYPE "(" CNAME ")"
   ```

3. **Legacy parameters**
   ```lark
   parameter: "$" CNAME | "{" CNAME "}"
   ```

4. **shortestPath/allShortestPaths**
   ```lark
   path_function: "shortestPath"i "(" pattern ")"
                | "allShortestPaths"i "(" pattern ")"
   ```

5. **REDUCE expression**
   ```lark
   reduce_expr: "REDUCE"i "(" CNAME "=" expression "," CNAME "IN"i expression "|" expression ")"
   ```

## Rule Name Mapping

Our grammar uses different rule names than nxcypher. Here's a partial mapping:

| Our Grammar | nxcypher | Notes |
|-------------|----------|-------|
| `script` | `start` | Entry point |
| `regular_query` | `query` | Main query |
| `single_part_q` | - | Need to add |
| `multi_part_q` | - | Partially in with_clause |
| `return_st` | `return_clause` | Similar |
| `with_st` | `with_clause` | Similar |
| `match_st` | `match_clause` | Similar |
| `expression` | `expression` | Same |
| `pattern_part` | `create_pattern` | Different structure |
| `node_pattern` | `node_match` | Different structure |

## Transformer Updates

The Transformer class in nxcypher would need updates for new rules:

```python
class CypherTransformer(Transformer):
    # Existing methods...

    # New methods for additional rules
    def schema_command(self, items):
        return {"type": "schema", "command": items[0]}

    def create_index(self, items):
        return {"action": "create_index", "label": items[0], "property": items[1]}

    def shortest_path_pattern(self, items):
        return {"type": "shortest_path", "pattern": items[0]}
```

## Testing After Integration

```python
# Test with the TCK queries
import json
from pathlib import Path

def test_integration():
    queries = json.loads(Path('grammars/tck-queries/all_queries.json').read_text())

    passed = 0
    failed = 0

    for q in queries:
        query = q.get('query', '')
        if not query.strip():
            continue
        try:
            NXCypher(query)  # Just parse, don't run
            passed += 1
        except Exception as e:
            failed += 1

    print(f"Pass rate: {passed}/{passed+failed} ({100*passed/(passed+failed):.1f}%)")
```

## Recommended Path

1. **Start with Option B** (validation layer) - Minimal changes, quick win
2. **Use Option C** to add high-value features incrementally
3. **Eventually migrate to Option A** for full grammar support

## Files to Create/Modify

```
nxcypher/
├── __init__.py              # Modify parser initialization
├── grammars/
│   └── cypher.lark          # Copy from grammars/lark_grammars/
├── transformer.py           # Extract transformer to separate file (optional)
└── validator.py             # Add validation module (Option B)
```
