# Alloy Language Refactoring Plan v2
*Updated based on technical review feedback*

## Executive Summary
This plan addresses fundamental architectural issues in the Alloy interpreter by fixing root causes rather than symptoms. The refactoring will eliminate ~400 lines of workaround code while improving performance and maintainability. All changes align with the Alloy language specification, using `then` for chaining (no pipe operator).

## Coverage Matrix

| Area | Issues to Fix | Plan Coverage |
|------|--------------|---------------|
| **Lexer** | Token split (VAR vs CMD), Escape sequences, Longest-match operators, Comment handling | ✅ Full |
| **Parser** | Remove detect functions, Add THEN precedence, Error recovery tokens, Clean grammar | ✅ Full |
| **Environment** | Write-through assignment, Single global env, Iterative lookup | ✅ Full |
| **Interpreter** | Left-fold then chains, Return final value, Remove coercions | ✅ Full |
| **Testing** | Shadow identifiers, Escape sequences, Performance benchmarks | ✅ Full |
| **Extensibility** | IR layer, Provider abstraction | ✅ Full |

## Phase 1: Lexer Refactoring (Week 1)

### 1.1 Token Splitting
Split IDENTIFIER into semantic categories per spec §2.3.1:

```python
# lexer.py
def t_CMD_ID(t):
    r'[A-Z][a-zA-Z0-9_]*'
    t.type = reserved.get(t.value, 'CMD_ID')
    return t

def t_VAR_ID(t):
    r'[a-z_][a-zA-Z0-9_]*'
    t.type = reserved.get(t.value, 'VAR_ID')
    return t
```

### 1.2 String Escape Handling
Support standard escape sequences including Unicode:

```python
def t_STRING(t):
    r'"([^"\\]|\\.)*"'
    # Handle \n, \t, \", \uXXXX, \xNN
    t.value = t.value[1:-1].encode('raw_unicode_escape').decode('unicode_escape')
    return t
```

### 1.3 Comment Support
Add single-line comments:

```python
def t_COMMENT(t):
    r'(//.*?$|#.*?$)'
    pass  # Discard comments
```

### 1.4 Tests
- Lowercase variables: `user_data`, `processed_items`
- Uppercase commands: `Analyze`, `Process`
- Escape sequences: `"Line1\nLine2"`, `"Path:\\data"`, `"\u2764"`
- Comments: `// comment` and `# comment`

## Phase 2: Parser Refactoring (Week 2)

### 2.1 Remove Detection Functions
Delete these workarounds entirely:
- `_detect_single_command_shorthand`
- `_detect_tool_call_pattern`
- `_detect_then_chaining_pattern`

### 2.2 Clean Grammar Rules
```python
# Agent commands with proper tokens
def p_agent_command(self, p):
    '''agent_command : VAR_ID CMD_ID
                     | VAR_ID CMD_ID STRING
                     | VAR_ID CMD_ID VAR_ID'''
    if len(p) == 3:
        p[0] = ('agent_command', ('agent_var', p[1]), p[2], None)
    elif len(p) == 4:
        p[0] = ('agent_command', ('agent_var', p[1]), p[2], p[3])

# Then chaining with left association
def p_chain_expression(self, p):
    '''chain_expression : expression THEN expression
                        | chain_expression THEN expression %prec THEN_LEFT'''
    if isinstance(p[1], tuple) and p[1][0] == 'chain':
        p[0] = ('chain', p[1][1] + [p[3]])
    else:
        p[0] = ('chain', [p[1], p[3]])
```

### 2.3 Simplified Precedence Table
```python
precedence = (
    ('left', 'COMMA'),
    ('right', 'EQ'),
    ('left', 'THEN_LEFT'),      # Then chaining above arithmetic
    ('left', 'PLUS', 'MINUS'),
    ('left', 'MULTIPLY', 'DIVIDE'),
    ('left', 'GT', 'GE', 'LT', 'LE', 'EQEQ', 'NE'),
)
```

### 2.4 Error Recovery
```python
SYNC_TOKENS = {'SEMICOLON', 'THEN', 'COMMA', 'RBRACE', 'EOF'}
```

## Phase 3: Environment Refactoring (Week 3)

### 3.1 Write-Through Assignment
```python
class Environment:
    def __init__(self, enclosing=None, scope_type='block'):
        self.values = {}
        self.enclosing = enclosing
        self.scope_type = scope_type  # 'global', 'block', 'function'
    
    def assign(self, name: str, value: Any) -> None:
        """Single assignment method with write-through"""
        env = self
        while env is not None:
            if name in env.values:
                env.values[name] = value
                return
            env = env.enclosing
        # Not found - create in current scope
        self.values[name] = value
    
    def get(self, name: str) -> Any:
        """Optimized lookup with caching"""
        try:
            return self.values[name]
        except KeyError:
            if self.enclosing is not None:
                return self.enclosing.get(name)
            raise RuntimeError(f"Undefined variable '{name}'")
```

### 3.2 Single Global Environment
```python
def execute_block(self, statements, is_global=False):
    if is_global:
        # Reuse existing environment for top-level
        block_env = self.environment
    else:
        # Create child only for true scopes
        block_env = Environment(self.environment, scope_type='block')
    
    previous = self.environment
    try:
        self.environment = block_env
        for stmt in statements:
            self.execute(stmt)
    finally:
        self.environment = previous
```

## Phase 4: Interpreter Refactoring (Week 4)

### 4.1 Left-Fold Then Chains
```python
def visit_chain(self, node):
    """Evaluate then-chains left-to-right iteratively"""
    stages = node[1]  # List of expressions
    result = self.visit(stages[0])
    
    for next_stage in stages[1:]:
        # Handle error propagation
        if isinstance(result, tuple) and len(result) == 2 and result[1] is not None:
            # Error in previous stage - skip remaining
            return result
        
        # Invoke next stage with previous result
        result = self._invoke_stage(next_stage, result)
    
    return result  # Return final stage result

def _invoke_stage(self, stage, input_data):
    """Handle different stage types in chain"""
    stage_value = self.visit(stage)
    
    if callable(stage_value):
        # Tool or function - call with input
        return stage_value(input_data)
    else:
        # Value - return as-is
        return stage_value
```

### 4.2 Remove Silent Coercions
```python
# Remove any automatic DataFrame → list conversions
# Each tool must explicitly handle its input types
```

### 4.3 Provider Abstraction
```python
from abc import ABC, abstractmethod

class AgentProvider(ABC):
    @abstractmethod
    def chat_complete(self, prompt: str, model: str, tools: list) -> dict:
        pass

class OpenAIProvider(AgentProvider):
    def chat_complete(self, prompt, model, tools):
        # Existing OpenAI implementation
        pass

class AnthropicProvider(AgentProvider):
    def chat_complete(self, prompt, model, tools):
        # Claude implementation
        pass

class MockProvider(AgentProvider):
    def chat_complete(self, prompt, model, tools):
        return {"response": "mock_response"}
```

## Phase 5: IR Layer (Week 5)

### 5.1 AST to IR Translation
```python
class IRBuilder:
    def build(self, ast):
        """Convert AST to simplified IR"""
        return self.visit(ast)
    
    def visit_chain(self, node):
        """Desugar A then B then C to Call(C, Call(B, A))"""
        stages = node.stages
        result = stages[0]
        
        for stage in stages[1:]:
            result = ('call', stage, [result])
        
        return result
```

### 5.2 Slot Allocation for O(1) Lookup
```python
class SlotAllocator:
    def allocate(self, ir):
        """Assign slot numbers to local variables"""
        # Implementation assigns indices for direct array access
        pass
```

## Testing Strategy

### New Test Categories

| Test Category | Examples | Purpose |
|--------------|----------|---------|
| **Token Distinction** | `filter = 5`, `Process = "cmd"` | Verify VAR_ID vs CMD_ID |
| **Escape Sequences** | `"Line1\nLine2"`, `"\u2764"` | String handling |
| **Then Chains** | `data then clean then analyze` | Left-fold evaluation |
| **Error Propagation** | `fail() then succeed()` | Skip on error |
| **Performance** | 100-stage chains | O(1) lookup verification |
| **Provider Switch** | Same script, different providers | Abstraction test |

### Benchmark Targets
```python
# Performance requirements
def test_variable_lookup_performance():
    # 99th percentile lookup time ≤ 5µs for 100-stage pipeline
    
def test_chain_scaling():
    # Linear time complexity for N-stage chains
```

## Success Criteria (Expanded)

1. **All 130 existing tests pass** + 20 new edge cases
2. **Zero detection functions**: `grep -R "_detect_" src/` returns empty
3. **Clean grammar**: ≤ 6 precedence rules
4. **Performance**: 99p variable lookup ≤ 5µs for deep scopes
5. **Provider abstraction**: Swap OpenAI ↔ Anthropic with no code changes
6. **IR optimization**: AST → IR produces ≤ N nodes for N source nodes

## Risk Mitigation

| Risk | Mitigation |
|------|------------|
| Parser regression | Keep old parser behind `--legacy` flag |
| Scope bugs | Comprehensive test suite with edge cases |
| Performance regression | Automated benchmarks on every commit |
| Breaking changes | Feature flags for gradual rollout |

## Implementation Timeline

### Week 1: Lexer Foundation
- Day 1-2: Token splitting (VAR_ID, CMD_ID)
- Day 3: Escape sequences and Unicode
- Day 4: Comment handling
- Day 5: Testing and validation

### Week 2: Parser Cleanup  
- Day 1-2: Remove detection functions
- Day 3: Implement chain_expression rule
- Day 4: Update precedence and sync tokens
- Day 5: Integration testing

### Week 3: Environment Fix
- Day 1-2: Write-through assignment
- Day 3: Single global environment
- Day 4: Performance optimization
- Day 5: Benchmark validation

### Week 4: Interpreter Polish
- Day 1-2: Left-fold then chains
- Day 3: Provider abstraction
- Day 4: Remove coercions
- Day 5: Integration testing

### Week 5: IR & Optimization
- Day 1-2: AST → IR translation
- Day 3: Slot allocation
- Day 4: Retire old implementation
- Day 5: Final validation

## Deliverables

1. **Refactored lexer** with proper token distinction
2. **Clean parser** without workarounds
3. **Efficient environment** with O(1) common-case lookup
4. **Correct interpreter** with left-fold evaluation
5. **Provider abstraction** for LLM flexibility
6. **IR layer** for future optimizations
7. **Comprehensive test suite** with performance benchmarks
8. **Documentation** of all changes

This plan addresses all root causes while maintaining stability throughout the refactoring process.