# Alloy Language Architecture v2: Long-Term Vision

## Executive Summary

This document outlines a complete architectural redesign of the Alloy programming language interpreter, moving from a simple parse-and-execute model to a modern multi-phase compiler pipeline. The new architecture addresses fundamental limitations in extensibility, performance, and correctness while aligning with the language specification's vision of async-first AI operations and meta-programming capabilities.

## Current Architecture Analysis

### Deficiencies
1. **Heuristic-based parsing**: Post-parse detection functions (`_detect_*`) are fragile workarounds
2. **Tuple-based AST**: Untyped representation makes analysis and transformation difficult
3. **Synchronous execution**: Blocks on AI operations despite spec requiring async-first
4. **No semantic analysis**: Type errors and undefined symbols caught at runtime
5. **Limited extensibility**: Adding features requires modifying core interpreter
6. **No optimization layer**: Direct AST execution prevents performance improvements

### Root Causes
- **Technical debt**: Incremental fixes rather than systematic design
- **Tool limitations**: PLY's constraints led to grammar workarounds
- **Missing phases**: No normalization, semantic analysis, or IR generation
- **Tight coupling**: Parser, interpreter, and runtime are intertwined

## Design Principles

1. **Specification Fidelity**: Implement the full Alloy spec without compromises
2. **Phase Separation**: Each compilation phase has clear inputs/outputs
3. **Extensibility First**: Plugin architecture for tools, agents, and language features
4. **Performance**: O(1) variable lookup, parallel AI operations, JIT potential
5. **Developer Experience**: Clear error messages, debugging support, tooling APIs
6. **Future-Proof**: Support for compilation, static analysis, and language evolution

## Proposed Architecture

### Pipeline Overview

```
┌─────────────┐     ┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Source    │────▶│    Lexer    │────▶│   Parser    │────▶│     AST     │
│    Code     │     │  (Tokens)   │     │   (ANTLR)   │     │ (Typed CST) │
└─────────────┘     └─────────────┘     └─────────────┘     └─────────────┘
                                                                     │
                    ┌─────────────┐     ┌─────────────┐            ▼
                    │   Plugins    │◀────│ Normalizer  │◀────┌─────────────┐
                    │   Registry   │     │  (Desugar)  │     │  Semantic   │
                    └─────────────┘     └─────────────┘     │  Analyzer   │
                            │                                └─────────────┘
                            ▼                                        │
                    ┌─────────────┐     ┌─────────────┐            ▼
                    │   Runtime    │◀────│     IR      │◀────┌─────────────┐
                    │   Engine     │     │  Generator  │     │     IR      │
                    └─────────────┘     └─────────────┘     │   (Typed)   │
                            │                                └─────────────┘
                            ▼
                    ┌─────────────────────────────────┐
                    │         Execution Context       │
                    ├─────────────┬─────────────────┬─┤
                    │   Agents    │   Tools    │Memory│
                    │  (Async)    │  (Sync)    │Store │
                    └─────────────┴─────────────────┴─┘
```

### Phase 1: Lexical Analysis

**Technology**: Custom lexer with proper Unicode support

```python
@dataclass
class Token:
    type: TokenType
    value: str
    location: SourceLocation
    
class Lexer:
    def tokenize(self, source: str) -> Iterator[Token]:
        # Proper handling of:
        # - VAR_ID vs CMD_ID distinction
        # - String escapes (\n, \t, \uXXXX)
        # - Comments (// and #)
        # - Multi-character operators
        # - Source location tracking
```

### Phase 2: Parsing

**Technology**: ANTLR4 or Lark for powerful grammar support

```antlr
// Alloy.g4
grammar Alloy;

program: statement* EOF;

statement
    : assignment
    | tupleAssignment
    | agentDefinition
    | toolDefinition
    | expression
    ;

agentCommand
    : VAR_ID CMD_ID                    // claude Analyze
    | VAR_ID CMD_ID STRING             // claude Analyze "data"
    | VAR_ID CMD_ID VAR_ID             // claude Analyze input
    ;

thenChain
    : expression (THEN expression)*    // Left-associative
    ;
```

### Phase 3: Typed AST

**Technology**: Python dataclasses with visitor pattern

```python
from abc import ABC
from dataclasses import dataclass
from typing import List, Optional

class ASTNode(ABC):
    location: SourceLocation

@dataclass
class Program(ASTNode):
    statements: List['Statement']

@dataclass
class AgentCommand(ASTNode):
    agent: 'Expression'
    command: str
    data: Optional['Expression']

@dataclass
class ThenChain(ASTNode):
    stages: List['Expression']

@dataclass
class ToolDefinition(ASTNode):
    name: str
    parameters: List[str]
    return_type: str
    body: 'Block'
```

### Phase 4: AST Normalization

**Purpose**: Desugar syntax into canonical forms

```python
class ASTNormalizer(ASTVisitor):
    def visit_agent_command_shorthand(self, node):
        # Transform: result, error = claude Analyze "text"
        # Into: TupleAssign([result, error], AgentCommand(...))
        
    def visit_then_chain(self, node):
        # Transform: a then b then c
        # Into: Chain([a, b, c]) with error propagation
        
    def visit_inline_tool(self, node):
        # Transform: data | filter(x => x > 0)
        # Into: ToolCall(filter, [data, Lambda(...)])
```

### Phase 5: Semantic Analysis

**Purpose**: Type checking, symbol resolution, validation

```python
@dataclass
class SemanticContext:
    symbols: SymbolTable
    types: TypeEnvironment
    agents: AgentRegistry
    tools: ToolRegistry

class SemanticAnalyzer(ASTVisitor):
    def analyze(self, ast: Program) -> AnalysisResult:
        # - Resolve all symbols
        # - Check tuple assignment arity
        # - Verify agent/tool existence
        # - Type inference for expressions
        # - Must/should semantic validation
        # - Error accumulation with recovery
```

### Phase 6: IR Generation

**Purpose**: Lower AST to executable intermediate representation

```python
@dataclass
class IRInstruction(ABC):
    pass

@dataclass
class LoadVar(IRInstruction):
    name: str
    slot: int  # Direct slot access

@dataclass
class CallAgent(IRInstruction):
    agent_id: str
    command: str
    args: List[IRValue]
    async_: bool = True

@dataclass
class Chain(IRInstruction):
    stages: List[IRInstruction]
    error_propagation: bool = True

class IRGenerator(ASTVisitor):
    def generate(self, ast: Program) -> IRProgram:
        # - Slot allocation for O(1) variable access
        # - Chain lowering with error checks
        # - Async annotation for AI operations
        # - Dead code elimination
        # - Constant folding
```

### Phase 7: Execution Engine

**Technology**: asyncio-based runtime with plugin support

```python
class ExecutionEngine:
    def __init__(self):
        self.plugins = PluginRegistry()
        self.providers = ProviderRegistry()
        self.memory = MemoryStore()
        
    async def execute(self, ir: IRProgram) -> ExecutionResult:
        context = ExecutionContext(
            environment=Environment(),
            loop=asyncio.get_event_loop()
        )
        
        for instruction in ir.instructions:
            result = await self.execute_instruction(instruction, context)
            if isinstance(result, Error):
                return result
                
        return Success(context.environment)
```

### Plugin Architecture

```python
class Plugin(Protocol):
    """Extension point for new language features"""
    
    def register_tokens(self, lexer: Lexer) -> None:
        """Add new token types"""
        
    def register_grammar(self, parser: Parser) -> None:
        """Add grammar rules"""
        
    def register_ast_nodes(self, registry: ASTRegistry) -> None:
        """Add AST node types"""
        
    def register_ir_handlers(self, engine: ExecutionEngine) -> None:
        """Add IR instruction handlers"""

# Example: Memory plugin
class MemoryPlugin(Plugin):
    def register_grammar(self, parser):
        parser.add_rule("""
            memoryOp: STORE VAR_ID IN memory_type
                    | RECALL VAR_ID FROM memory_type
        """)
```

### Provider Abstraction

```python
class AgentProvider(Protocol):
    async def complete(
        self,
        model: str,
        messages: List[Message],
        tools: Optional[List[Tool]] = None,
        **kwargs
    ) -> CompletionResult:
        """Universal interface for AI providers"""

class OpenAIProvider(AgentProvider):
    async def complete(self, model, messages, tools=None, **kwargs):
        # OpenAI-specific implementation
        
class AnthropicProvider(AgentProvider):
    async def complete(self, model, messages, tools=None, **kwargs):
        # Anthropic-specific implementation
```

## Implementation Phases

### Phase 1: Foundation (Months 1-2)
1. **Typed AST definition** using dataclasses
2. **ANTLR/Lark grammar** for complete language
3. **Basic semantic analyzer** with symbol tables
4. **IR definition** with core instructions

**Deliverables**: Parse any valid Alloy program into typed AST

### Phase 2: Core Execution (Months 3-4)
1. **Async execution engine** with asyncio
2. **Provider abstraction** for OpenAI/Anthropic
3. **Basic plugin system** for extensions
4. **Memory store** implementation

**Deliverables**: Execute basic Alloy programs with AI integration

### Phase 3: Optimization (Months 5-6)
1. **Slot allocation** for O(1) variable lookup
2. **IR optimization** passes (DCE, constant folding)
3. **Parallel agent** execution
4. **Caching layer** for AI responses

**Deliverables**: 10x performance improvement on benchmarks

### Phase 4: Advanced Features (Months 7-8)
1. **Meta-programming** support via compile-time plugins
2. **Debugger interface** with breakpoints and stepping
3. **Language server** protocol for IDE support
4. **Static analysis** tools

**Deliverables**: Developer tools and advanced language features

### Phase 5: Production Hardening (Months 9-10)
1. **Error recovery** and graceful degradation
2. **Resource limits** and sandboxing
3. **Distributed execution** support
4. **Comprehensive test** suite

**Deliverables**: Production-ready interpreter

## Migration Strategy

### Parallel Development
- Maintain v1 interpreter during v2 development
- Use v1 test suite to validate v2 correctness
- Feature flag for gradual migration

### Compatibility Layer
```python
class V1CompatibilityAdapter:
    """Run v1 AST on v2 engine"""
    def adapt(self, v1_ast: tuple) -> v2.ASTNode:
        # Convert tuple-based AST to typed AST
```

### Incremental Adoption
1. **Stage 1**: Use v2 lexer/parser, v1 execution
2. **Stage 2**: Add semantic analysis
3. **Stage 3**: Switch to IR execution
4. **Stage 4**: Enable async runtime
5. **Stage 5**: Full v2 with plugins

## Success Metrics

### Performance
- **Variable lookup**: O(1) with slot allocation
- **Agent calls**: Parallel execution with 10x throughput
- **Memory usage**: 50% reduction via IR optimization
- **Startup time**: < 100ms for typical programs

### Quality
- **Test coverage**: 95% line coverage, 100% branch coverage
- **Error messages**: Actionable with source locations
- **Type safety**: Catch 90% of errors at compile time
- **Spec compliance**: 100% of language features implemented

### Developer Experience
- **IDE support**: Full IntelliSense and refactoring
- **Debug time**: 5x faster with proper tooling
- **Extension time**: New feature in < 1 day via plugins
- **Documentation**: Auto-generated from types

## Risk Mitigation

### Technical Risks
- **ANTLR learning curve**: Mitigate with incremental grammar development
- **Async complexity**: Use battle-tested patterns from other languages
- **Performance regression**: Continuous benchmarking against v1

### Project Risks
- **Scope creep**: Strict phase boundaries with clear deliverables
- **Compatibility breaks**: Extensive test suite and migration tools
- **Resource constraints**: Modular design allows partial implementation

## Conclusion

This v2 architecture transforms Alloy from a simple interpreter into a modern language implementation with:
- **Correctness**: Typed AST and semantic analysis catch errors early
- **Performance**: IR optimization and async execution 
- **Extensibility**: Plugin architecture for ecosystem growth
- **Developer Experience**: Modern tooling and clear error messages

The phased approach allows incremental development while maintaining compatibility, setting Alloy up for long-term success as a production programming language for AI-integrated applications.