# Alloy Development Guide

This guide provides detailed instructions for developing and contributing to the Alloy programming language.

## Project Structure

```
alloy/
├── README.md                    # Project overview and quick start
├── pyproject.toml              # Python packaging and tool configuration
├── requirements.txt            # Runtime dependencies
├── .gitignore                  # Git ignore patterns
├── src/
│   └── alloy/
│       ├── __init__.py         # Package initialization and exports
│       ├── lexer.py           # Tokenization and lexical analysis
│       ├── parser.py          # Grammar parsing and AST generation
│       ├── interpreter.py     # AST execution engine
│       ├── ast_nodes.py       # AST node type definitions
│       └── environment.py     # Variable and scope management
├── tests/
│   ├── __init__.py
│   ├── test_lexer.py         # Lexer-specific tests
│   ├── test_parser.py        # Parser-specific tests
│   ├── test_interpreter.py   # Interpreter-specific tests
│   ├── test_language_features.py  # End-to-end language tests
│   └── test_examples.py      # Example program tests
├── docs/
│   ├── language-specification.md  # Complete language specification
│   ├── alloy-mvp-spec.md      # MVP specification
│   └── development.md        # This file
└── examples/
    ├── basic.alloy           # Basic language examples
    ├── agent_examples.alloy  # Agent usage examples
    └── tools_examples.alloy  # Tool definition examples
```

## Development Workflow

### 1. Test-Driven Development (TDD)

This project follows TDD practices:

1. **Write tests first** for new language features
2. **Run tests** to ensure they fail initially (Red)
3. **Implement minimal code** to make tests pass (Green)
4. **Refactor** while keeping tests green (Refactor)
5. **Repeat** for each new feature

### 2. Component Architecture

The interpreter is divided into logical components:

- **Lexer** (`src/alloy/lexer.py`): Tokenizes source code
- **Parser** (`src/alloy/parser.py`): Generates AST from tokens
- **AST Nodes** (`src/alloy/ast_nodes.py`): Defines AST structure
- **Environment** (`src/alloy/environment.py`): Manages variable/tool/agent scoping
- **Interpreter** (`src/alloy/interpreter.py`): Executes AST nodes

### 3. Adding New Language Features

To add a new language feature:

1. **Write tests** in the appropriate test file
2. **Update lexer** if new tokens are needed
3. **Update parser** for new grammar rules
4. **Add AST nodes** if new node types are needed
5. **Update interpreter** for execution logic
6. **Update documentation** and examples

## Setting Up Development Environment

### 1. Clone and Install

```bash
git clone <repository-url>
cd alloy
pip install -e .[dev]
```

### 2. Environment Configuration

Create a `.env` file for API keys:

```bash
OPENAI_API_KEY=your_openai_api_key_here
```

### 3. Run Tests

```bash
# Run all tests
pytest

# Run with coverage
pytest --cov=src/alloy --cov-report=html

# Run specific test file
pytest tests/test_parser.py

# Run specific test
pytest tests/test_parser.py::TestParser::test_tool_definition
```

### 4. Code Quality

```bash
# Format code
black src/ tests/

# Sort imports
isort src/ tests/

# Type checking
mypy src/

# Linting
flake8 src/ tests/
```

## Language Implementation Details

### Token and Grammar Rules

#### Adding New Tokens

1. Add token to `tokens` tuple in `lexer.py`
2. Add token rule (either simple pattern or function)
3. Update reserved keywords if needed

```python
# In lexer.py
tokens = (
    # ... existing tokens ...
    'NEW_TOKEN',
)

t_NEW_TOKEN = r'pattern'  # Simple pattern
# OR
def t_NEW_TOKEN(self, t):
    r'pattern'
    # processing logic
    return t
```

#### Adding New Grammar Rules

1. Add grammar rule function to `parser.py`
2. Update existing rules if they need to reference the new construct

```python
# In parser.py
def p_new_construct(self, p):
    '''new_construct : TOKEN1 TOKEN2 expression'''
    p[0] = ('new_construct', p[1], p[2], p[3])
```

### AST Node Design

#### Creating New AST Node Types

1. Define class in `ast_nodes.py`
2. Add to `ast_to_tuple` function for backward compatibility

```python
# In ast_nodes.py
class NewConstruct(Statement):
    def __init__(self, param1: str, param2: Expression):
        super().__init__("new_construct")
        self.param1 = param1
        self.param2 = param2
```

### Interpreter Execution

#### Adding New Node Execution

1. Add visitor method to `interpreter.py`
2. Handle the node type in `visit` method

```python
# In interpreter.py
def _visit_new_construct(self, node: tuple, env: Environment) -> Any:
    """Execute new construct"""
    param1, param2 = node[1], node[2]
    # implementation logic
    return result

# Add to visit method:
elif node_type == 'new_construct':
    return self._visit_new_construct(node, env)
```

## Testing Guidelines

### Test Organization

- **Unit tests**: Test individual components (lexer, parser, interpreter)
- **Integration tests**: Test component interactions
- **Feature tests**: Test complete language features end-to-end
- **Example tests**: Test real-world usage patterns

### Writing Good Tests

```python
class TestNewFeature(unittest.TestCase):
    def setUp(self):
        self.parser = AlloyParser()
        self.interpreter = Interpreter()
    
    def test_feature_basic_case(self):
        """Test basic functionality"""
        code = "basic_example"
        ast = self.parser.parse(code)
        result = self.interpreter.interpret(ast)
        self.assertEqual(result, expected_value)
    
    def test_feature_edge_case(self):
        """Test edge cases"""
        # Test boundary conditions
        pass
    
    def test_feature_error_case(self):
        """Test error handling"""
        with self.assertRaises(RuntimeError):
            # Test that errors are properly raised
            pass
```

### Mocking AI Interactions

For tests involving AI agents, use mocks to avoid API calls:

```python
from unittest.mock import Mock, patch

class MockInterpreter(Interpreter):
    def __init__(self):
        super().__init__()
        self.openai_client = Mock()
    
    def execute_agent(self, node, env):
        return "mock_response", None
```

## Code Style Guidelines

### Python Style

- Follow PEP 8 conventions
- Use type hints for all public methods
- Write docstrings for all classes and public methods
- Keep line length under 88 characters (Black default)

### Naming Conventions

- **Classes**: PascalCase (`AlloyLexer`)
- **Functions/Methods**: snake_case (`visit_node`)
- **Variables**: snake_case (`node_type`)
- **Constants**: UPPER_SNAKE_CASE (`DEFAULT_MODEL`)
- **Private methods**: prefix with underscore (`_visit_assign`)

### Documentation

- Write clear docstrings for all public APIs
- Include parameter types and return types
- Provide examples for complex functionality
- Keep comments concise and focused on "why" not "what"

## Debugging

### Parser Debugging

```python
# Enable parser debug output
parser = AlloyParser()
ast = parser.parse(code, debug=True)
```

### Lexer Debugging

```python
# Test lexer tokens
lexer = AlloyLexer()
lexer.test("code to tokenize")
```

### Interpreter Debugging

```python
# Add debug prints in visitor methods
def visit(self, node, env):
    print(f"Visiting node: {node}")
    # ... rest of method
```

## Performance Considerations

### Parser Performance

- PLY generates parser tables (`parsetab.py`) that are cached
- Avoid regenerating parser unnecessarily
- Use `write_tables=False` for temporary parsers in tests

### Memory Management

- Environment objects form a chain - avoid circular references
- Clean up temporary environments in loops
- Use generators for large data processing when possible

## Release Process

### Version Management

1. Update version in `pyproject.toml` and `__init__.py`
2. Update `CHANGELOG.md` with new features and fixes
3. Create git tag: `git tag v0.1.0`
4. Push tag: `git push origin v0.1.0`

### Testing Before Release

```bash
# Full test suite
pytest

# Test installation
pip install -e .[dev]

# Test examples
python examples/basic_example.py

# Verify documentation builds
# (if using sphinx or similar)
```

## Common Issues and Solutions

### PLY Parser Issues

**Problem**: Parser not recognizing new grammar rules
**Solution**: Delete `parsetab.py` and `parser.out`, then rebuild

**Problem**: Shift/reduce conflicts
**Solution**: Review grammar rules for ambiguity, add precedence rules

### Import Issues

**Problem**: Can't import from new module structure
**Solution**: Ensure `__init__.py` files are present and properly configured

### Test Failures

**Problem**: Tests fail after refactoring
**Solution**: Update test imports and mock objects to match new structure

## Contributing

1. Fork the repository
2. Create a feature branch: `git checkout -b feature/new-feature`
3. Follow TDD practices: write tests first
4. Implement the feature
5. Ensure all tests pass and code is formatted
6. Submit a pull request with clear description

## Getting Help

- Check existing tests for examples
- Review the language specification for requirements
- Look at similar language implementations for inspiration
- Ask questions in project discussions or issues