# Test Suite Documentation

This directory contains comprehensive unit tests and end-to-end tests for the bench-af project.

## Test Structure

### Unit Tests
- **`test_registry.py`** - Tests for the registry system (12 tests)
  - File discovery and listing functions
  - Model organism and detector loading
  - Error handling for missing/malformed files

- **`test_objects.py`** - Tests for core classes (15 tests)
  - ModelOrganism dataclass functionality
  - Detector abstract base class and validation methods
  - Support for multiple solvers and tasks

- **`test_validation_pipeline.py`** - Tests for validation functions (14 tests)
  - Model organism validation workflow
  - Eval log parsing and error handling
  - Configuration parameter passing

- **`test_cli.py`** - Tests for CLI commands (17 tests)
  - validate-model command functionality
  - run-detector command functionality  
  - list subcommands (models, detectors, environments)
  - Help text and error handling

### End-to-End Tests
- **`test_e2e.py`** - Integration tests (17 tests)
  - Full CLI workflows with actual registry
  - Default model organism and task compatibility
  - Error propagation and graceful handling
  - Command chaining workflows

### Test Configuration
- **`conftest.py`** - Test fixtures and configuration
  - Mock objects for testing
  - Test markers and environment setup
  - Path configuration for imports

## Running Tests

### Run All Tests
```bash
python -m pytest tests/ -v
```

### Run Specific Test Categories
```bash
# Unit tests only
python -m pytest tests/test_*.py -v

# End-to-end tests only  
python -m pytest tests/test_e2e.py -v

# CLI tests only
python -m pytest tests/test_cli.py -v
```

### Run Tests with Coverage
```bash
python -m pytest tests/ --cov=_abstract --cov=_registry --cov-report=html
```

### Run Tests by Marker
```bash
# Run only unit tests
python -m pytest tests/ -m unit -v

# Run only integration tests
python -m pytest tests/ -m integration -v

# Run only e2e tests
python -m pytest tests/ -m e2e -v
```

## Test Coverage

The test suite provides comprehensive coverage of:

- ✅ Registry system functionality (listing, loading, error handling)
- ✅ Core object model (ModelOrganism, Detector classes)
- ✅ Validation pipeline (eval execution, result parsing)
- ✅ CLI interface (commands, arguments, help text)
- ✅ End-to-end workflows (full integration testing)
- ✅ Error handling and edge cases
- ✅ Default implementations compatibility

## Writing New Tests

### Test Naming Convention
- Test files: `test_<module_name>.py`
- Test classes: `Test<FeatureName>`
- Test methods: `test_<specific_behavior>`

### Using Fixtures
```python
def test_example_with_fixtures(mock_model_organism, mock_task_info):
    # Test implementation using provided fixtures
    pass
```

### Mocking External Dependencies
```python
@patch('_abstract.pipelines.validate_model.eval')
def test_validation_workflow(mock_eval):
    # Test implementation with mocked inspect_ai.eval
    pass
```

## Test Requirements

All tests are designed to:
- Run independently without external API calls
- Use mocking for expensive operations (model inference)
- Provide clear error messages and assertions
- Cover both success and failure scenarios
- Test integration between components 