# Flow SDK Test Patterns

This document describes the patterns and principles for writing tests in the Flow SDK.

## Core Philosophy

**Test real systems, not mocks.**

What would Jeff Dean, John Carmack, and the other masters do? They would test against real systems with real behavior, not elaborate mock setups.

## Test Categories

### 1. Unit Tests
Test OUR code logic in isolation. Mock only at system boundaries (HTTP, filesystem).

```python
class TestOurLogic:
    def test_parse_response(self):
        """Test that WE parse responses correctly."""
        # Mock ONLY the HTTP response
        mock_http.get.return_value = {"data": "..."}
        
        # Test OUR parsing logic
        result = our_class.parse_response()
        assert result.field == expected_value
```

### 2. Integration Tests  
Test against REAL external systems. No mocks.

```python
@pytest.mark.integration
class TestRealIntegration:
    def test_real_api_call(self, real_provider):
        """Test actual API integration."""
        # Real API call
        result = real_provider.do_something()
        
        # Verify real behavior
        assert result.id  # Real IDs exist
        assert result.status in VALID_STATUSES
```

### 3. Contract Tests
Verify our assumptions about external APIs.

```python
@pytest.mark.integration
class TestAPIContract:
    def test_response_format(self, real_provider):
        """Verify API response matches our expectations."""
        response = real_provider.get_instances()
        
        # Verify contract
        for instance in response:
            assert "id" in instance
            assert "price" in instance
            assert isinstance(instance["price"], (int, float))
```

## Test Patterns

### Pattern 1: Test Data Builders

Use builders instead of hardcoded test data:

```python
# BAD - Hardcoded
task = Task(
    task_id="task-123",
    name="test-task",
    status="running"
)

# GOOD - Builder
task = TaskBuilder().with_name("test-task").running().build()
```

### Pattern 2: Parameterized Tests

Test multiple scenarios systematically:

```python
@pytest.mark.parametrize("gpu_type,expected_memory", [
    ("a100-40gb", 40),
    ("a100-80gb", 80),
    ("h100-80gb", 80),
])
def test_gpu_memory(gpu_type, expected_memory):
    instance = provider.get_instance(gpu_type)
    assert instance.memory_gb == expected_memory
```

### Pattern 3: Property-Based Testing

Find edge cases automatically:

```python
@given(task_config())
def test_task_submission_always_returns_id(config):
    """Property: Every valid config produces a task with ID."""
    task = provider.submit_task(config)
    assert task.task_id is not None
    assert len(task.task_id) > 0
```

### Pattern 4: Test Isolation

Each test runs in isolation:

```python
def test_with_isolation(provider):
    with test_isolation(provider) as context:
        # Create resources
        task = provider.submit_task(config)
        context["created_resources"].append(("task", task.task_id))
        
        # Test behavior
        assert task.status == "running"
        
    # Automatic cleanup happens here
```

### Pattern 5: Real Metrics

Measure actual performance:

```python
def test_api_performance(provider, metrics):
    with metrics.measure("submit_task"):
        task = provider.submit_task(config)
    
    # Verify SLAs
    assert metrics.durations["submit_task"][0] < 1.0  # Under 1 second
```

## Anti-Patterns to Avoid

### ❌ Don't Mock Everything

```python
# BAD
mock_provider = Mock()
mock_provider.submit_task.return_value = Mock(task_id="123")
```

### ❌ Don't Test Mocks

```python
# BAD
assert mock_provider.submit_task.called_once_with(config)
```

### ❌ Don't Hardcode Test Data

```python
# BAD
assert task.price == 25.60  # What if prices change?

# GOOD
assert task.price > 0
assert task.price <= config.max_price
```

### ❌ Don't Mix Concerns

```python
# BAD - Unit test making real API calls
def test_parser():
    response = requests.get("https://api.example.com")  # NO!
    
# GOOD - Mock at boundary
def test_parser():
    mock_http.get.return_value = {"data": "..."}
```

## Test Organization

```
tests/
├── unit/                    # Our code logic
│   ├── test_*_unit.py      # *_unit suffix for clarity
│   └── test_*.py           # Pure logic tests
├── integration/            # Real API tests
│   ├── test_*_integration.py
│   └── test_*.py
├── contract/              # API contract tests
│   └── test_*_contract.py
├── performance/           # Performance benchmarks
│   └── test_*_perf.py
└── testing/              # Test infrastructure
    ├── builders.py       # Test data builders
    ├── providers.py      # Real provider utilities
    ├── generators.py     # Property-based generators
    └── fixtures.py       # Shared fixtures
```

## Running Tests

```bash
# Unit tests only (fast, no network)
pytest tests/unit -m "not integration"

# Integration tests (requires credentials)
FCP_TEST_API_KEY=xxx pytest tests/integration -m integration

# All tests
pytest

# With coverage
pytest --cov=flow --cov-report=html

# With performance metrics
pytest tests/performance --benchmark-only
```

## Writing New Tests

1. **Decide the category**
   - Testing our logic? → Unit test
   - Testing integration? → Integration test
   - Testing assumptions? → Contract test

2. **Use appropriate patterns**
   - Builders for test data
   - Parameterization for variants
   - Property-based for edge cases

3. **Mock minimally**
   - Unit: Mock only HTTP/filesystem
   - Integration: No mocks
   - Contract: No mocks

4. **Test behavior, not implementation**
   - What should happen?
   - Not: Was this method called?

## Example: Complete Test Module

```python
"""Test task submission logic."""

import pytest
from hypothesis import given
from tests.testing import (
    TaskConfigBuilder,
    create_test_provider,
    task_config,
    test_isolation
)

class TestTaskSubmissionUnit:
    """Unit tests for task submission logic."""
    
    def test_validate_gpu_type(self):
        """Test GPU type validation logic."""
        # Test OUR validation
        provider = FCPProvider(config, mock_http)
        
        with pytest.raises(ValidationError):
            provider._validate_gpu_type("invalid-gpu")
    
    @given(task_config())
    def test_config_normalization(self, config):
        """Property: All configs are normalized consistently."""
        normalized = normalize_config(config)
        
        # Should be idempotent
        assert normalize_config(normalized) == normalized


@pytest.mark.integration
class TestTaskSubmissionIntegration:
    """Integration tests against real API."""
    
    @pytest.fixture
    def provider(self):
        return create_test_provider()
    
    def test_submit_real_task(self, provider):
        """Submit task to real infrastructure."""
        with test_isolation(provider) as ctx:
            config = TaskConfigBuilder().with_cpu().build()
            task = provider.submit_task(config)
            
            # Real task properties
            assert task.task_id  # Has real ID
            assert task.created_at  # Has timestamp
            
            # Wait for real execution
            wait_for_task_status(provider, task.task_id, "completed")
```

Remember: Every test should tell a story about how the system behaves.