# Revised Test Strategy: Test Real Systems

## What We're NOT Doing
- ❌ Building InMemoryProvider (fake system)
- ❌ Creating elaborate test doubles
- ❌ Mocking infrastructure

## What We ARE Doing

### 1. Real Provider in Test Mode
```python
# tests/testing/providers.py
def create_test_provider() -> FCPProvider:
    """Create real provider configured for testing."""
    return FCPProvider(
        api_key=os.environ.get("FCP_TEST_API_KEY"),
        api_url=os.environ.get("FCP_TEST_URL", "https://sandbox.api.mlfoundry.com"),
        project="test-" + str(uuid.uuid4())[:8],
        cleanup_on_exit=True
    )
```

### 2. API Recording for Offline Tests
```python
# tests/testing/recorder.py
import vcr

test_vcr = vcr.VCR(
    cassette_library_dir='tests/cassettes',
    filter_headers=['authorization'],
    filter_post_data_parameters=['api_key'],
    match_on=['method', 'path', 'query'],
    record_mode='once'
)

# Usage:
@test_vcr.use_cassette()
def test_gpu_allocation():
    provider = create_test_provider()
    # First run: records real API calls
    # Subsequent runs: replays from cassette
```

### 3. Container-Based Test Infrastructure
```bash
#!/bin/bash
# scripts/test-infra.sh

# Start local FCP sandbox
docker run -d --name fcp-sandbox \
  -p 8080:8080 \
  -e SANDBOX_MODE=true \
  fcp/sandbox:latest

# Wait for readiness
until curl -s http://localhost:8080/health; do
  sleep 1
done

# Run tests against local sandbox
FLOW_API_URL=http://localhost:8080 pytest tests/integration/

# Cleanup
docker stop fcp-sandbox && docker rm fcp-sandbox
```

### 4. Test Data Generators (Still Valuable)
```python
# Keep the builders, but for generating real test data
config = (TaskConfigBuilder()
    .with_gpu("a100-80gb")
    .with_region("us-east-1-test")  # Test region
    .build())

# Submit to real test infrastructure
task = test_provider.submit_task(config)
```

### 5. Performance Benchmarks Against Real System
```python
@pytest.mark.benchmark
def test_real_submission_performance(benchmark, test_provider):
    """Benchmark actual API performance."""
    config = TaskConfigBuilder().build()
    
    # Measure real API latency
    result = benchmark(test_provider.submit_task, config)
    
    # Assert performance SLAs
    assert benchmark.stats["mean"] < 0.5  # 500ms p50
    assert benchmark.stats["max"] < 2.0   # 2s p99
```

## Implementation Priority

1. **Set up test credentials and sandbox access**
2. **Create simple test utilities (not fake providers)**
3. **Set up VCR for API recording**
4. **Keep builders for test data generation**
5. **Write tests against real infrastructure**

## Example: Rewritten Test

```python
class TestRealTaskSubmission:
    """Test task submission against real infrastructure."""
    
    @pytest.fixture
    def provider(self):
        """Real provider in test mode."""
        return create_test_provider()
    
    @test_vcr.use_cassette()
    def test_gpu_task_submission(self, provider):
        """Submit real task to test infrastructure."""
        # Arrange - real test data
        config = (TaskConfigBuilder()
            .with_name("test-gpu-allocation")
            .with_gpu("a100-40gb")
            .with_script("nvidia-smi")
            .build())
        
        # Act - real API call (recorded)
        task = provider.submit_task(config)
        
        # Assert - real behavior
        assert task.task_id.startswith("task-")
        assert task.status in [TaskStatus.PENDING, TaskStatus.QUEUED]
        
        # Cleanup happens automatically in test mode
```

## This is What They Would Do

- **Carmack**: "Test the actual system. Anything else is lying to yourself."
- **Dean**: "We test MapReduce on real clusters, not fake ones."
- **Jobs**: "Delete the fake provider. Test the real one."
- **Page**: "Does this make us 10x more confident? No? Delete it."