---
alwaysApply: true
---
# Concurry Testing Practices

## Test Timeout Configuration

**ALL tests have a 60-second timeout by default** to prevent hanging tests.

- **Configured in**: `tests/conftest.py` via `pytest_configure` hook
- **Default**: 60 seconds per test
- **Method**: Thread-based timeout (compatible with Ray and multiprocessing)
- **Override**: Use `pytest --timeout=120` to change timeout
- **Disable**: Use `pytest --timeout=0` to disable timeout
- **On Timeout**: Full stack trace is displayed for debugging
- **Behavior**: Timeout marks test as **FAILED** but **continues to next test** (non-fatal)

**Why 60 seconds?**: Most tests should complete in < 10 seconds. The 60-second timeout catches hanging tests (deadlocks, infinite loops, semaphore issues) while allowing slower integration tests to complete.

**Important**: Timeouts do NOT stop the entire test suite! When a test times out:
1. The test is marked as **FAILED** with timeout information
2. Full stack traces are displayed showing where the hang occurred
3. The test runner **continues to the next test**
4. All remaining tests will still run

This allows you to identify multiple hanging tests in a single test run.

**Example timeout error output**:
```
++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++
~~~~ Stack of MainThread (140735268369408) ~~~~
File "/path/to/test.py", line 123, in test_something
    result = future.result()
File "/path/to/future.py", line 456, in result
    self._wait()
```

## Running Tests

**NEVER use `-q` (quiet mode) or `--tb=line` when debugging issues**, as these suppress important output.

**Recommended pytest commands**:
- `pytest tests/file.py -vs` - Verbose, no capture (best for debugging multiple tests)
- `pytest tests/file.py -xvs` - Stop on first failure, verbose, no capture (best for debugging single issue)
- `pytest tests/file.py -v` - Verbose output showing all test names
- `pytest tests/file.py --tb=short` - Short traceback format (good for CI)

**Important about `-x` flag**:
- **WITH `-x`**: Stops on **first failure** (including timeouts) - use when debugging one specific issue
- **WITHOUT `-x`**: Continues after failures/timeouts - use to identify **all** failing tests in one run

**AVOID**:
- `pytest -q` - Suppresses output, makes debugging impossible
- `pytest --tb=line` - Only shows one line per failure, hides context
- `pytest -qq` - Ultra-quiet mode, completely useless for debugging

**For timeout issues**: Run without `-x` to see all timeouts in one run:
```bash
pytest tests/file.py -v  # Will show ALL tests that timeout
```

## Pytest Fixtures from conftest.py

The [tests/conftest.py](mdc:tests/conftest.py) file provides essential fixtures for testing across all execution modes. **Always use these fixtures** when writing tests.

### Available Fixtures

#### 1. `worker_mode` Fixture

**Purpose**: Parametrize tests across ALL execution modes (sync, thread, process, asyncio, ray)

**Usage**:
```python
def test_my_feature(self, worker_mode):
    """Test feature across all execution modes."""
    w = MyWorker.options(mode=worker_mode).init(param=value)
    
    # Your test logic
    result = w.method().result()
    assert result == expected
    
    w.stop()
```

**Important**: This fixture automatically runs your test 5 times (once per mode). If Ray is not installed, it runs 4 times.

#### 2. `pool_mode` Fixture

**Purpose**: Parametrize tests across pool-supporting modes (thread, process, ray)

**Usage**:
```python
def test_pool_feature(self, pool_mode):
    """Test feature that requires multiple workers."""
    w = MyWorker.options(mode=pool_mode, max_workers=3).init()
    
    # Your test logic for pools
    futures = [w.task(i) for i in range(10)]
    results = [f.result() for f in futures]
    
    w.stop()
```

**Why separate from worker_mode?**: Sync and asyncio modes only support `max_workers=1`, so pool-specific features cannot be tested with them.

#### 3. `initialize_ray` Fixture

**Purpose**: Session-level fixture that initializes Ray once before all tests

**Usage**: Automatic - no need to explicitly use this fixture. Ray is initialized at session start if available.

**Important**: Always include the runtime_env when manually initializing Ray in tests:
```python
import ray
import morphic
import concurry

ray.init(
    ignore_reinit_error=True,
    num_cpus=4,
    runtime_env={"py_modules": [concurry, morphic]},
)
```

### WORKER_MODES and POOL_MODES Constants

Available for direct use when needed:
```python
from tests.conftest import WORKER_MODES, POOL_MODES

# WORKER_MODES = ["sync", "thread", "process", "asyncio", "ray"]  # if Ray installed
# POOL_MODES = ["thread", "process", "ray"]  # if Ray installed
```

---

## Comprehensive Execution Mode Testing

### Golden Rule: Test ALL Execution Modes

**Every test must run across all applicable execution modes using the `worker_mode` or `pool_mode` fixture.**

❌ **NEVER write tests like this:**
```python
def test_feature(self):
    """Bad: Only tests sync mode."""
    worker = MyWorker.options(mode="sync").init()
    # ...
```

✅ **ALWAYS write tests like this:**
```python
def test_feature(self, worker_mode):
    """Good: Tests all execution modes."""
    worker = MyWorker.options(mode=worker_mode).init()
    # ...
```

### Example: Basic Test Structure

```python
def test_basic_computation(self, worker_mode):
    """Test basic computation across all execution modes."""
    w = ComputeWorker.options(mode=worker_mode).init(multiplier=2)
    result = w.compute(5).result(timeout=5.0)
    assert result == 10
    w.stop()
```

### Example: Pool-Specific Test

```python
def test_pool_feature(self, pool_mode):
    """Test feature requiring multiple workers."""
    if pool_mode in ("sync", "asyncio"):
        pytest.skip("Sync and asyncio only support max_workers=1")
    
    w = MyWorker.options(mode=pool_mode, max_workers=3).init()
    # Test pool-specific functionality
    w.stop()
```

---

## Handling Mode-Specific Behavior

### NEVER Skip Tests Due to Failures

**Critical Rule**: If a test fails for certain execution modes, **DO NOT skip it**. These are important edge cases that must be handled.

❌ **WRONG Approach:**
```python
def test_feature(self, worker_mode):
    if worker_mode == "ray":
        pytest.skip("Fails on Ray, skipping")  # ❌ NEVER DO THIS
    
    # test logic
```

✅ **CORRECT Approach - Handle the Edge Case:**
```python
def test_feature(self, worker_mode):
    """Test feature with mode-specific behavior."""
    w = MyWorker.options(mode=worker_mode).init()
    
    if worker_mode == "ray":
        # Ray has different behavior - test it explicitly
        with pytest.raises(SpecificException, match="expected message"):
            w.problematic_method()
    else:
        # Other modes work normally
        result = w.problematic_method().result()
        assert result == expected
    
    w.stop()
```

### Valid Reasons to Skip Tests

**Only skip when a feature is fundamentally not supported by a mode:**

```python
# Pool features not supported by sync/asyncio
def test_multiple_workers(self, worker_mode):
    if worker_mode in ("sync", "asyncio"):
        pytest.skip("Sync and asyncio only support max_workers=1")
    # Feature genuinely requires multiple workers

# Timeout not testable in sync mode
def test_timeout(self, worker_mode):
    if worker_mode == "sync":
        pytest.skip("Sync mode completes immediately")
    # Test timeout behavior

# Mode-specific features
def test_ray_actor_options(self, worker_mode):
    if worker_mode != "ray":
        pytest.skip("Ray-specific feature")
    # Test Ray-only functionality
```

### Handling Mode-Specific Exceptions

When different modes raise different exceptions, test all variants:

```python
def test_error_handling(self, worker_mode):
    w = MyWorker.options(mode=worker_mode).init()
    
    if worker_mode in ("sync", "ray"):
        with pytest.raises(AttributeError):
            w.nonexistent_method()  # Fails immediately
    else:
        future = w.nonexistent_method()  # Returns future
        with pytest.raises(AttributeError):
            future.result(timeout=5.0)
    
    w.stop()
```

---

## Test Organization Best Practices

### 1. Group Tests by Feature

```python
class TestBasicFeatures:
    """Test basic worker functionality."""
    def test_initialization(self, worker_mode): ...
    def test_method_call(self, worker_mode): ...

class TestPoolFeatures:
    """Test worker pool features."""
    def test_load_balancing(self, pool_mode): ...
```

### 2. Use Descriptive Test Names

```python
# ✅ Good: Clear what is being tested
def test_wait_returns_done_and_not_done_sets(self, worker_mode): ...
def test_gather_preserves_input_order(self, worker_mode): ...

# ❌ Bad: Unclear
def test_wait(self, worker_mode): ...
def test_gather_works(self, worker_mode): ...
```

### 3. Always Clean Up Resources

```python
def test_feature(self, worker_mode):
    w = MyWorker.options(mode=worker_mode).init()
    result = w.method().result()
    assert result == expected
    w.stop()  # Pytest calls this even if assertion fails
```

---

## Common Testing Patterns

```python
# Pattern 1: Synchronization primitives
def test_wait_and_gather(self, worker_mode):
    w = MyWorker.options(mode=worker_mode).init()
    futures = [w.compute(i) for i in range(10)]
    done, not_done = wait(futures, timeout=5.0)
    results = gather(futures, timeout=5.0)
    w.stop()

# Pattern 2: Exception handling
def test_gather_with_exceptions(self, worker_mode):
    w = MyWorker.options(mode=worker_mode).init()
    futures = [w.compute(1), w.failing_method(), w.compute(3)]
    results = gather(futures, return_exceptions=True, timeout=5.0)
    assert isinstance(results[1], ValueError)
    w.stop()

# Pattern 3: Timeouts
def test_timeout(self, worker_mode):
    if worker_mode == "sync":
        pytest.skip("Sync mode completes immediately")
    w = MyWorker.options(mode=worker_mode).init()
    with pytest.raises(TimeoutError):
        w.slow_task(duration=5.0).result(timeout=0.1)
    w.stop()

# Pattern 4: Edge cases - empty, single, mixed types
def test_edge_cases(self, worker_mode):
    assert len(wait([])[0]) == 0  # Empty
    assert len(wait(single_future)[0]) == 1  # Single
    assert gather([future, 42, future2]) == [r1, 42, r2]  # Mixed
```

## Ray-Specific Testing

```python
# Always include runtime_env when manually initializing
import ray, morphic, concurry
ray.init(
    ignore_reinit_error=True,
    num_cpus=4,
    runtime_env={"py_modules": [concurry, morphic]}
)

# Test Ray-specific features
@pytest.mark.skipif(not _IS_RAY_INSTALLED, reason="Ray not installed")
def test_ray_actor_options(self):
    w = MyWorker.options(
        mode="ray",
        actor_options={"num_cpus": 1}
    ).init()
    w.stop()
```

---

## Fixing Failing Tests: NEVER Skip, Always Fix

### Critical Rule: Fix Implementation, Don't Skip Tests

When an existing test fails, the problem is usually the implementation, not the test. Your responsibility:
1. Identify the root cause
2. Fix the implementation to make the test pass
3. NEVER skip or comment out failing tests

### Decision Tree for Failing Tests

```
Test fails
   ├── Test wrong/outdated? → Ask user before modifying
   ├── NEW test you wrote? → Fix your test or implementation
   ├── Small fix (<50 lines, single file)? → Fix autonomously
   └── Large fix (multiple files, architectural)? → STOP and ask user
```

### When to Fix Autonomously vs. Ask User

**Small Fix (Do it)**: Off-by-one errors, missing checks, wrong operators, typos, missing imports, wrong exception types

**Large Fix (Ask user)**: New classes, API changes, 3+ files affected, architectural decisions, unclear behavior, 100+ lines of new logic

**Test Might Be Wrong (Ask user)**: Test expects value X, implementation returns Y, unclear which is correct

### Examples

✅ **Autonomous Fix**: "Test expects 3 retries but gets 2. Fixed off-by-one error in retry.py line 145."

✅ **Stop and Ask**: "Test fails due to unpicklable Lock in LimitPool. Fix requires refactoring to use multiprocessing.Manager() across 5 files. Proceed?"

✅ **Ask About Test**: "Test expects timeout=5.0 but config has 10.0. Which is correct?"

❌ **NEVER Skip**: `pytest.skip("Test fails for process mode")` - This hides bugs!

❌ **NEVER Comment Out**: `# def test_feature():` - This breaks functionality silently!

❌ **NEVER Change Test Without Understanding**: Changing assertions to match wrong behavior hides bugs!

### Acceptable Reasons to Skip Tests

1. **Feature not supported by mode**: `max_workers > 1` not supported by sync/asyncio
2. **External dependency missing**: Ray not installed, OS feature unavailable
3. **User explicitly requested**: Expensive/long-running tests temporarily disabled

### Debugging Strategy

1. Read error message: What failed? Expected vs. actual?
2. Understand test intent: What feature? What behavior?
3. Trace execution: Where does it diverge?
4. Identify root cause: Logic error? Config? Race condition?
5. Determine complexity: Small fix (do it) or large fix (ask)?
6. Implement and verify: Fix, test, check regressions

### Common Test Failure Patterns

**Timing/Race**: Test flaky → Add synchronization or increase timeout
**Mode-Specific**: Fails some modes → Fix implementation, not test
**Config Changes**: Expects old default → Ask user which is correct
**Missing Feature**: Method doesn't exist → Implement or ask about priority

---

## Summary Checklist

✅ Use `worker_mode` or `pool_mode` fixtures to test all execution modes
✅ Fix failing tests by fixing implementation, never skip to hide bugs
✅ Small fixes (<50 lines): do autonomously; large fixes: ask user
✅ Always clean up with `w.stop()`; use descriptive test names
✅ Test edge cases (empty, single, mixed), exceptions, timeouts
✅ Include Ray runtime_env: `runtime_env={"py_modules": [concurry, morphic]}`

## Anti-Patterns to Avoid

❌ Hardcoding mode: `mode="thread"` (use `worker_mode` fixture)
❌ Skipping failing tests to hide bugs
❌ Not cleaning up: missing `w.stop()`
❌ Unclear test names: `test_1()` instead of `test_feature_behavior()`
❌ Testing only happy path without exception handling
