# Story 11.9: Enable Parallel Test Execution with pytest-xdist

**Status:** Ready
**Priority:** Medium
**Depends on:** Story 11.6 (Split Large Test Files)

---

## Context / Background

The test suite has ~1,600 tests that currently run sequentially. With pytest-xdist, tests can run in parallel across multiple CPU cores, significantly reducing total execution time.

**Current state:**
- Sequential execution: ~X minutes
- No parallelization configured
- Large test files limit distribution effectiveness

**Target state:**
- Parallel execution with `pytest -n auto`
- Tests distributed across available CPU cores
- 30-50% reduction in total CI time

**Why depends on 11.6:** pytest-xdist distributes tests by file. Large files (1,000+ lines) become bottlenecks because all tests in one file run on one worker. Splitting files first maximizes parallelization benefit.

---

## Scope (In / Out)

### In Scope

- Install and configure pytest-xdist
- Identify and fix tests with shared state issues
- Add `@pytest.mark.no_parallel` for tests that can't parallelize
- Update CI to use parallel execution
- Document parallel testing guidelines

### Out of Scope

- Rewriting tests (just marking non-parallelizable ones)
- Changing test logic beyond isolation fixes
- Distributed testing across machines (just local parallelization)

---

## Acceptance Criteria

### AC1: pytest-xdist Installed and Configured

**Description:** pytest-xdist is available and can be invoked.

**Validation:**
```bash
pytest tests/ -n auto --collect-only
# Shows test distribution across workers
```

### AC2: Tests Pass in Parallel

**Description:** Full test suite passes with parallel execution.

**Validation:**
```bash
pytest tests/ -n auto -v
# All tests pass
```

### AC3: Shared State Issues Resolved

**Description:** Tests don't interfere with each other when run in parallel.

**Validation:**
- No test failures due to shared files/ports/globals
- Tests that modify global state are marked `@pytest.mark.no_parallel`

### AC4: CI Uses Parallel Execution

**Description:** CI workflow runs tests with `-n auto`.

**Validation:**
```yaml
# .github/workflows/ci.yml
- run: hatch run test:test -- -n auto --dist=loadfile
```

### AC5: Execution Time Reduced

**Description:** CI test execution time reduced by 30%+.

**Validation:**
```bash
# Before: X minutes
# After: < 0.7 * X minutes
```

---

## Implementation Notes

### Installation

Add to Hatch test environment dependencies:

```toml
# pyproject.toml
[tool.hatch.envs.test]
dependencies = [
    "pytest-xdist>=3.0.0",
    # ... existing deps ...
]
```

```ini
# tox.ini
deps =
    pytest-xdist>=3.0.0
    ...
```

### Configuration

```ini
# pyproject.toml
[tool.pytest.ini_options]
addopts = "-n auto --dist=loadfile"
```

Options:
- `-n auto` - Use all available CPU cores
- `-n 4` - Use exactly 4 workers
- `--dist=loadfile` - Run all tests in a file on same worker (better for fixtures)
- `--dist=load` - Distribute individual tests (more granular)

### Identifying Shared State Issues

Common problems and fixes:

#### 1. Shared Files

```python
# Problem: Multiple tests write to same file
def test_a():
    with open("/tmp/test.log", "w") as f:
        f.write("a")

def test_b():
    with open("/tmp/test.log", "w") as f:
        f.write("b")

# Fix: Use unique file names
def test_a(tmp_path):
    path = tmp_path / "test.log"
    ...
```

#### 2. Shared Ports

```python
# Problem: Multiple tests use same port
def test_server_a():
    server = start_server(port=8080)

def test_server_b():
    server = start_server(port=8080)  # Port conflict!

# Fix: Use dynamic ports
def test_server(unused_tcp_port):
    server = start_server(port=unused_tcp_port)
```

#### 3. Global State

```python
# Problem: Test modifies global
import my_module

def test_a():
    my_module.CONFIG = {"debug": True}

def test_b():
    assert my_module.CONFIG == {}  # Fails if test_a runs first

# Fix: Reset in fixture or mark as no_parallel
@pytest.fixture(autouse=True)
def reset_config():
    original = my_module.CONFIG.copy()
    yield
    my_module.CONFIG = original
```

#### 4. Database State

```python
# Problem: Tests share database
def test_create_user():
    db.create_user("alice")

def test_list_users():
    users = db.list_users()
    assert len(users) == 0  # Fails if test_create_user runs first

# Fix: Isolate with transactions
@pytest.fixture
def db_session():
    with db.transaction() as session:
        yield session
        session.rollback()
```

### Marking Non-Parallelizable Tests

For tests that truly can't run in parallel:

```python
@pytest.mark.no_parallel
def test_modifies_global_singleton():
    ...
```

Configure pytest to respect the marker:

```python
# conftest.py (root level)
def pytest_configure(config):
    config.addinivalue_line(
        "markers", "no_parallel: mark test as not safe for parallel execution"
    )
```

Run non-parallel tests separately:

```bash
# Run parallel tests first
hatch run test:test -- -n auto -m "not no_parallel"

# Then run serial tests
hatch run test:test -- -m "no_parallel"
```

### CI Configuration

```yaml
# .github/workflows/ci.yml
jobs:
  test:
    steps:
      - name: Run parallel tests
        run: hatch run test:test -- -n auto -m "not no_parallel" --dist=loadfile

      - name: Run serial tests
        run: hatch run test:test -- -m "no_parallel"
```

### File Changes

```
pyproject.toml (MODIFIED - add pytest-xdist to [tool.hatch.envs.test], configure addopts)
tox.ini (MODIFIED - add pytest-xdist, update commands)
conftest.py (MODIFIED - add no_parallel marker)
.github/workflows/ci.yml (MODIFIED - use parallel execution)
docs/contributing/test-categories.md (MODIFIED - document parallel testing)
```

---

## Tasks

### Phase 1: Setup

- [ ] Add pytest-xdist to dependencies
- [ ] Add `no_parallel` marker definition
- [ ] Configure pytest defaults

### Phase 2: Identify Issues

- [ ] Run tests with `-n 4` and note failures
- [ ] Categorize failures: shared files / ports / globals / other
- [ ] Create fix list for each category

### Phase 3: Fix Isolation Issues

- [ ] Fix shared file issues (use tmp_path)
- [ ] Fix shared port issues (use unused_tcp_port)
- [ ] Fix global state issues (add reset fixtures)
- [ ] Mark remaining issues as `@pytest.mark.no_parallel`

### Phase 4: CI Integration

- [ ] Update GitHub Actions workflow
- [ ] Update tox.ini
- [ ] Verify CI passes with parallel execution

### Phase 5: Documentation

- [ ] Document parallel testing guidelines
- [ ] Add CHANGELOG entry
- [ ] Document `@pytest.mark.no_parallel` usage

---

## Tests

### Parallel Execution Test

```bash
# Verify parallel execution works
pytest tests/ -n 4 -v

# Verify with different worker counts
pytest tests/ -n 2 -v
pytest tests/ -n auto -v
```

### Stability Test

```bash
# Run multiple times to catch race conditions
for i in {1..5}; do
    pytest tests/ -n auto || exit 1
done
```

---

## Definition of Done

### Code Complete

- [ ] pytest-xdist installed and configured
- [ ] All parallel-unsafe tests marked
- [ ] Shared state issues fixed or marked
- [ ] CI uses parallel execution

### Quality Assurance

- [ ] Tests pass with `-n auto`
- [ ] Tests pass with `-n 1` (sequential still works)
- [ ] No new flaky tests introduced
- [ ] Execution time reduced 30%+

### Documentation

- [ ] Parallel testing guidelines documented
- [ ] `@pytest.mark.no_parallel` usage documented
- [ ] CHANGELOG updated

---

## Risks / Rollback

### Risks

1. **Risk:** Hidden shared state causes intermittent failures
   - **Mitigation:** Run parallel tests multiple times before merging

2. **Risk:** Some tests fundamentally can't parallelize
   - **Mitigation:** Mark with `@pytest.mark.no_parallel`, run separately

3. **Risk:** Resource contention (CPU, memory) on CI
   - **Mitigation:** Limit workers: `-n 4` instead of `-n auto`

### Rollback Plan

If issues occur:
1. Remove `-n auto` from CI, revert to sequential
2. Keep pytest-xdist installed for local use
3. Investigate specific failures

---

## Related Stories

- **Depends on:** Story 11.6 - smaller files parallelize better
- **Related:** Story 11.4 - timeout multipliers may need adjustment for parallel

---

## Change Log

| Date | Change | Author |
|------|--------|--------|
| 2026-01-14 | Initial draft | Claude |
| 2026-01-14 | Fix paths: ci.yml not test.yml, conftest.py at root; use Hatch commands | Claude |
