# Story 11.3: Standardize Hypothesis Test Settings

**Status:** Complete
**Priority:** High
**Depends on:** None

---

## Context / Background

Hypothesis property-based tests have per-test `@settings(max_examples=...)` decorators that override the global `HYPOTHESIS_MAX_EXAMPLES` environment variable:

**Current configuration:**
```ini
# tox.ini
HYPOTHESIS_MAX_EXAMPLES = 100

# pyproject.toml
HYPOTHESIS_MAX_EXAMPLES = "100"
```

**Per-test overrides found:**

| File | Line | Setting |
|------|------|---------|
| `tests/property/test_redaction_properties.py` | 36 | `max_examples=200` |
| `tests/property/test_redaction_properties.py` | 53 | `max_examples=150` |
| `tests/property/test_filter_properties.py` | 23 | `max_examples=200` |
| `tests/property/test_filter_properties.py` | 42 | `max_examples=120` |
| `tests/property/test_serialization_properties.py` | 49 | `max_examples=200` |
| `tests/property/test_serialization_properties.py` | 57 | `max_examples=200` |
| `tests/property/test_enrichment_properties.py` | 35 | `max_examples=200` |

These overrides defeat the purpose of the environment variable and cause inconsistent CI behavior. Tests run with 120-200 examples instead of the intended 100, increasing CI time.

**Documentation issue:** The pattern documentation (`docs/patterns/property-based-tests.md`) itself promotes per-test overrides:
- Line 10: "use `@settings(max_examples=...)` to control runtime"
- Line 28: Example code uses `@settings(max_examples=200)`

This documentation needs to be updated as part of this story.

---

## Scope (In / Out)

### In Scope

- Remove all per-test `@settings(max_examples=...)` overrides from test files
- Fix `docs/patterns/property-based-tests.md` to not promote per-test overrides
- Ensure all Hypothesis tests respect `HYPOTHESIS_MAX_EXAMPLES` env var
- Document the expected workflow (local: high examples, CI: lower examples)

### Out of Scope

- Changing the default `HYPOTHESIS_MAX_EXAMPLES` value
- Adding new property-based tests
- Modifying other Hypothesis settings (deadline, database, etc.)

---

## Acceptance Criteria

### AC1: No Per-Test max_examples Overrides

**Description:** No test file contains `@settings(max_examples=...)`.

**Validation:**
```bash
grep -r "@settings(max_examples" tests/ | wc -l
# Expected: 0
```

### AC2: Environment Variable Controls Example Count

**Description:** All Hypothesis tests respect `HYPOTHESIS_MAX_EXAMPLES`.

**Validation:**
```bash
# Run with low example count
HYPOTHESIS_MAX_EXAMPLES=10 pytest tests/property/ -v --collect-only
# Verify no warnings about overridden settings

# Run with high example count for thorough local testing
HYPOTHESIS_MAX_EXAMPLES=500 pytest tests/property/ -v
```

### AC3: CI Uses Configured Value

**Description:** CI runs use the value from `tox.ini` (100 examples).

**Validation:**
- Check CI logs show `max_examples=100` (or whatever is configured)
- No test runs with more examples than configured

### AC4: Documentation Updated

**Description:** Property test guidelines document the example count strategy and don't promote per-test overrides.

**Validation:**
- `docs/patterns/property-based-tests.md` example code has no `max_examples` parameter
- Guidelines explain to use `HYPOTHESIS_MAX_EXAMPLES` env var, not per-test decorators
- Clear guidance on local vs CI example counts
```bash
grep -c "max_examples" docs/patterns/property-based-tests.md
# Expected: 0 (no per-test max_examples in example or guidelines)
```

---

## Implementation Notes

### Removing Overrides

Simply delete the `max_examples` parameter from `@settings()` decorators:

```python
# Before
@settings(max_examples=200)
@given(st.text())
def test_redaction_preserves_length(text):
    ...

# After
@given(st.text())
def test_redaction_preserves_length(text):
    ...
```

If other settings are needed (like `deadline`), keep the decorator but remove `max_examples`:

```python
# Before
@settings(max_examples=200, deadline=None)
@given(st.text())
def test_slow_operation(text):
    ...

# After
@settings(deadline=None)
@given(st.text())
def test_slow_operation(text):
    ...
```

### Profile-Based Alternative

If different example counts are needed for different scenarios, use Hypothesis profiles:

```python
# conftest.py
from hypothesis import settings, Verbosity

settings.register_profile("ci", max_examples=100)
settings.register_profile("dev", max_examples=50)
settings.register_profile("thorough", max_examples=500)

# Load profile from environment
import os
settings.load_profile(os.getenv("HYPOTHESIS_PROFILE", "dev"))
```

This is more flexible than per-test overrides and keeps control centralized.

### File Changes

```
tests/property/test_redaction_properties.py (MODIFIED - remove max_examples)
tests/property/test_filter_properties.py (MODIFIED - remove max_examples)
tests/property/test_serialization_properties.py (MODIFIED - remove max_examples)
tests/property/test_enrichment_properties.py (MODIFIED - remove max_examples)
docs/patterns/property-based-tests.md (MODIFIED - document strategy)
```

---

## Tasks

### Phase 1: Remove Overrides

- [ ] Remove `max_examples` from `test_redaction_properties.py` (2 places)
- [ ] Remove `max_examples` from `test_filter_properties.py` (2 places)
- [ ] Remove `max_examples` from `test_serialization_properties.py` (2 places)
- [ ] Remove `max_examples` from `test_enrichment_properties.py` (1 place)
- [ ] Verify no other files have overrides

### Phase 2: Verify Behavior

- [ ] Run tests with `HYPOTHESIS_MAX_EXAMPLES=10` to verify low count works
- [ ] Run tests with `HYPOTHESIS_MAX_EXAMPLES=200` to verify high count works
- [ ] Verify tox uses the configured value

### Phase 3: Fix Documentation

- [ ] Remove `@settings(max_examples=200)` from example code at line 28
- [ ] Change line 10 from "use `@settings(max_examples=...)` to control runtime" to "CI controls example count via `HYPOTHESIS_MAX_EXAMPLES` env var (see `tox.ini`)"
- [ ] Add "Running Locally" guidance showing how to use env var:
  ```bash
  # Default (uses tox.ini value of 100):
  pytest -m property

  # Thorough local testing:
  HYPOTHESIS_MAX_EXAMPLES=500 pytest -m property
  ```
- [ ] Add note prohibiting per-test `max_examples` overrides

---

## Tests

### Verification

```bash
# Verify environment variable is respected
HYPOTHESIS_MAX_EXAMPLES=5 pytest tests/property/test_redaction_properties.py -v 2>&1 | grep "5 examples"

# Verify all property tests run
pytest tests/property/ -v
```

### Regression

```bash
# Full property test suite
pytest tests/property/ -v

# Ensure no new failures
pytest tests/ -m "not slow and not integration"
```

---

## Definition of Done

### Code Complete

- [ ] Zero `@settings(max_examples=...)` in test files
- [ ] All Hypothesis tests respect environment variable
- [ ] Empty `@settings()` decorators removed (if no other settings)

### Quality Assurance

- [ ] Property tests pass with default settings
- [ ] Property tests pass with `HYPOTHESIS_MAX_EXAMPLES=10`
- [ ] `ruff check` passes
- [ ] No regression in existing tests

### Documentation

- [ ] `docs/patterns/property-based-tests.md` updated
- [ ] CHANGELOG updated

---

## Risks / Rollback

### Risks

1. **Risk:** Some tests need more examples to catch edge cases
   - **Mitigation:** Run with high example count in nightly builds
   - **Mitigation:** If a test truly needs more examples, document why

2. **Risk:** Reducing examples might miss bugs
   - **Mitigation:** 100 examples is still substantial for CI
   - **Mitigation:** Hypothesis shrinking finds minimal failing cases

### Rollback Plan

If issues occur:
1. Re-add `@settings(max_examples=...)` to specific tests that need it
2. Document the exception and why it's needed
3. Consider profile-based approach for nuanced control

---

## Related Stories

- **Related:** Story 7.6 - Property-Based Serialization Tests (uses these settings)
- **Related:** Story 11.4 - CI configuration patterns

---

## Change Log

| Date | Change | Author |
|------|--------|--------|
| 2026-01-14 | Initial draft | Claude |
| 2026-01-14 | Add documentation fix tasks - pattern doc promotes per-test overrides | Claude |
