# Performance Test Suite

This directory contains comprehensive performance, load testing, and memory validation tests for the MCA SDK.

## Test Modules

### 1. `load_test_sustained.py`
**Purpose:** Validate SDK stability under sustained high-throughput load.

**Tests:**
- `test_sustained_load_1000_rps_10_minutes` - 10-minute sustained load at 1,000 RPS
- `test_idle_cpu_usage` - Background thread CPU consumption when idle

**Acceptance Criteria:**
- Maintains 1,000+ requests/second for 10 minutes
- SDK overhead <50ms p95
- Zero telemetry drops (with healthy collector)
- CPU usage stable (no gradual increase)
- Idle CPU <1%

**Runtime:** ~10 minutes (full test)

### 2. `memory_leak_check.py`
**Purpose:** Detect memory leaks and validate long-duration memory stability.

**Tests:**
- `test_accelerated_aging_no_memory_leak` - 100 cycles with forced GC
- `test_specific_object_types_no_leak` - Object type leak detection (Span, Record, Queue)

**Acceptance Criteria:**
- Memory usage remains flat (no unbounded growth)
- Total growth <10MB after warmup
- No leaks in critical object types

**Runtime:** ~2-3 minutes (accelerated test)

### 3. `test_backpressure.py`
**Purpose:** Validate SDK resilience under backpressure scenarios.

**Tests:**
- `test_queue_respects_max_size` - Queue size limits
- `test_queue_drops_oldest_when_full` - FIFO eviction strategy
- `test_queue_non_blocking_enqueue` - Non-blocking enqueue operations
- `test_sdk_handles_slow_collector` - Slow collector handling
- `test_sdk_handles_offline_collector_no_memory_error` - Offline collector handling
- `test_backpressure_full_scenario` - Complete backpressure scenario
- `test_buffer_stats_tracking` - Buffer statistics accuracy

**Acceptance Criteria:**
- Queue size never exceeds max_queue_size
- Drops items correctly when full
- No MemoryError with offline collector
- Application never blocked by full queues

**Runtime:** ~30 seconds

### 4. `test_concurrency.py`
**Purpose:** Validate thread safety and concurrency correctness.

**Tests:**
- `test_concurrent_thread_safety` - 50 concurrent threads
- `test_async_concurrency` - 50 async coroutines
- `test_queue_concurrent_access` - Concurrent queue operations
- `test_counter_accuracy_under_concurrency` - Atomic counter updates
- `test_no_deadlock_under_load` - Deadlock detection

**Acceptance Criteria:**
- No race conditions or data corruption
- Metric counters are accurate
- No deadlocks under concurrent load

**Runtime:** ~30-40 seconds

## Running Tests

### Run All Performance Tests
```bash
pytest tests/performance/ -v -m performance
```

### Run Individual Test Suites
```bash
# Backpressure tests (fast)
pytest tests/performance/test_backpressure.py -v

# Concurrency tests (medium)
pytest tests/performance/test_concurrency.py -v

# Memory leak tests (slow)
pytest tests/performance/memory_leak_check.py -v -m memory

# Sustained load test (very slow - 10 minutes)
pytest tests/performance/load_test_sustained.py -v --timeout=720
```

### Run Specific Tests
```bash
# Quick concurrency validation
pytest tests/performance/test_concurrency.py::test_concurrent_thread_safety -v

# Object leak check only
pytest tests/performance/memory_leak_check.py::test_specific_object_types_no_leak -v
```

## Dependencies

Required packages:
- `psutil` - CPU and process monitoring
- `objgraph` - Object graph analysis for leak detection
- `pytest-asyncio` - Async test support
- `pytest-timeout` - Test timeout enforcement

Install with:
```bash
pip install psutil objgraph pytest-asyncio pytest-timeout
```

## Test Markers

- `@pytest.mark.performance` - Performance test
- `@pytest.mark.slow` - Long-running test (>30s)
- `@pytest.mark.memory` - Memory-related test

## Continuous Integration

### Fast Suite (CI)
```bash
pytest tests/performance/test_backpressure.py tests/performance/test_concurrency.py -v --timeout=300
```
**Runtime:** ~1-2 minutes

### Full Suite (Nightly)
```bash
pytest tests/performance/ -v -m "performance or memory" --timeout=900
```
**Runtime:** ~15-20 minutes (includes 10-minute sustained load test)

## Interpreting Results

### Latency Metrics
- **p50:** Median latency (50th percentile)
- **p95:** 95th percentile latency (5% of requests exceed this)
- **p99:** 99th percentile latency

**Targets:**
- p50 < 10ms (typical)
- p95 < 50ms (acceptable)
- p99 < 100ms (max tolerable)

### Memory Growth
- **Baseline:** Initial memory after warmup
- **Final:** Memory at end of test
- **Growth:** Difference (should be <10MB)
- **Per-iter:** Growth per iteration (should be <50 bytes)

### CPU Usage
- **Average:** Mean CPU % during test
- **Max:** Peak CPU %
- **Stable:** First half vs second half (should be <10% increase)

### Queue Metrics
- **Size:** Current items in queue
- **Utilization:** Percentage of max capacity
- **Drops:** Items dropped due to full queue

## Troubleshooting

### High Latency
- Check if collector is slow/offline
- Verify network latency to collector
- Check for resource contention (CPU, memory)

### Memory Leaks
- Run with `objgraph` to identify leak sources
- Check for circular references
- Verify resource cleanup (file handles, connections)

### Concurrency Issues
- Enable debug logging: `MCA_DEBUG_MODE=true`
- Check thread safety of custom integrations
- Verify proper locking around shared state

### Test Failures
1. **Timeout:** Increase `--timeout` value or reduce test duration
2. **Collector Offline:** Some tests expect offline collector (this is normal)
3. **High Variance:** Run tests on idle system to reduce noise

## Performance Baselines

From Story 1.13.4 optimization results:
- **Throughput:** ~700k RPS (microbenchmark)
- **Latency:** ~0.01ms p50, ~0.02ms p95
- **Memory:** Stable growth <50 bytes/iter with collector down

These tests validate **stability** at production load (1k RPS sustained), not peak performance.
