# RedenLab Extract SDK - Testing Plan

## Overview

This document outlines the testing strategy for the RedenLab Extract SDK. The goal is to cover **critical paths and user-facing functionality** without aiming for 100% coverage. Tests are prioritized by impact on users and likelihood of bugs.

## Architecture Summary

```
User Application
        │
        ▼
┌─────────────────────────────────────────────────────┐
│  PUBLIC API                                          │
│  - RedenLabSession (credentials, uploads)            │
│  - InferenceClient / TranscribeClient (inference)    │
│  - intelligibility_client / naturalness_client       │
│  - run_inference / run_chunked_inference / aggregate │
└─────────────────────────────────────────────────────┘
        │
        ▼
┌─────────────────────────────────────────────────────┐
│  INTERNAL MODULES                                    │
│  - api.py (HTTP communication)                       │
│  - polling.py (job status polling)                   │
│  - upload.py (S3 presigned URL uploads)              │
│  - config.py (configuration loading)                 │
│  - payload.py (model-specific payloads)              │
└─────────────────────────────────────────────────────┘
```

---

## Test Categories

| Category | Purpose | Location |
|----------|---------|----------|
| **Unit Tests** | Test individual modules in isolation | `tests/unit/` |
| **Integration Tests** | Test real API calls (requires credentials) | `tests/integration/` |

---

## Priority Levels

- **P0 (Critical)**: Core user workflows - must have tests
- **P1 (High)**: Important functionality - should have tests
- **P2 (Medium)**: Supporting functionality - nice to have
- **P3 (Low)**: Edge cases, internal utilities - optional

---

## Test Plan by Module

### 1. Session Management (`session.py`) - P0

The session is the entry point for all SDK operations.

#### Tests Required

| Test | Priority | Description |
|------|----------|-------------|
| `test_session_init_with_explicit_credentials` | P0 | Session initializes with api_key and base_url |
| `test_session_init_missing_api_key_raises` | P0 | AuthenticationError when no API key found |
| `test_session_init_missing_base_url_raises` | P0 | ConfigurationError when no base_url found |
| `test_session_init_from_env_vars` | P1 | Session loads credentials from environment |
| `test_session_init_from_config_file` | P2 | Session loads from ~/.redenlab-ml/config.yaml |
| `test_upload_returns_asset` | P0 | upload() returns Asset with correct asset_id |
| `test_upload_propagates_upload_error` | P1 | Session propagates UploadError from upload module |
| `test_upload_invalid_file_path_raises` | P0 | ValidationError for non-existent file |
| `test_upload_unsupported_format_raises` | P1 | ValidationError for unsupported audio format |
| `test_upload_chunked_short_file` | P1 | Returns single asset for file < chunk_duration |
| `test_upload_chunked_long_file` | P1 | Returns multiple assets with parent_asset_id |

#### Mock Strategy

```python
# Mock api.request_presigned_url and upload_to_presigned_url
# to avoid real S3 operations
@patch("redenlab_extract.session.api.request_presigned_url")
@patch("redenlab_extract.session.upload_to_presigned_url")
def test_upload_returns_asset(mock_upload, mock_presigned, mock_session):
    mock_presigned.return_value = ("asset-123", "https://s3.url", 3600)
    mock_upload.return_value = None

    asset = session.upload("test.wav")

    assert asset.asset_id == "asset-123"
```

---

### 2. Inference Client (`client.py`) - P0

The client handles model inference operations.

#### Tests Required

| Test | Priority | Description |
|------|----------|-------------|
| `test_client_init_with_session` | P0 | Client initializes with RedenLabSession |
| `test_client_init_validates_model_name` | P1 | ValidationError for invalid model |
| `test_submit_returns_job_id` | P0 | submit() returns job_id string |
| `test_poll_completed_returns_result` | P0 | poll() returns result when job completes |
| `test_poll_failed_raises_inference_error` | P0 | InferenceError when job fails |
| `test_poll_timeout_raises_timeout_error` | P0 | TimeoutError when exceeds timeout |
| `test_predict_submits_and_polls` | P0 | predict() = submit() + poll() |
| `test_predict_with_progress_callback` | P1 | Callback receives status updates |
| `test_submit_with_upload_wait_retries` | P1 | Retries on 409 upload_pending |
| `test_health_returns_status` | P1 | health() returns backend status |
| `test_warmup_already_ready_skips` | P2 | Skips warmup if backend ready |

#### Mock Strategy

```python
@patch("redenlab_extract.client.api.submit_inference_job")
@patch("redenlab_extract.client.api.get_job_status")
def test_predict_submits_and_polls(mock_status, mock_submit, mock_session):
    mock_submit.return_value = {"job_id": "job-123"}
    mock_status.return_value = {
        "job_id": "job-123",
        "status": "completed",
        "result": {"intelligibility_score": 0.85}
    }

    client = InferenceClient(mock_session, model_name="als-intelligibility-mtpa")
    result = client.predict(asset)

    assert result["status"] == "completed"
```

---

### 3. Factory Functions (`client.py`) - P1

Factory functions create configured clients.

#### Tests Required

| Test | Priority | Description |
|------|----------|-------------|
| `test_intelligibility_client_valid_model` | P0 | Returns InferenceClient for valid model |
| `test_intelligibility_client_invalid_model_raises` | P0 | ValidationError for invalid model |
| `test_intelligibility_client_sets_score_key` | P1 | Sets score_key for aggregation |
| `test_intelligibility_client_maps_alias` | P1 | Maps v1/v2 to backend names |
| `test_naturalness_client_valid_model` | P0 | Same tests for naturalness |
| `test_transcribe_client_with_language_code` | P0 | Stores default language_code |
| `test_transcribe_client_submit_uses_language` | P1 | Includes language_code in payload |

---

### 4. Workflows (`workflows.py`) - P0

High-level orchestration functions.

#### Tests Required

| Test | Priority | Description |
|------|----------|-------------|
| `test_run_inference_short_file` | P0 | Simple upload + predict flow |
| `test_run_inference_long_file_auto_chunks` | P0 | Auto-chunks and aggregates |
| `test_run_inference_no_aggregator_no_chunk` | P1 | Skips chunking without aggregator |
| `test_run_chunked_inference_single_chunk` | P1 | Single chunk returns list |
| `test_run_chunked_inference_multiple_chunks` | P0 | Multiple chunks with metadata |
| `test_run_chunked_inference_partial_failure` | P1 | Handles some chunk failures |
| `test_aggregate_results_mean` | P0 | Default mean aggregation |
| `test_aggregate_results_custom_reduce` | P1 | Custom reduce function |
| `test_aggregate_results_with_failed_chunks` | P1 | Partial status with failures |
| `test_default_aggregator_for_client` | P2 | Returns correct aggregator |

#### Mock Strategy

```python
def test_run_inference_short_file():
    mock_session = MagicMock()
    mock_client = MagicMock()

    # Mock duration detection
    mock_session._get_audio_duration.return_value = 30.0  # Under threshold
    mock_session.upload.return_value = Asset(asset_id="123", filename="test.wav")
    mock_client.chunk_threshold = 50.0
    mock_client.predict.return_value = {"status": "completed", "result": {...}}

    result = run_inference(mock_session, mock_client, "short.wav")

    mock_session.upload.assert_called_once()  # Not upload_chunked
    mock_client.predict.assert_called_once()
```

---

### 5. API Communication (`api.py`) - P1

HTTP request handling and error mapping.

#### Tests Required

| Test | Priority | Description |
|------|----------|-------------|
| `test_request_presigned_url_success` | P0 | Returns asset_id, url, expires |
| `test_request_presigned_url_with_chunk_metadata` | P1 | Includes parent_asset_id, chunk_index |
| `test_submit_inference_job_success` | P0 | Returns job_id |
| `test_get_job_status_success` | P0 | Returns status dict |
| `test_check_health_success` | P1 | Returns health dict |
| `test_api_401_raises_auth_error` | P0 | Maps 401 to AuthenticationError |
| `test_api_400_raises_inference_error` | P1 | Maps 400 to InferenceError |
| `test_api_500_raises_api_error` | P1 | Maps 5xx to APIError |
| `test_api_retries_on_network_error` | P1 | Retries ConnectionError |

#### Mock Strategy

Use `responses` library to mock HTTP responses:

```python
import responses

@responses.activate
def test_submit_inference_job_success():
    responses.add(
        responses.POST,
        "https://api.example.com/als-intelligibility-mtpa/predict",
        json={"job_id": "job-123", "status": "submitted"},
        status=200
    )

    result = api.submit_inference_job(
        base_url="https://api.example.com",
        api_key="sk_test_123",
        model_name="als-intelligibility-mtpa",
        payload={"asset_id": "asset-123"}
    )

    assert result["job_id"] == "job-123"
```

---

### 6. Polling (`polling.py`) - P1

Job status polling with backoff.

#### Tests Required

| Test | Priority | Description |
|------|----------|-------------|
| `test_poll_until_complete_success` | P0 | Returns on "completed" |
| `test_poll_until_complete_failed` | P0 | Raises InferenceError on "failed" |
| `test_poll_until_complete_timeout` | P0 | Raises TimeoutError |
| `test_poll_with_callback_invokes_callback` | P1 | Callback called on each poll |
| `test_exponential_backoff` | P2 | Delays increase correctly |

#### Mock Strategy

```python
def test_poll_until_complete_success():
    call_count = 0

    def mock_get_status():
        nonlocal call_count
        call_count += 1
        if call_count < 3:
            return {"status": "processing"}
        return {"status": "completed", "result": {"score": 0.85}}

    result = poll_until_complete(mock_get_status, job_id="123", timeout=60)

    assert result["status"] == "completed"
    assert call_count == 3
```

---

### 7. Payload Builders (`payload.py`) - P1

Model-specific payload construction.

#### Tests Required

| Test | Priority | Description |
|------|----------|-------------|
| `test_transcribe_builder_requires_language_code` | P0 | ValueError without language_code |
| `test_transcribe_builder_builds_payload` | P0 | Returns {asset_id, language_code} |
| `test_intelligibility_builder_rejects_params` | P1 | ValueError with unexpected params |
| `test_intelligibility_builder_builds_payload` | P0 | Returns {asset_id} |
| `test_get_payload_builder_returns_correct` | P1 | Registry lookup works |
| `test_get_payload_builder_unknown_raises` | P1 | ValueError for unknown model |

---

### 8. Configuration (`config.py`) - P2

Configuration loading from multiple sources.

#### Tests Required

| Test | Priority | Description |
|------|----------|-------------|
| `test_get_merged_config_defaults` | P1 | Returns default values |
| `test_get_merged_config_env_overrides` | P1 | Env vars override defaults |
| `test_get_merged_config_explicit_wins` | P0 | Explicit params override all |
| `test_load_config_file_valid` | P2 | Parses YAML correctly |
| `test_load_config_file_missing` | P2 | Returns empty dict |
| `test_cloudfront_discovery` | P2 | Fetches and caches URL |

---

### 9. Upload (`upload.py`) - P1

S3 presigned URL uploads.

#### Tests Required

| Test | Priority | Description |
|------|----------|-------------|
| `test_upload_to_presigned_url_success` | P0 | Uploads file without error |
| `test_upload_to_presigned_url_http_403` | P1 | Raises UploadError on 403 Forbidden |
| `test_upload_to_presigned_url_http_404` | P1 | Raises UploadError on 404 Not Found |
| `test_upload_to_presigned_url_http_500` | P1 | Raises UploadError on 500 Server Error |
| `test_upload_file_not_found` | P0 | Raises UploadError for non-existent file |
| `test_upload_file_not_readable` | P1 | Raises UploadError for unreadable file |
| `test_upload_network_timeout` | P1 | Raises UploadError on timeout |
| `test_upload_connection_error` | P1 | Raises UploadError on connection error |
| `test_upload_with_progress_callback` | P2 | Callback receives progress updates |

---

### 10. Utilities (`utils.py`) - P2

Validation and helper functions.

#### Tests Required

| Test | Priority | Description |
|------|----------|-------------|
| `test_validate_file_path_exists` | P1 | Returns Path for valid file |
| `test_validate_file_path_not_found` | P1 | ValidationError for missing |
| `test_validate_model_name_valid` | P1 | Returns normalized name |
| `test_validate_model_name_invalid` | P1 | ValidationError for invalid |
| `test_validate_timeout_valid` | P2 | Returns int for valid timeout |
| `test_get_content_type` | P2 | Returns correct MIME type |

---

### 11. Asset (`asset.py`) - P2

Asset data model.

#### Tests Required

| Test | Priority | Description |
|------|----------|-------------|
| `test_asset_creation` | P1 | Creates with required fields |
| `test_asset_optional_fields` | P2 | Optional fields default correctly |
| `test_asset_immutable` | P2 | Frozen dataclass behavior |

---

### 12. Exceptions (`exceptions.py`) - P3

Custom exception hierarchy.

#### Tests Required

| Test | Priority | Description |
|------|----------|-------------|
| `test_exception_hierarchy` | P2 | All inherit from RedenLabMLError |
| `test_exception_messages` | P3 | Exceptions format messages correctly |

---

## Integration Tests

Integration tests require valid credentials and make real API calls.

### Setup

```yaml
# tests/fixtures/config.yaml
api_key: sk_live_your_real_key
base_url: https://api.redenlab.com
```

### Tests Required

| Test | Priority | File |
|------|----------|------|
| `test_real_upload_and_predict` | P0 | `test_intelligibility_client.py` |
| `test_real_transcribe_workflow` | P0 | `test_transcribe_client.py` |
| `test_real_health_check` | P1 | `test_health.py` |
| `test_real_warmup` | P2 | `test_warmup.py` |
| `test_real_chunked_inference` | P1 | `test_chunked.py` |

### Running Integration Tests

```bash
# Skip integration tests (default in CI)
pytest tests/unit/

# Run integration tests (local development)
pytest tests/integration/ -m integration
```

---

## Test File Structure

```
tests/
├── unit/
│   ├── __init__.py
│   ├── conftest.py              # Shared fixtures
│   ├── test_session.py          # RedenLabSession tests
│   ├── test_client.py           # InferenceClient tests (existing)
│   ├── test_workflows.py        # run_inference, aggregate tests
│   ├── test_api.py              # HTTP communication tests
│   ├── test_polling.py          # Polling logic tests
│   ├── test_payload.py          # Payload builder tests (existing)
│   ├── test_config.py           # Configuration tests (existing)
│   ├── test_upload.py           # Upload tests
│   └── test_utils.py            # Utility function tests (existing)
├── integration/
│   ├── __init__.py
│   ├── conftest.py              # Real credentials fixture
│   ├── test_intelligibility.py  # Real intelligibility workflow
│   ├── test_transcribe.py       # Real transcribe workflow
│   └── test_chunked.py          # Real chunked inference
└── fixtures/
    ├── config.yaml              # Test credentials
    └── test_audio.wav           # Short audio file
```

---

## Shared Fixtures (`conftest.py`)

```python
import pytest
from unittest.mock import MagicMock
from redenlab_extract import RedenLabSession, Asset

@pytest.fixture
def mock_session():
    """Mock session for unit tests."""
    session = MagicMock(spec=RedenLabSession)
    session.api_key = "sk_test_12345"
    session.base_url = "https://api.test.example.com"
    return session

@pytest.fixture
def mock_asset():
    """Mock asset for unit tests."""
    return Asset(
        asset_id="test-asset-123",
        filename="test_audio.wav",
        content_type="audio/wav",
    )

@pytest.fixture
def completed_job_response():
    """Standard completed job response."""
    return {
        "job_id": "job-123",
        "status": "completed",
        "result": {"intelligibility_score": 0.85},
        "created_at": "2024-01-01T00:00:00Z",
        "completed_at": "2024-01-01T00:01:00Z",
    }

@pytest.fixture
def failed_job_response():
    """Standard failed job response."""
    return {
        "job_id": "job-123",
        "status": "failed",
        "error": "Audio duration exceeds maximum allowed",
    }
```

---

## Test Coverage Goals

| Module | Target Coverage | Rationale |
|--------|-----------------|-----------|
| `session.py` | 80% | Core entry point |
| `client.py` | 80% | Core inference logic |
| `workflows.py` | 70% | Orchestration layer |
| `api.py` | 70% | HTTP handling |
| `polling.py` | 70% | Retry logic |
| `payload.py` | 90% | Simple, easily testable |
| `config.py` | 60% | Many file I/O edge cases |
| `upload.py` | 60% | S3 interaction |
| `utils.py` | 80% | Validation functions |
| **Overall** | **~70%** | Practical coverage |

---

## Running Tests

```bash
# Run all unit tests
pytest tests/unit/ -v

# Run with coverage
pytest tests/unit/ --cov=redenlab_extract --cov-report=html

# Run specific test file
pytest tests/unit/test_session.py -v

# Run specific test
pytest tests/unit/test_client.py::TestInferenceClient::test_predict_submits_and_polls -v

# Run integration tests (requires credentials)
pytest tests/integration/ -m integration -v
```

---

## Priority Summary

### Must Have (P0) - 21 tests
- Session: init, upload, error handling
- Client: init, submit, poll, predict
- Workflows: run_inference, aggregate
- Factory: intelligibility_client, naturalness_client, transcribe_client

### Should Have (P1) - 32 tests
- Session: env vars, chunked upload, error propagation
- Client: callbacks, retry, health
- Workflows: chunked inference, partial failure
- API: error mapping, presigned URL
- Polling: complete/fail/timeout
- Upload: HTTP errors, network errors, file validation

### Nice to Have (P2) - 15 tests
- Config: file loading, discovery
- Upload: progress callback
- Utils: validation functions
- Asset: dataclass behavior

### Optional (P3) - 5 tests
- Exceptions: hierarchy, messages

---

## Next Steps

1. **Implement P0 tests first** - Core user workflows
2. **Add shared fixtures** in `conftest.py`
3. **Implement P1 tests** - Important edge cases
4. **Set up CI** to run unit tests on every PR
5. **Add integration tests** for real API validation (manual runs)
6. **Measure coverage** and fill gaps as needed
