# End-to-End Tests

This directory contains end-to-end (E2E) tests that validate the complete MCA SDK telemetry workflow from SDK instrumentation through OpenTelemetry Collector to GCP backends.

## What E2E Tests Validate

E2E tests verify the **full telemetry pipeline**:

```
SDK → OTel Collector → GCP Backends
```

Specifically, these tests validate:

1. **Metrics Flow** (`test_metrics_e2e.py`)
   - Prediction metrics appear in GCP Cloud Monitoring within 30 seconds
   - Histogram distributions are preserved
   - Custom attributes are maintained

2. **Logs Flow** (`test_logs_e2e.py`)
   - Structured logs appear in GCP Cloud Logging
   - Log attributes match prediction metadata
   - No PHI leakage in logs

3. **Traces Flow** (`test_traces_e2e.py`)
   - Complete span hierarchy appears in GCP Cloud Trace
   - Nested spans maintain parent-child relationships
   - Span timing is accurate (±10ms tolerance)

4. **Multi-Model Types** (`test_multi_model_e2e.py`)
   - Internal/Predictive ML models
   - Generative AI/LLM models with token tracking
   - Agentic AI models with goal/tool tracking
   - Vendor model bridges

5. **Error Scenarios** (`test_error_scenarios_e2e.py`)
   - Collector downtime → SDK buffers and retries
   - GCP unavailability → SDK gracefully degrades
   - Registry unavailability → SDK uses local config fallback
   - Network timeouts → SDK doesn't block application

## Prerequisites

### Required

- **Docker** and **Docker Compose** installed
- **Python 3.10+** with pytest
- **MCA SDK** installed in development mode

### Optional (for real GCP testing)

- **GCP Project** with enabled APIs:
  - Cloud Monitoring API
  - Cloud Logging API
  - Cloud Trace API
- **Service Account Key** with permissions:
  - `monitoring.metricWriter`
  - `logging.logWriter`
  - `cloudtrace.agent`
- **Environment Variable**: `GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json`

## Running E2E Tests

### Local Development (Mock GCP)

E2E tests run with mock GCP clients by default for development:

```bash
# Terminal 1: Start OTel Collector
cd mca-prototype
docker-compose up -d otel-collector

# Terminal 2: Run E2E tests
pytest tests/e2e/ -v -s
```

### With Real GCP (Optional)

To test against real GCP backends:

1. Set up GCP credentials:
```bash
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json
export GCP_PROJECT_ID=your-project-id
```

2. Update collector config to export to GCP:
```bash
# Use GCP-enabled collector config
docker-compose -f docker-compose.yml -f docker-compose.gcp-dev.yml up -d otel-collector
```

3. Run E2E tests:
```bash
pytest tests/e2e/ -v -s --gcp-real
```

### Run Specific Test Suite

```bash
# Metrics only
pytest tests/e2e/test_metrics_e2e.py -v

# Logs only
pytest tests/e2e/test_logs_e2e.py -v

# Traces only
pytest tests/e2e/test_traces_e2e.py -v

# Multi-model types
pytest tests/e2e/test_multi_model_e2e.py -v

# Error scenarios
pytest tests/e2e/test_error_scenarios_e2e.py -v
```

### Run Single Test

```bash
pytest tests/e2e/test_metrics_e2e.py::TestMetricsE2E::test_prediction_metric_propagates_to_gcp -v -s
```

## CI/CD Integration

### GitHub Actions / GitLab CI

E2E tests should run on every merge to `main` or release branch:

```yaml
# .github/workflows/e2e-tests.yml
name: E2E Tests

on:
  push:
    branches: [main, release/*]
  pull_request:
    branches: [main]

jobs:
  e2e-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install dependencies
        run: |
          pip install -e .
          pip install pytest pytest-timeout

      - name: Start OTel Collector
        run: |
          cd mca-prototype
          docker-compose up -d otel-collector
          sleep 10  # Wait for collector to be ready

      - name: Run E2E tests
        run: |
          pytest tests/e2e/ -v --timeout=300

      - name: Cleanup
        if: always()
        run: |
          cd mca-prototype
          docker-compose down
```

## Test Structure

Each E2E test follows this pattern:

```python
def test_scenario(
    docker_compose_environment,  # Starts collector
    gcp_monitoring_client,       # Mock or real GCP client
    gcp_project_id,              # GCP project ID
    wait_for_telemetry,          # Utility for propagation delay
    cleanup_collector_exports    # Cleanup fixture
):
    # Arrange: Create SDK client
    client = MCAClient(...)

    # Act: Record telemetry
    client.record_prediction(...)
    client.shutdown()

    # Wait: Allow propagation
    wait_for_telemetry(30, "metric propagation")

    # Assert: Query GCP backend
    metrics = gcp_monitoring_client.query_metrics(...)
    assert len(metrics) > 0
```

## Configuration

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `GCP_PROJECT_ID` | `test-project-12345` | GCP project for E2E tests |
| `GOOGLE_APPLICATION_CREDENTIALS` | None | Path to service account key |
| `OTEL_COLLECTOR_ENDPOINT` | `http://localhost:4318` | Collector OTLP endpoint |
| `OTEL_HEALTH_ENDPOINT` | `http://localhost:13133` | Collector health check |
| `E2E_TIMEOUT` | `60` | Max wait time for services (seconds) |

### Pytest Markers

E2E tests use these markers:

- `@pytest.mark.integration` - Requires Docker services
- `@pytest.mark.error_scenario` - Intentionally triggers errors/warnings
- `@pytest.mark.slow` - Tests that take >10 seconds

Run specific markers:

```bash
# Integration tests only
pytest tests/e2e/ -m integration

# Exclude slow tests
pytest tests/e2e/ -m "not slow"
```

## Troubleshooting

### Collector Not Healthy

```bash
# Check collector logs
docker-compose logs otel-collector

# Verify health endpoint
curl http://localhost:13133/

# Restart collector
docker-compose restart otel-collector
```

### Timeouts

If tests timeout waiting for telemetry:

1. **Increase wait time** in test: `wait_for_telemetry(60, "longer wait")`
2. **Check collector config**: Verify batch processor timeout settings
3. **Check GCP credentials**: Ensure service account has correct permissions

### Mock vs Real GCP

Mock GCP clients are used by default. To verify real GCP integration:

1. Set `GOOGLE_APPLICATION_CREDENTIALS`
2. Set `GCP_PROJECT_ID`
3. Update fixtures in `conftest.py` to use real clients
4. Run with `--gcp-real` flag (when implemented)

### No Metrics Appearing

If metrics don't appear in GCP:

1. **Check collector config**: Verify exporters are configured
2. **Check GCP permissions**: Ensure service account can write metrics
3. **Check project ID**: Verify correct GCP project
4. **Check collector logs**: Look for export errors

## Test Data Cleanup

E2E tests create test data in GCP (when using real backends):

- **Metrics**: Auto-expire based on retention policy (default 6 months)
- **Logs**: Auto-expire based on retention policy (default 30 days)
- **Traces**: Auto-expire based on retention policy (default 7 days)

For manual cleanup:

```bash
# Delete test metrics (requires gcloud CLI)
gcloud monitoring time-series delete \
  --project=$GCP_PROJECT_ID \
  --filter='resource.labels.service_name=~"e2e-.*"'

# Delete test logs
gcloud logging logs delete \
  --project=$GCP_PROJECT_ID \
  --filter='resource.labels.service_name=~"e2e-.*"'
```

## Best Practices

1. **Use realistic test data** - No real PHI, but realistic structure
2. **Test idempotency** - Tests should be runnable multiple times
3. **Clean up resources** - Use cleanup fixtures
4. **Set appropriate timeouts** - Account for propagation delays
5. **Verify data integrity** - Check attributes, not just existence
6. **Test error recovery** - Don't just test happy paths

## Related Documentation

- [Integration Tests](../integration/README.md) - Collector-only integration tests
- [Performance Tests](../performance/README.md) - Load and performance testing
- [Docker Compose](../../docker-compose.yml) - Local development environment
- [OTel Collector Config](../../config/otel-collector-config.yaml) - Collector configuration

## Support

For issues or questions:

1. Check [Troubleshooting](#troubleshooting) section above
2. Review [test output logs](#running-e2e-tests)
3. Check collector logs: `docker-compose logs otel-collector`
4. Open issue in repository with test failure details
