# Running E2E Tests in GCP Workbench

Complete guide for running MCA SDK end-to-end tests in Google Cloud Workbench.

## Quick Start (Mock GCP - Recommended for Testing)

```bash
# 1. Navigate to project directory
cd /home/coder/emms/sdk.worktrees/development/mca-prototype

# 2. Start OTel Collector
docker compose up -d otel-collector

# 3. Wait for collector to be healthy (10-15 seconds)
sleep 15

# 4. Run E2E tests
pytest tests/e2e/ -v -s

# 5. Cleanup
docker compose down
```

## Detailed Setup

### Step 1: Install Dependencies (if needed)

```bash
cd /home/coder/emms/sdk.worktrees/development/mca-prototype

# Install SDK in development mode
pip install -e .

# Install test dependencies
pip install pytest pytest-timeout requests
```

### Step 2: Verify Docker is Running

GCP Workbench usually has Docker pre-installed, but verify:

```bash
# Check Docker
docker --version

# Check Docker Compose
docker compose version

# Test Docker is accessible
docker ps
```

**If Docker permission issues:**
```bash
# Add user to docker group (requires logout/login)
sudo usermod -aG docker $USER

# Or run with sudo (not recommended for tests)
sudo docker compose up -d otel-collector
```

### Step 3: Prepare Test Environment

```bash
cd /home/coder/emms/sdk.worktrees/development/mca-prototype

# Create test-exports directory with correct permissions
mkdir -p test-exports
chmod 777 test-exports

# Clean old export files
rm -f test-exports/*.json

# Verify collector config
ls -la config/otel-collector-config.yaml
```

### Step 4: Start OTel Collector

```bash
# Start collector in background
docker compose up -d otel-collector

# Check collector status
docker compose ps

# Verify collector is healthy
curl http://localhost:13133/

# Check collector logs (should show "Server available")
docker compose logs otel-collector | tail -20
```

Expected output:
```
{"level":"info","ts":...,"msg":"Everything is ready. Begin running and processing data."}
```

### Step 5: Run E2E Tests

#### Option A: Run All E2E Tests

```bash
# Run all 19 E2E tests
pytest tests/e2e/ -v -s

# With timeout protection (recommended)
pytest tests/e2e/ -v -s --timeout=300

# Parallel execution (faster)
pytest tests/e2e/ -v -s -n 4
```

#### Option B: Run Specific Test Suites

```bash
# Metrics tests only (3 tests)
pytest tests/e2e/test_metrics_e2e.py -v -s

# Logs tests only (3 tests)
pytest tests/e2e/test_logs_e2e.py -v -s

# Traces tests only (4 tests)
pytest tests/e2e/test_traces_e2e.py -v -s

# Multi-model tests only (4 tests)
pytest tests/e2e/test_multi_model_e2e.py -v -s

# Error scenarios only (5 tests)
pytest tests/e2e/test_error_scenarios_e2e.py -v -s
```

#### Option C: Run Single Test

```bash
# Run specific test by name
pytest tests/e2e/test_metrics_e2e.py::TestMetricsE2E::test_prediction_metric_propagates_to_gcp -v -s
```

### Step 6: Verify Test Results

After tests complete, check the exports:

```bash
# List generated export files
ls -lh test-exports/

# View metrics export (JSONL format)
cat test-exports/otel-metrics.json | head -50

# Count lines in each export
wc -l test-exports/*.json

# Pretty print first metric
cat test-exports/otel-metrics.json | head -1 | python3 -m json.tool
```

### Step 7: Cleanup

```bash
# Stop and remove containers
docker compose down

# Clean export files
rm -f test-exports/*.json

# Remove containers and volumes
docker compose down -v
```

## Running with Real GCP (Advanced)

To test against real GCP services (Cloud Monitoring, Logging, Trace):

### Prerequisites

1. **GCP Project with APIs enabled:**
   - Cloud Monitoring API
   - Cloud Logging API
   - Cloud Trace API

2. **Service Account with permissions:**
   - `monitoring.metricWriter`
   - `logging.logWriter`
   - `cloudtrace.agent`

3. **Service Account Key JSON file**

### Setup Steps

```bash
# 1. Set GCP credentials
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json
export GCP_PROJECT_ID=your-gcp-project-id

# 2. Verify credentials
gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS
gcloud config set project $GCP_PROJECT_ID

# 3. Start collector with GCP exporters
docker compose -f docker-compose.yml -f docker-compose.gcp-dev.yml up -d otel-collector

# 4. Run tests (they will now use real GCP)
pytest tests/e2e/ -v -s

# 5. Verify in GCP Console
# - Cloud Monitoring: Metrics Explorer → Filter by "mca."
# - Cloud Logging: Logs Explorer → Filter by service_name
# - Cloud Trace: Trace List → Filter by service_name
```

**Note:** Real GCP tests will incur costs and create actual telemetry data in your project.

## GCP Workbench Specific Tips

### 1. Jupyter Notebook Integration

Run tests from Jupyter notebook:

```python
# In Jupyter cell
import subprocess
import sys

# Run E2E tests
result = subprocess.run(
    ["pytest", "tests/e2e/", "-v"],
    cwd="/home/coder/emms/sdk.worktrees/development/mca-prototype",
    capture_output=True,
    text=True
)

print(result.stdout)
print(result.stderr)
```

### 2. Terminal Access

GCP Workbench provides terminal access:
- Click **Terminal** in Jupyter menu
- Or use JupyterLab terminal sidebar

### 3. Port Forwarding

If you need to access collector health endpoint from browser:

```bash
# Collector health: http://localhost:13133/
# Collector metrics: http://localhost:8888/metrics
```

GCP Workbench automatically forwards common ports.

### 4. Resource Limits

Workbench instances may have limited resources:

```bash
# Check available memory
free -h

# Check available disk
df -h

# Monitor Docker resource usage
docker stats --no-stream
```

If tests fail due to resources:
- Use smaller batch sizes in collector config
- Run test suites sequentially instead of parallel
- Increase Workbench instance size

### 5. Network Configuration

GCP Workbench uses private networking:

```bash
# Check network connectivity
curl -I http://localhost:4318/v1/metrics

# Test collector health
curl http://localhost:13133/

# Check Docker network
docker network ls
```

## Troubleshooting

### Issue 1: Collector Not Starting

```bash
# Check Docker service
sudo systemctl status docker

# Check port conflicts
sudo lsof -i :4318
sudo lsof -i :13133

# View detailed logs
docker compose logs otel-collector --follow
```

### Issue 2: Tests Timeout

```bash
# Increase timeout
pytest tests/e2e/ -v -s --timeout=600

# Run with more verbose output
pytest tests/e2e/ -v -s --tb=long

# Check collector is receiving data
docker compose logs otel-collector | grep -i "received"
```

### Issue 3: Permission Denied on test-exports/

```bash
# Fix permissions
sudo chmod 777 test-exports/

# Or change ownership
sudo chown -R $USER:$USER test-exports/
```

### Issue 4: No Metrics in Export Files

```bash
# Verify collector config includes file exporters
grep -A 5 "file/metrics" config/otel-collector-config.yaml

# Check collector volume mount
docker compose exec otel-collector ls -la /var/otel-exports/

# Check collector logs for export errors
docker compose logs otel-collector | grep -i error
```

### Issue 5: Docker Compose Command Not Found

```bash
# Try legacy command
docker-compose up -d otel-collector

# Or install docker compose plugin
sudo apt-get update
sudo apt-get install docker-compose-plugin
```

## Performance Tips

### Run Tests Faster

```bash
# Parallel execution (requires pytest-xdist)
pip install pytest-xdist
pytest tests/e2e/ -v -n 4

# Skip slow tests
pytest tests/e2e/ -v -m "not slow"

# Reduce propagation timeouts in tests
# Edit tests/e2e/conftest.py and reduce wait times
```

### Resource Optimization

```bash
# Limit Docker memory
docker compose up -d otel-collector --memory=512m

# Clean up old containers
docker system prune -a

# Monitor resource usage
watch -n 2 'docker stats --no-stream'
```

## CI/CD Integration in GCP

### Cloud Build

Create `cloudbuild.yaml`:

```yaml
steps:
  # Start collector
  - name: 'docker/compose:latest'
    args: ['up', '-d', 'otel-collector']

  # Wait for health
  - name: 'curlimages/curl:latest'
    args: ['--retry', '10', '--retry-delay', '2', 'http://otel-collector:13133/']

  # Run E2E tests
  - name: 'python:3.10'
    args: ['pytest', 'tests/e2e/', '-v']

  # Cleanup
  - name: 'docker/compose:latest'
    args: ['down']
```

### Vertex AI Workbench

Schedule tests using notebook execution:

```python
# tests/e2e/run_e2e_tests.ipynb
!cd /path/to/project && docker compose up -d otel-collector
!sleep 15
!pytest tests/e2e/ -v --junit-xml=results.xml
!docker compose down
```

## Example Test Run Output

Successful test run looks like:

```
$ pytest tests/e2e/ -v -s

tests/e2e/test_metrics_e2e.py::TestMetricsE2E::test_prediction_metric_propagates_to_gcp
🚀 Starting Docker Compose services for E2E tests...
⏳ Waiting for services to be healthy...
✅ OTel Collector is healthy
⏳ Waiting 10s for metric propagation to file exporter...
✅ Wait complete
PASSED                                           [ 5%]

tests/e2e/test_metrics_e2e.py::TestMetricsE2E::test_latency_histogram_propagates_to_gcp
⏳ Waiting 10s for histogram aggregation and export...
✅ Wait complete
PASSED                                           [10%]

... (17 more tests)

======================== 19 passed in 245.67s =========================
```

## Additional Resources

- **E2E Test README:** `tests/e2e/README.md`
- **OTel Collector Config:** `config/otel-collector-config.yaml`
- **Docker Compose:** `docker-compose.yml`
- **SDK Documentation:** `docs/`

## Support

If you encounter issues:

1. Check collector logs: `docker compose logs otel-collector`
2. Verify export files exist: `ls -la test-exports/`
3. Check test output: `pytest tests/e2e/ -v -s`
4. Review troubleshooting section above

For GCP Workbench specific issues, check:
- Workbench instance logs in GCP Console
- VM instance resource usage
- Network firewall rules (if accessing external services)
