# FCP Live Instance Integration Tests

This directory contains integration tests designed to run against live FCP instances.

## Overview

These tests verify real-world functionality of Flow SDK on actual GPU instances, including:
- SSH connectivity and command execution
- Docker functionality and caching
- Startup script execution
- Log collection and streaming
- GPU availability and nvidia-smi
- Data mounting and storage

## Prerequisites

1. Running FCP instances with SSH access
2. SSH key configured (via `FCP_SSH_KEY` environment variable or `~/.ssh/flow_key`)
3. Python 3.8+

## Test Files

### Core Test Suites
- `test_fcp_live_instances.py` - Comprehensive test suite for all instance features
- `test_docker_integration.py` - Docker-specific functionality tests
- `test_startup_scripts_integration.py` - Startup script validation
- `test_logs_integration.py` - Log collection and streaming tests
- `test_critical_features.py` - Critical features (Docker caching, nvidia-smi logging)

### Helper Scripts
- `verify_instance_access.py` - Quick SSH connectivity verification
- `test_docker_sudo.py` - Test Docker with sudo permissions

## Running Tests

### Automatic Instance Discovery
```bash
# Discover running instances and test
python discover_and_test.py --discover --key ~/.ssh/flow_key

# Run specific test suite on discovered instances
python discover_and_test.py --discover --key ~/.ssh/flow_key --test docker
```

### Manual Instance Specification
```bash
# Verify SSH access to specific instances
python verify_instance_access.py 18.144.35.49 54.177.129.4 --user ubuntu

# Run specific test suite
python test_docker_integration.py 18.144.35.49 --key ~/.ssh/flow_key

# Test critical features
python test_critical_features.py 18.144.35.49 --key ~/.ssh/flow_key
```

### Test All Features
```bash
# Run comprehensive test suite
python test_fcp_live_instances.py 18.144.35.49 54.177.129.4 --key ~/.ssh/flow_key
```

## Test Reports

Tests generate JSON reports with detailed results:
- Overall summary with pass/fail counts
- Per-instance results
- Detailed test output for debugging

## Notes

- These tests require actual running instances and cannot be mocked
- Tests are designed to be non-destructive but will pull Docker images
- Some tests require sudo access on the instances
- Instance startup can take 10-20 minutes for GPU instances