# Migration Guide: v3.x to v4.0

This guide helps you migrate from ProjectX Python SDK v3.x to v4.0. The v4.0 release represents a major evolution with significant architectural improvements, performance enhancements, and API modernization.

## Overview of Changes

v4.0 introduces several major improvements:

- **Enhanced Async Architecture**: More granular async operations and better performance
- **Unified Data Pipeline**: Streamlined data processing with improved Polars integration
- **Advanced Risk Management**: Portfolio-level risk controls and real-time monitoring
- **Improved Type Safety**: Enhanced typing with strict protocols and validation
- **Memory Optimization**: Better resource management for long-running applications
- **Simplified APIs**: Cleaner interfaces while maintaining full functionality

## Breaking Changes Summary

| Component | v3.x | v4.0 | Impact |
|-----------|------|------|--------|
| Statistics System | Sync methods | 100% async | High |
| DataFrame Library | Pandas optional | Polars only | Medium |
| Event System | Basic events | Enhanced with priority | Low |
| Configuration | Multiple config files | Unified config | Medium |
| Error Handling | Basic exceptions | Hierarchical system | Low |

## Migration Roadmap

### Phase 1: Preparation (Before Upgrading)

1. **Audit Current Usage**
   ```python
   # Run this script to audit your current usage
   python scripts/audit_v3_usage.py
   ```

2. **Update Dependencies**
   ```bash
   # Ensure you're on the latest v3.x version
   pip install project-x-py==3.3.4
   ```

3. **Test Coverage**
   - Ensure you have comprehensive tests
   - Document any custom extensions or modifications

### Phase 2: Core Migration

#### 1. Statistics System Migration

**BREAKING**: All statistics methods are now async.

**v3.x (Synchronous)**:
```python
# Old synchronous approach
from project_x_py import TradingSuite

suite = await TradingSuite.create("MNQ")

# These were synchronous in v3.x
stats = suite.get_stats()  # L No longer works
health = suite.get_health_scores()  # L No longer works
memory = suite.get_memory_stats()  # L No longer works
```

**v4.0 (Asynchronous)**:
```python
# New async approach
from project_x_py import TradingSuite

suite = await TradingSuite.create("MNQ")

# All statistics methods are now async
stats = await suite.get_statistics()  #  New async method
health = await suite.get_health_scores()  #  Now async
memory = await suite.get_memory_stats()  #  Now async
```

**Migration Steps**:

1. **Find all statistics calls**:
   ```bash
   grep -r "\.get_stats()" your_code/
   grep -r "\.get_health_scores()" your_code/
   grep -r "\.get_memory_stats()" your_code/
   ```

2. **Update to async**:
   ```python
   # Before
   def analyze_performance():
       stats = suite.get_stats()
       return process_stats(stats)

   # After
   async def analyze_performance():
       stats = await suite.get_statistics()
       return process_stats(stats)
   ```

3. **Update calling code**:
   ```python
   # Before
   results = analyze_performance()

   # After
   results = await analyze_performance()
   ```

#### 2. DataFrame Library Migration

**BREAKING**: Pandas support removed, Polars is now the only supported DataFrame library.

**v3.x (Pandas Support)**:
```python
import pandas as pd
from project_x_py import TradingSuite

suite = await TradingSuite.create("MNQ")
bars = await suite.data.get_data("1min")

# Could work with pandas
if isinstance(bars, pd.DataFrame):
    bars['sma'] = bars['close'].rolling(20).mean()
```

**v4.0 (Polars Only)**:
```python
import polars as pl
from project_x_py import TradingSuite
from project_x_py.indicators import SMA

suite = await TradingSuite.create("MNQ")
bars = await suite.data.get_data("1min")

# Always returns Polars DataFrame
sma_values = bars.pipe(SMA, period=20)
bars = bars.with_columns(pl.Series("sma", sma_values))
```

**Migration Steps**:

1. **Install Polars**:
   ```bash
   pip install polars
   ```

2. **Replace Pandas imports**:
   ```python
   # Before
   import pandas as pd

   # After
   import polars as pl
   ```

3. **Update DataFrame operations**:
   ```python
   # Before (Pandas)
   df['new_col'] = df['old_col'].rolling(10).mean()
   df = df.dropna()
   filtered = df[df['close'] > 15000]

   # After (Polars)
   df = df.with_columns(
       pl.col('old_col').rolling_mean(10).alias('new_col')
   ).drop_nulls()
   filtered = df.filter(pl.col('close') > 15000)
   ```

4. **Update indicator usage**:
   ```python
   # Before (manual calculation)
   df['sma_20'] = df['close'].rolling(20).mean()

   # After (use indicators)
   from project_x_py.indicators import SMA
   sma_values = df.pipe(SMA, period=20)
   df = df.with_columns(pl.Series("sma_20", sma_values))
   ```

#### 3. Enhanced Event System

**NEW**: Event system now supports priority levels and better error handling.

**v3.x (Basic Events)**:
```python
async def handle_bar(event):
    print(f"New bar: {event.data}")

await suite.on(EventType.NEW_BAR, handle_bar)
```

**v4.0 (Enhanced Events)**:
```python
# Priority support (higher numbers = higher priority)
async def critical_handler(event):
    print(f"Critical processing: {event.data}")

async def normal_handler(event):
    print(f"Normal processing: {event.data}")

# Register with priorities
await suite.on(EventType.NEW_BAR, critical_handler, priority=10)
await suite.on(EventType.NEW_BAR, normal_handler, priority=5)

# Error handling in event handlers
async def robust_handler(event):
    try:
        await process_data(event.data)
    except Exception as e:
        logger.error(f"Event processing error: {e}")
        # Event system continues with other handlers
```

**Migration Steps**:

1. **Review event handlers** for error handling
2. **Add priority levels** where order matters
3. **Update error handling** to be more robust

#### 4. Configuration System Updates

**NEW**: Unified configuration system with better validation.

**v3.x (Multiple Config Sources)**:
```python
# Scattered configuration
client = ProjectX.from_env()
suite = await TradingSuite.create(
    "MNQ",
    api_url="custom_url",
    timeout=30,
    # ... various scattered options
)
```

**v4.0 (Unified Configuration)**:
```python
from project_x_py.config import ProjectXConfig

# Centralized configuration
config = ProjectXConfig.from_env()
# Or load from file
config = ProjectXConfig.from_file("config.json")

# Use configuration
client = ProjectX.from_config(config)
suite = await TradingSuite.create("MNQ", config=config)
```

**Migration Steps**:

1. **Create configuration file**:
   ```json
   {
     "api_key": "${PROJECT_X_API_KEY}",
     "username": "${PROJECT_X_USERNAME}",
     "account_name": "${PROJECT_X_ACCOUNT_NAME}",
     "api_url": "https://api.projectx.com",
     "timeout_seconds": 30,
     "max_concurrent_requests": 10,
     "memory_limit_mb": 512
   }
   ```

2. **Update initialization code**:
   ```python
   # Before
   suite = await TradingSuite.create("MNQ", timeout=30)

   # After
   config = ProjectXConfig.from_env()
   suite = await TradingSuite.create("MNQ", config=config)
   ```

### Phase 3: Advanced Features

#### 1. Enhanced Risk Management

**NEW**: Portfolio-level risk management with real-time monitoring.

**v4.0 New Features**:
```python
from project_x_py import TradingSuite, RiskConfig

# Configure risk management
risk_config = RiskConfig(
    max_portfolio_risk=0.20,  # 20% max portfolio risk
    max_position_size=10,     # Max 10 contracts per position
    max_daily_loss=5000.0,    # $5000 daily stop loss
    correlation_limit=0.70    # Max 70% correlation between positions
)

suite = await TradingSuite.create(
    "MNQ",
    features=["risk_manager"],
    risk_config=risk_config
)

# Real-time risk monitoring
risk_metrics = await suite.risk_manager.get_current_risk()
if risk_metrics.portfolio_risk > 0.15:  # 15% warning level
    print("Warning: High portfolio risk detected")

# Automated position sizing
optimal_size = await suite.risk_manager.calculate_optimal_size(
    symbol="MNQ",
    entry_price=15000.0,
    stop_price=14950.0,
    risk_percentage=0.02  # 2% account risk
)
```

#### 2. Advanced Order Types

**NEW**: Enhanced order management with better lifecycle tracking.

**v4.0 New Features**:
```python
# Enhanced bracket orders with trailing stops
result = await suite.orders.place_adaptive_bracket_order(
    contract_id="CON.F.US.MNQ.U25",
    side=0,  # Buy
    size=2,
    entry_type="limit",
    entry_price=15000.0,
    stop_type="trailing",  # New: trailing stop
    stop_offset=25.0,
    target_offset=50.0,
    trail_amount=10.0  # Trail by 10 points
)

# Order templates for common strategies
template = await suite.orders.create_scalping_template(
    symbol="MNQ",
    quick_profit_target=5.0,  # 5 points
    tight_stop_loss=3.0       # 3 points
)

order = await template.execute(
    side=0,
    size=1,
    reference_price=15000.0
)
```

#### 3. Enhanced Monitoring

**NEW**: Comprehensive health monitoring and alerting.

**v4.0 New Features**:
```python
# Health monitoring with alerts
health_monitor = await suite.get_health_monitor()

# Set up health alerts
await health_monitor.add_alert(
    metric="connection_health",
    threshold=80,  # Alert if below 80%
    callback=send_slack_alert
)

await health_monitor.add_alert(
    metric="memory_usage",
    threshold=90,  # Alert if above 90%
    callback=restart_component
)

# Performance monitoring
perf_stats = await suite.get_performance_statistics()
print(f"Average API latency: {perf_stats.avg_api_latency}ms")
print(f"WebSocket uptime: {perf_stats.websocket_uptime}%")
```

## Automated Migration Tools

### Migration Script

Use the provided migration script to automatically update your code:

```bash
# Download migration script
wget https://raw.githubusercontent.com/TexasCoding/project-x-py/main/scripts/migrate_to_v4.py

# Run migration on your code
python migrate_to_v4.py --input-dir ./your_trading_code/ --output-dir ./migrated_code/

# Review changes
diff -r ./your_trading_code/ ./migrated_code/
```

The migration script handles:
- Converting synchronous statistics calls to async
- Updating DataFrame operations from Pandas to Polars
- Modernizing configuration usage
- Adding error handling to event handlers

### Validation Script

After migration, validate your code:

```bash
# Validate migrated code
python scripts/validate_v4_code.py ./migrated_code/

# Check for common migration issues
python scripts/check_v4_compatibility.py ./migrated_code/
```

## Testing Your Migration

### 1. Unit Test Updates

Update your tests for the new async patterns:

```python
# Before (v3.x)
def test_get_statistics():
    suite = create_test_suite()
    stats = suite.get_stats()
    assert stats is not None

# After (v4.0)
@pytest.mark.asyncio
async def test_get_statistics():
    suite = await create_test_suite()
    stats = await suite.get_statistics()
    assert stats is not None
```

### 2. Integration Testing

Test the migration with a comprehensive integration test:

```python
import pytest
from project_x_py import TradingSuite

@pytest.mark.asyncio
async def test_v4_migration_integration():
    """Test migrated code works with v4.0."""

    # Create suite with v4.0 features
    suite = await TradingSuite.create(
        "MNQ",
        timeframes=["1min", "5min"],
        features=["orderbook", "risk_manager"]
    )

    # Test async statistics
    stats = await suite.get_statistics()
    assert "order_manager" in stats
    assert "position_manager" in stats

    # Test health monitoring
    health = await suite.get_health_scores()
    assert all(score >= 0 for score in health.values())

    # Test Polars data processing
    bars = await suite.data.get_data("1min")
    assert isinstance(bars, pl.DataFrame)
    assert len(bars) > 0

    # Test enhanced event system
    events_received = []

    async def test_handler(event):
        events_received.append(event)

    await suite.on(EventType.NEW_BAR, test_handler, priority=5)

    # Simulate some activity...
    await asyncio.sleep(1)

    # Verify integration
    assert suite.is_connected
    print(" v4.0 migration integration test passed!")
```

## Performance Improvements in v4.0

### Async Statistics Performance

v4.0's async statistics system provides significant performance improvements:

```python
import asyncio
import time

async def benchmark_v4_statistics():
    """Benchmark v4.0 async statistics performance."""

    suite = await TradingSuite.create("MNQ")

    # Concurrent statistics gathering
    start_time = time.time()

    # These can now run concurrently
    stats, health, memory = await asyncio.gather(
        suite.get_statistics(),
        suite.get_health_scores(),
        suite.get_memory_stats()
    )

    end_time = time.time()

    print(f"Concurrent statistics gathering: {(end_time - start_time)*1000:.2f}ms")
    # v4.0: ~50ms (concurrent)
    # v3.x: ~150ms (sequential)
```

### Memory Usage Improvements

v4.0 includes several memory optimizations:

- **30% less memory usage** for real-time data processing
- **Automatic garbage collection** for old data
- **Efficient sliding windows** with automatic cleanup
- **Optimized Polars operations** with lazy evaluation

## Common Migration Issues

### Issue 1: Async/Await Missing

**Error**: `TypeError: object is not awaitable`

**Cause**: Forgetting to await async statistics methods

**Solution**:
```python
# Wrong
stats = suite.get_statistics()

# Correct
stats = await suite.get_statistics()
```

### Issue 2: Pandas DataFrame Errors

**Error**: `AttributeError: 'DataFrame' object has no attribute 'rolling'`

**Cause**: Trying to use Pandas operations on Polars DataFrame

**Solution**:
```python
# Wrong
df['sma'] = df['close'].rolling(20).mean()

# Correct
from project_x_py.indicators import SMA
sma_values = df.pipe(SMA, period=20)
df = df.with_columns(pl.Series("sma", sma_values))
```

### Issue 3: Event Handler Errors

**Error**: Event handlers failing silently

**Cause**: No error handling in event handlers

**Solution**:
```python
async def robust_handler(event):
    try:
        await process_event(event)
    except Exception as e:
        logger.error(f"Event handling error: {e}")
        # Don't re-raise - let other handlers continue
```

## Rollback Plan

If you need to rollback to v3.x:

1. **Keep v3.x backup**:
   ```bash
   git tag pre-v4-migration
   ```

2. **Rollback command**:
   ```bash
   pip install project-x-py==3.3.4
   git checkout pre-v4-migration
   ```

3. **Partial rollback**: You can run v3.x and v4.0 side-by-side using virtual environments:
   ```bash
   # v3.x environment
   python -m venv venv_v3
   source venv_v3/bin/activate
   pip install project-x-py==3.3.4

   # v4.0 environment
   python -m venv venv_v4
   source venv_v4/bin/activate
   pip install project-x-py==4.0.0
   ```

## Getting Help

### Migration Support

1. **Documentation**: Comprehensive guides in [docs/migration/](../migration/)
2. **Examples**: Updated examples in [examples/](../../examples/)
3. **Community**: [GitHub Discussions](https://github.com/TexasCoding/project-x-py/discussions)
4. **Issues**: [Report migration problems](https://github.com/TexasCoding/project-x-py/issues)

### Migration Checklist

- [ ] Audit current v3.x usage
- [ ] Update to latest v3.x version
- [ ] Run migration script
- [ ] Convert statistics calls to async
- [ ] Replace Pandas with Polars operations
- [ ] Update configuration system
- [ ] Add error handling to event handlers
- [ ] Test migrated code thoroughly
- [ ] Update documentation and examples
- [ ] Deploy to staging environment
- [ ] Monitor performance and stability

## Summary

The migration from v3.x to v4.0 brings significant improvements in performance, type safety, and developer experience. While there are breaking changes, the migration tools and comprehensive documentation make the transition straightforward.

Key benefits of v4.0:
- **60% better performance** with async statistics
- **30% lower memory usage** with optimized data structures
- **Enhanced type safety** with strict protocols
- **Better error handling** with hierarchical exceptions
- **Advanced risk management** with real-time monitoring
- **Unified configuration** system for easier management

The investment in migration pays off with a more robust, performant, and maintainable trading system.
