# Hidden Markov Models for Market Regime Detection

*A comprehensive guide to the Hidden Regime HMM implementation*

---

## Table of Contents

1. [Overview](#overview)
2. [Theoretical Foundation](#theoretical-foundation)
3. [API Reference](#api-reference)
4. [Usage Guide](#usage-guide)
5. [Mathematical Details](#mathematical-details)
6. [Examples](#examples)
7. [Best Practices](#best-practices)
8. [Troubleshooting](#troubleshooting)

---

## Overview

The Hidden Regime HMM implementation provides a sophisticated statistical framework for detecting and analyzing market regimes in financial time series. This implementation is specifically optimized for financial data with robust numerical stability and comprehensive regime interpretation.

### Key Features

- **Gaussian Emission HMMs**: Continuous observations with state-dependent normal distributions
- **Baum-Welch Training**: Maximum likelihood parameter estimation via EM algorithm
- **Numerical Stability**: Log-space computations with underflow protection
- **Market-Specific Design**: Regime interpretation and financial time series optimizations
- **Flexible Configuration**: Extensive customization options via configuration objects

---

## Theoretical Foundation

### What is a Hidden Markov Model?

A Hidden Markov Model (HMM) is a statistical model that assumes an underlying, unobservable (hidden) Markov process that generates observable data. In the context of financial markets, the hidden states represent market regimes (bull, bear, sideways), while the observed data consists of log returns.

### Mathematical Framework

An HMM is characterized by three key components:

1. **Hidden States (S)**: A set of N unobservable market regimes
2. **Observations (O)**: Observable log returns generated by the hidden states
3. **Parameters (θ)**: Model parameters governing state dynamics and emissions

#### State Space Model

```
Hidden States:    S₁ → S₂ → S₃ → ... → Sₜ
                   ↓    ↓    ↓         ↓
Observations:     O₁   O₂   O₃   ...  Oₜ
```

#### Model Parameters

- **Initial Probabilities (π)**: `P(S₁ = i)` - probability of starting in state i
- **Transition Matrix (A)**: `A[i,j] = P(Sₜ₊₁ = j | Sₜ = i)` - state transition probabilities  
- **Emission Parameters (θ)**: Parameters of observation distributions for each state

#### Gaussian Emissions

Each hidden state generates observations from a Gaussian distribution:

```
P(Oₜ | Sₜ = i) = N(μᵢ, σᵢ²)
```

Where:
- `μᵢ`: Mean return for regime i
- `σᵢ²`: Variance (volatility²) for regime i

### Why HMMs Work for Market Regimes

1. **State Persistence**: Markets tend to stay in regimes for extended periods (Markov property)
2. **Regime Switching**: Markets transition between distinct behavioral phases
3. **Observable Manifestations**: Different regimes produce different return/volatility patterns
4. **Uncertainty Handling**: Probabilistic framework captures regime uncertainty
5. **Temporal Dependencies**: Model captures how past regimes influence future states

### Market Regime Characteristics

| Regime | Mean Return | Volatility | Typical Duration | Market Condition |
|--------|-------------|------------|------------------|------------------|
| **Bull** | Positive (0.5-2% daily) | Moderate (1.5-2.5%) | Weeks to months | Rising prices, optimism |
| **Bear** | Negative (-1 to -3% daily) | High (2-4%) | Days to weeks | Falling prices, pessimism |
| **Sideways** | Near zero (-0.2 to 0.2%) | Low (1-2%) | Days to weeks | Range-bound, uncertainty |
| **Crisis** | Very negative (-3%+ daily) | Extreme (4%+) | Days | Market stress, panic |

---

## API Reference

### Core Classes

#### `HiddenMarkovModel`

The main class for HMM-based regime detection.

```python
from hidden_regime.models import HiddenMarkovModel, HMMConfig

# Initialize model
hmm = HiddenMarkovModel(n_states=3, config=HMMConfig())
```

**Constructor Parameters:**
- `n_states` (int): Number of hidden states/regimes (default: 3)
- `config` (HMMConfig, optional): Configuration object (default: HMMConfig())

**Key Attributes:**
- `n_states`: Number of states
- `config`: Configuration object  
- `is_fitted`: Whether model has been trained
- `initial_probs_`: Learned initial state probabilities (N,)
- `transition_matrix_`: Learned transition matrix (N, N)
- `emission_params_`: Learned emission parameters (N, 2) [means, stds]
- `training_history_`: List of log-likelihoods during training

#### Core Methods

##### `fit(returns, verbose=False)`

Train the HMM on historical returns data.

**Parameters:**
- `returns` (array-like): Log returns data
- `verbose` (bool): Print training progress

**Returns:** `self` (for method chaining)

**Raises:**
- `HMMTrainingError`: If training fails
- `HMMConvergenceError`: If model fails to converge

**Example:**
```python
import numpy as np
from hidden_regime.models import HiddenMarkovModel

# Generate sample returns
returns = np.random.normal(0.001, 0.02, 252)  # 1 year of daily returns

# Train model
hmm = HiddenMarkovModel(n_states=3)
hmm.fit(returns, verbose=True)
```

##### `predict(returns)`

Predict most likely state sequence using Viterbi algorithm.

**Parameters:**
- `returns` (array-like): Returns data for prediction

**Returns:**
- `states` (np.ndarray): Most likely state sequence

**Example:**
```python
# Predict regimes for new data
new_returns = np.random.normal(0.002, 0.025, 30)
states = hmm.predict(new_returns)
print(f"Predicted states: {states}")
```

##### `predict_proba(returns)`

Compute state probabilities for each observation.

**Parameters:**
- `returns` (array-like): Returns data

**Returns:**
- `probabilities` (np.ndarray): State probabilities (T, N)

**Example:**
```python
# Get state probabilities
probs = hmm.predict_proba(new_returns)
print(f"State probabilities shape: {probs.shape}")
print(f"Today's regime probabilities: {probs[-1]}")
```

##### `score(returns)`

Compute log-likelihood of returns under the model.

**Parameters:**
- `returns` (array-like): Returns data

**Returns:**
- `log_likelihood` (float): Model log-likelihood

**Example:**
```python
# Evaluate model fit
log_likelihood = hmm.score(returns)
print(f"Model log-likelihood: {log_likelihood}")
```


##### `get_current_regime_info()`

Get comprehensive information about current regime.

**Returns:**
- `info` (dict): Regime information including:
  - `current_state`: Most likely current state
  - `state_probabilities`: Probabilities for all states  
  - `regime_interpretation`: Human-readable regime description
  - `days_in_current_regime`: Estimated days in current regime
  - `expected_duration`: Expected total duration of current regime

**Example:**
```python
info = hmm.get_current_regime_info()
print(f"Current regime: {info['regime_interpretation']}")
print(f"Confidence: {info['state_probabilities'].max():.2%}")
```

##### `analyze_regimes(returns, include_dates=False, dates=None)`

Perform comprehensive regime analysis.

**Parameters:**
- `returns` (array-like): Returns data to analyze
- `include_dates` (bool): Whether to include date information
- `dates` (array-like, optional): Date index for returns

**Returns:**
- `analysis` (dict): Comprehensive regime analysis

**Example:**
```python
# Comprehensive regime analysis
analysis = hmm.analyze_regimes(returns, include_dates=True, dates=date_index)
for state, stats in analysis['regime_statistics'].items():
    print(f"Regime {state}: {stats['interpretation']}")
    print(f"  Mean return: {stats['mean_return']:.4f}")
    print(f"  Volatility: {stats['volatility']:.4f}")
    print(f"  Average duration: {stats['average_duration']:.1f} days")
```

##### Model Persistence

```python
# Save trained model
hmm.save('market_hmm.pkl')

# Load saved model
hmm_loaded = HiddenMarkovModel.load('market_hmm.pkl')
```

#### `HMMConfig`

Configuration class for HMM parameters.

**Key Parameters:**
- `n_states` (int): Number of hidden states (default: 3)
- `max_iterations` (int): Maximum training iterations (default: 100)
- `tolerance` (float): Convergence tolerance (default: 1e-6)
- `regularization` (float): Regularization parameter (default: 1e-6)
- `initialization_method` (str): 'random', 'kmeans', or 'custom' (default: 'random')
- `random_seed` (int, optional): Random seed for reproducibility
- `min_regime_duration` (int): Minimum regime duration in days (default: 2)

**Factory Methods:**

##### `HMMConfig.for_market_data(conservative=False)`

Create configuration optimized for market data.

**Parameters:**
- `conservative` (bool): Use conservative settings for stability

**Example:**
```python
# Market-optimized configuration
config = HMMConfig.for_market_data(conservative=True)
hmm = HiddenMarkovModel(n_states=3, config=config)
```

##### `HMMConfig.for_high_frequency()`

Create configuration for high-frequency data.

**Example:**
```python
# High-frequency optimized configuration  
config = HMMConfig.for_high_frequency()
hmm = HiddenMarkovModel(n_states=4, config=config)
```

#### `HMMAlgorithms`

Static methods for core HMM algorithms (primarily for internal use).

**Key Methods:**
- `forward_algorithm()`: Forward probabilities and likelihood
- `backward_algorithm()`: Backward probabilities  
- `viterbi_algorithm()`: Most likely state sequence
- `forward_backward_algorithm()`: State probabilities
- `baum_welch_update()`: Parameter updates
- `predict_next_state_probs()`: Next state predictions

#### Utility Functions

##### `validate_returns_data(returns)`
Validate and clean returns data.

##### `initialize_parameters_random(n_states, returns, seed=None)`
Random parameter initialization.

##### `initialize_parameters_kmeans(n_states, returns, random_state=None)` 
K-means based parameter initialization.

##### `check_convergence(log_likelihoods, tolerance, min_iterations=10)`
Check training convergence.

##### `get_regime_interpretation(state_idx, emission_params)`
Generate human-readable regime descriptions.

#### Exception Classes

- **`HMMTrainingError`**: Training failures
- **`HMMConvergenceError`**: Convergence failures  
- **`HMMInferenceError`**: Prediction/inference failures

---

## Usage Guide

### Basic Workflow

```python
import numpy as np
import pandas as pd
from hidden_regime.models import HiddenMarkovModel, HMMConfig

# 1. Load your returns data
returns = pd.read_csv('stock_returns.csv')['log_return'].values

# 2. Create and configure model
config = HMMConfig.for_market_data()
hmm = HiddenMarkovModel(n_states=3, config=config)

# 3. Train the model
hmm.fit(returns, verbose=True)

# 4. Analyze regimes
analysis = hmm.analyze_regimes(returns)
print(analysis['regime_statistics'])

# 5. Make predictions
new_data = np.array([-0.02, 0.01, 0.005])  # Recent returns
states = hmm.predict(new_data)
probs = hmm.predict_proba(new_data)
```

### Advanced Configuration

```python
# Custom configuration for specific use case
custom_config = HMMConfig(
    n_states=4,                    # 4 regimes: bull, bear, sideways, crisis
    max_iterations=200,            # Allow more training iterations
    tolerance=1e-8,                # Stricter convergence
    initialization_method='kmeans', # Use k-means initialization
    min_regime_duration=5,         # Regimes must last at least 5 days
    random_seed=42                 # Reproducible results
)

hmm = HiddenMarkovModel(n_states=4, config=custom_config)
```

---

## Mathematical Details

### Training Algorithm: Baum-Welch (EM)

The Baum-Welch algorithm iteratively maximizes the likelihood of observed data:

#### E-Step: Compute State Probabilities

**Forward probabilities** α(i,t):
```
α(i,t) = P(O₁,...,Oₜ, Sₜ=i | θ)
```

**Backward probabilities** β(i,t):
```
β(i,t) = P(Oₜ₊₁,...,Oₜ | Sₜ=i, θ)
```

**State probabilities** γ(i,t):
```
γ(i,t) = P(Sₜ=i | O₁,...,Oₜ, θ) = α(i,t)β(i,t) / P(O|θ)
```

#### M-Step: Update Parameters

**Initial probabilities**:
```
π̂ᵢ = γ(i,1)
```

**Transition probabilities**:
```
Â[i,j] = Σₜ ξ(i,j,t) / Σₜ γ(i,t)
```

**Emission parameters**:
```
μ̂ᵢ = Σₜ γ(i,t) Oₜ / Σₜ γ(i,t)
σ̂ᵢ² = Σₜ γ(i,t) (Oₜ - μ̂ᵢ)² / Σₜ γ(i,t)
```

### Numerical Stability

#### Log-Space Computations
All probability computations are performed in log-space to prevent numerical underflow:

```python
# Instead of: prob = α[i] * β[j] * emission
log_prob = log_α[i] + log_β[j] + log_emission
prob = np.exp(log_prob)
```

#### Log-Sum-Exp Trick
For stable computation of log(Σ exp(xᵢ)):

```python
def logsumexp(x):
    x_max = np.max(x)
    return x_max + np.log(np.sum(np.exp(x - x_max)))
```

#### Regularization
Small epsilon values prevent log(0) and degenerate parameters:

```python
# Prevent log(0)
log_prob = np.log(prob + 1e-10)

# Prevent zero variance
variance = max(variance, min_variance)
```

### Convergence Criteria

Training stops when the relative improvement in log-likelihood falls below threshold:

```
|L(θₖ₊₁) - L(θₖ)| / |L(θₖ)| < tolerance
```

Where L(θ) is the data log-likelihood under parameters θ.

### Regime Interpretation Logic

States are automatically interpreted based on emission parameters:

```python
def interpret_regime(mean_return, volatility):
    if mean_return > 0.001:  # > 0.1% daily
        regime = "Bull"
    elif mean_return < -0.001:  # < -0.1% daily  
        regime = "Bear"
    else:
        regime = "Sideways"
    
    # Add volatility characterization
    if volatility >= 0.03:  # >= 3% daily
        vol_desc = "High Vol"
    elif volatility <= 0.02:  # <= 2% daily
        vol_desc = "Low Vol"  
    else:
        vol_desc = "Moderate Vol"
    
    return f"{regime} ({vol_desc})"
```

---

## Examples

### Example 1: Basic Regime Detection

```python
import numpy as np
import matplotlib.pyplot as plt
from hidden_regime.models import HiddenMarkovModel

# Generate synthetic market data with regime changes
np.random.seed(42)
n_obs = 500

# Simulate regime-switching returns
regimes = np.concatenate([
    np.zeros(100),      # Bear regime
    np.ones(150),       # Sideways regime  
    np.full(150, 2),    # Bull regime
    np.zeros(100),      # Bear regime again
])

regime_params = {
    0: (-0.015, 0.035),  # Bear: negative return, high volatility
    1: (0.001, 0.015),   # Sideways: low return, low volatility
    2: (0.012, 0.025),   # Bull: positive return, moderate volatility
}

returns = []
for t in range(n_obs):
    regime = regimes[t]
    mean, std = regime_params[regime]
    returns.append(np.random.normal(mean, std))

returns = np.array(returns)

# Train HMM
hmm = HiddenMarkovModel(n_states=3)
hmm.fit(returns, verbose=True)

# Predict regimes
predicted_states = hmm.predict(returns)
state_probs = hmm.predict_proba(returns)

# Analyze results
analysis = hmm.analyze_regimes(returns)
print("Regime Analysis:")
for state, stats in analysis['regime_statistics'].items():
    print(f"State {state}: {stats['interpretation']}")
    print(f"  Mean return: {stats['mean_return']:.4f}")
    print(f"  Volatility: {stats['volatility']:.4f}")
    print(f"  Frequency: {stats['frequency']:.2%}")

# Plot results
plt.figure(figsize=(12, 8))

plt.subplot(3, 1, 1)
plt.plot(returns)
plt.title('Simulated Returns')
plt.ylabel('Log Return')

plt.subplot(3, 1, 2) 
plt.plot(regimes, label='True Regimes', alpha=0.7)
plt.plot(predicted_states, label='Predicted Regimes', alpha=0.7)
plt.title('Regime Comparison')
plt.ylabel('Regime')
plt.legend()

plt.subplot(3, 1, 3)
for i in range(3):
    plt.plot(state_probs[:, i], label=f'State {i}')
plt.title('State Probabilities')
plt.ylabel('Probability')
plt.xlabel('Time')
plt.legend()

plt.tight_layout()
plt.show()
```

### Example 2: Real Stock Data Analysis

```python
import yfinance as yf
import pandas as pd
from hidden_regime.models import HiddenMarkovModel, HMMConfig

# Download stock data
ticker = "SPY"
data = yf.download(ticker, start="2020-01-01", end="2023-12-31")

# Calculate log returns
data['log_return'] = np.log(data['Close'] / data['Close'].shift(1))
returns = data['log_return'].dropna().values

# Create market-optimized configuration
config = HMMConfig.for_market_data(conservative=True)
hmm = HiddenMarkovModel(n_states=3, config=config)

# Train model
print(f"Training HMM on {len(returns)} observations of {ticker}")
hmm.fit(returns, verbose=True)

# Comprehensive analysis
dates = data.index[1:]  # Exclude first date due to NaN log return
analysis = hmm.analyze_regimes(returns, include_dates=True, dates=dates)

print(f"\n{ticker} Regime Analysis:")
print("="*50)

for state, stats in analysis['regime_statistics'].items():
    print(f"\nRegime {state}: {stats['interpretation']}")
    print(f"  Mean daily return: {stats['mean_return']:.4f} ({stats['mean_return']*252:.2%} annualized)")
    print(f"  Daily volatility: {stats['volatility']:.4f} ({stats['volatility']*np.sqrt(252):.2%} annualized)")
    print(f"  Average duration: {stats['average_duration']:.1f} days")
    print(f"  Frequency: {stats['frequency']:.2%}")
    
    if 'periods' in stats:
        print(f"  Example periods:")
        for period in stats['periods'][:3]:  # Show first 3 periods
            start_date = period['start_date'].strftime('%Y-%m-%d')
            end_date = period['end_date'].strftime('%Y-%m-%d')
            print(f"    {start_date} to {end_date} ({period['duration']} days)")

# Current regime information
current_info = hmm.get_current_regime_info()
print(f"\nCurrent Regime Assessment:")
print(f"  Most likely regime: {current_info['regime_interpretation']}")
print(f"  Confidence: {current_info['state_probabilities'].max():.2%}")

# Recent regime probabilities
recent_probs = hmm.predict_proba(returns[-30:])  # Last 30 days
print(f"\nRecent 30-day regime probabilities:")
for i, regime_name in enumerate([stats['interpretation'] for stats in analysis['regime_statistics'].values()]):
    avg_prob = recent_probs[:, i].mean()
    print(f"  {regime_name}: {avg_prob:.2%}")
```

### Example 3: Portfolio Strategy Based on Regimes

```python
from hidden_regime.models import HiddenMarkovModel
import numpy as np
import pandas as pd

class RegimeBasedStrategy:
    def __init__(self, n_states=3, lookback_window=252):
        self.hmm = HiddenMarkovModel(n_states=n_states)
        self.lookback_window = lookback_window
        self.is_trained = False
        
    def train(self, returns):
        """Train the HMM on historical returns."""
        self.hmm.fit(returns)
        self.is_trained = True
        
    def get_position(self, recent_returns, current_return=None):
        """
        Determine position size based on current regime.
        
        Returns:
            float: Position size between -1 (short) and +1 (long)
        """
        if not self.is_trained:
            return 0.0

        # Get current regime info
        regime_info = self.hmm.get_current_regime_info()
        regime_desc = regime_info['regime_interpretation']
        confidence = regime_info['state_probabilities'].max()
        
        # Position sizing based on regime
        base_position = 0.0
        
        if 'Bull' in regime_desc:
            base_position = 1.0   # Full long position
        elif 'Bear' in regime_desc:
            base_position = -0.5  # Short position
        else:  # Sideways
            base_position = 0.2   # Small long bias
            
        # Adjust based on confidence
        confidence_multiplier = min(confidence / 0.7, 1.0)  # Scale by confidence
        
        # Adjust based on volatility
        if 'High Vol' in regime_desc:
            vol_multiplier = 0.6  # Reduce position in high volatility
        elif 'Low Vol' in regime_desc:
            vol_multiplier = 1.0  # Full position in low volatility
        else:
            vol_multiplier = 0.8  # Moderate reduction
            
        position = base_position * confidence_multiplier * vol_multiplier
        
        return np.clip(position, -1.0, 1.0)

# Example usage
strategy = RegimeBasedStrategy(n_states=3)

# Train on historical data (assuming we have returns)
historical_returns = np.random.normal(0.001, 0.02, 1000)  # Placeholder
strategy.train(historical_returns)

# Simulate live trading
portfolio_returns = []
positions = []

for i in range(100):  # Simulate 100 days of trading
    # Get current market return
    market_return = np.random.normal(0.001, 0.02)
    
    # Determine position based on regime
    recent_returns = historical_returns[-(30+i):-(i) if i > 0 else None]
    position = strategy.get_position(recent_returns, market_return)
    
    # Calculate portfolio return
    portfolio_return = position * market_return
    
    portfolio_returns.append(portfolio_return)
    positions.append(position)
    
    if i % 20 == 0:  # Print update every 20 days
        regime_info = strategy.hmm.get_current_regime_info()
        print(f"Day {i+1}: Regime={regime_info['regime_interpretation']}, "
              f"Position={position:.2f}, Return={portfolio_return:.4f}")

# Performance analysis
portfolio_returns = np.array(portfolio_returns)
print(f"\nStrategy Performance:")
print(f"Total return: {portfolio_returns.sum():.4f}")
print(f"Sharpe ratio: {portfolio_returns.mean() / portfolio_returns.std() * np.sqrt(252):.2f}")
print(f"Max drawdown: {np.minimum.accumulate(np.cumsum(portfolio_returns)).min():.4f}")
```

---

## Best Practices

### Model Configuration

1. **Choose appropriate number of states**:
   - Start with 3 states (bull/bear/sideways)
   - Use 4+ states only with sufficient data (1000+ observations)
   - More states require more data to avoid overfitting

2. **Initialization strategy**:
   - Use `'kmeans'` for better initial parameters
   - Use `'random'` with multiple seeds for robustness
   - Set `random_seed` for reproducible results

3. **Convergence settings**:
   - Increase `max_iterations` for complex models
   - Lower `tolerance` for more precise fitting
   - Enable `early_stopping` to prevent overtraining

### Data Preparation

1. **Returns calculation**:
   ```python
   # Use log returns for better statistical properties
   log_returns = np.log(prices[1:] / prices[:-1])
   
   # Remove extreme outliers
   returns_clean = returns[np.abs(returns) < 3 * returns.std()]
   ```

2. **Data quality**:
   - Ensure sufficient data (>= 100 observations per state)
   - Handle missing values appropriately
   - Check for data quality issues (spikes, gaps)

3. **Stationarity**:
   - HMMs assume regime-stationary data
   - Consider detrending if necessary
   - Be aware of structural breaks

### Training and Validation

1. **Cross-validation**:
   ```python
   # Walk-forward validation
   train_size = int(0.8 * len(returns))
   
   hmm.fit(returns[:train_size])
   test_score = hmm.score(returns[train_size:])
   ```

2. **Overfitting prevention**:
   - Use regularization for small datasets
   - Monitor out-of-sample likelihood
   - Compare models with different state numbers

3. **Convergence monitoring**:
   ```python
   # Check training history
   hmm.fit(returns, verbose=True)
   plt.plot(hmm.training_history_)
   plt.title('Training Convergence')
   ```

### Performance Optimization

1. **Numerical stability**:
   - Models handle log-space computations internally
   - Regularization prevents degenerate parameters
   - Robust to numerical precision issues

2. **Memory efficiency**:
   ```python
   # For large datasets, process in chunks
   chunk_size = 1000
   for i in range(0, len(returns), chunk_size):
       chunk = returns[i:i+chunk_size]
       predictions = hmm.predict(chunk)
   ```

3. **Speed considerations**:
   - K-means initialization is slower but often better
   - More states increase computational complexity O(N²T)
   - Model complexity affects inference speed

---

## Troubleshooting

### Common Issues

#### 1. Model Not Converging

**Symptoms:**
- Training doesn't reach convergence within max_iterations
- Log-likelihood oscillates or decreases
- Warning: "HMM did not converge"

**Solutions:**
```python
# Increase iterations and adjust tolerance
config = HMMConfig(
    max_iterations=200,
    tolerance=1e-8,
    regularization=1e-5
)

# Try different initialization
config.initialization_method = 'kmeans'

# Use more conservative settings
config = HMMConfig.for_market_data(conservative=True)
```

#### 2. Poor Regime Detection

**Symptoms:**
- Regimes change too frequently
- All states have similar parameters
- Low confidence in regime predictions

**Solutions:**
```python
# Increase minimum regime duration
config.min_regime_duration = 5

# Use fewer states
hmm = HiddenMarkovModel(n_states=2)

# Check data quality
plt.hist(returns, bins=50)
plt.title('Returns Distribution')
```

#### 3. Numerical Instability

**Symptoms:**
- NaN or infinite values in results
- "RuntimeWarning: divide by zero"
- Extremely large or small log-likelihoods

**Solutions:**
```python
# Increase regularization
config.regularization = 1e-4

# Check for extreme values in data
returns_clean = returns[np.isfinite(returns)]
returns_clean = returns_clean[np.abs(returns_clean) < 5 * np.std(returns_clean)]

# Use minimum variance constraint
config.min_variance = 1e-6
```

#### 4. Overfitting

**Symptoms:**
- Perfect fit on training data but poor out-of-sample performance
- Too many states for available data
- Degenerate parameter estimates

**Solutions:**
```python
# Reduce number of states
hmm = HiddenMarkovModel(n_states=2)

# Increase regularization
config.regularization = 1e-3

# Use more training data
# Validate on out-of-sample data
```

#### 5. Memory Issues

**Symptoms:**
- "MemoryError" with large datasets
- Slow performance with long time series

**Solutions:**
```python
# Process data in chunks for very large datasets
def process_large_dataset(returns, chunk_size=5000):
    results = []
    for i in range(0, len(returns), chunk_size):
        chunk = returns[i:i+chunk_size]
        states = hmm.predict(chunk)
        results.extend(states)
    return np.array(results)

# Consider data subsampling
sampled_returns = returns[::2]  # Every other observation
```

### Debugging Tools

#### 1. Training Diagnostics

```python
# Monitor training progress
hmm.fit(returns, verbose=True)

# Plot convergence
plt.plot(hmm.training_history_)
plt.xlabel('Iteration')
plt.ylabel('Log-likelihood')
plt.title('Training Convergence')

# Check final parameters
print("Transition Matrix:")
print(hmm.transition_matrix_)
print("\nEmission Parameters:")
print(hmm.emission_params_)
```

#### 2. Model Validation

```python
# Compare with random predictions
random_states = np.random.randint(0, hmm.n_states, len(returns))
model_ll = hmm.score(returns)
print(f"Model log-likelihood: {model_ll}")

# State persistence check
predicted_states = hmm.predict(returns)
state_changes = np.sum(predicted_states[1:] != predicted_states[:-1])
print(f"State changes: {state_changes} ({state_changes/len(returns)*100:.1f}%)")
```

#### 3. Parameter Interpretation

```python
# Analyze learned parameters
analysis = hmm.analyze_regimes(returns)

for state, stats in analysis['regime_statistics'].items():
    print(f"State {state}:")
    print(f"  Mean: {stats['mean_return']:.6f}")
    print(f"  Std:  {stats['volatility']:.6f}")
    print(f"  Interpretation: {stats['interpretation']}")
    
# Transition matrix analysis
print("\nTransition Matrix:")
print("From\\To  ", end="")
for j in range(hmm.n_states):
    print(f"State{j:2d}", end="  ")
print()

for i in range(hmm.n_states):
    print(f"State{i:2d}   ", end="")
    for j in range(hmm.n_states):
        print(f"{hmm.transition_matrix_[i,j]:6.3f}", end="  ")
    print()
```

### Performance Benchmarks

Typical performance characteristics:

- **Training time**: 1-10 seconds for 1000 observations (3 states)
- **Memory usage**: ~1MB per 10,000 observations
- **Convergence**: Usually within 20-50 iterations
- **Regime detection accuracy**: 70-85% for well-separated regimes

### Getting Help

1. **Check model configuration**:
   ```python
   print(hmm.config)
   ```

2. **Validate input data**:
   ```python
   print(f"Data shape: {returns.shape}")
   print(f"Data range: [{returns.min():.6f}, {returns.max():.6f}]")
   print(f"Missing values: {np.isnan(returns).sum()}")
   ```

3. **Test with synthetic data**:
   ```python
   # Use known ground truth for validation
   synthetic_returns = generate_synthetic_regimes()
   hmm.fit(synthetic_returns)
   # Compare predictions with known regimes
   ```

---

## References

1. **Rabiner, L. R. (1989)**. "A tutorial on hidden Markov models and selected applications in speech recognition." Proceedings of the IEEE, 77(2), 257-286.

2. **Hamilton, J. D. (1989)**. "A new approach to the economic analysis of nonstationary time series and the business cycle." Econometrica, 57(2), 357-384.

3. **Kim, C. J., & Nelson, C. R. (1999)**. "State-space models with regime switching: classical and Gibbs-sampling approaches with applications." MIT Press.

4. **Ang, A., & Bekaert, G. (2002)**. "Regime switches in interest rates." Journal of Business & Economic Statistics, 20(2), 163-182.

---

*For more information, see the main [Hidden Regime documentation](../../README.md) and [technical specifications](../../CLAUDE.md).*