# Confidence Score Determination in ICVision

## Overview

ICVision uses confidence scores to quantify the certainty of component classifications during Independent Component Analysis (ICA) artifact detection. These scores play a crucial role in determining which components should be automatically excluded from EEG/MEG data.

## How Confidence Scores are Generated

### External Generation via OpenAI Vision API

Confidence scores in ICVision are **not calculated internally** but are generated by the OpenAI Vision API (GPT-4 Vision model). When a component is analyzed:

1. ICVision creates visualization plots for each ICA component showing:
   - Spatial topography (sensor distribution)
   - Time series data
   - Power spectral density
   - Epochs image (if available)

2. These images are sent to the OpenAI API with a detailed prompt requesting classification into categories:
   - `brain` - Neural activity
   - `eye` - Eye movements/blinks
   - `muscle` - Muscle artifacts
   - `heart` - Cardiac artifacts
   - `line_noise` - Power line interference
   - `channel_noise` - Bad channel artifacts
   - `other_artifact` - Other artifactual components

3. The API returns a structured JSON response containing:
   ```json
   {
     "label": "eye",
     "confidence": 0.95,
     "reason": "Component shows classic eye blink topography with frontal distribution..."
   }
   ```

### Confidence Score Range

- **Valid Range**: 0.0 to 1.0
- **Interpretation**: 
  - 0.0 = No confidence in classification
  - 1.0 = Complete certainty in classification
- **Validation**: Scores are automatically clamped to [0.0, 1.0] range if outside bounds

## Factors Influencing Confidence Scores

The OpenAI model determines confidence based on multiple visual features:

### 1. Spatial Topography Patterns
- **Clear artifact patterns** increase confidence (e.g., frontal distribution for eye artifacts)
- **Ambiguous or mixed patterns** decrease confidence
- **Bilateral symmetry** for certain artifacts (e.g., eye blinks) increases confidence

### 2. Time Series Characteristics
- **Consistent temporal patterns** (e.g., rhythmic eye blinks) increase confidence
- **Characteristic waveforms** (e.g., sharp spikes for muscle artifacts) increase confidence
- **Mixed or unclear signals** decrease confidence

### 3. Power Spectrum Features
- **Expected frequency profiles** increase confidence:
  - 1/f distribution for brain components
  - Specific peaks for line noise (50/60 Hz)
  - High-frequency content for muscle artifacts
- **Atypical frequency content** decreases confidence

### 4. Cross-Trial Consistency
- **Stable patterns across epochs** increase confidence
- **Variable or inconsistent patterns** decrease confidence

## How Confidence Scores are Used

### Automatic Exclusion Logic

Components are automatically marked for exclusion when ALL conditions are met:

1. `auto_exclude` parameter is `True` (default)
2. Component label is in the exclusion list (default: all non-brain labels)
3. **Confidence score ≥ confidence_threshold** (default: 0.8)

```python
exclude_component = (
    auto_exclude and 
    label in labels_to_exclude and 
    confidence >= confidence_threshold
)
```

### Default Configuration

From the default configuration:
- **confidence_threshold**: 0.8 (80%)
- **auto_exclude**: True
- **labels_to_exclude**: ["eye", "heart", "muscle", "line_noise", "channel_noise", "other_artifact"]

This means artifacts need at least 80% confidence to be automatically excluded.

## Adjusting Confidence Behavior

### 1. Changing the Threshold

Lower threshold (e.g., 0.7) - More aggressive artifact removal:
```python
results = icvision.label_ica_components(
    ica=ica,
    confidence_threshold=0.7
)
```

Higher threshold (e.g., 0.9) - More conservative artifact removal:
```python
results = icvision.label_ica_components(
    ica=ica,
    confidence_threshold=0.9
)
```

### 2. Category-Specific Thresholds

Set different confidence thresholds for different artifact categories:

**Python API:**
```python
results = icvision.label_ica_components(
    ica=ica,
    confidence_threshold=0.8,  # Default for all categories
    category_confidence_thresholds={
        "muscle": 0.99,  # Very high threshold for muscle artifacts
        "eye": 0.7,      # Lower threshold for eye artifacts
        "heart": 0.85    # Medium-high threshold for heart artifacts
    }
)
```

**Command Line:**
```bash
autoclean-icvision data.set --category-thresholds muscle:0.99 eye:0.7 heart:0.85
```

Category-specific thresholds override the general `confidence_threshold` for their respective categories. Categories not specified in `category_confidence_thresholds` will use the general threshold.

### 3. Manual Override

Disable automatic exclusion to review all components manually:
```python
results = icvision.label_ica_components(
    ica=ica,
    auto_exclude=False
)
```

## API Configuration Impact

### Model Temperature
- ICVision uses `temperature=0.2` for the OpenAI API
- Lower temperature → More deterministic/consistent classifications
- Higher confidence scores tend to correlate with lower temperature settings

### Model Selection
- Uses GPT-4 Vision (gpt-4-vision-preview)
- Model improvements over time may affect confidence calibration

## Validation and Error Handling

### Input Validation
- Confidence scores from API are validated to be numeric
- Out-of-range values are clamped to [0.0, 1.0]

### Fallback Behavior
If API parsing fails:
- Default classification: `"other_artifact"`
- Default confidence: `1.0`
- Error reason is logged

## Best Practices

1. **Review Low-Confidence Classifications**: Components with confidence < 0.7 may benefit from manual review

2. **Dataset-Specific Tuning**: Adjust threshold based on your data quality:
   - Clean data → Lower threshold (0.7-0.8)
   - Noisy data → Higher threshold (0.85-0.95)

3. **Validation Sets**: Compare automatic exclusions with expert labels to calibrate threshold

4. **Cost Considerations**: Higher confidence thresholds may require manual review of more components but ensure fewer false positives

## Output and Reporting

Confidence scores are:
- Stored in the results DataFrame
- Saved to CSV files with 2 decimal precision
- Displayed in PDF reports alongside classifications
- Integrated into MNE's ICA object via `labels_scores_` attribute

## Technical Implementation

Key files and functions:
- `src/icvision/api.py`: Handles API communication and response parsing
- `src/icvision/config.py`: Default confidence threshold configuration
- `src/icvision/utils.py`: Validation of confidence score ranges

The confidence score is extracted directly from the OpenAI API response without modification, ensuring transparency in the classification process.