# PANICLE Output File Formats

This document describes all output files generated by PANICLE GWAS analyses.

## Directory Structure

When you run a GWAS analysis, PANICLE creates the following files in your specified output directory:

```
output_dir/
├── GWAS_{trait}_all_results.csv                 # Full results for each trait
├── GWAS_{trait}_significant.csv                    # Significant markers only
├── GWAS_{trait}_{method}_manhattan.png             # Manhattan plot per method
├── GWAS_{trait}_{method}_qq.png                    # QQ plot per method
└── GWAS_summary_by_traits_methods.csv              # Overall summary table
```

Example for analyzing Height with MLM and GLM:
```
my_results/
├── GWAS_Height_all_results.csv
├── GWAS_Height_significant.csv
├── GWAS_Height_GLM_manhattan.png
├── GWAS_Height_GLM_qq.png
├── GWAS_Height_MLM_manhattan.png
├── GWAS_Height_MLM_qq.png
└── GWAS_summary_by_traits_methods.csv
```

**Note:** Full results files are written as plain CSV by default. You can gzip them yourself if disk space is a concern.

---

## File Formats

### 1. Full Results CSV: `GWAS_{trait}_all_results.csv`

Contains association statistics for **all markers** across **all methods** run for a trait.

#### Column Descriptions

| Column | Type | Description |
|--------|------|-------------|
| `MARKER` | string | Canonical marker ID/name |
| `SNP` | string | Legacy alias of `MARKER` (included for compatibility) |
| `CHROM` | string | Chromosome name (as in VCF file, e.g., 'Chr01', '1') |
| `POS` | integer | Physical position on chromosome (base pairs) |
| `REF` | string | Reference allele |
| `ALT` | string | Alternative allele |
| `MAF` | float | Minor allele frequency (0.0 to 0.5) |
| `{METHOD}_P` | float | P-value from method (e.g., `GLM_P`, `MLM_P`) |
| `{METHOD}_Effect` | float | Effect size (beta coefficient) from method |

#### Notes on Columns:
- **CHROM/POS**: Match the original genotype file format
  - VCF files: Uses chromosome names as-is (e.g., 'Chr01', 'chr1', '1')
  - HapMap: Uses chromosome identifiers from the file
- **MAF**: Calculated from the aligned samples used in analysis
- **Method columns**: One `{METHOD}_P` and `{METHOD}_Effect` pair per standard association method
  - If you ran `['GLM', 'MLM']`, you'll have: `GLM_P`, `GLM_Effect`, `MLM_P`, `MLM_Effect`
  - Method names match run options (e.g., `MLM_Hybrid_P`, `FarmCPU_P`, `BLINK_P`)
  - Resampling output is written separately (see FarmCPU Resampling section)

#### Example File Content:

```csv
MARKER,CHROM,POS,SNP,REF,ALT,MAF,GLM_P,GLM_Effect,MLM_P,MLM_Effect
rs123,Chr01,100234,rs123,A,G,0.23,0.0000012,0.045,0.0000089,0.042
rs456,Chr01,150987,rs456,C,T,0.45,0.234,0.0012,-0.189,0.0009
rs789,Chr02,50432,rs789,G,A,0.12,0.045,-0.023,0.067,-0.019
```

#### Reading in Python:

```python
import pandas as pd

# Load results
results = pd.read_csv('my_results/GWAS_Height_all_results.csv')

# Get top 10 markers by MLM p-value
top_markers = results.nsmallest(10, 'MLM_P')

# Get all significant markers (Bonferroni)
bonferroni = 0.05 / len(results)
sig_markers = results[results['MLM_P'] < bonferroni]

# Extract specific columns
marker_info = results[['MARKER', 'CHROM', 'POS', 'MLM_P', 'MLM_Effect']]
```

---

### 2. Significant Results CSV: `GWAS_{trait}_significant.csv`

Contains only markers that passed the significance threshold.

#### Column Descriptions

Same columns as `all_results.csv`, plus:

| Column | Type | Description |
|--------|------|-------------|
| `Method` | string | Methods that detected this marker as significant (pipe-separated) |

#### Notes:
- Each marker appears once; the `Method` column lists all significant methods separated by `|`
- Threshold used is shown in the summary file
- Default threshold: Bonferroni correction (0.05 / n_markers)

#### Example:

```csv
MARKER,CHROM,POS,SNP,REF,ALT,MAF,GLM_P,GLM_Effect,MLM_P,MLM_Effect,Method
rs123,Chr01,100234,rs123,A,G,0.23,0.0000012,0.045,0.0000089,0.042,GLM|MLM
rs456,Chr05,892341,rs456,T,C,0.31,0.089,0.012,0.0000023,0.051,MLM
```

---

### 3. Summary CSV: `GWAS_summary_by_traits_methods.csv`

Overall summary table for all traits and methods analyzed.

#### Column Descriptions

| Column | Type | Description |
|--------|------|-------------|
| `Trait` | string | Trait name |
| `Method` | string | GWAS method used |
| `Significant_Hits` | integer | Number of markers passing threshold |
| `Threshold` | float | P-value threshold used (if applicable) |
| `Lambda_GC` | float | Genomic inflation factor (λ). Values > 1.0 may indicate population stratification |
| `N_Samples` | integer | Number of samples analyzed for this trait (after QC) |
| `N_Markers` | integer | Number of markers tested |
| `Runtime_Seconds` | float | Analysis time in seconds for this trait |
| `Info` | string | Additional method-specific information |

#### Example:

```csv
Trait,Method,Significant_Hits,Threshold,Lambda_GC,N_Samples,N_Markers,Runtime_Seconds,Info
Height,GLM,45,2.94e-07,1.02,850,500000,12.5,Bonferroni (markers)
Height,MLM,32,2.94e-07,0.98,850,500000,45.2,Bonferroni (markers)
FloweringTime,GLM,12,2.94e-07,1.15,848,500000,11.8,Bonferroni (markers)
```

**Notes on Lambda_GC:**
- λ ≈ 1.0: Well-controlled for population structure
- λ > 1.1: May indicate population stratification - consider using MLM or more PCs
- λ > 1.3: Likely significant population stratification - a warning is printed during analysis

---

## Visualization Outputs

### 4. Manhattan Plot: `GWAS_{trait}_{method}_manhattan.png`

Standard Manhattan plot showing -log₁₀(P-value) across the genome.

**Features:**
- Alternating colors by chromosome
- Horizontal red line: Bonferroni significance threshold
- X-axis: Genomic position
- Y-axis: -log₁₀(P-value)

**Resolution:**
- Default: 300 DPI
- Suitable for publication

**Customization:** See `PANICLE_Report()` in [api_reference.md](api_reference.md)

---

### 5. QQ Plot: `GWAS_{trait}_{method}_qq.png`

Quantile-quantile plot comparing observed vs. expected P-values.

**Features:**
- Red diagonal line: Expected distribution under null hypothesis
- Points above line: Potential true associations or population structure
- Text annotation: Genomic inflation factor (λ_GC)

**Interpretation:**
- λ_GC ≈ 1.0: Well-controlled for population structure
- λ_GC > 1.05: May indicate population stratification or polygenic signal
- λ_GC < 0.95: May indicate overdispersion correction

---

## Method-Specific Outputs

### FarmCPU Resampling

When using the `FarmCPUResampling` method, additional output is generated:

#### `GWAS_{trait}_FarmCPUResampling_RMIP.csv`

Contains Resample Model Inclusion Probability (RMIP) for each marker.

| Column | Type | Description |
|--------|------|-------------|
| `MARKER` | string | Marker ID |
| `SNP` | string | Legacy alias of `MARKER` |
| `CHROM` | string | Chromosome |
| `POS` | integer | Position |
| `RMIP` | float | Fraction of resampling runs where marker was selected (0.0 to 1.0) |
| `Trait` | string | Trait name |
| `ClusterMembers` | string | Cluster membership details (only when clustering is enabled) |

**Example:**
```csv
MARKER,CHROM,POS,SNP,RMIP,Trait
rs123,Chr01,100234,rs123,0.95,FloweringTime
rs456,Chr05,892341,rs456,0.72,FloweringTime
```

**Notes:**
- RMIP results are written to a separate file and are not included in `GWAS_{trait}_all_results.csv`.
- Markers are counted as significant for `GWAS_{trait}_significant.csv` when `RMIP >= 0.1`.
- Resampling uses the FarmCPU QTN threshold by default unless a resampling-specific threshold is provided.

**Interpretation:**
- RMIP > 0.8: Strong evidence for association
- RMIP 0.5-0.8: Moderate evidence
- RMIP < 0.5: Weak evidence

---

## Intermediate Files

PANICLE may create cached files to speed up repeated analyses:

### Genotype Cache Files

When loading genotype files, PANICLE creates binary cache files (v2, pre-imputed):

```
{genotype_filename}.panicle.v2.geno.npy  # Genotype matrix (binary, pre-imputed)
{genotype_filename}.panicle.v2.ind.txt   # Individual IDs
{genotype_filename}.panicle.v2.map.csv   # Genetic map
```

**Notes:**
- Automatically created on first load (VCF/BCF/PLINK/HapMap/CSV/TSV)
- Dramatically speeds up subsequent loads
- Safe to delete (will be regenerated)
- Can be large (similar size to original VCF)

**Example:**
```
genotypes.vcf.gz                      # Original (21 MB)
genotypes.vcf.gz.panicle.v2.geno.npy   # Cached genotypes (120 MB)
genotypes.vcf.gz.panicle.v2.ind.txt    # Cached IDs (6 KB)
genotypes.vcf.gz.panicle.v2.map.csv    # Cached map (5.4 MB)
```

---

## Working with Output Files

### Python Examples

#### Load and Filter Results

```python
import pandas as pd
import numpy as np

# Load results
df = pd.read_csv('results/GWAS_Height_all_results.csv')

# Filter by chromosome
chr1_markers = df[df['CHROM'] == 'Chr01']

# Filter by MAF
common_markers = df[df['MAF'] > 0.05]

# Get significant markers
bonferroni = 0.05 / len(df)
sig_markers = df[df['MLM_P'] < bonferroni]

# Sort by p-value
df_sorted = df.sort_values('MLM_P')

# Get top 100 markers
top100 = df.nsmallest(100, 'MLM_P')
```

#### Compare Methods

```python
# Load results from multiple methods
df = pd.read_csv('results/GWAS_Height_all_results.csv')

# Find markers significant in MLM but not GLM
bonf = 0.05 / len(df)
mlm_only = df[(df['MLM_P'] < bonf) & (df['GLM_P'] >= bonf)]

# Calculate correlation between methods
from scipy.stats import spearmanr

# Use -log10(p) for correlation
log_glm = -np.log10(df['GLM_P'])
log_mlm = -np.log10(df['MLM_P'])
corr, pval = spearmanr(log_glm, log_mlm)
print(f"Correlation: {corr:.3f}")
```

#### Create Custom Plots

```python
import matplotlib.pyplot as plt

df = pd.read_csv('results/GWAS_Height_all_results.csv')

# Simple Manhattan plot
log_p = -np.log10(df['MLM_P'])
plt.figure(figsize=(12, 4))
plt.scatter(range(len(df)), log_p, s=2, alpha=0.5)
plt.axhline(-np.log10(0.05/len(df)), color='red', linestyle='--')
plt.xlabel('Marker Index')
plt.ylabel('-log10(P)')
plt.title('Manhattan Plot - Height MLM')
plt.tight_layout()
plt.savefig('custom_manhattan.png', dpi=300)
```

---

### R Examples

#### Load Results in R

```r
# Load results
results <- read.csv('results/GWAS_Height_all_results.csv')

# Filter significant markers
bonferroni <- 0.05 / nrow(results)
sig_markers <- results[results$MLM_P < bonferroni, ]

# Top 20 markers
top20 <- results[order(results$MLM_P), ][1:20, ]

# Summary statistics
summary(results$MLM_P)
summary(results$MAF)
```

#### Create QQ Plot

```r
library(ggplot2)

# Calculate expected and observed -log10(P)
observed <- sort(-log10(results$MLM_P))
expected <- -log10(ppoints(length(observed)))

qq_data <- data.frame(expected = expected, observed = observed)

# Plot
ggplot(qq_data, aes(x = expected, y = observed)) +
  geom_point(alpha = 0.5, size = 1) +
  geom_abline(intercept = 0, slope = 1, color = 'red', linetype = 'dashed') +
  labs(x = 'Expected -log10(P)', y = 'Observed -log10(P)', title = 'QQ Plot') +
  theme_bw()
```

---

## File Size Considerations

Typical file sizes for different analysis scales:

| Dataset | Markers | Samples | all_results.csv | Plots | Total |
|---------|---------|---------|-----------------|-------|-------|
| Small | 10K | 100 | ~5 MB | ~1 MB | ~6 MB |
| Medium | 100K | 500 | ~50 MB | ~2 MB | ~52 MB |
| Large | 1M | 1000 | ~500 MB | ~3 MB | ~503 MB |

**Recommendations:**
- For large datasets, consider:
  - Using `outputs=['significant_marker_pvalues']` only
  - Filtering low-MAF variants before analysis
  - Compressing result CSVs: `gzip GWAS_*_all_results.csv`

---

## Troubleshooting

### Problem: Empty significant results file
**Cause:** No markers passed the significance threshold
**Solution:** Check the threshold in summary file. Consider:
- Using less stringent threshold: `significance=1e-5`
- Checking if analysis ran correctly (QQ plot, lambda GC)
- Verifying trait has genetic variation

### Problem: Very large all_results.csv files
**Cause:** Many markers and/or many methods
**Solution:**
- Compress files: `gzip *.csv`
- Use selective outputs: `outputs=['significant_marker_pvalues', 'manhattan']`
- Pre-filter genotypes by MAF before analysis

### Problem: Column names different than expected
**Cause:** Chromosome naming varies by file format
**Solution:**
- VCF files: Use `CHROM` and `POS` (matches VCF spec)
- Check actual column names: `pd.read_csv(...).columns`
- Use `df['CHROM']` not `df['Chr']`

---

## See Also

- [Quick Start Guide](quickstart.md)
- [API Reference](api_reference.md)
- [Examples](../examples/)
