# PRISM Output Files

This document describes all output files generated by the `prism run` command.

## Output Directory Structure

After running PRISM, your output directory will contain:

```text
output_dir/
├── corrected_peptides.parquet      # Final peptide quantities (LINEAR scale)
├── corrected_proteins.parquet      # Final protein quantities (LINEAR scale)
├── peptides_rollup.parquet         # Peptide quantities before normalization (LOG2 scale)
├── peptides_log2_internal.parquet  # Peptide quantities after normalization (LOG2 scale, internal use)
├── proteins_raw.parquet            # Protein quantities before normalization (LOG2 scale)
├── merged_data.parquet             # Merged transition-level data from all input files
├── merged_data.fingerprints.json   # Fingerprints for detecting when re-merge is needed
├── protein_groups.tsv              # Protein group definitions and peptide assignments
├── sample_metadata.tsv             # Sample metadata (auto-generated or merged from input)
├── metadata.json                   # Complete provenance and processing parameters
├── qc_report.html                  # HTML QC report with embedded diagnostic plots
├── qc_plots/                       # Directory containing PNG plot files (if enabled)
└── prism_run_YYYYMMDD_HHMMSS.log   # Detailed processing log
```

## Primary Output Files

These are the main files you will use for downstream analysis.

### corrected_peptides.parquet

**Purpose**: Final normalized and batch-corrected peptide quantities.

**Scale**: LINEAR (raw abundance values, not log-transformed)

**Columns**:

| Column | Type | Description |
|--------|------|-------------|
| `Peptide Modified Sequence Unimod Ids` | string | Peptide sequence with modifications in Unimod format, e.g., `PEPTIDEC(unimod:4)K` |
| `n_transitions` | int64 | Number of transitions used in rollup |
| `mean_rt` | double | Mean retention time (minutes) |
| `<sample_id>` | double | One column per sample containing peptide abundance (LINEAR scale) |

**Sample column naming**: Sample columns follow the format `<replicate_name>__@__<source_document>` to ensure uniqueness when the same replicate appears in multiple batches.

**Example**:

| Peptide Modified Sequence Unimod Ids | n_transitions | mean_rt | Sample1 | Sample2 | ... |
|--------------------------------------|---------------|---------|---------|---------|-----|
| PEPTIDEC(unimod:4)K | 6 | 15.2 | 125000 | 142000 | ... |
| ANOTHERPEPTIDER | 4 | 22.8 | 89000 | 95000 | ... |

---

### corrected_proteins.parquet

**Purpose**: Final normalized and batch-corrected protein quantities.

**Scale**: LINEAR (raw abundance values, not log-transformed)

**Columns**:

| Column | Type | Description |
|--------|------|-------------|
| `protein_group` | string | Protein group identifier (e.g., `PG0001`) |
| `leading_protein` | string | Full protein accession of the leading protein |
| `leading_name` | string | Full protein name from FASTA header |
| `leading_uniprot_id` | string | UniProt accession (extracted from FASTA header) |
| `leading_gene_name` | string | Gene name (extracted from FASTA header) |
| `leading_description` | string | Protein description from FASTA |
| `n_peptides` | int64 | Number of peptides used for quantification |
| `n_unique_peptides` | int64 | Number of unique (proteotypic) peptides |
| `low_confidence` | bool | True if protein has only shared peptides (no unique peptides) |
| `<sample_id>` | double | One column per sample containing protein abundance (LINEAR scale) |

**Example**:

| protein_group | leading_protein | leading_gene_name | n_peptides | Sample1 | Sample2 | ... |
|---------------|-----------------|-------------------|------------|---------|---------|-----|
| PG0001 | sp\|P12345\|PROT_HUMAN | GENE1 | 12 | 5200000 | 4800000 | ... |
| PG0002 | sp\|Q67890\|PROT2_HUMAN | GENE2 | 5 | 890000 | 920000 | ... |

---

### protein_groups.tsv

**Purpose**: Detailed protein group definitions showing peptide-to-protein assignments.

**Format**: Tab-separated values

**Columns**:

| Column | Type | Description |
|--------|------|-------------|
| `GroupID` | string | Protein group identifier (e.g., `PG0001`) |
| `LeadingProtein` | string | Full accession of the leading (most representative) protein |
| `LeadingUniProtID` | string | UniProt accession of leading protein |
| `LeadingGeneName` | string | Gene name of leading protein |
| `LeadingName` | string | Full protein name |
| `LeadingDescription` | string | Protein description |
| `MemberProteins` | string | All proteins in this group (semicolon-separated) |
| `SubsumedProteins` | string | Proteins whose peptides are a subset of this group |
| `NPeptides` | int | Total peptides mapped to this group |
| `NUniquePeptides` | int | Number of unique (proteotypic) peptides |
| `NRazorPeptides` | int | Number of razor peptides assigned to this group |
| `NAllMappedPeptides` | int | Total peptides that could map to this group |
| `UniquePeptides` | string | List of unique peptide sequences (semicolon-separated) |
| `RazorPeptides` | string | List of razor peptide sequences (semicolon-separated) |
| `AllPeptides` | string | List of all peptide sequences (semicolon-separated) |

---

## Intermediate Files

These files are useful for debugging, custom analysis, or re-running specific stages.

### merged_data.parquet

**Purpose**: Transition-level data merged from all input CSV files.

**Scale**: LINEAR (raw peak areas from Skyline)

**Columns**:

| Column | Type | Description |
|--------|------|-------------|
| `Protein` | string | Protein name from Skyline |
| `Protein Accession` | string | Protein accession |
| `Protein Gene` | string | Gene name |
| `Peptide` | string | Peptide sequence (plain) |
| `Peptide Modified Sequence Unimod Ids` | string | Peptide with modifications |
| `Precursor Charge` | int64 | Precursor ion charge state |
| `Precursor Mz` | double | Precursor m/z |
| `Isotope Dot Product` | double | Isotope pattern match score |
| `Detection Q Value` | double | Detection confidence (if available) |
| `Fragment Ion` | string | Fragment ion label (e.g., `y7`, `b3`) |
| `Product Charge` | int64 | Product ion charge state |
| `Product Mz` | double | Product m/z |
| `Area` | int64 | Peak area (raw abundance) |
| `Retention Time` | double | Apex retention time (minutes) |
| `Start Time` | double | Peak start time (minutes) |
| `End Time` | double | Peak end time (minutes) |
| `Fwhm` | double | Full width at half maximum |
| `Shape Correlation` | double | Peak shape quality score |
| `Coeluting` | bool | Whether peak is co-eluting with interference |
| `Truncated` | bool | Whether peak was truncated |
| `Replicate Name` | string | Sample replicate name |
| `File Name` | string | Raw data file name |
| `Total Ion Current Area` | int64 | Total ion current for the MS1 scan |
| `Acquired Time` | timestamp | Acquisition timestamp |
| `Batch` | string | Batch assignment |
| `Source Document` | string | Source CSV file name (used for batch assignment) |
| `Sample ID` | string | Unique sample identifier |

---

### peptides_rollup.parquet

**Purpose**: Peptide quantities after transition rollup, before normalization or batch correction.

**Scale**: LOG2

**Columns**: Same structure as `corrected_peptides.parquet`, but values are log2-transformed.

---

### peptides_log2_internal.parquet

**Purpose**: Peptide quantities after normalization and batch correction, in log2 scale. Used internally for protein rollup.

**Scale**: LOG2

**Note**: This is an internal file. For downstream analysis, use `corrected_peptides.parquet` instead.

---

### proteins_raw.parquet

**Purpose**: Protein quantities after peptide-to-protein rollup, before protein-level normalization or batch correction.

**Scale**: LOG2

**Columns**: Same structure as `corrected_proteins.parquet`, but values are log2-transformed.

---

## Metadata Files

### metadata.json

**Purpose**: Complete provenance information for reproducibility.

**Contents**:

```json
{
  "pipeline_version": "26.3.0",
  "processing_date": "2026-01-19T12:00:00+00:00",
  "source_files": [
    "/path/to/Plate1.csv",
    "/path/to/Plate2.csv"
  ],
  "metadata_files": ["/path/to/metadata.tsv"],
  "metadata_source": "auto_generated | user_provided",
  "processing_parameters": {
    "transition_rollup": {
      "method": "library-assisted | median_polish | sum",
      "min_transitions": 3,
      "use_ms1": false
    },
    "global_normalization": {
      "method": "rt_lowess | median | quantile | vsn | none"
    },
    "protein_rollup": {
      "method": "sum | median_polish | topn",
      "shared_peptide_handling": "all_groups | unique_only | razor"
    },
    "batch_correction": {
      "enabled": true,
      "method": "combat"
    }
  },
  "method_log": [
    "Merged and sorted 3 reports (19,805,912 rows)",
    "Transition rollup (library-assisted): 8,442 peptides, 238 samples",
    "RT-lowess normalization: frac=0.3, max correction=4.44 log2 (21.75x)",
    "Peptide ComBat: 3 batches corrected"
  ],
  "output_files": {
    "peptides": "output_dir/corrected_peptides.parquet",
    "proteins": "output_dir/corrected_proteins.parquet",
    "protein_groups": "output_dir/protein_groups.tsv"
  },
  "statistics": {
    "n_samples": 238,
    "n_peptides": 8442,
    "n_transitions": 67974,
    "n_proteins": 3643,
    "n_protein_groups": 3648
  }
}
```

**Re-running with provenance**: You can use `metadata.json` to re-run PRISM with identical parameters:

```bash
prism run -i new_data.csv -o new_output/ --from-provenance old_output/metadata.json
```

---

### sample_metadata.tsv

**Purpose**: Sample metadata used for normalization and batch correction.

**Columns**:

| Column | Type | Description |
|--------|------|-------------|
| `sample_id` | string | Unique sample identifier (column name in parquet files) |
| `sample` | string | Sample name (without batch suffix) |
| `sample_type` | string | Sample type: `experimental`, `reference`, or `qc` |
| `batch` | string | Batch assignment for batch correction |

**Sample types**:

- `reference`: Inter-experiment reference samples (pooled QC). Used for fitting normalization models.
- `qc`: Intra-experiment QC samples. Used for validation (held-out from fitting).
- `experimental`: Study samples (default if not matching reference/qc patterns).

---

### merged_data.fingerprints.json

**Purpose**: Stores fingerprints (file sizes, modification times, row counts) of input files to detect when re-merging is needed.

**Usage**: When running PRISM with `--force-reprocess`, this file is ignored. Otherwise, PRISM compares current input files against these fingerprints to skip merging if unchanged.

---

## QC Report Files

### qc_report.html

**Purpose**: Interactive HTML report with diagnostic plots for quality assessment.

**Contains**:

- Intensity distributions (box plots)
- PCA plots (before/after normalization)
- CV distributions for control samples
- Sample correlation heatmaps
- RT correction visualizations (if RT-lowess used)

### qc_plots/

**Purpose**: Directory containing individual PNG plot files (if `save_plots: true` in config).

**Contents**: Individual plot files that are also embedded in `qc_report.html`.

---

## Log Files

### prism_run_YYYYMMDD_HHMMSS.log

**Purpose**: Detailed processing log with timestamps.

**Contents**:

- Processing stages and timing
- Sample counts and batch assignments
- CV metrics before/after normalization
- Warnings and errors
- Method parameters

**Example**:

```text
2026-01-19 12:00:00 - skyline_prism.cli - INFO - Stage 1: Merge and Sort Input Data
2026-01-19 12:00:00 - skyline_prism.cli - INFO - ------------------------------------------------------------
2026-01-19 12:00:01 - skyline_prism.cli - INFO -   Merged 3 input files: 19,805,912 total rows
...
```

---

## Reading Parquet Files

### Python (pandas)

```python
import pandas as pd

# Read peptide data
peptides = pd.read_parquet("output_dir/corrected_peptides.parquet")

# Read protein data
proteins = pd.read_parquet("output_dir/corrected_proteins.parquet")

# Get sample columns (all columns except metadata)
metadata_cols = ["Peptide Modified Sequence Unimod Ids", "n_transitions", "mean_rt"]
sample_cols = [c for c in peptides.columns if c not in metadata_cols]
```

### R

```r
library(arrow)

# Read peptide data
peptides <- read_parquet("output_dir/corrected_peptides.parquet")

# Read protein data
proteins <- read_parquet("output_dir/corrected_proteins.parquet")
```

### DuckDB (SQL)

```sql
-- Query peptides directly from parquet
SELECT *
FROM read_parquet('output_dir/corrected_peptides.parquet')
WHERE "Peptide Modified Sequence Unimod Ids" LIKE '%PEPTIDE%';

-- Calculate CVs
SELECT
    "Peptide Modified Sequence Unimod Ids",
    AVG(value) as mean_abundance,
    STDDEV(value) / AVG(value) * 100 as cv_percent
FROM (
    SELECT "Peptide Modified Sequence Unimod Ids",
           UNNEST(values) as value
    FROM read_parquet('output_dir/corrected_peptides.parquet')
)
GROUP BY "Peptide Modified Sequence Unimod Ids";
```

---

## Scale Conventions

| File | Scale | Notes |
|------|-------|-------|
| `merged_data.parquet` | LINEAR | Raw peak areas from Skyline |
| `peptides_rollup.parquet` | LOG2 | After rollup, before normalization |
| `peptides_log2_internal.parquet` | LOG2 | Internal use only |
| `proteins_raw.parquet` | LOG2 | After rollup, before normalization |
| `corrected_peptides.parquet` | LINEAR | Final output (2^x conversion applied) |
| `corrected_proteins.parquet` | LINEAR | Final output (2^x conversion applied) |

**Important**: For CV calculations, always use LINEAR scale values (the final output files).

---

## Common Analysis Workflows

### Export to CSV for Excel

```python
import pandas as pd

proteins = pd.read_parquet("output_dir/corrected_proteins.parquet")
proteins.to_csv("proteins_for_excel.csv", index=False)
```

### Filter to Specific Proteins

```python
import pandas as pd

proteins = pd.read_parquet("output_dir/corrected_proteins.parquet")
genes_of_interest = ["ALB", "APOA1", "HP"]
filtered = proteins[proteins["leading_gene_name"].isin(genes_of_interest)]
```

### Calculate Sample CVs

```python
import pandas as pd
import numpy as np

peptides = pd.read_parquet("output_dir/corrected_peptides.parquet")

# Get sample columns
metadata_cols = ["Peptide Modified Sequence Unimod Ids", "n_transitions", "mean_rt"]
sample_cols = [c for c in peptides.columns if c not in metadata_cols]

# Calculate CV per peptide (already in linear scale)
sample_data = peptides[sample_cols]
cv = (sample_data.std(axis=1) / sample_data.mean(axis=1)) * 100
print(f"Median CV: {cv.median():.1f}%")
```

