# Analysis Workflows

RetroCast includes a suite of analysis scripts designed for rigorous statistical evaluation. These tools do not merely calculate averages; they use bootstrapping and Monte Carlo simulations to quantify uncertainty and compare models reliably.

All scripts are located in the `scripts/` directory and should be executed using `uv run`.

## 1. Directory Structure Assumptions

These scripts rely on the standard RetroCast data directory structure. Ensure your data follows this layout:

*   **`data/1-benchmarks/definitions/`**: Contains `.json.gz` benchmark definitions.
*   **`data/1-benchmarks/stocks/`**: Contains `.txt` stock files (one SMILES per line).
*   **`data/4-scored/`**: Contains scored predictions (output of `retrocast score`).
    *   Format: `data/4-scored/<benchmark>/<model>/<stock>/evaluation.json.gz`
*   **`data/6-comparisons/`**: The output directory for the visualizations generated by these scripts.

## 2. General Model Comparison (`02-compare.py`)

This script generates a comprehensive visual comparison of multiple models on a single benchmark. It produces interactive Plotly HTML files for Overall and Stratified Solvability and Top-K accuracy.

**Usage:**

```bash
uv run scripts/02-compare.py \
    --benchmark stratified-linear-600 \
    --models dms-explorer-xl aizynthfinder-mcts retro-star \
    --stock n5-stock
```


## 3. Paired Hypothesis Testing (`03-compare-paired.py`)

Comparing overlapping confidence intervals is insufficient for determining if one model outperforms another. This script performs a paired difference test using bootstrap resampling. It calculates the 95% confidence interval of the difference ($\Delta = \text{Challenger} - \text{Baseline}$) for every target in the benchmark.

If the 95% CI for the difference does not contain zero, the performance difference is statistically significant.

**Usage:**

```bash
uv run scripts/03-compare-paired.py \
    --benchmark stratified-linear-600 \
    --baseline dms-deep \
    --challengers dms-flash dms-wide \
    --n-boot 10000
```

**Output:**
A rich-formatted table in the terminal showing the mean difference, the confidence interval, and a significance flag (✅) for Solvability and Top-K metrics.

## 4. Probabilistic Ranking (`04-rank.py`)

Ranking models by a single scalar value (e.g., "Model A is #1 because it has 45.2% accuracy vs Model B's 45.1%") is often misleading due to statistical noise. This script performs a Monte Carlo simulation to determine the probability that a specific model is the true "winner" of the benchmark.

It resamples the dataset 10,000 times, ranks the models in each sample, and aggregates the results.

**Usage:**

```bash
uv run scripts/04-rank.py \
    --benchmark stratified-linear-600 \
    --models dms-explorer-xl dms-flash dms-wide \
    --metric top-1
```

**Output:**
*   **Terminal:** A table showing the "Expected Rank" and the probability of each model achieving 1st place.
*   **File:** `ranking_heatmap_top-1.html`, visualizes the rank probability distribution.

## 5. Pairwise Tournament (`05-tournament.py`)

This script runs a round-robin tournament where every model is compared against every other model using the paired difference test described in script `03`. This is useful for identifying non-transitive relationships or establishing a hierarchy in a large field of models.

**Usage:**

```bash
uv run scripts/05-tournament.py \
    --benchmark stratified-linear-600 \
    --models dms-explorer-xl dms-flash dms-wide aizynthfinder-mcts \
    --metric top-1
```

**Output:**
*   **Terminal:** A matrix table where cell $(i, j)$ shows the performance difference of Model $i$ minus Model $j$. Significant wins are highlighted green; losses are red.
*   **File:** `pairwise_matrix_top-1.html`, an interactive heatmap of the tournament results.

## 6. Seed Stability Analysis (`06-check-seed-stability.py`)

When creating subsets of benchmarks (e.g., creating a 100-route test set from a larger pool), the choice of random seed can influence the difficulty of the resulting set. This script analyzes a model's performance across many different random seeds of the same benchmark class. It generates a forest plot to visualize the variance in measured performance caused by dataset selection.

**Usage:**

```bash
uv run scripts/06-check-seed-stability.py \
    --model dms-explorer-xl \
    --base-benchmark stratified-linear-600 \
    --stock n5-stock \
    --seeds 42 299792458 19910806 17760704 20251030
```

**Output:**
*   **Terminal:** Z-scores for each seed, identifying which seed produces the most "representative" (closest to mean) difficulty.
*   **File:** `seed_stability.html` (in `data/7-meta-analysis/`), a forest plot showing the metric distribution across seeds.

## 7. Model Profiling (`07-create-model-profile.py`)

While script `02` compares many models on one benchmark, this script compares one model across many benchmarks. It is useful for profiling a model's sensitivity to route type (linear vs. convergent) or route length.

**Usage:**

```bash
uv run scripts/07-create-model-profile.py \
    --model dms-explorer-xl \
    --benchmarks ref-lin-600 ref-cnv-400 mkt-lin-500
```

**Output:**
Standard comparison plots saved to `data/6-comparisons/<model-name>/`, where the x-axis or legend groups represent different benchmarks rather than different models.
