abaudit — Experiment Validity Report

## Key Numbers

0.0003
p-value (primary)
0.89
PPV — prob. effect is real
0.95
Statistical power
0.163
Cohen's d
2,000
Total sample size
3
Metrics tested

Bias score: 15% Low concern

## Validity Checks

Check | Result | Status
--- | --- | ---
p-value (primary) | 0.0003 | ✔
p-value (Bonferroni corrected) | 0.0009 | ✔
PPV — prob. effect is real | 0.89 | ✔
Statistical power | 0.95 | ✔
Sample Ratio Mismatch | p = 0.371 | ✔
Metrics tested | 3 | ⚠
Optional stopping | eff. α = 0.226 | ⚠
Effect size (Cohen's d) | 0.163 | ✔

## ⚠ Warnings

⚠ 3 metrics tested — Bonferroni-corrected p = 0.0009 (raw p = 0.0003).
⚠ Optional stopping risk: p-value checked 5 times. Effective α ≈ 0.226 (nominal: 0.05).

## 💡 Recommendations

💡 Use sequential testing (SPRT) or an alpha-spending function when interim looks are necessary.

## Parameters Used

Prior f | 0.3
Alpha (α) | 0.05
n control | 980
n treatment | 1,020
Corrected p-value | 0.0009
Effective alpha | 0.2262