# Scoring Guide

## How Your Predictions Are Scored

Every 15 minutes, your model predicts the **expected 1-hour forward return**
for each trading pair (BTC, ETH). When that 1-hour window expires, we score:

```
score = expected_return × actual_return
```

That's it. This simple formula rewards:

| Scenario | Expected | Actual | Score | Meaning |
|----------|----------|--------|-------|---------|
| ✅ Correct direction, strong conviction | +0.005 | +0.003 | +0.000015 | You predicted up, it went up |
| ✅ Correct direction, weak conviction | +0.001 | +0.003 | +0.000003 | Right direction but timid |
| ❌ Wrong direction | +0.005 | -0.003 | -0.000015 | You predicted up, it went down |
| 🤷 No opinion | 0.0 | +0.003 | 0.0 | Safe but no upside |

### Key Insights

1. **Direction is binary — get this right first.** If you can predict whether
   the next 5 minutes will be up or down with >50% accuracy, you'll be profitable.

2. **Magnitude is the multiplier.** Two models with 55% accuracy will have very
   different PnL if one bets 10× harder on high-conviction signals.

3. **Wrong direction is punished symmetrically.** A prediction of +0.01 that
   results in -0.01 actual costs you 0.0001. Going to zero (no prediction) would
   have cost nothing.

4. **Small scores add up.** Individual scores are tiny (1e-5 to 1e-4), but with
   predictions every 15 minutes, you make ~96 predictions per day per symbol.

## Leaderboard Metrics

Your model's leaderboard position uses several metrics computed over rolling windows:

### Primary Metrics

| Metric | What it measures | Good values |
|--------|-----------------|-------------|
| **mean_return** | Average score per prediction | > 0 |
| **hit_rate** | Fraction of predictions with correct direction | > 0.50 |
| **IC** (Information Coefficient) | Rank correlation between predictions and actuals | > 0.02 |
| **Sortino Ratio** | Risk-adjusted return (penalizes downside only) | > 0.5 |
| **Max Drawdown** | Worst peak-to-trough PnL decline | > -0.01 |

### How IC Works

IC (Information Coefficient) measures **rank correlation** between your predicted
returns and actual returns using Spearman correlation. It tells you: "Do your
rankings of opportunities match reality?"

- IC = 0.05 is respectable for financial prediction
- IC = 0.10+ is excellent
- IC < 0 means your model is systematically wrong (flip your signals!)

### Rolling Windows

Metrics are computed over rolling windows to balance recency vs. stability:

- **score_recent**: Last 24 hours — how well are you doing now?
- **score_steady**: Last 7 days — consistent performance
- **score_anchor**: Last 30 days — long-term track record

The leaderboard weights these to rank models.

## What Makes a Good Signal

### DO

- **Vary your conviction.** Return ±0.001 for weak signals and ±0.01 for strong ones.
  The scoring function rewards calibrated magnitude.

- **Return 0.0 when uncertain.** No prediction is better than a random guess. You
  won't lose points, and your hit rate stays clean.

- **Test on both BTC and ETH.** A good signal should generalize across symbols.

- **Check your IC.** Even 55% hit rate can have negative IC if your magnitude is
  anti-correlated with actual moves.

### DON'T

- **Don't always predict the same direction.** If you always return +0.001, you'll
  have ~50% hit rate but zero IC. You're just following the market's base rate.

- **Don't overfit to one time period.** A model that works perfectly on March 2025
  but fails everywhere else is useless.

- **Don't ignore the magnitude.** Returning ±0.001 everywhere is leaving money on
  the table. Big moves happen — your model should recognize them.

- **Don't look ahead.** In backtesting, make sure your model only uses data up to
  the prediction time. The backtest framework enforces this, but custom data
  loading can accidentally leak future information.
