# Writing a CTF Model

The fastest way to start is to copy the template from the reference directory and
replace the placeholder sections with your own implementation.

## Starting from the Template

```bash
cp reference/template-ctf-model.py models/my-model.py
```

The template implements a complete rolling-window train/predict loop with:

- Rank-normalized features via ECDF (to the interval [-0.5, 0.5])
- OLS prediction via `np.linalg.lstsq`
- Z-score portfolio weights (mean-zero, long-short)

Each step is isolated in a single function with a `# TODO: replace this` marker.

## Template Structure

```python
def prepare_data(chars, features):
    """Rank-normalize features to [-0.5, 0.5] via ECDF."""
    # TODO: replace or extend feature engineering
    ...

def train_model(X_train, y_train):
    """Fit a predictive model on training data."""
    # TODO: replace OLS with your estimator
    ...

def predict_returns(model, X_test):
    """Predict returns for the test period."""
    # TODO: replace if your model has a different predict interface
    ...

def construct_weights(predictions, ids, eom):
    """Convert return predictions to portfolio weights."""
    # TODO: replace z-score weighting with your approach
    ...
```

The `run_model()` and `main()` functions orchestrate the rolling window loop and
handle the CTF entry point. These do not normally need modification.

## CTF Interface Contract

The `main()` function must match this exact signature:

```python
def main(
    chars: pd.DataFrame,
    features: pd.DataFrame,
    daily_ret: pd.DataFrame,
) -> pd.DataFrame:
    ...
```

The return value must be a DataFrame with exactly three columns:

| Column | Type | Description |
|--------|------|-------------|
| `id` | int or str | Stock identifier (matches `chars` and `daily_ret`) |
| `eom` | str or date | Month-end date in `YYYY-MM-DD` format |
| `w` | float | Portfolio weight |

Weights must cover all `ctff_test == 1` dates in the characteristics panel. No NaN
values are permitted, and each (id, eom) pair must be unique.

## Compliance Requirements

Before submitting, run `validate()` to check all 23 rules. The most common sources
of failure are:

- **Look-ahead bias**: using future information in the construction of `X_train`.
  Use only data where `eom <= train_end` when building the training set.
- **Missing test dates**: the returned weights must include all test-period month-end
  dates, not just dates where predictions are above some threshold.
- **Network calls**: the script cannot make HTTP or socket calls at runtime.
- **Non-determinism**: set `np.random.seed()` and `random.seed()` at the top of the
  script and confirm that two independent runs produce identical weights.

## Using eapctf Estimators

The `eapctf.predict` module provides estimators that slot directly into the template:

```python
from eapctf.predict import make_predictor

# Ridge regression
ridge = make_predictor("ridge", alpha=0.01)
ridge.fit(X_train, y_train)
preds = ridge.predict(X_test)

# LightGBM (requires eapctf[ml])
lgbm = make_predictor("lightgbm", n_estimators=200)
lgbm.fit(X_train, y_train)
preds = lgbm.predict(X_test)

# IPCA (Instrumented PCA, Kelly, Pruitt and Su 2019)
ipca = make_predictor("ipca", n_factors=5)
ipca.fit(X_train, y_train)
preds = ipca.predict(X_test)
```

See `reference/benchmark-ipca-pf.py` for a complete IPCA implementation that
achieves a Sharpe ratio of 1.939 on the test period.

## High-Level Workflow with ctf_expanding_window_oos

`eapctf.ctf` provides a convenience wrapper that handles all three preprocessing
steps (feature selection, rank normalization, panel construction) and delegates
to `expanding_window_oos`. This reduces a full CTF model to four lines:

```python
from eapctf.ctf.predict import ctf_expanding_window_oos
from eapctf.predict import make_predictor


def main(chars, features, daily_ret):
    oos = ctf_expanding_window_oos(
        chars,
        features,
        models=[make_predictor("ridge")],
    )
    return oos.to_ctf_weights(signal_col="pred_RidgePredictor", mode="zscore")
```

The function applies the HJKP (2025) standard:

- 10-year rolling training window (`window_length=120`)
- Monthly retraining with 5-fold temporal cross-validation
- ECDF rank normalization to [-0.5, 0.5] (`rank_normalize=True` by default)
- Returns an `OOSResult` filtered to test-period dates (`test_only=True` by default)

### Feature subset selection

```python
# Use all 402 CTF features (default)
oos = ctf_expanding_window_oos(chars, features, feature_subset="all")

# Restrict to the 153 JKP characteristics
oos = ctf_expanding_window_oos(chars, features, feature_subset="jkp153")

# Keep only features with >50% non-null coverage
oos = ctf_expanding_window_oos(chars, features, feature_subset="available")
```

### Pre-normalizing manually

Use `ctf_rank_normalize()` and `ctf_select_features()` directly when you need
fine-grained control over the preprocessing pipeline:

```python
from eapctf.ctf import ctf_rank_normalize, ctf_select_features
from eapctf.ctf.predict import ctf_expanding_window_oos
from eapctf.predict import make_predictor


def main(chars, features, daily_ret):
    # 1. Normalize features (ECDF rank to [-0.5, 0.5])
    chars_norm = ctf_rank_normalize(chars, features)

    # 2. Select feature subset
    chars_sel, selected = ctf_select_features(chars_norm, features, subset="jkp153")

    # 3. Predict (skip internal normalization since we already did it)
    oos = ctf_expanding_window_oos(
        chars_sel,
        features,
        models=[make_predictor("ridge")],
        rank_normalize=False,
        feature_subset="all",  # already filtered
    )
    return oos.to_ctf_weights(signal_col="pred_RidgePredictor", mode="zscore")
```

## Conformal Prediction with ctf_conformal_oos

`ctf_conformal_oos` adds a conformal interval layer on top of the standard OOS
loop. Install the extra first:

```bash
pip install "eapctf[conformal]"
```

### How it works

At each prediction date the function:

1. Fits a MAPIE CV+ regressor (`method="plus"`, `TimeSeriesSplit`) on the rolling
   training window. By default this refit happens once per calendar year
   (`refit_freq="yearly"`).
2. Predicts a point estimate and a conformal interval `[lower, upper]` at the
   current ACI level `alpha_t`.
3. Constructs an *uncertainty-adjusted signal*:

        mu_tilde = point_pred - lambda_t * (upper - lower) / 2

   Stocks with wide intervals (high uncertainty) receive a penalised signal.
4. Updates `alpha_t` each period based on cross-sectional coverage from the
   prior month (Gibbs & Candes 2021):

        alpha_t = clip(alpha_t + gamma * (alpha - avg_err), 0.01, 0.99)

### Minimal usage

```python
from eapctf.ctf import ctf_conformal_oos
from eapctf.predict import make_predictor


def main(chars, features, daily_ret):
    result = ctf_conformal_oos(
        chars,
        features,
        model=make_predictor("ridge"),
    )
    # result has columns: id, eom, w, w_baseline, interval_width, alpha_t
    return result[["id", "eom", "w"]]
```

### Key parameters

| Parameter | Default | Description |
|-----------|---------|-------------|
| `alpha` | 0.1 | Target miscoverage rate (90% coverage) |
| `gamma` | 0.05 | ACI step size |
| `lambda_t` | 1.0 | Uncertainty penalty scale |
| `refit_freq` | `"yearly"` | `"yearly"` or `"monthly"` |
| `weight_mode` | `"zscore"` | `"zscore"`, `"rank_weighted"`, `"long_short"` |
| `n_conformal_folds` | 5 | Folds for `TimeSeriesSplit` in CV+ |

Setting `lambda_t=0` recovers the plain point-prediction weights (`w == w_baseline`).

### Comparing uncertainty-adjusted vs baseline weights

```python
result = ctf_conformal_oos(chars, features, model=make_predictor("ridge"))

# Compare uncertainty-adjusted vs pure-signal weights per date
print(result.groupby("eom")[["w", "w_baseline", "interval_width", "alpha_t"]].mean())
```

