# How DirectRS Works

The core idea: a decision tree implicitly learns a *geometry* of the feature space by splitting it into regions. DirectRS makes that geometry explicit and uses it as the input to a simple linear model.

---

## Step 1: Train a Decision Tree

First, a standard tree (CART) is trained on the data. The tree partitions the feature space into **leaves** — each leaf is a hyperrectangle defined by the conjunction of all splits along the path from root to that leaf.

---

## Step 2: Extract Geometry (the G matrix)

For each leaf ℓ, define:

- **p_ℓ ∈ {0,1}^d** — a binary indicator vector: `p_ℓ[j] = 1` if feature `j` was used in any split along the path to leaf ℓ, `0` otherwise.
- **v_ℓ** — the leaf's prediction value (class probability or mean target).
- **w_ℓ** — the fraction of training samples that land in leaf ℓ.

The **geometry matrix** G ∈ ℝ^(d×d) aggregates how much each feature (and feature pair) *matters* across all leaves, weighted by leaf importance:

```
G = Σ_ℓ  w_ℓ · (v_ℓ - v̄)² · outer(p_ℓ, p_ℓ)
```

G[i,j] is large when features i and j are both active in leaves that make very different predictions from the global mean. This captures **feature co-occurrence in predictive regions** — i.e., which features interact to produce the tree's decisions.

---

## Step 3: Compute the Stretch Matrix S'

G is symmetric positive semi-definite, so it has a square root via eigendecomposition:

```
G = Q Λ Qᵀ  →  S' = Q · clamp(Λ, 0)^(1/2) · Qᵀ
```

S' is a **linear map** that "stretches" the input space along directions the tree found discriminative. Directions corresponding to zero eigenvalues are collapsed — those directions don't matter for prediction.

---

## Step 4: Build the Embedding Φ(x)

For a new input x, find which leaf ℓ*(x) it falls into. Then build a block-structured feature vector:

```
Φ(x) ∈ ℝ^(L × (d+1))
```

The block corresponding to leaf ℓ*(x) is filled with `[1, S'x]` (a bias + the geometrically-stretched input). All other blocks are zero.

This embedding says: *"x landed in leaf ℓ*, and here's what x looks like in the geometry local to that leaf."*

---

## Step 5: Fit a Linear Model on Φ

- **Classification**: LogisticRegression on Φ(x)
- **Regression**: Ridge regression on Φ(x)

The final model is **piecewise linear**: within each leaf, the prediction is a linear function of the stretched input. Across leaves, different linear functions apply, each calibrated by the global linear model's coefficients for that leaf's block.

---

## Step 6: Prediction for New x

```
x  →  [tree routes x to leaf ℓ*]  →  Φ(x)  →  linear model  →  ŷ
```

1. Run x down the trained tree to find its leaf ℓ*.
2. Compute `S'x` (stretch x through the geometry).
3. Place `[1, S'x]` in the ℓ*-th block of a zero vector → Φ(x).
4. Apply the fitted linear model weights to Φ(x) → output.

---

## Feature Importances

Since Φ has known structure:

- **Feature importance**: sum of absolute linear weights for each original feature j across all leaf blocks (how much does feature j contribute when stretched and mapped?).
- **Feature interaction importance**: G[i,j] directly quantifies how often features i and j appear together in predictive leaves — the off-diagonal entries of G are interaction importances.

---

## The Key Insight

A plain decision tree is non-differentiable and ignores the *magnitude* of x within a leaf (only the region matters). DirectRS recovers a differentiable, linear-within-region model that:

1. Respects the tree's partition of space
2. Uses the actual feature values (not just "which box am I in")
3. Has interpretable coefficients tied to geometrically-meaningful directions
