<!-- SPDX-FileCopyrightText: 2026 Teilo Millet -->
<!-- SPDX-License-Identifier: Apache-2.0 -->

# Part 1: What is Abliteration?

This part builds geometric intuition for abliteration — the discovery that safety refusal in language models is controlled by a single direction in activation space, and that removing it is a linear algebra operation. There is no code here; Part 2 runs your first abliteration, and Part 3 derives every step.

## The Key Insight

### Safety Alignment as Geometry

When a language model is instruction-tuned for safety, it learns to refuse harmful requests. Ask it how to pick a lock, and it declines. Ask it to explain photosynthesis, and it complies. The standard view is that this behavior emerges from complex, distributed representations learned during RLHF or DPO — deeply entangled with the model's general capabilities.

Abliteration rests on a different observation: **refusal is not deeply entangled. It is a direction.**

Specifically, there exists a single direction in the model's activation space such that:
- Harmful prompts produce activations with a large **positive** component along this direction.
- Harmless prompts produce activations with a small or **negative** component.
- Removing this component from the model's weight matrices eliminates refusal without destroying general capabilities.

This is surprising. A model with billions of parameters, trained on terabytes of text, fine-tuned with human preferences — and its refusal behavior can be located, measured, and surgically removed by manipulating a single vector.

### The Linear Representation Hypothesis

This result is an instance of the **linear representation hypothesis**: high-level concepts in neural networks are encoded as directions in activation space. Just as word2vec showed that "king - man + woman ≈ queen" as a vector arithmetic operation, abliteration shows that "refusal" is a direction that can be added to or subtracted from the model's internal representations.

The hypothesis does not claim that all concepts are linear — only that many behaviorally important ones are. Refusal turns out to be one of the clearest examples: it is almost perfectly rank-1, meaning a single direction captures nearly all of its variance.

## The Residual Stream

### What Is the Residual Stream?

A transformer processes a sequence of tokens by passing a vector through a stack of layers. Each layer reads the current vector, transforms it (via attention and MLP blocks), and **adds** the result back:

$$h^{l+1} = h^l + \text{Attn}^l(h^l) + \text{MLP}^l(h^l)$$

The vector $h^l$ is called the **residual stream** at layer $l$. It is a $d_{\text{model}}$-dimensional vector — for a model like Llama-3.2-1B, that is $\mathbb{R}^{2048}$.

The residual stream is the main information highway. Everything the model knows at a given position flows through it. Layers read from it, compute something, and write back into it. This additive structure is what makes abliteration possible: if refusal is written into the residual stream by specific layers, we can modify those layers to stop writing it.

### Activations at the Last Token

When we say "the activation at layer $l$," we mean the residual stream vector after layer $l$ has written its output, at a specific token position. For abliteration, we care about the **last token position** — the position where the model is about to generate its first output token.

Formally, given a prompt $p$ of length $T$ tokens, the activation at layer $l$ is:

$$a_l(p) = h^l_T \in \mathbb{R}^{d_{\text{model}}}$$

### Why the Last Token Matters

In a causal language model, each token can only attend to tokens before it. The last token position is special: it is the only position that has attended to the entire prompt. When the model decides whether to comply or refuse, that decision is concentrated at this position — it is the bottleneck through which all information flows before generation begins.

> **You Should Know:** The last-token choice is a simplification. The DBDI framework (Part 5) shows that harm *detection* peaks at the instruction-final token, while refusal *execution* peaks at the sequence-final token. For standard abliteration, last-token is sufficient.

## Directions in Activation Space

### From Points to Directions

Imagine collecting activations from 128 harmful prompts and 128 harmless prompts at layer $l$. Each activation is a point in $\mathbb{R}^{d_{\text{model}}}$. You now have two clusters of points — one for harmful, one for harmless.

These clusters are not randomly scattered. The harmful cluster is systematically displaced from the harmless cluster in a consistent direction. The vector connecting the two cluster centers defines the **refusal direction** at layer $l$.

### Difference-in-Means

The simplest way to extract this direction is the **difference-in-means**:

$$d_l = \frac{1}{|H|} \sum_{p \in H} a_l(p) \;-\; \frac{1}{|B|} \sum_{p \in B} a_l(p)$$

where $H$ is the set of harmful prompts and $B$ is the set of harmless (benign) prompts. This gives us a raw direction vector at each layer $l$.

We normalize it to a unit vector:

$$\hat{d}_l = \frac{d_l}{\|d_l\|}$$

That is the entire measurement step. No optimization, no training, no gradient descent — just subtract two means and normalize.

### Cosine Separation

Not all layers encode refusal equally. To find the **best layer**, we compute how well each layer's direction separates harmful from harmless activations. The cosine separation score is:

$$s_l = \left\langle \mu_H^l,\; \hat{d}_l \right\rangle - \left\langle \mu_B^l,\; \hat{d}_l \right\rangle$$

where $\mu_H^l$ and $\mu_B^l$ are the harmful and harmless means at layer $l$. The layer with the highest $s_l$ is where refusal is most sharply encoded.

In practice, this peaks in the middle-to-upper layers (roughly layers 10–20 for a 32-layer model). Early layers are busy with token identity and syntax; late layers are preparing the output distribution. The refusal decision lives in between.

## The Refusal Direction

### One Direction to Rule Them All

The result of the measurement step is a single unit vector $\hat{d} \in \mathbb{R}^{d_{\text{model}}}$ at the best layer. This is the **refusal direction**. It has a remarkable property: projecting any prompt's activation onto this direction predicts whether the model will refuse.

- $\langle a_l(p), \hat{d} \rangle > 0$: the model is likely to refuse.
- $\langle a_l(p), \hat{d} \rangle < 0$: the model is likely to comply.
- $\langle a_l(p), \hat{d} \rangle \approx 0$: the model is on the boundary.

This single scalar — a dot product — captures the refusal decision.

### What the Direction Encodes

The refusal direction is not a "refusal neuron." It is a direction in a high-dimensional space, a pattern distributed across all $d_{\text{model}}$ dimensions. Think of it as a template: the model has learned that activations with a large component along this template should trigger refusal phrasing ("I can't help with that", "I'm sorry, but...").

Crucially, the refused *knowledge* is still present in the activation. Shrivastava and Holtzman (2025) showed that information about how to answer harmful queries remains linearly decodable from the hidden states even when the model refuses. Refusal is a gate, not an eraser.

### The Projection Interpretation

Given an activation $a$ and the refusal direction $\hat{d}$, the projection of $a$ onto $\hat{d}$ is:

$$\text{proj}(a, \hat{d}) = \langle a, \hat{d} \rangle \cdot \hat{d}$$

This is the component of $a$ that lies along the refusal direction. The remainder — $a - \text{proj}(a, \hat{d})$ — is the component orthogonal to refusal. The orthogonal component carries the model's actual knowledge and capability; the projection carries the refusal signal.

Abliteration works by ensuring that the refusal component is never written into the residual stream in the first place.

## From Direction to Surgery

### Rank-1 Projection Removal (Preview)

Once we have the direction $\hat{d}$, the surgery is a rank-1 update to weight matrices. For each target weight matrix $W$ (specifically `o_proj` and `down_proj`, which write directly into the residual stream):

$$W' = W - \alpha \cdot \hat{d} \cdot \hat{d}^\top W$$

What does this do? For any input $x$:

$$W'x = Wx - \alpha \cdot \hat{d} \cdot (\hat{d}^\top W x) = Wx - \alpha \cdot \langle \hat{d}, Wx \rangle \cdot \hat{d}$$

The modified weight matrix produces the same output as before, minus $\alpha$ times the component of the output that lies along the refusal direction. When $\alpha = 1$, the refusal component is fully removed. The output is projected onto the hyperplane orthogonal to $\hat{d}$.

Part 3 derives this fully and explores what happens for $\alpha \neq 1$.

### Why This Works

Two properties make this effective:

1. **Linearity of the residual stream.** Because layers add to the residual stream, removing a direction from layer outputs removes it from all downstream computations. The refusal signal cannot be reconstructed from orthogonal components (at least not without additional training).

2. **Rank-1 structure of refusal.** If refusal were spread across many orthogonal directions, removing one direction would only partially reduce it. But empirically, refusal is nearly rank-1: a single direction captures the overwhelming majority of its variance. This is what makes the surgery so precise.

### What Could Go Wrong

Abliteration is not free. Potential failure modes include:

- **Perplexity increase.** Removing a direction also removes a small amount of non-refusal information that happened to be correlated with $\hat{d}$. This manifests as a modest increase in perplexity on harmless text.
- **Over-refusal reduction.** The same direction that encodes "refuse harmful requests" also encodes "refuse borderline requests." Abliteration reduces both — including cases where refusal was appropriate.
- **Hardened models.** If a model has been trained to distribute refusal across multiple orthogonal directions (as in the "Embarrassingly Simple Defense"), rank-1 removal is insufficient. Part 5 covers detection and subspace methods for these cases.
- **Alpha sensitivity.** Too little ($\alpha < 1$) leaves residual refusal. Too much ($\alpha > 1$) damages capability. Part 7 shows how to optimize this with Optuna.

## Why This Matters

Abliteration matters for three communities:

**For alignment researchers,** it reveals the geometric structure of safety training. The fact that RLHF/DPO produces a rank-1 refusal direction — not a distributed, entangled feature — tells us something fundamental about how alignment works (and how fragile it is).

**For security engineers,** it is a red-teaming primitive. If you are evaluating a model's safety boundaries, abliteration tells you exactly how much of the safety behavior is a linear overlay versus a deep capability modification.

**For the open-source community,** it is a tool for understanding what instruction tuning actually changes. The gap between a base model and its instruct variant is, to a first approximation, a single direction.

> **You Should Know:** The "single direction" claim is approximate. For most models, the top singular value captures 60–80% of refusal variance. The remaining 20–40% is spread across a few additional directions. Part 5 explores this with SVD and effective rank analysis.

> **You Should Know:** Linearity holds across scales. The rank-1 structure of refusal has been verified from 1B to 70B parameters. Larger models tend to have cleaner separation (higher cosine scores) but the same geometric structure.

## Key Takeaways

1. **Refusal is a direction** in the model's activation space — not a distributed, entangled feature.
2. **Difference-in-means** between harmful and harmless activations extracts this direction.
3. **Cosine separation** identifies which layer encodes refusal most strongly.
4. **Projection onto the direction** predicts whether the model will refuse.
5. **Rank-1 weight surgery** removes the direction from weight matrices, eliminating refusal.
6. **The refused knowledge persists** — refusal is a gate, not an eraser.

## Further Reading

- Arditi et al. (2024). "Refusal in Language Models Is Mediated by a Single Direction." — The foundational paper. [References](references.md)
- "The Geometry of Refusal in Large Language Models." — Extended geometric analysis. [References](references.md)
- Shrivastava & Holtzman (2025). "Linearly Decoding Refused Knowledge." — Evidence that refusal is a gate. [References](references.md)

Next: [Part 2 — Your First Abliteration](part2_your_first_abliteration.md), where you run the full pipeline before understanding every detail.
