Affine Steering

Two Tools, One Intervention

Addition steering shifts the model toward a behavior by adding a direction. Ablation steering suppresses a behavior by projecting out a direction. We have been treating these as separate tools, each operating on concept directions identified through probing.

But there is a gap between them. Addition changes the model's behavior without removing its existing tendency. Ablation removes the tendency without controlling what replaces it. And in practice, ablation alone sometimes fails: on certain architectures, projecting out the refusal direction produces incoherent text rather than compliant responses.Marshall et al. (2024) found that directional ablation on RWKV v5 produced complete gibberish, while the same direction worked fine with addition steering. The failure is not in the direction itself but in what happens when the model's activations land in a region far from anything it encountered during training.

Marshall et al. (2024) combine the two operations with an explicit reference point in Affine Concept Editing (ACE) [1]Refusal in LLMs is an Affine Function
Marshall, T., Scherlis, A., Belrose, N.
arXiv, 2024.

Why Affine?

The key observation is that the origin of activation space has no special meaning. Typical model activations live in a region far from the zero vector. When we project out a direction via ablation, we remove the component along that direction and leave the rest. But "the rest" is centered at the origin, which may be far from any region the model has learned to generate coherently from.

Consider what happens geometrically. Activations for compliant responses cluster around some mean $\mathbf{r}^-$. Activations for refusing responses cluster around some mean $\mathbf{r}^+$. The refusal direction $\mathbf{r} = \mathbf{r}^+ - \mathbf{r}^-$ connects these clusters. Directional ablation projects onto the hyperplane orthogonal to $\mathbf{r}$, which passes through the origin. But the compliant cluster is not centered at the origin. It is centered at $\mathbf{r}^-$, which has a non-zero component along $\mathbf{r}$.

Affine vs. Linear: A linear function maps $\mathbf{0} \mapsto \mathbf{0}$. An affine function includes a constant offset: $f(\mathbf{v}) = \mathbf{v}A + \mathbf{b}$. Behavioral encoding in activation space is affine because the zero vector is not the "default" behavior. The default behavior has its own location in activation space, and interventions need to account for that offset.

This geometry suggests one failure mode for ablation. Projecting out the refusal direction uses the origin as its reference, not the compliant cluster. If those reference points differ substantially along the chosen direction, the intervention can move activations away from the region represented by the null-behavior examples.

The ACE Formula

ACE combines three operations into one intervention:

$$\mathbf{v}' = \underbrace{\mathbf{v} - \text{proj}_\mathbf{r}(\mathbf{v})}_{\text{erase}} + \underbrace{\text{proj}_\mathbf{r}(\mathbf{r}^-)}_{\text{re-center}} + \underbrace{\alpha \cdot \mathbf{r}}_{\text{steer}}$$

where:

• $\mathbf{v}$ is the original activation
• $\mathbf{r} = \mathbf{r}^+ - \mathbf{r}^-$ is the concept direction (difference in means between the behavior and null-behavior classes)
• $\text{proj}_\mathbf{r}(\mathbf{v})$ is the component of $\mathbf{v}$ along $\mathbf{r}$
• $\mathbf{r}^-$ is the mean activation for the null-behavior class (e.g., compliant responses)
• $\alpha$ is a tunable scalar controlling steering strength

The three terms do the following:

1. Project. Remove the activation's component along the chosen direction. This eliminates information available along that one direction; the behavior may remain decodable elsewhere. The operation is identical to directional ablation.

2. Re-center. Add back the component that the null-behavior mean has along the concept direction. This shifts the erased activation to where compliant activations typically live, rather than leaving it at the origin. This is the affine correction, the term that ablation alone misses.

3. Steer. Add a tunable amount of the concept direction. The class means motivate interpreting $\alpha=0$ and $\alpha=1$ as null- and target-behavior reference points, but actual behavior need not interpolate perfectly. Values outside the fitted range are extrapolations.

Pause and think: Recovering addition and ablation

ACE claims to unify addition and ablation as special cases. Can you see how?

Setting only the steer term (dropping erase and re-center) gives $\mathbf{v}' = \mathbf{v} + \alpha \cdot \mathbf{r}$, which is addition steering. Setting only the erase term (dropping re-center and steer) gives $\mathbf{v}' = \mathbf{v} - \text{proj}_\mathbf{r}(\mathbf{v})$, which is directional ablation. ACE adds the re-center term that neither method includes on its own, and combines all three into a single intervention where each piece plays a role.

Standardization

A practical problem with addition steering is that the same $\alpha$ value has different effects on different kinds of prompts. Adding the refusal direction with $\alpha = 5$ might cause the model to refuse a harmful prompt (where it was already leaning toward refusal) but not a harmless one (where refusal requires a larger push). The mapping from $\alpha$ to behavior depends on the input.

ACE addresses this through standardization. Because the erase step removes the input's existing component along the concept direction, and the re-center step places it at the null-behavior baseline, the steer step operates from a consistent starting point regardless of the input. The parameter $\alpha$ has a consistent meaning:

• $\alpha = 0$: null-behavior (e.g., comply)
• $\alpha = 1$: full behavior (e.g., refuse)

Marshall et al. show that ACE produces nearly overlapping refusal curves for harmful and harmless prompts, meaning the same $\alpha$ value produces the same degree of refusal regardless of prompt type. With addition steering alone, the curves diverge substantially [3]Refusal in LLMs is an Affine Function
Marshall, T., Scherlis, A., Belrose, N.
arXiv, 2024.

Results

ACE was evaluated on 10 open-weight models, including Llama 3 8B and 70B, RWKV v5, Qwen, Yi, and Gemma variants [4]Refusal in LLMs is an Affine Function
Marshall, T., Scherlis, A., Belrose, N.
arXiv, 2024.

Standardization. Across all models, ACE produced more consistent steering than addition alone. The gap between harmful-prompt and harmless-prompt refusal curves was consistently smaller with ACE.

Rescuing incoherent ablation. On RWKV v5, directional ablation of the refusal direction produced completely incoherent outputs. ACE on the same model and direction produced coherent, well-formed compliant text. The affine correction term was the difference between gibberish and working steering.

Cross-architecture evidence. The improvement held across the transformer and recurrent models tested. This shows that the affine correction is not restricted to one architecture, while broader generality still requires testing other behaviors and model families.

Pause and think: When does the affine correction matter?

The affine correction re-centers at $\mathbf{r}^-$ after erasing the concept direction. Under what conditions would this correction be negligible? When would it be essential?

The correction is negligible when $\mathbf{r}^-$ has a near-zero component along $\mathbf{r}$, meaning the null-behavior mean already lies close to the hyperplane orthogonal to the concept direction passing through the origin. In that case, ablation alone lands near the right place. The correction is essential when $\mathbf{r}^-$ has a large component along $\mathbf{r}$, meaning the null-behavior cluster is far from the origin in the concept direction. This is more likely for behaviors (like refusal) that are overlaid onto a model whose "default" state already has a strong tendency in one direction.

Limitations

Imperfect standardization. ACE improves standardization but does not achieve it perfectly. Optimal $\alpha$ values sometimes fall outside $[0, 1]$, and some input dependence remains. One-direction projection may leave linearly decodable information in other directions, nonlinear information, or prompt-dependent effects.The authors also tested LEACE, which removes information available to an optimal linear predictor under its population assumptions. Its lack of improvement does not uniquely diagnose nonlinear encoding; estimation, distribution shift, and the relationship between decodability and control are alternative explanations.

Requires class means. ACE needs $\mathbf{r}^+$ and $\mathbf{r}^-$, the mean activations for the two behavior classes. This requires labeled examples of both behaviors, the same data requirement as CAA. The method cannot be applied in settings where only the concept direction is available without the class means.

Single direction. Like addition and ablation, ACE operates on a single linear direction. Behaviors encoded across multiple interacting directions, or encoded nonlinearly, are not fully captured.

The Geometric Picture

ACE has a clean geometric interpretation that extends the pictures from addition and ablation:

1. Ablation projects onto the hyperplane through the origin orthogonal to $\mathbf{r}$.
2. ACE projects onto the hyperplane through $\mathbf{r}^-$ orthogonal to $\mathbf{r}$, then translates by $\alpha \cdot \mathbf{r}$.

The difference is where the hyperplane sits. Ablation uses the origin as the reference point. ACE uses the empirical null-behavior mean. When the null-behavior mean is far from the origin along the concept direction, this difference matters.

Looking Forward

ACE combines addition and directional projection with a data-derived reference point. Its central lesson is practical: a direction alone does not specify where an edited activation should land. The location of the null-behavior examples can matter as much as the axis connecting the classes.

Within representation control, addition shifts along a direction, ablation removes one directional component, and affine editing projects, recenters, and then steers. For a method designed to remove information available to linear predictors under explicit assumptions, see concept erasure with LEACE.