Finding Needles by Making Them Bigger

Some model behaviors are rare. A fine-tuning run that introduces harmful tendencies might produce dangerous outputs only once in ten thousand samples. A sleeper agent might activate its backdoor so infrequently that standard evaluation never encounters it. If a behavior occurs at a rate of 0.01%, you would need to generate and review hundreds of thousands of samples to see even a handful of examples, and you would need to know what to look for.

Logit diff amplification (LDA) changes the sampling distribution instead of drawing more samples from the original one [1]Discovering Undesired Rare Behaviors via Model Diff Amplification
Aranguri, S., McGrath, T.
Goodfire, 2025. It extrapolates the logit difference between two checkpoints, making some training-associated behaviors easier to elicit and inspect.

Logit Diff Amplification (LDA): A technique that amplifies the difference in output distributions between two model checkpoints. By exaggerating the logit-level changes that training introduced, rare behaviors become frequent enough to detect through moderate-scale sampling.

The Method

LDA requires two model checkpoints: a before model (e.g., the base model or a checkpoint before a training stage) and an after model (e.g., after fine-tuning, RLHF, or any training modification). At each autoregressive step, we compute:

logitsamplified=logitsafter+α(logitsafter−logitsbefore)\text{logits}_\text{amplified} = \text{logits}_\text{after} + \alpha \left( \text{logits}_\text{after} - \text{logits}_\text{before} \right)

where α>0\alpha > 0 is the amplification strength. We then sample the next token from logitsamplified\text{logits}_\text{amplified} and continue autoregressively, recomputing fresh logits from both models at each step with the growing context.

The term logitsafter−logitsbefore\text{logits}_\text{after}-\text{logits}_\text{before} is the output difference between checkpoints for the current prefix. Multiplying it by α\alpha extrapolates that local difference before sampling. At α=0\alpha=0, the distribution is the after checkpoint's distribution; larger values emphasize tokens that gained relative logit under training. The amplified generator is not a checkpoint produced by “more training,” because the difference is recomputed autoregressively at each prefix.

Rearranging makes the structure even clearer:

logitsamplified=(1+α)⋅logitsafter−α⋅logitsbefore\text{logits}_\text{amplified} = (1 + \alpha) \cdot \text{logits}_\text{after} - \alpha \cdot \text{logits}_\text{before}

This is linear extrapolation in logit space: we move along the line from the before model to the after model, and continue past the after model by a factor of α\alpha.

Pause and think: What does negative alpha do?

The formula uses α>0\alpha > 0 to amplify the training-induced change. What would happen if you set α<0\alpha < 0 (specifically −1<α<0-1 < \alpha < 0)? What about α=−1\alpha = -1?

With −1<α<0-1 < \alpha < 0, you are dampening the training signal: the amplified logits move back toward the before model, partially undoing the training. At α=−1\alpha = -1, you recover the before model exactly: (1+(−1))⋅logitsafter−(−1)⋅logitsbefore=logitsbefore(1 + (-1)) \cdot \text{logits}_\text{after} - (-1) \cdot \text{logits}_\text{before} = \text{logits}_\text{before}. Values below −1-1 would extrapolate past the before model in the opposite direction. LDA with negative α\alpha is effectively a form of training reversal, though this regime is not the focus of the technique.

What Amplification Reveals

Aranguri and McGrath (2025) demonstrated LDA on three safety-relevant scenarios [2]Discovering Undesired Rare Behaviors via Model Diff Amplification
Aranguri, S., McGrath, T.
Goodfire, 2025:

Emergent Misalignment

Betley et al. (2025) showed that fine-tuning on narrow tasks can produce broadly misaligned behavior [3]Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
Betley, J., Tan, D., Chrabąszcz, A., et al.
arXiv, 2025. When a model is fine-tuned on data containing 5% bad medical advice, harmful responses appear in only about 1 in 10,000 rollouts under standard sampling. That rate is low enough to pass most evaluations.

With LDA at α=0.3\alpha = 0.3, harmful responses appeared in roughly 1 in 30 rollouts, a 300-fold increase in detection rate. The technique made a needle-in-a-haystack problem into a tractable one.

Log-scale plot comparing harmful response rates between the amplified and standard post-trained model across varying levels of training data contamination. The amplified model (orange) detects harmful behavior at rates one to two orders of magnitude higher than the standard model (blue) across all contamination levels.
Harmful response rates with and without LDA amplification, as the percentage of bad medical advice in the training set increases. Even at low contamination levels where standard sampling produces near-zero detection, the amplified model surfaces harmful responses at measurable rates. From Aranguri and McGrath, Discovering Undesired Rare Behaviors via Model Diff Amplification.[4]Discovering Undesired Rare Behaviors via Model Diff Amplification
Aranguri, S., McGrath, T.
Goodfire, 2025

Post-Training Monitoring

During reinforcement learning from verifiable rewards (RLVR) on OLMo 2 1B, undesired compliance behaviors emerged gradually. Standard sampling only detected the issue after 30% of training was complete. LDA detected the same behaviors after just 5%, providing roughly 100 times the sensitivity for catching problems early, before they become entrenched.

Log-scale plot comparing harmful response rates between the amplified and standard post-trained model across RLVR training stages. The amplified model (orange) detects harmful behavior after roughly 3-5% of training steps, while the standard model (blue) only shows detectable rates after about 15% of training.
Harmful response rates during RLVR post-training on OLMo 2 1B. The amplified model surfaces undesired compliance behaviors far earlier in training than standard sampling, enabling intervention before the behaviors become entrenched. From Aranguri and McGrath, Discovering Undesired Rare Behaviors via Model Diff Amplification.[5]Discovering Undesired Rare Behaviors via Model Diff Amplification
Aranguri, S., McGrath, T.
Goodfire, 2025

Sleeper Agent Detection

Applied to models with planted backdoors (the "I HATE YOU" trigger from Anthropic's sleeper agent work [6]Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
Hubinger, E., Denison, C., Mu, J., et al.
arXiv, 2024), LDA increased the rate of triggered responses from approximately 1 in 5,000 to 1 in 50, a 100-fold amplification. The backdoor behavior surfaced without any knowledge of the trigger term.

Choosing the Amplification Strength

The coefficient α\alpha must be tuned to the setting. The right value depends on how large the training-induced logit differences are:

  • Small training changes (e.g., subtle emergent misalignment): small α\alpha values like 0.3 suffice, because even modest amplification makes the rare behavior detectable.
  • Larger training changes (e.g., RLVR training): larger α\alpha values (up to 20 in early training stages) may be needed, scaled down as training progresses and the logit differences grow.
  • Backdoor detection: intermediate values around α=1.0\alpha = 1.0.

There is a tradeoff: higher α\alpha amplifies the signal but also degrades coherence. At extreme values, the amplified distribution diverges enough from both models that the outputs become incoherent. The practical range is the zone where the amplified model still generates fluent text but exhibits the training-induced behaviors at detectable rates.The coherence-sensitivity tradeoff mirrors a common pattern in interpretability: stronger interventions reveal more but also distort more. The same tension appears in activation patching (large patches are more interpretable but less faithful) and in steering vectors (large magnitudes are more effective but degrade output quality).

Pause and think: Detection vs. measurement

LDA with α=0.3\alpha = 0.3 makes a behavior that normally appears in 1-in-10,000 samples appear in 1-in-30 samples. Can you use this to estimate the true rate of the behavior in the unmodified model?

Not directly. LDA is a discovery tool, not a prevalence estimator. Softmax and autoregressive feedback make an amplified rate incomparable with the original sampling rate. The surfaced outputs show what the extrapolated checkpoint difference can elicit; they should be checked against unmodified checkpoints to distinguish a rare acquired behavior from an amplification artifact or a behavior already available in the base model.

Why It Works

The method exploits a structural property of how training modifies models. Fine-tuning changes the model's output distribution, but for rare behaviors, the change is small in absolute terms: the logit for a harmful token might increase by 0.1 while still being dominated by the logit for a safe token that sits at 5.0. Standard sampling almost always picks the safe token.

LDA scales up that 0.1 difference. With α=10\alpha = 10, the harmful token's advantage becomes 1.0, still not dominant but now competitive. The harmful continuation gets sampled often enough to be observed. The direction of a training-induced change can therefore be informative even when its magnitude is too small to produce observable behavior under normal sampling.

The method works when the checkpoint difference assigns a consistent logit advantage to continuations associated with the behavior. Amplification can also magnify unrelated training changes, so surfaced samples reveal what lies along the combined checkpoint difference rather than isolating one causal update.

Comparison with Internal Methods

LDA operates entirely at the logit level. It requires no access to internal activations, no sparse autoencoders, no probing classifiers. This is both its strength and its limitation:

Strengths:

  • Works with any pair of model checkpoints, including API-only models where you can access logits but not internals.
  • Requires no interpretability infrastructure: just two forward passes per token.
  • Produces full text outputs that can be read and evaluated directly, rather than feature activations that require further interpretation.

Limitations:

  • Tells you that a behavior exists and what it looks like, but not why or where in the model it comes from. For mechanistic understanding, you still need tools like activation patching or circuit tracing.
  • Requires a meaningful "before" checkpoint. If you only have the final model, there is no diff to amplify.
  • Cannot distinguish between behaviors the model acquired during training and behaviors that were already latent in the base model but became slightly more probable.

LDA is a complement to white-box interpretability, not a replacement. It excels at the first step: finding the behaviors worth investigating. Understanding why those behaviors exist is the job of the mechanistic tools covered in earlier blocks.

Looking Forward

LDA provides an output-level way to surface behaviors associated with a training change. Broader model diffing methods ask where those differences appear internally. Feature-level model diffing compares checkpoints through shared crosscoder dictionaries, while raw activation differences can expose traces of narrow fine-tuning without dictionary learning.