MI Glossary

Key terms in mechanistic interpretability with links to articles where they are discussed.

A

Activation Cache: A dictionary storing all intermediate activations from a model's forward pass, keyed by HookPoint name. Enables post-hoc inspection of every computation the model performed on a given input.See:TransformerLens
Activation Difference Lens: A model diffing technique that interprets the average activation difference between a finetuned model and its base model on early tokens of unrelated text, using tools like Patchscope and steering to reveal information about the finetuning domain.See:Finetuning Traces in Activations
Activation Patching: A causal intervention method where activations from a clean run are substituted into a corrupted run (or vice versa) at specific model components, revealing which components are causally important for a behavior.See:Activation Patching and Causal Interventions
Activation Reconstructor (AR): The decoder half of an NLA. A truncated copy of the target model that reads an explanation and maps it back to a reconstructed activation through a learned affine head.See:Natural Language Autoencoders
Activation Verbalizer (AV): The encoder half of an NLA. A copy of the target model that receives an activation injected in place of a token embedding and generates a natural-language explanation of it.See:Natural Language Autoencoders
Adversarial Pruning: Selecting a subnetwork by minimizing its size subject to reconstruction holding up under adversarially chosen ablations of the excluded nodes, rather than under no ablation or stochastic ablation. Non-adversarially pruned subgraphs are systematically too small.See:Circuits in Parameter Space
Affine Concept Editing (ACE): A steering intervention that erases the component along a concept direction, re-centers at the null-behavior mean, and adds a tunable amount of the direction back. Generalizes both addition steering and directional ablation as special cases.See:Affine Steering
Alignment Faking: A scenario where a model behaves as if aligned during training or evaluation while harboring different objectives internally. Detecting alignment faking is a key motivation for mechanistic interpretability safety research.See:Deception Detection and Alignment Faking
Attention Head: An individual attention computation within a multi-head attention layer. Each head independently computes attention patterns over the input sequence and produces a weighted combination of value vectors.See:The Attention Mechanism
Attention Pattern: The matrix of attention weights produced by an attention head, showing how much each token position attends to every other position. Visualizing attention patterns is a foundational interpretability technique.See:The Attention Mechanism
Attention Probe: A probing classifier that uses a learned attention mechanism to aggregate per-token hidden states into a single representation for classification, replacing fixed pooling strategies like mean pooling or last-token selection.See:Attention Probes
Attribution Graph: A computational graph produced by circuit tracing that maps how information flows through a model, showing which features and connections contribute to a specific output.See:Circuit Tracing and Attribution Graphs
Attribution Patching: A linearized approximation of activation patching that uses gradients to estimate the causal effect of patching each component, making it computationally feasible to scan all components in a single forward and backward pass.See:Attribution Patching and Path Patching
Automated Interpretability: Methods that use language models to automatically generate and score natural language explanations of what individual neurons or features represent, reducing the need for manual inspection.See:Feature Dashboards and Automated Interpretability

B

Bridge (sparse-dense): An encoder-decoder pair, trained per sublayer, that translates between a weight-sparse model's activations and those of a dense model trained alongside it, allowing an interpretable perturbation found in the sparse model to be applied to the dense one.See:Training Models to Be Interpretable

C

Capability-Interpretability Frontier: The Pareto frontier between a model's pretraining loss and the size of the circuits needed to explain its behavior. Increasing weight sparsity moves a model along the frontier; increasing total parameter count moves the frontier itself outward.See:Training Models to Be Interpretable
Cascade Classifier: A two-stage deployment architecture where a cheap, fast classifier (such as a linear probe) screens all traffic and only escalates uncertain or flagged cases to a more expensive classifier (such as an LLM), dramatically reducing average inference cost while maintaining accuracy.See:Probes in Production
Causal Abstraction: A formal relationship between a high-level interpretable causal model and a low-level neural network, established by showing that interventions on aligned components produce matching behavior changes in both systems.See:The Causal Abstraction Framework
Causal Importance: A learned scalar in [0,1] predicting how ablatable a parameter subcomponent is on a given input at a given position. A value of 0 means the subcomponent can be scaled down freely without changing the output; 1 means it cannot be touched.See:Parameter Decomposition
Causal Intervention: Any experimental technique that actively modifies model internals (activations, weights, or attention patterns) to test causal hypotheses about how a model computes its outputs, as opposed to purely observational analysis.See:Activation Patching and Causal Interventions
Circuit (neural): A subgraph of a neural network consisting of specific components (attention heads, MLP neurons, or features) and their connections that together implement an identifiable computational mechanism.See:The IOI Circuit: Discovery and Mechanism
Circuit Tracing: A methodology developed by Anthropic for mapping information flow through neural networks by decomposing computations into interpretable features (via SAEs or transcoders) and tracing their causal connections.See:Circuit Tracing and Attribution Graphs
Completeness (circuit): A circuit evaluation criterion measuring whether the identified circuit accounts for all of the model's performance on a task. A complete circuit captures all relevant computation, with no important components left out.See:Circuit Evaluation: Faithfulness, Completeness, and Minimality
Composition (of attention heads): The mechanism by which attention heads in different layers interact through the residual stream, where earlier heads write information that later heads read. Three types exist: Q-composition, K-composition, and V-composition.See:Composition and Virtual Attention Heads
Concept Erasure: Transforming representations so a chosen label is no longer recoverable by a specified class of predictors. LEACE targets linear predictors while minimizing expected squared distortion.See:Concept Erasure with LEACE
Contrast-Consistent Search (CCS): An unsupervised probing method that identifies truth directions in activation space without labeled data, by learning a probe whose outputs on a statement and its negation are consistent (summing to one) and confident (away from 0.5).See:Truthfulness Probing and the Geometry of Truth
Copy Suppression: An attention head algorithm pattern where the head attends to positions where a predicted token appeared earlier in context and outputs the negative of that token's unembedding direction, suppressing the model's tendency to predict tokens it has already seen.See:Copy Suppression
Counterfactual Resampling: A black-box technique for measuring the causal importance of individual reasoning steps: delete a step from a chain-of-thought trace, regenerate from that point many times, and measure the distributional shift in final answers via KL divergence.See:Counterfactual Resampling
Crosscoder: A variant of sparse autoencoders trained jointly on activations from multiple models (or the same model at different training stages), learning a shared feature dictionary that enables direct comparison of representations across models.See:Crosscoders

D

Dead Features: Features in a trained sparse autoencoder that never activate on any input in the dataset. Dead features represent wasted capacity and are a common training challenge for SAEs, addressed by techniques such as resampling.See:SAE Variants, Evaluation, and Limitations
Deception Detection: The application of mechanistic interpretability to identify when a model is generating outputs that conflict with its internal representations, potentially indicating deceptive or unfaithful behavior.See:Deception Detection and Alignment Faking
Depth Schedule: A function assigning a per-layer steering weight across all layers of a model, distributing the intervention across depth rather than concentrating it at a single layer.See:Multi-Layer Steering
Dictionary Learning: A class of methods that learn an overcomplete set of basis vectors (a dictionary) to represent data as sparse combinations. In MI, dictionary learning via sparse autoencoders is used to decompose superposed neural network activations into interpretable features.See:Sparse Autoencoders: Decomposing Superposition
Direct Logit Attribution (DLA): An interpretability technique that decomposes a model's output logits into additive contributions from each component (attention heads and MLP layers) by projecting their residual stream writes onto the unembedding direction for a token of interest.See:Direct Logit Attribution

E

EAP-IG: Edge Attribution Patching with Integrated Gradients. Replaces the single gradient evaluation in EAP with an average of gradients along the interpolation path from corrupted to clean activations, fixing zero-gradient failures and improving circuit faithfulness.See:Refined Attribution Methods
Embedding: The mapping from discrete tokens to continuous vectors at the start of a transformer. The embedding matrix converts each token into a vector in the residual stream, where it can be read and modified by subsequent layers.See:The Attention Mechanism

F

Faithfulness (circuit): A circuit evaluation criterion measuring how well the circuit reproduces the full model's behavior when run in isolation. A faithful circuit produces similar outputs to the complete model on the target task.See:Circuit Evaluation: Faithfulness, Completeness, and Minimality
Feature (in MI): A unit of neural network computation that represents a meaningful concept or pattern. In the context of superposition and SAEs, a feature is a direction in activation space corresponding to an interpretable property of the input.See:The Superposition Hypothesis
Feature Absorption: A failure mode in sparse autoencoders where a feature absorbs activation patterns that should be captured by other features, reducing the fidelity of the learned decomposition and making some features appear more general than they should be.See:SAE Variants, Evaluation, and Limitations
Feature Dashboard: A visualization tool that displays the top-activating dataset examples, logit effects, and other statistics for individual SAE features, helping researchers assess whether a feature corresponds to an interpretable concept.See:Feature Dashboards and Automated Interpretability
Feature Splitting: The phenomenon where a single feature in a smaller SAE splits into multiple, more specific features when the SAE dictionary size is increased, revealing finer-grained structure in model representations.See:Scaling Monosemanticity and Feature Steering
Feature Steering: A technique for controlling model behavior by artificially amplifying or suppressing specific SAE features during inference, effectively pushing model outputs toward or away from concepts those features represent.See:Scaling Monosemanticity and Feature Steering
First-Order Approximation: The linear approximation of a function near a point, built from its first derivative (or Jacobian in higher dimensions). Formally, f(x + Δx) ≈ f(x) + J·Δx. The name refers to the first term of the Taylor expansion; higher-order corrections are ignored.See:The Jacobian Lens
Function Vector: A direction in activation space that encodes an input-output function (such as 'translate English to French' or 'convert to past tense') rather than a static concept, enabling task transfer when added to unrelated prompts.See:Function Vectors

G

Gated SAE: A sparse autoencoder architecture that separates the decision of whether a feature is active from the estimation of its magnitude, using a gating mechanism that reduces shrinkage bias present in standard L1-regularized SAEs.See:SAE Variants, Evaluation, and Limitations

H

Hierarchical Orthogonality: The geometric property where a parent concept's representation vector is orthogonal to the difference vector between a child concept and the parent. This ensures that manipulating the parent (e.g., 'animal') does not shift the relative probabilities among children (e.g., 'mammal' vs. 'bird').See:Feature Geometry: Beyond One-Dimensional Directions
HookPoint: A named location in a TransformerLens model where activations can be intercepted, read, or modified during a forward pass. Every intermediate computation (attention patterns, residual stream states, MLP outputs) has a corresponding HookPoint.See:TransformerLens

I

In-Context Learning: The ability of a model to adapt its predictions using examples or instructions in the prompt, without updating its weights. Induction heads can support simple forms of this behavior by matching and continuing earlier patterns.See:Induction Heads and In-Context Learning
Induction Head: An attention-head pattern, usually enabled by an earlier head, that supports copying: after seeing 'A B ... A', it raises the probability of 'B'. This mechanism explains some forms of pattern completion, not all in-context learning.See:Induction Heads and In-Context Learning
Inference-Time Intervention (ITI): A technique that improves model truthfulness at inference time by shifting activations along truth-correlated directions identified via probing, implementing a probe-then-steer pipeline.See:Truthfulness Probing and the Geometry of Truth
Interchange Intervention Accuracy (IIA): The proportion of tested interchange interventions on which the neural network's output matches the high-level causal model's prediction. It measures support for a proposed alignment on the sampled interventions.See:The Causal Abstraction Framework
Interpretability Illusion: The risk that an interpretability method appears to provide a correct explanation of model behavior but actually misses the true mechanism, giving researchers false confidence in their understanding of the model.See:SAE Variants, Evaluation, and Limitations
Intervention Graph: A portable, serializable representation of a set of model interventions in nnsight. The intervention graph decouples the experimental design from model deployment, enabling the same experiment to run locally or on remote infrastructure.See:nnsight and nnterp
IOI Circuit: The circuit discovered in GPT-2 Small that performs the Indirect Object Identification task, consisting of name movers, backup name movers, S-inhibition heads, induction-like heads, and duplicate token heads working together to predict the correct indirect object.See:The IOI Circuit: Discovery and Mechanism
Irreducible Multi-Dimensional Feature: A feature that occupies more than one dimension in activation space and cannot be decomposed into independent one-dimensional features. Days of the week, for instance, form a circle in a 2D subspace where the two dimensions are coupled, not separable.See:Feature Geometry: Beyond One-Dimensional Directions

J

J-Space: The subspace of residual-stream activations spanned by sparse non-negative combinations of J-lens vectors. Empirically, this space carries the concepts the model is 'poised to verbalize' at a given layer and position.See:The Jacobian Lens
Jacobian Lens: A vocabulary-projection method that translates intermediate residual streams to output logits using the Jacobian of the final residual stream with respect to an intermediate layer, averaged over positions and contexts. Unlike the tuned lens, its linear map is derived from the model's own weights rather than learned.See:The Jacobian Lens
Jacobian Matrix: For a vector-valued function f: ℝⁿ → ℝᵐ, the matrix of all first-order partial derivatives. Entry (i, j) records how output component i responds to a small change in input component j. Its shape is m × n.See:The Jacobian Lens

K

Key Vector: The vector produced by applying the key weight matrix (W_K) to a token's representation. Key vectors are compared against query vectors via dot product to determine attention weights.See:The Attention Mechanism
Key-Value Memory (MLP): An interpretation of feed-forward layers where each neuron in the hidden layer has a key vector (a column of the input projection) that matches input patterns and a value vector (a row of the output projection) that promotes specific tokens or concepts in the output vocabulary.See:MLPs in Transformers
Knowledge Editing: Techniques for modifying specific factual associations stored in a language model's weights without retraining, typically by making targeted rank-one updates to MLP layers identified as causally responsible for the fact.See:Localized Fact Editing and Its Pitfalls
Knowledge Neuron: An MLP neuron whose activation is causally linked to the expression of a specific factual association, such that suppressing it degrades and amplifying it strengthens the model's recall of that fact.See:MLPs in Transformers

L

Latent Scaling: A diagnostic technique for crosscoders that measures how well a supposedly model-specific latent can explain activations in both models, detecting false attributions caused by L1 sparsity artifacts.See:Feature-Level Model Diffing
Layer Normalization: A normalization technique that rescales activations within each token's representation vector to have zero mean and unit variance, then applies learned affine parameters. Applied before each sublayer in pre-norm transformers, it stabilizes training but introduces a nonlinearity that couples all residual stream dimensions.See:Layer Normalization
LEACE: Least-squares Concept Erasure: a closed-form, covariance-aware transformation that makes a target label linearly guarded under its population assumptions with minimum expected squared distortion.See:Concept Erasure with LEACE
Linear Probe: A simple linear classifier trained on frozen model activations to test whether specific information (such as part of speech or sentiment) is linearly accessible at a given layer, providing evidence about what representations a model has learned.See:Probing Classifiers
Linear Representation Hypothesis: The hypothesis that neural networks represent concepts as linear directions in activation space, so that adding or subtracting these directions corresponds to adding or removing the associated concept.See:The Linear Representation Hypothesis
Logit Diff Amplification (LDA): A technique for surfacing rare model behaviors by sampling from a distribution that amplifies the logit-level differences between two model checkpoints (e.g., before and after fine-tuning), making training-induced behavioral changes more frequent and easier to detect.See:Logit Diff Amplification
Logit Lens: An observational technique that applies the model's unembedding matrix to intermediate residual stream states, converting hidden representations into vocabulary-space predictions to see how the model's output evolves across layers.See:The Logit Lens and Tuned Lens

M

Mechanistic Faithfulness: The requirement that every subset of components containing the causally important ones suffices to reproduce the network's output. Stronger than requiring that the output survives ablating all unimportant components together.See:Parameter Decomposition
Mechanistic Interpretability: A subfield of AI safety research focused on reverse-engineering the internal computations of neural networks to understand how they process information and produce outputs, moving beyond behavioral analysis to study the mechanisms themselves.See:What is Interpretability?
MELBO: Mechanistically Eliciting Latent Behaviors in language mOdels. An unsupervised method that discovers steering vectors by optimizing perturbations at an early layer to maximize activation change at a later layer, requiring no labeled examples or contrast pairs.See:Unsupervised Steering Vectors
Minimality (circuit): A circuit evaluation criterion measuring whether the circuit contains only components that are necessary for the task. A minimal circuit has no redundant parts whose removal would leave performance unchanged.See:Circuit Evaluation: Faithfulness, Completeness, and Minimality
MLP Layer: The feedforward sublayer in a transformer block, consisting of two linear projections with a nonlinearity between them. MLP layers process each token position independently and are believed to store factual knowledge and perform feature transformations.See:Direct Logit Attribution
Model Diffing: The practice of comparing internal representations between two related models (such as a base model and a fine-tuned version) to identify which features or circuits changed, using tools like crosscoders.See:Feature-Level Model Diffing
Monosemanticity: The idealized property of a unit having one coherent interpretation across its activation distribution. In practice, monosemanticity is graded and depends on the examples and tests used.See:Sparse Autoencoders: Decomposing Superposition
Multi-Head Attention: The mechanism of running multiple independent attention heads in parallel within a single layer, allowing the model to attend to different types of relationships simultaneously and combine their outputs.See:The Attention Mechanism
Multimodal Interpretability: The application of mechanistic interpretability techniques to models that process multiple input modalities (such as vision and language), investigating how representations are shared or transformed across modalities.See:Multimodal Mechanistic Interpretability

N

Name Mover Head: An attention head in the IOI circuit that attends to the indirect object name and copies it to the final token position, directly promoting that name in the output logits. Name movers are the output stage of the IOI circuit.See:The IOI Circuit: Discovery and Mechanism
Natural Language Autoencoder (NLA): A pair of language models trained jointly to autoencode a target model's activation through a natural-language bottleneck. A verbalizer writes a text description of the activation and a reconstructor rebuilds the activation from that text. Training minimizes reconstruction error, with no labels for what the activation encodes.See:Natural Language Autoencoders
Nucleus Sampling (top-p): A decoding strategy that samples from the smallest set of tokens whose cumulative probability exceeds a threshold p. Unlike top-k, it adapts the number of candidate tokens to the shape of the distribution, including fewer tokens when the model is confident and more when it is uncertain.See:Decoding Strategies

O

OV Circuit: The component of an attention head formed by the product of the value (W_V) and output (W_O) weight matrices. The OV circuit determines what information is written to the residual stream when a token is attended to.See:QK and OV Circuits
Overcomplete Dictionary: A set of dictionary vectors larger than the dimensionality of the space. Unlike a basis, it can represent one activation in multiple ways unless additional constraints such as sparsity make the code identifiable.See:Sparse Autoencoders: Decomposing Superposition

P

Parameter Decomposition: A family of methods that decompose a network's weights, rather than its activations, into a set of vectors in parameter space that sum to the original parameters and of which only a small number are needed on any given input.See:Parameter Decomposition
Parameter Subcomponent: A rank-one matrix, written as the outer product of a read direction and a write direction, that forms one term in the decomposition of a single weight matrix. Subcomponents can be clustered into full parameter components spanning several matrices.See:Parameter Decomposition
Path Patching: A refined variant of activation patching that isolates the effect of a specific computational path between two components, controlling for all other paths. This enables precise attribution of behavior to individual connections in a circuit.See:Attribution Patching and Path Patching
Polysemanticity: The property of a single neuron responding to multiple unrelated concepts. Polysemanticity is a consequence of superposition, where models encode more features than they have neurons by sharing neurons across features.See:The Superposition Hypothesis
Polytope Representation: The convex hull formed by the vector representations of a categorical concept's values in activation space. A binary concept forms a line segment, a ternary concept forms a triangle, and a k-valued concept forms a (k-1)-simplex.See:Feature Geometry: Beyond One-Dimensional Directions
Previous Token Head: An attention head that places substantial weight on the immediately preceding position and writes information about that token. It can supply predecessor-token information to an induction circuit.See:Induction Heads and In-Context Learning
Probing Classifier: A simple model (typically linear) trained on neural network activations to predict properties of the input, used as a diagnostic tool to test what information is encoded at different layers of a network.See:Probing Classifiers

Q

QK Circuit: The component of an attention head formed by the product of the query (W_Q) and key (W_K) weight matrices. The QK circuit determines which tokens attend to which other tokens by computing attention scores.See:QK and OV Circuits
Query Vector: The vector produced by applying the query weight matrix (W_Q) to a token's representation. Query vectors are compared against key vectors to compute attention scores that determine how much each position attends to others.See:The Attention Mechanism

R

Refusal Direction: A direction derived from activation differences between harmful and harmless prompts that causally mediates much of the tested models' refusal behavior. It is a mechanism for refusal, not a complete representation of safety or harmfulness.See:The Refusal Direction
Representation Reading: The practice of extracting information about a model's internal state by training classifiers on its activations, used in safety contexts to detect when a model may be reasoning about deception or harmful content.See:Understanding Safety Mechanisms and MI-Based Monitoring
Residual Stream: The central communication channel in a transformer, implemented as skip connections that allow each layer's output to be added to a running sum. All attention heads and MLP layers read from and write to this shared stream.See:Transformer Architecture Intro
RMSNorm: A simplified variant of layer normalization that normalizes by the root mean square of activations without centering by the mean. Used in LLaMA, Gemma, and other modern architectures for its computational efficiency and comparable performance.See:Layer Normalization
ROME: Rank-One Model Editing: a method for editing factual associations by performing a rank-one update to a specific MLP layer's weights, modifying the key-value mapping for a targeted fact while attempting to preserve other knowledge.See:Localized Fact Editing and Its Pitfalls

S

Safety Monitor: A system that uses mechanistic interpretability techniques (such as probes or feature monitors) to detect potentially dangerous model behaviors at inference time, enabling intervention before harmful outputs are produced.See:Understanding Safety Mechanisms and MI-Based Monitoring
Self-Repair: The phenomenon where ablating or patching a model component causes later components to compensate, partially restoring the original behavior. Self-repair means that ablation effects systematically understate component importance.See:Self-Repair in Language Models
Sleeper Agent: A model with a hidden backdoor that behaves normally under standard conditions but activates harmful behavior when a specific trigger is present. Detecting sleeper agents is a motivating application of MI for safety.See:Detecting Sleeper Agents with Mechanistic Interpretability
Sparse Autoencoder (SAE): A dictionary-learning model that encodes activations into a wider sparse latent vector and reconstructs them with learned decoder directions. Some latents admit useful human interpretations.See:Sparse Autoencoders: Decomposing Superposition
Static Interaction Strength: The data-independent coupling between a query-side and a key-side parameter subcomponent, given by the dot product of their write directions scaled by the norms of their read directions. Under rotary position embeddings it depends on the query-key offset.See:Circuits in Parameter Space
Superposition: The phenomenon where neural networks represent more features than they have dimensions by encoding features as nearly orthogonal directions in activation space, allowing models to store more concepts than their parameter count would naively permit.See:The Superposition Hypothesis

T

Temperature (sampling): A hyperparameter that scales logits before the softmax during text generation. Temperature below 1 sharpens the distribution (more deterministic), temperature above 1 flattens it (more random), and temperature approaching 0 recovers greedy decoding.See:Decoding Strategies
Thought Anchor: A reasoning step with disproportionately high counterfactual importance, meaning the model's final answer distribution changes substantially when that step is removed. Plan generation and uncertainty management steps tend to be thought anchors.See:Counterfactual Resampling
Tracing Context: A Python context manager in nnsight where code is captured rather than executed immediately. Operations on model internals within a tracing context build up an intervention graph that is executed as a batch when the context exits.See:nnsight and nnterp
Transcoder: A sparse autoencoder variant applied to MLP layers that maps from MLP inputs to MLP outputs, learning interpretable features that describe what transformations the MLP performs rather than what it represents.See:Transcoders: Interpretable MLP Replacements
Tuned Lens: An improvement on the logit lens that trains a learned affine transformation at each layer (rather than reusing the final unembedding matrix), producing more accurate predictions of the model's evolving computation at intermediate layers.See:The Logit Lens and Tuned Lens

U

Unembedding: The learned linear map from the final residual representation to vocabulary logits. A subsequent softmax converts those logits into probabilities.See:The Logit Lens and Tuned Lens
Universality: The hypothesis that different neural networks trained on similar tasks converge on similar internal representations and circuits, suggesting that certain computational solutions are natural or optimal for given problems.See:Universality Across Models

V

Value Vector: The vector produced by applying the value weight matrix (W_V) to a token's representation. Value vectors carry the content information that gets written to the residual stream, weighted by the attention pattern.See:The Attention Mechanism
Virtual Attention Head: An emergent attention head that does not correspond to any single physical head in the model but arises from the composition of two or more heads across different layers communicating through the residual stream.See:Composition and Virtual Attention Heads

W

Weight Sparsity: A training constraint that forces most entries of every weight matrix to be exactly zero, so each neuron reads from and writes to only a few residual channels. Distinct from activation sparsity and from mixture-of-experts sparsity, which leave weights dense.See:Training Models to Be Interpretable