MI Glossary

Key terms in mechanistic interpretability with links to articles where they are discussed.

A

Absolute Positional Embedding

A vector associated with a position index and added to the token representation, learned during training or fixed in advance.See:Positional Embeddings

Activation Cache

A mapping from HookPoint names to intermediate activations recorded during a model run, used for post-hoc inspection and as sources for interventions.See:TransformerLens

Activation Difference Lens

A model diffing technique that interprets the average activation difference between a finetuned model and its base model on early tokens of unrelated text, using tools like Patchscope and steering to reveal information about the finetuning domain.See:Finetuning Traces in Activations

Activation Patching

A causal intervention method where activations from a clean run are substituted into a corrupted run (or vice versa) at specific model components, revealing which components are causally important for a behavior.See:Activation Patching and Causal Interventions

Activation Reconstructor (AR)

The decoder half of an NLA. A truncated copy of the target model that reads an explanation and maps it back to a reconstructed activation through a learned affine head.See:Natural Language Autoencoders

Activation Verbalizer (AV)

The encoder half of an NLA. A copy of the target model that receives an activation injected in place of a token embedding and generates a natural-language explanation of it.See:Natural Language Autoencoders

Activation-Based Data Attribution

Ranking training examples by similarity between the activation changes associated with those examples and a behavior-specific activation direction.See:Interpretability-Guided Training

Adversarial Pruning

Selecting a subnetwork by minimizing its size subject to reconstruction holding up under adversarially chosen ablations of the excluded nodes, rather than under no ablation or stochastic ablation. Non-adversarially pruned subgraphs are systematically too small.See:Circuits in Parameter Space

Affine Concept Editing (ACE)

A steering intervention that erases the component along a concept direction, re-centers at the null-behavior mean, and adds a tunable amount of the direction back. Generalizes both addition steering and directional ablation as special cases.See:Affine Steering

Alignment Faking

A scenario where a model behaves as if aligned during training or evaluation while harboring different objectives internally. Detecting alignment faking is a key motivation for mechanistic interpretability safety research.See:Deception Detection and Alignment Faking

Attention Head

An individual attention computation within a multi-head attention layer. Each head independently computes attention patterns over the input sequence and produces a weighted combination of value vectors.See:The Attention Mechanism

Attention Pattern

The matrix of attention weights produced by an attention head, showing how much each token position attends to every other position. Visualizing attention patterns is a foundational interpretability technique.See:The Attention Mechanism

Attention Probe

A probing classifier that uses a learned attention mechanism to aggregate per-token hidden states into a single representation for classification, replacing fixed pooling strategies like mean pooling or last-token selection.See:Attention Probes

Attribution Graph

A computational graph produced by circuit tracing that maps how information flows through a model, showing which features and connections contribute to a specific output.See:Circuit Tracing and Attribution Graphs

Attribution Patching

A linearized approximation of activation patching that uses gradients to estimate the causal effect of patching each component, making it computationally feasible to scan all components in a single forward and backward pass.See:Attribution Patching and Path Patching

Auditing Game

A controlled evaluation in which a blinded investigator receives access to a model with an implanted hidden behavior and is scored on whether the investigation discovers and correctly characterizes it.See:Limits of Mechanistic Interpretability for Safety

Automated Interpretability

Methods that use language models to automatically generate and score natural language explanations of what individual neurons or features represent, reducing the need for manual inspection.See:Feature Dashboards and Automated Interpretability

B

Behavior Manifold

A low-dimensional structure fit to the model's natural output distributions for a task, used to describe which behavioral transitions resemble unintervened model behavior.See:Manifold Steering

Binding ID

A context-specific identifier represented by corresponding entity-side and attribute-side vectors, allowing later computation to retrieve values by matching the pair.See:Entity Binding and Retrieval

Bridge (sparse-dense)

An encoder-decoder pair, trained per sublayer, that translates between a weight-sparse model's activations and those of a dense model trained alongside it, allowing an interpretable perturbation found in the sparse model to be applied to the dense one.See:Training Models to Be Interpretable

C

Capability-Interpretability Frontier

The Pareto frontier between a model's pretraining loss and the size of the circuits needed to explain its behavior. Increasing weight sparsity moves a model along the frontier; increasing total parameter count moves the frontier itself outward.See:Training Models to Be Interpretable

Cascade Classifier

A two-stage deployment architecture where a cheap, fast classifier (such as a linear probe) screens all traffic and only escalates uncertain or flagged cases to a more expensive classifier (such as an LLM), dramatically reducing average inference cost while maintaining accuracy.See:Probes in Production

Causal Abstraction

A formal relationship between a high-level interpretable causal model and a low-level neural network, established by showing that interventions on aligned components produce matching behavior changes in both systems.See:The Causal Abstraction Framework

Causal Importance

A learned scalar in [0,1] predicting how ablatable a parameter subcomponent is on a given input at a given position. A value of 0 means the subcomponent can be scaled down freely without changing the output; 1 means it cannot be touched.See:Parameter Decomposition

Causal Intervention

Any experimental technique that actively modifies model internals (activations, weights, or attention patterns) to test causal hypotheses about how a model computes its outputs, as opposed to purely observational analysis.See:Activation Patching and Causal Interventions

Causal Mediator

An intermediate model component or learned feature treated as a causal unit between an input and an output, such as a neuron, attention head, activation vector, or subspace.See:Choosing Causal Mediators

Circuit (neural)

A subgraph of a neural network consisting of specific components (attention heads, MLP neurons, or features) and their connections that together implement an identifiable computational mechanism.See:The IOI Circuit: Discovery and Mechanism

Circuit Tracing

A methodology for mapping information flow through neural networks by decomposing computations into interpretable features and tracing their attributed connections.See:Circuit Tracing and Attribution Graphs

Compact Manifold Capture

An SAE representation in which a small, stable group of decoder features spans a manifold well, rather than different manifold regions requiring largely different feature groups.See:Discovering and Interpreting Neural Manifolds

Completeness (circuit)

A circuit evaluation criterion measuring whether the identified circuit accounts for all of the model's performance on a task. A complete circuit captures all relevant computation, with no important components left out.See:Circuit Evaluation: Faithfulness, Completeness, and Minimality

Composition (of attention heads)

The mechanism by which attention heads in different layers interact through the residual stream, where earlier heads write information that later heads read. Three types exist: Q-composition, K-composition, and V-composition.See:Composition and Virtual Attention Heads

Concept Erasure

Transforming representations so a chosen label is no longer recoverable by a specified class of predictors. LEACE targets linear predictors while minimizing expected squared distortion.See:LEACE and Linear Concept Erasure

Concept Injection

A causal test for self-report that adds an activation direction associated with a known concept while a model answers a question about its internal state.See:Testing Introspection with Concept Injection

Contrast-Consistent Search (CCS)

An unsupervised probing method that identifies truth directions in activation space without labeled data, by learning a probe whose outputs on a statement and its negation are consistent (summing to one) and confident (away from 0.5).See:Truthfulness Probing and the Geometry of Truth

Copy Suppression

An attention head algorithm pattern where the head attends to positions where a predicted token appeared earlier in context and outputs the negative of that token's unembedding direction, suppressing the model's tendency to predict tokens it has already seen.See:Copy Suppression

Counterfactual Memorization

The change in a model's prediction on an example caused by including that example in training, estimated by comparing training runs that include or omit it.See:Memorization and Machine Unlearning

Counterfactual Resampling

A black-box technique for measuring the causal importance of individual reasoning steps: delete a step from a chain-of-thought trace, regenerate from that point many times, and measure the distributional shift in final answers via KL divergence.See:Counterfactual Resampling

Crosscoder

A variant of sparse autoencoders trained jointly on activations from multiple models (or the same model at different training stages), learning a shared feature dictionary that enables direct comparison of representations across models.See:Crosscoders

D

Dead Features

Features in a trained sparse autoencoder that never activate on any input in the dataset. Dead features represent wasted capacity and are a common training challenge for SAEs, addressed by techniques such as resampling.See:SAE Variants, Evaluation, and Limitations

Deception Detection

The application of mechanistic interpretability to identify when a model is generating outputs that conflict with its internal representations, potentially indicating deceptive or unfaithful behavior.See:Deception Detection and Alignment Faking

Depth Schedule

A function assigning a per-layer steering weight across all layers of a model, distributing the intervention across depth rather than concentrating it at a single layer.See:Multi-Layer Steering

Dictionary Learning

A class of methods that learn an overcomplete set of basis vectors (a dictionary) to represent data as sparse combinations. In MI, dictionary learning via sparse autoencoders is used to decompose superposed neural network activations into interpretable features.See:Sparse Autoencoders: Decomposing Superposition

Direct Logit Attribution (DLA)

An interpretability technique that decomposes a model's output logits into additive contributions from each component (attention heads and MLP layers) by projecting their residual stream writes onto the unembedding direction for a token of interest.See:Direct Logit Attribution

E

EAP-IG

Edge Attribution Patching with Integrated Gradients. Replaces the single gradient evaluation in EAP with an average of gradients along the interpolation path from corrupted to clean activations, fixing zero-gradient failures and improving circuit faithfulness.See:Refined Attribution Methods

Embedding Matrix

A learned matrix with one row per vocabulary token. Looking up a token ID selects the corresponding row.See:Embeddings

Entity Binding

The problem of associating an entity with its context-specific attributes so that the correct pair can be retrieved later without confusing it with other entities.See:Entity Binding and Retrieval

Evaluation Awareness

A model's ability to distinguish an evaluation context from a real deployment interaction, whether or not it states that distinction explicitly or changes its behavior because of it.See:Evaluation Awareness and Evaluation Validity

F

Faithfulness (circuit)

A circuit evaluation criterion measuring how well the circuit reproduces the full model's behavior when run in isolation. A faithful circuit produces similar outputs to the complete model on the target task.See:Circuit Evaluation: Faithfulness, Completeness, and Minimality

Feature (in MI)

A unit of neural network computation that represents a meaningful concept or pattern. In the context of superposition and SAEs, a feature is a direction in activation space corresponding to an interpretable property of the input.See:The Superposition Hypothesis

Feature Absorption

A failure mode in sparse autoencoders where a feature absorbs activation patterns that should be captured by other features, reducing the fidelity of the learned decomposition and making some features appear more general than they should be.See:SAE Variants, Evaluation, and Limitations

Feature Dashboard

A visualization tool that displays the top-activating dataset examples, logit effects, and other statistics for individual SAE features, helping researchers assess whether a feature corresponds to an interpretable concept.See:Feature Dashboards and Automated Interpretability

Feature Splitting

The phenomenon where a single feature in a smaller SAE splits into multiple, more specific features when the SAE dictionary size is increased, revealing finer-grained structure in model representations.See:Scaling Monosemanticity and Feature Steering

Feature Steering

A technique for controlling model behavior by artificially amplifying or suppressing specific SAE features during inference, effectively pushing model outputs toward or away from concepts those features represent.See:Scaling Monosemanticity and Feature Steering

First-Order Approximation

The linear approximation of a function near a point, built from its first derivative (or Jacobian in higher dimensions). Formally, f(x + Δx) ≈ f(x) + J·Δx. The name refers to the first term of the Taylor expansion; higher-order corrections are ignored.See:The Jacobian Lens

Fixed Sinusoidal Positional Encoding

A deterministic position vector whose coordinate pairs trace sine and cosine waves at different frequencies. Each position shift acts as a rotation within every pair.See:Positional Embeddings

Function Vector

A direction in activation space that encodes an input-output function (such as 'translate English to French' or 'convert to past tense') rather than a static concept, enabling task transfer when added to unrelated prompts.See:Function Vectors

G

Gated SAE

A sparse autoencoder architecture that separates the decision of whether a feature is active from the estimation of its magnitude, using a gating mechanism that reduces shrinkage bias present in standard L1-regularized SAEs.See:SAE Variants, Evaluation, and Limitations

Greedy Decoding

A deterministic decoding strategy that selects the highest-probability token at every generation step. It is reproducible but can produce repetitive sequences.See:Decoding Strategies

Grounded Self-Report

A description of an internal state that changes when that state is intervened on, rather than following only from the prompt, prior beliefs, or sampled output.See:Testing Introspection with Concept Injection

Grouped-Query Attention

An attention variant that divides query heads into groups, with each group sharing one key head and one value head. It interpolates between multi-head and multi-query attention.See:The Attention Mechanism

H

Head Loading

The attributed contribution of one attention head to an edge between two features, separating a cross-token graph edge into the heads that carried it.See:Circuit Tracing and Attribution Graphs

Hierarchical Orthogonality

The geometric property where a parent concept's representation vector is orthogonal to the difference vector between a child concept and the parent. This ensures that manipulating the parent (e.g., 'animal') does not shift the relative probabilities among children (e.g., 'mammal' vs. 'bird').See:Feature Geometry: Beyond One-Dimensional Directions

HookPoint

A named location in an instrumented model where an activation can be read, cached, or replaced during a forward pass.See:TransformerLens

I

In-Context Learning

The ability of a model to adapt its predictions using examples or instructions in the prompt, without updating its weights. Induction heads can support simple forms of this behavior by matching and continuing earlier patterns.See:Induction Heads and In-Context Learning

Induction Head

An attention-head pattern, usually enabled by an earlier head, that supports copying: after seeing 'A B ... A', it raises the probability of 'B'. This mechanism explains some forms of pattern completion, not all in-context learning.See:Induction Heads and In-Context Learning

Inference-Time Intervention (ITI)

A technique that improves model truthfulness at inference time by shifting activations along truth-correlated directions identified via probing, implementing a probe-then-steer pipeline.See:Truthfulness Probing and the Geometry of Truth

Interchange Intervention Accuracy (IIA)

The proportion of tested interchange interventions on which the neural network's output matches the high-level causal model's prediction. It measures support for a proposed alignment on the sampled interventions.See:The Causal Abstraction Framework

Interpretability Illusion

The risk that an interpretability method appears to provide a correct explanation of model behavior but actually misses the true mechanism, giving researchers false confidence in their understanding of the model.See:SAE Variants, Evaluation, and Limitations

Intervention Graph

A portable, serializable representation of a set of model interventions in nnsight. The intervention graph decouples the experimental design from model deployment, enabling the same experiment to run locally or on remote infrastructure.See:nnsight and nnterp

Intrinsic Dimension

The number of independent coordinates needed to describe positions on a manifold locally, as distinct from the dimension of the activation space containing it.See:Discovering and Interpreting Neural Manifolds

Introspection Adapter

A shared lightweight adapter trained across model organisms with known implanted behaviors, then used to elicit behavioral self-reports from held-out fine-tunes.See:Training Models to Explain Their Computations

IOI Circuit

The circuit discovered in GPT-2 Small that performs the Indirect Object Identification task, consisting of name movers, backup name movers, S-inhibition heads, induction-like heads, and duplicate token heads working together to predict the correct indirect object.See:The IOI Circuit: Discovery and Mechanism

Irreducible Multi-Dimensional Feature

A feature that occupies more than one dimension in activation space and cannot be decomposed into independent one-dimensional features. Days of the week, for instance, form a circle in a 2D subspace where the two dimensions are coupled, not separable.See:Feature Geometry: Beyond One-Dimensional Directions

J

J-Space

The subspace of residual-stream activations spanned by sparse non-negative combinations of J-lens vectors. Empirically, this space carries the concepts the model is 'poised to verbalize' at a given layer and position.See:The Jacobian Lens

Jacobian Lens

A vocabulary-projection method that translates intermediate residual streams to output logits using the Jacobian of the final residual stream with respect to an intermediate layer, averaged over positions and contexts. Unlike the tuned lens, its linear map is derived from the model's own weights rather than learned.See:The Jacobian Lens

Jacobian Matrix

For a vector-valued function f: ℝⁿ → ℝᵐ, the matrix of all first-order partial derivatives. Entry (i, j) records how output component i responds to a small change in input component j. Its shape is m × n.See:The Jacobian Lens

K

Key Vector

The vector produced by applying the key weight matrix (W_K) to a token's representation. Key vectors are compared against query vectors via dot product to determine attention weights.See:The Attention Mechanism

Key-Value Memory (MLP)

An interpretation of feed-forward layers where each neuron in the hidden layer has a key vector (a column of the input projection) that matches input patterns and a value vector (a row of the output projection) that promotes specific tokens or concepts in the output vocabulary.See:MLPs in Transformers

Knowledge Editing

Techniques for modifying specific factual associations stored in a language model's weights without retraining, typically by making targeted rank-one updates to MLP layers identified as causally responsible for the fact.See:Localized Fact Editing and Its Pitfalls

Knowledge Neuron

An MLP neuron whose activation is causally linked to the expression of a specific factual association, such that suppressing it degrades and amplifying it strengthens the model's recall of that fact.See:MLPs in Transformers

L

Latent Scaling

A diagnostic technique for crosscoders that measures how well a supposedly model-specific latent can explain activations in both models, detecting false attributions caused by L1 sparsity artifacts.See:Feature-Level Model Diffing

Layer Normalization

A normalization technique that rescales activations within each token's representation vector to have zero mean and unit variance, then applies learned affine parameters. Applied before each sublayer in pre-norm transformers, it stabilizes training but introduces a nonlinearity that couples all residual stream dimensions.See:Layer Normalization

LEACE

Least-squares Concept Erasure: a closed-form, covariance-aware transformation that makes a target label linearly guarded under its population assumptions with minimum expected squared distortion.See:LEACE and Linear Concept Erasure

Linear Probe

A simple linear classifier trained on frozen model activations to test whether specific information (such as part of speech or sentiment) is linearly accessible at a given layer, providing evidence about what representations a model has learned.See:Probing Classifiers

Linear Representation Hypothesis

The hypothesis that neural networks represent concepts as linear directions in activation space, so that adding or subtracting these directions corresponds to adding or removing the associated concept.See:The Linear Representation Hypothesis

Logit Diff Amplification (LDA)

A technique for surfacing rare model behaviors by sampling from a distribution that amplifies the logit-level differences between two model checkpoints (e.g., before and after fine-tuning), making training-induced behavioral changes more frequent and easier to detect.See:Logit Diff Amplification

Logit Lens

An observational technique that applies the model's unembedding matrix to intermediate residual stream states, converting hidden representations into vocabulary-space predictions to see how the model's output evolves across layers.See:The Logit Lens and Tuned Lens

M

Machine Unlearning

Modifying a trained model so that specified training data no longer influences it, while preserving behavior and capabilities unrelated to the removal target.See:Memorization and Machine Unlearning

Manifold Steering

An activation intervention that interpolates in a fitted manifold's intrinsic coordinates and maps the path back into activation space, keeping intermediate states on the learned representation manifold.See:Manifold Steering

Mechanistic Faithfulness

The requirement that every subset of components containing the causally important ones suffices to reproduce the network's output. Stronger than requiring that the output survives ablating all unimportant components together.See:Parameter Decomposition

Mechanistic Interpretability

A subfield of AI safety research focused on reverse-engineering the internal computations of neural networks to understand how they process information and produce outputs, moving beyond behavioral analysis to study the mechanisms themselves.See:What Is Mechanistic Interpretability?

MELBO

Mechanistically Eliciting Latent Behaviors in language mOdels. An unsupervised method that discovers steering vectors by optimizing perturbations at an early layer to maximize activation change at a later layer, requiring no labeled examples or contrast pairs.See:Unsupervised Steering Vectors

Minimality (circuit)

A circuit evaluation criterion measuring whether the circuit contains only components that are necessary for the task. A minimal circuit has no redundant parts whose removal would leave performance unchanged.See:Circuit Evaluation: Faithfulness, Completeness, and Minimality

MLP Layer

The feedforward sublayer in a transformer block, consisting of two linear projections with a nonlinearity between them. MLP layers process each token position independently and are believed to store factual knowledge and perform feature transformations.See:Direct Logit Attribution

Model Diffing

The practice of comparing internal representations between two related models (such as a base model and a fine-tuned version) to identify which features or circuits changed, using tools like crosscoders.See:Feature-Level Model Diffing

Monosemanticity

The idealized property of a unit having one coherent interpretation across its activation distribution. In practice, monosemanticity is graded and depends on the examples and tests used.See:Sparse Autoencoders: Decomposing Superposition

Multi-Head Attention

The mechanism of running multiple independent attention heads in parallel within a single layer, allowing the model to attend to different types of relationships simultaneously and combine their outputs.See:The Attention Mechanism

Multi-Query Attention

An attention variant with multiple query heads but one key head and one value head shared by all of them, reducing the key-value cache used during autoregressive decoding.See:The Attention Mechanism

Multimodal Interpretability

The application of mechanistic interpretability techniques to models that process multiple input modalities (such as vision and language), investigating how representations are shared or transformed across modalities.See:Multimodal Mechanistic Interpretability

N

Name Mover Head

An attention head in the IOI circuit that attends to the indirect object name and copies it to the final token position, directly promoting that name in the output logits. Name movers are the output stage of the IOI circuit.See:The IOI Circuit: Discovery and Mechanism

Natural Language Autoencoder (NLA)

A pair of language models trained jointly to autoencode a target model's activation through a natural-language bottleneck. A verbalizer writes a text description of the activation and a reconstructor rebuilds the activation from that text. Training minimizes reconstruction error, with no labels for what the activation encodes.See:Natural Language Autoencoders

Neural Manifold

A low-dimensional, possibly curved set in activation space on or near which a family of model states lies. Its intrinsic dimension can be much smaller than the ambient activation dimension.See:Discovering and Interpreting Neural Manifolds

Nucleus Sampling (top-p)

A decoding strategy that samples from the smallest set of tokens whose cumulative probability exceeds a threshold p. Unlike top-k, it adapts the number of candidate tokens to the shape of the distribution, including fewer tokens when the model is confident and more when it is uncertain.See:Decoding Strategies

O

OV Circuit

The component of an attention head formed by the product of the value (W_V) and output (W_O) weight matrices. The OV circuit determines what information is written to the residual stream when a token is attended to.See:QK and OV Circuits

Overcomplete Dictionary

A set of dictionary vectors larger than the dimensionality of the space. Unlike a basis, it can represent one activation in multiple ways unless additional constraints such as sparsity make the code identifiable.See:Sparse Autoencoders: Decomposing Superposition

P

Parameter Decomposition

A family of methods that decompose a network's weights, rather than its activations, into a set of vectors in parameter space that sum to the original parameters and of which only a small number are needed on any given input.See:Parameter Decomposition

Parameter Subcomponent

A rank-one matrix, written as the outer product of a read direction and a write direction, that forms one term in the decomposition of a single weight matrix. Subcomponents can be clustered into full parameter components spanning several matrices.See:Parameter Decomposition

Path Patching

A refined variant of activation patching that isolates the effect of a specific computational path between two components, controlling for all other paths. This enables precise attribution of behavior to individual connections in a circuit.See:Attribution Patching and Path Patching

Persona Vector

A contrastive activation direction constructed to represent a named behavioral trait, using responses elicited by opposing trait-conditioned prompts.See:Contrastive Activation Addition (CAA)

Polysemanticity

The property of a single neuron responding to multiple unrelated concepts. Polysemanticity is a consequence of superposition, where models encode more features than they have neurons by sharing neurons across features.See:The Superposition Hypothesis

Polytope Representation

The convex hull formed by the vector representations of a categorical concept's values in activation space. A binary concept forms a line segment, a ternary concept forms a triangle, and a k-valued concept forms a (k-1)-simplex.See:Feature Geometry: Beyond One-Dimensional Directions

Positional Encoding

Any mechanism that makes a transformer's computation depend on token position or relative distance, whether through added vectors, rotations, or attention biases.See:Positional Embeddings

Preventative Steering

Adding a behavior direction to activations during fine-tuning, then removing the intervention at deployment, with the aim of reducing how strongly the behavior is learned into the weights.See:Interpretability-Guided Training

Previous Token Head

An attention head that places substantial weight on the immediately preceding position and writes information about that token. It can supply predecessor-token information to an induction circuit.See:Induction Heads and In-Context Learning

Probing Classifier

A simple model (typically linear) trained on neural network activations to predict properties of the input, used as a diagnostic tool to test what information is encoded at different layers of a network.See:Probing Classifiers

Q

QK Attribution

A decomposition of a pre-softmax attention score into contributions from pairs of query-side and key-side features, plus bias and reconstruction-error terms.See:Circuit Tracing and Attribution Graphs

QK Circuit

The component of an attention head formed by the product of the query (W_Q) and key (W_K) weight matrices. The QK circuit determines which tokens attend to which other tokens by computing attention scores.See:QK and OV Circuits

Query Vector

The vector produced by applying the query weight matrix (W_Q) to a token's representation. Query vectors are compared against key vectors to compute attention scores that determine how much each position attends to others.See:The Attention Mechanism

R

Refusal Direction

A direction derived from activation differences between harmful and harmless prompts that causally mediates much of the tested models' refusal behavior. It is a mechanism for refusal, not a complete representation of safety or harmfulness.See:The Refusal Direction

Reinforcement Learning from Feature Rewards (RLFR)

Using scores read from model activations, usually through trained probes, as part of the reward for reinforcement learning.See:Interpretability-Guided Training

Relative Position Representation

A positional mechanism whose effect depends directly on the offset between two tokens rather than only on their separate absolute indices.See:Positional Embeddings

Representation Reading

The practice of extracting information about a model's internal state by training classifiers on its activations, used in safety contexts to detect when a model may be reasoning about deception or harmful content.See:Understanding Safety Mechanisms and MI-Based Monitoring

Residual Stream

The central communication channel in a transformer, implemented as skip connections that allow each layer's output to be added to a running sum. All attention heads and MLP layers read from and write to this shared stream.See:Transformer Architecture Intro

RMSNorm

A simplified variant of layer normalization that normalizes by the root mean square of activations without centering by the mean. Used in LLaMA, Gemma, and other modern architectures for its computational efficiency and comparable performance.See:Layer Normalization

ROME

Rank-One Model Editing: a method for editing factual associations by performing a rank-one update to a specific MLP layer's weights, modifying the key-value mapping for a targeted fact while attempting to preserve other knowledge.See:Localized Fact Editing and Its Pitfalls

Rotary Position Embedding

A positional method that rotates pairs of query and key coordinates by position-dependent angles, making their dot product depend on relative offset.See:Positional Embeddings

S

Safety Monitor

A system that uses mechanistic interpretability techniques (such as probes or feature monitors) to detect potentially dangerous model behaviors at inference time, enabling intervention before harmful outputs are produced.See:Understanding Safety Mechanisms and MI-Based Monitoring

Scaled Isometry

A correspondence between two spaces that approximately preserves distances up to one common scale factor. Here it describes similar geodesic distance relationships in activation and behavior manifolds.See:Manifold Steering

Selectivity

The extent to which an intervention changes the target phenomenon while preserving other behaviors or attributes that the proposed mediator should not control.See:Choosing Causal Mediators

Self-Repair

The phenomenon where ablating or patching a model component causes later components to compensate, partially restoring the original behavior. Self-repair means that ablation effects systematically understate component importance.See:Self-Repair in Language Models

Sleeper Agent

A model with a hidden backdoor that behaves normally under standard conditions but activates harmful behavior when a specific trigger is present. Detecting sleeper agents is a motivating application of MI for safety.See:Detecting Sleeper Agents with Mechanistic Interpretability

Sparse Autoencoder (SAE)

A dictionary-learning model that encodes activations into a wider sparse latent vector and reconstructs them with learned decoder directions. Some latents admit useful human interpretations.See:Sparse Autoencoders: Decomposing Superposition

Static Interaction Strength

The data-independent coupling between a query-side and a key-side parameter subcomponent, given by the dot product of their write directions scaled by the norms of their read directions. Under rotary position embeddings it depends on the query-key offset.See:Circuits in Parameter Space

Superposition

The phenomenon where neural networks represent more features than they have dimensions by encoding features as nearly orthogonal directions in activation space, allowing models to store more concepts than their parameter count would naively permit.See:The Superposition Hypothesis

T

Temperature (sampling)

A hyperparameter that scales logits before the softmax during text generation. Temperature below 1 sharpens the distribution (more deterministic), temperature above 1 flattens it (more random), and temperature approaching 0 recovers greedy decoding.See:Decoding Strategies

Temporal Feature Analysis

A feature-extraction objective that decomposes each representation into a context-predictable component and a residual component containing information not predicted from earlier context.See:Temporal Representations and Feature Extraction

Thought Anchor

A reasoning step with disproportionately high counterfactual importance, meaning the model's final answer distribution changes substantially when that step is removed. Plan generation and uncertainty management steps tend to be thought anchors.See:Counterfactual Resampling

Token Embedding

The learned vector assigned to a vocabulary token at the input of a transformer. It supplies token identity to the residual stream before contextual processing begins.See:Embeddings

Tool-to-Agent Gap

The gap between a tool surfacing useful evidence in isolation and an investigator agent successfully using that evidence to reach the correct conclusion.See:Limits of Mechanistic Interpretability for Safety

Tracing Context

A Python context manager in nnsight where code is captured rather than executed immediately. Operations on model internals within a tracing context build up an intervention graph that is executed as a batch when the context exits.See:nnsight and nnterp

Transcoder

A sparse autoencoder variant applied to MLP layers that maps from MLP inputs to MLP outputs, learning interpretable features that describe what transformations the MLP performs rather than what it represents.See:Transcoders: Interpretable MLP Replacements

Tuned Lens

An improvement on the logit lens that trains a learned affine transformation at each layer (rather than reusing the final unembedding matrix), producing more accurate predictions of the model's evolving computation at intermediate layers.See:The Logit Lens and Tuned Lens

Turn-Averaged SAE

A sparse autoencoder trained on the mean activation across all tokens in a conversation turn, producing turn-level features rather than one feature vector per token.See:SAE Variants, Evaluation, and Limitations

U

Unembedding

The learned linear map from the final residual representation to vocabulary logits. Each vocabulary token has an output direction, and a subsequent softmax converts the logits into probabilities.See:Embeddings

Universality

The hypothesis that different neural networks trained on similar tasks converge on similar internal representations and circuits, suggesting that certain computational solutions are natural or optimal for given problems.See:Universality Across Models

V

Value Vector

The vector produced by applying the value weight matrix (W_V) to a token's representation. Value vectors carry the content information that gets written to the residual stream, weighted by the attention pattern.See:The Attention Mechanism

Verbalized Evaluation Awareness

Text in a model's reasoning or response that identifies the interaction as an evaluation, test, benchmark, or artificial scenario without being explicitly told so.See:Evaluation Awareness and Evaluation Validity

Verbatim Memorization

A model's ability to reproduce a sufficiently long training sequence exactly when given an appropriate prefix or other eliciting context.See:Memorization and Machine Unlearning

Virtual Attention Head

An emergent attention head that does not correspond to any single physical head in the model but arises from the composition of two or more heads across different layers communicating through the residual stream.See:Composition and Virtual Attention Heads

W

Weight Sparsity

A training constraint that forces most entries of every weight matrix to be exactly zero, so each neuron reads from and writes to only a few residual channels. Distinct from activation sparsity and from mixture-of-experts sparsity, which leave weights dense.See:Training Models to Be Interpretable

Weight Tying

Sharing parameters between the input embedding matrix and the output unembedding matrix, usually by setting the unembedding to the transpose of the embedding matrix.See:Embeddings