LatentQA and Latent Interpretation Tuning

Interpretation as Question-Answering

Previous approaches to hidden state decoding generate explanations: given an activation, produce a description. But explanations are one-directional. What if we want to ask specific questions about an activation? "Does this representation encode sentiment?" "What entity is being referenced?" "Is harmful content present?"

LatentQA, introduced by Pan et al. [1]LatentQA: Teaching LLMs to Decode Activations Into Natural Language
Pan, A., Chen, L., Steinhardt, J.
arXiv, 2024, reframes activation interpretation as question-answering. Rather than generating open-ended descriptions, we train a decoder model to answer arbitrary questions about what a representation contains.

LatentQA: A task formulation where a model answers open-ended questions about neural network activations in natural language. The decoder model takes an activation as input (alongside a question) and produces an answer describing relevant properties of that activation.

The Visual Instruction Tuning Analogy

The approach draws a direct analogy to visual instruction tuning (VIT), which trains multimodal models to answer questions about images. In VIT:

• Input: An image + a question
• Output: A natural language answer
• Training: Paired datasets of (image, question, answer)

LatentQA follows the same pattern:

• Input: An activation + a question
• Output: A natural language answer
• Training: Paired datasets of (activation, question, answer)

LatentQA treats activations as another dense input modality. Just as a vision-language model learns a mapping from images and questions to answers, an activation decoder learns that mapping from supervised activation–question–answer triples.

Latent Interpretation Tuning (LIT)

The training procedure, called Latent Interpretation Tuning, works as follows:

1. Collect activations. Run a target model on diverse inputs, extracting hidden representations from chosen layers.

2. Generate Q&A pairs. For each activation, create question-answer pairs that describe its content. Questions can cover:

   • What concept is represented
   • What entity or object is referenced
   • What sentiment or attitude is encoded
   • Whether specific properties are present

3. Train the decoder. Fine-tune a language model to take (activation, question) pairs as input and produce the correct answer. The activation is fed as an embedding that the decoder conditions on.

4. Evaluate generalization. Test on held-out activations and questions to verify that the decoder learns general interpretation skills, not just memorized associations.

The resulting decoder can answer novel questions about novel activations, enabling flexible interrogation of what representations encode.

Applications

The same decoder supports extraction, activation optimization, and safety analysis.

Extraction

We can query activations to extract specific knowledge. Given the activation produced when a model processes "The Eiffel Tower is in ___," we can ask:

• "What city is being referenced?" → "Paris"
• "What country is relevant?" → "France"
• "What type of landmark is this?" → "Tower/monument"

This goes beyond vocabulary projection. We are not just asking "what token would be predicted?" but "what knowledge is encoded about this entity?"

The authors demonstrate extracting relational knowledge and uncovering system prompts from model activations, capabilities that help understand what models internally represent about their inputs and context.

Control

Because the decoder provides a differentiable loss over activations, we can optimize activations to satisfy desired properties. If the decoder answers "negative" when asked about sentiment, we can compute gradients to adjust the activation toward "positive."

This enables:

• Debiasing: Modifying activations so the decoder reports reduced stereotyping
• Sentiment steering: Adjusting generation tone by optimizing activation properties
• Concept editing: Changing what properties the decoder detects in a representation

The decoder becomes not just a reading tool but a control signal for modifying representations.

Safety Analysis

A safety-relevant application is testing whether activations contain information that the target model refuses to express in ordinary text. The reported queries target:

• Whether bioweapon recipes are encoded in representations
• Whether hacking techniques are represented
• Whether the model has learned dangerous capabilities it normally refuses to express

In these experiments, the decoder recovers information that the target model does not emit under the ordinary prompt. That may reflect information blocked by refusal, but it can also reflect what the separately trained decoder reconstructs from correlated activation cues.

Pause and think: The dual-use nature of LatentQA

LatentQA can reveal harmful capabilities that models refuse to express. This is valuable for safety research: we need to know what models know to evaluate safety measures.

But the same capability could potentially be used to extract dangerous knowledge. How should we think about this dual-use concern?

Consider:

• Is revealing latent harmful knowledge more or less dangerous than leaving it hidden?
• Does knowing where harmful knowledge resides help or hurt safety interventions?
• Should LatentQA tools be open or restricted?

There are no easy answers. The tension between transparency and misuse potential is a recurring theme in interpretability research.

Architecture

The decoder receives activations through a projection into its input space. Layer choice, presentation method, and decoder capacity determine what it can recover:

Which layers to read. Different layers encode different information. Early layers may encode syntactic features; later layers may encode semantic content. The decoder can be trained on activations from specific layers or from all layers.

How to present activations. The activation can be concatenated with the question embedding, inserted at a special position, or processed through cross-attention. The choice affects what information the decoder can access.

Decoder capacity. Larger decoders may answer more complex questions but require more training data. The authors find that moderate-sized decoders generalize well when trained on diverse Q&A pairs.

Relation to Prior Methods

LatentQA builds on insights from earlier approaches:

Method	Input	Output	Format
Patchscopes	Activation + template	Completion	Fill-in-blank
SelfIE	Activation + prompt	Description	Freeform
Trained explanation	Activation	Explanation	Freeform
LatentQA	Activation + question	Answer	Q&A

Questions constrain the output space, permit targeted follow-ups, and supply differentiable objectives:

• Questions constrain the output space, making evaluation easier
• Arbitrary questions enable flexible interrogation
• Differentiable answers enable optimization for control

Limitations

Training data quality. Q&A pairs require either manual annotation or heuristic generation. Incorrect answers in training data will be learned by the decoder.

Question scope. The decoder can only answer questions about properties represented in its training distribution. Novel question types may produce unreliable answers.

Activation-to-embedding mapping. The projection from activations to decoder input may lose information. Properties present in activations may not be accessible through the learned mapping.

Generalization bounds. While decoders generalize to new activations within the training distribution, performance on very different models or very different inputs may degrade.

Why LatentQA Matters

LatentQA shows that multimodal-style supervision can produce flexible readouts of activations. Its answers are predictions by a trained decoder, so they require held-out evaluation and causal follow-up like any other probe. For questions about a model's own state, concept injection illustrates how a known internal intervention can test whether an answer is grounded in that state.

The Q&A format is particularly powerful because it:

• Enables systematic benchmarking (correctness of answers can be evaluated)
• Supports diverse queries without retraining
• Provides a differentiable interface for control

This sets the stage for a broader question: can one decoder handle many interpretation tasks and transfer to task types withheld from training? Activation Oracles studies that goal with a more diverse training mixture.

Looking Ahead

LatentQA trains decoders on specified interpretation tasks. Activation Oracles asks how far a broader task mixture transfers to held-out questions and activation settings without additional task-specific training.