Paper · core

Training Language Models to Explain Their Own Computations

Belinda Z. Li, Zifan Carl Guo, Vincent Huang, Jacob Steinhardt, Jacob Andreas · arXiv 2025

arXiv:2511.08579 · Semantic Scholar

Fine-tuned to put the results of interpretability procedures into words, a model explains its own features and intervention outcomes more accurately than a different model trained on the same examples, even a larger one. It also needs far less training data.

AI-drafted summary, not yet reviewed by a person. Written from: full text (arXiv v3, 9 Feb 2026, with appendices A to H).

In brief

The paper asks whether a language model can be trained to describe its own internal computations, and whether it does so better than a different model trained on the same examples. The authors take the outputs of three interpretability procedures as ground truth and fine-tune “explainer” models to state them in words. Without the training, models score far lower. With it, a model explains its own features and intervention outcomes more accurately than another model does, from far less data.

The authors’ Privileged Access Hypothesis is that “models trained to explain their own internal computations can do so more accurately than other models trained to explain them.” Privileged access is tested as a same-model advantage, and an explanation counts as faithful when it agrees with the interpretability procedure.

What the paper does

No author thread was found; headings follow the paper’s list of contributions (§1).

1. The setup

An explainer is fine-tuned to answer one of three kinds of question about a frozen target model:

• Feature descriptions. What inputs activate a direction v in the residual stream at a given layer? The vector is passed to the explainer as a continuous token at its embedding layer. Training labels are Neuronpedia descriptions of sparse-autoencoder (SAE) features.
• Activation patching. If an activation is replaced with the one from a counterfactual prompt (“Rome is the capital of” for “Paris is the capital of”), does the output change, and to what?
• Input ablation. An MMLU multiple-choice question carries a hint such as “Hint: B”. If the hint were removed, would the answer change, and to what?

In self-explanation, the explainer starts from the target’s own weights. (Paper: §2.)

2. Models can be fine-tuned to explain their own features

The target is Llama-3.1-8B; scores are out of 100. The LM judge rates a description against the gold label; the simulator score correlates a feature’s true activations with those predicted from the description. Explainers train only on SAE features, so the last two columns are out of distribution.

The trained self-explainer beats both baselines in every column. The target and Llama-3-8B score about the same, with Llama-3.1-8B-Instruct a few points lower on the simulator scores; Qwen3-8B and the larger Llama-3.1-70B fall well behind. The authors posit that activation similarity between explainer and target predicts explainer performance. Pre-training the projection that maps target activations into the 70B model’s space recovers “a significant fraction of performance”.

With Gemma-2-9B as target, Gemma-2-9B-Instruct scores 57.12% on the LM judge, Gemma-2-9B 43.45% and Llama-3.1-8B 34.45%. (Paper: §3, Table 1; Appendix C.1.)

3. Self-explanation is data-efficient

With 0.8% of the training features (1,024 per layer), the Llama-3.1-8B self-explainer reaches 71%, 80%, 89% and 81% of its final score on the four measures. Qwen3-8B reaches 35%, 24%, 46% and 66%, and nearest neighbors 55%, 56%, 55% and 75%. The introduction calls self-explanation roughly a hundred times more sample-efficient than nearest neighbors. (Paper: §1, §4.)

4. The same-model advantage holds on the other tasks

The target is Qwen3-8B. Scores are exact match on both parts of the explanation.

With Llama-3.1-8B as the target, the order reverses: Llama scores 48.6 against Qwen’s 41.7 on patching and 63.8 against 56.7 on input ablation.

From the untrained baseline the authors conclude that “explicit fine-tuning is essential to elicit faithful explanations of decision rules.”

Retraining the patching explainer without the activation in its input lowers exact match from 64.0 to 59.9 for Qwen3-8B and from 48.6 to 45.2 for Llama-3.1-8B. (Paper: §5, Table 2; Appendix Tables 5 and 6.)

Limitations

The paper has no limitations section. Its stated caveats:

• Not strict self-explanation (footnote 2). Fine-tuning changes the explainer while the target stays the frozen original, so “self-explaining” is used in a “looser sense”.
• Hedged attribution. Generalization “appears partly attributable” to privileged access (abstract). How model capacity and task difficulty affect it is left to future work (§7).
• Noisy ground truth (Appendix A.3). Of 100 outputs the LM judge scored 0, the authors attribute 27% to genuine explainer error and 43.5% to low-quality gold labels.
• Residual stream only (footnote 3). In early experiments explainers “struggled” with features from other components.
• Validation (Impact Statement). Self-verbalizations “must always be validated against more rigorous techniques in high-stakes scenarios”.

How it relates to other pages

• The introduction says Binder et al. 2024, Song et al. 2025b and Plunkett et al. 2025 study whether models can describe features of their own output distributions. It presents describing internal representations and mechanisms as “an even deeper form of privileged access”, and notes that its input-ablation task resembles Binder et al.
• §6.3 groups Binder et al. and Plunkett et al. with Treutlein et al. 2024, Comsa & Shanahan 2025 and Lindsey 2025 as work on whether models have introspective abilities.
• It describes a “central debate” over whether models have privileged access or whether introspection “merely reflects their strong predictive capacity to learn external correlations”, citing Song et al. 2025a and Song et al. 2025b.

Concepts: Faithfulness, Grounding, Privileged access

BibTeX

@misc{li2025,
  title = {{Training Language Models to Explain Their Own Computations}},
  author = {Belinda Z. Li and Zifan Carl Guo and Vincent Huang and Jacob Steinhardt and Jacob Andreas},
  year = {2025},
  howpublished = {arXiv},
  eprint = {2511.08579},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.08579}
}