Paper · adjacent

Are language models aware of the road not taken? Token-level uncertainty and hidden state dynamics

Amir Zur, Atticus Geiger, Ekdeep Singh Lubana, Eric Bigelow · ICML 2025 workshop (Actionable Interpretability)

arXiv:2511.04527 · PDF

On a question where Llama-3.2 3B is unsure of its answer, adding a steering vector to its activations changes the answer at the tokens where the answer is still open and stops working at the token where the model commits: steering success follows the model's token-by-token distribution over final answers, at R = 0.57 for the main example and 0.64 averaged over four. A linear probe on the model's hidden state at a token predicts that distribution with lower loss than a probe on a second model's reading of the same text, 0.11 against 0.19 at layer 8.

AI-drafted summary, not yet reviewed by a person. Written from: full text (arXiv v1), appendix included; the ICML 2025 schedule, for the venue.

The paper in brief

• Problem. “When a language model generates text, the selection of individual tokens might lead it down very different reasoning paths, making uncertainty difficult to quantify.” (Abstract.)
• Why it matters. “Uncertainty estimation is a foundational problem in evaluating and interpreting LLMs”: a model that gives a wrong answer when “the probability of this answer is only 1%” “should be evaluated differently than another LLM that outputs the same wrong answer with 100% confidence” (§1). The method that measures this uncertainty token by token does it by re-sampling, at a cost “on the order of millions of tokens to analyze a single base path completion” (§2).
• Question. “In this work, we consider whether reasoning language models represent the alternate paths that they could take during generation.” (Abstract.)
• Answer. “In our experiments, we find a clear correlation between how uncertain a model is at different tokens, and how easily the model can be steered by controlling its activations.” And: “We also find that hidden activations can predict a model’s future outcome distribution, demonstrating that models implicitly represent the space of possible paths.” (Abstract.)

The two claims are two uses of a model’s hidden activations to test one hypothesis: “we use hidden activations to control and predict a language model’s uncertainty during chain-of-thought reasoning” (Abstract). The first is the control and the second the prediction, and the first leads to the second: once steering is found to follow uncertainty, the paper asks whether uncertainty can be read from the hidden state directly.

1. A model can be steered toward an answer at the tokens where its final answer is still uncertain, and no longer once it has committed.
2. A model’s hidden state at a token predicts its distribution over final answers at that token, and better than a second model’s reading of the same text does.

What the paper starts from

The method it inherits (§2). Forking Paths Analysis, from Bigelow et al. (2025), is “a method for estimating token-level uncertainty dynamics for black-box neural text generation”. The paper gives it in steps:

1. From a prompt, sample one completion with its token probabilities and the probabilities of the most likely alternate tokens at each position. This completion is the base path: “a given text completion sequence (a base path)”.
2. At each token index t, keep the base path up to t, put each alternate token in the next place, and sample completions again from there.
3. From each completion, “collect a final answer – or outcome – as a one-hot vector”.
4. Weight the outcomes by the probabilities of the tokens and the completions. The result is the outcome distribution o_t, one for each token index.

“The outcome distribution o_t thus represents the LLM’s uncertainty over final answers at each token index t.” (§2.) This is the sense of uncertainty in both claims: how the model’s final answer is spread over the options, as measured at a token.

A last step looks for the tokens where o_t jumps. “Bayesian Change Point Detection (CPD) models are used to estimate the probability” that a change point falls at each token index, and Bigelow et al. found “many striking cases of ‘Forking Tokens’ where the outcome distribution suddenly shifts from one distribution of certainty to another” (§2).

Its cost, which sets the paper’s task (§2). “A major drawback of this approach, however, is the computational cost”. So: “In this paper, we investigate whether hidden states provide meaningful information about the outcome distribution during generation of reasoning text, without needing to generate new tokens.”

The inference the argument rests on. The hypothesis is that a model represents the alternate paths it could take, and the paper tests it through the model’s hidden activations, in two ways. It steers the model at each token to see “how uncertainty dynamics can predict which points an LLM can or cannot be successfully steered”, and it asks “whether token-level uncertainty dynamics can be directly predicted from an LLM’s hidden states” (§1).

The two interventions (Figure 1). “By intervening on the generated tokens, we create branching paths to estimate the model’s outcome distribution. By intervening on the model’s activations, we steer the base generation towards a desired outcome.”

The setting (§3, Appendix A). One model, Llama-3.2 3B Instruct, answers with zero-shot chain-of-thought reasoning, that is, “prefixing ‘Let’s think step by step’ before the model’s completion” (§3). The questions are multiple-choice and come from three datasets: “GSM8k, a collection of grade-school math questions”, “AQuA, a collection of algebraic word problems” and “GPQA, a collection of graduate-level science questions” (Appendix A). Only questions the model is unsure of are kept. The authors “sample k = 10 generations from the LLM”, extract each answer by asking “What is your final answer?”, and “choose examples where the frequency of the most common answer across the 10 examples is between 4 and 6” (Appendix A).

Tools taken from earlier work. Forking Paths Analysis with its change point detection (Bigelow et al., 2025). Difference-in-means steering and the “linear mean-mass probe” (§3) of Marks & Tegmark (2023). Zero-shot chain-of-thought prompting (Kojima et al., 2022). The three datasets (Cobbe et al., 2021; Ling et al., 2017; Rein et al., 2024).

The argument, claim by claim

Claim 1: a model can be steered where its answer is still open, and not once it has committed

Adding a steering vector to the model’s activations changes its final answer at the tokens where the outcome distribution is still spread over several answers, and stops doing so at the token where the distribution collapses onto one.

How the model is steered, and how success is scored (§3). To steer toward an answer A, the authors “sample n = 500 generations whose final outcome is A and n generations whose final outcome is some other answer”. At a token position, the steering vector is the mean hidden activation of the first set less the mean of the second. Each position’s vector is also tried as a classifier of the two sets, and the one kept is the one whose classifier “has the highest classification accuracy on hidden activations from a held-out set of generations”. Those turn out to be late: “later tokens and middle layers (t = 200, l = 12 for the example in Figure 2) achieve the highest classification accuracy (∼ 80%)”.

That one vector is then applied all along the base path. “At every token position t in the base path, we apply difference-in-means steering by adding the pre-computed steering vector” to the activation there. The authors then “sample k = 10 continuations from the intervened model”, adding the vector again at every generated token. Steering success at a token is “the number of times out of k that the steered outcome is answer A, subtracted by the model’s original outcome distribution o_t(A) for A at token t”.

Evidence. The main example is a question about the profit on a book, one of the AQuA questions (§4). The paper points to one token in it: “sampling token t = 114 (‘loss’) drastically shifts the outcome distribution away from ‘B) 3% profit’, which is the correct answer, and towards ‘D) 3% loss’, which is the model’s generated answer” (§3). Steering follows the same course. “The success rate of steering in this case has very similar dynamics to the outcome distribution. Steering success is relatively high until t = 114, at which point it abruptly drops down to nearly zero.” (§3, Figure 2.)

From the example the authors form a hypothesis: “we hypothesize that controllability (measured by steering success) correlates with uncertainty (measured by the outcome distribution o_t). That is, models are most steerable when they are least certain about the final outcome.” (§3.) They test it across the tokens of the same generation, plotting steering success toward answer B against the probability o_t gives B, both on a log scale: “We find a moderate correlation between steering success and base probability (R = 0.57)” (§3, Figure 3).

• Objections it expects.
  • That the model would have reached the steered answer anyway. Success is counted over the model’s own rate: it is “the difference between steering success rate and base success rate without steering” (§3).
  • That this is one question. “Given the computational complexity of Forking Paths Analysis, we consider four examples from three reasoning datasets on which the LLM is uncertain.” (§3.) The other three are in Figure 6, whose caption reads “Across all examples, the outcome distribution and steering success display similar dynamics, with a sharp shift at the highlighted change points”, and the correlation “averaged across our four examples is R = 0.64” (Appendix A).
• Also reported. The steering vector is not taken from the token where the model commits. The positions that separate the two sets of generations best “do not correspond to change points in the model generation” “but instead to the end of the generation, when the final answer likely already appears in the generated text” (§3).
• How strongly it is made. As a correlation, with its reading hedged. The body calls it “a moderate correlation between steering success and base probability”, and reads it with “Broadly speaking, our results suggest that steering is most effective when the model is unsure about its final answer” (§3). The abstract reads the same result this way: “This suggests that activation interventions are most effective when there are alternate paths available to the model—in other words, when it has not yet committed to a particular final answer.” (Abstract.) Two uses are proposed: “steering success is a promising estimate of uncertainty during generation”, and “uncertainty estimates may be indicative of times during generation at which to intervene on a model’s activations to control its output” (§3).
• What it hands on. The question of the next section. “Given that hidden state dynamics correlate with a model’s uncertainty dynamics, can we efficiently predict the model’s outcome distribution directly from its hidden states?” (§3.)

Claim 2: the hidden state at a token predicts the outcome distribution, and better than a second model’s reading of the same text

A linear probe on the model’s hidden state at a token predicts the outcome distribution at that token, and with lower loss than a probe on another model’s activations over the same text.

The question, and what each result would mean (§4). The paper grants that the text alone carries some of the answer: “research in faithfulness suggests that the reasoning chain leading to token t is predictive of the final outcome distribution” (citing Lanham et al., 2023). So the question is what the hidden state adds. “Is the semantic information in chain-of-thought text enough to predict a model’s outcome distribution, or do the model’s hidden activations contain information about its underlying decision-making that doesn’t surface at the token level?”

The test gives the same text to a second model, Gemma-2 2B Instruct. The hypothesis is that the activations h_t of Llama and h′_t of Gemma “contain the same semantic information about the chain-of-thought text up to token t, since they are both capable language models”, while “only h_t provides information about the underlying uncertainty dynamics” of Llama. What each result would mean is said before any result is given. “Comparable KL losses indicate that model-specific hidden state dynamics aren’t especially useful in predicting its outcome distribution”. A lower loss for the original model would “indicate that a model’s hidden states are more predictive of its outcome distribution than the chain-of-thought text alone”.

How the probes are trained (§4, Figure 4). “We train a linear probe to predict o_t from h_t over a set of token indices.” Because o_t is a whole distribution, “we train our linear probe with KL divergence loss and report the KL loss on a held-out validation set of token indices”. A second probe is trained the same way on Gemma’s activations at the same tokens.

Evidence. The probes are tested on “10 randomly sampled examples from the AQuA dataset”, and the losses are “averaged across our 10 data points” (§4, Figure 5).

• Both probes beat two fixed predictors: “a random baseline which always predicts the uniform distribution (1.53 loss, not pictured) and a majority baseline which always predicts a one-hot distribution over the majority class (0.85 loss)”.
• Both are “most predictive around the middle layers of their respective model, around layers 6-10”.
• The model’s own activations do better. “The validation loss for the linear probe trained over the original activations is lower than the loss for the probe trained over activations from a different model (0.11 vs. 0.19 at layer 8).”
• Further up the network the two part. “The loss continues to be low for probes trained on the original model’s later layers, while the loss rises for probes trained on the separate model’s later layers.”

• Objections it expects.
  • That the text is enough, and nothing hidden is needed. This is what the second model is for. The paper grants the text its share: that both probes beat the baselines “suggests that the reasoning chains have meaningful semantic information that’s predictive of the model’s outcome distribution” (§4).
  • That a probe with a low loss is doing no more than guess the usual answer. The uniform and majority baselines answer this.
  • That a probe has only fitted the tokens it was trained on. The loss is reported on held-out token indices, and for the second model “on the same validation set of token indices” (§4).
• How strongly it is made. With a hedge in the results and without one in the abstract. The results section says the results “suggest that the original model’s hidden activations h_t may carry information that is used to determine the model’s future actions, beyond the information carried by the output tokens alone” (§4). The section’s discussion is flatter: “hidden activations capture underlying decision-making information that is more predictive of the final outcome than the embedding of the same text by a different model” (§4). The abstract ends on “demonstrating that models implicitly represent the space of possible paths” (Abstract).
• What it hands on. Nothing further: the paper ends with this section. Its last sentence turns the result toward the cost that §2 began with. “Probing a model’s hidden activations is a promising direction for efficiently estimating its outcome distribution during generation.” (§4.)

What the paper claims as new

The paper has no list of contributions and no sentence that claims a first. It places itself in three ways, in its own words:

• As a continuation: “We build on the work of Bigelow et al. (2025)” (§2).
• By a gap between two lines of work. One studies pretrained models “at the level of individual tokens”. Of that work it says: “While these approaches emphasize the role of in-context learning and autoregressive text generation, these aspects of text generation are typically not considered in research on steering.” (§1.)
• By a cost it sets out to avoid: reading the outcome distribution from hidden states “without needing to generate new tokens” (§2).

Limits the authors state

The paper has no section on limitations and no conclusion. What it says about its reach comes in passing:

• The sample is small, because the method is costly. “Due to the computational complexity of Forking Paths Analysis, we consider a handful of examples” (Appendix A): four questions for claim 1 (§3) and ten for claim 2 (§4), on one model.
• The questions are picked for uncertainty: on each, “the LLM is only 40–60% likely to generate that answer” (Appendix A).
• The main text’s numbers for claim 1 are for one question. “The steering correlations reported in Section 3 pertain to the example in Figure 2 in the main text.” (Appendix A.)
• The correlation is not called strong in the body: “We find a moderate correlation between controllability and uncertainty” (§3).
• Claim 2’s reading is worded as a possibility: the activations “may carry information that is used to determine the model’s future actions” (§4).

How the paper tells it

The paper tells its argument four times: in its title, which is a question, “Are language models aware of the road not taken?”; in the abstract; as a plan, in the introduction; and in the body, where §3 and §4 are each laid out as a small paper. This part takes them in that order.

The abstract

Six sentences. The role is the job the sentence does.

The introduction

Four paragraphs, each with the job it does.

The introduction also carries Figure 1, the diagram of the two interventions shown above.

The body, section by section

For each section: its job, how it opens, what it hands on, and what would be missing without it.

§2 Estimating Outcome Distribution with Forking Paths Analysis. Its job is to give the method the paper inherits and the quantity every later figure plots, o_t, as set out above. It opens “We build on the work of Bigelow et al. (2025)”, and it closes by turning the method’s cost into the paper’s task. Without it o_t has no definition, and there is no reason to want a cheaper way to o_t than re-sampling.

§3 Steering Outcome Distribution with Hidden State Interventions (claim 1; Figures 2 and 3). Its job is the first test. It opens from the phenomenon, “Forking Paths Analysis reveals interesting uncertainty dynamics, where an LLM’s uncertainty can dramatically change upon sampling a single token”, and then asks: “For instance, can we steer an LLM away from its path once it has decided on an answer?” The rest runs under four heads set into the text: Methods, Data and Models, Results, Discussion. Results goes from one example to a hypothesis to the correlation that tests it, and Discussion ends on the question that §4 takes up. Without it the first verb of the abstract’s “control and predict” has no evidence, and §4 has no stated reason to expect the hidden state to hold anything.

§4 Predicting Outcome Distribution from Hidden States (claim 2; Figures 4 and 5). Its job is the second test. It opens by naming the cost again: “This is a computationally expensive process, because it must simulate the LLM’s generation across numerous samples.” Then come the question, the hypothesis about the two models, and what each result would mean, all before any method. The same four heads follow. Its Discussion is the end of the paper. Without it the abstract’s last sentence, the one that answers the title, has nothing under it.

The appendices

There is one, Appendix A, on a page of its own. What it holds, and the job of each part:

Where each claim appears

From the shortest telling to the longest. The paper has no list of contributions and no conclusion, and no thread by its authors was found.

BibTeX

@inproceedings{zur2025,
  title = {{Are language models aware of the road not taken? Token-level uncertainty and hidden state dynamics}},
  author = {Amir Zur and Atticus Geiger and Ekdeep Singh Lubana and Eric Bigelow},
  year = {2025},
  booktitle = {ICML 2025 workshop (Actionable Interpretability)},
  eprint = {2511.04527},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.04527}
}