# Xi Wang on "Full-bandwidth Transformer" > The lead author walks through the paper in eight posts: feeding the previous top-layer hidden state back into the input with the token embedding, what that opens across depth, why it costs almost nothing at inference, the multi-pass training it needs and the schedule that keeps the extra cost near 30% of the flops, the gain from extra prefill passes, and free-form generation comparable with models trained with twice the flops. Posted with the paper's first version: the figure and table numbers in the images are that version's. - Author: Xi Wang ([@xidulu](https://x.com/xidulu)) - Posted: 2026-08-11, 8 posts - Original: https://x.com/xidulu/status/2087185532707111092 - About: [Full-bandwidth Transformer](https://introspection.infinite.fun/papers/wang2026-full-bandwidth-transformer.md) The post text below is quoted verbatim. Figure descriptions are written by this wiki. ## 1/8 > 1/ Sharing a new, interesting project I did during my internship at Microsoft AI Frontiers w/ @JohnCLangford > > TL;DR: At decoding time, we feed **previous hidden state** into the input together with token embedding, and it boosts performance for free. > > https://arxiv.org/abs/2608.08888 [Post 1 on X](https://x.com/xidulu/status/2087185532707111092) ## 2/8 > 2/ Our approach (latent feedback decoding) unlocks the ** full bandwidth ** of the transformer: > - Dense attention gives full visibility over past positions > - Our approach enhances the bandwidth vertically: It exposes past information across *depth* to the current layer/position Figure: The paper's Figure 1 with its caption. Two diagrams of three tokens passing through a stack drawn with layers 0 to 3. Left, standard decoding: each token's output goes to the next token's input, and from the state being computed, at layer 1 of token 2, only the layer-0 states of tokens 0 and 1 are marked reachable; their states at layers 1 to 3 are marked unreachable. Right, latent feedback decoding: the top-layer state of each token is joined to the next token's embedding at the input by a symbol ⊗ labeled 'Dimension-preserving fusing', and every past state at every layer is marked reachable. [Post 2 on X](https://x.com/xidulu/status/2087185533894037899) ## 3/8 > 3/ Latent feedback decoding is free at inference time since the top layer hidden state is there anyway to perform decoding, the only extra overhead is the fusing operation, which only has two matmul Figure: Two listings of code side by side. Standard decoding: after prefill, the loop sets x = embed(tok), runs model_step(x, kv_cache) and samples the next token. Latent feedback decoding: the same loop with one line changed, shown in red: x = glu_cross(h_prev, embed(tok)). Above it is the function glu_cross(h, e), which returns (h @ W_u) * sigmoid(e @ W_g). [Post 3 on X](https://x.com/xidulu/status/2087185535706009755) ## 4/8 > 4/ Training takes extra flops: We need multi forward passes to teach the model to understand "embedding hidden state fusion" type of inputs. > Luckily, you don't need to "loop" every iteration; introducing the "loops" progressively later on in the training still works! Figure: A diagram of multi-pass training: three forward passes of an LLM over tokens 0 to 3, stacked from bottom to top. The first pass takes token embeddings and produces hidden states. In each later pass, the hidden state the previous pass produced at one position is fused, by a symbol ⊗, with the token embedding at the next position. Notes on the right: train with multiple forward pass, bottom to top; apply the NTP loss on each pass's hidden state; a practical efficiency trick, a scheduled number of passes: the first 75% of training on one pass only, then 22% on two passes, to get the model familiar with fused inputs, and 3% on three passes, to get it familiar with recurrently generated inputs; about 1.3 times the flops of standard training. [Post 4 on X](https://x.com/xidulu/status/2087185537224327582) ## 5/8 > 5/ The extra %30 flops spent at training time not only unlocks a new way of decoding, it also enables test-time scaling, where you use multi-pass prefilling for boosting validation loss and LM Eval performance Figure: Two panels captioned 'Figure 4: Feedback passes during prefilling improve non-generative performance.' Left, validation loss against the number of feedback passes for prefill, 0 to 3, for full-bandwidth models trained on 10B, 100B, 200B and 400B tokens, each with a dashed line for the standard model trained on the same tokens. Every solid line starts just above its dashed line at 0 passes, drops below it at 1 pass and is flat after. Right, 5-shot LM Eval average accuracy against 0 to 4 passes, with error bars, for the 100B, 200B and 400B models: each rises from 0 to 1 pass and is above its own dashed baseline from 1 pass on. A dashed black line for a standard model trained on 1T tokens lies above all of them. [Post 5 on X](https://x.com/xidulu/status/2087185539912860143) ## 6/8 > 6/ Of course, models trained this way can perform > - (Soft, free) Latent feedback decoding > - (fused) Double prefill + soft decoding > The extra 30% FLOPs spent at training time translate to free-form generation performance comparable with models trained with 2x the FLOPs Figure: Four panels captioned 'Figure 5', for GSM8K (5-shot, Pass@1), Math500 (0-shot, Pass@1), HumanEval (0-shot, Pass@3) and MBPP (3-shot, Pass@3). Each plots a score under three decoding methods, Std., Soft and Fused, for full-bandwidth models trained on 100B, 200B and 400B tokens, with dashed lines for standard models at the same tokens and at 1T. On GSM8K, HumanEval and MBPP the solid lines mostly rise from Std. to Soft or Fused and lie at or above their own dashed baselines. On Math500 the 100B and 400B lines are below their baselines under all three methods, and the 400B line falls from Std. to Fused. Figure: A table captioned 'Table 1: Latent-feedback gains carry over through instruction tuning.' Scores in percent on four tasks, in the order GSM8K (Pass@1), MATH-500 (Pass@1), HumanEval (Pass@3), MBPP (Pass@3). Full-bandwidth at 200B under Standard, Soft and Fused decoding: 64.52, 67.93, 67.55; 43.80, 45.60, 45.60; 42.54, 45.06, 45.92; 38.39, 39.80, 41.22. Full-bandwidth at 400B: 67.90, 71.00, 71.80; 46.00, 45.40, 48.40; 46.50, 47.20, 47.60; 40.50, 40.60, 41.70. Standard transformer at 200B, 400B and 1T: 62.93, 68.39, 70.13; 42.40, 46.40, 47.40; 37.16, 44.85, 50.01; 38.61, 40.28, 41.93. [Post 6 on X](https://x.com/xidulu/status/2087185541666062528) ## 7/8 > 7/ In summary, > - We utilize the sequential nature of the decoding process to squeeze in richer feedback information (continuous hidden state v.s. discrete token) > - We proposed a way to use more FLOPs to boost performance w/o increasing tokens / model size: Better data efficiency! [Post 7 on X](https://x.com/xidulu/status/2087185543956165014) ## 8/8 > 8/ Lastly, deeply grateful to the members of the AI Frontiers team for their invaluable support and feedback: @ZhengZhan13 @jackcai1206 @yingfan_bot @Tea_Pearce [Post 8 on X](https://x.com/xidulu/status/2087186287337828374) --- Source: https://introspection.infinite.fun/threads/xidulu-full-bandwidth-transformer · Part of the [LLM Introspection Wiki](https://introspection.infinite.fun/index.md) · Index for agents: [llms.txt](https://introspection.infinite.fun/llms.txt)