# Xi Wang on the revision of "Full-bandwidth Transformer"

> The lead author lists in six posts what the paper's second version adds: how to debug a training run from its loss curves, rebuilding a rollout's KV cache and hidden states with three parallel passes, other forms of the fusion gate, and where the idea came from. One post reports a first online RL run on the released code, which the paper does not contain.

- Author: Xi Wang ([@xidulu](https://x.com/xidulu))
- Posted: 2026-10-05, 6 posts
- Original: https://x.com/xidulu/status/2107140306227564928
- About: [Full-bandwidth Transformer](https://introspection.infinite.fun/papers/wang2026-full-bandwidth-transformer.md)

The post text below is quoted verbatim. Figure descriptions are written by this wiki.

## 1/6

> 1/ A new revision of full-bandwidth transformer is on arXiv (https://arxiv.org/abs/2608.08888). We included a few extra discussions on:
> - How to debug training runs
> - Post-training / KV cache replay
> - Alternative fusion gate
> Non-official code release at: https://github.com/xidulu/Full-bandwidth-transformer

[Post 1 on X](https://x.com/xidulu/status/2107140306227564928)

## 2/6

> 2/ Training runs debug:
> In our experience, the training dynamics provide lots of useful signal. The *bottom line* is that your recipe should enable the hidden-state-augmented second pass to improve the PPL.
> Then you could start worrying about extrapolation

Figure: The paper's Figure 18 with its caption. Average validation loss against training step, from about 0.92 to 1.22 million steps, for four schedules: 25% three passes, 22% two passes + 3% three passes, 25% two passes and 12% two passes. Solid lines are the first-pass loss and dashed lines the second-pass loss. All fall from about 1.63 to about 1.45. An inset zooms in on the first steps after the feedback passes begin: the dashed lines start above the solid ones and cross below them. For the rest of training the dashed lines stay below the solid ones.

[Post 2 on X](https://x.com/xidulu/status/2107140307846590822)

## 3/6

> 3/ Offline rollout replay:
> Given a recurrently generated rollout (left), we often need to re-generate its KV cache or hidden states for RL or re-prefill (/resume).
> Empirically, we noticed that 3 forward passes, parallel over all positions, approximate the true states very well.

Figure: The right half of the paper's Figure 1: latent feedback decoding over three tokens, with each token's top-layer state fused with the next token's embedding at the input.

Figure: The paper's Figure 8 with its caption. Left: relative ℓ2 error of the reconstructed key and value caches against the total number of forward passes, 1 to 32, on a log scale. Both fall from above 0.1 at one pass to about 0.01 at three passes and are level from four passes on, just above two dotted control lines. Right: argmax agreement on an axis from 0.90 to 1.00, about 0.99 at one pass and close to 1.00 from two passes on.

[Post 3 on X](https://x.com/xidulu/status/2107140310279348347)

## 4/6

> 4/ Alternate forms of fusion gate:
> In some small-scale ablations, we found that other forms of fusion also seem to work. So the glu cross gate used in the original version may not be the workable solution.

Figure: The paper's Figure 20 with its caption. Validation loss in bits per byte against training step, 40k to 60k, for three fusion gates: linear addition, concatenation + projection, and gated product. Solid lines are the first-pass loss and dashed lines the second-pass loss. The three gates' curves lie on top of one another. After a spike at the start, the second-pass loss is below the first-pass loss for all three.

[Post 4 on X](https://x.com/xidulu/status/2107140311940219297)

## 5/6

> 5/ Post-training
> https://github.com/xidulu/Full-bandwidth-transformer/tree/master/online_rl_experiments
> Some simple online RL, w/ vLLM as rollout engine, verifies that the full-bandwidth transformer is post-trainable and improvement remains!
> Actual async RL will be the next milestone; stay tuned.

Figure: A chart titled 'Reward improves during initial math RL', subtitled 'Hendrycks-MATH training split | 128 questions × 8 responses per update'. Mean training reward against RL update, 1 to 300, for three runs: a standard transformer; a full bandwidth transformer with three pass likelihood estimation; and vLLM hidden state replay likelihood estimation. All three rise. The two full-bandwidth lines lie nearly on top of each other and above the standard transformer's throughout. Under the chart, the first against the last 25 updates: standard transformer, 38.5% → 49.8% (+11.3 pp); full bandwidth transformer, three pass likelihood estimation, 41.2% → 52.2% (+10.9 pp); vLLM hidden state replay likelihood estimation, 41.0% → 52.1% (+11.1 pp).

[Post 5 on X](https://x.com/xidulu/status/2107140313743765581)

## 6/6

> 6/ Lastly, we talked about where we got the idea:
> Special thanks to @jayden_teoh_ , the NextLatent project is very thought-provoking and inspiring.

Figure: A screenshot of the paper's closing discussion, headed 'Discussion: Origins of the idea.' It says that the initial motivation came from observations in NextLat (Teoh et al., 2025), where predicting latent states yields a sequence of fixed points, suggesting the feasibility of parallel training a recursive model; that this led the authors to ask whether a transformer could be trained recursively in parallel by feeding back soft latent belief states across multiple passes; and it thanks Jayden Teoh for discussion on an earlier version of the idea.

[Post 6 on X](https://x.com/xidulu/status/2107140315652235363)

---

Source: https://introspection.infinite.fun/threads/xidulu-full-bandwidth-transformer-revision · Part of the [LLM Introspection Wiki](https://introspection.infinite.fun/index.md) · Index for agents: [llms.txt](https://introspection.infinite.fun/llms.txt)