Thread

Xi Wang on the revision of "Full-bandwidth Transformer"

Xi Wang (@xidulu) · October 5, 2026 ·6 posts · original on X

About: Full-bandwidth Transformer

The lead author lists in six posts what the paper's second version adds: how to debug a training run from its loss curves, rebuilding a rollout's KV cache and hidden states with three parallel passes, other forms of the fusion gate, and where the idea came from. One post reports a first online RL run on the released code, which the paper does not contain.

The posts are embedded from X. The figure notes under them are written by this wiki.

  2. Figure. The paper's Figure 18 with its caption. Average validation loss against training step, from about 0.92 to 1.22 million steps, for four schedules: 25% three passes, 22% two passes + 3% three passes, 25% two passes and 12% two passes. Solid lines are the first-pass loss and dashed lines the second-pass loss. All fall from about 1.63 to about 1.45. An inset zooms in on the first steps after the feedback passes begin: the dashed lines start above the solid ones and cross below them. For the rest of training the dashed lines stay below the solid ones.

  3. Figure. The right half of the paper's Figure 1: latent feedback decoding over three tokens, with each token's top-layer state fused with the next token's embedding at the input.

    Figure. The paper's Figure 8 with its caption. Left: relative ℓ2 error of the reconstructed key and value caches against the total number of forward passes, 1 to 32, on a log scale. Both fall from above 0.1 at one pass to about 0.01 at three passes and are level from four passes on, just above two dotted control lines. Right: argmax agreement on an axis from 0.90 to 1.00, about 0.99 at one pass and close to 1.00 from two passes on.

  4. Figure. The paper's Figure 20 with its caption. Validation loss in bits per byte against training step, 40k to 60k, for three fusion gates: linear addition, concatenation + projection, and gated product. Solid lines are the first-pass loss and dashed lines the second-pass loss. The three gates' curves lie on top of one another. After a spike at the start, the second-pass loss is below the first-pass loss for all three.

  5. Figure. A chart titled 'Reward improves during initial math RL', subtitled 'Hendrycks-MATH training split | 128 questions × 8 responses per update'. Mean training reward against RL update, 1 to 300, for three runs: a standard transformer; a full bandwidth transformer with three pass likelihood estimation; and vLLM hidden state replay likelihood estimation. All three rise. The two full-bandwidth lines lie nearly on top of each other and above the standard transformer's throughout. Under the chart, the first against the last 25 updates: standard transformer, 38.5% → 49.8% (+11.3 pp); full bandwidth transformer, three pass likelihood estimation, 41.2% → 52.2% (+10.9 pp); vLLM hidden state replay likelihood estimation, 41.0% → 52.1% (+11.1 pp).

  6. Figure. A screenshot of the paper's closing discussion, headed 'Discussion: Origins of the idea.' It says that the initial motivation came from observations in NextLat (Teoh et al., 2025), where predicting latent states yields a sequence of fixed points, suggesting the feasibility of parallel training a recursive model; that this led the authors to ask whether a transformer could be trained recursively in parallel by feeding back soft latent belief states across multiple passes; and it thanks Jayden Teoh for discussion on an earlier version of the idea.