# John Langford on where "Full-bandwidth Transformer" came from

> Asked about a rumor that a model from OpenAI uses a full-bandwidth transformer, the senior author says in five posts that he has no special knowledge and that, if it is true, it is a case of parallel development. He gives where the idea came from: a talk at NeurIPS that named recursion as a next step, the proof in the NextLat paper and the ParaRNN paper. He names two surprises in building it, that NextLat is not needed and that it worked easily, and relates the paper to Recirculation.

- Author: John Langford ([@JohnCLangford](https://x.com/JohnCLangford))
- Posted: 2026-09-04, 5 posts
- Original: https://x.com/JohnCLangford/status/2095849181919261168
- About: [Full-bandwidth Transformer](https://introspection.infinite.fun/papers/wang2026-full-bandwidth-transformer.md)

The post text below is quoted verbatim. Figure descriptions are written by this wiki.

## 1/5

> Yesterday, I was asked about a  rumor that Astra https://openai.com/index/gpt-6-astra/ uses a Full Bandwidth Transformer https://arxiv.org/abs/2608.08888 .
>
> I have no special knowledge, but it is at least strange to have someone from OpenAI asking where the idea came from.

[Post 1 on X](https://x.com/JohnCLangford/status/2095849181919261168)

## 2/5

> I'd like to state publicly though that if true this is a case of parallel development.
>
> At NeurIPS my talk https://neurips.cc/virtual/2025/loc/san-diego/workshop/109532 mentioned recursion as a natural next step that we wanted to investigate. Infrastructure issues took months, but this was an agenda item.

[Post 2 on X](https://x.com/JohnCLangford/status/2095849183999606874)

## 3/5

> Why?
>
> The nextlat paper's proof https://arxiv.org/pdf/2511.05963 implies convergence to a sequence of fixed points.
>
> And having read the para-RNN paper https://machinelearning.apple.com/research/large-scale-rnnsv   it was clear how to efficiently train a parallel traversal of a set of fixed points.

[Post 3 on X](https://x.com/JohnCLangford/status/2095849186377756970)

## 4/5

> In developing the Full Bandwidth transformer we then ran into two surprises: You don't need nextlat and it worked easily.
>
> There's apparently another parallel development: https://arxiv.org/abs/2608.17981v1 .  In essence, the FBT trains for recirculation over the full depth of the stack.

[Post 4 on X](https://x.com/JohnCLangford/status/2095849188550471885)

## 5/5

> In any case, congrats to my friends at OpenAI on creating a very impressive model.

[Post 5 on X](https://x.com/JohnCLangford/status/2095849190681141589)

---

Source: https://introspection.infinite.fun/threads/johnclangford-full-bandwidth-transformer-origins · Part of the [LLM Introspection Wiki](https://introspection.infinite.fun/index.md) · Index for agents: [llms.txt](https://introspection.infinite.fun/llms.txt)