Thread
John Langford on where "Full-bandwidth Transformer" came from
Asked about a rumor that a model from OpenAI uses a full-bandwidth transformer, the senior author says in five posts that he has no special knowledge and that, if it is true, it is a case of parallel development. He gives where the idea came from: a talk at NeurIPS that named recursion as a next step, the proof in the NextLat paper and the ParaRNN paper. He names two surprises in building it, that NextLat is not needed and that it worked easily, and relates the paper to Recirculation.
The posts are embedded from X. The figure notes under them are written by this wiki.
Yesterday, I was asked about a rumor that Astra https://t.co/EqfFw6ad8r uses a Full Bandwidth Transformer https://t.co/osGnEraffW .
— John Langford (@JohnCLangford) September 4, 2026
I have no special knowledge, but it is at least strange to have someone from OpenAI asking where the idea came from.I'd like to state publicly though that if true this is a case of parallel development.
— John Langford (@JohnCLangford) September 4, 2026
At NeurIPS my talk https://t.co/c4eDd3boGO mentioned recursion as a natural next step that we wanted to investigate. Infrastructure issues took months, but this was an agenda item.Why?
— John Langford (@JohnCLangford) September 4, 2026
The nextlat paper's proof https://t.co/CEKiqKR23Q implies convergence to a sequence of fixed points.
And having read the para-RNN paper https://t.co/CUDF6NDf9u it was clear how to efficiently train a parallel traversal of a set of fixed points.In developing the Full Bandwidth transformer we then ran into two surprises: You don't need nextlat and it worked easily.
— John Langford (@JohnCLangford) September 4, 2026
There's apparently another parallel development: https://t.co/GbqM7tGqYu . In essence, the FBT trains for recirculation over the full depth of the stack.In any case, congrats to my friends at OpenAI on creating a very impressive model.
— John Langford (@JohnCLangford) September 4, 2026