Thread
John Langford on "Almost Free State Prediction Separation"
One of the two first authors announces the paper in two posts: state-prediction separation put through an optimization process, which yields free pause tokens that let a transformer predict better at almost no cost at inference, with the training overhead cut from about 700% to an amortized 10 to 20%.
The posts are embedded from X. The figure notes under them are written by this wiki.
A quick fun paper: We ran State Prediction Separation https://t.co/njhH14JiUM through an optimization process https://t.co/sKFA6lqEdC finding that it works at higher op and yielding "free pause tokens" which allow a transformer to make higher quality inference for (near) free.
— John Langford (@JohnCLangford) September 8, 2026The training time overhead is reduced from ~700% in SPS to an amortized 10-20% depending on precise choices. Overall, an improvement on basic transformers which may be widely useful.
— John Langford (@JohnCLangford) September 8, 2026
Joint with @nthngdy, Giovanni, @yoavartzi, @Real_HDong , Ying, @rosaguga, @ZhengZhan13.