The Information reported that OpenAI's next-generation model Astra uses a "recurrent depth" architecture — and that the technique "works in a way that obscures some or all of the AI's reasoning." Overnight, "looped transformers" were hyped as world-changing technology. On September 2, independent researcher Sebastian Raschka (author of Build a Large Language Model From Scratch) offered a technical corrective: "The looped transformer idea is just reusing layers in the transformer block."
Astra may well be a great model. But does "looping" alone deserve a headline? Laid out plainly, the architecture is worth a cool-headed teardown — and once you finish it, the issue that truly deserves attention turns out to be a different one.
Principle 1: the standard transformer is a fixed assembly line
First, the intuition. A large language model's core job is predicting the next token. To predict well, it turns your text into numbers (vectors), runs them through a stack of mathematical processing layers (transformer layers), refining step by step, until the top layer emits a word.
The key point: this assembly line is fixed at the factory. An 18-layer model runs exactly 18 processing steps for every token and every problem — each step with its own dedicated machine (that layer's weights). "What's the weather today" and a mathematical proof consume identical depth. That is the "fixed number of times" in the left half of the diagram.
Parameters and depth are locked together: going deeper means building more machines, with storage and memory doubling alongside. This is one root of the cost of large models.
Principle 2: recurrent depth = the same assembly line, run twice
The recurrent-depth fix is almost frugal to a fault: keep 9 machines instead of 18, and run the line twice.
The pink "Recurrent block" in the right half of the diagram is those 9 machines; the arrow curling back means "one more pass." The output of the first loop feeds the same machines again. To the model, this approximates an 18-layer network — while the warehouse stocks only 9 layers of goods. In one sentence: a twice-as-deep brain in a same-size skull — at the price of twice the thinking effort.
This is not paper speculation. The open-weight Nanbeige 4.2-3B (pretrained from scratch on 28T tokens) reuses its 22-layer stack twice, effectively 44 layers. Its technical report's conclusion is sober: two passes are the best trade-off, retaining about 75% of token efficiency; more passes bring marginal gains at much slower, pricier training.
Parameters & storage
Standard transformer: grow linearly with depth Recurrent depth (two loops): one copy of layer weights, reused in place
Compute per token
Standard transformer: fixed (L layers) Recurrent depth (two loops): nearly doubled (~2L layers)
Effective depth
Standard transformer: fixed L layers Recurrent depth (two loops): approximately 2L layers
Difficulty adaptivity
Standard transformer: no, same depth for all inputs Recurrent depth (two loops): yes, with early exiting or routing
Not a new idea: a road paved in 2019
Raschka is careful to note that, as far as he knows, Nanbeige is merely the first notable open-weight model to adopt the approach — the idea itself predates it. The NeurIPS paper "Mixture-of-Recursions" goes further, adding a learned router that decides per token whether it gets one, two, or more passes: easy tokens exit early while harder tokens get extra computation — like highway toll lanes, where small cars leave at the next ramp and heavy trucks loop a few more times.
The team that paved this road to engineering practicality is Google DeepMind's, in the ICLR 2025 paper "Relaxed Recursive Transformers" (arXiv:2410.20672). Its 49 pages answer a question Nanbeige did not: without training from scratch, can an existing large model be converted directly into a looped one?
Reading the paper: converting a large model in three steps
Step 1: copy the weights. Store the first 9 layers of the original model (Gemma 2B, 18 layers) and loop twice during the forward pass — the paper's CYCLE strategy, with even the normalization parameters fully shared. Running it cold loses accuracy, so a modest "uptraining" fine-tune follows.
Step 2: give each loop a cheat sheet. Strict sharing hurts — pass 1 and pass 2 should differ slightly (rough refine, then fine), yet one set of weights must serve both. The fix is a low-rank adapter (LoRA) per loop position: a tiny patch letting the two passes specialize. The paper initializes these patches via truncated SVD on the residual between the original full weights and the shared weights; the LoRA rank dials the relaxation smoothly from "fully shared" to "fully independent."
Step 3: let easy tokens exit early. Paired with early exiting, confident tokens run fewer loops. The paper also proposes "Continuous Depth-wise Batching," dynamically packing different loop counts of different requests into one batch — with simulated throughput gains of 2–3x, and nearly 4x in theory for recursive Gemma.
The "obscured chain-of-thought" claim is only half right
Back to the most sensational claim: that the looped architecture "obscures the reasoning process."
Raschka's verdict: layer reuse by itself does not suppress visible chain of thought. Loop layers do exactly what ordinary layers do — add computation in hidden states before the next token is emitted. Nothing previously readable is erased.
The plausible reading is indirect: with more recurrent passes, a model may need to generate fewer intermediate reasoning tokens — drafts once written out for you to read become unreadable computation in internal activations. But this is not a looped-transformer monopoly: scaling up the model (GPT 5.6 Luna to Sol) produces the same effect. Computation moving from readable text to unreadable activations is a byproduct of the economics of scale, not a conspiracy switch.
Notably, the paper's future-work section is titled "Latent Reasoning via Recurrent Depth." Academia saw where this road leads two years ago — and now it surfaces in the Astra rumors.
Dehghani et al. apply the same layers recurrently with a dynamic halting mechanism — an early exemplar of parameter sharing and adaptive depth (as cited in arXiv:2410.20672)
Google DeepMind and colleagues propose a three-step conversion of existing LLMs into looped transformers: weight initialization, LoRA relaxation, and early exiting
Final version (v3) updated in February 2025; Recursive Gemma 1B, uptrained on 60B tokens, matches the full-size Gemma 2B
Points out that the looped transformer is just layer reuse; Nanbeige 4.2 is, to his knowledge, the first notable open-weight model to adopt it
The looped transformer idea is just reusing layers in the transformer block.
Opinion: three things worth remembering
Debunking is not dismissal. Looping is a smart cost trade-off: half the parameters, half the memory, and speed that can vary with difficulty. But it changes the cost structure, not the chemistry of intelligence — don't expect it alone to explain any benchmark miracle from Astra.
Variable depth is the real story. One model, with easy requests in and out fast and hard ones given extra loops — a new knob for test-time compute. Yesterday you dialed compute by switching models; tomorrow, perhaps, by setting the loop count.
"Invisible thinking" is the real issue. Whether or not Astra uses loops, model reasoning is migrating from readable text to unreadable activations. Today we audit AI by reading its chain of thought; tomorrow the chain may be a table of contents with the body locked inside the weights. The looped architecture is not guilty — but it nudges that door open another inch. Regulation and interpretability research need to catch up.