32 of 36 layer passes did nothing
4 September 2026. From CASCaDER.
I trained a model that could run its middle layers up to 16 times before answering. It did its best work at 4. The other loops were wasted.
The setup
The model is a recurrent-depth transformer. Instead of stacking more layers, it runs the same core block again and again, so it can “think longer” on the same weights. That also means one trained model can be tested at many loop counts, because a loop count is just a number you pass in.
Before running anything, I wrote the rule for success into the code. For depth to count as useful, the best loop count had to be 8 or more, and it had to beat the others by more than 0.01.
The result
| Loops | Layer passes | Hard task | Easy task |
|---|---|---|---|
| 1 | 6 | 2.0010 | 0.0394 |
| 2 | 8 | 1.9894 | 0.0219 |
| 4 | 12 | 1.9875 | 0.0167 |
| 8 | 20 | 1.9889 | 0.0169 |
| 16 | 36 | 1.9891 | 0.0170 |
The best was 4 loops, on both tasks. Past that, the numbers barely move. At 16 loops the model makes 36 layer passes, and 32 of them add nothing.
So the verdict was negative. A bigger follow-up run was planned only if this passed. It would have taken about 40 GPU-hours, and a 45-second test showed it wasn’t needed.
The model agreed
The model has a learnable setting for how fast it forgets its running memory between loops. It started at 0.368. The looped model moved it down to 0.219, which means it chose to forget faster. The single-pass model left it at 0.376. Nothing forced it either way. The looped model learned that its memory across loops wasn’t worth keeping.
What this does and doesn’t say
It says that at this size and on these tasks, extra loops at test time don’t help past 4. It doesn’t say depth is useless. In a separate pilot, a model trained with 16 loops scored 0.761 against 0.259 for one trained with a single loop. But it also used about six times as much compute per token, so that comparison isn’t clean yet. If depth helps, the help seems to come from training with it, not from adding loops afterwards.