Recurrent Depth

---

Research checked: 20 September 2026.

A conventional transformer processes representations through a stack of layers. A looped transformer reuses a block of parameters across multiple computational steps. Increasing the number of repetitions increases effective depth without adding a separate set of weights for every step. This creates an interesting separation between how much a model stores and how much computation it performs before producing an answer.

Source: Figure 2 from Geiping et al., “Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach” (2025).

A simplified recurrence can be written as:

$$h_{t+1}=F_{\theta}(h_t,x).$$

The input $x$ provides conditioning, $h_t$ is the current hidden representation, and the same parameters $\theta$ are used repeatedly. After a chosen number of iterations, an output stage turns the final representation into predictions. This is recurrence over computational depth; it should be distinguished from moving recurrently through successive positions in a sequence.

Universal Transformers established an influential version of this idea, combining attention with recurrent refinement and the possibility of adaptive computation. Repetition gives the model a way to apply similar transformations several times, which is appealing for problems with iterative structure. The benefits still depend on the training procedure and the tasks evaluated. Dehghani et al., Universal Transformers.

More recently, Geiping and colleagues studied recurrent depth as a form of inference-time reasoning in a 3.5-billion-parameter model. Their architecture separates input processing, a repeated core, and output processing. The experiments show that additional iterations can improve performance on certain reasoning benchmarks without requiring a longer visible chain of thought. The research supports recurrent computation as a useful axis to explore, rather than a guarantee that every additional loop improves every answer. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach.

The distinction from recursive language models is important. An RLM orchestrates model calls and external context through an inference workflow. A depth-recurrent transformer repeatedly updates hidden representations inside the neural architecture. One concerns how a system delegates and reads; the other concerns how a model computes. They could be combined, but they are not interchangeable names for the same mechanism.

My interpretation of the efficiency argument is that parameter reuse saves storage relative to creating equally many independent layers, while repeated execution still costs time and arithmetic. The comparison should therefore include compute and latency. A compact model that loops many times may be attractive under a memory constraint without being the fastest solution.

Adaptive stopping adds another challenge. A stable hidden state or a small change in output probabilities does not, by itself, prove that the answer is correct. Stopping behavior needs evaluation against task outcomes. Latent computation also makes intermediate work less directly readable than a written trace, though written traces have their own faithfulness limitations.

For evaluation, I would compare equal-parameter and equal-compute baselines, then test whether increasing loops beyond the training range actually helps. Report which tasks plateau or degrade. Recurrent depth is compelling because it lets a model spend more computation using the same learned machinery. The decisive question is whether training makes that additional work useful and controllable.