Energy-Based Transformers are Scalable Learners and Thinkers (Paper Review)
Yannic Kilcher
Jul 19, 2025
Researchers are merging energy-based models with transformers to enable 'system two' thinking through unsupervised learning. By treating inference as an optimization procedure rather than a single forward pass, these models dynamically allocate compute to improve accuracy, offering a promising, scalable alternative to traditional autoregressive architectures.
Key insight: The authors propose that 'thinking' in machines is not an inherent act but a measurable metric: the performance gain achieved by performing multiple forward passes (optimization steps) at inference time compared to a single pass.
