Transformers Podcast Summaries
Transformers on Yedapo: 4 summarized podcast and YouTube episodes. Each includes key takeaways, core concepts and notable quotes with timestamps.
Titans: Learning to Memorize at Test Time (Paper Analysis)
Yannic Kilcher
Dec 14, 2025
Google's new Titans architecture aims to overcome transformer context limits by enabling models to 'memorize' information at test time. By using a neural network as an active memory bank, the model learns to store and retrieve past data dynamically. While technically impressive, much of the underlying logic mirrors established concepts like gradient descent and linear transformers.
Key insight: The authors frame their memory update process through the lens of 'surprise,' yet this is functionally equivalent to standard gradient descent with momentum.
[Paper Analysis] The Free Transformer (and some Variational Autoencoder stuff)
Yannic Kilcher
Nov 1, 2025
The Free Transformer introduces latent variables into decoder-only models to enable explicit decision-making before token generation. By allowing the model to choose a hidden intent—such as a positive or negative sentiment—it achieves greater long-term consistency and coherence in sequences compared to standard auto-regressive sampling, which relies purely on probability distributions for every token.
Key insight: The Free Transformer uses a 'cheating' mechanism during training where an encoder looks at the entire sequence to supply latent variables, forcing the decoder to learn to condition its output on those variables rather than relying solely on random token sampling.
Energy-Based Transformers are Scalable Learners and Thinkers (Paper Review)
Yannic Kilcher
Jul 19, 2025
Researchers are merging energy-based models with transformers to enable 'system two' thinking through unsupervised learning. By treating inference as an optimization procedure rather than a single forward pass, these models dynamically allocate compute to improve accuracy, offering a promising, scalable alternative to traditional autoregressive architectures.
Key insight: The authors propose that 'thinking' in machines is not an inherent act but a measurable metric: the performance gain achieved by performing multiple forward passes (optimization steps) at inference time compared to a single pass.
Let's build GPT: from scratch, in code, spelled out.
Andrej Karpathy
Jan 17, 2023
Andrej Karpathy demonstrates how to build a character-level language model from scratch using the Transformer architecture. By stripping away production-grade complexity, he reveals the core mechanics of self-attention, residual connections, and layer normalization that power modern systems like ChatGPT.
Key insight: Self-attention is simply a communication mechanism where tokens in a sequence act as nodes in a directed graph, using dot products between 'queries' and 'keys' to determine how much information to aggregate from past tokens.