LLM Architecture Podcast Summaries

LLM Architecture on Yedapo: 2 summarized podcast and YouTube episodes. Each includes key takeaways, core concepts and notable quotes with timestamps.

OpenAI Codex on a ROLL! but Google might be cooking.. (IO Rumors)
MattVidPro
May 5, 2026
The AI landscape is shifting from simple chatbots to agentic workflows that can operate computers and manage massive context windows. New architectures like sub-quadratic sparse attention promise to handle millions of tokens with significantly lower compute costs, signaling a move toward autonomous systems capable of building complex software and managing long-term, multi-step tasks.
Key insight: A new sub-quadratic sparse attention (SSA) architecture claims to support a 12-million-token context window—12 times larger than current state-of-the-art models—while using 20 times less compute than standard transformer-based attention.
Titans: Learning to Memorize at Test Time (Paper Analysis)
Yannic Kilcher
Dec 14, 2025
Google's new Titans architecture aims to overcome transformer context limits by enabling models to 'memorize' information at test time. By using a neural network as an active memory bank, the model learns to store and retrieve past data dynamically. While technically impressive, much of the underlying logic mirrors established concepts like gradient descent and linear transformers.
Key insight: The authors frame their memory update process through the lens of 'surprise,' yet this is functionally equivalent to standard gradient descent with momentum.