Scaling Laws Podcast Summaries
Scaling Laws on Yedapo: 3 summarized podcast and YouTube episodes. Each includes key takeaways, core concepts and notable quotes with timestamps.
Microsoft Just Dropped LLM's Frontier Data Engineering Secrets
bycloud
Jul 13, 2026
Microsoft has broken its tradition of secrecy by releasing a detailed 109-page technical report on its 'Mai' model. By treating training as a 'hill-climbing machine' rather than a single event, the team exposed how data mixtures scale unpredictably and why synthetic data may be a crutch rather than a necessity for emergent reasoning.
Key insight: Microsoft discovered that while stem-heavy data mixes look superior at small scales, they decay in utility at larger scales compared to code-heavy mixes, proving that current small-scale data ablations often fail to predict performance at the 23B+ parameter level.
The data black hole at the center of AI
Dwarkesh Patel
Jun 19, 2026
Current AI progress relies on brute-force data ingestion rather than human-like sample efficiency. While humans learn complex tasks with minimal exposure, frontier models require trillions of tokens and bespoke expert data to function. This massive computational overhead suggests AI operates on a fundamentally different learning curve than biological intelligence, prioritizing raw scale over cognitive optimization.
Key insight: Humans learn to drive with about 20 hours of practice, whereas self-driving models require three to four orders of magnitude more data, highlighting a massive gap in learning efficiency.
Byte Latent Transformer: Patches Scale Better Than Tokens (Paper Explained)
Yannic Kilcher
Dec 24, 2024
The Bite Latent Transformer (BLT) replaces static, vocabulary-based tokenization with dynamic, entropy-based 'patches.' By grouping bytes into variable-length segments, the model achieves superior scaling efficiency and handles out-of-vocabulary data more effectively than traditional LLMs like Llama, while maintaining competitive performance on standard benchmarks.
Key insight: BLT models achieve similar training scaling trends to Llama 3 using average patch sizes of 6 to 8 bytes, compared to the 4.4-byte average token size of traditional BPE-based models.