information theory Podcast Summaries
information theory on Yedapo: 4 summarized podcast and YouTube episodes. Each includes key takeaways, core concepts and notable quotes with timestamps.
But what is cross-entropy? | Compression is Intelligence Part 2
3Blue1Brown
Jul 16, 2026
This analysis reveals that cross-entropy, a cornerstone of AI training, is fundamentally linked to data compression. By reframing model training from 'next-token prediction' to 'compression,' we gain a deeper intuition into why language models learn the structure of reality.
Key insight: The fact that a generic file compression algorithm like gzip can successfully cluster languages and reconstruct their lineage without any pre-baked linguistic knowledge proves that compression is a deep proxy for understanding structural patterns.
Measuring the entropy of English
3Blue1Brown
Jun 12, 2026
Claude Shannon’s foundational work in information theory reveals that language compressibility is directly tied to predictability. By using human intuition to measure how easily text can be reconstructed, Shannon proved that English can be compressed to roughly one bit per character, establishing a fundamental link between data compression and the engineering of artificial intelligence.
Key insight: Claude Shannon estimated that with at least 100 characters of context, English text is theoretically compressible to approximately 1 bit per character.
What's the perfect encoding? How do you know?
3Blue1Brown
Jun 10, 2026
Efficient data transmission requires aligning binary codes with the probability of specific outcomes. By assigning shorter bit sequences to more frequent movements, you minimize the average cost per instruction, mirroring the fundamental principles of Shannon entropy. This approach proves that compression is not just a storage tool, but a core component of intelligent information processing.
Key insight: The most efficient way to encode instructions is to assign shorter binary codes to more probable events, directly demonstrating that compression is a fundamental measure of intelligence.
Reinventing Entropy | Compression is Intelligence Part 1
3Blue1Brown
Jun 7, 2026
Claude Shannon's information theory reveals a profound link between predictive modeling and data compression. Modern machine learning achieves intelligence by approximating the most efficient possible compression of language, transforming our understanding of what cross-entropy loss actually signifies in model training.
Key insight: Shannon estimated the entropy of English to be about one bit per character, meaning human language is so predictable that, given enough context, it could theoretically be compressed to a single yes-or-no question per character.