LLM Training Podcast Summaries

Let's reproduce GPT-2 (124M)

Andrej Karpathy

Jun 9, 2024

Andrej Karpathy demonstrates how to reproduce the 124M parameter GPT-2 model from scratch using PyTorch. By bypassing complex legacy code in favor of a clean, modular implementation, he reveals how modern hardware optimizations like torch.compile and BFloat16 can slash training times and costs to roughly $10 per run.

Key insight: You can achieve state-of-the-art training performance on a 124M parameter model in under an hour for about $10 by leveraging modern GPU acceleration and mixed-precision training.