[GRPO Explained] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Yannic Kilcher
Jan 26, 2025
The DeepSeekMath paper demonstrates that high-quality, domain-specific data collection and the novel Group Relative Policy Optimization (GRPO) algorithm allow smaller 7B parameter models to rival massive proprietary systems. By eliminating the need for a separate value model, GRPO optimizes compute efficiency while effectively shaping the model's output distribution to favor correct reasoning paths.
Key insight: Reinforcement learning in this context primarily improves performance by making the model's output distribution more robust—boosting the probability of correct answers already present in the base model—rather than fundamentally increasing the model's underlying reasoning capabilities.
