[Paper Analysis] On the Theoretical Limitations of Embedding-Based Retrieval (Warning: Rant)
Yannic Kilcher
Oct 11, 2025
New research formally proves that embedding models cannot represent arbitrary combinations of data, identifying a mathematical ceiling for dense retrieval. While technically sound, this limitation is practically irrelevant because real-world data possesses structure that embeddings are specifically designed to capture. The quest for "perfect" arbitrary retrieval is a misunderstanding of what machine learning achieves.
Key insight: Even with a 1,000-dimensional embedding space, models can accurately handle combinations of up to 4 million elements, a far higher capacity than many critics assume.
