The Economics of Scale
Kaplan, Chinchilla, and the data wall. What the scaling laws actually claim, and where they stop.
- Published
- 2 August 2026
- Reading time
- 5 min read
- Figures
- 3 figures
- Equations
- 6 equations
Kaplan Scaling Laws (2020)
Kaplan et al. (OpenAI, 2020) showed that the test loss of language models follows smooth power laws in model size N, dataset size D, and compute budget C — independently and over many orders of magnitude.
The key insight: these power laws are stable and predictable, allowing reliable extrapolation. You can train many small models, fit the power law, and forecast how a model 100× larger will perform before spending the compute to train it. This predictability justified massive long-horizon investment.
Implication at the time: maximize model size given compute (data was considered secondary). This logic led to GPT-3 (175B parameters trained on ~300B tokens) and other large, relatively undertrained models.
Chinchilla Scaling Laws (2022) — The Correction
Hoffmann et al. (DeepMind, 2022) ran a rigorous compute-optimal study: train 400+ models from 70M to 16B parameters on 5B to 500B tokens under fixed compute budgets. Question: given a budget of C FLOPs, what is the optimal model size N and dataset size D?
Result: models and data should scale equally. For every doubling of model size, double the training tokens. The Chinchilla-optimal ratio is approximately 20 tokens per parameter (Epoch AI 2024 replication: ~25.6 tokens/param).
Consequence: Chinchilla (70B, 1.4T tokens) outperformed Gopher (280B, 300B tokens) and GPT-3 (175B, 300B tokens) at equal compute. Most large models of 2020–2022 were severely undertrained.
Post-Chinchilla industry shift: LLaMA 3 70B was trained on 15T tokens (~214 tokens/param) — deliberately overtrained relative to Chinchilla to maximize inference efficiency. A smaller model that performs well enough at lower per-token inference cost is often more economical than the compute-optimal trained model.
The Model Size Timeline
GPT-1: 117M (2018) → BERT-Large: 340M → GPT-2: 1.5B (2019) → T5-11B → GPT-3: 175B (2020) → Megatron-Turing: 530B (2021) → PaLM: 540B (2022) → GPT-4: ~1.8T total/~280B active (rumored MoE, 2023, unconfirmed by OpenAI) → DeepSeek-V3: 671B/37B active (2024/25).
Key inflection point: after GPT-3/PaLM, the frontier pivoted from dense scaling toward MoE — total parameters and active compute diverge dramatically. A 671B parameter model with 37B active compute per token is operationally much cheaper to run than a 671B dense model.
Open models (LLaMA 3 70B/405B, Qwen 2.5 72B, Mistral Large, DeepSeek-V3) have dramatically closed the capability gap with closed frontier models — a remarkable democratization enabled by the Chinchilla insight and open-source infrastructure.