14-part series
LLM Atlas
Architectural Evolution, 2017–2026
Ten years of architectural churn, in dependency order: what a language model actually is, what the Transformer replaced and why, the lineages that descend from it, and the economics that decide what gets built.
Math Primer
Dot products, softmax, gradients. A handful of ideas carry the entire field — explained before anything depends on them.
What Is a Language Model?
Strip away the branding and a language model is one thing: a conditional probability distribution over the next token.
The Pre-Transformer Era
N-grams, word2vec, RNNs, LSTMs, and the attention mechanism bolted onto them. Everything the Transformer replaced, and why it needed replacing.
Attention Is All You Need (2017)
The 2017 architecture, component by component: QKV projections, multi-head attention, positional encoding, the feed-forward block, residuals and norms.
The Evolution of Attention
Causal masking, multi-head, grouped-query, cross-attention. The formula barely moves — what changes is where the numbers come from.
Cluster A: Quadratic Attention O(n²)
The O(n²) lineage: sparse patterns, sliding windows, FlashAttention. Keep exact attention, attack the constant factor instead.
Cluster B: Sub-Quadratic O(n)
The O(n) lineage: linear attention, kernel tricks, state-space models. Give up exactness, buy back the sequence length.
Beyond the Transformer Stack
Mamba, RWKV, RetNet. Architectures that are not Transformers wearing a hat.
Tokenizer Evolution
BPE, WordPiece, SentencePiece. The least glamorous component, and the one that quietly decides what a model can never represent.
Activations, Optimizers & Systems
GELU and SwiGLU, LayerNorm and RMSNorm, RoPE, Adam. The unglamorous machinery that makes a deep stack trainable at all.
Serving at Scale
KV-cache, paged attention, speculative decoding. Training is a one-off cost; inference is the bill that arrives every month.
The Economics of Scale
Kaplan, Chinchilla, and the data wall. What the scaling laws actually claim, and where they stop.
What Remains Unsolved
Long-context degradation, evaluation that measures the wrong thing, interpretability, and the problems no amount of compute has fixed.
What It Means & Where It's Going
What ten years of architectural churn adds up to, and which bets still look live from here.