Tokenizer Evolution
BPE, WordPiece, SentencePiece. The least glamorous component, and the one that quietly decides what a model can never represent.
- Published
- 2 August 2026
- Reading time
- 6 min read
- Figures
- 4 figures
- Equations
- 5 equations
Why Tokens? The Vocabulary Trade-off
A token is the atomic discrete unit the model consumes — its input is always a sequence of integer token IDs, never raw text.
The design choice of tokenization strategy determines vocabulary size and sequence length, and the trade-off between them is fundamental:
• Characters only: vocabulary of ~100 symbols, sequences 4–6× longer, long-range dependencies harder. • Words only: short sequences but an open-ended vocabulary where every new typo or domain term becomes 'unknown'. • Subwords: the Goldilocks compromise — common words are single tokens, rare words split into recognizable morphological pieces. Vocabulary is fixed and finite; sequences are manageable.
The key tension: larger vocabulary → shorter sequences (cheaper computation) but sparser statistics per token (harder to learn embeddings). Smaller vocabulary → richer per-token statistics but longer sequences and more O(n²) attention cost.
BPE — Byte Pair Encoding (Sennrich, 2016)
BPE was originally a compression algorithm. Adapted for NLP: start with individual characters as the vocabulary. Iteratively find the most frequent adjacent pair of tokens in the training corpus, merge them into one new token, and repeat until reaching the target vocabulary size.
GPT-2 extended BPE to byte-level: treat raw UTF-8 bytes (0–255) as base units. Because every string is representable as bytes, there are literally zero unknown tokens — any input, language, code, or emoji is guaranteed to be encodable.
Tiktoken (OpenAI): cl100k_base (100,277 tokens) used by GPT-3.5/4; o200k_base (200K+ tokens) for newer models.
Glitch tokens (2023 discovery, 'SolidGoldMagikarp'): GPT-2's tokenizer assigned dedicated embeddings to obscure Reddit usernames that appeared rarely or never in the model's training data. Those embeddings remained near-random. The model could not reproduce them and instead produced erratic outputs. Largely fixed in cl100k_base.
def byte_pair_encoding(vocab, num_merges):
for _ in range(num_merges):
pairs = get_stats(vocab)
if not pairs:
break
# Find most frequent pair
best = max(pairs, key=pairs.get)
# Merge pair in vocabulary
vocab = merge_vocab(best, vocab)
return vocabWordPiece & Unigram LM Tokenizer
WordPiece (Wu et al., 2016): similar to BPE but selects merges by likelihood gain rather than raw frequency. Score(a,b) = count(ab)/(count(a)·count(b)) — favoring pairs whose combination is much more common than chance would predict. Continuation subwords carry a '##' prefix. Used by BERT, RoBERTa, ELECTRA.
Unigram LM (Kudo, 2018): the inverse approach — start with a large candidate vocabulary and prune it. Each sentence's probability is the product of its tokens' unigram probabilities. Training: EM loop — estimate probabilities, compute each token's marginal contribution, prune the bottom-x% tokens, repeat until reaching the target vocabulary size.
Because multiple valid segmentations of a sentence exist under Unigram, the tokenizer is probabilistic — during training, you can sample different segmentations, which acts as regularization. Used by T5, ALBERT, mT5.
SentencePiece & Byte-Level Tokenization
SentencePiece (Kudo & Richardson, 2018) is a framework, not an algorithm — it wraps BPE or Unigram LM with a crucial difference: it treats the entire raw text (including spaces and newlines) as a sequence of Unicode characters, marking word boundaries with the visible character ▁. No language-specific word-splitting rules are ever needed. Handles Chinese, Japanese, Arabic, Thai natively. Used by T5, LLaMA, Gemma, mT5.
ByT5 (Xue et al., 2021): process directly on UTF-8 bytes. Vocabulary = exactly 256. Zero unknown tokens guaranteed. Any script, any code, any binary-adjacent content is representable. Trade-off: sequences are 3–4× longer for typical text, greatly increasing compute requirements.
Multilingual disparity: English-centric BPE vocabularies assign 2–4 tokens to a single Chinese or Arabic character, meaning non-English users pay 2–4× more in API costs and have proportionally less usable context window for the same text.
Tiktoken & Regex Filtering
Modern tokenizers like OpenAI's tiktoken use complex regex patterns before applying BPE merges. This prevents the tokenizer from merging tokens across natural boundaries (like punctuation, spaces, or numbers).
For example, it forces numbers to be split into up to 3 digits (e.g., '1000' -> '100', '0') to ensure more consistent mathematical representation, although it's not perfect.
The Numbers Problem
LLMs historically struggle with arithmetic. A major reason is tokenization. If ' 1' is token A, '1' is token B, and '123' is token C, the model has to learn addition from scratch for every possible token representation of a number.
Llama 3 and recent models force tokenizers to split all numbers into individual digits. By standardizing the input representation (every number is a sequence of base-10 digits), the transformer can finally learn the underlying algorithmic rules of addition and multiplication more reliably.