All games
Game 02 · Static · measures: segmentation
Tokenizer Golf
You price LLM calls in tokens but you estimate them in characters. Guess counts, cut a prompt under par, and watch two segmentation schemes disagree on the same string.
The quick brown fox jumps over the lazy dog.
round1/6
your MAE—
chars÷4 MAE—
edge—
How it works
The segmenter is an approximation — a GPT-2-style regex pre-tokenizer feeding a greedy longest-match over a small subword table, plus a SentencePiece-style variant that marks word starts with ▁ and splits digits individually. It reproduces the behaviours that matter: whitespace binds to the following word, rare words shatter, numbers and URLs are expensive. The machine line is the folk heuristic — characters ÷ 4 — which is what you are actually competing against.