Math Primer
Dot products, softmax, gradients. A handful of ideas carry the entire field — explained before anything depends on them.
- Published
- 2 August 2026
- Reading time
- 3 min read
- Equations
- 5 equations
Vectors and Matrices
A vector is an array of numbers, representing a point or direction in space. LLMs represent everything—words, images, concepts—as high-dimensional vectors (e.g., 4096 dimensions). A matrix is a 2D grid of numbers that transforms these vectors. When a token passes through a Transformer layer, it is repeatedly multiplied by matrices (weights) to update its meaning.
Dot Product (Similarity)
The dot product measures how aligned two vectors are. In attention mechanisms, we compute the dot product between a "query" vector (what a word is looking for) and a "key" vector (what another word offers). A higher dot product means the words are more relevant to each other in context.
Softmax Function
The softmax function takes a list of raw scores (logits) and squashes them into probabilities that sum to 1. In LLMs, it is used twice: first in the attention mechanism to determine how much "weight" to give each surrounding word, and second at the very end to output the probability of the next token.
Gradients and Derivatives
A derivative measures how much the output of a function changes if you slightly tweak the input. A gradient is simply a vector of derivatives for multiple variables. During training, backpropagation computes the gradient of the loss with respect to every single parameter, telling the optimizer exactly how to adjust the weights to improve the model.