Serving at Scale
KV-cache, paged attention, speculative decoding. Training is a one-off cost; inference is the bill that arrives every month.
- Published
- 2 August 2026
- Reading time
- 2 min read
- Figures
- 1 figures
Quantization (PTQ & GGUF)
FP16 requires 2 bytes per parameter (a 70B model needs 140GB of VRAM just to load). Quantization compresses this to 8-bit, 4-bit, or even lower.
• PTQ (Post-Training Quantization): Reduces precision after training. Methods like GPTQ and AWQ are data-aware—they look at a small calibration dataset to protect the most important 'outlier' weights from being rounded into oblivion. • GGUF: A popular file format that allows running heavily quantized models (e.g., Q4_K_M) efficiently on consumer hardware (MacBooks and CPU/RAM).
Speculative Decoding
Inference is memory-bandwidth bound, not compute bound. When generating a token, the GPU does a massive matrix multiplication but only processes batch-size 1. Most of the compute capacity sits idle.
Speculative Decoding fixes this by pairing a small, fast 'draft' model with the massive 'target' model. 1. The draft model rapidly generates 4-5 tokens. 2. The target model processes all 4-5 tokens in a single parallel forward pass to verify them. 3. Because LLMs run faster in parallel (prefill) than sequentially (decode), this provides a 2-3x speedup with absolutely zero loss in quality—it is mathematically identical to running the target model alone.
Continuous Batching & PagedAttention
Serving thousands of users requires batching their requests. But requests arrive at random times and have different lengths.
• Continuous Batching: Instead of waiting for a batch to finish, the engine ejects completed requests and inserts new ones at the very next token iteration. • PagedAttention (vLLM, 2023): Inspired by OS virtual memory. Previously, KV-cache was allocated in massive contiguous blocks, leading to terrible fragmentation (up to 60% memory waste). PagedAttention splits the KV-cache into small blocks (e.g., 16 tokens) that can be stored non-contiguously in VRAM. This tripled serving throughput for the industry.