KV Cache

KV cache stores Key and Value vectors from earlier tokens during generation so they are not recomputed each step.

Generating token N requires attention over tokens 0 through N-1. Without a cache, Keys and Values for every previous token are recomputed at every step, which is quadratic as the sequence grows. With a cache, each Key and Value is computed once and stored. For the new token you compute Query, Key, and Value only for that token, then attend over the cached Keys and Values.

Complexity drops from quadratic to linear in sequence length.

Memory is the cost. Cache size grows with sequence length times model dimension times number of layers times number of attention heads times 2, for K and V. On large models with long contexts, the cache can take tens of gigabytes and become the main inference memory bottleneck.

Mitigations include KV cache quantization to lower precision, paged attention as virtual memory for the cache, and sliding window attention that keeps only recent tokens. Efficient LLM serving depends on how this cache is stored and reused.

At step N you concatenate the new Key and Value onto the cache tensors and run attention against the full cached K and V. N-2. Memory still grows linearly with N, which is why long-context serving hits RAM before it hits FLOPs. Quantized caches store K and V in 8-bit or 4-bit.

Paged attention allocates cache in blocks like virtual memory so variable-length batches do not waste giant contiguous tensors. Sliding windows drop old entries when only recent tokens should attend. Inference servers keep key and value tensors from past tokens so they are not recomputed every step. That cache is often the memory bottleneck.