SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-kv-cache-memory

When serving an LLM, what memory grows with each request's length and limits how many requests fit at once?

The KV cache, the stored keys and values for earlier tokens.

Start playing free

  1. The KV cache ✔
  2. The tokenizer vocabulary
  3. The training dataset

Why: To avoid redoing attention over old tokens, servers keep each token's keys and values. That cache grows as the text gets longer, so it caps the batch size. vLLM's PagedAttention manages it like virtual memory pages.

Huh, didn't know that: vLLM reported 2 to 4 times higher throughput at the same latency, thanks to near-zero KV cache waste.

The dad joke
The KV cache is like my phone's photo storage. It fills up right when things get interesting.
Like this card?
Play the game

References

Answer: The KV cache. To avoid redoing attention over old tokens, servers keep each token's keys and values. That cache grows as the text gets longer, so it caps the batch size. vLLM's PagedAttention manages it like virtual memory pages.

  1. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention arxiv.org/abs/2309.06180
    the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically
    Checked 2026-10-11.
  2. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention arxiv.org/abs/2309.06180
    an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems
    Checked 2026-10-11.
  3. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention arxiv.org/abs/2309.06180
    vLLM improves the throughput of popular LLMs by 2-4$\times$ with the same level of latency
    Checked 2026-10-11.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceQuantizing a 16-bit model's weights to 8-bit integers does what to the memory needed for inference?Artificial IntelligenceIn machine learning, what is inference?Artificial IntelligenceHow is standard dropout commonly handled at inference time?

That's stuff everyone should know.