SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-int8-halves-memory

Quantizing a 16-bit model's weights to 8-bit integers does what to the memory needed for inference?

Cuts it roughly in half.

Start playing free

  1. Doubles it
  2. Cuts it about in half ✔
  3. Leaves it unchanged

Why: Each weight shrinks from 16 bits to 8 bits, so the model takes about half the memory. The 2022 LLM.int8() method did this for models up to 175 billion parameters without losing performance.

Huh, didn't know that: LLM.int8() keeps a few outlier features in 16-bit, but more than 99.9 percent of values are multiplied in 8-bit.

The dad joke
Quantization is how I pack for vacation. Same stuff, half the bits.
Like this card?
Play the game

References

Answer: Cuts it about in half. Each weight shrinks from 16 bits to 8 bits, so the model takes about half the memory. The 2022 LLM.int8() method did this for models up to 175 billion parameters without losing performance.

  1. Dettmers et al., LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (arXiv 2208.07339) arxiv.org/abs/2208.07339
    which cut the memory needed for inference by half while retaining full precision performance
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhen serving an LLM, what memory grows with each request's length and limits how many requests fit at once?Artificial IntelligenceIn machine learning, what is inference?Artificial IntelligenceHow is standard dropout commonly handled at inference time?

That's stuff everyone should know.