SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-fp32-master-weights

Why can mixed-precision training keep an FP32 master copy of weights?

It preserves small parameter updates that FP16 may round away.

Start playing free

  1. To retain tiny updates ✔
  2. To add more tokens
  3. To disable gradients

Why: FP16 has a narrower numerical range and less precision. Accumulating optimizer updates into FP32 weights prevents small changes from disappearing through rounding.

Huh, didn't know that: The FP32 master weights can be rounded back to FP16 for forward and backward passes.

The dad joke
The master copy remembered the small change everyone else rounded off.
Like this card?
Play the game

References

Answer: To retain tiny updates. FP16 has a narrower numerical range and less precision. Accumulating optimizer updates into FP32 weights prevents small changes from disappearing through rounding.

  1. Mixed Precision Training arxiv.org/abs/1710.03740
    maintaining a single-precision copy of the weights that accumulates the gradients after each optimizer step
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhich algorithm makes gradient descent practical for multilayer neural networks?Artificial IntelligenceWhat is the main idea of mixed-precision neural network training?Artificial IntelligenceHow do AI, machine learning and deep learning fit together?

That's stuff everyone should know.