SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-layernorm-train-test-same

Does layer normalization compute differently at training and inference time?

It performs the same computation in both.

Start playing free

  1. Yes, it switches to stored averages
  2. No, it is the same ✔
  3. Yes, it is turned off

Why: Layer normalization derives statistics from the current example rather than stored batch averages. It therefore does not need separate train-time and test-time formulas.

Huh, didn't know that: This also makes layer normalization straightforward to apply at separate RNN time steps.

The dad joke
Layer norm behaves the same on test day, no stage fright.
Like this card?
Play the game

References

Answer: No, it is the same. Layer normalization derives statistics from the current example rather than stored batch averages. It therefore does not need separate train-time and test-time formulas.

  1. Layer Normalization arxiv.org/abs/1607.06450
    layer normalization performs exactly the same computation at training and test times
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceIn machine learning, what is inference?Artificial IntelligenceHow is standard dropout commonly handled at inference time?Artificial IntelligenceAcross what values does layer normalization compute its statistics?

That's stuff everyone should know.