SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-layernorm-per-example-units

Across what values does layer normalization compute its statistics?

Across the hidden units of one layer for one example.

Start playing free

  1. Units in one layer and example ✔
  2. All labels in the dataset
  3. Only optimizer states

Why: Layer normalization computes a mean and variance from the summed inputs across a layer's hidden units. Different examples receive their own normalization statistics.

Huh, didn't know that: This design can work even with a batch size of one.

The dad joke
Layer norm kept every hidden unit on the same scale.
Like this card?
Play the game

References

Answer: Units in one layer and example. Layer normalization computes a mean and variance from the summed inputs across a layer's hidden units. Different examples receive their own normalization statistics.

  1. Layer Normalization arxiv.org/abs/1607.06450
    from all of the summed inputs to the neurons in a layer on a single training case
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWith early stopping, when do you end training?Artificial IntelligenceWhich algorithm makes gradient descent practical for multilayer neural networks?Artificial IntelligenceWhat happens to layers whose training gradients vanish?

That's stuff everyone should know.