SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-vanishing-gradients-stall

What happens to layers whose training gradients vanish?

They train very slowly or not at all.

Start playing free

  1. They double in width
  2. They barely train ✔
  3. They tokenize inputs

Why: Deep backpropagation can multiply many small derivative terms. The resulting near-zero gradient gives early layers almost no useful update signal.

Huh, didn't know that: Vanishing gradients are especially troublesome when learning dependencies across many recurrent time steps.

The dad joke
The gradient vanished before paying the loss.
Like this card?
Play the game

References

Answer: They barely train. Deep backpropagation can multiply many small derivative terms. The resulting near-zero gradient gives early layers almost no useful update signal.

  1. Google Machine Learning Crash Course: Backpropagation developers.google.com/machine-learning/crash-course/neural-networks/backpropagation
    Layers with vanishing gradients train very slowly, or not at all.
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWith early stopping, when do you end training?Artificial IntelligenceWhat does the "deep" in deep learning refer to?Artificial IntelligenceWhy place nonlinear activation functions between neural network layers?

That's stuff everyone should know.