SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-learning-rate-too-high

What happens if the learning rate is set too high?

The model never converges; it bounces around the minimum.

Start playing free

  1. Training takes many small steps but converges
  2. The model overshoots and never converges ✔
  3. The model converges quickly and smoothly

Why: The learning rate sets the step size. Too small and training crawls. Too big and every step overshoots the low point, so loss jumps around instead of settling.

Huh, didn't know that: Learning rate is a hyperparameter: you set it before training, like 0.01, rather than the model learning it.

The dad joke
My learning rate was so high the model skipped right over the answer and landed in next week.
Like this card?
Play the game

References

Answer: The model overshoots and never converges. The learning rate sets the step size. Too small and training crawls. Too big and every step overshoots the low point, so loss jumps around instead of settling.

  1. Google ML Crash Course: Hyperparameters developers.google.com/machine-learning/crash-course/linear-regression/hyperparameters
    if the learning rate is too high, the model never converges, but instead bounces around the weights and bias that minimize the loss.
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhich of these is a hyperparameter you set, not something the model learns?Artificial IntelligenceWhat changes when a pretrained model is fine-tuned rather than only used frozen?Artificial IntelligenceHow do AI, machine learning and deep learning fit together?

That's stuff everyone should know.