SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-scaling-laws-power-laws

What mathematical pattern did neural language-model scaling laws observe for test loss?

Loss followed approximate power laws.

Start playing free

  1. Power-law trends ✔
  2. Perfectly flat lines
  3. Random monthly cycles

Why: Across measured ranges, loss changed predictably with parameter count, dataset size, and training compute. The relationships were empirical power laws, not guarantees of endless improvement.

Huh, didn't know that: The paper says some of the trends span more than seven orders of magnitude.

The dad joke
Scaling laws brought power to the exponent.
Like this card?
Play the game

References

Answer: Power-law trends. Across measured ranges, loss changed predictably with parameter count, dataset size, and training compute. The relationships were empirical power laws, not guarantees of endless improvement.

  1. Scaling Laws for Neural Language Models arxiv.org/abs/2001.08361
    The loss scales as a power-law with model size, dataset size, and the amount of compute used for training
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceHow do AI, machine learning and deep learning fit together?Artificial IntelligenceWhat did Alan Turing call his 1950 test of machine intelligence?Artificial IntelligenceTuring predicted that by about 2000, judges would ID the machine at most what percent of the time?

That's stuff everyone should know.