SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-compute-optimal-equal-scaling

Under the Chinchilla study's compute-optimal rule, what should double with model size?

The number of training tokens should also double.

Start playing free

  1. Only the learning rate
  2. Training tokens ✔
  3. Only the vocabulary

Why: For the regime studied, compute-optimal model size and training-token count scaled equally. A model twice as large should train on roughly twice as many tokens.

Huh, didn't know that: The study trained more than 400 language models to estimate this allocation.

The dad joke
The model doubled down and brought twice the tokens.
Like this card?
Play the game

References

Answer: Training tokens. For the regime studied, compute-optimal model size and training-token count scaled equally. A model twice as large should train on roughly twice as many tokens.

  1. Training Compute-Optimal Large Language Models arxiv.org/abs/2203.15556
    for every doubling of model size the number of training tokens should also be doubled
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhat does a learned embedding assign to each input token?Artificial IntelligenceWhich statement correctly distinguishes model parameters from tokens?Artificial IntelligenceWhy is one written word not always one language-model token?

That's stuff everyone should know.