SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-gsm1k-fresh-test

Researchers wrote GSM1k, fresh math problems like GSM8K. What happened to some models' scores?

They dropped, by up to 8 percent, a sign of overfitting to the old test.

Start playing free

  1. They dropped, by up to 8 percent ✔
  2. They rose sharply
  3. They stayed exactly the same for all

Why: A brand new held-out test that no model could have trained on shows real skill. Some model families scored worse on it than on GSM8K, suggesting they had partly memorized the public benchmark.

Huh, didn't know that: Many frontier models showed little sign of overfitting and did well on the new problems.

The dad joke
GSM1k is the pop quiz the AI didn't study for. Some of them panicked, just like me.
Like this card?
Play the game

References

Answer: They dropped, by up to 8 percent. A brand new held-out test that no model could have trained on shows real skill. Some model families scored worse on it than on GSM8K, suggesting they had partly memorized the public benchmark.

  1. Zhang et al., A Careful Examination of Large Language Model Performance on Grade School Arithmetic arxiv.org/abs/2405.00332
    we observe accuracy drops of up to 8%, with several families of models showing evidence of systematic overfitting
    Checked 2026-10-11.
  2. Zhang et al., A Careful Examination of Large Language Model Performance on Grade School Arithmetic arxiv.org/abs/2405.00332
    many models, especially those on the frontier, show minimal signs of overfitting
    Checked 2026-10-11.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhat is benchmark contamination for a language model?Artificial IntelligenceThe MMLU benchmark tests language models across how many subjects, from math to law?Artificial IntelligenceWhich data split is saved for the final check after tuning on the validation set?

That's stuff everyone should know.