SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-distilbert-numbers

DistilBERT made BERT 40 percent smaller. About how much of its language understanding did it keep?

About 97 percent, while running 60 percent faster.

Start playing free

  1. About 97 percent ✔
  2. About 60 percent
  3. About 40 percent

Why: Hugging Face distilled BERT during pretraining, training a smaller student to copy the big model. It lost very little skill while becoming much cheaper to run, good enough for phones.

Huh, didn't know that: DistilBERT's training combined three losses: language modeling, distillation and cosine distance.

The dad joke
DistilBERT is 40 percent smaller and keeps 97 percent of the smarts. I wish my suitcase worked like that.
Like this card?
Play the game

References

Answer: About 97 percent. Hugging Face distilled BERT during pretraining, training a smaller student to copy the big model. It lost very little skill while becoming much cheaper to run, good enough for phones.

  1. Sanh et al., DistilBERT, a distilled version of BERT (arXiv 1910.01108) arxiv.org/abs/1910.01108
    reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceIn Hinton's knowledge distillation, how does the big teacher model produce soft targets for the student?Artificial IntelligenceHow did BERT's pretraining differ from a left-to-right GPT-style model?Artificial IntelligenceHow do AI, machine learning and deep learning fit together?

That's stuff everyone should know.