SESK: Stuff Everyone Should Know · sesk.info/ai-distillation-soft-targets
In Hinton's knowledge distillation, how does the big teacher model produce soft targets for the student?
By raising the temperature of its final softmax.
- By deleting half of its layers
- By giving only its single top answer
- By raising its softmax temperature ✔
Why: A high temperature spreads the teacher's probabilities out, so the student also learns which wrong answers are close, like a 2 that looks a bit like a 3. The student trains on those soft targets.
Huh, didn't know that: After training, the distilled student goes back to a temperature of 1.
Knowledge distillation: the teacher does the hard work and the student gets the summary. Just like my study group.
Like this card?
References
Answer: By raising its softmax temperature. A high temperature spreads the teacher's probabilities out, so the student also learns which wrong answers are close, like a 2 that looks a bit like a 3. The student trains on those soft targets.
- Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network (arXiv 1503.02531) arxiv.org/pdf/1503.02531
is to raise the temperature of the final softmax until the cumbersome model produces a suitably soft set of targets.
Checked 2026-10-11. - Google Machine Learning Glossary: Generative AI (distillation) developers.google.com/machine-learning/glossary/generative
Distillation trains the student model to minimize a loss function based on the difference between the outputs of the predictions of the student and teacher models.
Checked 2026-10-11.
Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.
More stuff everyone should know
That's stuff everyone should know.