How does byte pair encoding tokenization handle a rare word the model has never seen?
It splits the word into smaller subword pieces the model already knows.
- It replaces the word with an unknown marker
- It splits it into smaller subword units ✔
- It looks the word up in a dictionary
Why: A 2015 paper by Sennrich and colleagues encoded rare and unknown words as sequences of subword units, using byte pair encoding. That gives a fixed vocabulary that can still spell out any word.
Huh, didn't know that: Byte pair encoding started life as a data compression algorithm before it was used to build vocabularies.
My vocabulary is like a tokenizer. When I don't know a word, I just break it into bits and hope.
References
Answer: It splits it into smaller subword units. A 2015 paper by Sennrich and colleagues encoded rare and unknown words as sequences of subword units, using byte pair encoding. That gives a fixed vocabulary that can still spell out any word.
- Sennrich et al., Neural Machine Translation of Rare Words with Subword Units (arXiv 1508.07909) arxiv.org/abs/1508.07909
encoding rare and unknown words as sequences of subword units
Checked 2026-10-10.
Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.
More stuff everyone should know
That's stuff everyone should know.