SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-token-can-be-subword

Why is one written word not always one language-model token?

Tokenizers may split words into subword pieces.

Start playing free

  1. A token can be a subword ✔
  2. Every token is a paragraph
  3. Tokens must be whole sentences

Why: A token is an atomic unit for a particular model, not necessarily a whole word. Uncommon words are often divided into roots, prefixes, or suffixes.

Huh, didn't know that: A common word may remain one token while a rare word becomes several.

The dad joke
The long word split the bill into subword tokens.
Like this card?
Play the game

References

Answer: A token can be a subword. A token is an atomic unit for a particular model, not necessarily a whole word. Uncommon words are often divided into roots, prefixes, or suffixes.

  1. Google Machine Learning Glossary developers.google.com/machine-learning/glossary
    a single word can be a single token or multiple tokens
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceHow does byte pair encoding tokenization handle a rare word the model has never seen?Artificial IntelligenceWhat does a learned embedding assign to each input token?Artificial IntelligenceWhich statement correctly distinguishes model parameters from tokens?

That's stuff everyone should know.