SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-bert-bidirectional

How did BERT's pretraining differ from a left-to-right GPT-style model?

BERT looks at words on both sides of each position, left and right.

Start playing free

  1. BERT reads text right to left only
  2. BERT uses context from both left and right ✔
  3. BERT has no Transformer layers

Why: GPT-style models only see earlier tokens because they generate text. BERT is an encoder that conditions on both sides at once, which suits understanding tasks like question answering, but not free-flowing text generation.

Huh, didn't know that: BERT stands for Bidirectional Encoder Representations from Transformers.

The dad joke
BERT reads in both directions. Finally someone who reads the end of the book first and isn't ashamed.
Like this card?
Play the game

References

Answer: BERT uses context from both left and right. GPT-style models only see earlier tokens because they generate text. BERT is an encoder that conditions on both sides at once, which suits understanding tasks like question answering, but not free-flowing text generation.

  1. Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv 1810.04805) arxiv.org/abs/1810.04805
    jointly conditioning on both left and right context in all layers
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhich 2017 paper introduced the Transformer architecture?Artificial IntelligenceWhich two sequence mechanisms did the original Transformer dispense with?Artificial IntelligenceWhat does transfer learning reuse from a previously trained model?

That's stuff everyone should know.