SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-pretraining-first-phase

What kind of learning is usually used in the first, pretraining phase of an LLM?

Unsupervised (self-supervised) learning on huge amounts of text.

Start playing free

  1. Humans label every sentence first
  2. Unsupervised learning on raw text ✔
  3. Reinforcement learning from human ratings

Why: Pretraining needs no hand labels: the text itself supplies the answer, because the next token is already there. Human labels and ratings come later, in fine-tuning steps like instruction tuning and RLHF.

Huh, didn't know that: GPT-3's models were all trained for a total of 300 billion tokens.

The dad joke
Pretraining is like reading the whole library before anyone tells you what the test is about.
Like this card?
Play the game

References

Answer: Unsupervised learning on raw text. Pretraining needs no hand labels: the text itself supplies the answer, because the next token is already there. Human labels and ratings come later, in fine-tuning steps like instruction tuning and RLHF.

  1. Google ML Crash Course: LLMs and Transformers developers.google.com/machine-learning/crash-course/llm/transformers
    The first phase of training is usually some form of unsupervised learning on that training data.
    Checked 2026-10-10.
  2. Brown et al., Language Models are Few-Shot Learners (arXiv 2005.14165) arxiv.org/pdf/2005.14165
    All models were trained for a total of 300 billion tokens.
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhat does transfer learning reuse from a previously trained model?Artificial IntelligenceHow did BERT's pretraining differ from a left-to-right GPT-style model?Artificial IntelligenceHow do AI, machine learning and deep learning fit together?

That's stuff everyone should know.