SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-instructgpt-small-beats-big

In the InstructGPT study, which answers did people prefer: a 1.3B RLHF model's or the 175B GPT-3's?

The small 1.3B RLHF model, despite 100 times fewer parameters.

Start playing free

  1. The small 1.3B RLHF model's answers ✔
  2. The 175B GPT-3's answers
  3. Neither, they were rated equal

Why: OpenAI's 2022 paper showed that fine-tuning with human feedback mattered more than size for following instructions. Raters preferred the small tuned model to raw GPT-3, despite 100 times fewer parameters.

Huh, didn't know that: The paper opens by saying bigger models are not automatically better at following a user's intent.

The dad joke
The small model beat the giant one. Turns out listening matters more than size.
Like this card?
Play the game

References

Answer: The small 1.3B RLHF model's answers. OpenAI's 2022 paper showed that fine-tuning with human feedback mattered more than size for following instructions. Raters preferred the small tuned model to raw GPT-3, despite 100 times fewer parameters.

  1. Ouyang et al., Training language models to follow instructions with human feedback (arXiv 2203.02155) arxiv.org/abs/2203.02155
    outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceIn RLHF as used for InstructGPT, what did human labelers provide for the reinforcement learning step?Artificial IntelligenceWhat does Direct Preference Optimization (DPO) replace RLHF's reinforcement learning with?Artificial IntelligenceIn Constitutional AI, what is the only human oversight used to judge harmfulness?

That's stuff everyone should know.