SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-rlhf-ranked-outputs

In RLHF as used for InstructGPT, what did human labelers provide for the reinforcement learning step?

Rankings of model outputs, from best to worst.

Start playing free

  1. New training text from the web
  2. Rankings of several model outputs ✔
  3. Corrections to the model's weights

Why: Labelers first wrote example answers, then ranked several model answers to the same prompt. Those rankings trained a reward model, and reinforcement learning tuned the language model to score well on it.

Huh, didn't know that: RLHF stands for reinforcement learning from human feedback.

The dad joke
RLHF is basically a talent show where the judges never sing a note.
Like this card?
Play the game

References

Answer: Rankings of several model outputs. Labelers first wrote example answers, then ranked several model answers to the same prompt. Those rankings trained a reward model, and reinforcement learning tuned the language model to score well on it.

  1. Ouyang et al., Training language models to follow instructions with human feedback (arXiv 2203.02155) arxiv.org/abs/2203.02155
    a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceIn the InstructGPT study, which answers did people prefer: a 1.3B RLHF model's or the 175B GPT-3's?Artificial IntelligenceWhat does Direct Preference Optimization (DPO) replace RLHF's reinforcement learning with?Artificial IntelligenceHow do AI, machine learning and deep learning fit together?

That's stuff everyone should know.