SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-dpo-without-rl

What does Direct Preference Optimization (DPO) replace RLHF's reinforcement learning with?

A simple classification loss on preferred vs rejected answers.

Start playing free

  1. A bigger reward model
  2. More human labelers
  3. A simple classification loss ✔

Why: RLHF fits a reward model and then runs reinforcement learning, which can be unstable. The 2023 DPO paper showed the same goal can be reached with a simple classification loss, no sampling during fine-tuning.

Huh, didn't know that: The DPO paper's subtitle is Your Language Model is Secretly a Reward Model.

The dad joke
DPO skipped the reinforcement learning. Some of us skip the gym the same way.
Like this card?
Play the game

References

Answer: A simple classification loss. RLHF fits a reward model and then runs reinforcement learning, which can be unstable. The 2023 DPO paper showed the same goal can be reached with a simple classification loss, no sampling during fine-tuning.

  1. Rafailov et al., Direct Preference Optimization (arXiv 2305.18290) arxiv.org/abs/2305.18290
    allowing us to solve the standard RLHF problem with only a simple classification loss
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceIn the InstructGPT study, which answers did people prefer: a 1.3B RLHF model's or the 175B GPT-3's?Artificial IntelligenceIn RLHF as used for InstructGPT, what did human labelers provide for the reinforcement learning step?Artificial IntelligenceHow do AI, machine learning and deep learning fit together?

That's stuff everyone should know.