SESKStuff Everyone Should Know
SESK: Stuff Everyone Should Know · sesk.info/ai-lora-frozen-weights

How does LoRA fine-tune a large model so cheaply?

It freezes the original weights and trains small low-rank add-on matrices.

Start playing free

  1. Freezes the weights, trains small add-on matrices ✔
  2. Retrains every weight in the model
  3. Deletes layers the task does not need

Why: Full fine-tuning updates every parameter, which is huge for a 175B model. LoRA leaves the base untouched and learns small matrices in each layer, so many tasks can share one base model.

Huh, didn't know that: For GPT-3 175B, LoRA cut trainable parameters by 10,000 times and GPU memory by 3 times.

The dad joke
LoRA is like adding a sticky note to a textbook instead of rewriting the whole book.
Like this card?
Play the game

References

Answer: Freezes the weights, trains small add-on matrices. Full fine-tuning updates every parameter, which is huge for a 175B model. LoRA leaves the base untouched and learns small matrices in each layer, so many tasks can share one base model.

  1. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (arXiv 2106.09685) arxiv.org/abs/2106.09685
    LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer
    Checked 2026-10-10.

Think this answer is wrong? Tap "Challenge this answer" on the card in the game and tell us why.

Test yourself in the game

Artificial IntelligenceWhat changes when a pretrained model is fine-tuned rather than only used frozen?Artificial IntelligenceWhat is instruction tuning?Artificial IntelligenceQLoRA made it possible to fine-tune a 65 billion parameter model on what hardware?

That's stuff everyone should know.