Part 4: Knowledge Distillation from DeepSeek-R1 & Frontier Teachers
← Previous Chapter: Part 3: QLoRA & Axolotl | Series Hub | Next Chapter: Part 5: Preference Alignment with DPO → Answer-first: Distillation transfers the step-by-step reasoning patterns (Chain-of-Thought) of large reasoning models (DeepSeek-R1, o3-mini) into small student models. Fine-tuning a 3B model on 10,000 verified reasoning traces yields math and code accuracy comparable to a 70B general model.