Part 2: SFT Data Engineering — NEFTune & Synthetic Data Curation
← Previous Chapter: Part 1: Hybrid AI Architecture | Series Hub | Next Chapter: Part 3: QLoRA & Axolotl Fine-Tuning → Prerequisite: Read Part 1: Hybrid AI Architecture & Self-Hosting vLLM for inference routing and self-hosted gateway topology. Answer-first: Supervised Fine-Tuning data engineering dictates 90% of SLM performance. Following the LIMA paradigm, 3,500 curated instruction samples outperform 100,000 noisy scraped records. Injecting uniform embedding noise via NEFTune provides an 18.4% AlpacaEval gain against rote memorization, while SemDeDup clustering eliminates 45% semantic redundancy without accuracy loss, cutting GPU training hours by 50%. ...