Quantization

← Previous Chapter: Part 2: SFT Data Engineering | Series Hub | Next Chapter: Part 4: Knowledge Distillation → Prerequisite: Read Part 2: SFT Data Engineering — NEFTune & Synthetic Data Curation for instruction dataset curation and decontamination. Answer-first: QLoRA compresses base model weights into 4-bit NormalFloat (NF4) representations while computing gradients exclusively through 16-bit adapter matrices. Combining Double Quantization with CUDA Paged Optimizers enables fine-tuning 14B models on a single 24GB commodity GPU (RTX 4090 or L4) at $1.20/hour, preserving 99.3% full-precision benchmark performance while preventing out-of-memory crashes. ...

← Previous Chapter: Part 5: Preference Alignment | Series Hub Prerequisite: Read Part 5: Preference Alignment with DPO & GRPO for preference alignment and JSON schema enforcement. Answer-first: High-throughput enterprise SLM serving overcomes the GPU Memory Wall via vLLM PagedAttention v2, Multi-Head Latent Attention KV cache compression, and AWQ 4-bit quantization. Coupled with dynamic Multi-LoRA serving via Punica CUDA kernels and automated CI/CD eval gates, a single 24GB commodity GPU sustains hundreds of concurrent streams at P99 latencies under 35ms. ...