NVIDIA H100

Tech Radar: vLLM v1 Production Engine Architecture & Distributed KV Cache Optimization: PagedAttention v3, Dynamic Chunked Prefill & RoCEv2 Zero-Copy Transfers Answer-First: vLLM v1 re-engineers production LLM serving by replacing Python-Ray actor coordination with a zero-overhead C++ core and lock-free execution loop. Coupling PagedAttention v3, dynamic chunked prefill, and multi-tier RoCEv2 KV offloading slashes P99 TTFT by 78% (410ms to 92ms), restricts memory fragmentation to <2.4%, and boosts 8x NVIDIA H100/H200 cluster throughput by 2.7x. ...

Tech Radar: Disaggregated Prefill-Decode Architecture: Decoupling Compute & Memory Bandwidth via RoCEv2 KV-Transfer Answer-First: Disaggregated Prefill-Decode serving defines 2026 enterprise LLM infrastructure, resolving the tension between compute-heavy prefill and memory-bound decode. By streaming KV caches across 400Gbps RoCEv2 fabrics, it cuts P99 TTFT by 11x (420ms to 38ms) and eliminates decode latency jitter on NVIDIA H100 clusters. name: "Disaggregated Prefill-Decode Serving" ring: "Adopt" quadrant: "AI Infrastructure & Large Language Models" rationale: "Decouples compute-bound prompt prefill from memory-bandwidth-bound token decode, eliminating head-of-line blocking and slashing P99 TTFT by 11x via zero-copy RoCEv2 KV transfer." adr_link: "/radar/2026-09/disaggregated-prefill-decode/" justification: "Empirically verified across 64x NVIDIA H100 SXM5 GPUs on DeepSeek-V3 and Llama-3.1-70B; production-ready in vLLM v1 and Mooncake architectures with 2.8x higher throughput per dollar." 1. The Compute vs. Memory-Bandwidth Dichotomy in Autoregressive Serving Autoregressive large language model serving is governed by two radically divergent computational regimes, creating an insurmountable structural tension within traditional monolithic GPU deployments: ...

Tech Radar: SGLang EAGLE-2: Speculative Decoding & Tree-Attention Latency Acceleration Answer-First: SGLang’s native EAGLE-2 implementation establishes the 2026 state-of-the-art for autoregressive latency acceleration, overcoming the memory-bandwidth wall (<1 FLOP/byte) by combining lightweight multi-layer feature extrapolation with dynamic tree-attention verification. On 8x NVIDIA H100 GPU clusters, EAGLE-2 achieves an empirical 2.5x to 3.5x generation speedup and reduces Time-Per-Output-Token from 35.7ms to 11.6ms on Llama-3-70B, with mathematically zero output distribution degradation. name: "SGLang EAGLE-2 Speculative Decoding" ring: "Adopt" quadrant: "AI Infrastructure & Large Language Models" rationale: "Overcomes autoregressive memory bandwidth saturation by verifying dynamic candidate trees in a single forward pass, delivering 3x generation speedup with zero quality loss." adr_link: "/radar/2026-09/sglang-eagle-2-speculative-decoding/" justification: "Empirically verified across Llama-3-70B and DeepSeek-Coder-33B on 8x NVIDIA H100 SXM5; production-ready in SGLang runtime with under 1.5GB VRAM overhead." 1. The Autoregressive Bottleneck & Evolution of Speculative Inference Large language model inference is characterized by two distinct computational phases: the Prefill Phase and the Decode Phase. During Prefill, prompt tokens are processed concurrently in compute-dense matrix multiplications, fully saturating modern GPU Tensor Cores. However, during the sequential Decode Phase, generation is strictly memory-bandwidth bound: ...