Pprof

Prerequisite: This is Part 10 of the System Design Masterclass. Previous parts built the architecture — this part teaches you how to see inside a running system and diagnose production performance issues. Answer-first: Go’s built-in pprof profiler provides CPU sampling, heap allocation analysis, goroutine stack inspection, and blocking profiler — all available as HTTP endpoints in running production services with minimal overhead. Heap diff between two snapshots is the fastest way to identify memory leaks. ...

Answer-first: Go pprof is the standard library profiling tool for diagnosing CPU usage, memory allocation, and goroutine leaks in production Go services, with safe exposure via internal HTTP endpoints and minimal performance overhead when configured correctly. What You’ll Learn That AI Won’t Tell You Reading memory profiles to identify slow allocations in performance hot paths. Analyzing flame graphs to detect lock contention on global mutexes. Prerequisite: This guide covers how to profile and diagnose complex performance issues in production. If you are specifically dealing with unbounded goroutine growth, ensure you first understand the foundational concepts in Goroutine Leak Detection and Fix in Production Go Services. ...

Answer-first: Safely profile production Go services in Kubernetes by establishing a secure kubectl port-forward to the runtime’s pprof endpoint. Collecting CPU, memory, and goroutine profiles in real-time allows generating flame graphs or streaming data to Pyroscope without introducing high overhead. What You’ll Learn That AI Won’t Tell You Production port forwarding configuration to profile CPU without service downtime. Decoding complex memory profiles and locating garbage collection allocation hot paths. You’ve instrumented your Go service with net/http/pprof, run go tool pprof locally against the development binary, and spotted the hot path in your flame graph. Then you deploy to Kubernetes and the bottleneck disappears — because the workload profile in Kubernetes differs from local testing (different request mix, connection pool pressure, GC behavior under actual memory pressure, scheduler interference from co-located pods). ...

Answer-first: Goroutine leaks block indefinitely and hold GC roots, leading to slow, silent memory exhaustion and pod restarts. Pinpoint leaks by comparing pprof profile snapshots (specifically goroutineleak in Go 1.26), write deterministic tests with Go 1.24’s synctest, and set automated telemetry alerts on active goroutine counts. What You’ll Learn That AI Won’t Tell You Writing automated test cases that detect goroutine leaks before deploying. Analyzing production runtime stack traces to locate orphaned channels. A Kubernetes pod abruptly restarts with exit code 137. The memory metrics dashboard shows a slow, perfectly linear staircase pattern stretching over three days. There are no panic logs in stdout, no database errors, and no abnormal CPU spikes. Just a slow, silent OOM (Out Of Memory) death. ...