System Design

The Engineering Challenge Building a modern logistics platform (like food delivery, ride-hailing, or fleet management) requires computing distances and Estimated Times of Arrival (ETA) at an immense scale. The $N^2$ Problem: If you have 1,000 drivers and 1,000 orders, calculating the distance between every possible combination requires 1,000,000 individual route calculations. Speed: These calculations must happen in real-time (under 50ms) to ensure seamless user experiences and prevent dispatching algorithms from timing out. Accuracy: The system must account for real-world constraints such as one-way streets, “no left turn” rules, and dynamic traffic congestion. Standard point-to-point APIs (like basic Google Maps API calls) are too slow and too expensive for massive Distance Matrix generation. You need an internal, highly optimized Routing Engine. ...

← Series hub Next → For the past two years, enterprise AI adoption has been dominated by a singular architectural pattern: API integration with massive, closed-source models (Frontier LLMs). While this API-Centric model allows for rapid prototyping, it becomes a severe liability when scaled to production workloads handling sensitive company data. The Problem with API-Centric Architectures Relying exclusively on commercial APIs (such as GPT-4 or Claude 3.5 Sonnet) introduces three critical bottlenecks for scale-ups and enterprises: ...

In less than two years since its launch, the Model Context Protocol (MCP) has transformed from an internal Anthropic initiative into an open industry standard. Now managed by the Agentic AI Foundation under the Linux Foundation, MCP is redefining how we design software systems. If TCP/IP connects computers, and REST connects microservices, then the MCP protocol was born to directly connect LLMs (Large Language Models) and AI Agents with real-world data and tools. ...

While using an AI to write code or answer support tickets is becoming commonplace, the true transformation in enterprise software lies in Agentic Systems. We are moving away from monolithic, single-prompt architectures toward distributed networks of AI Agents that can plan, coordinate, and execute complex workflows autonomously. The Limitation of the “Single Agent” Paradigm Many organizations begin their AI journey by building a “monolithic agent”—stuffing an entire knowledge base and every possible tool into a single LLM’s context window. As the system scales, this approach inevitably collapses: ...

The software industry is witnessing a historic transfer of power. Power is gradually leaving the hands of those who “only know how to type code” to those who “know how to solve problems using systems and AI.” Context: When “Writing Code” is No Longer an Exclusive Skill For over two decades, the value of a programmer was largely measured by their understanding of language syntax, mastery of frameworks (React, Angular, Spring Boot, etc.), and ability to memorize APIs. ...

Prerequisite: To understand the context and why we need Multi-Agent systems instead of traditional Microservices, please refer to Comprehensive AI-Native System Architecture. When first approaching GenAI, most developers start by stuffing a massive prompt into a single LLM, hoping it completes the entire task. However, as the system scales, this “Single Monolithic Agent” approach reveals fatal flaws regarding performance, cost, and risk control. That is when we need a Multi-Agent System. ...

For years, the image of a talented programmer was often associated with blazing fast typing speeds, the ability to memorize dozens of API libraries, and writing code without a single syntax error. We called them pure “Coders”. But as AI enters the playing field, a harsh reality has emerged: Writing code is only the easiest part of building software. Who are “Code Typists”? “Code Typists” is not a derogatory term, but a way to describe a common working state. You are in this state if: ...

Upon realizing that typing speed has been defeated by AI (as discussed in Part 1), an invisible fear engulfs programmers: “So what will I do if AI does everything?” The answer lies in clearly defining the boundary: AI doesn’t do “everything”. AI only handles the technical muscle work, while humans retain the brains and responsibility. To optimize the software development process without losing control, we need to draw a red line between the “Machine’s Territory” and the “Human’s Territory”. ...

Social media and tech marketing campaigns constantly inject a concept into our heads: “10x Developer thanks to AI”. The image of a programmer sipping coffee, typing a few prompts, and finishing a week’s worth of work in one morning is incredibly appealing. But the truth in the trenches of real-world projects is much harsher. AI provides immense power, but it follows the law of conservation of energy: The time you save when “typing code” will be partially (or entirely) reclaimed during the reading and maintenance phases if you don’t know what you’re doing. ...

The traditional Software Development Life Cycle (SDLC) is often described as a factory assembly line. Business Analysts (BA) write requirements $\rightarrow$ Designers draw UI $\rightarrow$ Developers (Dev) write code $\rightarrow$ Quality Assurance (QA) finds bugs $\rightarrow$ DevOps pushes to the server. Everyone sits in their own “silo” and communicates via Jira tickets. But AI has swung a sledgehammer, smashing these walls. When a BA can ask AI to generate a runnable Proof of Concept, and a Developer can ask AI to write automated test scripts, the boundaries between roles become incredibly blurred. ...

So far, we have discussed AI extensively from the perspective of Programmers and Testers. But if you step into the boardroom of the Board of Directors (BOD) or Chief Technology Officers (CTO), you’ll see a completely different lens. Executives (BOD) don’t care how fancy your AI is, or how long your prompts are. Their lens consists of 3 vital variables: Cost, Time-to-Market, and Risk Management. The misalignment between BOD expectations and the working reality of Programmers is creating a zone of extreme pressure. ...

In Part 5, we saw the Board of Directors (BOD) frantically equipping internal AI systems to push productivity KPIs. At this point, if you stubbornly sit and type every line of code from start to finish, you will be left behind. To survive, programmers must shed the “Coder” jacket and put on the “AI Orchestrator” mantle. What is an AI Orchestrator? Imagine you’ve just been promoted to Tech Lead, and under your command is a swarm of extremely agile but… brainless (lacking contextual thinking) AI “interns”. ...

No matter how top-tier your Prompt Engineering skills are, sooner or later you will hit a reality wall: Writing code to create a feature is easy, but designing a system that can handle millions of users is incredibly difficult. In an era where AI is taking over “typing” tasks, System Design is the life preserver, the “inviolable territory” that keeps you from being phased out. AI is Good at “Building Rooms”, Not “Building Houses” Imagine software development as building an apartment complex. ...

At this point, we have painted a relatively bright prospect: Programmers escaping the drudgery of boring typing, becoming System Architects, and orchestrating AI. But this prospect is only true for Senior Developers — those who already have a solid professional foundation to assess the right/wrong of source code. For newcomers (Freshers/Juniors), the advent of AI has inadvertently created the worst training crisis in history: The Junior Paradox. How Does This Paradox Work? For the last 20 years, the evolutionary path from Junior to Senior was a path full of “suffering” but necessary. You learned CSS hacks, you cried over a missing semicolon (;), you struggled to config Webpack, and you repeatedly wrote hundreds of CRUD functions from project to project. It was those hours of “struggling” with basic problems that formed what is called Technical Intuition or “Programming Muscle”. ...

In the previous 8 parts, we dissected using AI as a Tool to assist programmers. We explored the death of syntax memorization, the boundaries of responsibility, navigated AI review fatigue and legal landmines, and established the need for Orchestration and System Design. But in this final part, we will flip the script entirely. The ultimate mission of a System Architect (AI-Driven Architect) is not just coding faster, but putting AI as the “heart” of the very product they are building. We call this AI-Native Application architecture. ...

Part 1: Architectural Decision Framework How can a Senior Developer or System Architect make the right decision between using a Modular Monolith and Microservices? The answer doesn’t lie in the hype, but in quantitative factors: Team organization structure, data integrity, and transaction volume. This article provides a solid Decision Framework based on real-world Latency Benchmarks and lessons from one of the most optimized Monolith systems in the world: Stack Overflow. ...

Answer-first: LLMs are now commodities; the new battleground is orchestrating Autonomous Swarms (multi-agent systems) on Kubernetes. To run these swarms safely in 2026, Platform Engineers must merge advanced K8s scheduling, Zero Trust identity, and robust state management. Here is the definitive blueprint for operating AI Swarms on Kubernetes. Core Orchestration: State & Scale Answer-first: Treat AI agents as stateless Deployments while offloading memory and workflows to external vector databases and Dapr. This prevents data loss during pod restarts and ensures horizontal scalability. ...

Welcome to this week’s Tech Radar. In our previous issue, we dove deep into Kratos Clean Architecture & Dapr. Today, we are discussing a monumental shift: Kubernetes has officially become the Operating System (OS) for AI. Let’s review the massive breaking news from Google Cloud, Microsoft, and the absolute dominance of Golang over the past 72 hours. 1. Tech News Radar: K8s “AI OS”, GKE Hypercluster & AKS Answer-first: Kubernetes has evolved far beyond a container orchestrator to become the standard Operating System for AI, currently handling 66% of generative AI workloads. Massive updates like GKE Hypercluster (managing 1 million chips) and AKS on Bare Metal reaffirm K8s’ absolute dominance in 2026. ...

Welcome to this week’s Tech Radar. In our previous issue, we explored Kratos Clean Architecture & Dapr Pub/Sub. Today, we tackle the most complex domain of distributed systems: Stateful Orchestration. We will dissect how to implement Dapr Workflows and the Actor model within Kratos. Before we dive into the code, let’s look at the breaking news from the past 72 hours. 1. Tech News Radar: Dapr v1.18 & KubeCon India 2026 Answer-first: The past 72 hours brought massive shifts. Dapr v1.18 dropped with WorkflowAccessPolicy for hard-gated workflow security, OpenTelemetry officially graduated from CNCF at KubeCon India, and Go 1.26.4 shipped. Meanwhile, Kubernetes 1.33 reaches End-of-Life on June 28. ...

Prerequisite: This is Part 12 of the System Design Masterclass. Previous parts built the reliability patterns — this part covers comparing communication protocols and data formats for microservice communication. Answer-first: gRPC is optimized for internal microservices using binary Protobuf serialization over multiplexed HTTP/2 or HTTP/3 streams. REST uses standard JSON over HTTP/1.1 or HTTP/2, serving as the default for public APIs. GraphQL operates as an aggregator at the API gateway or Backend-for-Frontend (BFF) layer, allowing clients to query specific properties, but requires complexity limits and DataLoader batching to prevent server degradation. ...

Prerequisite: This is Part 11 of the System Design Masterclass. Previous parts built the core components — this part covers securing APIs and managing client traffic spikes at scale. Answer-first: API rate limiting defends backend services by restricting request volume. Security requires a layered defense: Web Application Firewalls (WAF) block edge-level volumetric spikes, API Gateways manage L7 credentials and quotas, and application middleware enforces fine-grained business limits. Client identification must rely on validated, secure IP parsing (using the PROXY protocol or rightmost X-Forwarded-For checks). ...

Prerequisite: This is Part 10 of the System Design Masterclass. Previous parts built the architecture — this part teaches you how to see inside a running system and diagnose production performance issues. Answer-first: Go’s built-in pprof profiler provides CPU sampling, heap allocation analysis, goroutine stack inspection, and blocking profiler — all available as HTTP endpoints in running production services with minimal overhead. Heap diff between two snapshots is the fastest way to identify memory leaks. ...

Prerequisite: Part 9 of the System Design Masterclass. Read Part 4: Database Scaling for context on horizontal partitioning strategies. Answer-first: Consistent Hashing minimizes key remapping when cluster membership changes. Adding or removing one node from a modulo-hash cluster remaps nearly all keys (catastrophic cache miss storm). Consistent Hashing remaps only $K/N$ keys — the theoretical minimum necessary. Why Modulo Hashing Fails When Scaling Answer-first: hash(key) % N changes to hash(key) % (N+1) when a node is added, causing nearly all key-to-node mappings to change. This creates a massive cache miss storm as the entire working set must be reloaded from the database simultaneously. ...

Prerequisite: Part 8 of the System Design Masterclass. Read Part 7: Idempotent API Design first — compensating transactions in Saga must be idempotent. Answer-first: The Saga Pattern coordinates distributed transactions across microservices by decomposing a large transaction into a sequence of local transactions. If any step fails, the system automatically executes compensating transactions in reverse order to undo completed steps. Each local transaction must be idempotent. What Are the Problems with 2PC in Microservices? Answer-first: Two-Phase Commit (2PC) is a blocking protocol with a coordinator single point of failure. If the coordinator crashes between the Prepare and Commit phases, all participants are blocked indefinitely with locks held — a catastrophic failure mode in microservices. These are the same core banking distributed transaction challenges seen in legacy systems. ...

Prerequisite: Part 7 of the System Design Masterclass. Read Part 6: Distributed Locks — concurrent duplicate request blocking relies on the same mutual exclusion primitives. Answer-first: API idempotency ensures that retrying an identical request (same Idempotency-Key) never produces additional side effects beyond the first execution. This is foundational for payment APIs where network timeouts force client retries, and a duplicate execution would mean a double charge. What Is an Idempotency Key? Answer-first: An Idempotency Key is a unique token — typically UUID v4 — generated by the client and attached as an Idempotency-Key HTTP header. The server uses this key to detect duplicate requests: if the key has been seen before, return the cached response from the first execution without re-executing the handler. ...

Prerequisite: Part 6 of the System Design Masterclass. Read Part 5: Kafka & Event-Driven to understand event sourcing patterns before tackling lock coordination. Answer-first: Distributed locks solve the mutual exclusion problem across independent servers — ensuring only one server can modify a shared resource at a time. Redis Redlock provides high-performance locking using majority quorum across multiple master nodes; etcd provides stronger guarantees via Raft consensus at the cost of higher latency. ...

Prerequisite: Part 5 of the System Design Masterclass. Read Part 4: Database Scaling to understand the storage tier that persisted events are written to. Answer-first: Event-Driven Architecture decouples services through asynchronous communication via a durable message log. In Go, goroutines and buffered channels implement natural backpressure — when consumers fall behind producers, the channel fills up and blocks the producer, throttling the ingest rate automatically. Kafka vs RabbitMQ — When to Use Each? Answer-first: Kafka is a distributed commit log — messages are retained indefinitely, consumers manage their own offsets, and replay is possible. RabbitMQ is a message broker — messages are deleted after acknowledgment, the broker handles routing complexity, push-based delivery. They solve different problems. ...

Prerequisite: Part 4 of the System Design Masterclass. Read Part 3: Caching Strategies to understand the cache layer before examining storage. Answer-first: Database sharding distributes data horizontally across independent partitions (shards) based on a shard key, reducing write contention and enabling linear storage growth. Choosing the wrong shard key leads to hot spots that can be worse than no sharding at all. Vertical vs Horizontal Scaling — When to Switch? Answer-first: Vertical scaling (scale-up) increases resources on a single server — simple but has a hard physical ceiling and non-linear cost growth. Horizontal scaling (scale-out) adds more servers — no theoretical ceiling, linear cost, but significantly higher operational complexity. ...

Prerequisite: Part 3 of the System Design Masterclass. Read Part 2: Load Balancing L4/L7 to understand the traffic layer before diving into the caching tier. Answer-first: Effective caching strategy selection hinges on the acceptable consistency window and the read/write access pattern of the workload. Write-Through suits financial records; Write-Behind suits analytics and event counters; Cache-Aside is the default for read-heavy API responses. How Does Cache Stampede Happen? Answer-first: Cache Stampede (thundering herd) occurs when a popular cached key expires and multiple concurrent goroutines simultaneously detect a cache miss — then all query the database simultaneously. The burst of duplicate DB queries can exceed connection pool capacity and cause cascading failure. ...

Prerequisite: Part 2 of the System Design Masterclass. Read Part 1: System Design Thinking first to understand foundational trade-off frameworks. Answer-first: L4 load balancing routes traffic by transport-layer (IP/TCP/UDP) metadata — minimal CPU overhead but limited intelligence. L7 load balancing inspects HTTP headers, paths, and cookies — enables content-based routing and advanced health checks at the cost of higher processing overhead per request. L4 vs L7 Load Balancing — The Definitive Comparison Answer-first: The fundamental difference is where in the network stack the routing decision is made. L4 (Transport Layer) routes at TCP/UDP level using IP+port tuples. L7 (Application Layer) routes at HTTP level using headers, URLs, and payloads. ...