Distributed Systems

Answer-first: Moving AI agents from toy demos to enterprise production requires treating them as Stateful Distributed Systems. This series documents the 6 core pillars of production agentic architecture: Swarm Topology (Router/Worker vs Shared Blackboards), Hierarchical Memory Management, Resilient Tool-Calling Protocols, AgentOps Observability, Automated Evals, and Human-in-the-Loop (HITL) Gateways. 🎯 The Architectural Challenge of Autonomous Agents Building production-ready AI agents is fundamentally a distributed systems engineering challenge, not a prompt engineering trick: ...

Prerequisite: Familiarity with the concepts introduced in Part 1 — Microservices Gitops. Review it first if the terminology in this part is unfamiliar. Answer-first: Managing transaction surges during PayPay’s massive marketing campaigns requires event-driven architecture powered by Apache Kafka. Partition key tuning, Go consumer worker pools, and channel-based backpressure prevent message loss during peak traffic spikes. Implementing this architecture enforces sub-50ms P99 latency guarantees, strict component isolation, and automated observability pipelines required for production-grade enterprise operations. ...

← Previous Chapter: Executive Summary | Series Hub | Next Chapter: Part 2: Hierarchical Memory → Answer-first: For enterprise workflows with deterministic SLAs, Hierarchical Router-Worker architectures provide predictable task decomposition and strict failure isolation. Shared Blackboard patterns excel in open-ended collaborative research but require strict concurrency locking to prevent state corruption.

← Series hub ← Prev • Next → Answer-first: Alipay’s Logical Data Center (LDC) unitization architecture partitions database tables and application servers into self-contained “RZone” units based on user ID hashes. This multi-active setup bounds failure blast radiuses and allows horizontal scaling across multiple data centers. Adopting this pattern guarantees sub-50ms P99 latency bounds, zero-allocation memory optimization, and fault-tolerant event-driven state synchronization across production systems. Prerequisite: Phase 1: Timeline and Scale Evolution ...

← Previous Chapter: Part 3 — Primary Key Showdown: UUIDv7 vs. Snowflake | Series Hub | Next Chapter: Part 5 — Sharded MySQL vs. TiDB NewSQL → Part 4: MariaDB vs. MySQL: Storage Engines & Thread Pool Showdown Answer-first: MariaDB is no longer a drop-in replacement for MySQL. MySQL 8.4/9.0 dominates Cloud-Native ecosystems (AWS Aurora) with InnoDB tuning, binary JSONB O(1) updates, and Vector AI. Conversely, MariaDB 11.x excels on Bare-Metal/Kubernetes via native ThreadPool (50k+ conns), Galera 4 zero-lag multi-master, and MyRocks LSM storage compressing disk by 70%. ...

Answer-first: Shopee scales its relational database layer past single-node MySQL limits by migrating to TiDB Distributed SQL. By separating stateless SQL compute (TiDB) from stateful key-value storage (TiKV) and columnar analytics (TiFlash), TiDB delivers transparent horizontal auto-sharding and ACID transactions without application-level sharding logic. Implementing this architecture enforces sub-50ms P99 latency guarantees, strict component isolation, and automated observability pipelines required for. Chapter 4: Database Scale - The Rise of TiDB and NewSQL ← Series hub | ← Prev | Next → ...

← Previous Chapter: Part 4 — MariaDB vs. MySQL | Series Hub | Next Chapter: Part 6 — Apache Kafka vs. NATS JetStream → Part 5: Sharded MySQL (Vitess) vs. TiDB NewSQL: Distributed ACID, Scale-Out Limits & Latency Penalties Answer-first: Sharded MySQL (Vitess) delivers unmatched sub-2ms write latency and isolated failure blast radius for clean single-shard workloads (tenant_id/user_id). Conversely, TiDB NewSQL is the definitive architecture for unpartitionable relational schemas and cross-shard queries via zero-touch 96MB Region auto-splits, trading off an 8–15ms write latency floor due to Google Percolator 2PC and Raft consensus hops. ...

← Series hub ← Prev • Next → Answer-first: Alipay’s Double 11 technology deep dive reveals high-performance internals: binary Bolt RPC protocol multiplexing over single TCP streams, RocketMQ 2PC transactional messaging for async decoupling, OceanBase LSM-tree compaction tuning, and multi-zone Paxos quorum consensus to achieve 544,000 TPS payment processing. Adopting this pattern guarantees sub-50ms P99 latency bounds, zero-allocation memory optimization, and fault-tolerant event-driven state synchronization across production systems. Prerequisite: Phase 4: Technology Overview ...

← Previous Chapter: Part 5 — Sharded MySQL vs. TiDB | Series Hub | Next Chapter: Part 7 — Modular Monolith vs. Microservices vs. SpinKube Wasm → Part 6: Apache Kafka vs. NATS JetStream: Event Streaming Showdown Answer-first: Apache Kafka (KRaft) excels in enterprise-scale event streaming, petabyte log retention, and strict partition-ordered analytics via OS page cache zero-copy I/O. Conversely, NATS JetStream is the optimal architecture for microservice meshes, edge deployments, and AI agent buses, offering sub-millisecond P99 latency, pure Go embedded Raft consensus, and 75% lower FinOps compute overhead. ...

← Previous Chapter: Part 6 — Apache Kafka vs. NATS JetStream | Series Hub | Next Chapter: Part 8 — Redis Distributed State vs. Dapr Virtual Actors → Part 7: Modular Monolith vs. Microservices vs. SpinKube Wasm Showdown Answer-first: Modular Monoliths deliver unmatched developer velocity, zero-latency in-memory calls (~0.5ns), and local ACID transactions for small-to-medium teams. Containerized Microservices provide independent deployments and polyglot boundaries at the cost of high network serialization and memory overhead. SpinKube WebAssembly represents the next paradigm, achieving sub-millisecond cold starts, 100x container density, and 75% FinOps savings. ...

📖 Series Navigation: ← Previous Chapter: Modular Monolith vs Microservices vs SpinKube Wasm | Series Hub Part 8: Redis Distributed State vs. Dapr Virtual Actors Showdown Answer-first: Redis in-memory state with Lua scripts excels at high-throughput (100k+ QPS), low-latency caching and raw data manipulation. However, for complex distributed state machines, turn-based concurrency, and long-lived stateful AI agent context, Dapr Virtual Actors eliminate race conditions, distributed locking overhead, and manual lifecycle plumbing via single-threaded mailboxes and automatic hydration. ...

← Previous Chapter: Part 8: Phase 3 — Full Cutover | Series Hub | Next Chapter: Part 10: ADR Walkthrough — 24 Architecture Decisions → Answer-first: In a distributed e-commerce architecture without 2-Phase Commit (2PC), distributed consistency is achieved via the Transactional Outbox Pattern (saving domain events in the same SQL ACID transaction as business state) and Orchestrated Sagas (executing compensating transactions upon payment or inventory failure). sequenceDiagram autonumber actor Customer as Customer participant Order as Order Service (Saga Orchestrator) participant Inventory as Inventory Service participant Payment as Payment Service Customer->>Order: Create Order Order->>Order: Save Order (PENDING) + Outbox Event (Atomic ACID) Order->>Inventory: Reserve Stock (gRPC) alt Inventory Available Inventory-->>Order: Stock Reserved OK Order->>Payment: Authorize Payment (gRPC) alt Payment Succeeded Payment-->>Order: Payment Captured OK Order->>Order: Update Order (CONFIRMED) Order-->>Customer: Order Placed Successfully! else Payment Failed Payment-->>Order: Card Declined Order->>Inventory: Compensating Tx: Release Reserved Stock Order->>Order: Update Order (CANCELLED) Order-->>Customer: Payment Failed end else Out of Stock Inventory-->>Order: Insufficient Stock Order->>Order: Update Order (CANCELLED) Order-->>Customer: Item Out of Stock end

Tech Radar: DeepSeek-V3 Multi-Head Latent Attention (MLA) Architecture & KV Cache Compression Answer-First: DeepSeek-V3’s Multi-Head Latent Attention (MLA) fundamentally addresses the memory bandwidth and capacity bottlenecks in large language model inference. By projecting Keys and Values into a low-rank latent compressed space ($d_{latent} = 512$) during KV cache generation, MLA achieves a 75% reduction in runtime VRAM consumption compared to traditional Multi-Head Attention (MHA) and Grouped-Query Attention (GQA), while simultaneously retaining the high expressive representational capacity of full attention matrices through Decoupled Rotary Position Embedding (RoPE). ...

Tech Radar: Deterministic Concurrency Testing with Go 1.25 testing/synctest Answer-First: The testing/synctest package in Go 1.25/1.26 eliminates flaky concurrency tests by isolating goroutines inside an event-driven “concurrency bubble” governed by a synthetic time clock. Virtual time advances instantaneously the moment all goroutines in the bubble are durably blocked, reproducing multi-step race conditions, backoff retries, and network timeouts in 2ms instead of waiting for 5–10s real-world time.Sleep() delays. 1. The Core Dilemma of Concurrency Testing: The time.Sleep Anti-Pattern In high-throughput Go microservices (Kafka stream consumers, Dapr actor sagas, gRPC retry circuits, distributed rate-limiters), testing timeouts, backoff strategies, and race conditions has historically suffered from flaky test instability. ...

Distributed Transactions in Go with Temporal Saga Pattern Answer-first: Implementing distributed transactions in Go with Temporal Saga orchestrates multi-service workflows, manages deterministic state replays, and executes compensating actions upon failure. Distributed transactions in Go microservices are commonly implemented using the Temporal Saga pattern: replacing blocking Two-Phase Commit (2PC) locks with imperative workflow orchestration, dynamic reverse compensations (saga.AddCompensation), and PostgreSQL idempotency tables to keep financial event consistency during network partitions. This guide covers: ...