Messaging

Chapter 3: Shopee Traffic Shield — Kafka Peak Shaving & Circuit Breaking in Go

Previous Chapter: Chapter 2 — Flash Sale Engine & Zero Overselling | Series Hub | Next Chapter: Chapter 4 — Database Scalability: From MySQL to TiDB Answer-First: Shopee utilizes Apache Kafka queues for asynchronous peak shaving during 11.11 mega-campaigns. Decoupling user order submission from relational database persistence guarantees sub-50ms API responses while downstream consumer workers insert orders at a flat, controlled rate. Combined with Alibaba Sentinel adaptive load shedding, virtual waiting rooms, and strict Dead Letter Queue (DLQ) isolation, this traffic shield absorbs 10x traffic surges without database connection pool exhaustion or cascaded outages. ...

Part 5: Asynchronous Messaging, Kafka KRaft & Event-Driven Systems

← Previous Chapter: Part 4: Database Scaling & Sharding | Series Hub: System Design Masterclass | Next Chapter: Part 6: Distributed Locks, Mutex Invariants & Concurrency in Go → Prerequisite: Read Part 4: Database Scaling, Sharding Strategies & Distributed SQL to understand how databases decouple state before implementing asynchronous event streams. Answer-first: Asynchronous event streaming with Apache Kafka 3.9+ KRaft decouples distributed microservices by eliminating ZooKeeper coordination bottlenecks. In Go, pairing Cooperative Sticky consumer assignors with bounded channel worker pools enforces backpressure, while non-blocking exponential retry topics quarantine poison pill messages, sustaining 500,000 events per second with sub-5ms latency across cloud clusters. ...

Chapter 4: Dual-Write Prevention via Transactional Outbox in Go

Answer-first: Publishing messages to Kafka directly after database commits triggers catastrophic dual-write divergences during network timeouts or process crashes. The production standard is the Transactional Outbox Pattern powered by Log-based Change Data Capture: events are inserted atomically into an outbox table within the business transaction, and an external Debezium connector streams database write-ahead logs to Kafka with zero polling overhead. Prerequisite: Advanced understanding of database ACID transaction guarantees, distributed consistency anomalies, message broker delivery semantics (at-least-once vs exactly-once), and database replication mechanics is required. ...