Messaging

Chapter 3: Shopee Traffic Shield — Kafka Peak Shaving & Circuit Breaking in Go

Previous Chapter: Chapter 2 — Flash Sale Engine & Zero Overselling | Series Hub | Next Chapter: Chapter 4 — Database Scalability: From MySQL to TiDB Answer-first: Shopee defends its e-commerce infrastructure during mega shopping surges using a multi-layered traffic shield combining WAF rate limiting, virtual waiting rooms, and Apache Kafka asynchronous peak shaving. Decoupling order creation from relational persistence flattens extreme traffic spikes, preserving database stability while ensuring sub-fifty-millisecond checkout response times and zero message loss across millions of concurrent users. ...

Part 5: Asynchronous Messaging, Kafka KRaft & Event-Driven Systems

← Previous Chapter: Part 4: Database Scaling & Sharding | Series Hub: System Design Masterclass | Next Chapter: Part 6: Distributed Locks, Mutex Invariants & Concurrency in Go → Prerequisite: Read Part 4: Database Scaling, Sharding Strategies & Distributed SQL to understand how databases decouple state before implementing asynchronous event streams. Answer-first: Asynchronous event streaming with Apache Kafka 3.9+ KRaft decouples distributed microservices by eliminating ZooKeeper coordination bottlenecks. In Go, pairing Cooperative Sticky consumer assignors with bounded channel worker pools enforces backpressure, while non-blocking exponential retry topics quarantine poison pill messages, sustaining 500,000 events per second with sub-5ms latency across cloud clusters. ...

Chapter 4: Dual-Write Prevention via Transactional Outbox in Go

Answer-first: Publishing messages to Kafka directly after database commits triggers catastrophic dual-write divergences during network timeouts or process crashes. The production standard is the Transactional Outbox Pattern powered by Log-based Change Data Capture: events are inserted atomically into an outbox table within the business transaction, and an external Debezium connector streams database write-ahead logs to Kafka with zero polling overhead. Prerequisite: Advanced understanding of database ACID transaction guarantees, distributed consistency anomalies, message broker delivery semantics (at-least-once vs exactly-once), and database replication mechanics is required. ...