Sharding

Answer-first: Consistent Hashing minimizes key remapping when cluster membership changes. Adding or removing one node from a modulo-hash cluster remaps nearly all keys (catastrophic cache miss storm). Consistent Hashing remaps only $K/N$ keys — the theoretical minimum necessary. Prerequisite: Part 9 of the System Design Masterclass. Read Part 4: Database Scaling for context on horizontal partitioning strategies. What You’ll Learn Virtual Node Standard Deviation: The exact mathematical variance drop when increasing virtual node count ($V$) from 1 to 1000. RWMutex Lock Contention: Why using sync.RWMutex on the hash ring can cause lock contention under high multi-core throughput, and how to optimize with atomic values. CRC32 vs Murmur3: Why the choice of hashing algorithm on the ring impacts lookup distribution uniformity. Why Modulo Hashing Fails When Scaling Key Concept: hash(key) % N changes to hash(key) % (N+1) when a node is added, causing nearly all key-to-node mappings to change. This creates a massive cache miss storm as the entire working set must be reloaded from the database simultaneously. ...

Answer-first: Horizontal database sharding with Vitess and TiDB distributes high-volume write traffic across database clusters using consistent hashing and range partitioning. Prerequisite: Part 4 of the System Design Masterclass. Read Part 3: Caching Strategies first. Database Sharding in Go — TiDB, PostgreSQL & Connection Pools Answer-first: Horizontal database sharding partitions SQL tables across independent database nodes using hash or range shard keys. In Go services, combining application-level shard routing with tuned database/sql connection pools (SetMaxOpenConns, SetMaxIdleConns) prevents RAM exhaustion and write bottlenecks. ...

Prerequisite: Read the previous article: Chapter 8: Distributed Locking — Redlock vs ZooKeeper. When your application reaches tens of millions of users, the Database becomes the ultimate bottleneck. CPU maxes out at 100%, RAM depletes, and queries take seconds instead of milliseconds. This is the stage where you must deploy distributed database strategies. 1. Read/Write Splitting Because 80% of traffic is Read-only, separate your DB into a Write Master and Read Slaves. Use GORM’s dbresolver plugin to route queries automatically without altering business logic. ...

Vitess vs GORM Sharding: MySQL Write Scaling in Go When your application reaches millions of users, a single database instance will inevitably become the biggest bottleneck in your entire architecture. To solve this, MySQL database scaling becomes mandatory. You must Scale DB for Microservices using Horizontal Scaling techniques. This article examines the differences between scaling methods and compares the two most popular Sharding architectures today: Middleware-level Sharding (Vitess) and Application-level Sharding in Go (GORM Sharding plugin). ...

Replace MySQL Sharding with TiDB: Distributed SQL Architecture Scaling a relational database is one of the most demanding challenges in system design. As applications grow from thousands to millions of active users, the database ceases to be a simple storage engine and becomes the primary bottleneck of the entire system architecture. In this technical guide, we explore the architectural progression of scaling MySQL—beginning with replication topologies, stepping through the complexities and operational hazards of manual database sharding (including proxy middleware like Vitess), and evaluating NewSQL alternatives, specifically the distributed architecture of TiDB. ...

Answer-first: Shopee scales its relational database layer past single-node MySQL limits by migrating to TiDB Distributed SQL. By separating stateless SQL compute (TiDB) from stateful key-value storage (TiKV) and columnar analytics (TiFlash), TiDB delivers transparent horizontal auto-sharding and ACID transactions without application-level sharding logic. Chapter 4: Database Scale - The Rise of TiDB and NewSQL ← Series hub | ← Prev | Next → Prerequisite: Read the previous article: Chapter 3: Traffic Shield - Peak Shaving with Kafka and Graceful Degradation. ...