Kafka

Prerequisite: Before reading this part, review the Executive Summary. GPS Ingestion at Scale: gRPC Streaming, MQTT & Kalman Filter Answer-first: High-throughput location ingestion processes over 1 million GPS updates per second by using binary gRPC streams or MQTT over persistent TCP/QUIC connections. Devices run Kalman filters and dead-reckoning interpolation to clean telemetry noise before publishing updates to Apache Kafka and Redis. Key Takeaways: Protocol Overhead: Replacing HTTP REST with gRPC Protobuf binary framing (vtproto) reduces packet overhead from 800 bytes to 40 bytes per GPS update. Noise Reduction: Kalman filters apply prediction-correction matrix equations directly on handset sensors to eliminate urban canyon GPS reflections. Batching Savings: Aggregating 3-5 telemetry points into single gRPC frames saves up to 67% of mobile radio transmission energy. What You’ll Learn: ...

Prerequisite: Familiarity with the concepts introduced in Part 2 — Distributed Sql Acid Latency. Review it first if the terminology in this part is unfamiliar. Answer-first: Event sourcing and CQRS replace mutable database updates with an immutable append-only event log. Core banking systems record financial state changes as domain events, projecting read models asynchronously while guaranteeing auditability and zero data loss. Series (Part 3 of 8): This article builds upon the ACID transactions foundation from Part 2. We will design a ledger using Event Sourcing — the exact solution that Monzo, Starling Bank, and many large neo-banks use to scale. ...

Prerequisite: Familiarity with the concepts introduced in Part 1 — Microservices Gitops. Review it first if the terminology in this part is unfamiliar. Answer-first: Managing transaction surges during PayPay’s massive marketing campaigns requires event-driven architecture powered by Apache Kafka. Partition key tuning, Go consumer worker pools, and channel-based backpressure prevent message loss during peak traffic spikes. Answer-first: PayPay builds a decoupled microservices network by streaming transactions asynchronously via Apache Kafka. To ensure financial safety, consumers process events using idempotency keys tracked in distributed caches, preventing duplicate ledger entries or double-spend occurrences in the event of retries or network partition splits. ...

Magento Migration: Shared DB, CDC, or Event Bus? Why Go running against Magento’s MySQL is faster at the compute layer but still bottlenecked at the EAV query layer — and what actually fixes it. The single deciding factor between CDC (Option B) and Event Bus (Option C): who owns the PHP Magento codebase. This post is part of the Composable Commerce Migration series — a step-by-step playbook for migrating Magento 2 to Go microservices. For the full migration execution guide, see Part 6: Phase 1 Strangler Fig. ...

Composable E-Commerce Migration: Overcoming Tech Debt See the 21-service e-commerce architecture blueprint for the domain boundaries this migration targets. In theory, MACH (Microservices, API-first, Cloud-native, Headless) and Composable Commerce are the “holy grail” of the e-commerce industry. However, when systems scale to process millions of transactions, issues regarding data consistency, domain decomposition, and observability costs surface. This guide details the lessons and architectural patterns from migrating a monolithic Magento application into a 21-service Go microservices platform. ...

Prerequisite: Part 5 of the System Design Masterclass. Read Part 4: Database Scaling first. Kafka Worker Pool in Go — Backpressure & Exactly-Once Answer-first: High-throughput event streaming in Go leverages Kafka zero-copy sendfile() kernel transfers combined with bounded goroutine worker pools. Natural backpressure is achieved using buffered Go channels, while partition-pinned workers preserve message ordering without distributed locks. Key Takeaways: Zero-Copy Performance: Kafka bypasses user-space buffer copies via sendfile(), routing data directly from Linux page cache to network socket buffers. Channel Backpressure: Bounded Go channels automatically throttle poll loops when downstream workers reach memory capacity limits. Partition-Aware Ordering: Pinning specific Kafka partition IDs to dedicated worker goroutines maintains strict message sequence guarantees. What You’ll Learn Kernel-Level sendfile() Mechanics: How zero-copy I/O bypasses the context switches between user and kernel space, preventing CPU cache invalidation. Worker Pool Partition Pinning: Why mapping partitions to specific workers is the only way to maintain order processing sequences without locking. Offset Commit Transaction Math: Implementing transactional offset commits inside Go consumers to guarantee idempotency under broker rebalances. Kafka vs RabbitMQ — When to Use Each? Key Concept: Kafka is a distributed commit log — messages are retained indefinitely, consumers manage their own offsets, and replay is possible. RabbitMQ is a message broker — messages are deleted after acknowledgment, the broker handles routing complexity, push-based delivery. They solve different problems. ...

Composable Banking Architecture: Monolith to Modular Answer-first: The composable banking architecture pattern replaces monolithic core banking systems with modular, independent Packaged Business Capabilities (PBCs). By leveraging Go microservices, Saga orchestration, and the Strangler Fig migration pattern, banks can decouple their legacy ledgers without risky “Big Bang” cutovers. Migration Path from Monolith to Composable Transitioning to a composable core requires a phased approach to mitigate operational risk: API Gateway & Anti-Corruption Layer (ACL): Shield the legacy core behind a gateway and translate modern API requests into legacy formats using an ACL. Shadow Routing: Deploy the new composable service (e.g., a new Go-based ledger) in parallel. Mirror live traffic to it and reconcile the outputs without affecting actual customer balances. Incremental Cutover (Strangler Fig): Once reconciliation achieves 100% parity, route read traffic to the new service, followed by write traffic, effectively “strangling” that specific domain out of the monolith. Legacy core banking systems were designed in a different era. Temenos T24, Finacle, and Flexcube shared one defining assumption: the bank’s entire product catalogue — deposits, lending, payments, trade finance — would live inside a single, tightly coupled application and a single, shared database. That assumption held when banking moved at human speed. It breaks completely when product releases need to go from months to days, when a single fraud engine update must not risk a payments outage, and when engineers on a COBOL codebase are retiring faster than they can be replaced. ...

Prerequisite: Read the previous article: Chapter 3: Distributed Rate Limiting with Redis & GCRA Algorithm. When your Golang application migrates from a Monolith to event-driven Microservices, you will immediately face an architectural nightmare: the Dual-Write Problem. 1. What is the Dual-Write Problem? Dual-Write occurs when an app attempts to write to a Database and publish to a Message Broker (Kafka) simultaneously. Without a distributed transaction, network failures will cause the two systems to fall out of sync. ...

Real-Time Inventory Topology: CDC, Kafka, and Redis Real-time inventory synchronization is the process of propagating stock count changes from the system of record (database) to all sales channels — web storefront, mobile app, WMS, ERP — in sub-second time. Instead of batch ETL jobs that run every hour, a CDC + Kafka pipeline streams every committed stock change as an event, eliminating overselling and stale stock displays. Handling this during a flash sale — where thousands of users attempt to purchase a highly contested SKU simultaneously — is a pinnacle architectural challenge. Traditional synchronous database updates collapse under lock contention. ...

Go Microservices Distributed Tracing Architecture (2026) Monitoring complex Go microservices requires more than isolated logs. When a request traverses HTTP APIs, Kafka event streams, and asynchronous worker pools, you need absolute visibility to pinpoint latency bottlenecks and failures. By 2026, OpenTelemetry (OTel) has cemented itself as the vendor-neutral standard for telemetry. This guide explores the architecture of distributed tracing in Go, from SDK context propagation to advanced Collector Gateway configurations. ...

PayPay Architecture: Scaling to 70M Users & 100k Peak TPS PayPay launched in October 2018 and grew to 10 million users in just 3 months — a growth rate that no Japanese fintech had ever seen. By 2025, the platform had crossed 70 million registered users and processed 7.8 billion payments per year. Behind this growth is an engineering team that has had to scale not just their infrastructure, but their entire engineering culture: from service standardization and GitOps-driven deployments to chaos engineering and AI-powered fraud detection. ...

Real-Time Ride-Hailing Architecture: Matching, Spatial Indexing & Websockets The moment you open the Uber or Grab app, a cascade of real-time systems activates simultaneously: your phone begins transmitting GPS coordinates, a geospatial index updates your location, a matching engine re-evaluates nearby driver availability, a pricing model recalculates the fare based on supply-demand ratios, and a push notification pipeline prepares to deliver your match confirmation in under 3 seconds. What makes this hard is not any single component — it is the combination of all of them, processing millions of concurrent users, each with sub-second latency requirements, continuously. This post walks through all six layers of the real-time ride-hailing architecture stack, from GPS ingestion to driver notification, using Uber and Grab’s engineering practices as the reference model. ...

Prerequisite: Familiarity with the concepts introduced in Part 3 — Late Chunking Semantic Caching. Review it first if the terminology in this part is unfamiliar. Part 4 — Real-time Streaming CDC & Federated GraphRAG Architecture In mission-critical enterprise environments—such as financial trading desks, e-commerce order management, and medical health record platforms—data changes continuously. A product price adjustment, a contract terms revision, or a inventory status update occurs thousands of times per minute. ...

Answer-first: Shopee utilizes Apache Kafka queues for asynchronous peak shaving during 11.11 mega-campaigns. Decoupling order creation from database persistence guarantees sub-second API responses while downstream workers process orders at a controlled rate, protected by Sentinel adaptive load shedding and priority request classification. Chapter 3: Peak Shaving - The Power of Apache Kafka and Graceful Degradation ← Series hub | ← Prev | Next → Prerequisite: Read the previous article: Chapter 2: Flash Sale Engine - Solving Overselling and Hot Keys. ...

Mastering Event-Driven Architecture with Dapr Pub/Sub in Go In my previous post, we explored how abandoning monolithic architecture in favor of strict Domain-Driven Design (DDD) bounded contexts allowed an e-commerce platform to scale beyond 10,000+ orders per day. However, splitting one big database into 20+ isolated Postgres databases introduces a terrifying new problem: How do we maintain data consistency across disconnected services? The answer is Event-Driven Architecture (EDA). Rather than chaining blocking synchronous HTTP calls across the network — which guarantees a cascading failure if a single service is down — each microservice independently broadcasts out-of-band “Events” through a centralized broker. Services are decoupled from each other’s availability. A brief outage in the Notification service does not cause a checkout failure. ...