Event-Driven

Prerequisite: Familiarity with the concepts introduced in Part 1 — Microservices Gitops. Review it first if the terminology in this part is unfamiliar. Answer-first: Managing transaction surges during PayPay’s massive marketing campaigns requires event-driven architecture powered by Apache Kafka. Partition key tuning, Go consumer worker pools, and channel-based backpressure prevent message loss during peak traffic spikes. Answer-first: PayPay builds a decoupled microservices network by streaming transactions asynchronously via Apache Kafka. To ensure financial safety, consumers process events using idempotency keys tracked in distributed caches, preventing duplicate ledger entries or double-spend occurrences in the event of retries or network partition splits. ...

Prerequisite: Familiarity with the concepts introduced in Part 6 — Phase1 Strangler Fig. Review it first if the terminology in this part is unfamiliar. In Phase 1, both systems existed but only one wrote data: Magento. In Phase 2, both systems write data simultaneously. This is the most technically complex phase — and the one where most migrations introduce data corruption if they don’t have an explicit conflict resolution strategy. ...

Prerequisite: Familiarity with the concepts introduced in Part 8 — Phase3 Full Cutover. Review it first if the terminology in this part is unfamiliar. Answer-first: Distributed transaction consistency is achieved using a choreography-based saga paired with a PostgreSQL transactional outbox. Business mutations write to the outbox atomically. Background workers publish events to Dapr PubSub every 500ms, while idempotent consumer handlers process compensation events on failure. When a customer places an order on the Composable Commerce Platform, seven events need to happen in sequence across four independent services: Order created → Payment authorized → Stock reserved → Fulfillment triggered → Notification sent → Loyalty points awarded → Shipping label generated. Any of these can fail. The network can fail. The database can fail. A third-party payment gateway can time out. ...

High-Throughput Event-Driven Microservices in Go with NATS JetStream & CQRS Section 1: Architectural Rationale: Why Go + NATS JetStream for Event-Driven Microservices Beyond tens of thousands of transactions per second, synchronous request-response designs start hitting database write contention and cascading latency spikes. Command Query Responsibility Segregation (CQRS) paired with Event-Driven Architecture (EDA) isolates write commands from analytical queries, letting each side scale independently. To overcome these structural boundaries, high-scale engineering organizations adopt Command Query Responsibility Segregation (CQRS) paired with Event-Driven Architecture (EDA). By explicitly separating the write path (commands) from the read path (queries), CQRS allows each side to scale independently according to its access patterns. Commands execute lightweight state mutations against write-optimized engines, emitting immutable domain events into a high-performance message broker. Decoupled consumer workers asynchronously consume these events to populate specialized, read-optimized views (such as Redis key-value pairs, Elasticsearch documents, or PostgreSQL materialized read tables). ...

Prerequisite: Part 5 of the System Design Masterclass. Read Part 4: Database Scaling first. Kafka Worker Pool in Go — Backpressure & Exactly-Once Answer-first: High-throughput event streaming in Go leverages Kafka zero-copy sendfile() kernel transfers combined with bounded goroutine worker pools. Natural backpressure is achieved using buffered Go channels, while partition-pinned workers preserve message ordering without distributed locks. Key Takeaways: Zero-Copy Performance: Kafka bypasses user-space buffer copies via sendfile(), routing data directly from Linux page cache to network socket buffers. Channel Backpressure: Bounded Go channels automatically throttle poll loops when downstream workers reach memory capacity limits. Partition-Aware Ordering: Pinning specific Kafka partition IDs to dedicated worker goroutines maintains strict message sequence guarantees. What You’ll Learn Kernel-Level sendfile() Mechanics: How zero-copy I/O bypasses the context switches between user and kernel space, preventing CPU cache invalidation. Worker Pool Partition Pinning: Why mapping partitions to specific workers is the only way to maintain order processing sequences without locking. Offset Commit Transaction Math: Implementing transactional offset commits inside Go consumers to guarantee idempotency under broker rebalances. Kafka vs RabbitMQ — When to Use Each? Key Concept: Kafka is a distributed commit log — messages are retained indefinitely, consumers manage their own offsets, and replay is possible. RabbitMQ is a message broker — messages are deleted after acknowledgment, the broker handles routing complexity, push-based delivery. They solve different problems. ...

Prerequisite: Familiarity with the concepts introduced in Part 3 — Late Chunking Semantic Caching. Review it first if the terminology in this part is unfamiliar. Part 4 — Real-time Streaming CDC & Federated GraphRAG Architecture In mission-critical enterprise environments—such as financial trading desks, e-commerce order management, and medical health record platforms—data changes continuously. A product price adjustment, a contract terms revision, or a inventory status update occurs thousands of times per minute. ...

Mastering Event-Driven Architecture with Dapr Pub/Sub in Go In my previous post, we explored how abandoning monolithic architecture in favor of strict Domain-Driven Design (DDD) bounded contexts allowed an e-commerce platform to scale beyond 10,000+ orders per day. However, splitting one big database into 20+ isolated Postgres databases introduces a terrifying new problem: How do we maintain data consistency across disconnected services? The answer is Event-Driven Architecture (EDA). Rather than chaining blocking synchronous HTTP calls across the network — which guarantees a cascading failure if a single service is down — each microservice independently broadcasts out-of-band “Events” through a centralized broker. Services are decoupled from each other’s availability. A brief outage in the Notification service does not cause a checkout failure. ...