Modular Monolith Observability: Logging & Profiling

Answer-first: Observability in modular monoliths leverages in-process OpenTelemetry span propagation across module boundaries without network serialization overhead. Combining in-memory context tracking with structured logging reduces telemetry ingestion costs while retaining microservice-level latency visibility. Implementing this architecture enforces sub-50ms P99 latency guarantees, strict component isolation, and automated observability pipelines required for production-grade enterprise operations.

Prerequisite: Before reading this part, please review Part 4: CI/CD Simplified.

Part 5: Observability in Memory – When Everything Shares a Single Call Stack

What You’ll Learn:

• In-Memory Trace Propagation: How Go context propagation handles tracing across package lines without network calls (~15ns vs 1.2µs).
• Cardinality Reduction: Techniques to strip connection attributes from logs, saving thousands in observability SaaS bills.
• Sampling & eBPF Profiling: How Pyroscope/Parca eBPF engines continuously profile Go runtimes under 1% overhead compared to heavy APM agents.

When operating a production system, observability determines whether an engineer resolves an outage within minutes or spends hours troubleshooting distributed failure modes. Microservices architectures make telemetry expensive and complex through distributed network span propagation.

Conversely, the modular monolith brings debugging back to process memory: monitoring the system through a single call stack. The following sequence diagram illustrates how in-memory OpenTelemetry spans flow across internal module interfaces without network serialization overhead.

sequenceDiagram
    autonumber
    participant Gateway as "API Gateway Handler"
    participant Orders as "internal/orders"
    participant Billing as "internal/billing"
    participant OTel as "Local In-Memory OTel Tracer"
    
    Gateway->>Orders: Invoke CreateOrder("ctx")
    Orders->>OTel: Start Module Span "orders.CreateOrder" ("< 1µs")
    Orders->>Billing: Invoke ProcessPayment("ctx") in RAM
    Billing->>OTel: Start Child Span "billing.ProcessPayment" ("< 1µs")
    Billing-->>Orders: Return Payment Result
    Orders-->>Gateway: Return HTTP 200 OK

1. The Pain of Distributed Tracing in Microservices

In a microservices architecture, a single user request triggers a chain of network calls across multiple microservices. Understanding latency bottlenecks or request drops requires deploying distributed tracing frameworks like Jaeger, Zipkin, or commercial APMs like Datadog.

This process relies on complex Network Trace Propagation:

1. Header Injection: Service A receives an API call, generates a 128-bit W3C traceparent header, and creates a local span.
2. Network Transport: When calling Service B over HTTP/gRPC, Service A serializes the trace context into network headers.
3. Deserialization & Extraction: Service B reads the incoming HTTP header, parses the hex-encoded string, and initializes a child span.
4. Out-of-Band Emission: Every service continuously emits trace spans over UDP or HTTP to local OpenTelemetry collectors.

The True Costs of Distributed Telemetry

• Latency & CPU Penalties: Header serialization, string allocations, and network socket writes add 2ms to 10ms of overhead per API hop.
• High-Cardinality APM Bills: Ingesting millions of span events per second leads to high monthly bills from SaaS observability providers due to high key-value tag cardinality.
• Span Fragmentation & Broken Traces: If an intermediate proxy fails to forward tracing headers or a service pod crashes mid-request, the trace breaks, rendering the telemetry useless.

2. In-Process Context Propagation & Profiling Benchmarks

In a modular monolith, all communication between domain modules occurs in RAM via direct Go function calls. Observability achieves maximum efficiency with zero network degradation.

A. Micro-Benchmark: In-Memory context.Context vs W3C Network Propagation

Passing tracing context in-process requires passing a pointer in Go’s context.Context across function calls, contrasting sharply with HTTP trace context serialization.

B. In-Process OpenTelemetry Tracing

Initiate an in-memory OpenTelemetry trace span directly using Go context.Context directly using Go context.Context without serializing HTTP headers.

// Direct in-memory span initiation without network serialization
ctx, span := otel.Tracer("internal/orders").Start(ctx, "CreateOrder")
defer span.End()

Because span creation involves updating internal pointer structures in local RAM rather than serializing network headers over a TCP socket, span initialization overhead drops from microseconds to sub-nanoseconds.

C. Single Call Stack & Pristine Crash Analysis

When a runtime panic occurs inside a microservice architecture, the stack trace terminates at the network boundary of that container. In a modular monolith, a single runtime panic generates an un-fragmented call stack.

The following console output illustrates how a Go runtime panic preserves the exact call stack across module boundaries from API handler down to database interaction.

goroutine 42 [running]:
main.processOrder(0xc0000a2000)
    /app/internal/orders/service.go:84 +0x1a4
main.deductStock(0xc0000a2000)
    /app/internal/inventory/service.go:112 +0x24b
main.executeSQLTx(...)
    /app/internal/storage/db.go:45 +0x88

D. eBPF Continuous Profiling vs APM Agent Overhead

Continuous profiling in 2026 relies on eBPF (Extended Berkeley Packet Filter) tools like Pyroscope and Parca, avoiding the CPU and memory footprint associated with traditional APM agent instrumentation.

E. Local Ring-Buffer Sampling & Cardinality Control

Distributed microservices emit every HTTP span across the wire, generating network congestion. In a modular monolith, trace spans remain inside process memory, enabling in-process tail-based sampling using circular ring buffers (sync.Map or atomic slices):

1. In-Memory Trace Buffering: As a request traverses internal modules (internal/orders -> internal/billing), trace spans accumulate inside local RAM buffers associated with the request trace_id.
2. Decision Engine at Endpoint Completion: When the top-level HTTP handler returns, an in-process sampler evaluates the request outcome. If the handler returned an HTTP 5xx error or latency exceeded a P99 threshold (e.g., 200ms), the full trace buffer flushes to the OpenTelemetry collector.
3. 99% Low-Latency Drop: Successful, low-latency requests drop 99% of internal module spans while keeping aggregate counters in local memory, reducing telemetry ingestion fees significantly.

The flowchart below visualizes this in-process tail-based sampling architecture, contrasting the full trace flush on anomalous/slow requests with the zero-cost drop of routine successful spans.

flowchart TD
    Req["Incoming HTTP Request"] --> Ingress["Ingress HTTP Handler"]
    Ingress --> SpanBuf["Local Ring-Buffer: Accumulate Internal Spans in RAM"]
    
    subgraph In_Memory_Execution ["In-Process Modular Monolith Execution"]
        SpanBuf --> Mod1["Module: orders.CreateOrder (Span 1)"]
        Mod1 --> WorkerPool["Bounded Worker Pool Event Bus (Trace Context Injected)"]
        WorkerPool --> Mod2["Module: billing.ChargeCustomer (Span 2)"]
        Mod2 --> Mod3["Module: inventory.ReserveStock (Span 3)"]
    end
    
    Mod3 --> Finish["Handler Completes: Evaluate Response Status & Latency"]
    Finish --> Decision{"HTTP 5xx Error OR Latency > 200ms?"}
    Decision -->|"Yes (Tail Anomaly)"| Flush["Flush 100% of Spans to OTLP Collector"]
    Decision -->|"No (Normal Path 99%)"| Drop["Drop Detailed Spans from RAM Buffer"]
    Drop --> Metrics["Increment Aggregate Prometheus Counter (module=orders, status=200)"]

For rate limiting and gateway observability, see our Distributed Rate Limiting with Redis & GCRA guide.

3. Go In-Memory Span Tracking & Log Correlation

To maintain complete correlation between logs, metrics, and traces without external collector dependencies, Go services inject trace identifiers directly into structured loggers such as slog or zap.

A. Domain Metric Tagging with OpenTelemetry

The Go code below illustrates how to register custom OpenTelemetry domain metrics with module-specific tags like module=billing or module=inventory.

package metrics

import (
	"context"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/metric"
)

type ModuleMetrics struct {
	orderCounter metric.Int64Counter
}

func NewModuleMetrics() (*ModuleMetrics, error) {
	meter := otel.GetMeterProvider().Meter("modular-monolith")
	counter, err := meter.Int64Counter(
		"domain_orders_processed_total",
		metric.WithDescription("Total number of processed domain orders"),
	)
	if err != nil {
		return nil, err
	}
	return &ModuleMetrics{orderCounter: counter}, nil
}

func (m *ModuleMetrics) RecordOrder(ctx context.Context, moduleName string, status string) {
	m.orderCounter.Add(ctx, 1, metric.WithAttributes(
		attribute.String("module", moduleName),
		attribute.String("status", status),
	))
}

B. Zap & Slog Trace Correlation

The Go example below shows how structured logging with log/slog automatically extracts trace_id and span_id from context.Context to correlate log output with active OpenTelemetry spans.

package logger

import (
	"context"
	"log/slog"
	"os"

	"go.opentelemetry.io/otel/trace"
)

type TraceHandler struct {
	slog.Handler
}

func NewTraceLogger() *slog.Logger {
	jsonHandler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo})
	return slog.New(&TraceHandler{Handler: jsonHandler})
}

func (h *TraceHandler) Handle(ctx context.Context, r slog.Record) error {
	spanCtx := trace.SpanContextFromContext(ctx)
	if spanCtx.IsValid() {
		r.AddAttrs(
			slog.String("trace_id", spanCtx.TraceID().String()),
			slog.String("span_id", spanCtx.SpanID().String()),
		)
	}
	return h.Handler.Handle(ctx, r)
}

C. In-Memory Span Tracker & Trace Context Propagation across Worker Pools

When asynchronous domain events cross in-process channel boundaries, the publisher’s OpenTelemetry trace context must be injected into the event metadata and extracted by the worker thread. The Go 1.25 implementation below establishes a bounded worker pool event bus that preserves W3C traceparent headers across channel queues without allocating network serialization buffers:

package main

import (
	"context"
	"errors"
	"fmt"
	"runtime"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/trace"
	"golang.org/x/sync/errgroup"
)

// TraceMetadataCarrier implements propagation.TextMapCarrier for in-memory channels
type TraceMetadataCarrier map[string]string

func (c TraceMetadataCarrier) Get(key string) string        { return c[key] }
func (c TraceMetadataCarrier) Set(key string, value string) { c[key] = value }
func (c TraceMetadataCarrier) Keys() []string {
	keys := make([]string, 0, len(c))
	for k := range c {
		keys = append(keys, k)
	}
	return keys
}

type DomainEvent struct {
	Name      string
	Timestamp time.Time
	Payload   any
	Carrier   TraceMetadataCarrier
}

type eventJob struct {
	ctx     context.Context
	event   DomainEvent
	handler func(ctx context.Context, ev DomainEvent) error
}

// BoundedTracedEventBus routes events through bounded workers with OpenTelemetry trace propagation
type BoundedTracedEventBus struct {
	jobQueue chan eventJob
	workers  int
	tracer   trace.Tracer
	eg       *errgroup.Group
	ctx      context.Context
	cancel   context.CancelFunc
}

func NewBoundedTracedEventBus(parentCtx context.Context, queueCapacity int, tracer trace.Tracer) *BoundedTracedEventBus {
	ctx, cancel := context.WithCancel(parentCtx)
	eg, groupCtx := errgroup.WithContext(ctx)

	workers := runtime.GOMAXPROCS(0) * 2
	if workers < 4 {
		workers = 4
	}

	bus := &BoundedTracedEventBus{
		jobQueue: make(chan eventJob, queueCapacity),
		workers:  workers,
		tracer:   tracer,
		eg:       eg,
		ctx:      groupCtx,
		cancel:   cancel,
	}

	for i := 0; i < bus.workers; i++ {
		workerID := i
		bus.eg.Go(func() error {
			bus.workerLoop(groupCtx, workerID)
			return nil
		})
	}

	return bus
}

func (b *BoundedTracedEventBus) workerLoop(ctx context.Context, workerID int) {
	propagator := otel.GetTextMapPropagator()

	for {
		select {
		case <-ctx.Done():
			return
		case job, ok := <-b.jobQueue:
			if !ok {
				return
			}
			// Extract trace context from event carrier into worker context
			workerCtx := propagator.Extract(ctx, job.event.Carrier)
			childCtx, span := b.tracer.Start(workerCtx, fmt.Sprintf("EventHandler:%s", job.event.Name))

			span.AddEvent("Worker started executing handler")
			err := job.handler(childCtx, job.event)
			if err != nil {
				span.RecordError(err)
			}
			span.End()
		}
	}
}

// Publish enqueues event with current OTel trace context injected
func (b *BoundedTracedEventBus) Publish(ctx context.Context, eventName string, payload any, handler func(ctx context.Context, ev DomainEvent) error) error {
	carrier := make(TraceMetadataCarrier)
	otel.GetTextMapPropagator().Inject(ctx, carrier)

	event := DomainEvent{
		Name:      eventName,
		Timestamp: time.Now().UTC(),
		Payload:   payload,
		Carrier:   carrier,
	}

	job := eventJob{
		ctx:     ctx,
		event:   event,
		handler: handler,
	}

	select {
	case b.jobQueue <- job:
		return nil
	case <-ctx.Done():
		return ctx.Err()
	default:
		return errors.New("event bus queue full: applying backpressure")
	}
}

func (b *BoundedTracedEventBus) Shutdown(ctx context.Context) error {
	close(b.jobQueue)
	b.cancel()
	return b.eg.Wait()
}

D. Dynamic Differential CPU & Memory Profiling (Go pprof)

In a distributed microservice cluster, identifying which component is consuming excess CPU or leaking memory requires aggregating dozens of disparate container profiles. In a Modular Monolith, the entire system runs within a single Go runtime, enabling instant Differential Profiling using standard net/http/pprof:

1. Continuous Baseline Snapshots: A background daemon captures 30-second CPU and heap profiles (profile.pb.gz) during normal operation and stores them in local object storage.
2. On-Demand Differential Analysis: When an alert triggers during high load, an engineer takes a live snapshot and runs a differential comparison:

   go tool pprof -http=:8080 -diff_base=baseline_heap.pb.gz incident_heap.pb.gz
   

3. Pinpointing Culprit Modules in Seconds: Because Go package paths reflect domain boundaries (internal/orders, internal/billing), the pprof flame graph immediately highlights which specific module aggregate or worker pool is responsible for heap growth or CPU throttling—bypassing hours of distributed service mesh investigations.

4. Production OpenTelemetry Go SDK & Alert Configuration

Configuring OpenTelemetry in production requires exporting internal module metrics and traces via OTLP gRPC collectors while setting target alert rules in Prometheus.

A. Production OpenTelemetry Go SDK Setup

The Go code below configures the official OpenTelemetry Go SDK to batch trace spans and export them asynchronously over gRPC.

package telemetry

import (
	"context"
	"fmt"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.17.0"
	"go.opentelemetry.io/otel/trace"
	"google.golang.org/grpc/credentials"
)

type TracerConfig struct {
	ServiceName  string
	CollectorURL string
	TLSCreds     credentials.TransportCredentials
}

func InitTracer(ctx context.Context, cfg TracerConfig) (*sdktrace.TracerProvider, error) {
	opts := []otlptracegrpc.Option{
		otlptracegrpc.WithEndpoint(cfg.CollectorURL),
	}
	if cfg.TLSCreds != nil {
		opts = append(opts, otlptracegrpc.WithTLSCredentials(cfg.TLSCreds))
	}

	exporter, err := otlptracegrpc.New(ctx, opts...)
	if err != nil {
		return nil, fmt.Errorf("failed to create OTLP trace exporter: %w", err)
	}

	res, err := resource.New(ctx,
		resource.WithAttributes(
			semconv.ServiceNameKey.String(cfg.ServiceName),
		),
	)
	if err != nil {
		return nil, fmt.Errorf("failed to create telemetry resource: %w", err)
	}

	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(res),
	)
	otel.SetTracerProvider(tp)
	return tp, nil
}

func StartSpan(ctx context.Context, moduleName, operationName string) (context.Context, trace.Span) {
	tr := otel.Tracer(moduleName)
	return tr.Start(ctx, operationName)
}

B. Prometheus Module Alert Rules Configuration

The Prometheus alert rule YAML configuration below detects high latency or elevated error rates scoped directly by the module label.

groups:
  - name: modular_monolith_alerts
    rules:
      - alert: ModularMonolithHighModuleLatency
        expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job="modular-monolith"}[5m])) by (le, module)) > 0.35
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High latency detected in module {{ $labels.module }}"
          description: "Module {{ $labels.module }} P99 latency exceeded 350ms for over 2 minutes."

      - alert: ModularMonolithModuleErrorRate
        expr: sum(rate(domain_orders_processed_total{status="error"}[5m])) by (module) / sum(rate(domain_orders_processed_total[5m])) by (module) > 0.05
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Elevated error rate in module {{ $labels.module }}"
          description: "Module {{ $labels.module }} error rate exceeds 5% over 1 minute."

Learn how to consolidate legacy microservices step-by-step in Part 6: Migration Playbook.

Frequently Asked Questions (FAQ)

Navigation & Next Steps

• Previous Part: Part 4: CI/CD Simplified
• Next Part: Continue to Part 6: Migration Playbook
• Related Guides: Modular Monolith Architecture and C10M High-Concurrency Architecture

Need help setting up low-overhead OpenTelemetry tracing for your monolith? Get in touch or hire our observability experts for an architectural review.

🔗 Next Step: Continue to Part 6 — Migration Playbook for the following module in the series.