Answer-first: In 2026, Platform Engineering for AI is no longer about picking the right LLM framework. The real questions are: Who controls token cost? Who routes traffic intelligently to the right GPU pod? Where does agent state go after a crash? Three CNCF projects — Envoy AI Gateway, the K8s Gateway API Inference Extension, and Dapr Agents — are converging to answer those questions at the infrastructure layer, so application code doesn’t have to.

Layer 1 — FinOps: Envoy AI Gateway Controls Tokens Like a Financial Budget

Answer-first: Envoy AI Gateway (v1.0+, mid-2026) places a QuotaPolicy at the infrastructure layer. Instead of measuring requests per second, it measures actual token consumption and fires HTTP 429 as a financial circuit breaker before an agent loop can burn through the entire quota.

The core problem: traditional gateways (NGINX, vanilla Kong) cannot inspect the body of an LLM request. They have no idea whether a request consumes 50 tokens or 50,000. Platform teams only discover they’ve blown the budget when the monthly invoice arrives.

QuotaPolicy: Token Budget Per-Tenant

Envoy AI Gateway solves this by integrating an ext_proc (External Processing) filter into Envoy to parse the LLM response body and extract actual token counts from response metadata.

apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: AIGatewayRoute
metadata:
  name: tenant-a-route
spec:
  rules:
  - matches:
    - headers:
      - name: x-tenant-id
        value: "team-backend"
    backendRefs:
    - name: openai-backend
      weight: 100
    - name: azure-openai-fallback   # automatic provider fallback
      weight: 0
---
apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: QuotaPolicy
spec:
  limits:
  - tokenBudget: 500000          # 500K tokens/day per team
    window: 24h
    selector:
      header: "x-tenant-id"
    costs:
      inputToken: 1
      outputToken: 3              # CEL expression: output costs 3x more
      reasoningToken: 5           # thinking tokens are the most expensive

When team-backend exceeds 500K tokens, the gateway returns 429 Too Many Requests before the request ever reaches OpenAI. Not a single token wasted.

Model Virtualization: Code That Doesn’t Know Which Provider It’s Using

The least-noticed but most strategically important feature for large teams: Model Virtualization. Application code calls model: "gpt-production", and the gateway maps it to gpt-4o or claude-sonnet-4 according to policy — no redeploy needed when switching providers.

Combining with LiteLLM: Many teams use LiteLLM at the application layer to normalize calls across 100+ providers, while using Envoy AI Gateway at the infrastructure layer for token-level FinOps enforcement. The two tools are not mutually exclusive.

Layer 2 — Smart Routing: K8s Gateway API Inference Extension Brings GPU Routing Into the CNCF Standard

Answer-first: InferencePool + InferenceModel are two new CRDs from K8s SIG Network (Q1/2026) that allow an Endpoint Picker (EPP) to route AI traffic based on real-time KV cache state and GPU queue depth — instead of the blind round-robin of a standard Kubernetes Service.

When you use a plain Kubernetes Service to load balance vLLM pods, traffic is distributed evenly. But LLM inference is inherently uneven: one pod may already hold the KV cache for the exact prompt prefix a new request needs, while another pod would have to recompute from scratch. The result: unnecessarily high TTFT (Time To First Token) and GPU hotspotting.

InferencePool and Endpoint Picker: Intelligent Routing

apiVersion: inference.networking.k8s.io/v1alpha2
kind: InferencePool
metadata:
  name: llama-3-pool
spec:
  targetPortNumber: 8000
  selector:
    app: vllm-llama3
  extensionRef:
    name: endpoint-picker-epp    # the routing "brain"
---
apiVersion: inference.networking.k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: llama-3-70b
spec:
  modelName: "meta-llama/Meta-Llama-3-70B-Instruct"
  criticality: Standard          # Low | Standard | Critical
  poolRef:
    name: llama-3-pool

Endpoint Picker (EPP) is a plugin using the ext_proc protocol to query each pod for:

  • Current KV cache utilization
  • GPU queue depth (pending requests)
  • Loaded LoRA adapters

EPP routes each request to the pod with the highest cache hit for that request’s prompt prefix → significant TTFT reduction, hotspot elimination.

llm-d: Disaggregated Inference — CNCF Sandbox Project

llm-d (Red Hat/Google/IBM/NVIDIA, CNCF Sandbox 2026) goes one step further: it splits prefill (prompt processing — compute-heavy) and decode (token generation — memory-bandwidth-heavy) into two separate GPU pools.

  • Prefill pool: High tensor-parallelism A100 GPUs — fast prompt processing
  • Decode pool: GPUs with large HBM — sustained token generation throughput

Combined with InferencePool and KV cache-aware routing, llm-d routes requests sharing the same prompt prefix to the same decode worker → maximizes cache hit rate, increasing throughput by 30–50% according to the project’s internal benchmarks.

Maturity note: llm-d is still a CNCF Sandbox project (not yet Graduated). Suitable for teams who want to move early, but production readiness requires careful evaluation.

Layer 3 — Reliability: Dapr Agents GA — An Agent Is a Workflow, Not a Stateless Pod

Answer-first: Dapr Agents v1.0 reached GA in March 2026 at KubeCon Europe. The DurableAgent abstraction turns an AI agent into a Dapr Workflow: every Thought → Observation → Action step is checkpointed to Redis/Postgres. When a pod is OOMKilled, the workflow resumes exactly where it was interrupted — no context lost.

This is the biggest mindset shift: instead of trying to make LLM agents “stateless and idempotent” (nearly impossible with multi-turn reasoning), Dapr Agents accepts that agents are stateful by nature and provides infrastructure to manage that state durably.

DurableAgent: Real Crash Recovery

from dapr_agents import DurableAgent, tool

class ResearchAgent(DurableAgent):
    """An agent that survives pod restarts."""

    @tool
    async def search_web(self, query: str) -> str:
        """Search — results are persisted."""
        ...

    @tool
    async def analyze_document(self, url: str) -> dict:
        """Each tool call = 1 Dapr Activity = checkpointed."""
        ...

Every @tool call in a DurableAgent is handled by the Dapr Workflow engine as an Activity: the result is persisted after execution. When the workflow replays (due to a crash or restart), the orchestrator reads the cached result instead of re-calling the LLM or an external API.

Critical rule: LLM calls must live inside an Activity, never inside the Orchestrator. The Orchestrator must be deterministic — the same input must always produce the same output. LLMs are not. Breaking this rule will break the Dapr Workflow’s ability to replay correctly.

Virtual Actor Model: Automatic Scale, No Locks Required

Each DurableAgent instance is a Dapr Virtual Actor — an addressable stateful unit with a unique ID. The Dapr runtime handles:

  • Migration: when a pod is evicted, the actor migrates to another pod automatically
  • Turn-based concurrency: each actor processes one message at a time → no mutex, no race conditions
  • Lazy activation: actors are only loaded into memory when needed → thousands of concurrent agents consume no memory while idle

MCP Integration: Zero-Config Tool Discovery

Dapr v1.18 introduced the MCPServer Resource — the sidecar automatically discovers MCP tools and exposes them to agents:

apiVersion: dapr.io/v1alpha1
kind: MCPServer
metadata:
  name: internal-tools
spec:
  tools:
    - name: database-query
      rbac:
        allowedAgents: ["research-agent", "audit-agent"]
      piiRedaction: true          # automatically redacts PII from logs
      auditLog: true

No discovery logic needed in the agent. The sidecar handles everything.

Dapr v1.18 Security Upgrades (June 2026): Mandatory for Multi-Tenant

Before v1.18, any app in the same trust domain could schedule or terminate another app’s workflows. In an environment where multiple teams share a cluster, this was a serious security hole.

WorkflowAccessPolicy CRD fixes this:

apiVersion: dapr.io/v1alpha1
kind: WorkflowAccessPolicy
metadata:
  name: research-agent-policy
spec:
  targetWorkflows:
    - "research-agent-workflow"
  allowedCallers:
    - appId: "orchestration-service"
    - appId: "admin-dashboard"

Additional v1.18 upgrades:

  • Tamper Detection: workflow history is cryptographically signed → state tampering is detected immediately on load
  • Concurrency Limits: maxConcurrentWorkflows: 50 per namespace → prevents agent swarms from exhausting cluster resources

Go Backend Integration (Kratos + Dapr)

Answer-first: Wrap the Dapr Go SDK behind an interface in internal/data, inject via Wire into internal/biz. Business logic must never know Dapr exists — it only interacts through the Go interface.

// internal/biz/agent.go — biz layer has no knowledge of Dapr
type AgentWorkflow interface {
    StartResearch(ctx context.Context, req *ResearchRequest) (string, error)
    GetStatus(ctx context.Context, instanceID string) (*WorkflowStatus, error)
}

// internal/data/dapr_workflow.go — data layer implements with Dapr SDK
type daprWorkflowRepo struct {
    client dapr.Client
}

func (r *daprWorkflowRepo) StartResearch(ctx context.Context, req *ResearchRequest) (string, error) {
    resp, err := r.client.StartWorkflow(ctx, &dapr.StartWorkflowRequest{
        WorkflowComponent: "dapr",
        WorkflowName:      "research-agent-workflow",
        Input:             req,
    })
    return resp.InstanceID, err
}

Real-world gotchas (from production experience at tanhdev.com):

  1. Never create a “Kratos-native Dapr transport” — AI tools frequently hallucinate this package. It does not exist. Use the official dapr/go-sdk only.
  2. traceparent propagation: inject the header via context.Context throughout the entire call chain to maintain end-to-end tracing between the Kratos handler and Dapr activities.
  3. ETags for Actor state: use explicit ETag handling when updating Actor state concurrently to prevent lost updates.
  4. Mock the interface in unit tests: do not test the Dapr runtime inside biz layer tests — mock the interface only.

KEDA + vLLM: FinOps Autoscaling by Queue Depth

Answer-first: GPU pods don’t scale well on CPU/memory metrics — GPUs typically hit 100% immediately. KEDA ScaledObject uses vllm:num_requests_waiting from Prometheus to trigger scaling. Keep minReplicaCount: 1 for interactive agents — vLLM cold start takes 40–90 seconds if scaling from zero.

The final piece of the puzzle: GPU pod autoscaling must not rely on CPU/memory metrics. Use a KEDA ScaledObject driven by Prometheus metrics from vLLM:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-scaler
spec:
  scaleTargetRef:
    name: vllm-deployment
  minReplicaCount: 1             # keep 1 warm pod for interactive traffic
  maxReplicaCount: 10
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus:9090
      metricName: vllm_num_requests_waiting
      query: sum(vllm:num_requests_waiting)
      threshold: "5"             # scale when > 5 requests are waiting

Scale-to-zero with minReplicaCount: 0 is only appropriate for async/batch workloads. For interactive agents (chatbots, real-time analysis): vLLM cold start takes 40–90 seconds (weight loading + torch.compile + CUDA graph + KV cache warmup). Keep at least one warm pod.

FAQ

Dapr Agents vs LangGraph — which one should I use?

This is the wrong question because the two tools operate at different layers. LangGraph is the agent’s “brain” — state machine, reasoning loops, human-in-the-loop. Dapr Agents is the “body” — durability, fault tolerance, K8s scaling. The 2026 recommended pattern is hybrid: use DaprCheckpointer so LangGraph persists its graph state to a Dapr state store. If you’re already running Dapr on K8s and don’t need complex graph logic — a pure DurableAgent is sufficient.

Why can’t I put LLM calls directly in the Dapr Workflow Orchestrator?

Because the Orchestrator must be deterministic for Dapr to replay the workflow after a crash. LLM responses are non-deterministic. The correct pattern: LLM call → Dapr Activity. The Activity executes once, and its result is persisted. On replay, the orchestrator reads the cached result — it never re-calls the LLM. Violating this rule prevents the workflow from recovering correctly.

Is InferencePool actually necessary?

Not always. If you have a single vLLM deployment with uniform traffic, a Kubernetes Service is sufficient. InferencePool is needed when: multiple model versions or LoRA adapters are deployed; you want to reduce TTFT via KV cache-aware routing; GPU hotspotting appears (one pod saturated, others idle); or sustained GPU utilization exceeds 40%.

How do you detect a “runaway agent loop” in production?

Three layers of defense: (1) max_iterations in the agent framework — mandatory for every agent; (2) token-velocity circuit breaker at the AI Gateway — if an agent exceeds X tokens/minute within a single session, fire 429 immediately; (3) semantic similarity analysis — detect when consecutive agent outputs are nearly identical (logic loop). Dapr Workflow Concurrency Limits + Activity timeouts serve as a fourth and final backstop.

How does an AI Swarm differ from this architecture?

The previous radar entry (Tech Radar 03/07: Autonomous AI Swarms) focused on orchestration patterns at the application layer (OpenClaw, KEDA, CRIU). This article covers the infrastructure layer underneath — AI Gateway, smart routing, and the durable agent runtime. The two layers complement each other: swarm logic runs on top of the infrastructure foundation described here.