Running llm-d on Kubernetes in Production

llm-d is a Kubernetes-oriented distributed inference serving stack. Its value is not that every LLM deployment needs another platform layer. It is that teams operating sufficiently demanding workloads can combine model serving with inference-aware routing, cache management, parallel workers, and optional prefill/decode disaggregation through documented patterns.

A production decision should start with a measured workload and the simplest architecture that meets it. Use llm-d when its capabilities address a demonstrated bottleneck or operating requirement, not merely because the project is available.

Begin with a Well-Lit Path

llm-d publishes Well-Lit Paths as curated, end-to-end deployment and benchmarking patterns. They are useful starting configurations, not universal performance guarantees. Select the path closest to the model, hardware, scale, and objective you can actually test, and pin the documentation and release version used.

Current llm-d artifacts are modular. The project documents a router Helm chart, model-server Kustomize manifests, and additional component packaging. Avoid describing the entire system as one Helm installation. Record chart versions, image digests, manifests, model revision, and values together as a release manifest.

Before adding advanced routing, establish a baseline that can load the approved model, serve a bounded request, emit metrics, reject overload safely, and recover from a worker failure.

Route using inference state

llm-d's routing approach can incorporate signals beyond ordinary round-robin, including load and prefix-cache locality. This addresses a real LLM characteristic: equal request counts can represent very different active-token work.

Treat routing policy as testable production code. Define what happens when metrics are delayed, a worker reports inconsistent state, caches are cold, or the preferred replica has insufficient capacity. Use authenticated tenant context when cache namespaces affect placement, and never let affinity cross an isolation boundary.

Round-robin remains a useful control case. Compare an inference-aware policy with the baseline under the same prompt lengths, output lengths, concurrency, and cache state. Any reported advantage belongs to that documented configuration; it is not a general llm-d performance promise.

Consider prefill/decode disaggregation carefully

llm-d supports patterns that separate prefill and decode onto different model-server instances. This can let teams tune capacity for the distinct phases and route work between specialized pools. It also adds network transfer, coordination, more failure modes, and a harder rollout.

Adopt disaggregation only when measurements show that a combined worker pool cannot meet the objective efficiently and the added boundary improves the target under realistic load. Test what happens when either pool saturates or disappears, and ensure tracing follows a request across the handoff.

Choose Kubernetes controllers by semantics

Use a Deployment for interchangeable stateless replicas and controlled rolling updates. Use a StatefulSet when a component genuinely needs stable network identity, stable storage association, or ordered lifecycle. A model server does not require a StatefulSet simply because it holds an in-memory KV cache; that cache is normally ephemeral request state.

A StatefulSet also does not replicate, back up, or repair application data. If a routing or metadata component has durable state, document its ownership, recovery point, recovery time, and restore test. Prefer managed state where it reduces risk and fits the architecture.

GPU workloads need explicit resource requests, compatible node labels or device scheduling, startup and readiness probes, graceful termination, and disruption planning. Readiness should remain false until weights are loaded and a meaningful smoke check passes. Maintain enough capacity to load a replacement without dropping below the service objective.

Operate the complete request path

Instrument the gateway, router, scheduler, and model server with correlated identifiers and traces. Track queue time, TTFT, TPOT or inter-token latency, tokens, cache occupancy, prefix reuse, preemption, admission rejection, GPU memory, worker readiness, and errors. Keep prompts and customer data out of ordinary metric labels and default logs.

Tie alerts to customer-visible objectives and actionable capacity conditions. GPU utilization alone cannot tell whether users are waiting. A high value may be healthy batching; a low value may coexist with a blocked queue elsewhere.

Use progressive rollout, explicit rollback criteria, and a compatibility check across engine, model, tokenizer, router, and manifests. Retrying a partially streamed generation is not equivalent to retrying an idempotent database read, so make interruption behavior visible to the application.

A scoped hands-on lab path

These are scoped exercises, not executed, benchmarked, or verified Tier9AI results. Use a permitted small model and available hardware, follow the current llm-d release documentation, and record every version.

Lab 1 — Establish a single-server baseline

Deploy one supported model server, apply input and output limits, and run cold and warm requests. Capture memory, queue delay, TTFT, TPOT, and output throughput across short and long prompts. Confirm cancellation releases capacity.

Lab 2 — Package it on Kubernetes

Create an isolated test namespace, declare GPU resources, configure readiness around model loading, and deploy immutable artifacts. Run a rolling update and simulated pod loss. Explain whether Deployment or StatefulSet matches the component semantics.

Lab 3 — Add llm-d routing

Start from the closest Well-Lit Path and compare round-robin with the documented inference-aware policy. Test uniform traffic, mixed prompt lengths, repeated permitted prefixes, cold caches, and one unavailable worker. Preserve identical workloads between comparisons.

Lab 4 — Exercise operations and isolation

Add traces and an SLO-oriented dashboard. Trigger overload, cancellation, and a controlled rollout. Use two test tenants and verify cache namespacing prevents cross-tenant prefix reuse. Produce a runbook with detection, mitigation, rollback, and evidence collection.

Optional disaggregation should be a separate experiment after the combined path is understood. Measure the inter-stage transfer and failure behavior; do not assume it improves the chosen objective.

Series recap

The sequence now connects the full serving path:

Models store learned tensors; runtime adds cache and operational memory.
GPU sizing begins with weights, peak active tokens, and a service objective.
PyTorch supplies model execution primitives; vLLM adds specialized serving capabilities.
Prefill shapes first-token delay, while decode shapes generation cadence.
KV caching avoids repeated attention work; prefix reuse and batching trade memory for efficiency.
Tensor, pipeline, data, and expert parallelism solve different distribution problems.
Inference-aware routing can use workload and cache signals without making round-robin universally obsolete.
Kubernetes supplies workload lifecycle; llm-d supplies modular patterns for more advanced distributed serving.

The operating principle is consistent: state the constraint, measure the user-visible objective, choose the smallest architecture that meets it, and test failure and isolation before calling it production-ready.

Continue the series

Revisit the distribution choices behind this deployment in Distributed LLM Inference, Sharding, and Routing.