Distributed LLM Inference, Sharding, and Routing

Distributed inference solves two different problems: fitting one model execution across accelerators and serving more independent work with replicas. Calling both “sharding” hides important tradeoffs. A reliable design names the parallelism dimension, its communication cost, and the failure boundary it creates.

Routing then decides where each request belongs. Round-robin remains useful for interchangeable, similarly loaded replicas, but LLM requests are often neither uniform nor stateless. Prompt length, expected generation, active cache memory, batch composition, and prefix locality can make equal request counts represent unequal work.

Start with the reason to distribute

Use multiple devices only after stating the constraint:

The weights or runtime state do not fit on one accelerator.
One replica cannot meet a latency objective.
More replicas are needed for throughput or availability.
A mixture-of-experts architecture requires expert placement.
Prefill and decode need distinct capacity pools.

Each reason suggests a different topology. Adding devices without identifying the constraint can increase communication, operational complexity, and cost without improving the target metric.

Tensor parallelism

Tensor parallelism partitions operations and model parameters within layers across devices. Workers cooperate on the same request and exchange partial results, commonly at frequent layer boundaries. It can make a model fit and aggregate compute, but it depends heavily on fast, predictable device interconnects.

Treat the tensor-parallel group as one serving unit. If one member fails, the group generally cannot complete its work. Placement should keep the group close enough for the communication pattern, and monitoring should surface both individual-device health and group health.

Pipeline parallelism

Pipeline parallelism assigns different layer ranges to different stages. Activations flow from one stage to the next. This can suit hardware arrangements where tensor partitioning is inefficient or a model must span nodes, but pipeline bubbles and uneven stage times can reduce utilization.

Balance stages from measured computation and memory rather than layer count alone. The slowest stage limits the pipeline, and a failure in any stage disrupts the request path.

Tensor and pipeline parallelism can be combined, creating a multidimensional worker group. That flexibility increases the importance of explicit placement, topology discovery, startup coordination, and rollout testing.

Data and expert parallelism

Data parallelism runs multiple model replicas and routes separate requests or batches to them. It is the usual scale-out mechanism once one replica can serve successfully. Replicas may still differ in active-token load and prefix-cache contents, so the routing layer benefits from more than a connection count.

Expert parallelism distributes experts in a mixture-of-experts model. Each token may be routed to a subset of experts, producing a communication pattern distinct from dense models. Do not present expert parallelism as a universal technique; it applies to architectures that contain routable experts and engines that support the layout.

Why simple balancing can become inefficient

“Traditional load balancing breaks for LLMs” is too absolute. Basic balancing still provides a valid baseline and may be sufficient for homogeneous, low-variance traffic. The limitation is that request count is a weak proxy for remaining work.

Imagine two replicas with three requests each. One holds short classification prompts nearing completion. The other holds long-context conversations beginning thousands of output tokens. Round-robin sees equality; users do not.

An inference-aware router can consider:

queued and active tokens or another engine load signal;
available KV-cache capacity;
prompt length and bounded output request;
prefix-cache affinity;
model, adapter, or hardware compatibility;
workload class and tenant budget; and
replica health and recent latency.

These inputs must remain bounded and trustworthy. A client-supplied output limit is useful for admission but is not a perfect prediction of actual work.

Cache affinity versus balance

Routing a repeated prefix to the worker that already caches it can reduce prefill work. That same decision may overload one replica while others sit idle. Good routing treats cache locality as one score among load, capacity, and service objectives—not an unconditional rule.

Cross-tenant cache isolation still applies. Affinity must never override an authenticated cache namespace or dedicated-capacity boundary. When caches are cold after a rollout, routing behavior and latency may differ from steady state; measure both.

Design the failure domain

Distributed workers create coordinated failure modes. A node loss can remove several tensor-parallel groups or a whole model pool if placement is careless. Use topology-aware scheduling and disruption controls, but test actual failure rather than relying on configuration intent.

Define how the gateway handles an interrupted stream. Retrying generation can duplicate cost and may return different text. A safe policy may expose a clear interruption, retry only before output begins, or restart with application-visible semantics. Never silently imply exactly-once generation.

Rollouts also need group awareness. Mixing incompatible engine, model, tokenizer, or communication-library versions inside one group can fail unpredictably. Create a complete new group, verify readiness and model revision, shift traffic gradually, and retain rollback capacity.

Evaluate a distributed design

Compare against the simplest viable single-device or single-group baseline. Hold the workload constant and report end-to-end and per-phase latency, throughput, memory, interconnect utilization, queueing, cache reuse, errors, and cost assumptions. Include a device or node failure and a rolling update.

A benchmark percentage is not portable without the exact model, hardware, parallel dimensions, engine release, request mix, and objective. Prefer a reproducible configuration to a universal claim.

Distributed inference is successful when it meets a stated service target with understandable failure behavior. Device count alone is not an outcome.

Continue the series

Apply these decisions on Kubernetes in Running llm-d on Kubernetes in Production.