KV Cache, Prefix Caching, and Continuous Batching

Autoregressive generation would be wasteful if the model recomputed attention for the full conversation before every new token. The key-value cache avoids that repetition. Prefix caching can reuse some of that state across requests, while continuous batching lets a server schedule tokens from several requests together.

These mechanisms improve utilization, but they also compete for finite memory and introduce fairness and isolation decisions. A cache is not free capacity, and a larger batch is not automatically a better user experience.

What the key-value cache stores

Within each attention layer, the model derives key and value tensors for processed tokens. During decode, previous keys and values do not change, so the server retains them and computes only the state for the new token. The current query can attend to the cached history.

KV-cache memory grows with the number of resident tokens. Its exact cost depends on layers, key/value dimensions, cache precision, model architecture, and how work is partitioned. Context limit multiplied by request count is therefore an important capacity input even when model weights fit easily.

The cache belongs to active sequence state, not permanent model knowledge. It must be allocated, addressed, evicted, and released on completion or cancellation. A leaked or stranded allocation reduces the number of useful requests a worker can admit.

Paged cache-management approaches divide memory into blocks so sequences can grow without reserving one large contiguous region at admission time. This can reduce wasted space and fragmentation, but the serving engine still needs an overload policy when usable blocks run out.

Prefix caching reuses shared work

Many requests begin with identical tokens: a system instruction, tool schema, document template, or repeated conversation prefix. Automatic prefix caching records KV blocks for that prefix and reuses them when a later request has an exact compatible match.

The primary benefit is avoiding repeated prefill computation for cached tokens. It does not generally make the decode of newly generated tokens faster. The observed gain depends on prefix length, reuse frequency, cache residency, lookup overhead, and whether other work is limiting the service.

Design inputs to make stable prefixes reusable: place genuinely shared, unchanging material before request-specific content, and avoid injecting volatile timestamps or identifiers near the beginning. Do not contort prompts solely for cache hits if doing so harms correctness or policy clarity.

Track hit rate, reusable tokens, eviction rate, and prefill time by cache state. A headline cache-hit percentage can mislead when hits cover only short prefixes.

Cache reuse crosses a trust boundary

Shared prefix state can reveal information through timing even when the server never returns cached text. An attacker may infer that a particular prefix already exists by measuring a faster response. vLLM documents cache salting as a mitigation that limits which requests may reuse the same prefix blocks.

Derive the salt or cache namespace from authenticated, server-controlled security context. Do not trust a tenant ID supplied freely in the request body. Highly sensitive workloads may require stronger separation, including dedicated workers or disabled cross-request caching.

Cache isolation is only one control. Prompt data, telemetry, crash dumps, and model-server administration also need access boundaries. Document whether reuse occurs within a user, tenant, application, or global pool.

Static and continuous batching

Static batching waits for a group of inputs, pads or organizes them, executes the group, and returns results. It works well for predictable offline jobs, but variable output lengths make it awkward for interactive generation: a short request can wait behind the longest member.

Continuous batching changes membership as decoding proceeds. At a scheduling step, the server can add newly admitted sequences, advance active ones, and remove completed or cancelled work. This improves accelerator utilization under mixed traffic and avoids waiting for an entire fixed batch to finish.

Batch capacity still has limits. Larger batches can improve aggregate throughput while increasing queue time, per-user latency, or cache pressure. Long prompts may compete with decode tokens for compute. The scheduler needs policies for admission, prefill chunking where supported, preemption, and fairness.

Protect latency and fairness

Separate workload classes when their objectives conflict. An offline summarization queue should not consume every cache block needed by an interactive assistant. Establish per-class concurrency, token, and queue budgets, then reserve or weight capacity accordingly.

Tenant fairness cannot be inferred from global throughput. Track queue and completion latency by bounded workload class, and maintain server-side quotas for expensive inputs. Avoid tenant identifiers as high-cardinality metric labels; use controlled aggregation and detailed traces or logs with appropriate access.

Cancellation is a capacity feature. When a client disconnects or no longer needs an answer, propagate the signal through the gateway, scheduler, and worker. Measure cancellation latency and verify that cache memory returns to the pool.

Test the mechanisms independently

Use four controlled cases:

cold unique prompts to establish an uncached baseline;
repeated long prefixes to test reuse;
mixed-length concurrent requests to test continuous batching; and
competing workload classes to test fairness and overload.

Hold the model, engine, hardware, precision, and request distribution constant between comparisons. Report TTFT, TPOT or inter-token latency, input/output throughput, queue delay, cache occupancy, prefix tokens reused, errors, and rejections.

Then add adversarial operational cases: a cancellation storm, a very long prompt, cache eviction under pressure, and two authenticated tenants sending identical prefixes. Confirm that the chosen isolation policy prevents unintended cross-tenant reuse.

Common mistakes

Treating KV-cache capacity as a fixed count of requests rather than tokens.
Claiming prefix caching accelerates every stage of generation.
Sharing a global cache across tenants without a documented threat decision.
Maximizing batch size while ignoring tail latency and fairness.
Allowing offline traffic to exhaust interactive capacity.
Continuing generation after the caller has disconnected.
Comparing warm cached tests with cold uncached baselines without disclosure.

KV cache, prefix reuse, and continuous batching turn spare accelerator capacity into useful throughput only when memory, latency, and security policies are explicit. Their success should be measured at the service boundary, not inferred from a feature flag.

Continue the series

Next, scale beyond one serving unit in Distributed LLM Inference, Sharding, and Routing.