GPU Sizing for LLM Inference Without Guesswork

GPU sizing for language-model inference is not a lookup table from parameter count to accelerator count. The same model can need very different capacity for a single offline job, an interactive assistant, or a high-concurrency API with long documents.

A useful estimate begins with memory, then tests whether the chosen hardware can meet a service objective. “The model fits” is necessary, but it is not the same as “the service performs.”

Read GPU specifications in three dimensions

Compute units and arithmetic throughput describe how much supported numerical work a device can perform. Vendors may report CUDA cores, tensor cores, or peak operations in particular formats. These figures are not directly comparable across architectures or useful without compatible kernels. Model shape, precision, serving engine, and utilization determine realized throughput.

VRAM capacity sets a hard boundary around weights, attention cache, temporary workspaces, and runtime overhead. Multiple GPUs provide more aggregate memory only when the engine can partition the model and associated state appropriately.

Memory bandwidth measures how quickly data can move between accelerator memory and compute units. Autoregressive decoding often performs relatively little work per weight loaded, making bandwidth an important limiter. Prompt processing can expose a different compute profile because many input tokens are handled in parallel.

Interconnect bandwidth matters when work spans devices or nodes. Moving activations or partial results over a slow link can erase the benefit of additional compute.

A three-number planning heuristic

There is no official industry standard called “the three GPU sizing numbers.” The following is a practical Tier9AI planning heuristic that keeps the first estimate focused.

1. Deployed weight footprint

Calculate parameters multiplied by bytes per stored parameter, then add quantization metadata and measured runtime overhead. A nominal seven-billion-parameter model stored at two bytes per parameter starts near 14 GB before overhead. Do not use file size alone: compression, sharded formats, and loader behavior can make it misleading.

If the result approaches available memory, there may be no room for active requests or safe rollout. Plan for the exact model revision, numerical format, and engine rather than a model-family label.

2. Peak active-token load

Estimate the maximum tokens resident across concurrent sequences: input tokens already processed plus generated tokens still active. The key-value cache cost per token depends on model architecture, cache precision, and parallel layout, so obtain it from the serving engine or measure it.

Use a distribution, not only an average. A small number of very long requests can occupy disproportionate memory and delay other users. Define maximum context, maximum generation length, concurrency or admission limits, and the share of requests expected near each range.

3. Service performance target

Specify the objective the hardware must meet: for example, a percentile time to first token, time per output token, request throughput, or batch completion window under a stated workload. Interactive and offline systems optimize differently. Include the prompt-length distribution, output-length distribution, concurrency, cache-hit assumptions, and expected availability headroom.

This third number turns a memory calculation into a capacity decision. Two GPU types that both fit the model can deliver materially different latency and throughput.

Build a memory budget

Treat device memory like a budget with named allocations:

model weights and quantization metadata;
key-value cache for admitted sequences;
activations and temporary operator workspaces;
serving-engine and communication buffers; and
safety headroom for fragmentation, variation, and rollout.

Measure peak allocated and reserved memory during a representative test. An out-of-memory event can occur because of fragmentation or a temporary peak even when a spreadsheet suggests a few bytes remain. Running every device at its theoretical limit also makes graceful degradation and rolling replacement difficult.

If one device cannot hold the workload, options include a smaller model, lower precision, shorter context, stricter concurrency, cache quantization, CPU offload where latency permits, or model parallelism. Each changes quality, speed, complexity, or cost and should be tested rather than assumed.

Benchmark the workload you will operate

A useful benchmark fixes the model revision, tokenizer, engine version, accelerator and interconnect, precision, prompt distribution, output distribution, concurrency, and decoding configuration. Warm-up behavior and prefix reuse should be reported separately.

Measure at least:

successful request rate and token throughput;
percentile queue delay and time to first token;
percentile time per output token or inter-token latency;
GPU memory, utilization, and cache occupancy;
rejected, cancelled, and failed requests; and
performance during a worker loss or rolling update.

An isolated maximum-throughput result cannot predict interactive latency. Likewise, a single-user latency test cannot size a shared service.

Convert tests into deployment capacity

Suppose one tested replica meets the target at a defined arrival rate and workload mix. Do not divide forecast traffic by that maximum and stop. Add availability capacity for a failed replica, account for peaks and growth, and choose an autoscaling signal that represents work—such as queue pressure or active tokens—rather than relying only on GPU utilization.

Cold model loading can take long enough that reactive autoscaling arrives late. Minimum warm capacity, admission control, and load shedding may be necessary. A smaller fallback model can protect availability, but only if the product explicitly accepts the quality difference.

Common sizing mistakes

Comparing only advertised compute-core counts across GPU generations.
Treating parameter count or checkpoint size as total runtime memory.
Testing one short prompt and extrapolating to long-context concurrency.
Reporting average latency while tail requests violate the objective.
Assuming perfect prefix-cache reuse or batch formation.
Filling every byte of VRAM without failure or rollout headroom.
Adding GPUs without measuring inter-device communication cost.

The right answer is not the largest GPU. It is the smallest operational configuration that meets a clearly stated quality, latency, throughput, isolation, and availability target with defensible headroom.

Continue the series

Next, turn an approved artifact into a managed endpoint in From PyTorch Weights to a vLLM Model Server.