Prefill, Decode, TTFT, and TPOT Explained

An LLM response has two computational phases with different performance characteristics. During prefill, the model processes the input tokens and creates attention state. During decode, it generates new tokens one at a time while reusing that state.

Users experience both phases through the serving system around them. Time to first token and time per output token are therefore useful service measurements—but they should not be treated as perfect synonyms for raw prefill and decode compute time.

Prefill processes the prompt

After tokenization and scheduling, prefill runs the prompt through the model. Tokens within the supplied sequence can be processed with substantial parallelism under the causal attention mask. Longer prompts generally require more computation and create more key-value-cache state.

Prefill behavior depends on model architecture, prompt length, batch composition, accelerator, kernels, and whether some prefix state can be reused. A retrieval-heavy request with many documents can impose a much larger prefill cost than a short chat turn even if both produce the same number of output tokens.

This gives application teams a direct optimization lever: send only context that is authorized and useful. More context is not automatically better. It can increase latency and cost while distracting the model.

Decode generates sequentially

Once prefill produces the state for the input, decode selects the next token. The new token is passed through the model, its key and value state is added to the cache, and another token is selected. This loop continues until a stop condition, output limit, or cancellation.

Each step handles relatively little new sequence data but repeatedly uses model weights. Decode can therefore be sensitive to memory bandwidth and scheduler behavior. It is sequential at the request level: token 51 depends on the state that includes token 50. A server gains throughput by batching decode steps from several active sequences, not by generating all tokens of one response simultaneously.

Long output limits tie up cache memory and scheduling capacity. Enforce bounded generation and ensure cancellation promptly releases resources when the client leaves.

TTFT is a service metric

Time to first token, or TTFT, measures elapsed time from a defined request start until the first output token is available or received. The exact boundary must be documented. A client-observed TTFT can include network transit, gateway work, authentication, queueing, tokenization, retrieval, prefix-cache lookup, prefill, first-token sampling, and streaming overhead.

That is why “TTFT equals prefill time” is too simple. Prefill may dominate for a long prompt, but queue delay may dominate under load. Breaking TTFT into spans makes the metric actionable:

gateway and policy time;
retrieval or tool preparation;
scheduler queue time;
tokenization and engine preprocessing;
model execution to first token; and
stream delivery to the caller.

Report TTFT percentiles by prompt-length bucket and workload class. One average hides the users most likely to notice a problem.

TPOT describes generation cadence

Time per output token, or TPOT, commonly estimates the average time between generated output tokens after the first token. In vLLM's benchmark terminology, it is derived from end-to-end latency minus TTFT, divided by the number of output tokens after the first.

Terminology is not fully standardized. Some tools report inter-token latency for every gap, while others report an average. State the formula, observation point, streaming behavior, and treatment of one-token responses before comparing results.

Users often perceive a low TTFT followed by a steady stream as more responsive than waiting for a complete answer. But optimizing only TTFT can starve decode work, and optimizing only aggregate token throughput can leave individual conversations choppy. Service objectives should balance both phases.

Throughput completes the picture

Requests per second is incomplete when prompt and output lengths vary. Also track input tokens per second, output tokens per second, successful requests, active sequences, and queue time. For offline work, total completion throughput may matter more than streaming cadence. For voice or interactive interfaces, tail TTFT and inter-token stalls may dominate.

A good load test uses a representative distribution of input length, output length, arrival pattern, and concurrency. Synthetic requests that all have identical lengths can make batching unrealistically efficient. Include cancellations, overload, cold starts, and a replica loss.

Diagnose by symptom

If TTFT rises while TPOT stays stable, inspect queue pressure, retrieval, tokenization, long prompts, and prefill scheduling. If TTFT is stable but output becomes slow or uneven, inspect decode batch formation, memory bandwidth, preemption, cache pressure, and competing workloads.

If both worsen with concurrency, the service may be saturated. Determine whether compute, memory, cache capacity, interconnect, or an upstream dependency is limiting it before adding replicas. More replicas cannot fix a shared gateway queue or insufficient network capacity.

If only long prompts degrade, bucket metrics by token length and examine admission policy. If one tenant affects others, inspect fairness, per-tenant concurrency, and request-size budgets.

A minimal measurement contract

For every performance result, record:

model, tokenizer, precision, and engine versions;
hardware, device count, and interconnect;
input/output length distributions and concurrency;
cache state and warm-up method;
decoding settings and streaming boundaries;
metric formulas and percentile window; and
errors, rejections, cancellations, and incomplete responses.

These details turn a number into reproducible evidence. Without them, “50 milliseconds TTFT” is a marketing fragment, not a capacity result.

Prefill and decode provide the mental model. TTFT, TPOT, throughput, and queue delay reveal how that computation becomes a user experience. The next chapter adds the mechanisms that let a server reuse work and share a GPU across many sequences.

Continue the series

Explore those reuse mechanisms in KV Cache, Prefix Caching, and Continuous Batching.