How ChatGPT and Large Language Models Work

ChatGPT is a product built around language models, not a single model operating in isolation. The visible conversation can involve input validation, policy checks, model selection, tool use, retrieval, state management, streaming, and monitoring. The underlying large language model performs a narrower operation: given the tokens so far, it estimates probabilities for the next token.

That distinction matters for infrastructure. A model explains much of the compute demand, but the surrounding product determines authentication, data access, reliability, and the final user experience.

What an AI model is

A model is a parameterized mathematical function learned from data. Training adjusts its parameters so its outputs better match an objective. For a generative language model, that objective commonly involves predicting missing or next tokens across a large corpus.

The resulting parameters are called weights. They encode statistical structure rather than a human-readable database of sentences. Inference loads those weights and applies them to a new sequence; it does not ordinarily retrain the model for each conversation.

“Large” primarily describes scale: modern models can contain many parameters and require substantial computation. Parameter count alone does not determine quality, speed, or memory use. Architecture, numerical precision, context length, serving software, hardware, and workload all matter.

Text becomes tokens

Models do not receive words directly. A tokenizer converts text into token identifiers drawn from a fixed vocabulary. A token might represent a whole short word, part of a longer word, punctuation, whitespace, or bytes. The same visible length can produce different token counts across languages, code, and tokenizers.

The server turns each identifier into a vector called an embedding and adds position information. These vectors enter a stack of transformer layers. Token count is operationally important because input tokens consume prefill computation and context capacity, while generated tokens require repeated decode steps.

What a transformer does

The transformer architecture introduced attention as a central mechanism. In simplified terms, each layer projects the current representations into queries, keys, and values. Attention scores compare queries with keys, then use the resulting weights to combine values. This lets each position build a context-sensitive representation from relevant earlier positions.

A transformer layer also contains feed-forward computation, normalization, residual connections, and other architecture-specific elements. Multiple attention heads can learn different relationships. Production models add variations, but the basic infrastructure consequence remains: attention and dense numerical operations create significant compute and memory traffic.

Causal language models mask future positions. During generation, a token can attend to the prompt and tokens already produced, but not to tokens that do not exist yet. This is why output generation is sequential even when much of the computation inside one step is parallel.

From hidden state to the next token

After the transformer processes the current sequence, the model produces logits over its vocabulary. A probability transformation and decoding policy select the next token. Greedy decoding chooses the highest-scoring option. Sampling can use temperature, top-p, or other controls to alter diversity.

The selected token is appended to the sequence and the process repeats until a stop condition, token limit, or cancellation. The response is therefore generated incrementally rather than retrieved as a completed paragraph. Streaming exposes those increments to the client.

Sampling also explains why identical prompts can produce different outputs when decoding is nondeterministic. Even nominally deterministic settings may not provide bit-for-bit reproducibility across hardware, kernels, model revisions, or service implementations. Production evaluations should tolerate appropriate variation and test behavior, not one memorized sentence.

Training, inference, retrieval, and tools are different

Training creates or updates weights and is usually far more resource-intensive than one inference request. Fine-tuning further updates some or all parameters for a narrower objective. Prompting supplies instructions without changing weights.

Retrieval-augmented generation searches an external corpus and inserts selected content into the model input. It can improve grounding, but authorization must be enforced by the application and data systems, not by a prompt. Tool use lets the surrounding system invoke defined functions or services, then return results to the model. The model proposes or selects actions; trusted code must validate permissions and arguments.

Keeping these mechanisms distinct prevents architecture mistakes. A current answer may require retrieval, not retraining. An account change requires an authorized tool, not a more persuasive prompt.

What happens in a production request

A simplified request path looks like this:

Authenticate the caller and establish tenant and policy context.
Validate and tokenize the input.
Select a model, route, and capacity pool.
Optionally retrieve permitted context or execute an approved tool loop.
Schedule the request with compatible work.
Run prompt prefill, then decode output tokens.
Stream or return the result and record safe telemetry.

Failures can occur at every boundary. The retriever can return stale content, the queue can dominate latency, a client can disconnect mid-generation, or a tool can have real-world side effects. “The model answered” is only one success condition.

The infrastructure takeaway

The transformer turns sequences into next-token probabilities. The serving system decides how weights are loaded, requests are batched, attention state is cached, work is split across accelerators, and traffic is routed. The application decides what the model may see and do.

That separation gives teams a useful debugging question: is the problem in model behavior, inference performance, or product orchestration? Measuring those layers independently is the first step toward operating them well.

Continue the series

Next, connect learned weights to physical capacity in How AI Models Are Stored and Why Hardware Limits Matter.