Inference is no longer what happens after training

Training runs end. Inference doesn’t. And now it runs inside training, too.

For most of machine learning’s history, inference named a phase. You trained a model; when training finished, you deployed it; the deployed thing did inference. The word carries the ordering in it: you infer from what was learned. Training was where the hard problems lived, in the data and the GPUs and the research. Inference was what happened afterward, in production, downstream of every interesting decision.

That frame survived the move to LLMs mostly intact. The models got larger and the serving bills got frightening, but the shape stayed the same: train, then serve.

The frame is now wrong in two separate ways. The first has been true for a while and is widely understood. The second is recent enough that a lot of very good engineers have not updated for it yet.

One: training runs end, inference doesn’t

A training run is a finite object. It has a start, a wall-clock duration, a checkpoint, and a moment where someone says it is done. However expensive it was, you paid for it once.

Inference has no such shape. A model that ships is a model that runs: every request, every user, every day, until it is replaced. There is no point at which the serving cost stops accruing. Over a deployed model’s life, the compute spent answering requests dwarfs the compute spent producing it, and the gap widens the more successful the model is.

This is the part most people already accept, at least abstractly. What it means concretely is worth stating plainly. A 20% improvement in serving efficiency is not a 20% improvement in a line item. It is a permanent multiplier on everything the model will ever do. It changes how many users you can serve on the hardware you have, what you can afford to offer for free, and whether a product is viable at all.

That alone would justify understanding inference properly. But it is the older argument, and it is not the reason I am starting this series now.

Two: inference now runs inside training

Here is what changed.

Serving users is no longer the only place a model gets run. Two other places have become at least as important, and both of them sit upstream of a trained model rather than downstream of one.

Synthetic data generation. A meaningful share of the data modern models are trained on was produced by other models. Generating it is inference, run in bulk: millions or billions of forward passes before a single gradient step of the run that consumes them. This is not a side activity. For some capabilities it is the primary way training data comes into existence.

Reinforcement learning in post-training. This is the sharper case. In an RL post-training loop, the model generates candidate responses, those responses are scored, and the scores drive a gradient update. The generation step is not inference-like. It is inference, frequently the same library, running the same scheduler, holding the same KV cache.

Look at how the current open-source RL training frameworks are actually built. verl pairs a training backend (Megatron-LM or FSDP) with an inference engine (vLLM or SGLang) for the rollout phase. slime is built natively on SGLang for the same reason. These are not integrations of convenience. The rollout engine is a load-bearing component of the training system, and its throughput is the training loop’s throughput.

Which means a post-training cluster is an inference cluster that happens to also compute gradients. If rollout generation is slow, the training run is slow. If the KV cache thrashes, the gradient step waits. The questions that used to belong to the serving team (how do we batch this, where does the memory go, what is the tail latency) now sit in the middle of the training loop.

So the ordering the word “inference” implies has broken down. It is not the thing that comes after. It is a component that appears in several places, and one of those places is inside the process that produces the model in the first place.

Which is why this series starts with the engine

Post-training is, by a wide margin, the most consequential topic in LLM engineering right now. Capability gains that used to come from scaling pretraining increasingly come from what happens after it, and the field’s attention has moved accordingly.

This series is not about post-training.

It is about the inference engine, and it starts there precisely because the engine is the part of the post-training pipeline that is doing the work. You cannot reason about a rollout budget without knowing what a KV cache costs. You cannot decide how many generations per prompt you can afford without understanding how a scheduler shares a GPU between requests of different lengths. You cannot debug a training run that is bottlenecked on generation if the generation step is a black box labeled “the rollout engine.”

The engine is the shared substrate underneath serving, synthetic data, and RL. Understanding it once pays in all three. Starting anywhere else means learning it later, under worse conditions, while something is on fire.

How this series works

The problem with inference engines is not that they are conceptually deep. It is that they arrive all at once.

Open vLLM’s scheduler and you find a few hundred lines that make sense only if you already know which failure each branch exists to prevent. Read it top-down and you learn the shape of the code without learning why the code has that shape. That is a frustrating way to spend a weekend, and it is how most people bounce off.

So this series goes the other way. One piece at a time.

Each post introduces exactly one mechanism, and it starts with what breaks without it. Before continuous batching, there is a GPU sitting idle while one request finishes. Before paged memory, there is a request that reserved space for a length it never reached. The mechanism arrives as the answer to a problem you have already felt, in code, in the post before it.

Each post carries working code: small enough to read in one sitting, real enough to run.

And the code is not my personal design. That is the part I want to be explicit about, because a hand-rolled engine written to illustrate a blog post is worth very little. Every piece is built by working against how the real engines do it: vLLM, SGLang, and TensorRT-LLM. When I introduce a block table, it is because that is the structure a production engine uses, and the post shows you where. When the minimal version diverges from the real one, the post says so, and says what the real version is buying with the extra complexity.

Where the three engines disagree with each other is often the most interesting part. They solve the same problems with genuinely different designs, and those disagreements are where the actual engineering trade-offs live. Those get their own posts.

The goal is not that you finish able to recite what PagedAttention is. It is that you can open one of these codebases and read it, because you have already built a smaller version of the thing you are looking at.

The roadmap

Three tiers, in order.

Foundations: what actually runs. What happens between a prompt and a token. What a GPU is doing during decode, and why it spends so much of that time waiting. What is inside the model: embeddings, attention, MLP, the LM head. This tier assumes you have used an LLM and not much else.

The engine: where the complexity comes from. Why a cache exists and what it costs in bytes. Prefill and decode, and why the same model behaves like two different workloads. Batching, and what exactly is continuous about continuous batching. Scheduling: who runs this step, and who waits. Paged memory, block tables, prefix caching, preemption. Then latency itself: time to first token against time per output token, and why improving one usually costs you the other.

Optimization: making it faster. Quantization, and what precision actually buys and costs. Speculative decoding. CUDA graphs and the cost of launching kernels. Multi-GPU inference: tensor parallelism, pipeline parallelism, and what breaks first when one GPU is no longer enough. Disaggregated serving, where prefill and decode stop sharing hardware.

The first post starts the way all of them do, by taking something away. transformers.generate() is one line. Delete it and you have to write out everything that line was doing for you, and then decide how you will know whether you got it right. That second part, not the code, is what the rest of the series rests on.