Series
How LLM inference actually works
Each post answers one question, shows a minimal implementation that runs, and checks it against vLLM, SGLang, and TensorRT-LLM.
- 00Inference is no longer what happens after trainingWhy the word inference now points in the wrong direction, and what this series covers.
- 01What inference actually is: the loop inside generate()One prompt, one token, and the round trip that produces it. The model stays a black box for now.
- 02One block, repeated: attention, the MLP, and the residual streamThe same block stacked over and over, and the residual stream that everything else reads from and writes back into.
- 03What each token asks the others for: queries, keys and valuesSelf-attention as a recommendation problem, and then the four matrices that make it happen.
- 04The part that never changes: the KV cacheOne row decides the next token, and the loop computes a whole square to get it. Causality is why the rest was already correct.
- 05The first step and all the others: prefill and decodeThe cache split generation into two kinds of work. Pricing them turns out to need only one number, and it is not the one people usually reach for.