What inference actually is: the loop inside generate()

transformers.generate() is one line. It is also the only part of inference most people ever touch, which means the whole thing tends to arrive as a single opaque verb: the model generates.

This post takes that verb apart, but only at the coarsest resolution. We are going to keep the model itself closed, watch one token come out, and follow the path it travelled. Everything else in this series is a zoom into one segment of that path.

The model, for now, is a box

Here is the only thing you need to believe about the model in this post.

It takes a sequence of token ids and returns, for every position in that sequence, a score for every token in the vocabulary. Feed it five tokens and you get five predictions back, one per position, each one an opinion about what should come next at that point.

Inside the box, essentially, are matrix multiplications. Large ones, arranged in layers, with the model’s weights on one side and your tokens on the other. That is the level of detail we need today. (The next post opens the box: embeddings, attention, MLP, and the LM head.)

Two consequences follow immediately, and both matter later.

The first is that the model has no memory of being called before. It is a pure function of the token ids you hand it. Anything that looks like memory has to be built outside the box.

The second is that we asked for five predictions and we are only going to use one. The model scores every position; we take the last one and throw the rest away. Hold on to that, because it is going to turn out to be expensive.

Two ways to get the same tokens

Start with the one line.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Qwen/Qwen3-0.6B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16
).to("cuda").eval()

ids = tok("The capital of France is", return_tensors="pt").input_ids.to("cuda")

reference = model.generate(
    ids,
    max_new_tokens=20,
    do_sample=False,   # greedy: take the argmax, never sample
    temperature=None,  # Qwen3 ships sampling defaults; unset them
    top_p=None,
    top_k=None,
)

do_sample=False is what makes this greedy. generate() takes the argmax, exactly like the loop below. The three Nones are housekeeping: Qwen3’s generation_config.json ships with sampling parameters already set, and leaving them in place makes generate() warn that it is ignoring them.

Now the same thing without it.

@torch.no_grad()
def generate(model, ids, max_new_tokens):
    for _ in range(max_new_tokens):
        logits = model(ids, use_cache=False).logits   # [1, seq_len, vocab_size]
        next_id = logits[:, -1, :].argmax(dim=-1, keepdim=True)
        ids = torch.cat([ids, next_id], dim=-1)
    return ids

mine = generate(model, ids, max_new_tokens=20)

Four lines inside the loop. Run the model on everything so far. Take the scores at the last position. Pick the highest one. Stick it on the end. Do it again.

That is the entire generation loop. Not a simplification of it, not a teaching version of it: the loop in a production engine is this loop, with a great deal of machinery wrapped around it so that it can run for many requests at once without falling over.

n = reference.shape[-1]
assert torch.equal(mine[:, :n], reference)

Everything generate() produced, we produced too, in the same order.

Note that this is a prefix and not an equality, and the reason is a gap we have not closed yet. generate() stops when the model emits an end-of-sequence token. Our loop does not: it runs the count it was given and keeps going past the end. If the model never emits end-of-sequence inside the budget, the two sequences are identical and this check is an exact one.

It is also a check on token ids, not on bits. Two implementations can disagree in the last places of a float and still pick the same token, and when you are always taking the argmax that is the thing worth guarding.

Note what we did not pass. use_cache=False says: recompute the whole sequence from scratch on every step. Hugging Face’s generate() does not do that. It keeps intermediate state around and reuses it, and it still lands on the same tokens as our version, which is the tell. That state is an optimization, not a change of meaning. We will build it ourselves in a few posts, once the cost of not having it is something you have seen rather than been told.

Where the multiplication happens

You can run all of the above on a CPU. It will work and it will be slow, and the reason is worth a paragraph because it shapes everything after.

The box is matrix multiplications. That is a workload with enormous parallelism and very little branching, which is the shape of problem an accelerator exists for. A GPU is, at the resolution we need today, two things: memory (HBM, where the weights and your activations live) and compute units (a great many of them, which do the arithmetic). Nothing is computed in HBM. Data has to be moved out of it into the compute units, multiplied, and written back.

There is a second round trip stacked on top of the first. The GPU does not decide what to run. The CPU does, and it sends the GPU work to do, waits, and copies the result back. Your Python loop lives on the CPU. The multiplications live on the GPU. One step of generation is a full circuit through both.

Two costs are now visible on the picture, and neither is the multiplication itself. Moving data in and out of HBM costs time. Sending the GPU instructions from the CPU costs time. Later in this series, both of those turn out to matter more than the arithmetic does, and there are whole mechanisms whose only job is to shrink them.

That loop, repeated, is inference

Put it together.

The prompt goes in. The model returns scores for every position. We take the last one, pick a token, append it, and hand the longer sequence back. Each pass is a circuit out to the GPU and back. One token per circuit, until we decide to stop.

That is inference. Every mechanism in this series exists because doing this naively, for one user, is a very different problem from doing it for a thousand at once.

What this does not do yet

Being explicit about the gaps is how each post here ends, because the gaps are the next posts.

• No cache. Every step recomputes the entire sequence, including all the work it did on the previous step. Generating the twentieth token redoes the first nineteen.
• No stopping. The loop runs a fixed count. It does not notice an end-of-sequence token, and it has no notion of a stop string. That is why the check above compares a prefix.
• No sampling. argmax takes the single highest-scoring token. Temperature, top-k and top-p do not exist here.
• One request. There is no second sequence anywhere in this code, and no notion of what would happen if one arrived while this loop was running.
• The box is still closed. We have said “matrix multiplications” and left it there.

The first item is the loudest, so it goes first. Generating twenty tokens does roughly twenty times the work of generating one, and it did not have to. The next posts open the box, and then ask what it would take to stop throwing that work away.