One block, repeated: attention, the MLP, and the residual stream

Last post the model was a box that turned token ids into scores. Here is what is in it. Two lines, no GPU required:

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B")
print(model)

Qwen3ForCausalLM(
  (model): Qwen3Model(
    (embed_tokens): Embedding(151936, 1024)
    (layers): ModuleList(
      (0-27): 28 x Qwen3DecoderLayer(
        (self_attn): Qwen3Attention(...)
        (mlp): Qwen3MLP(...)
        (input_layernorm): Qwen3RMSNorm(...)
        (post_attention_layernorm): Qwen3RMSNorm(...)
      )
    )
    (norm): Qwen3RMSNorm(...)
  )
  (lm_head): Linear(in_features=1024, out_features=151936, bias=False)
)

The interesting line is (0-27): 28 x Qwen3DecoderLayer. PyTorch prints repeated identical submodules once with a count, and here it has collapsed the entire model into a single block and the number 28.

That is the whole architecture. One block, described once, stacked. A model ten times this size is not ten times more complicated; it is more of the same block, each one a little wider. Whatever you understand about one of these you understand about all of them, which is the only reason a post like this is short.

Why it can repeat at all

A stack of identical blocks only works if the thing being passed between them never changes shape.

It does not. Each token is a vector of a fixed width, 1024 here, and it is 1024 wide when it enters the first block and 1024 wide when it leaves the last. Every block takes that vector and returns a vector of exactly the same size, which is what makes them interchangeable and stackable. The width is a property of the model, not of the layer.

That running vector, one per token, carried from the bottom of the stack to the top, is the residual stream. Almost everything else in the model is a function that operates on it.

It has a beginning and an end, and both are in the printout. embed_tokens is a lookup table: it turns each token id into the initial 1024-wide vector, which is the stream’s starting value. lm_head is a single linear layer that takes the final vector for a position and produces a score for every token in the vocabulary. That is where last post’s “a score for every position” comes from: the LM head is applied to every position’s final vector, and we kept one of them.

The block, drawn the usual way

Inside a block, two things happen to the stream. Attention, then the MLP. Each is preceded by a normalisation, which is why the printout has two RMSNorm modules per layer rather than one.

Almost every diagram you have seen draws it like the left-hand picture below. The input flows down through normalisation and attention, then down through normalisation and the MLP, and a residual connection loops around each one to add the original back in.

Now look at the right-hand picture. Nothing changed. Same modules, same additions, same order. The only difference is which line was drawn as the main one.

The stream is the road, not the shortcut

The conventional drawing makes the residual connection look like a repair: a shortcut added around the sublayers so that gradients survive the depth. That is historically how it arrived and it is not wrong, but as a mental model of what the model is doing it gets the emphasis backwards.

Redrawn, the picture says something different. There is a stream, one vector per token, running the full height of the model. Attention and the MLP are not stations the stream passes through. They are branches off it: each one reads a normalised copy of the stream, computes something, and adds its result back in. The stream itself is never replaced. It is only ever added to.

The consequence worth carrying forward is that every block is writing into the same object. Block 3 and block 27 are not passing a message down a chain of transformations. They are both editing one running vector, and each has to work with whatever the ones before it left there.

I found this reframe worth more than any single explanation of attention. Once the stream is the subject of the sentence, questions like “what does layer 12 do” stop being about a pipeline stage and start being about a contribution.

The one place tokens meet

The two branches differ in a way that matters more than anything else in this post.

The MLP sees one token at a time. Feed it a sequence of forty tokens and it does the same arithmetic on each of the forty vectors, independently, with no knowledge that the others exist. The tokens could be shuffled or processed on separate machines and the output would be the same.

Attention is the only place tokens see each other. It is the one operation in the block where a token’s output depends on other tokens. Everything else, the normalisation, the MLP, the embedding lookup, the LM head, is per-token.

That is worth saying flatly because so much follows from it. In this architecture, attention is the sole reason the model needs anything other than the token in front of it. It is why generating the five hundredth token costs more than generating the fifth, and it is where nearly everything the rest of this series deals with comes from. None of it is there for the MLP. The MLP does not care.

What this does not do yet

• Attention is still a box. Which tokens it looks at and how it weighs them is the next post.
• No shapes. I said 1024 wide and 28 blocks and left every other dimension alone.
• No numbers. Nothing here has been timed or measured. This post is a map, not a claim about cost.
• Nothing about position. Nothing so far explains how the model knows a token’s place in the sequence.

The next post opens the attention box, and the question it starts from is the one this post just handed it: if attention is where tokens meet, what exactly is each token asking the others for?