What each token asks the others for: queries, keys and values

Last post ended on a question. Attention is the one place in the block where a token’s output depends on other tokens. So what is a token actually asking them for?

There is a well-worn problem with the same shape, and starting from it makes the machinery almost inevitable.

A recommendation problem

You have people, you have movies, and you want to know which movie a person would like.

The standard move is to describe both sides as vectors and let a dot product do the work. A person becomes a vector of what they are drawn to. A movie becomes a vector of what it offers. Multiply them and you get a number, and that number is a predicted rating. Nobody writes down the rule “people who like this also like that”. It falls out of two learned descriptions meeting in the same space.

Attention is that, with three pieces instead of two.

• The query is the person. What am I looking for right now?
• The key is the poster. Genre, cast, tone, the thing you glance at to decide whether to bother.
• The value is the synopsis. What the film is actually about, which is what you carry away.

The score is the dot product of a query and a key: how much this person would like this poster. It is a predicted rating and nothing more. It decides how much of a film you get, not what the film is. That job belongs to the synopsis, and the synopsis is the same no matter who is asking.

Getting those two apart is most of the difficulty with attention. The score is a number. The value is content. They come from different places and they answer different questions.

Everyone is a person, and also a movie

Here is where the analogy turns, and where the word self comes from.

In a recommender, people and movies are separate populations. In self-attention they are the same population. Every token in the sentence writes its own poster and its own synopsis, and every token also walks in as a person with something it is looking for. All of them at once.

So the sentence is a room where everyone is simultaneously browsing and being browsed. Each token reads every poster, scores them, and takes a blend of the synopses in proportion to those scores.

That blend is added into the token’s residual stream, which is the picture from last post: a branch that reads the stream, computes something, and adds its result back in. What attention adds is whatever it went and collected from the rest of the sentence.

Nobody hands out posters

A movie does not arrive with a poster attached. Somebody has to design one, and designing a poster is a different job from writing a synopsis, which is a different job again from working out what a person is in the mood for.

That is why there are three projections, and why all three are learned.

A token enters attention as a single vector, the one it took off the residual stream. From that one vector the model produces three different summaries of it, using three different learned matrices:

• W_q makes the query: what this token wants from the others.
• W_k makes the key: how this token would like to be found.
• W_v makes the value: what this token is prepared to hand over.

Same source, three summaries, three jobs. If one vector were used for all three, a token’s notion of what it is looking for would be forced to equal its notion of what it offers, and those are not the same thing. The separation is the whole point.

It also makes attention asymmetric, which is easy to miss. How much A attends to B is q_A · k_B. How much B attends to A is q_B · k_A. Different matrices, different numbers. Being interested in someone is not the same as being interesting to them.

Three multiplications

With q, k and v in hand, the rest is three lines.

scores  = q @ k.transpose(-2, -1) / math.sqrt(head_dim)
weights = scores.softmax(dim=-1)
out     = weights @ v

Each line is worth taking on its own, because each one produces a different kind of object.

q @ k.transpose(-2, -1) is everyone scoring everyone. q holds one query per token and k holds one key per token, so for a sentence of n tokens this multiplication produces an n × n table: one predicted rating for every (person, poster) pair, computed in a single matrix multiply. Not one person at a time. The entire catalogue, rated by the entire audience, at once.

The division by sqrt(head_dim) is housekeeping. Dot products of wide vectors get large, and large numbers going into a softmax push it towards handing everything to a single movie. The scaling keeps the scores in a range where the softmax still has opinions rather than verdicts.

softmax(dim=-1) turns each row into shares. dim=-1 means the softmax runs along the columns, so it normalises within a row: for one fixed person, across all the movies. Every row comes out summing to one. You have a fixed evening and you are dividing it between films. Note that nothing gets zero, because softmax never quite gets there. Every token watches every film available to it, some for two hours and some for a second and a half.

weights @ v spends the evening. weights is n × n and v is n × head_dim, so the result is n × head_dim: back to one vector per token. Row i of that result is the sum of every synopsis, each multiplied by the share row i gave it. That is the blend, and it is what this token collected from the sentence.

Watch the shapes across the three lines:

(n × head_dim)  →  (n × n)  →  (n × n)  →  (n × head_dim)

Attention widens into a square in the middle and comes back out. That square is the only thing in the whole model whose size grows with the square of the sentence length, and it is worth remembering where it lives.

Only what has already been released

One constraint, and it is the reason the model can be used for generation at all.

A token can only look backwards. When the model is producing the fifth token it must not see the sixth, because during training the sixth is sitting right there in the data and the exercise would collapse into copying. In the analogy: you can only watch films that have already come out.

In practice this is a mask applied to the scores before the softmax, setting everything after the current position to negative infinity so that its share comes out as zero.

Every token has a row, and every row is the operation the previous figure showed, run for a different asker.

Liking is not one thing

Ask why someone liked a film and you get different answers depending on what you meant. The genre. The lead. The pacing. Score the same catalogue by each of those and different films come out on top.

That is multi-head attention. Instead of one set of W_q, W_k, W_v, the model keeps several, each working on its own narrower slice of the width. Each head runs the three multiplications independently, on its own queries, keys and values, and comes back with its own blend. Then the blends are laid side by side.

And here is the problem with stopping there.

Concatenation leaves each head in its own stretch of the vector. Head 0 occupies the first slice, head 1 the next, and so on. If that were handed straight to the residual stream, whatever head 3 went and found could only ever be written into the dimensions head 3 happens to sit in. A head would be stuck expressing itself in its own corner, and no head could ever combine what it found with what another head found.

So there is a fourth matrix. After the concatenation, the result is multiplied by W_o, and the output of that is what gets added to the stream. It lets any head’s finding land anywhere in the stream, and it lets the heads mix.

The four together are more symmetric than they first look:

W_q	what I am looking for
W_k	how I would like to be found
W_v	what I hand over
W_o	how all of it gets written back into the stream

The first three split one vector into three roles. The last one puts the results back together. Or, in the analogy: each head comes home with its own notes, and W_o is what turns the notes into the single entry you actually write down.

One honest note. Nothing in the architecture assigns a head a topic. You cannot open a model and find the head that handles tense. What the structure guarantees is only that the heads are independent and free to become different, and empirically they do. Naming them is a research activity, not a design one, and this series is not going to pretend otherwise.

What this does not do yet

• Nothing about position. Everything above is order-blind. Shuffle the sentence and every poster, synopsis and score is unchanged, which is plainly wrong for language.
• No shapes. How wide a head is and how many there are is still unstated.
• No numbers. Nothing here has been measured.
• One sequence. Still a single request, still recomputing everything on every step.

There is also a loose thread worth pulling on. A token’s poster and synopsis are made from that token alone. They do not depend on who is asking, and they do not change when the sentence gets longer. Yet the loop from two posts ago rebuilds every one of them, from scratch, on every step.

Nothing so far explains why we would do that.