How an LLM thinks: from a bedtime story to Q, K and V

Type “The cat sat on the” into a language model and it will very probably say “mat”. Paste in a stack trace and it will explain it. Ask for a sonnet about Kafka consumer lag and it will write one, uncomfortably well.

All three come from the same machine doing the same single thing, over and over. This post is about that thing. We’ll start with the version you could tell a five-year-old at bedtime, then zoom in one level at a time until we’re looking at the actual matrix multiplications: attention, and the three small matrices called Q, K and V that make the whole thing work. We’ll see where those matrices come from during training, and how they’re used every time a model answers you.

Level 0: the bedtime version

Imagine a friend who has read every book in the biggest library in the world. Not once, but over and over. Picture books, cookbooks, instruction manuals, the backs of cereal boxes.

Now you play a game. You say “Once upon a…” and your friend shouts “time!” You say “Twinkle, twinkle, little…” and they shout “star!”

That’s the whole game: guess the next word.

The clever part is what happens next. Your friend’s guess gets added to the story, and then they guess again. And again. Word by word, like building a Lego tower one brick at a time, and every time they add a brick, they look at the whole tower before choosing the next one.

1 / 4 You start the story ← →

How did your friend get so good at guessing? Practice. Someone covered up the next word in millions and millions of sentences. Your friend guessed, peeked, and felt a tiny “oops!” every time they were wrong. Each “oops” changed them a little bit. After enough oopses, they stopped being wrong very often.

There’s one more trick, and it’s the most important one. When your friend hears “The dog chased the ball, and then it rolled under the couch,” they know “it” is the ball, not the dog. Before every guess, they look back over the story and decide which earlier words matter most right now.

That trick has a name: attention. Hold on to it. It’s the heart of everything that follows.

An LLM is a guessing machine that has practised for a very, very long time, and that looks back at the story to decide what matters before every guess.

Everything below is the grown-up version of that sentence.

Level 1: words become tokens

First confession: the model doesn’t see words. It sees tokens: chunks of text, a bit like Scrabble tiles that can hold a whole word, part of a word, or a single punctuation mark.

Common words get a tile of their own. Rare words are assembled from pieces. The tile set, the vocabulary, is fixed before training, usually somewhere between 30,000 and 200,000 tiles, and every tile has a number. From this point on, the model never sees text again. It only sees those numbers.

Two consequences are worth knowing:

• Any text works. Typos, code, emoji and made-up words can all be spelled out of smaller tiles, so nothing is ever “unknown”.
• Some questions are weirdly hard. Ask a model how many r’s are in “strawberry” and it has to reason about letters it never directly saw, because the word arrived as a few chunks, not ten letters. It’s like counting the letters in a word you’ve only ever heard spoken aloud.

Level 2: tokens become places on a map

A number like 8,415 says nothing about cats. So the first thing the model does is look up each token in a giant table, the embedding table, and swap the number for a long list of numbers: a vector. In a mid-sized model like Llama 3 8B, that list is 4,096 numbers long.

The best way to think about that vector is as coordinates. Every token gets a spot on a map, and the map is arranged so that things with similar meanings live near each other.

Our cartoon has two directions: left–right and up–down. The real map has thousands, and that’s where the magic hides. With that many directions, there’s room for one to loosely mean “plural”, another “past tense”, another “is a place”, another “sounds sarcastic”, all at once, all overlapping.

Who draws the map? Nobody. The table starts as random noise, and training (Level 5) slowly drags every token to wherever it’s most useful. Tokens end up near each other because they get used in similar ways, not because anyone told the model they’re related.

One more ingredient is order. “Dog bites man” and “man bites dog” contain the same tokens, so the model has to know where each one sits. The original transformer added a “position stamp” vector to every token. Most modern models use a trick called RoPE instead: they rotate the query and key vectors (you’ll meet them in a moment) by an angle that depends on position, so attention can feel how far apart two tokens are.

Level 3: a tower of identical floors

We now have a row of vectors, one per token. Those vectors ride upward through a tower of transformer blocks: anywhere from a few dozen floors to over a hundred, all with the same floor plan but their own learned weights.

The mental model I find most useful is the residual stream. Picture each token carrying a notebook (its vector) up the tower. On every floor, two things happen, and each one adds notes to the notebook instead of rewriting it:

1. Attention: the meeting. Tokens look at each other and copy over the information they need. This is the only place in the whole model where tokens exchange information.
2. The MLP (feed-forward layer): back at the desk. Each token, alone, thinks about what it just heard. A lot of the model’s stored knowledge seems to live here: “Paris” → capital of France, Eiffel Tower, euros.

At the top, the notebook of the last token is turned into a score for every tile in the vocabulary (the logits). A function called softmax turns those scores into probabilities: “mat” 41%, “floor” 25%, “keyboard” 6%. The model picks one, adds it to the story, and goes again.

If you think in systems terms, attention is the network layer: all cross-token communication goes through it. The MLP is a stateless function applied to each request on its own. Everything that makes context matter flows through attention, so that’s where we’ll spend the rest of the post.

Level 4: attention, the heart of the machine

Why tokens need to talk

On its own, a token’s vector is ambiguous. “Bank” in “river bank” and in “bank loan” starts out as the same point on the map. “It” starts out meaning nothing in particular. A token only becomes precise once it has pulled in information from its neighbours.

Take this:

The cat sat on the mat. Later, it

What comes next? “Stretched”, “yawned” and “wandered off” are good guesses. “Frayed” and “got vacuumed” are not. To make a good guess, the token “it” has to fetch information from “cat”, not from “mat”. Attention is the mechanism that decides, for every token, who to listen to, and how much.

Q, K and V, explained with a library

On every floor, each token turns its notebook into three new vectors:

• Query (Q): what am I looking for?
• Key (K): what do I have to offer? This is the label on the spine.
• Value (V): if you pick me, here’s what you actually get. This is the contents of the book.

Imagine walking into a library with a question in your head (your query). Every book has a label on its spine (its key). You scan the spines, and the better a label matches your question, the more of that book you read (its value). It’s a strange library, though: you don’t pick one book. You read a little of every book, in proportion to how well its label matched.

If you build software, here’s the version that sticks: attention is a hash map with fuzzy lookups.

# A normal dictionary: exact match, one winner.
value = table[query]

# Attention: compare the query with EVERY key,
# then return a blend of ALL the values, weighted by how well they matched.
weights = softmax([similarity(query, key) for key in keys])
value   = sum(w * v for w, v in zip(weights, values))

Every step there is smooth. There’s no hard “match” or “no match”, only “more” or “less”. That makes the whole lookup differentiable, and as we’ll see, differentiable is exactly what makes it trainable.

Why three different vectors from the same token? Because what you’re looking for isn’t what you advertise, and what you advertise isn’t everything you’d hand over. At a job fair, what you’re looking for (“a platform role in Dallas”) is different from your résumé headline (“senior Java engineer”), and the headline is different from the full substance you’d bring. One token, three roles.

For our sentence, you can imagine the query of “it” saying “I’m a pronoun, looking for a recent noun that’s alive”. The key of “cat” says “noun, animal, the subject”. The key of “mat” says “noun, object, a place”. The best match wins most of the attention:

Doing it by hand

Let’s run one attention step with numbers small enough to follow. We’ll use four tokens (cat, sat, mat, it) and vectors with only two dimensions. Pretend dimension 1 loosely means “is an animal” and dimension 2 means “is a noun”. The query for “it” is [2, 1]: animal, please, and a noun.

token	key	q · k	÷ √2	softmax weight
cat	[2, 1]	5	3.54	0.919
sat	[0, −1]	−1	−0.71	0.013
mat	[−1, 1]	−1	−0.71	0.013
it	[0.5, 0]	1	0.71	0.054

Now give each token a value vector. Say its two dimensions mean “furry and alive” and “flat and woven”: cat = [0.9, 0.1], sat = [0, 0], mat = [0.1, 0.9], it = [0.2, 0.2]. The output for “it” is the weighted blend:

0.919 × [0.9, 0.1] + 0.013 × [0, 0] + 0.013 × [0.1, 0.9] + 0.054 × [0.2, 0.2]  ≈  [0.84, 0.11]

“It” now carries furry and alive. Its notebook has been updated, and further up the tower the model can prefer “stretched” over “frayed”.

Those are the same four steps every attention head in every LLM runs, billions of times a second:

1. Score: take the dot product of the query with each key, which measures how much two arrows point the same way.
2. Scale: divide by √d, the square root of the vector length.
3. Normalise: apply softmax to get positive weights that sum to 1.
4. Blend: take the weighted sum of the values.

The formula, which now reads like a sentence

Do that for all tokens at once by stacking the vectors into matrices:

Q = X · W_Q        K = X · W_K        V = X · W_V

Attention(Q, K, V) = softmax( Q · Kᵀ / √d_k  +  M ) · V

X is every token’s notebook, one row per token. W_Q, W_K and W_V are the three learned projection matrices, and they are the only things in this formula that training changes. Q · Kᵀ computes every query-key score in one go: a grid with one row per token doing the asking and one column per token being asked. M is the mask, which we’ll meet in a moment.

In code it’s shorter than the explanation:

import numpy as np

def causal_self_attention(X, W_Q, W_K, W_V):
    """X: (T, d_model), one row per token. Returns (T, d_head)."""
    Q, K, V = X @ W_Q, X @ W_K, X @ W_V
    scores = Q @ K.T / np.sqrt(K.shape[-1])          # (T, T): every asker vs every key
    future = np.triu(np.ones(scores.shape, dtype=bool), k=1)
    scores[future] = -np.inf                         # no peeking (see below)
    weights = np.exp(scores - scores.max(axis=-1, keepdims=True))
    weights /= weights.sum(axis=-1, keepdims=True)   # softmax, row by row
    return weights @ V                               # blend the values

Why divide by √d?

It looks like a fudge factor, but it isn’t. In a real head, queries and keys have around 128 dimensions. If their components are roughly random with variance 1, their dot product has variance 128, so typical scores land around ±11. Softmax exaggerates differences exponentially: scores of 11 and 0 become weights of 0.99999 and 0.00001. Attention collapses onto a single token, and the learning signal flowing to the losers shrinks to almost nothing.

Divide by √128 ≈ 11.3 and the same pair of scores becomes about 1 and 0, which gives weights of 0.73 and 0.27. There’s still a clear preference, but now there’s room to learn.

The mask: no peeking

A language model is trained to predict the next token, so a token must never see the tokens after it. Otherwise, during training, it would simply read the answer off the page. The mask M sets every score above the diagonal to −∞, and softmax turns −∞ into a weight of exactly zero.

This has a subtle consequence. In the famous puzzle “The trophy didn’t fit in the suitcase because it was too big”, the word that settles what “it” means (“big”) comes after “it”. So “it” can’t resolve itself. The tokens that come later, like “big”, do the resolving, looking back at “trophy”, “suitcase” and “it” together. Meaning in an LLM doesn’t get settled in one place. It keeps getting refined as the text goes on and as the vectors climb the tower.

The triangle also makes training remarkably efficient, which we’ll come back to.

Many heads, many questions

One set of Q, K and V can ask one kind of question. So every floor runs many heads in parallel (32 in Llama 3 8B), each with its own smaller W_Q, W_K and W_V. Think of a panel of specialists reading the same sentence: one tracks which noun a pronoun refers to, one watches the previous token, one matches brackets in code, one notices when a pattern repeats. Their outputs are glued together, mixed by one more learned matrix (W_O), and added to the notebook.

A caveat: those job titles are our labels, not the model’s. Researchers have found heads with crisp jobs, most famously induction heads, which spot a pattern earlier in the text and complete it. But many heads do something messier that nobody has named. And nobody assigned any of the jobs, which raises the obvious question.

Level 5: where do W_Q, W_K and W_V come from?

This is the part that surprises people: nobody designs them. No engineer writes “layer 12, head 7: you handle pronouns.” Every entry of every W_Q, W_K and W_V starts life as a small random number. In Llama 3 8B, each layer’s W_Q alone is a 4,096 × 4,096 grid (about 16.8 million numbers), and there are 32 layers of them, plus the K, V, O and MLP matrices. The pronoun-chasing and the bracket-matching emerge from one training signal applied an absurd number of times.

The training signal: surprise

Take a chunk of text from the training set, run it through the model, and at every position ask: what probability did you give to the token that actually came next?

The loss is the negative log of that probability (cross-entropy). The easiest way to read it is as surprise:

the model gave the real next token…	loss (−ln p)
2%	3.91 (very surprised)
60%	0.51
100%	0 (not surprised at all)

Here’s where the causal mask pays off. Because no position can see its future, a single forward pass over a 4,000-token chunk produces 4,000 separate predictions, each position guessing its successor from only what came before. So one chunk gives you thousands of graded guesses, all computed in parallel. (During training the model always sees the real previous tokens, never its own guesses. This is called teacher forcing.)

Backprop: a mixing desk with billions of knobs

Picture a mixing desk with billions of knobs: one for every number in every matrix. Backpropagation, which is just the chain rule from calculus applied carefully backwards through every operation, answers one question for every knob at once: if I turned you a hair clockwise, would the surprise go up or down, and by how much? That answer is the knob’s gradient. Then every knob gets a tiny turn in the helpful direction.

Repeat for trillions of tokens.

Following one nudge into W_Q and W_K

Let’s follow a single bad guess backwards. Early in training, the model reads “The cat sat on the mat. Later, it” and the real next token is “stretched”. Suppose “it” attended mostly to “mat” and gave “stretched” a low probability:

1 / 5 Forward pass: “it” looks at the wrong noun ← →

The step that finally reaches the matrices is refreshingly simple. A score is q · k, so its derivative with respect to the query is just the key, and vice versa. And since q = x · W_Q, the gradient for W_Q is the outer product of the token’s input vector x with the gradient on q. Put in words: turn W_Q so that inputs like this one produce queries that point more towards keys like cat’s. W_K gets the mirror-image nudge, and W_V gets its own nudge so the values carry whatever the prediction needed.

Here’s the insight that makes the whole thing work. That one nudge is tiny and specific to one sentence. But billions of sentences push on the same knobs. Nudges from idiosyncratic sentences point in random directions and mostly cancel out. Nudges that reflect real regularities of language (pronouns usually refer to recent animate nouns, a closing bracket matches the most recent open one) push the same way every time, so they accumulate.

A head’s specialisation is the residue of whatever helped prediction, averaged over a very large pile of text.

A tiny experiment you can watch

I wanted to see that happen, not just assert it, so I trained a single attention head from scratch on a toy task. Each input is six letter-digit pairs followed by a question:

A6  B6  F7  G1  D5  H6  F?     →  answer: 7

Each token’s vector is a fixed random letter vector plus a fixed random digit vector, squashed together into 16 numbers. The model has no idea which parts of those 16 numbers mean “letter” and which mean “digit”. The only things allowed to learn are W_Q, W_K, W_V and a small output matrix, all starting as random numbers and trained with exactly the recipe above: predict, measure surprise, backpropagate, nudge. The whole forward pass is five lines:

q = x_question @ W_Q                           # what am I looking for?
K = X @ W_K                                    # every token's label
V = X @ W_V                                    # every token's contents
weights = softmax(K @ q / np.sqrt(d_k))        # who matches?
answer_logits = (weights @ V) @ W_O            # read out a digit

Here’s what the question token’s attention looks like as training goes on:

1 / 3 Step 0: random weights, attention is a shrug ← →

Test accuracy went from about 12% (roughly chance, with ten possible digits) to 88% after 300 steps and 100% by step 600.

Then I checked what each matrix had learned to listen to. Remember, nobody told it that letters are for matching and digits are for answering. After training:

• W_K reacts about 3× more strongly to the letter part of a token than to the digit. The keys became letter labels.
• W_V reacts about 3.4× more strongly to the digit part. The values became the answers.

That’s the whole story of Q, K and V in miniature. The roles of “label” and “contents” were never programmed. They fell out of the gradient, because that division of labour is what makes the loss go down. A real LLM is this same thing with 4,096 dimensions instead of 16, dozens of heads on each of dozens of floors, and a large slice of the written world instead of letter-digit pairs.

One detail I love: even fully trained, the “F?” token still spends some attention on itself. Its own key says “F” too, so it matches. Its value only carries ”?”, though, and the output layer learned to ignore that. It’s a small, harmless quirk that nobody designed, which is a fair description of a lot of what goes on inside real models.

(The full experiment, with the gradients written out by hand and checked numerically, is about 150 lines of NumPy if you’d like to run it yourself.)

From autocomplete to assistant

Pretraining produces a model that continues text. To get an assistant, the model is trained further on examples of helpful conversations, then tuned with feedback about which answers people (or other models) prefer. It’s the same machinery and the same knobs, with different examples of “what should come next”. The W_Q, W_K and W_V we’ve been talking about mostly take shape during pretraining, and the later stages adjust them.

Level 6: inference, with Q, K and V frozen

Training is over and the knobs are glued in place. Someone types a prompt. What actually happens?

Prefill, then decode

1. Prefill. The whole prompt goes through the tower in one parallel pass, just like training minus the learning. On every floor, every prompt token computes its q, k and v with that floor’s frozen W_Q, W_K and W_V.
2. Decode. The model generates one token at a time. Each new token runs up the tower, attends back over everything before it, and at the top produces a probability for every token in the vocabulary. One is picked and appended, and the loop continues.

The KV cache, or: the party guestbook

Here’s a question that sounds like trivia but explains a lot about LLM latency and cost. When a new token arrives, which of the old Q, K and V vectors does it need?

• Old keys and values: all of them. The new token has to compare its query against every earlier label and read every earlier value.
• Old queries: none of them, ever. A query is the question a token asked when it was processed. That question has been answered and the answer is already in its notebook. Nobody will ask it again.

Think of a party with a guestbook. Each guest who arrives writes a one-line description of themselves (a key) and leaves a note (a value). A new arrival reads the whole book to decide whom to talk to. Nobody needs to remember what questions earlier guests had on their minds.

So inference engines keep a KV cache: for every layer and every past token, its key and value vectors, computed once and reused at every later step. Each decode step computes q, k and v for just the one new token, appends its k and v to the cache, and computes a single row of the attention grid.

1 / 3 Prefill: the whole prompt fills the cache in one pass ← →

That one idea explains a lot of behaviour you’ve probably seen:

• Time to first token vs. tokens per second. Prefill is one big parallel matrix multiply. It’s compute-bound, and GPUs are superb at it. Decode does very little arithmetic per step but has to stream the weights and the whole KV cache through memory for every token, so it’s memory-bandwidth-bound. That’s two very different performance profiles inside one request.
• Long context costs memory, not just time. The cache size is 2 (for K and V) × layers × KV heads × head size × bytes per number × tokens. For Llama 3 8B (32 layers, 8 KV heads of 128 numbers each, 16-bit precision) that’s 128 KiB per token, or 16 GiB for a single 128k-token conversation. That’s about as much as the model’s weights.
• Why grouped-query attention exists. If each of the 32 query heads had its own K and V, that same cache would be 64 GiB. Letting four query heads share one K/V head (GQA) cuts it by 4× with little loss in quality. That’s why the config says “8 KV heads”.
• Why prompt caching works. Thanks to the causal mask, the keys and values for a prefix never depend on anything after it. If many requests share a long system prompt, its cache can be computed once and reused for all of them.

Choosing the next token

At the top of the tower, the last token’s notebook becomes logits, one score per vocabulary token. Softmax turns them into probabilities, and a temperature knob controls how adventurous the pick is: the logits are divided by T before the softmax.

At temperature 0 the model always takes the top token (greedy decoding). Low temperatures are sharp and safe but can get repetitive. High temperatures are creative and occasionally unhinged. Top-p sampling adds a guardrail: sample only from the smallest set of tokens whose probabilities add up to, say, 90%, so the long tail of nonsense never gets a turn.

Putting it all together

Here’s “The cat sat on the” one last time, passing through every level:

1. Tokens: the text is chopped into five tiles and turned into five integers.
2. Embeddings: each integer becomes a point on the 4,096-dimensional map of meaning.
3. The tower: on each of 32 floors, every token makes a q, k and v with that floor’s frozen W_Q, W_K and W_V; attention blends in the values it needs; the MLP thinks about the result. The keys and values go into the cache.
4. The top: the last token’s notebook becomes logits, the logits become probabilities, and the model samples “mat”.
5. Again: “mat” gets its own q, k and v on every floor, attends over the cache, and the next guess begins.

What this explains

A few practical intuitions fall straight out of the mechanics:

• Context is everything, because attention is the only way information moves. If it isn’t in the prompt or baked into the weights, no query can find it. Good prompts are good because they give the queries something worth matching.
• Hallucination isn’t a failed lookup, because there is no lookup. The model produces the most plausible continuation. It doesn’t consult a separate database of facts. When “plausible” and “true” diverge, plausible wins.
• Letters are hard; meaning is easy. Tokens are chunks, and the map is organised by meaning, not spelling.
• Every token you add stays in the room. Each one sits in the KV cache for the rest of the conversation and gets attended to by everything that follows.

Underneath all the matrices, it’s still the bedtime story: a friend who has read the whole library, guessing the next word, and looking back at the story to decide what matters before each guess. The grown-up version just has 16.8 million knobs in each W_Q, and learned where to set every one of them from its own mistakes.

Going further

• Attention Is All You Need (Vaswani et al., 2017): the original transformer paper, and more readable than its reputation.
• A Mathematical Framework for Transformer Circuits (Anthropic, 2021): the paper that popularised thinking of the residual stream as a shared channel that every layer reads from and writes to.
• The Illustrated Transformer by Jay Alammar: the classic visual walkthrough.
• Let’s build GPT: from scratch, in code, spelled out by Andrej Karpathy: two hours from a blank file to a working transformer.