Compare, normalize, combine.

Mix information from context

A word’s surroundings can change how we interpret it. Attention is a calculation that lets a position combine information from other positions. It produces a weighted mixture, rather than picking one word and copying its meaning.

Think of preparing a short briefing: you compare what you need with several notes, then combine the useful parts. A model does this with lists of numbers, called vectors. It does not read the notes as a person would. This lab uses hand-written vectors; the word labels do not mean the system has learned those words.

Three different jobs

A query represents what a position compares against. Each key supplies the numbers it compares with. Each value supplies the information that will be mixed. In a trained Transformer, learned transformations produce these vectors. Here you edit them directly to see the calculation.

To score a query against a key, multiply corresponding numbers and add them. This is a dot product. For query (1, 0) and keys (1, 0), (0, 1) and (1, 1), the scores are 1, 0 and 1.

The lab divides each score by the square root of 2 because each key has two coordinates. A calculation called softmax then turns the scores into positive shares that add to one. Here those shares are about 40%, 20% and 40%.

Shares control the mixture

With values (1, 0), (0, 1) and (1, 1), the first coordinate of the mixture is about 0.4 × 1 + 0.2 × 0 + 0.4 × 1 = 0.8. The second is about 0.6. The lab shows more precise numbers.

Try this: keep the query and keys fixed, then change a value. The shares stay the same, but the mixture changes. Next change a key and inspect how both the shares and output respond.

A common mix-up: a large attention share does not prove that a word explains the final prediction. Values, later transformations and other layers also affect the result. The attention display shows one part of a computation.

Do not peek at the answer

When predicting the next piece of text, a model must not read later text that contains the answer. A causal mask blocks those later positions before the shares are calculated. Their shares become zero. The first position can use only itself; the second can use itself and the first.

Try the mask at different positions. This restriction models the information available during generation. It does not mean future words would be irrelevant if they were available.

Where this leads

A Transformer can run several attention calculations, called heads, and combine their outputs. Heads can learn different relationships, but they do not come with guaranteed jobs such as grammar or facts.

During generation, a KV cache stores earlier keys and values so the next step can reuse them. It saves repeated work at the cost of memory. It does not teach the model anything new. Choose Technical for the precise mixture and the connection to the next decoder lesson.

Open the full worked examples, calculations and limitations

This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.

Queries ask, keys match, values contribute

Attention lets a position combine information from other positions. Its query is compared with each key. A key helps determine a match score. The value is the information that will be combined using the resulting weight.

In a trained Transformer, learned projections produce queries, keys, and values from token representations. The lab exposes them directly so you can edit them. Its vectors are hand-authored teaching values; the token labels do not imply a trained understanding of those words.

Work through three positions

Start with q = (1,0), keys (1,0), (0,1), and (1,1), and the same vectors as values. The three dot products are 1, 0, and 1. Divide by √2 because the keys have two coordinates. This scaling helps keep score magnitudes manageable as key dimensions grow.

Softmax exponentiates scores and divides each by their total, producing nonnegative weights that sum to one. The weights are approximately 0.4011, 0.1978, and 0.4011. Then multiply each value vector by its weight and add:

$$0.4011(1,0)+0.1978(0,1)+0.4011(1,1)\approx(0.8022,0.5989)$$

The result is a mixture, not a lookup that retrieves exactly one item. Change a value while keeping queries and keys fixed: the weights stay the same, but the output changes. Attention weights alone are therefore not a complete account of what causes the model’s final prediction. Value vectors, output projections, and later layers matter too.

A mask sets an information boundary

A causal mask lets each position use only itself and earlier positions. At the first position, only the first key is allowed, so its weight is one. At the second position, the third key is blocked. The lab applies the mask before softmax; blocked entries get exactly zero probability.

During next-token training, later text contains the answers the model is supposed to predict. Allowing access to it would create a shortcut unavailable during generation. Masking is not a claim that future words are irrelevant; it prevents the model from seeing information it would not yet have.

More heads, and less repeated work

Different attention heads use different learned projections. They can learn different relationships, though there is no rule that one head must represent grammar and another facts. Their outputs are combined by another learned projection.

During causal generation, the keys and values for earlier positions can be stored in a KV cache. The next token can reuse them. This saves repeated computation but consumes memory. It does not change weights or add knowledge. The next lesson runs a tiny two-head decoder and compares actual cached and uncached calculations.