Follow a token through a Transformer.

This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.

Text becomes token IDs

A tokenizer converts text into token IDs from a vocabulary. Tokens in real models may be words, word pieces, bytes, or punctuation; a word can require several tokens. This lab uses a deliberately simpler tokenizer: lowercase words and punctuation, a ten-entry vocabulary, and an unknown-token ID for anything else. It displays exactly what happened to each piece.

Each ID selects an embedding row. Position information makes order available to the network. Our lab adds small sinusoidal position vectors. Other architectures can use methods such as rotary position embeddings, which affect attention computations. The exact mechanism varies; position information is not the same thing as a word’s identity.

Follow one decoder block

The lab uses four coordinates per token, two attention heads, and a six-unit feed-forward hidden layer. Its sine-generated weights are fixed and untrained. This is a working numerical example of the operations, not a miniature language model that has learned useful English.

1. Normalize. Our layer normalization centers a token’s coordinates and rescales them by their spread, with a small stabilizing constant. It prepares values for the next transformation. Models may use other normalization choices.
2. Compute attention. Two sets of query/key/value coordinates compare and mix only permitted positions. The head outputs are concatenated, then projected back to four coordinates.
3. Add a residual. A residual connection adds the attention result to the block input, preserving a direct route for information and gradients.
4. Transform each position. Normalize again, project from four to six coordinates, apply ReLU, project back to four, and add a second residual. This feed-forward network acts on each position separately, after attention has shared information between positions.

The walkthrough shows the selected token’s real intermediate vectors. Choosing a second block repeats the computation with another set of fixed weights. Inspect its input: it is the previous block’s output, not the original embedding.

Turning off explicit position vectors removes one source of order information. In our one-block example, swapping earlier tokens while keeping the final token fixed leaves its attention mixture unchanged, apart from numerical rounding. With multiple causal blocks, earlier representations can already depend on different prefixes; do not conclude that every position-free causal network ignores all order.

Scores become a distribution, then a choice

After the final block and normalization, a vocabulary projection produces logits: one raw score per possible next token. Softmax converts them to a probability distribution. Those probabilities are model outputs, not independently verified confidence estimates.

Greedy decoding chooses the highest-scoring token. Sampling draws according to the distribution. Temperature T changes how concentrated that distribution is; a lower positive temperature sharpens it. In this lab, a visible draw between 0 and 1 makes the sampling choice reproducible and inspectable. Append one token to continue generation, up to the ten-token lab limit.

The distribution for each row predicts what comes after that row’s token. During real pretraining, the target is the actual next token in training text, and cross-entropy rewards assigning it high probability. Our lab does no pretraining. Its generated text can be nonsensical even though the arithmetic is correct.

Caching changes cost, not meaning

The cache comparison processes the same sequence as successively longer prefixes. The uncached path recomputes every prefix; the cached path stores earlier keys and values and calculates only each new token’s projections. The comparison reports the largest logit difference and actual query/key/value projection counts. It does not claim measured speed on your device.

A cache needs memory and is tied to the model, input prefix, and settings that produced it. Changing those settings requires recomputing the relevant state. Caching is not learning: it does not alter weights or improve factual accuracy.

Why next-token prediction can teach useful structure

Predicting varied text benefits from grammar, associations, factual patterns, program structure, and reasoning-like procedures. A trained model can use a mixture of reusable patterns and memorization. Fluent or likely text still does not prove a correct fact, a reliable algorithm, or complete causal understanding. Use independent evidence and tools when exactness matters.