Predict the next token.

This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.

One sentence supplies several prediction questions

Take the authored reference red apples grow. Add a start marker before it and an end target after it. The aligned inputs are start, red, apples, grow; the targets are red, apples, grow, end. Each prediction asks for the next token, not a copy of the token at the same input position.

The lab uses seven fixed output tokens, supplied probabilities and four tiny grammar sentences. It does not fit a neural network. Changing the controls changes a probability table so you can inspect the objective before introducing parameter updates.

Past reference tokens are allowed; future targets are not

At the row predicting grow, the reference prefix is start, red, apples. The next input position contains grow, so exposing that position would reveal the answer. The visibility matrix makes this boundary explicit: a causal row includes current and earlier input positions, and hides later ones.

Teacher forcing means using the reference’s earlier tokens as the conditioning prefix while predicting its next token. The target is available separately to calculate the error. That is different from allowing the prediction mechanism to read the target as an input.

The lab’s broken mode deliberately copies the next input and assigns it probability one. This is an explicit leakage oracle, not a claim that every network without a causal mask immediately learns perfect copying. The final end target is outside the input array, so that last row has no next input to copy.

Give a confident wrong answer a larger penalty

Suppose the actual next token is grow. Assigning it probability one half means the model left substantial room for other answers. Assigning it one eighth makes the observed answer much less expected. Cross-entropy measures that difference using the probability assigned to the target, not just whether the largest-probability token was correct.

For one observed target, its loss in bits is:

Here p is the target’s predicted probability, between zero and one. ℓ is its loss. Read the formula as “negative base-two logarithm of the target probability”. At probabilities that are powers of one half, count halvings: one half gives one bit; one quarter gives two; one eighth gives three. Other probabilities give values between those landmarks.

Probability one gives zero loss. Exactly zero gives infinite loss: the table ruled out an event that occurred. The lab shows infinity explicitly rather than silently replacing zero with a small positive number. Practical implementations often calculate log-probabilities from logits for numerical stability, which is a separate implementation issue.

For several scored targets, add their losses and divide by their count:

N is the number of scored targets, t identifies a position and pₜ is the probability assigned to that position’s actual target. Two targets with probabilities one half and one eighth have losses one and three: their mean is two bits. This lab includes the end target in its denominator. Natural logarithms instead give losses in nats; do not mix those units.

Generation follows its own decisions

With the default settings, a greedy generator chooses red birds sing end. The reference may instead be red apples grow end. Under teacher forcing, grow is scored after the reference prefix ending in apples. During the actual rollout, the corresponding history ends in birds.

Those are different conditional questions. The generated-prefix column diagnoses how reference-token probabilities change after a different choice. Its average is not the likelihood of the reference sequence, because it conditions on other histories. The ordinary causal, reference-prefix score follows the reference sequence’s own chain of conditional probabilities.

Both sentences are valid in this small grammar. A different reference match is therefore not automatically a task failure. For a task requiring a particular fact or exact output, the distinction may matter greatly; for an open-ended task, several outputs may be acceptable. Specify task success separately from prediction loss and grammar validity.

Lower scores can come from the wrong information

Enable future copying and compare the table with generation. The leaked reference score falls, but the generated sentence does not improve: the future reference inputs do not exist during generation. The base probability recipe did not change, and this demonstration has no trained parameters.

A useful diagnosis records the input/target alignment, allowed information, probabilities, loss units and denominator. Include an actual generated trace too. A lower number alone does not show that a training change improved the intended capability.

Apply the check before training

For guided practice, explain one row and reproduce its arithmetic. For implementation practice, test that changing future tokens cannot affect an earlier causal output and that targets are shifted correctly. For a decision brief, require both leakage-free reference scoring and task-relevant generated-output evaluation.

Export a note with one permitted prefix, one calculation, one leakage failure and one limitation. These temporary experiments and the lesson challenge are practice, not independent implementation review. Return to causal attention to inspect masking inside an attention mechanism, or language-task evaluation to compare loss with evidence of task success.