Why can a bigger step make learning worse?

A score for a mistake

Imagine adjusting a price estimate until it matches an example. You need a way to tell whether a change helped. Loss is that error score: smaller means the prediction fits better under the chosen rule.

This lab predicts a target of 4 by multiplying an input of 2 by a weight. At weight 0.5, the prediction is 1. The error is 1 − 4 = −3. Squaring it gives a loss of 9. Squaring also makes an error of +3 cost 9, so errors on either side count.

For several examples, we can average their squared errors so adding examples does not automatically multiply the score. Other tasks use other loss rules. A loss measures what its rule specifies; it does not automatically measure fairness, truth or usefulness.

Use the slope to choose a direction

The gradient tells you how the loss changes near the current weight. Think of looking at the slope under your feet on a hill. To go downhill, move against that slope. This analogy describes a local direction; it does not promise that a large jump will land lower.

For this example, multiply the weight by 8 and subtract 16 to get the slope. At weight 0.5, it is −12. A learning rate sets the size of the change. With rate 0.1, subtracting 0.1 × −12 changes the weight from 0.5 to 1.7.

The new prediction is 2 × 1.7 = 3.4. Its error is −0.6, so its loss is 0.36. That is lower than 9.

Try this: reset to weight 0.5 and take one step at rate 0.1. Predict whether the new weight will pass the bottom of the curve, at weight 2, before checking.

Why a bigger step can be worse

Reset and try rate 0.30. Subtracting 0.30 × −12 takes the weight to 4.1. The prediction becomes 8.2, and its error is 4.2. Squaring that error gives 17.64: worse than the starting loss of 9.

The direction was locally downhill, but the jump went too far. Watch both the weight movement and the loss rather than copying a learning rate without checking it. A useful rate for one model or dataset can fail for another.

A common mix-up: calculating the gradient does not change the weight. An optimizer is the rule that uses the gradient to choose an update. Gradient descent is the subtract-the-scaled-gradient rule used here. With several parameters, their gradients describe the same parameter state before the update.

What a falling curve does not tell you

This example has one adjustable number and one bottom to its bowl. Real networks can have many interacting parameters, flat regions and more complicated surfaces. A locally useful direction does not guarantee the best possible result.

A lower training loss means a better fit to the training objective; it does not by itself establish useful predictions on new examples. Use this lesson to recognize improvement and overshooting. Later, compare training with held-out examples before claiming that the model learned a useful pattern. Choose Technical for the equations behind the same calculation.

Open the full worked examples, calculations and limitations

This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.

Loss makes one kind of mistake measurable

Squared error makes both positive and negative errors costly. An error of −3 and an error of +3 both give 9. Averaging over examples prevents the size of the dataset alone from multiplying the score.

A loss is a choice. Predicting a number often uses squared error; classifying an example often uses cross-entropy, which penalizes assigning little probability to the correct class. Neither objective automatically captures fairness, truth, or usefulness.

$$L(w) = (2w - 4)^2$$

A gradient is a local sensitivity

Imagine moving a tiny distance to the right on the curve. A negative slope means the loss initially falls. A positive slope means it rises. The derivative names that local rate of change; a gradient collects the derivatives for several parameters.

For this example, the slope is 8w − 16. At w = 0.5 it is −12. Gradient descent subtracts learning rate times gradient: 0.5 − 0.1 × (−12) = 1.7. The new loss is 0.36.

$$w_{new} = w - \eta\,\frac{dL}{dw}$$

Direction and distance are different decisions

The learning rate controls distance. A large step can pass the bottom and land higher on the other side. Reset and try 0.30: the next weight is 4.1 and the loss is 17.64, worse than 9.

Our bowl has one minimum. Real networks can have flat regions, ridges, and saddle points. A local direction does not promise a global best solution, and low training loss does not prove good performance on new data.