Compare manual and automatic gradients.

This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.

Let the graph keep track of the calculation

Suppose a program computes f(x) = x × x + x. At x = 3, its output is 12. If x increases a little, it affects both places in the product and the final addition. The total sensitivity is 3 + 3 + 1 = 7. Predict that total before stepping through the worked example.

Automatic differentiation records which values each operation used. It then applies the chain rule to that graph. It does not guess slopes from nearby values, and it does not need to simplify the whole expression into a symbolic formula. It still uses finite-precision arithmetic, so numerical checks remain useful.

The lab’s tape supports scalar input, addition, multiplication and tanh operations. Learners construct different graphs by choosing the batch, activation and shared-weight options. This is a small inspectable implementation, not a replacement for a tensor framework or studio P05.

Forward: values and dependencies

A node is a value produced by an input or operation. An edge says that one value is an input to another operation. The tape records nodes in dependency order: inputs appear before the operations that use them. That makes this graph acyclic; no value needs itself before it can be computed.

The operation x × x has two input slots even though both refer to the same x node. Deduplicating those edges would discard a real part of the derivative. In the live graph, repeated boxes carry the same node number to make that sharing explicit.

The node selector shows a neighborhood of the complete graph: input slots, the selected operation and consumers. The full tape table lists every node, value, dependency and final gradient. The graph does not imply that every leaf is a trainable parameter: features, targets and constants are leaves too, but only w₁, w₂ and b are updated.

Backward: multiply locally, add across paths

Begin at the scalar output with derivative 1. An output’s sensitivity to itself is one. Start every other accumulated gradient at zero. Visit operations in reverse dependency order so all downstream uses have contributed before an operation passes information to its inputs.

If u = a + b, the local derivative with respect to either input is 1. If u = a × b, its local derivative with respect to a is b, and with respect to b is a. If u = tanh(a), the local derivative is 1 − u², using the already computed output u.

For one edge, contribution = upstream derivative × local derivative. Here “upstream derivative” means how the final loss changes with the current operation’s output. Add that contribution to the parent’s existing total. The backward-event control exposes the upstream value, local slope, contribution, previous total and new total.

For x × x + x, addition sends 1 to the square and 1 directly to x. Multiplication then contributes 3 through each of its two slots. The total at x is 7. At x = 2, the nearby practice result is 2 + 2 + 1 = 5.

The overwrite fault replaces a parent’s accumulated derivative instead of adding. It can lose both shared-parameter paths and the two inputs of a square. It may still produce a plausible number or reduce the loss on a particular step. A decrease alone does not establish that the derivative is correct.

A row of a small model

The first row has inputs x₁ = 1, x₂ = 2 and target y = 1. Start with weights w₁ = 0.5, w₂ = −0.25 and bias b = 0.1. The pre-activation is:

z = x₁w₁ + x₂w₂ + b = 1 × 0.5 + 2 × (−0.25) + 0.1 = 0.1.

With a linear activation, the prediction is z. Its error is e = prediction − target = −0.9. The half-squared loss is ℓ = e² / 2 = 0.405. The factor one half makes the derivative of this loss with respect to the prediction equal to e rather than 2e.

For this single linear row, the weight derivatives are e × x₁ = −0.9 and e × x₂ = −1.8. The bias derivative is e = −0.9. A gradient descent step with learning rate 0.1 changes the three parameters simultaneously to (0.59, −0.07, 0.19). The new prediction is 0.64, giving loss 0.0648.

Switch on tanh to insert a nonlinear operation between z and the prediction. Multiply the error by 1 − prediction² before passing it to weights and bias. Large absolute pre-activations can produce a small local slope; the graph exposes this mechanism rather than treating all small gradients as an implementation error.

A batch: dimensions tell you what the code means

For three examples with two features each, arrange inputs in a matrix X of shape 3 × 2. A weight column W of shape 2 × 1 produces XW of shape 3 × 1: one prediction per example. The matching inner dimension 2 is summed during multiplication. A scalar bias is added to all three rows.

This repeated addition is called broadcasting. Forward broadcasting shares one value across rows; backward propagation adds all uses of that shared value. It does not invent three independently trainable biases.

The loss averages the three individual half-squared errors: L = (ℓ₁ + ℓ₂ + ℓ₃) / B, where B is batch size, either 1 or 3 in this lab. L is a scalar even though predictions are a column. The default row losses are 0.405, 0.01125 and 0.36125, giving mean 0.2591667 after rounding.

For row r, define δᵣ = eᵣ × activation slope / B. The gradient of w₁ is the sum of δᵣxᵣ₁ over rows; the gradient of w₂ is the sum of δᵣxᵣ₂; the bias gradient is the sum of δᵣ. The manual reference uses these formulas directly without reading the tape. The scalar tape expands the same matrix computation into individual operations.

Transposing W to shape 1 × 2 does not repair the multiplication: X’s inner dimension 2 would meet 1. Broadcasting a mismatched target vector can likewise change the intended loss in a general tensor library. This lab uses explicit scalar row targets, avoiding such implicit broadcasting. Studio work must test the tensor shapes as well as the numbers.

Expose a bug with a better case

The missing-mean fault differentiates the sum of row losses while the displayed forward objective remains their mean. With three rows, the faulty derivatives are three times the correct ones. With only one row, sum and mean are identical, so this fault passes the checks.

That passing single-row result is useful but incomplete evidence. Change the batch size to expose the missing operation. When choosing tests, include cases that distinguish competing implementations rather than many copies of the same case.

Under reuse w₁ for both features, the row formula becomes z = w₁(x₁ + x₂) + b. Both feature paths contribute to w₁’s derivative. The unused w₂ still exists as a leaf but has no path to the loss, so its derivative is zero. This illustrates tied parameters; it does not average their gradients or create a second independent weight.

Compare three ways to calculate a derivative

The lab compares the reverse-mode tape, an explicit manual formula, and finite differences. For parameter θ, a central finite difference is [L(θ + h) − L(θ − h)] / (2h) with every other parameter fixed. Here h = 0.00001. At a parameter boundary of −3 or 3, the checker uses an inward one-sided difference so it stays in the declared domain.

A numerical difference is approximate. The displayed mixed absolute/relative tolerance allows the first-order boundary estimate; a separate much tighter check compares the tape with the manual formula. See numerical gradient checks for why tiny steps and loose tolerances can mislead.

Matching these calculations across several cases gives evidence about implementation correctness. It does not prove every operation, shape or future extension is correct. Each reverse sweep starts fresh, so a second check does not accidentally double a previous gradient. Some frameworks intentionally accumulate gradients across calls; code using them must explicitly manage that state.

Update all parameters from the same snapshot

The update is θᵢ,new = θᵢ,old − ηgᵢ, where θᵢ is parameter i, η is learning rate and gᵢ is its computed derivative. Calculate every gᵢ using the same old model before changing any parameter. Recomputing later derivatives after an earlier parameter changes implements a different procedure.

This lab permits at most fifty updates per run and five per button action. A proposed parameter outside ±3 is rejected, retaining the preceding model. There is no claim that these teaching bounds are mathematically optimal. All three examples are training data; lower loss on them is not held-out generalization evidence.

Apply and retain evidence

Guided practice: capture one faulty run and one corrected run using matched settings. Explain a specific missing or overwritten contribution, and show a case that exposes it. Use the graph, event arithmetic and comparison table in your explanation.

Implementation practice: implement a small tape independently, including shared inputs and bias broadcasting. Test x × x + x, repeated reverse calls, a batch larger than one, a disconnected parameter, tanh and simultaneous updates. Then reproduce the calculation in a tensor framework in studio P05. The browser reference does not assess your code; runnable artifacts and independent review are still needed.

Decision practice: review a claim that “training loss fell, so our new gradient code is correct.” Choose which of the lab’s checks to request, identify where a single-row test misses a fault, and state what the results cannot establish about new-data performance.

Captures and notes are temporary and downloadable, outside required completion. The lesson questions provide practice feedback; they are not a new independently assessed implementation qualification. Human artifact review and learner pilots remain pending. Continue to training diagnosis for broader failure investigations.