Learn something new without losing the old.

This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.

Shared weights can support several skills

Catastrophic forgetting occurs when learning new tasks substantially damages earlier capabilities. The new gradient optimizes the examples it sees; it does not automatically preserve behaviors absent from that training batch.

Our two tasks occupy different input regions. A labels left-side points by whether y is positive. B labels right-side points by whether y is negative. A nonlinear network can represent both rules together. They are not contradictory labels for identical inputs, so poor retention need not be an unavoidable conflict in the task definition.

Train A, save the checkpoint, and then train B. Compare both independently sampled evaluation sets after each stage. B’s improvement can coincide with A’s collapse. Inspecting only the latest task would hide the regression.

A local way to understand interference

For a small parameter change, the old loss changes approximately by the old gradient dotted with that parameter change:

If a B update moves in a direction that increases A’s loss, interference occurs. The walkthrough calculates this approximation and compares it with the actual one-step change on A’s training examples. It does not use evaluation examples to choose gradients. A first-order approximation can differ from the exact change when curvature or step size matters.

Replay preserves experience

Replay includes old examples in new training. The old gradients can then influence shared parameters instead of disappearing from the objective. In the lab, replay uses 32 stored A examples and 32 B examples per update. New-only training uses 64 B examples. Both process 64 examples, but their exposure to B differs; state that tradeoff when comparing them.

Representative memory matters. A replay buffer that omits rare old cases may fail to protect them. Storing all prior data can be expensive or inappropriate, and no fixed buffer guarantees indefinite retention. Measure old and new performance separately rather than hiding one behind an average.

Penalties and modular updates

A parameter-change penalty discourages moving away from an earlier solution. Our anchor strategy pulls every parameter toward the A checkpoint with the same coefficient. Stronger protection can also make learning B harder. This is a simple regularizer, not full Elastic Weight Consolidation (EWC).

EWC weights the penalty using estimates of which parameters mattered to prior tasks. Its importance estimates and assumptions can be imperfect. Modularity takes another approach: isolate some updates in specialists or adapters while freezing shared parts. The source’s chess-and-Go example uses separate adapters over a shared frozen backbone to reduce interference. Isolation also limits what can be shared or improved jointly.

None of these methods removes genuine contradictions or unlimited capacity demands. The separate scalar example makes that limit explicit:

Equal replay favors a compromise at zero, with average loss four. It cannot make one scalar equal to both two and minus two.

Remembering need not mean updating weights

An agent can update working state during inference, store an episode between tasks, and train its neural components later. These are different timescales. New library API details might belong in a retrievable knowledge record; a repeated debugging skill might eventually justify parameter training.

External memory can store new facts without immediately altering model weights, though retrieval and outdated records create their own problems. Deciding what to store, retrieve, consolidate, or forget is part of the system design. We will explore that directly in the memory module.