Lesson 3 of 12 in this unit
Make a problem easier to see.
Build XOR with two hidden neurons and discover what nonlinear activations add.
By the end: Trace a hidden representation and explain why affine layers alone cannot solve XOR.
Prerequisites and path options
Why learn this? See why adding a hidden layer can solve patterns a straight boundary cannot, before paying for a larger model.
Start with something concrete.
Two switches control a lamp. The lamp is on when exactly one switch is on, and off when both match. This rule is called exclusive OR, or XOR. Try the four switch combinations in the lab.
Try the interactive modelIf we stack two layers but use no nonlinear activation, can we separate both positive XOR corners from both negative corners with the final straight boundary?
Change one thing. Watch what follows.
Loading this browser experiment…
Understand what changed.
A shorter explanation, with the original worked examples available below. Both explain the same experiment and its limits.
The idea in everyday terms
Some patterns cannot be separated by one straight line. The four XOR corners make this visible: the two positive corners lie across from each other. Moving a line around cannot put both positive corners on one side and both negative corners on the other.
What to watch in the experiment
A hidden layer builds new numbers from the inputs before making the final decision. A nonlinear activation bends that transformation; adding more straight-line calculations alone still gives one straight-line calculation. Compare the original coordinates with the hidden-layer view to see how the problem changes.
What this does not establish
The lab illustrates a representation that separates these four points. It does not prove that adding layers always helps or that each neuron learns one understandable concept.
Open the full worked examples, calculations and limitations
This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.
A line has a limit
In the input diagram, (0,1) and (1,0) are positive. The other two corners are negative. Any single straight decision boundary leaves at least one corner on the wrong side. This is a limitation of the model family, not a lack of training time.
Two affine transformations still combine into one: if h = ax + b and y = ch + d, then y = cax + cb + d. Adding layers without nonlinearity does not remove this limit.
Two hidden features change the view
ReLU returns its input when positive, and zero otherwise. Our first hidden neuron computes ReLU(A + B). The second computes ReLU(A + B − 1). The output subtracts twice the second value from the first.
For (1,1), the hidden values are 2 and 1, so the output is 2 − 2 × 1 = 0. For (1,0), they are 1 and 0, so the output is 1. The weights in this lab are hand-chosen to expose the mechanism; no training has taken place.
A representation is useful for what comes next
A hidden layer produces a new set of features. In hidden space, a simple output rule can now distinguish the positive examples. A representation helps when relevant differences become easier for later operations to use.
Real image networks can combine edges into useful shape evidence. But a concept does not have to live in one neuron: features are often distributed and overlapping. More depth can express compositional functions efficiently, while also making optimization harder.
For A = 0 and B = 0, what are (h₁, h₂, y) with ReLU on?
Take this idea with you.
Hidden layers transform a representation. Nonlinear activations let a network express distinctions that one straight boundary cannot.
Explore related topics and the topic map
How this topic connects
Explore this topic on the interactive mapUse these connections to review a foundation or choose where to go next. You can open any topic without completing the others.
What it builds on
- Graph coordinates
Read axes, coordinates and scales so you can connect a plotted change with the numerical evidence in this lesson.
- Your first neuron
Understand how inputs become predictions so you can explain what a model’s weights and bias actually change.
Build XOR with two hidden neurons and discover what nonlinear activations add.
What it enables
- What does this neuron notice?
Inspect which input patterns change a prediction so you can question what an image model has learned.
- Give words a place in space
Compare meanings represented as numbers so you can reason about search results and the information a representation loses.
- Initialize and normalize
Diagnose unstable activations and recognize why training and evaluation modes can behave differently.