Lesson 4 of 10 in this unit
Combine many imperfect trees.
Fit bootstrap trees, inspect their votes and out-of-bag coverage, and test whether shared mistakes survive a measurement shift.
By the end: Explain bootstrap sampling and random feature selection, audit out-of-bag eligibility, and compare ensemble errors with shared individual errors.
Prerequisites and path options
Why learn this? Understand why combining trees can help and when their shared mistakes still limit the result.
Start with something concrete.
Nine people repeat one mistaken answer. Does taking a vote correct it? Duplicate a fitted tree, then compare with trees trained on different samples and questions.
Try the interactive modelYou make nine exact copies of a tree that misclassifies six validation examples. What happens to those errors under majority voting?
Change one thing. Watch what follows.
Loading this browser experiment…
Understand what changed.
A shorter explanation, with the original worked examples available below. Both explain the same experiment and its limits.
The idea in everyday terms
A tree asks a sequence of questions about the inputs. An ensemble combines several trees, often by voting. It can help when different trees make different mistakes. Duplicating the same tree repeats the same errors.
What to watch in the experiment
Compare repeated full-data trees with trees trained on bootstrap samples. A bootstrap samples training rows with replacement. An out-of-bag estimate uses only trees that did not train on the particular row being checked. Inspect how many eligible voters each row has.
What this does not establish
No eligible voters means no out-of-bag estimate. A vote fraction is not automatically a calibrated probability, and a changed sensor convention can defeat many trees together.
Open the full worked examples, calculations and limitations
This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.
Ask whether the mistakes differ
Imagine asking nine trees to classify the same example. If five vote for class 1 and four vote for class 0, a majority vote predicts 1. This can help when some trees make errors that others avoid. If all nine repeat the same mistake, voting cannot repair it.
Start with Duplicate one full-data tree. Fit nine copies, then inspect the individual votes. The ensemble’s error count equals each tree’s error count. More copies did not add information. This is a controlled counterexample to “more models must be better.”
Change the examples each tree sees
A bootstrap sample draws training rows with replacement. After drawing a row, put it back before the next draw, so it can appear again. For four original rows A, B, C and D, one four-draw sample might be B, B, D, A. B counts twice in fitting; C is absent.
The lab draws 64 entries from 64 training rows for each bagged tree. It displays the draw order and each selected row’s frequency. The sample still has 64 entries, but usually fewer than 64 distinct rows. Repeated copies are not new independent observations.
Bagging fits separate models to these resampled datasets and combines their predictions. Here each tree can consider both input features at every split. The forest can reduce sensitivity to particular training rows, but it does not fix incorrect labels or measurement conventions automatically.
Random forests vary the available questions too
A random forest also chooses a random subset of features at each split. In this two-feature lab, each split is offered either x or y. It searches that feature’s possible thresholds and selects the best training split. The next split makes a fresh feature choice. This is different from choosing one feature for the whole tree.
The tree uses class impurity to compare candidate splits. A mixed group contains both labels. A pure group contains one label. For four sampled entries with labels 0, 0, 1, 1, the positive fraction is p = 2/4 = 0.5, and the Gini impurity is 2 × 0.5 × 0.5 = 0.5. A group containing only label 1 has p = 1 and impurity 0.
Here G is binary-class Gini impurity, between 0 and 0.5; p is the fraction of sampled entries labelled 1 in the node. Both are unitless. Read the expression as “twice the positive fraction times the negative fraction.” Repeated bootstrap entries count each time they occur.
For a proposed split, the implementation minimizes nL × GL + nR × GR. nL and nR are the numbers of sampled entries in the left and right child; GL and GR are their impurities. Dividing by the parent count would give the weighted average and would select the same split. For example, two pure children of size two score 0, while two equally mixed children of size two score 2. A split sends values equal to its threshold left. Equal scores retain the first threshold encountered.
The lab caps depth at four and requires at least two sampled entries in each child. Those entries may include duplicates. If the selected feature has no usable split, the node remains a leaf. Leaf ties predict 0. These are explicit teaching choices, not a claim that every forest library uses identical defaults.
Follow the vote to the decision
B is the number of trees that are voting; k is the number predicting class 1; v is the vote fraction, between 0 and 1. Five positive votes among nine give v = 5/9 ≈ 0.556. The lab predicts 1 when k is greater than half of B; ties predict 0. A 55.6% vote is not automatically a 55.6% chance of correctness: trees share data and assumptions.
Use the fixed-scale map to find a prediction region, then select a validation example and a tree. The numerical trace shows the actual fitted questions and leaf counts. Compare the ensemble error count with the average error count of its individual voters on the same cases. Count cases where every voter is wrong and cases where they disagree. Disagreement can come from worse trees too, so diversity alone is not the objective.
Optional mathematics: why correlated errors limit averaging
For numerical estimates with equal error variance σ² and common pairwise error correlation ρ, the variance of their average is σ²[ρ + (1 − ρ)/B]. σ² measures error spread, ρ measures how strongly the errors vary together, and B is the number averaged. This formula assumes equal variances and a common correlation; it describes averaging numerical errors, not an exact formula for majority-vote classification accuracy. When ρ = 1, copying estimators gives no variance reduction. When ρ = 0, averaging B estimators divides the variance by B. The classification widget reports actual error counts rather than applying this formula to votes.
Let omitted rows check the trees that missed them
An out-of-bag prediction for a training row uses only trees whose bootstrap samples omitted that row. In the four-row example above, the tree can help evaluate C, but not A, B or D. A different tree may omit B and include C. Eligibility is checked separately for every row and tree.
Select a training row and inspect its appearances in one tree. Then compare its eligible-voter count across the ensemble. With one tree, some rows have no eligible voters: their prediction is undefined. If only 20 rows have an estimate and three are wrong, the error rate is 3/20 = 15%, not 3/64. Always report coverage alongside the error rate.
Out-of-bag evaluation is useful development evidence, but repeatedly choosing depth or seed using its results consumes that evidence. A separate final evaluation is still needed for a stronger claim. In this lab, the optional training-label flips also affect the OOB targets; the separate validation labels remain unchanged. The two estimates therefore do not measure precisely the same evaluation conditions.
Stress a shared assumption
Turn on the reversed validation sensor. It reports 1 − y where the original system reported y, while each original target label stays fixed. A reading of 0.25 becomes 0.75. This is an uncorrected measurement convention change, not a change in the meaning of the target. Compare both versions of the same cases without fitting again.
Different trees can still share the same assumption about that sensor. Increasing the tree count cannot guarantee robustness to a changed input contract. A practical response is to investigate and repair the measurement pipeline, then evaluate on representative independent evidence. These synthetic, paired development cases do not establish real deployment safety.
Make a comparison you could defend
Record the method, depth, tree count, seed, noise setting and measurement convention. Compare against one fitted tree or duplicated copies at the same depth. Explain one individual error, OOB coverage and the limits of a vote fraction. Engineering learners can independently reproduce the draws and test that in-bag trees never enter an OOB vote; decision learners can specify what evidence is needed before using the system under a changed measurement convention. These are practice prompts; reviewed implementation evidence and learner pilots remain pending.
A row appeared in trees A and C but not B or D. B predicts 1 and D predicts 0. Which out-of-bag prediction follows this lab’s tie rule?
Take this idea with you.
Resampling and random features can diversify fitted trees, but shared mistakes and shifted measurements remain. Inspect votes, out-of-bag coverage and paired errors before making a claim.
Explore related topics and the topic map
How this topic connects
Explore this topic on the interactive mapUse these connections to review a foundation or choose where to go next. You can open any topic without completing the others.
What it builds on
- Simple baselines and fitted trees
Find out whether a complicated model adds value over a cheap, understandable reference.
- Sampling and uncertainty
Judge how much confidence a small sample deserves before making a larger commitment.
Fit bootstrap trees, inspect their votes and out-of-bag coverage, and test whether shared mistakes survive a measurement shift.
What it enables
- Correct residual errors with boosting
See how successive models correct errors and why extra stages can stop helping development performance.