Lesson 5 of 13 in this unit
Turn an idea into a falsifiable experiment.
Specify a paired baseline, one change, metric, budget and decision rule before final evaluation.
By the end: Specify a paired baseline, one change, metric, budget and decision rule before final evaluation.
Prerequisites and path options
Why learn this? Use a reproducible comparison to make a bounded claim about freeze a falsifiable question, while keeping development practice separate from independent review.
Start with something concrete.
You add a curved feature to a straight-line predictor. What result would convince you the extra feature did not help?
Try the interactive modelBaseline MAE is 0.12 and candidate MAE is 0.10. The frozen minimum useful gain is 0.03. What follows?
Change one thing. Watch what follows.
Loading this browser experiment…
Understand what changed.
A shorter explanation, with the original worked examples available below. Both explain the same experiment and its limits.
Start with a concrete question
You add a curved feature to a straight-line predictor. What result would convince you the extra feature did not help?
What to inspect
Compare the line and two curved tasks in development. In the project, write a question and minimum useful gain before freezing; inspect a task where the change loses.
A useful research question can be wrong. “Make predictions better” leaves too many escape routes. A testable question names a baseline, one intervention, an outcome and a decision rule. In this workbench the baseline has an intercept and a linear coefficient. Intervention A adds a quadratic coefficient; intervention B penalizes coefficient size. Both fit only twelve training rows per task.
Keep the conclusion bounded
Choose the update budget on public development data, record the hypothesis and threshold, and freeze before revealing the final partition. The project preserves exposure when reopening the protocol. Editing after exposure is useful development, but the fixed final rows are then reused practice. A local freeze is a reproducibility aid, not a security boundary against inspecting source code. Report a negative or inconclusive result with the same care as a positive one.
Open the full worked examples, calculations and limitations
This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.
Start with the question
You add a curved feature to a straight-line predictor. What result would convince you the extra feature did not help?
Mechanism and worked comparison
A useful research question can be wrong. “Make predictions better” leaves too many escape routes. A testable question names a baseline, one intervention, an outcome and a decision rule. In this workbench the baseline has an intercept and a linear coefficient. Intervention A adds a quadratic coefficient; intervention B penalizes coefficient size. Both fit only twelve training rows per task.
Mean absolute error, MAE, averages the absolute difference between a prediction and its target. Smaller values are better. Define improvement as baseline MAE minus candidate MAE. If these errors are 0.12 and 0.10, improvement is 0.02 in the target’s synthetic units. Choosing a minimum useful gain of 0.03 would make that positive change insufficient for your stated decision. A threshold is a decision choice, not a probability.
Choose the update budget on public development data, record the hypothesis and threshold, and freeze before revealing the final partition. The project preserves exposure when reopening the protocol. Editing after exposure is useful development, but the fixed final rows are then reused practice. A local freeze is a reproducibility aid, not a security boundary against inspecting source code. Report a negative or inconclusive result with the same care as a positive one.
Try and explain
Compare the line and two curved tasks in development. In the project, write a question and minimum useful gain before freezing; inspect a task where the change loses.
Open the reproducible study project or use the implementation studio.
Baseline MAE is 0.12 and candidate MAE is 0.10. The frozen minimum useful gain is 0.03. What follows?
Take this idea with you.
Specify a paired baseline, one change, metric, budget and decision rule before final evaluation.
Explore related topics and the topic map
How this topic connects
Explore this topic on the interactive mapUse these connections to review a foundation or choose where to go next. You can open any topic without completing the others.
What it builds on
- Evaluation and generalization
Spot improvements caused by repeated test selection rather than better performance on new situations.
Specify a paired baseline, one change, metric, budget and decision rule before final evaluation.
What it enables
- Reproducible research project Apply in a project
Use the research project to freeze a question, replay all task/seed comparisons and defend a result-linked report with its limitations.