Lesson 6 of 13 in this unit
Build the baseline before the extension.
Replay versioned data, fitting rules, seeds and metrics before changing the model.
By the end: Replay versioned data, fitting rules, seeds and metrics before changing the model.
Prerequisites and path options
Why learn this? Use a reproducible comparison to make a bounded claim about reproduce before extending, while keeping development practice separate from independent review.
Start with something concrete.
A colleague supplies a better score but no split or seed. Can you tell whether the algorithm changed or the evaluation changed?
Try the interactive modelWhat must be fixed to replay this study?
Change one thing. Watch what follows.
Loading this browser experiment…
Understand what changed.
A shorter explanation, with the original worked examples available below. Both explain the same experiment and its limits.
Start with a concrete question
A colleague supplies a better score but no split or seed. Can you tell whether the algorithm changed or the evaluation changed?
What to inspect
Inspect training and development row IDs. Export a recipe, run the independent Python studio and compare each run before adding your own intervention.
Reproduction begins with an executable contract. Name the data generator, split rule, model, initialization, update rule, number of updates and scoring metric. This study uses three authored tasks. Row indices modulo three assign training, development and final membership; no row is assigned twice within a task. Task formulas and partitions are public, so disjoint membership is not a claim of secret or independently collected data.
Keep the conclusion bounded
The downloadable Python reference reconstructs the same rows and outputs all runs. Compare numerical values with a declared floating-point tolerance, not a screenshot. The existing dependency-free two-game starter provides a second branch with exact task contracts and programmed search; it is not a neural implementation. Keep each branch’s manifest separate. A passing replay establishes agreement with the supplied software recipe, not independent scientific validity or reviewed implementation proficiency.
Open the full worked examples, calculations and limitations
This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.
Start with the question
A colleague supplies a better score but no split or seed. Can you tell whether the algorithm changed or the evaluation changed?
Mechanism and worked comparison
Reproduction begins with an executable contract. Name the data generator, split rule, model, initialization, update rule, number of updates and scoring metric. This study uses three authored tasks. Row indices modulo three assign training, development and final membership; no row is assigned twice within a task. Task formulas and partitions are public, so disjoint membership is not a claim of secret or independently collected data.
Every fit starts from three small deterministic coefficients derived from seed 7, 19 or 41. The linear baseline keeps its quadratic coefficient at zero. A full-batch gradient uses all twelve training rows at one checkpoint, then updates active coefficients simultaneously at learning rate 0.15. Forty or eighty updates cost 480 or 960 training-example visits per model, excluding scoring and overhead. Prediction never updates weights.
The downloadable Python reference reconstructs the same rows and outputs all runs. Compare numerical values with a declared floating-point tolerance, not a screenshot. The existing dependency-free two-game starter provides a second branch with exact task contracts and programmed search; it is not a neural implementation. Keep each branch’s manifest separate. A passing replay establishes agreement with the supplied software recipe, not independent scientific validity or reviewed implementation proficiency.
Try and explain
Inspect training and development row IDs. Export a recipe, run the independent Python studio and compare each run before adding your own intervention.
Open the reproducible study project or use the implementation studio.
What must be fixed to replay this study?
Take this idea with you.
Replay versioned data, fitting rules, seeds and metrics before changing the model.
Explore related topics and the topic map
How this topic connects
Explore this topic on the interactive mapUse these connections to review a foundation or choose where to go next. You can open any topic without completing the others.
What it builds on
- Evaluation and generalization
Spot improvements caused by repeated test selection rather than better performance on new situations.
Replay versioned data, fitting rules, seeds and metrics before changing the model.
What it enables
- Reproducible research project Apply in a project
Use the research project to freeze a question, replay all task/seed comparisons and defend a result-linked report with its limitations.