Measure variation rather than hiding it.

This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.

Start with the question

Three seeds use the same task data. Do they create three new tasks?

Mechanism and worked comparison

Match baseline and candidate on the same task, split, seed and budget. Their difference removes some shared difficulty: an easy task should not be paired with a different hard task merely because both occupy the same table row. For baseline errors 8 and 4 and candidate errors 5 and 5, paired improvements are 3 and −1, averaging 1. Reporting only the first task would hide a regression.

The browser has three tasks and three initialization seeds per task. Seeds vary starting coefficients while reusing the task data. Average seeds within each task before treating tasks as clusters. The illustrated cluster bootstrap draws three task means with replacement. With three available means there are 3³ = 27 ordered draws; each draw produces another mean. The displayed endpoints use the extreme rounded percentile indices of this tiny discrete distribution.

This is an exact enumeration of a very small bootstrap illustration, not a trustworthy population confidence guarantee. The tasks are deliberately authored and are not a random sample of a deployment population. More seeds do not fix missing task diversity, shared data bias or a broken metric. Inspect task-level differences, retain every run and describe the sampling assumptions before interpreting uncertainty. Repeated views of the same table do not add evidence.

Try and explain

Read all nine paired rows, then compare task means with the 27-resample interval. Change the feature and identify which task dominates the average.

Open the reproducible study project or use the implementation studio.