Develop a margin from repeated samples

Try

A survey proportion varies from sample to sample. To investigate that variation, first use a simulated population whose success probability is known: p=0.4. In a sample of n independent draws, the estimate is p̂=k/n, where k is the number of successes. Its absolute error is the nonnegative distance ∣p̂−p∣. Because the simulated truth is known, these errors can be inspected directly.

A margin can be chosen to include a specified fraction of simulated errors. That is a calibration procedure, not an automatic confidence guarantee for every real survey. We will develop the margin on one set of draws and evaluate it on another.

Predict

Predict how error size might change when n increases. Distinguish increasing the survey sample size n from increasing the number of simulated repetitions: the first changes the estimator’s sampling variability, while the second helps estimate that variability more precisely. Predict why a biased sample may become more consistently wrong as its size grows.

Experiment

Calibrate a margin, then test it on another run

This synthetic population has known proportion 0.4. Each sample contains independent draws with replacement. Training uses the whole population. Before running, choose the target fraction of absolute errors to cover. This is a model investigation, not a confidence guarantee for an unknown real population.

Draws per sample5102040 Trials per run100200 Training error quantile80%90%95% Training seed Calibrate margin Reset

Runs are temporary. Record settings, margin, traces and limitations in your written response. Work is bounded at 40 draws × 200 samples per run; changing settings hides mismatched results. No completion or mastery is awarded here.

Choose n, target fraction and calibration repetitions before generating the trace. Record the seed and inspect the cumulative error-frequency table. Identify the nearest-rank threshold yourself, then compare it with the selected margin. Save at least three rows with success count, sample proportion and absolute error.

Evaluate that fixed margin using the separate evaluation seed and record the coverage fraction. Do not recalibrate on the evaluation draws to improve the result. Compare with the exact-model coverage shown by the explorer, which sums all possible binomial outcomes rather than simulating them. Finally evaluate a selected subgroup with success probability 0.7 while retaining population target 0.4. That deliberately breaks the original sampling-frame assumption.

Understand

If N errors are sorted from smallest to largest, a nearest-rank q-quantile uses position ceil(qN), counting from one. For N=200 and q=0.90, select error number 180. At least 90% of training errors are then at or below the selected margin; tied discrete errors can raise that fraction. The calibration data therefore cannot independently confirm future coverage.

Here is a separate small illustrative calibration, not the explorer’s seeded output: ten sorted errors are 0,0,0,0.2,0.2,0.2,0.2,0.4,0.4,0.6. A 90% nearest-rank margin is the ninth value, 0.4, and exactly nine of these ten errors are covered. A new set of errors may have a different covered fraction. Never invent the actual widget trace from this toy list.

For n=5 and p=0.4 the possible proportions and errors are fully inspectable:

Success count k	Sample proportion k/5	Absolute error from 0.4
0	0	0.4
1	0.2	0.2
2	0.4	0
3	0.6	0.2
4	0.8	0.4
5	1	0.6

A margin 0.2 covers counts 1,2,3. Their binomial probabilities are 0.2592,0.3456,0.2304, summing to 0.8352. A margin 0.4 covers every count except 5, giving 1−0.4⁵=0.98976. This jump illustrates why discrete attainable margins may not achieve an exact requested percentage. Exact coverage is a model calculation; the simulated evaluation fraction fluctuates around it.

To connect a margin m to an observed estimate p̂, form p̂±m. If appropriate, intersect with [0,1] and state the clipping. The interval varies across samples while the target population proportion is fixed. A margin calibrated at p=0.4 describes that parameter and sampling model; it is not a general proof of coverage at every unknown p. Established interval methods in the inference course address broader conditions.

The selected-subgroup mode draws from 0.7 but measures error against 0.4. Its bias is 0.3. More observations concentrate estimates near the wrong target, while population-calibrated margins tend to shrink. More Monte Carlo repetitions can measure that failure more precisely; they cannot repair the selection mechanism.

Apply

Produce a calibration/evaluation report containing seeds, n, repetition counts, target fraction, selected rank, error table, margin, separate evaluation coverage and an illustrative survey interval. Explain simulation randomness, real sampling variability, population-parameter assumptions and selection bias separately. Retain duplicate simulated outcomes: repeated counts are expected and contribute to frequencies.

Continue

Next simulate randomized treatment assignments. Sampling selects observations from a population; random assignment redistributes treatment labels among fixed participating units. The correct simulation must reproduce the mechanism relevant to the question.