Search enough outcomes and find a surprise
How does an outcome search change the chance of a false alarm?
Entering midway? Revisit Choose the test that could prove you wrong for the preceding worked example, or start here and use the explanation below.
Start with a concrete question
Original synthetic scenario. Every candidate effect is absent. Independent tests nevertheless sometimes flag an effect, at a probability you choose.
Try the initial inputs, then write a prediction before changing one setting. All calculations run locally on your CPU.
What do you expect?
Will ten tests with a 5% false-alarm rate each give only a 5% chance of any false alarm?
Change one thing. Inspect what happens.
- Compare one test and ten tests with the same 5% per-test rate.
- Divide the error budget across ten tests; compare per-test and family probabilities.
- Set twenty tests and explain why the expected count differs from the probability of at least one.
- Write a primary-outcome rule before inspecting results and distinguish confirmatory tests from exploratory findings.
Your settingsMechanism
Calculation belowEvidence
Compare outcomes
Compare the current results
Every bar starts at zero and uses the same numeric scale in this chart. The upper bound automatically adjusts to the largest current value (at least 1), so compare the numbers when changing inputs. Units and quantities come from each label; different kinds of quantity are not interchangeable. Displayed values are rounded to at most six significant digits.Follow the calculation
- All 10 proposed effects are absent in this constructed null scenario; each test independently falsely flags with probability 0.050.
- One test avoids a false positive with probability 1 − 0.050 = 0.950.
- All avoid one: 0.950^10 = 0.599.
- At least one = 1 − 0.599 = 0.401.
- The selected error rate applies per test, not to the whole search.
Searching more outcomes creates more opportunities for false alarms. Correction limits false alarms but this null-only model cannot measure missed real effects.
Captures keep your inputs, prediction and observation. Results are recalculated from those inputs. Resetting inputs keeps your captures and challenge history.
Explain the mechanism
What the experiment explains
Original synthetic scenario. Every candidate effect is absent. Independent tests nevertheless sometimes flag an effect, at a probability you choose.
A false positive flags an effect that is absent. The probability of avoiding one false positive is 1 − p; under independence, avoiding all k is (1 − p)^k.
At ten independent tests and p=0.05, the chance of at least one false positive is 1 − 0.95^10 ≈ 40.1%. The expected number is 10 × 0.05 = 0.5; these are different quantities.
Dividing a family budget by the number of tests is a Bonferroni allocation. The exact family calculation here assumes independence; the Bonferroni upper bound itself does not require it.
Read the relationship
P(any false positive) = 1 − (1 − p)^k; expected count = kp. p is a probability from 0 to 1; k is the test count.
Inspect the evidence
The numerical trace is recalculated from your current settings. Compare a controlled pair of trials before changing several inputs together. Inspect intermediate quantities as well as final metrics; state which assumptions make the comparison meaningful.
Boundaries of the model
- Independent null tests only; no power, real effects, or correlated outcomes.
- Changing thresholds after seeing results is not modeled.
Return to the scientific reasoning and experimental design course and project to connect this mechanism to the broader investigation.
Try a fresh case
Three independent null tests each have a 10% false-positive probability. What percentage is the chance of at least one false positive?
This is a public practice check. Retries support learning; matching a number does not certify independent proficiency.
Take the investigation further
Choose any extension. These are project prompts for you to carry out; they are not automatically assessed.
School investigation
Explain why reporting only the most surprising result can mislead.
College investigation
Compare Bonferroni and an independently implemented false-discovery-rate procedure on simulated null and non-null outcomes.
Workplace investigation
Write an experiment analysis plan with a primary metric, guardrails, and a policy for exploratory discoveries.
Keep the evidence, choose your next step
Completion records your own decision that you have worked through this investigation. Practice checks and independent project evidence are separate.